A pose measurement method of a soft manipulator based on a neural network

Abstract

This invention provides a method for measuring the pose of a soft manipulator based on a neural network, comprising: S1, setting marker points on the outer surface of the soft manipulator to control its bending deformation; wherein the soft manipulator has a first cavity, a second cavity, and a third cavity inside, and its outer wall has an intermittently distributed concave-convex structure; and applying air pressure to the first cavity, the second cavity, and the third cavity to control the bending deformation of the soft manipulator; S2, acquiring images captured by a camera; S3, extracting the bending deformation image of the soft manipulator, wherein a recognition neural network is established to extract the bending deformation image of the soft manipulator from the images captured by the camera; and S4, measuring the bending deformation image of the soft manipulator using a measurement neural network to output the pose information of the bending deformation of the soft manipulator. This invention utilizes two independent neural networks to perform "recognition" and "measurement" of the soft manipulator respectively, completing the pose measurement of the soft manipulator based on a visual sensing method.

Description

Technical Field

[0001] This invention relates to the field of soft manipulator pose measurement technology, and in particular to a method for measuring the pose of a soft manipulator based on a neural network. Background Technology

[0002] Compared to traditional rigid manipulators, soft manipulators are made of flexible materials, possess multiple degrees of freedom, are inherently safe and less prone to damage during human-computer interaction, and can deform continuously and flexibly to adapt to various material structures. They exhibit high flexibility, adaptability, and safety, especially in confined and variable spaces. Therefore, soft manipulators are gaining increasing attention and are widely used in surgical, exploration, and transportation tasks. However, due to their theoretically unlimited degrees of freedom and deformation methods, the pose measurement accuracy of soft manipulators cannot yet reach the level of rigid manipulators. Furthermore, traditional sensors can affect the soft structure, limiting the movement of the manipulator itself, thus restricting its further widespread adoption to some extent.

[0003] Machine vision-based pose measurement technology, with its simple structure and ability to be integrated with existing endoscopic techniques, has become one of the most feasible methods and has therefore received extensive research. Currently mature machine vision pose measurement methods rely on feature point recognition, depth information, and model-image matching, but software actuators theoretically possess infinite degrees of freedom in shape transformation, making these methods difficult to apply directly. However, emerging neural network technology possesses high data processing and fitting capabilities, enabling intelligent processing based on machine vision. Summary of the Invention

[0004] To address the technical problem of low pose measurement accuracy in existing soft manipulators, one objective of this invention is to provide a pose measurement method for a soft manipulator based on a neural network, the method comprising:

[0005] S1. Marking points are set on the outer side of the soft operating arm to control the bending deformation of the soft operating arm. The soft operating arm has a first cavity, a second cavity and a third cavity inside, and the outer wall is distributed with concave and convex structures at intervals. The bending deformation of the soft operating arm is controlled by applying air pressure to the first cavity, the second cavity and the third cavity.

[0006] S2. Acquire images captured by the camera;

[0007] S3. Extract the bending deformation image of the soft manipulator, wherein the bending deformation image of the soft manipulator is extracted from the image captured by the camera by establishing a recognition neural network;

[0008] S4. Measure the bending deformation image of the soft manipulator by measuring the neural network, and output the pose information of the bending deformation of the soft manipulator.

[0009] Preferably, in step S3, the image captured by the camera is cropped by a recognition neural network to extract the bending deformation image of the soft manipulator, and the extracted bending deformation image of the soft manipulator is used to construct a sample set.

[0010] Preferably, in step S4, marker points are extracted from the bending deformation image of the soft manipulator to construct a piecewise DH matrix of the soft manipulator bending.

[0011] The segmented DH matrix is ​​used as input information to the measurement neural network, and the pose information of the bending deformation of the soft manipulator is output.

[0012] Preferably, in step S4, a residual network ResNet is introduced into the measurement neural network.

[0013] This invention provides a method for measuring the pose of a soft manipulator based on a neural network. The method extracts the bending deformation image of the soft manipulator by establishing a recognition neural network, and maps the bending deformation image of the soft manipulator to the pose information of the soft manipulator by establishing a measurement neural network. The method uses two independent neural networks to "recognize" and "measure" the soft manipulator respectively, thus completing the pose measurement of the soft manipulator based on a visual sensing method.

[0014] This invention provides a pose measurement method for a soft manipulator based on a neural network. The visual sensing method used is a non-contact sensing approach, which does not affect the deformation of the soft manipulator. The pose measurement method based on visual sensing has good scalability and can be easily promoted in similar application scenarios. With the help of the neural network, the visual sensing method achieves greatly enhanced anti-interference capabilities, improving the stability of the entire system. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a pose measurement method for a neural network-based soft manipulator is shown.

[0017] Figure 2 A schematic diagram of the structure of the soft manipulator of the present invention is shown.

[0018] Figure 3 A top view of the soft operating arm of the present invention is shown.

[0019] Figure 4 The image shown is an image of the bending deformation of the soft manipulator extracted by the present invention.

[0020] Figure 5 A schematic diagram of the measurement neural network structure of the ResNet residual network introduced in this invention is shown.

[0021] Figure 6 This invention illustrates the relationship between the pose error of a soft manipulator measured by a neural network and changes in air pressure in one embodiment of the invention. Detailed Implementation

[0022] To make the above and other features and advantages of the present invention clearer, the invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explanation to those skilled in the art and are exemplary only, not restrictive.

[0023] like Figure 1 The flowchart shown illustrates a pose measurement method for a soft manipulator based on a neural network. According to an embodiment of the present invention, a pose measurement method for a soft manipulator based on a neural network is provided, comprising:

[0024] Step S1: Control the bending and deformation of the software operating arm.

[0025] like Figure 2 The diagram shown is a structural schematic of the soft manipulator of the present invention. Figure 3 The diagram shows a top view of the soft operating arm of the present invention. A first cavity 102, a second cavity 103, and a third cavity 104 are formed inside the soft operating arm 100. The outer wall of the soft operating arm 100 is provided with intermittently distributed concave and convex structures 101. Air pressure is applied to the first cavity 102, the second cavity 103, and the third cavity 104 to control the bending deformation of the soft operating arm 100.

[0026] The intermittently distributed concave-convex structures 101 on the outer side wall of the soft manipulator 100 give it stronger resistance to expansion in the radial direction and stronger extension capacity in the longitudinal direction. In one embodiment, the soft manipulator 100 has a length of 108 mm and a maximum outer diameter of 26 mm.

[0027] Marking points are set on the outer surface of the soft manipulator 100 to control the bending deformation of the soft manipulator 100. Specifically, the bending deformation of the soft manipulator 100 is controlled by applying air pressure to the first cavity 102, the second cavity 103 and the third cavity 104.

[0028] Step S2: Acquire images captured by the camera.

[0029] During the process of controlling the bending and deformation of the soft manipulator 100, images of the bending and deformation process of the soft manipulator 100 are captured by a camera from different angles.

[0030] Step S3: Extract the bending deformation image of the soft manipulator, wherein the bending deformation image of the soft manipulator is extracted from the image captured by the camera by establishing a recognition neural network.

[0031] Images captured by a camera from different angles during the bending and deformation process of the soft manipulator 100 contain other images (redundant information) unrelated to the soft manipulator 100. This invention extracts the bending and deformation images of the soft manipulator 100 from the images captured by the camera by establishing a recognition neural network.

[0032] Specifically, the bending and deformation images of the soft manipulator 100 in the images captured by the camera are marked, the images captured by the camera are cropped by a recognition neural network, and images unrelated to the bending and deformation images of the soft manipulator 100 are deleted, thus extracting the bending and deformation images of the soft manipulator 100.

[0033] A sample set was constructed using the extracted bending deformation images of the soft manipulator 100, such as... Figure 4 The image shown is an image of the bending deformation of the soft manipulator extracted by this invention.

[0034] This invention, by establishing a recognition neural network, not only removes redundant information in images and improves measurement accuracy, but also divides an entire end-to-end neural network task into two consecutive subtasks (recognition neural network and measurement neural network), reducing task complexity and lowering the requirements of the subsequent measurement neural network on the amount of training data.

[0035] Step S4: Measure the bending deformation image of the soft manipulator through a measurement neural network, and output the pose information of the bending deformation of the soft manipulator.

[0036] In step S3, the bending deformation images of the soft manipulator 100 extracted are used to construct a sample set. Marker points set on the outer surface of the soft manipulator 100 are collected. The collected marker points are used to construct a segmented DH matrix of the soft manipulator 100. The segmented DH matrix is ​​used as the input information of the measurement neural network to output the pose information of the bending deformation of the soft manipulator 100.

[0037] This invention establishes a measurement neural network, takes a segmented DH matrix constructed from marker points set on the outer surface of the soft manipulator 100 as input, and outputs a column vector representing the pose information of the soft manipulator 100, thereby realizing the mapping from the image of the soft manipulator 100 to the pose information of the soft manipulator 100.

[0038] This invention measures the super-strong fitting ability of neural networks to data and the ability to extract high-level information from low-level data by abstracting it into high-level features, enabling the soft manipulator 100 to play a unique advantage in attitude recognition and estimation.

[0039] In one embodiment, since the pose estimation of the soft manipulator 100 is a regression problem, the pose model of the soft manipulator 100 has 90 parameters. Therefore, the measurement neural network needs to extract a 90-dimensional feature vector (column vector of pose information), which places requirements on the depth of the measurement neural network.

[0040] To avoid problems such as gradient vanishing, gradient explosion, and network degradation caused by increasing the depth of the measurement neural network, a residual network ResNet is introduced into the measurement neural network.

[0041] like Figure 5 The diagram shows the structure of the measurement neural network introduced in this invention, which incorporates the ResNet residual network. A regularization layer is added to the measurement neural network to standardize features and optimize their distribution, thus accelerating network convergence and addressing gradient vanishing and gradient explosion issues as depth increases. Furthermore, the data layers in the measurement neural network are connected in an alternate-layer manner, weakening the connections between adjacent layers and mitigating network degradation.

[0042] In an embodiment of the present invention, a sample set is constructed by extracting 2338 bending deformation images of the soft manipulator 100, and the marker points set on the outer surface of the soft manipulator 100 in 70% of the bending deformation images of the soft manipulator 100 are randomly selected to construct a segmented DH matrix of the soft manipulator 100 as a training set.

[0043] The remaining 30% of the soft manipulator 100 bending deformation images are used to set marker points on the outer surface of the soft manipulator 100, and a segmented DH matrix of the soft manipulator 100 is constructed as a test set.

[0044] By comparing the test results (the pose error of the soft manipulator measured by the measurement neural network) on the test set with changes in air pressure, the relationship between the pose error of the soft manipulator measured by the measurement neural network and changes in air pressure is plotted. For example... Figure 6 The diagram illustrates the relationship between the pose error of a soft manipulator measured by a neural network and changes in air pressure in one embodiment of the present invention.

[0045] As can be seen, as the air pressure applied to the first cavity 102, the second cavity 103 and the third cavity 104 increases, the measurement error obtained by the measurement neural network increases accordingly. It is speculated that this is due to the irreversible deformation of the soft manipulator 100 material caused by multiple high-intensity deformations.

[0046] The average error measured by the measurement neural network under extreme air pressure was 6.5%, indicating good accuracy of the measurement neural network. At the same time, as the air pressure approached the limit, the change in measurement error tended to be gradual, because the deformation of the soft manipulator 100 had reached its limit at this time.

[0047] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A pose measurement method of a soft robot arm based on a neural network, characterized by, The measurement method comprises: S1, a mark point is arranged outside the soft operating arm, and the soft operating arm is controlled to bend and deform, wherein a first cavity, a second cavity and a third cavity are arranged inside the soft operating arm, and concave-convex structures are arranged on the outer side wall in a spaced manner, and the soft operating arm is controlled to bend and deform by applying air pressure in the first cavity, the second cavity and the third cavity; S2, an image captured by a camera is acquired; S3, a bending and deforming image of the soft operating arm is extracted, wherein the bending and deforming image of the soft operating arm is extracted from the image captured by the camera by establishing an identification neural network; S4, a measurement neural network is used to measure the bending and deforming image of the soft operating arm, and pose information of the bending and deforming of the soft operating arm is outputted; In step S3, the image captured by the camera is cropped by the identification neural network, the bending and deforming image of the soft operating arm is extracted, and the extracted bending and deforming image of the soft operating arm is constructed into a sample set; In step S4, mark points in the bending and deforming image of the soft operating arm are extracted, and a segmented D-H matrix of the soft operating arm is constructed, The segmented D-H matrix is taken as input information of the measurement neural network, and the pose information of the bending and deforming of the soft operating arm is outputted.

2. The pose measurement method according to claim 1, characterized in that, In step S4, a residual network ResNet is introduced into the measurement neural network.

Images