A method, device and apparatus for sensing flexible cable status with robust occlusion

Abstract

The embodiment of the present invention provides an occlusion-robust flexible cable state perception method, device and equipment, the method comprising: obtaining a three-dimensional point cloud of a flexible cable; extracting features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features; using an end-to-end regression branch network to map the point cloud depth features to a first predicted three-dimensional position of M key nodes evenly distributed on the flexible cable; using a point-to-point voting branch network to map the point cloud depth features to a second predicted three-dimensional position of M key nodes of the flexible cable; based on a non-rigid point cloud registration network, the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network are fused to obtain a flexible cable state perception result. The embodiment of the present invention can realize accurate state perception of complex shapes of flexible cables in an obstructed environment, and can be widely used for flexible cables of different lengths, thicknesses and materials.

Description

Technical Field

[0001] The present invention relates to the technical field of flexible cable identification, and in particular to an occlusion-robust flexible cable state sensing method, device and equipment. Background Art

[0002] Flexible cables are widely used in human production and life, and there is a large demand for the operation of flexible cables in industrial manufacturing, medical health, home services and other scenarios. Through the automation and autonomous operation of flexible cables, a large number of manpower can be liberated from complicated and hard repetitive labor, and human daily life can be made more convenient.

[0003] Accurate state perception of flexible cables is a necessary prerequisite for planning and operating flexible cables. Unlike rigid objects, whose states can be represented by simple six-dimensional poses and have been widely studied, the complex deformation of flexible cables makes them easily occluded by other objects in unstructured scenes or self-occluded, which poses a great challenge to the state estimation of the occluded parts. At the same time, there are no obvious features on the surface of flexible cables, making it difficult to distinguish different local areas in the flexible cables. Flexible cables are often thin, making it impossible for depth cameras with limited accuracy to accurately perceive the depth information of the cable surface, and the acquired cable depth images contain a lot of noise.

[0004] Existing flexible cable perception methods cannot fully and reliably handle the above challenges and achieve accurate and occlusion-robust state perception of cables in unstructured scenes. There are problems such as the need for manual annotation, serious cumulative errors, and poor perception results under occlusion. Summary of the invention

[0005] In view of the above problems, embodiments of the present invention provide a method, apparatus and device for sensing the state of a flexible cable with robust shielding, so as to overcome the above problems or at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present invention, a flexible cable state sensing method with robust occlusion is disclosed, the method comprising:

[0007] Acquire a three-dimensional point cloud of the flexible cable, wherein the three-dimensional point cloud includes N input points;

[0008] Extracting features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features;

[0009] Using an end-to-end regression branch network, the point cloud depth feature is mapped into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable, and a prediction result of the end-to-end regression branch network is obtained;

[0010] Mapping the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable using a point-to-point voting branch network to obtain a prediction result of the point-to-point voting branch network;

[0011] Based on the non-rigid point cloud registration network, the prediction result of the end-to-end regression branch network and the prediction result of the point-to-point voting branch network are fused to obtain the flexible cable state perception result.

[0012] Optionally, extracting features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features includes:

[0013] The three-dimensional point cloud of the flexible cable is sampled based on the farthest point sampling method, multiple center points are selected from N input points in the three-dimensional point cloud, multiple neighboring points are found as regions in the neighborhood of each center point, and feature extraction is performed in the region to obtain the depth feature of each center point;

[0014] The depth feature of each center point is transferred to the input point in the three-dimensional point cloud, and the depth feature of each input point is obtained by weighted interpolation as the depth feature of the flexible cable point cloud.

[0015] Optionally, mapping the point cloud depth features into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable using an end-to-end regression branch network includes:

[0016] Performing a maximum pooling operation on the point cloud depth feature in the dimension of the number of points to obtain a global shape feature;

[0017] The global shape feature is mapped into first predicted three-dimensional positions of M key nodes on the flexible cable through a fully connected network.

[0018] Optionally, mapping the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable using a point-to-point voting branch network includes:

[0019] The feature of each input point is mapped into a thermal value and a relative displacement offset from the input point to the key node to be predicted through a fully connected network, wherein the thermal value represents the relative distance from the input point to the key node to be predicted, and the relative displacement offset represents the direction from the input point to the key node to be predicted;

[0020] Calculating a three-dimensional absolute offset from each input point to each key node based on the thermal value and the relative displacement offset;

[0021] A weighted vote is performed on the three-dimensional absolute offset of each input point to obtain a second predicted three-dimensional position of each key node.

[0022] Optionally, the non-rigid point cloud registration network is based on fusing the prediction result of the end-to-end regression branch network and the prediction result of the point-to-point voting branch network to obtain a flexible cable state perception result, including:

[0023] Calculate the probability that the M key nodes are not blocked respectively to determine whether the key node is blocked;

[0024] The unobstructed key nodes are retained, and the obstructed key nodes are removed from the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network;

[0025] The prediction results of the end-to-end regression branch network after removing some key nodes are aligned with the prediction results of the point-to-point voting branch network after removing some nodes, and the global deformation field is calculated;

[0026] A spatial non-rigid transformation is performed on the prediction result of the end-to-end regression branch network according to the global deformation field to obtain a flexible cable state perception result.

[0027] Optionally, respectively calculating the probabilities that the M key nodes are not blocked to determine whether the key node is blocked includes:

[0028] For each key node, the maximum thermal value of all input points in the key node is used as the probability that the key node is not blocked;

[0029] When the probability that the key node is not blocked is greater than or equal to a preset threshold, determining that the key node is not blocked;

[0030] When the probability that the key node is not blocked is less than a preset threshold, it is determined that the key node is blocked.

[0031] Optionally, the prediction result of the end-to-end regression branch network after removing some key nodes is aligned with the prediction result of the point-to-point voting branch network after removing some nodes, and the global deformation field is calculated, including:

[0032] Constructing a Gaussian radial basis network according to the prediction results of the end-to-end regression branch network after removing some key nodes;

[0033] The weight matrix and variance size of the Gaussian radial basis network are iteratively optimized so that the prediction results of the end-to-end regression branch network after removing some key nodes are continuously aligned and close to the prediction results of the point-to-point voting branch network after removing some nodes, thereby obtaining the global deformation field.

[0034] Optionally, the method further comprises:

[0035] Constructing a dual-branch neural network model, the dual-branch neural network model includes: a point cloud feature extraction network, an end-to-end regression branch network, a point-to-point voting branch network and a non-rigid point cloud registration network;

[0036] A training data set is constructed, wherein the data in the training data set is the flexible cable motion data collected in the simulator.

[0037] Optionally, the flexible cable motion data collected in the simulator includes:

[0038] A plurality of flexible cables of different lengths, thicknesses and rigidities are arranged in the simulator, and both ends of the flexible cables are fixedly clamped by a mechanical arm in the simulator, and the mechanical arm moves freely in the workspace;

[0039] Recording the color image and depth image of the flexible cable at each moment in the simulator, as well as the three-dimensional position of the key nodes of the flexible cable at the current moment;

[0040] The depth image of the area where the flexible cable is located is back-projected into three-dimensional space to obtain a three-dimensional point cloud of the flexible cable, and the true value of the thermal value and the true value of the relative offset from each point in the three-dimensional point cloud to the key node are calculated. The three-dimensional point cloud of the flexible cable, the three-dimensional position of the key node recorded at the corresponding time, the true value of the thermal value and the true value of the relative offset constitute a set of training data.

[0041] Optionally, the dual-branch neural network model is trained in the following manner:

[0042] Inputting the data in the training set into the two-branch neural network model;

[0043] For each set of training data, the dual-branch neural network model calculates the three-dimensional position of the key node predicted by the end-to-end regression branch network, the predicted value of the thermal value from each input point to each key node in the point voting branch network, and the predicted value of the relative offset according to the three-dimensional point cloud of the flexible cable in the training data;

[0044] Calculate the loss function according to the true value of the three-dimensional position of the key node, the true value of the thermal value of each key node and the true value of the relative offset, the three-dimensional position of the key node predicted by the end regression branch network, and the predicted value of the thermal value of each key node to be considered and the predicted value of the relative offset;

[0045] According to the loss function, optimizing the parameters of the dual-branch neural network model based on a gradient descent algorithm;

[0046] After the training is completed, a trained dual-branch neural network model is obtained.

[0047] A second aspect of an embodiment of the present invention discloses an occlusion-robust flexible cable state sensing device, the device comprising:

[0048] A point cloud acquisition module, used to acquire a three-dimensional point cloud of the flexible cable, wherein the three-dimensional point cloud includes N input points;

[0049] A feature extraction module, used to extract features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features;

[0050] An end-to-end regression module, used to map the point cloud depth features into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable using an end-to-end regression branch network, and obtain a prediction result of the end-to-end regression branch network;

[0051] A point-to-point voting module, used to map the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable using a point-to-point voting branch network, and obtain a prediction result of the point-to-point voting branch network;

[0052] The result fusion module is used to fuse the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network based on the non-rigid point cloud registration network to obtain the flexible cable state perception result.

[0053] According to a third aspect of an embodiment of the present invention, an electronic device is disclosed, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the occlusion-robust flexible cable state perception method as described in the first aspect of an embodiment of the present invention.

[0054] The embodiments of the present invention include the following advantages:

[0055] In an embodiment of the present invention, a flexible cable state perception method with robust occlusion is provided. First, the point cloud depth features are extracted from the three-dimensional point cloud of the flexible cable to effectively encode the overall shape and local spatial information of the flexible cable; then, the point cloud depth features are mapped to the first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable through an end-to-end regression branch network, so as to effectively reconstruct the shape of the occluded part of the cable; the point cloud depth features are mapped to the second predicted three-dimensional positions of M key nodes of the flexible cable through a point-to-point voting branch network, so as to effectively predict the positions of the key nodes of the local cable, especially for the key nodes outside the occluded part, and the prediction performance is good; then, the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network are fused to obtain an accurate and occlusion-robust state perception result. Since the flexible cable state perception method provided by this embodiment combines the advantages of the end-to-end regression network and the point-to-point voting network, it can pay attention to the global shape features and local features of the flexible cable at the same time, and thus can accurately perceive the flexible cable completely and reliably, and can still accurately perceive the flexible cable in the case of occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0057] Figure 1 It is a flowchart of the steps of a flexible cable state perception method with robust occlusion provided by an embodiment of the present invention;

[0058] Figure 2 It is a schematic diagram of the overall structure of an occlusion-robust flexible cable state sensing method provided by an embodiment of the present invention;

[0059] Figure 3 It is a schematic diagram of a method for processing point cloud depth features by a point voting branch network provided by an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of a non-rigid registration network processing method provided by an embodiment of the present invention;

[0061] Figure 5 It is a structural schematic diagram of an occlusion-robust flexible cable state sensing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The embodiment of the present invention provides a flexible cable state sensing method with robust occlusion. Figure 1 As shown, Figure 1 A flowchart of a method for sensing the state of a flexible cable with robust shielding provided by an embodiment of the present invention includes steps S101 to S105:

[0064] Step S101: Acquire a three-dimensional point cloud of a flexible cable, wherein the three-dimensional point cloud includes N input points.

[0065] In this embodiment, the three-dimensional point cloud of the flexible cable is a series of unordered points in the three-dimensional space to characterize the state of the flexible cable in the actual environment, and the three-dimensional point cloud of the flexible cable can be directly collected by the sensor. Specifically, for the flexible cable to be sensed, the method for obtaining the three-dimensional point cloud is: obtaining the color image and depth image of the flexible cable (expressing the distance between the object and the camera in grayscale), identifying the flexible cable area in the color image based on color segmentation or other segmentation algorithms, and back-projecting the depth image information in the area into the three-dimensional space, thereby obtaining the three-dimensional point cloud X of the flexible cable, the dimension of the three-dimensional point cloud is N×3, where N is the number of points in the three-dimensional point cloud.

[0066] In practical applications, a depth camera can be used to simultaneously capture the color image and depth image of the flexible cable. After obtaining the color image and depth image, the area where the flexible cable is located is separated from the color image, and the cable depth information is back-projected into three-dimensional space to obtain a three-dimensional point cloud. The three-dimensional point cloud of the flexible cable is shown in Figure 2. Figure 2 As shown in the input point cloud in , in the subsequent steps, processing is performed based on the acquired three-dimensional point cloud to obtain the flexible cable state perception result.

[0067] Step S102: extracting features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features.

[0068] In this embodiment, the point cloud feature extraction network is a pre-constructed network, and the point cloud feature extraction network can continuously stack feature extraction layers and feature propagation layers to construct a neural network according to the PointNet++ network rules. Among them, the feature extraction layer includes the steps of farthest point sampling, neighboring area recognition, feature extraction within the area, etc., which reduces the dimensionality of the original input three-dimensional point cloud of the flexible cable into the features of several center points; wherein, the feature propagation layer can adopt a cross-layer connection method to restore the depth features of each input point from the features of several center points. After feature extraction of the three-dimensional point cloud, the depth feature F(X) dimension output by the point cloud feature extraction network is N×C, where C is the feature dimension of each point.

[0069] In an optional embodiment, extracting features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features includes:

[0070] The three-dimensional point cloud of the flexible cable is sampled based on the farthest point sampling method, multiple center points are selected from N input points in the three-dimensional point cloud, multiple neighboring points are found as regions in the neighborhood of each center point, and feature extraction is performed in the region to obtain the depth feature of each center point;

[0071] The depth feature of each center point is transferred to the input point in the three-dimensional point cloud, and the depth feature of each input point is obtained by weighted interpolation as the depth feature of the flexible cable point cloud.

[0072] In this embodiment, since the points in the three-dimensional point cloud of the flexible cable are disordered, directly extracting features from the three-dimensional point cloud requires a high computational cost. Therefore, by downsampling the three-dimensional point cloud, the operations on the entire point cloud are converted to the sampling points obtained by downsampling, thereby achieving the purpose of reducing the amount of calculation.

[0073] In this embodiment, the farthest point sampling method is used to downsample the three-dimensional point cloud of the flexible cable to obtain multiple sampling points (i.e., multiple center points are selected), and then multiple neighboring points are found as regions based on the adjacent domain of each center point. Finally, the points in the region are feature extracted to obtain the depth features of each center point. After obtaining the depth features of the center point, the depth features of the center point are calculated according to weighted interpolation to obtain the depth features of each input point in the center point region. For example, assuming that there are 100 input points in the three-dimensional point cloud, 10 center points are selected, and then feature extraction is performed on 10 regions to obtain the corresponding depth features of the 10 center points. Compared with not downsampling and directly extracting features from 100 points, the method provided in this embodiment greatly reduces the amount of feature extraction calculations.

[0074] like Figure 2As shown, after obtaining the point cloud depth features of the flexible cable, the point cloud depth features are simultaneously input into the end-to-end regression branch network and the point-to-point voting branch network for processing, that is, step S103 and step S104 are processed to predict the three-dimensional positions of the key nodes of the flexible cable in different ways.

[0075] Step S103: using an end-to-end regression branch network to map the point cloud depth features into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable, and obtaining a prediction result of the end-to-end regression branch network.

[0076] In this embodiment, the end-to-end regression branch network can focus on the global characteristics of the flexible network. Therefore, especially for the obscured flexible cable, the end-to-end regression branch network can effectively reconstruct the shape of the obscured part of the cable. The end-to-end regression branch network uses the three-dimensional positions of M key nodes that are orderly and evenly distributed on the flexible cable as the state representation of the cable (i.e., the first predicted three-dimensional position), where the number of key nodes M is a preset constant that is large enough to ensure that there are enough key nodes to represent the cable shape, and the state dimension to be predicted is M×3.

[0077] In an optional embodiment, mapping the point cloud depth features into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable using an end-to-end regression branch network includes:

[0078] A maximum pooling operation is performed on the point cloud depth feature in the dimension of the number of points to obtain a global shape feature; and the global shape feature is mapped into a first predicted three-dimensional position of M key nodes on the flexible cable through a fully connected network.

[0079] In this embodiment, if Figure 2 As shown, the point cloud depth feature is subjected to a maximum pooling operation (MaxPool) in the dimension of the number of points to obtain a one-dimensional vector as a global shape feature, and the global shape feature is expressed as MaxPool(F(X)). The global shape feature has the global shape information of the flexible cable, and the first predicted three-dimensional positions of the M key nodes are finally obtained, which are evenly and smoothly distributed and have strong robustness to occluded cables.

[0080] Step S104: using a point-to-point voting branch network to map the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable, and obtaining a prediction result of the point-to-point voting branch network.

[0081] In this embodiment, the M key nodes obtained by processing using the point-to-point voting branch network correspond to the M key points in step S103. The point-to-point voting branch network fully utilizes the information of the local area of ​​each node through point-by-point estimation and voting mechanism. The voting method has better prediction performance for nodes outside the occluded part because it focuses on rich local information.

[0082] In an optional embodiment, mapping the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable using a point-to-point voting branch network includes:

[0083] The feature of each input point is mapped into a thermal value and a relative displacement offset from the input point to the key node to be predicted through a fully connected network, wherein the thermal value represents the relative distance from the input point to the key node to be predicted, and the relative displacement offset represents the direction from the input point to the key node to be predicted;

[0084] Calculating a three-dimensional absolute offset from each input point to each key node based on the thermal value and the relative displacement offset;

[0085] A weighted vote is performed on the three-dimensional absolute offset of each input point to obtain a second predicted three-dimensional position of each key node.

[0086] Specifically, Figure 2 As shown, a fully connected network can be shared among all input points, and through normalization operation, the thermal value and relative offset from each input point to each key node to be predicted are obtained from the deep features of each input point. The thermal value H and relative offset U of all input points can be expressed as:

[0087] H = Sigmoid(FC(F(X)))

[0088] U = Normalized(FC(F(X)))

[0089] Among them, FC() represents a fully connected layer, that is, it represents a linear mapping of the point cloud depth features, Sigmoid() is the activation function, and Normalize() is the normalization function.

[0090] like Figure 3As shown in the figure, the point-to-point voting branch network processes the deep features of the point cloud to obtain the thermal value and relative offset of the input point to the key point. The larger the thermal value, the closer the relative distance from the input point to the key node to be predicted. After obtaining the thermal value and relative offset, the point-to-point voting branch network can further use the predicted thermal value and relative offset to take the input point with a larger thermal value as the more credible prediction source, and perform weighted voting on the predicted value of the key node according to the size of the thermal value to obtain the state of the flexible cable predicted by the point-to-point voting branch. When performing weighted voting, the larger the thermal value, the closer the distance from the input point to the key node to be predicted, and the input point is given a greater weight.

[0091] Step S105: Based on the non-rigid point cloud registration network, the prediction result of the end-to-end regression branch network and the prediction result of the point-to-point voting branch network are integrated to obtain a flexible cable state perception result.

[0092] like Figure 4 As shown in the figure, the prediction results of the end-to-end regression branch network are uniform and smooth, but the prediction results are not accurate enough. The prediction results of the point-to-point voting branch network are relatively accurate in the unoccluded part, but the results in the occluded part are more deviated from the actual results. Therefore, in order to effectively fuse the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network, according to the correspondence between the two, a non-rigid registration method is used to overcome the problem that the prediction results of the end-to-end regression branch are not accurate enough, and the prediction results of the point-to-point voting branch are not robust enough for occluded flexible cables.

[0093] Specifically, a non-rigid transformation from the prediction result of the end-to-end regression branch network of the unobstructed part to the prediction result of the point-to-point voting branch network is pre-estimated, and then the non-rigid transformation is applied to the prediction results of the end-to-end regression branch network of all key nodes to obtain the final flexible cable state perception result. The unobstructed part corresponding to the state perception result is as accurate as the prediction result of the point-to-point voting branch network, and the obstructed part is as smooth and reasonable as the prediction result of the end-to-end regression branch network.

[0094] For example, for the three-dimensional positions of 50 key nodes predicted by the end-to-end regression branch network and the three-dimensional positions of 50 key nodes predicted by the point-to-point voting branch network, 10 of the key nodes are occluded key nodes. First, the three-dimensional positions of the 40 unoccluded key nodes predicted by the end-to-end regression branch network are estimated in advance, and a non-rigid transformation is performed on the three-dimensional positions of the 40 unoccluded key nodes predicted by the point-to-point voting branch network. Then, the non-rigid transformation is applied to the three-dimensional positions of the 50 key nodes predicted by the end-to-end regression branch network, and the final flexible cable state perception result is obtained.

[0095] In an alternative embodiment, the non-rigid point cloud registration network fuses the prediction results of the end-to-end regression branch network and the point-to-point voting branch network to obtain the flexible cable state perception result, including steps A1 to A4:

[0096] Step A1: Calculate the probabilities that M key nodes are not occluded respectively to determine whether a key node is occluded.

[0097] Specifically, for each key node, the maximum heat value of all input points in the key node is used as the probability that the key node is not occluded; when the probability that the key node is not occluded is greater than or equal to a preset threshold, it is determined that the key node is not occluded; when the probability that the key node is not occluded is less than the preset threshold, it is determined that the key node is occluded.

[0098] For example, for the key node y to be predicted i , select the maximum heat value among all input points as the probability that the key node y i is not occluded, that is, p j = max i H ij , where H ij represents the heat value of the input point x i to the key node y to be predicted j . According to a preset threshold T with a value between 0 and 1, when p j ≥T, it is determined that the key node is not occluded, and when p j <T, it is determined that the key node is occluded.

[0099] Step A2: Retain the unoccluded key nodes and remove the occluded key nodes from the prediction results of the end-to-end regression branch network and the point-to-point voting branch network.

[0100] Step A3: Register the prediction result of the end-to-end regression branch network after removing some key nodes to the prediction result of the point-to-point voting branch network after removing some nodes, and calculate the global deformation field.

[0101] In this embodiment, the global deformation field represents the non-rigid transformation relationship between the prediction result of the end-to-end regression branch network and the prediction result of the point-to-point voting branch network. The global deformation field is a non-rigid transformation relationship obtained by registering the three-dimensional positions of the unoccluded key nodes predicted by the end-to-end regression branch network to the three-dimensional positions of the unoccluded key nodes predicted by the point-to-point voting branch network.

[0102] Specifically, the prediction results of the end-to-end regression branch network after removing some key nodes are aligned with the prediction results of the point-to-point voting branch network after removing some nodes, and the global deformation field is calculated, including step A31 and step A32:

[0103] Step A31: constructing a Gaussian radial basis network according to the prediction results of the end-to-end regression branch network after removing some key nodes and the prediction results of the point-to-point voting branch network after removing some nodes.

[0104] In this step, the prediction results of the end-to-end regression branch network after removing some key nodes are used as the registration source point set, and the prediction results of the point-to-point voting branch network after removing some nodes are used as the target point set. First, a Gaussian radial basis network is constructed. For the three-dimensional point cloud Z, the Gaussian kernel function in the radial basis network is expressed as:

[0105] G ij (Z) = exp(-||z i -z j || 2 / 2β 2 )

[0106] Among them, G ij is a Gaussian kernel function with a dimension of k×k, G ij For the jth element in its i-th row, z j is the jth point in the point cloud, z i is the i-th point in the point cloud, and β is the preset Gaussian kernel function parameter.

[0107] The non-rigid transformation (global deformation field) of the prediction result of the end-to-end regression branch network after removing some key nodes can be expressed by the corresponding Gaussian radial basis network as follows:

[0108]

[0109] in, It means that the back-end regression branch network predicts the three-dimensional position of the key node after removing the occluded key node, W is the weight matrix to be solved, It represents the displacement of the key node predicted by the end-to-end regression branch network after removing the occluded key node. The two are added together to obtain the non-rigid transformation of the key node predicted by the end-to-end regression branch network after removing the occluded key node.

[0110] Step A32: Iteratively optimize the weight matrix and variance size of the Gaussian radial basis network so that the prediction results of the end-to-end regression branch network after removing some key nodes are continuously aligned with the prediction results of the point-to-point voting branch network after removing some nodes, thereby obtaining the global deformation field.

[0111] In this step, the Gaussian radial basis network weight matrix and the variance size constructed in step A31 are iteratively optimized. Since the corresponding relationship between the key nodes predicted by the end-to-end regression branch network and the key nodes predicted by the point-to-point voting branch network is known in this embodiment, only a few iterations are needed to calculate the weight matrix W and the variance size σ 2 , and then get the global deformation field, weight matrix W and variance size σ 2 The iterative calculation formulas are:

[0112]

[0113]

[0114] in, and are the prediction results of the end-to-end regression branch and the point-to-point voting branch network after removing the occluded key nodes, λ is the preset parameter that determines the smoothness of the non-rigid transformation, G is the Gaussian kernel function, D represents the dimension, and D is equal to 3. W is the weight matrix to be solved, σ 2 is the variance to be solved.

[0115] According to the above weight matrix W and variance size σ 2 Substitute the iterative calculation formula into the current Solve the weight matrix W and the variance size σ 2 After one iteration, the solution is used as the new weight matrix and variance size, and substituted into the above two equations for the next iteration. , where ε is the preset threshold, indicating that the prediction result of the end-to-end regression branch network after registration is close enough to the prediction result of the point-to-point voting branch network, the iterative calculation process ends, and the current weight matrix W is compared with the variance size σ 2 As the final calculation result (i.e., representing the global deformation field).

[0116] Step A4: performing a spatial non-rigid transformation on the prediction result of the end-to-end regression branch network according to the global deformation field to obtain a flexible cable state perception result.

[0117] Specifically, the flexible cable status perception result The specific calculation process is expressed as:

[0118]

[0119] in, is the prediction result of the end-to-end regression branch network, according to the global deformation field calculated in step A3 Perform a non-rigid transformation to obtain the result after the non-rigid transformation The non-rigid transformation result is used as the flexible cable state perception result.

[0120] In the above embodiment, the dual-branch neural network model for sensing the state of the flexible cable is pre-constructed, and the construction of the dual-branch neural network model includes steps B1 and B3:

[0121] Step B1: construct a dual-branch neural network model, which includes: a point cloud feature extraction network, an end-to-end regression branch network, a point-to-point voting branch network and a non-rigid point cloud registration network.

[0122] like Figure 2 As shown in the figure, the dual-branch neural network model includes four parts. The point cloud feature extraction network is used to extract the point cloud deep features from the three-dimensional point cloud of the flexible cable to effectively encode the overall shape and local spatial information of the flexible cable. Specifically, the feature extraction layer and the feature propagation layer are stacked continuously according to the PointNet++ network rules to construct the point cloud feature extraction network. The end-to-end regression branch network is used to effectively reconstruct the shape of the occluded part of the cable. The end-to-end regression branch network includes: maximum pooling operation and fully connected network. The global shape features of the flexible cable are obtained through the maximum pooling operation. The global shape features are mapped to the state of the flexible cable predicted by the end-to-end regression branch through the fully connected network. The point-to-point voting branch network is used to effectively estimate the position of the key nodes of the local cable. The point-to-point voting branch network includes: a fully connected network and a fully connected network shared between the input points through normalization operation. Through the normalization operation, the thermal value and relative offset from each input point to each key node to be predicted are obtained from the features of each input point, and then the state of the flexible cable predicted by the point-to-point voting branch is obtained through weighted voting. The non-rigid point cloud registration network is used to fuse the prediction results of the end-to-end regression branch network and the point-to-point voting branch network to obtain accurate and occlusion-robust state perception results.

[0123] Step B2: construct a training data set, wherein the data in the training data set is the flexible cable motion data collected in the simulator.

[0124] In this embodiment, flexible cables of different lengths, thicknesses, and stiffnesses are randomly generated in the simulator, and two independent mechanical arms are used to clamp the two ends of the flexible cables and move freely in the workspace to generate random motion data of the flexible cables and collect as many different shapes as possible. Among them, a set of training data is the three-dimensional point cloud corresponding to a motion moment (a posture) of the flexible cable at a certain length, thickness, and stiffness, the three-dimensional position of the key nodes, the true value of the thermal value from each input point to each key node, and the true value of the relative offset.

[0125] In this embodiment, the flexible cable motion data collected by the simulator can cover the postures of the flexible cable in different shapes, stiffnesses and lengths as much as possible, increasing the richness of the data. Furthermore, the dual-branch neural network model trained based on the training data set has good generalization ability.

[0126] Specifically, the flexible cable motion data collected in the simulator includes:

[0127] A plurality of flexible cables of different lengths, thicknesses and rigidities are arranged in the simulator, and both ends of the flexible cables are fixedly clamped by a mechanical arm in the simulator, and the mechanical arm moves freely in the workspace;

[0128] Recording the color image and depth image of the flexible cable at each moment in the simulator, as well as the three-dimensional position of the key nodes of the flexible cable at the current moment;

[0129] The depth image of the area where the flexible cable is located is back-projected into three-dimensional space to obtain a three-dimensional point cloud of the flexible cable, and the true value of the thermal value and the true value of the relative offset from each point in the three-dimensional point cloud to the key node are calculated. The three-dimensional point cloud of the flexible cable, the three-dimensional position of the key node recorded at the corresponding time, the true value of the thermal value and the true value of the relative offset constitute a set of training data.

[0130] In this embodiment, the actual value H of the thermal value in the point-to-point voting branch network is calculated. gt The relative offset value U gt And can be calculated according to the following formula:

[0131]

[0132]

[0133] Among them, r is the preset neighborhood radius, and the thermal value and relative offset are both 0 outside this neighborhood. i is the i-th input point, y j is the jth key node to be predicted.

[0134] Step B3: training a dual-branch neural network model based on the training data set, wherein the dual-branch neural network model training comprises:

[0135] Inputting the data in the training set into the two-branch neural network model;

[0136] For each set of training data, the dual-branch neural network model calculates the three-dimensional position of the key node predicted by the end-to-end regression branch network, the predicted value of the thermal value from each input point to each key node in the point voting branch network, and the predicted value of the relative offset according to the three-dimensional point cloud of the flexible cable in the training data;

[0137] Calculate the loss function according to the true value of the three-dimensional position of the key node, the true value of the thermal value of each key node and the true value of the relative offset, the three-dimensional position of the key node predicted by the end regression branch network, and the predicted value of the thermal value of each key node to be considered and the predicted value of the relative offset;

[0138] According to the loss function, optimizing the parameters of the dual-branch neural network model based on a gradient descent algorithm;

[0139] After the training is completed, a trained dual-branch neural network model is obtained.

[0140] Among them, the specific calculation of the loss function Loss can be expressed as:

[0141]

[0142] in, Predict the 3D position of key nodes for the end-to-end regression branch, Y gt is the true value of the three-dimensional position of the key node, μ is the weight coefficient of the end-to-end regression branch network and the point-to-point voting branch network, N is the number of input points in the three-dimensional point cloud, M is the number of key nodes to be predicted, and are the predicted value of the thermal value and the predicted value of the relative offset from the i-th input point to the j-th key node to be predicted in the point-to-point voting branch network, and They are respectively the true value of the thermal value and the true value of the relative offset from the i-th input point to the j-th key node to be predicted in the point-to-point voting branch network.

[0143] In the actual training process, the data in the training data set is randomly divided, such as selecting 80% of the data as the training set and the other 20% of the data as the test set. The data in the training set is input into the two-branch neural network model in batches for training. After the training is completed, the two-branch neural network model is verified using the verification set, and the two-branch neural network model parameters with the best verification results are selected as the final two-branch neural network model parameters.

[0144] In this embodiment, a flexible cable state perception method with robust occlusion is provided. First, the point cloud depth features are extracted from the three-dimensional point cloud of the flexible cable to effectively encode the overall shape and local spatial information of the flexible cable; then, the point cloud depth features are mapped to the first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable through an end-to-end regression branch network, so as to effectively reconstruct the shape of the occluded part of the cable; the point cloud depth features are mapped to the second predicted three-dimensional positions of M key nodes of the flexible cable through a point-to-point voting branch network, so as to effectively predict the positions of the key nodes of the local cable, especially for the key nodes outside the occluded part, which has good prediction performance; then, the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network are fused to obtain an accurate and occlusion-robust state perception result. Since the flexible cable state perception method provided by this embodiment combines the advantages of the end-to-end regression network and the point-to-point voting network, it can pay attention to the global shape features and local features of the flexible cable at the same time, and thus can accurately perceive the flexible cable completely and reliably, and can still accurately perceive the flexible cable in the case of occlusion.

[0145] The embodiment of the present invention also provides a flexible cable state sensing device with robust shielding. Figure 5 As shown, Figure 5 A schematic diagram of a structural diagram of a flexible cable state sensing device with robust shielding provided by an embodiment of the present invention, the device comprising:

[0146] A point cloud acquisition module 51 is used to acquire a three-dimensional point cloud of the flexible cable, wherein the three-dimensional point cloud includes N input points;

[0147] A feature extraction module 52 is used to extract features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features;

[0148] An end-to-end regression module 53, used to map the point cloud depth features into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable using an end-to-end regression branch network, and obtain a prediction result of the end-to-end regression branch network;

[0149] A point-to-point voting module 54 is used to map the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable using a point-to-point voting branch network to obtain a prediction result of the point-to-point voting branch network;

[0150] The result fusion module 55 is used to fuse the prediction result of the end-to-end regression branch network and the prediction result of the point-to-point voting branch network based on the non-rigid point cloud registration network to obtain the flexible cable state perception result.

[0151] In an optional embodiment, the feature extraction module includes:

[0152] A first extraction submodule is used to sample the three-dimensional point cloud of the flexible cable based on the farthest point sampling method, select multiple center points from N input points in the three-dimensional point cloud, find multiple neighboring points as regions in the neighborhood of each center point, perform feature extraction in the region, and obtain the depth feature of each center point;

[0153] The second extraction submodule is used to transfer the depth feature of each center point to the input point in the three-dimensional point cloud, and use weighted interpolation to obtain the depth feature of each input point as the depth feature of the flexible cable point cloud.

[0154] In an optional embodiment, the end-to-end regression module includes:

[0155] A pooling module is used to perform a maximum pooling operation on the point cloud depth feature in the dimension of the number of points to obtain a global shape feature;

[0156] A mapping module is used to map the global shape feature into first predicted three-dimensional positions of M key nodes on the flexible cable through a fully connected network.

[0157] In an optional embodiment, the peer-to-peer voting module includes:

[0158] A calculation module is used to map the features of each input point into a thermal value and a relative displacement offset from the input point to the key node to be predicted through a fully connected network, wherein the thermal value represents the relative distance from the input point to the key node to be predicted, and the relative displacement offset represents the direction from the input point to the key node to be predicted;

[0159] An absolute offset module, for calculating a three-dimensional absolute offset from each input point to each key node according to the thermal value and the relative displacement offset;

[0160] The voting module is used to perform weighted voting on the three-dimensional absolute offset of each input point to obtain a second predicted three-dimensional position of each key node.

[0161] In an optional embodiment, the result fusion module includes:

[0162] The probability module is used to calculate the probability of M key nodes being unobstructed respectively, so as to determine whether the key node is obstructed;

[0163] A removal module, used to retain unobstructed key nodes and remove obstructed key nodes from the end-to-end regression branch network prediction results and the point-to-point voting branch network prediction results;

[0164] A registration module is used to register the prediction results of the end-to-end regression branch network after removing some key nodes with the prediction results of the point-to-point voting branch network after removing some nodes, and calculate the global deformation field;

[0165] The transformation module is used to perform spatial non-rigid transformation on the prediction result of the end-to-end regression branch network according to the global deformation field to obtain a flexible cable state perception result.

[0166] In an optional embodiment, the probability module includes:

[0167] A first probability submodule is used to, for each key node, use the maximum thermal value of all input points in the key node as the probability that the key node is not blocked;

[0168] A second probability submodule, configured to determine that the key node is not blocked when the probability that the key node is not blocked is greater than or equal to a preset threshold;

[0169] The third probability submodule is used to determine that the key node is blocked when the probability that the key node is not blocked is less than a preset threshold.

[0170] In an optional embodiment, the registration module includes:

[0171] A first registration module is used to construct a Gaussian radial basis network according to the prediction result of the end-to-end regression branch network after removing some key nodes;

[0172] The second registration module is used to iteratively optimize the weight matrix and variance size of the Gaussian radial basis network so that the prediction results of the end-to-end regression branch network after removing some key nodes are continuously aligned to the prediction results of the point-to-point voting branch network after removing some nodes, thereby obtaining the global deformation field.

[0173] In an optional embodiment, the device further includes:

[0174] A network construction module, used to construct a dual-branch neural network model, wherein the dual-branch neural network model includes: a point cloud feature extraction network, an end-to-end regression branch network, a point-to-point voting branch network, and a non-rigid point cloud registration network;

[0175] The data construction module is used to construct a training data set, wherein the data in the training data set is the flexible cable motion data collected in the simulator.

[0176] In an optional embodiment, the data construction module includes:

[0177] A first data construction submodule is used to set a plurality of flexible cables of different lengths, thicknesses and stiffnesses in the simulator, wherein both ends of the flexible cables are fixedly clamped by a mechanical arm in the simulator, and the mechanical arm moves freely in the workspace;

[0178] A second data construction submodule is used to record the color image and depth image of the flexible cable at each moment in the simulator, and the three-dimensional position of the key node of the flexible cable at the current moment;

[0179] The third data construction submodule is used to back-project the depth image of the area where the flexible cable is located back to the three-dimensional space to obtain the three-dimensional point cloud of the flexible cable, calculate the true value of the thermal value and the true value of the relative offset from each point in the three-dimensional point cloud to the key node, and form a set of training data with the three-dimensional point cloud of the flexible cable, the three-dimensional position of the key node recorded at the corresponding time, the true value of the thermal value and the true value of the relative offset.

[0180] In an optional embodiment, the device further includes:

[0181] A training input module, used for inputting the data in the training set into the dual-branch neural network model;

[0182] A prediction calculation module, for each set of training data, the dual-branch neural network model calculates the three-dimensional position of the key node predicted by the end-to-end regression branch network, the predicted value of the thermal value from each input point to each key node in the point voting branch network, and the predicted value of the relative offset according to the three-dimensional point cloud of the flexible cable in the training data;

[0183] A loss calculation module is used to calculate a loss function according to the true value of the three-dimensional position of the key node, the true value of the thermal value of each key node and the true value of the relative offset, the three-dimensional position of the key node predicted by the end regression branch network, and the predicted value of the thermal value of each key node to be considered and the predicted value of the relative offset;

[0184] A parameter optimization module, used for optimizing the parameters of the dual-branch neural network model based on the gradient descent algorithm according to the loss function;

[0185] The training result module is used to obtain the trained two-branch neural network model after the training is completed.

[0186] An embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the occlusion-robust flexible cable state perception method described in the embodiment of the present invention is implemented.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses and devices according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0191] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0192] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0193] The above is a detailed introduction to the occlusion-robust flexible cable state perception method, device and equipment provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A flexible cable state sensing method with robust occlusion, characterized in that: The method comprises: Acquire a three-dimensional point cloud of the flexible cable, wherein the three-dimensional point cloud includes N input points; Extracting features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features; Using an end-to-end regression branch network, the point cloud depth feature is mapped into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable, and a prediction result of the end-to-end regression branch network is obtained; Mapping the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable using a point-to-point voting branch network to obtain a prediction result of the point-to-point voting branch network; Based on the non-rigid point cloud registration network, the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network are fused to obtain the flexible cable state perception result; wherein the point-to-point voting branch network is used to estimate the position of the local cable key node; the end-to-end regression branch network is used to reconstruct the shape of the obscured cable part; The non-rigid point cloud registration network is based on fusing the prediction result of the end-to-end regression branch network and the prediction result of the point-to-point voting branch network to obtain a flexible cable state perception result, including: Calculate the probability that the M key nodes are not blocked respectively to determine whether the key node is blocked; The unobstructed key nodes are retained, and the obstructed key nodes are removed from the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network; The prediction results of the end-to-end regression branch network after removing some key nodes are aligned with the prediction results of the point-to-point voting branch network after removing some nodes, and the global deformation field is calculated; A spatial non-rigid transformation is performed on the prediction result of the end-to-end regression branch network according to the global deformation field to obtain a flexible cable state perception result.

2. The method according to claim 1, characterized in that The step of extracting features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features includes: The three-dimensional point cloud of the flexible cable is sampled based on the farthest point sampling method, multiple center points are selected from N input points in the three-dimensional point cloud, multiple neighboring points are found as regions in the neighborhood of each center point, and feature extraction is performed in the region to obtain the depth feature of each center point; The depth feature of each center point is transferred to the input point in the three-dimensional point cloud, and the depth feature of each input point is obtained by weighted interpolation as the depth feature of the flexible cable point cloud.

3. The method according to claim 1, characterized in that The method of mapping the point cloud depth features into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable using an end-to-end regression branch network includes: Performing a maximum pooling operation on the point cloud depth feature in the dimension of the number of points to obtain a global shape feature; The global shape feature is mapped into first predicted three-dimensional positions of M key nodes on the flexible cable through a fully connected network.

4. The method according to claim 1, characterized in that The method of mapping the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable by using a point-to-point voting branch network includes: The feature of each input point is mapped into a thermal value and a relative displacement offset from the input point to the key node to be predicted through a fully connected network, wherein the thermal value represents the relative distance from the input point to the key node to be predicted, and the relative displacement offset represents the direction from the input point to the key node to be predicted; Calculating a three-dimensional absolute offset from each input point to each key node based on the thermal value and the relative displacement offset; A weighted vote is performed on the three-dimensional absolute offset of each input point to obtain a second predicted three-dimensional position of each key node.

5. The method according to claim 1, characterized in that The respectively calculating the probabilities that the M key nodes are not blocked to determine whether the key nodes are blocked includes: For each key node, the maximum thermal value of all input points in the key node is used as the probability that the key node is not blocked; When the probability that the key node is not blocked is greater than or equal to a preset threshold, determining that the key node is not blocked; When the probability that the key node is not blocked is less than a preset threshold, it is determined that the key node is blocked.

6. The method according to claim 1, characterized in that The prediction results of the end-to-end regression branch network after removing some key nodes are aligned with the prediction results of the point-to-point voting branch network after removing some nodes, and the global deformation field is calculated, including: Constructing a Gaussian radial basis network according to the prediction results of the end-to-end regression branch network after removing some key nodes; The weight matrix and variance size of the Gaussian radial basis network are iteratively optimized so that the prediction results of the end-to-end regression branch network after removing some key nodes are continuously aligned and close to the prediction results of the point-to-point voting branch network after removing some nodes, thereby obtaining the global deformation field.

7. The method according to claim 1, characterized in that The method further comprises: Constructing a dual-branch neural network model, the dual-branch neural network model includes: a point cloud feature extraction network, an end-to-end regression branch network, a point-to-point voting branch network and a non-rigid point cloud registration network; A training data set is constructed, wherein the data in the training data set is the flexible cable motion data collected in the simulator.

8. The method according to claim 7, characterized in that The flexible cable motion data collected in the simulator includes: A plurality of flexible cables of different lengths, thicknesses and rigidities are arranged in the simulator, and both ends of the flexible cables are fixedly clamped by a mechanical arm in the simulator, and the mechanical arm moves freely in the workspace; Recording the color image and depth image of the flexible cable at each moment in the simulator, as well as the three-dimensional position of the key nodes of the flexible cable at the current moment; The depth image of the area where the flexible cable is located is back-projected into three-dimensional space to obtain a three-dimensional point cloud of the flexible cable, and the true value of the thermal value and the true value of the relative offset from each point in the three-dimensional point cloud to the key node are calculated. The three-dimensional point cloud of the flexible cable, the three-dimensional position of the key node recorded at the corresponding time, the true value of the thermal value and the true value of the relative offset constitute a set of training data.

9. The method according to claim 7, characterized in that: The dual-branch neural network model is trained in the following way: Inputting the data in the training data set into the two-branch neural network model; For each set of training data, the dual-branch neural network model calculates the three-dimensional position of the key node predicted by the end-to-end regression branch network, the predicted value of the thermal value from each input point to each key node in the point voting branch network, and the predicted value of the relative offset according to the three-dimensional point cloud of the flexible cable in the training data; Calculate the loss function according to the true value of the three-dimensional position of the key node, the true value of the thermal value of each key node and the true value of the relative offset, the three-dimensional position of the key node predicted by the end regression branch network, the predicted value of the thermal value of each key node and the predicted value of the relative offset; According to the loss function, optimizing the parameters of the dual-branch neural network model based on a gradient descent algorithm; After the training is completed, a trained dual-branch neural network model is obtained.

10. A flexible cable state sensing device with robust occlusion, characterized in that: The device comprises: A point cloud acquisition module, used to acquire a three-dimensional point cloud of the flexible cable, wherein the three-dimensional point cloud includes N input points; A feature extraction module, used to extract features from the three-dimensional point cloud through a point cloud feature extraction network to obtain point cloud depth features; An end-to-end regression module, used to map the point cloud depth features into first predicted three-dimensional positions of M key nodes evenly distributed on the flexible cable using an end-to-end regression branch network, and obtain a prediction result of the end-to-end regression branch network; A point-to-point voting module, used to map the point cloud depth features into second predicted three-dimensional positions of M key nodes of the flexible cable using a point-to-point voting branch network, and obtain a prediction result of the point-to-point voting branch network; The result fusion module is used to fuse the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network based on the non-rigid point cloud registration network to obtain the flexible cable state perception result; wherein the point-to-point voting branch network is used to estimate the position of the local cable key node; the end-to-end regression branch network is used to reconstruct the shape of the occluded part of the cable; the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network based on the non-rigid point cloud registration network are fused to obtain the flexible cable state perception result, including: Calculate the probability that the M key nodes are not blocked respectively to determine whether the key node is blocked; The unobstructed key nodes are retained, and the obstructed key nodes are removed from the prediction results of the end-to-end regression branch network and the prediction results of the point-to-point voting branch network; The prediction results of the end-to-end regression branch network after removing some key nodes are aligned with the prediction results of the point-to-point voting branch network after removing some nodes, and the global deformation field is calculated; A spatial non-rigid transformation is performed on the prediction result of the end-to-end regression branch network according to the global deformation field to obtain a flexible cable state perception result.

11. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the occlusion-robust flexible cable state perception method according to any one of claims 1 to 9.

Images