A Bolt Pose Estimation Method Based on Monocular RGB Images

Abstract

This invention discloses a bolt pose estimation method based on monocular RGB images, applicable to industrial production and intelligent manufacturing. The method involves inputting a monocular RGB image or video frame into a pre-defined feature extraction network to obtain a high-dimensional feature map of the image or video frame to be detected. This high-dimensional feature map is then input into a convolutional pose estimation network to predict the bolt's 2D keypoint confidence map and affinity map. The prediction results are used to search for the optimal match among all potential symmetrical poses of the bolt, and the network loss is calculated to update the network parameters. A heuristic algorithm is used to search within the prediction results of the confidence map and affinity map to obtain the 2D coordinates of all bolt keypoints. Finally, a direct linear transformation algorithm is used to establish the connection between the 2D and 3D keypoints of the bolt, calculating the 6D pose of the bolt.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a bolt pose estimation method based on monocular RGB images. Background Technology

[0002] The "14th Five-Year Plan for Intelligent Manufacturing Development" points out the need to deeply implement intelligent manufacturing projects, develop intelligent manufacturing system solutions, and build intelligent manufacturing systems that are virtual-physical integrated, knowledge-driven, dynamically optimized, safe, and efficient. In manufacturing, computer vision focuses on creating artificial systems that can capture, process, and thus understand visual input from the physical world to elicit appropriate responses and assist humans in completing various production-related tasks, providing operators or control mechanisms with navigation, grasping, and other location information related to environmental interaction.

[0003] In discrete manufacturing production lines, controlling a robotic arm to grasp and twist bolts is a relatively complex task. The robotic arm requires computer vision information for navigation during its movement and must provide the control unit with the position and orientation information of the target bolt in real time. Therefore, a bolt orientation detection algorithm that can be deployed in industrial manufacturing scenarios is needed. Currently, monocular RGB orientation detection methods based on two-dimensional keypoints of objects have made significant progress. These methods first detect the positions of all two-dimensional keypoints of the object, and then establish a 2D-3D correspondence between the CAD model of the target object and the DLT (Direct Linear Transform) algorithm to achieve the detection of the object's six degrees of freedom orientation. However, these methods still face challenges when applied to bolt targets: ① Bolt targets are relatively small, occupying as little as 40×40 pixels in a 400×400 resolution image. Their feature extraction process is easily affected by background information and downsampling operations, and existing methods cannot effectively utilize the semantic information in the image. ② Existing orientation detection methods often treat target localization as an upstream task and keypoint detection as a corresponding downstream task. When multiple target objects appear in the image, the lack of connection between key points and their respective objects during the reasoning process hinders the correct matching of key points with their objects. ③ For objects with multiple rotational symmetries, such as hexagonal bolts, the main problem faced by this type of algorithm is how to uniquely predict the pose of symmetrical objects, since the same observation results may be produced from different orientations. Summary of the Invention

[0004] To address the shortcomings and deficiencies of the existing technologies, a bolt pose estimation method based on monocular RGB images is provided. This method solves the problems of poor performance of existing pose detection methods for multi-target and small-target detection, as well as their inability to handle objects with symmetry.

[0005] The solution proposed in this invention application is as follows:

[0006] A bolt pose estimation method based on monocular RGB images, the method comprising the following steps:

[0007] S11: Acquire 8k-10k RGB images containing bolt targets with a resolution of 400×400. Simultaneously, the rotation matrix, translation matrix, and corresponding 2D keypoint coordinates of each bolt in the RGB images containing bolt targets relative to the camera coordinate system are used as the true pose values ​​and saved in the form of image-pose label pairs as the network training dataset. Each image contains no more than 10 bolt targets of random number and pose for network training.

[0008] S12, input the RGB image obtained in step S11 into the feature extraction network to obtain a high-dimensional feature map of size 128×50×50;

[0009] An RGB image is input into a feature extraction network to obtain a low-dimensional feature map. When the low-dimensional feature map passes through each feature module in the feature extraction network, channel-level up-dimensional convolution, channel-level attention filtering, and channel-level down-dimensional convolution are performed. During the forward propagation process, the low-dimensional feature map is fused with the deep features output by the bypass.

[0010] S13, input the high-dimensional feature map obtained in step S12 into the convolutional pose network module to obtain 9 confidence maps and 16 affinity maps of the two-dimensional key points corresponding to the bolt target. The position of the peak in the confidence map is the candidate key point.

[0011] The nine key points of the bolt target each correspond to a confidence map, and the eight key points of the bolt target other than the center point each correspond to two affinity maps. The confidence maps and affinity maps together serve as two-dimensional key point features extracted from the RGB image.

[0012] When a high-dimensional feature map is input into the convolutional pose network module, the convolutional pose network module calculates the network loss using the true values ​​of the two-dimensional keypoint confidence map generated by the pose label and the true values ​​of the affinity map. The calculated network loss is then backpropagated and the network parameters are updated.

[0013] S14, Combining the affinity map obtained in step S13, a heuristic algorithm is used to search for the candidate key points to obtain the two-dimensional key point coordinates of the bolt target;

[0014] S15, the 2D key point coordinates of the bolt target searched in step S14 are compared with the known CAD model of the bolt to be detected by solving the 2D-3D correspondence of the key points of the bolt target through the direct linear transformation algorithm, so as to obtain the 6-DOF pose of the bolt target. The 6-DOF pose of the bolt target is represented as the rotation matrix and translation matrix of the bolt target relative to the camera coordinate system, thus completing the bolt pose estimation based on monocular RGB image.

[0015] In step S12, the feature extraction network is used to extract features from the input RGB image. The feature extraction network consists of 5 feature modules M1, 5 feature modules M2, a feature module Bottom, and a feature module Top.

[0016] The feature module M1 consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 1-1, ascending and descending channel convolutional module 1-2, and ascending and descending channel convolutional module 1-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 1-1, ascending and descending channel convolutional module 1-2, channel attention module, and ascending and descending channel convolutional module 1-3.

[0017] The feature module M2 consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 2-1, ascending and descending channel convolutional module 2-2, and ascending and descending channel convolutional module 2-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 2-1, ascending and descending channel convolutional module 2-2, channel attention module, and ascending and descending channel convolutional module 2-3.

[0018] The feature module Bottom consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 3-1, ascending and descending channel convolutional module 3-2, and ascending and descending channel convolutional module 3-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 3-1, ascending and descending channel convolutional module 3-2, channel attention module, and ascending and descending channel convolutional module 3-3.

[0019] The feature module Top is an ascending and descending channel convolutional module;

[0020] The method for constructing the feature extraction network is as follows:

[0021] The Bottom feature module is the first feature module in the feature extraction network, and it is connected to the first feature module M1. Five feature modules M1 are stacked vertically, and each M1 is connected horizontally to a feature module M2. Each feature module M2 outputs deep feature information, which is then element-wise added to the output of its corresponding feature module M1 to serve as the input feature map for the next feature module M1. The last feature module M1 is connected to the Top feature module, which serves as the terminal feature module of the feature extraction network.

[0022] The ascending and descending channel convolutional module is composed of a convolutional layer, a batch normalization layer and a Swish activation function connected in series, which serves as the basic structure of the feature module.

[0023] The channel attention module is used to perform global pooling on the current input feature map to obtain a feature vector with a size of 1×1 channels, and then multiply the feature vector as channel weights with the current input feature map to achieve channel filtering effect;

[0024] The rising and falling channel convolutional layers 1-1, 2-1, and 3-1 are used to increase the number of feature channels, while the rising and falling channel convolutional layers 1-3, 2-3, and 3-3 are used to decrease the number of feature channels.

[0025] In step S13, the convolutional attitude network module is used to predict the confidence map and affinity map of the two-dimensional key points of the bolt target. The convolutional attitude network module includes five key point prediction modules: key point prediction module 1, key point prediction module 2, key point prediction module 3, key point prediction module 4, and key point prediction module 5. Key point prediction module 1 includes two prediction channels, used to predict the two-dimensional key point confidence map and affinity map respectively, and each prediction channel includes two feature modules M1. Key point prediction module 2 also includes two prediction channels, used to predict the two-dimensional key point confidence map and affinity map respectively, and each prediction channel includes four feature modules M1. Key point prediction modules 2, 3, 4, and 5 have the same structure.

[0026] The high-dimensional feature map is input into five keypoint prediction modules, each of which outputs a confidence map of size 9×50×50 and an affinity map of size 16×50×50. The output of the first keypoint prediction module is added element-wise with the high-dimensional feature map output by the feature extraction network, and the two are used as the input features of the next keypoint prediction module. The confidence map and affinity map output by the last keypoint prediction module are used as the final prediction result.

[0027] The confidence plot represents the probability distribution of the 2D key point locations of the bolt target;

[0028] The true values ​​of the bolt target two-dimensional key point confidence map are generated based on the positions of the bolt target two-dimensional key points, and the generation method is as follows:

[0029] The initial true values ​​of the confidence graph for the k-th keypoint are:

[0030]

[0031] make This indicates that the coordinates (i,j) represent the k-th key point (i,j) of the n-th bolt target. (k,n),j (k,n) The correlation magnitude of ) is determined by ε = 8, which is a hyperparameter; let Traversing the key point confidence graph M k blf Elements in;

[0032] Repeat the above steps until the true values ​​M1 of the confidence plot for the 9 key points are obtained. blf M2 blf …M9 blf ;

[0033] The affinity map is a feature representation of a two-dimensional key point of a bolt target with respect to the two-dimensional center point of the bolt target to which the two-dimensional key point belongs. The method for constructing the true value of the affinity map is as follows:

[0034] The initial values ​​of the two keypoint affinity maps corresponding to the k-th keypoint are:

[0035]

[0036] Obtain the point set consisting of the k-th key point of all bolt targets:

[0037]

[0038] in, Let the coordinates of the k-th key point of the n-th bolt target be denoted as . The coordinates of the center point of the nth bolt target are represented as follows:

[0039]

[0040]

[0041] Repeat the above steps to obtain a total of 16 true affinity maps.

[0042] In step S13, the network loss during training is calculated from the confidence map and affinity map of the five outputs of the convolutional pose network module. The method for calculating the network loss is as follows:

[0043] The network loss function is expressed as:

[0044]

[0045] in, Represent the confidence map and affinity field loss on the q-th convolutional pose network module, respectively. For each convolutional pose network module, and The calculation method is as follows:

[0046]

[0047]

[0048]

[0049]

[0050] in, Let represent the network's predicted value and the actual value of the confidence graph for the k-th key point, respectively. Let represent the predicted and actual values ​​of the affinity graph corresponding to the k-th keypoint, respectively. This represents the predicted values ​​of the two affinity maps corresponding to the k-th keypoint. This represents the true values ​​of the two affinity maps corresponding to the k-th keypoint, where k = 1, 2, ..., 9, and γ = 50 is the resolution of the confidence map and affinity map. This represents the rotation of the k-th 3D keypoint of the object around the bolt target's |θ| axis of symmetry. The confidence graph corresponding to the projection of the value λ times onto a two-dimensional plane. This represents the rotation of the k-th 3D keypoint of the object around the bolt target's |θ| axis of symmetry. After multiplying by λ, the corresponding two affinity maps are projected onto a two-dimensional plane. For the bolt target, the symmetry axis weight |θ|=6, and the confidence map output from the last stage is taken as the result. The location where the confidence peak appears on this map is marked as a candidate key point;

[0051] In step S14, a heuristic search is performed on the candidate key points, and the candidate key points are assigned to their corresponding bolt targets. The heuristic search method is as follows:

[0052] Obtain the set of all m candidate keypoints for the prediction result of the k-th keypoint by the convolutional pose network module:

[0053]

[0054] Obtain the feature vectors of the candidate keypoints at the corresponding positions in the affinity map prediction results:

[0055]

[0056] in, This represents the eigenvalue of the m-th candidate point at the corresponding position in the two affinity maps. The eigenvectors formed;

[0057] For the nth bolt target, the coordinates of the kth key point for:

[0058]

[0059] in, This represents the prediction result of the convolutional pose network module for the center point of the nth bolt target, where δ = 0.5 is a hyperparameter.

[0060] Repeat the above steps until the coordinates of k key points of all n bolt targets are obtained;

[0061] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a bolt attitude estimation method based on a monocular RGB image.

[0062] A computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement a bolt attitude estimation method based on a monocular RGB image.

[0063] Beneficial effects

[0064] (1) This application proposes a bolt attitude estimation method based on monocular RGB images, which improves the defects of current related methods in bolt attitude detection.

[0065] (2) To address the common problem of poor feature extraction performance for small targets in existing pose detection methods, a feature extraction network module with high resolution and channel attention mechanism is designed. To integrate semantic information between feature layers of different depths, the backbone of the network forms a mesh structure through lateral connections during forward propagation. To avoid a large number of irrelevant redundant feature maps during forward propagation and to ensure the network focuses on the detected target itself, a channel attention module is introduced. This module uses a bypass network to obtain the importance of each channel in the input features and assigns weights to each channel based on its importance. Furthermore, to maintain feature resolution, the network avoids downsampling operations (such as convolutional layers or pooling layers with a stride of 2), ensuring that the feature map resolution remains at least 50×50 throughout the entire network module computation. Compared to benchmark feature extraction networks such as ResNet and VGG, the network structure proposed in this application significantly improves the model's utilization of the semantic information of bolt targets.

[0066] (3) Existing methods generally lack effective strategies for matching keypoints with their respective objects. To address this issue, this application proposes predicting affinity maps of keypoints using a convolutional pose network. The affinity map is a feature representation of the relationship between a keypoint and its corresponding bolt center point, used to filter and determine the object affiliation of detected keypoints. For a keypoint (i,j) and its corresponding bolt center point (i0,j0), its affinity feature is represented as a vector (i-i0,j-j0) pointing from the current bolt's 2D center point to the keypoint. The two feature values ​​of this vector are stored at their corresponding positions in the two affinity maps corresponding to the keypoint. During the prediction process, using this feature representation in conjunction with the aforementioned heuristic search algorithm can correctly and effectively match keypoints with their respective bolts, avoiding incorrect connections between keypoints of different bolts in multi-object scenes, which would otherwise make it difficult for the algorithm to recover the 6D pose of the bolts.

[0067] (4) To address the problem that existing methods cannot stably detect rotationally symmetric objects, this application proposes a network loss function that considers all potential symmetric poses of the object. During network training and loss function calculation, the bolt is rotated six times along its rotational symmetry axis (consistent with the multiplicity of the symmetry axis). The loss is calculated by comparing the confidence map predicted by the network with the projection of each symmetric pose of the bolt onto a two-dimensional plane, and the minimum value is taken. When the network correctly predicts any symmetric pose of the bolt, the network loss is minimized. This loss function significantly improves the convergence of the network during training, eliminates the phenomenon of multi-peak values ​​in the network keypoint confidence map caused by the bolt's symmetry, and improves the accuracy of keypoint localization.

[0068] (5) This invention discloses a bolt pose estimation method and device based on monocular RGB images, which can be applied to industrial production and intelligent manufacturing. The method includes: inputting a monocular RGB image or video frame into a preset feature extraction network to obtain a high-dimensional feature map of the image or video frame to be detected; inputting the obtained high-dimensional feature map into a convolutional pose network to predict the 2D keypoint confidence map and affinity map of the bolt; searching for the optimal match among all potential symmetrical poses of the bolt using the prediction results and calculating the network loss to update the network parameters; using a heuristic algorithm to search in the prediction results of the confidence map and affinity map to obtain the 2D coordinates of all bolt keypoints; and using a direct linear transformation algorithm to establish the connection between the 2D and 3D keypoints of the bolt and calculate the 6D pose of the bolt to be tested. In the pose detection method proposed in this invention, a high-resolution feature extraction network structure is proposed, which improves the network's ability to extract semantic information from small targets such as bolts. A method using the bolt target affinity feature to assist in matching its keypoints with the object to which it belongs is proposed, improving the robustness of the algorithm in multi-target scenarios. Furthermore, due to the rotational symmetry of bolts, the same observation results may be obtained from different angles. This method proposes a loss function that can be used to train symmetrical objects. During training, the network loss is minimized for all symmetrical poses of the bolt, eliminating the ambiguity caused by the bolt's rotational symmetry in 2D keypoint prediction. This method aims to detect bolt pose using monocular RGB image information, exhibits strong robustness to multi-object scenes, and has a certain degree of versatility. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0070] Figure 2 This is a schematic diagram of the feature extraction network structure according to an embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the convolutional pose network module structure according to an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram of the monocular RGB image bolt pose estimation device based on a convolutional neural network according to an embodiment of the present invention. Detailed Implementation

[0073] To make the technical solutions contained in this invention application clearer and easier to understand, the invention application will be described and introduced in detail below with reference to specific embodiments. When the description involves drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Based on the embodiments in this invention application, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of this invention.

[0074] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings.

[0075] Figure 1 The diagram shown is a flowchart of a bolt posture detection method provided by an embodiment of the present invention. The posture detection method includes:

[0076] S11: Acquire 8k-10k RGB images containing bolt targets with a resolution of 400×400. Simultaneously, the rotation matrix, translation matrix, and corresponding 2D keypoint coordinates of each bolt in the RGB images containing bolt targets relative to the camera coordinate system are used as the true pose values ​​and saved in the form of image-pose label pairs as the network training dataset. Each image contains no more than 10 bolt targets of random number and pose for network training.

[0077] Specifically, the eight vertices of the bounding box of the bolt cuboid and the center point of the bolt are defined as key points, for a total of nine three-dimensional key points; the nine projection points of the three-dimensional key points on the imaging plane are the corresponding two-dimensional key points; the training dataset is generated in batches using dataset generation software based on Unity 3D or Unreal Engine to generate rendering samples and corresponding labels.

[0078] In this embodiment, the R, G, and B channels of the image are first normalized, and the calculation method is as follows: The mean parameters are (0.5, 0.5, 0.5), and the standard deviation parameters are (0.5, 0.5, 0.5). Geometric transformations and mosaic data augmentation are used to randomly rotate and translate the dataset, and to change the brightness and saturation of the images. In particular, if geometric transformations are performed on the input images, the confidence maps and affinity maps corresponding to that data need to undergo the same transformations to ensure data consistency.

[0079] S12, input the RGB image obtained in step S11 into the feature extraction network to obtain a high-dimensional feature map of size 128×50×50;

[0080] An RGB image is input into a feature extraction network to obtain a low-dimensional feature map. When the low-dimensional feature map passes through each feature module in the feature extraction network, channel-level up-dimensional convolution, channel-level attention filtering, and channel-level down-dimensional convolution are performed. During the forward propagation process, the low-dimensional feature map is fused with the deep features output by the bypass.

[0081] S13, input the high-dimensional feature map obtained in step S12 into the convolutional pose network module to obtain 9 confidence maps and 16 affinity maps corresponding to the two-dimensional key points of the bolt target. The position of the peak in the confidence map is the candidate key point. The 9 key points of the bolt target correspond to one confidence map, and the 8 key points of the bolt target other than the center point correspond to two affinity maps. The confidence map and the affinity map together serve as the two-dimensional key point features extracted from the RGB image.

[0082] When a high-dimensional feature map is input into the convolutional pose network module, the convolutional pose network module calculates the network loss using the true values ​​of the two-dimensional keypoint confidence map generated by the pose label and the true values ​​of the affinity map. The calculated network loss is then backpropagated and the network parameters are updated.

[0083] Specifically, the values ​​in the confidence map represent the probability that an object keypoint appears at that location. The predicted result of the confidence map is a 9×50×50 vector, corresponding to 9 different keypoints, with each keypoint having a confidence map resolution of 50. The affinity map is a feature representation of the relationship between a keypoint and its corresponding bolt center point, used to determine the object affiliation of the detected keypoint. The predicted result of the affinity map is a 16×50×50 vector, with the 8 keypoints (excluding the center point) each corresponding to two affinity maps, with each keypoint having two affinity maps with a resolution of 50. During network training, the output of each stage of the convolutional pose network is used to calculate the network loss. In the actual detection process, the output of the 5th stage of the network serves as the final prediction result of the confidence map and affinity map.

[0084] During network training, the ground truth values ​​of the bolt target's 2D keypoint confidence map are generated based on the positions of the bolt target's 2D keypoints. The generation method is as follows:

[0085] The initial true values ​​of the confidence graph for the k-th keypoint are:

[0086]

[0087] make This indicates that the coordinates (i,j) are relative to the k-th key point (i,j) of the n-th object. (k,n) ,j (k,n) The correlation magnitude of ) is determined by ε = 8, which is a hyperparameter; let Traversing the key point confidence graph M k blf Elements in;

[0088] Repeat the above steps until the true values ​​M1 of the confidence plot for the 9 key points are obtained. blf M2 blf …M9 blf ;

[0089] During network training, the affinity map is a feature representation of a 2D keypoint of a bolt target with respect to the 2D center point of the bolt target to which that keypoint belongs. The method for constructing the true value of the affinity map is as follows:

[0090] The initial values ​​of the two keypoint affinity maps corresponding to the k-th keypoint are:

[0091]

[0092] Obtain the point set consisting of the k-th keypoint of all objects:

[0093]

[0094] in, Let represent the coordinates of the k-th key point of the n-th object. The coordinates of the object's center point are represented; the affinity diagram is calculated as follows;

[0095]

[0096]

[0097] Repeat the above steps to obtain a total of 16 true affinity maps.

[0098] The network loss during training is calculated from the confidence maps and affinity maps of the five outputs of the convolutional pose network module. The method for calculating the network loss is as follows:

[0099]

[0100] in, Represent the confidence map and affinity field loss on the q-th convolutional pose network module, respectively. For each convolutional pose network module, and The calculation method is as follows:

[0101]

[0102]

[0103]

[0104]

[0105] in, Let represent the network's predicted value and the actual value of the confidence graph for the k-th key point, respectively. Let represent the predicted and actual values ​​of the affinity graph corresponding to the k-th keypoint, respectively. This represents the predicted values ​​of the two affinity maps corresponding to the k-th keypoint. This represents the true values ​​of the two affinity maps corresponding to the k-th keypoint, where k = 1, 2, ..., 9, and γ = 50 is the resolution of the confidence map and affinity map. This represents the rotation of the k-th 3D keypoint of the object around the bolt target's |θ| axis of symmetry. The confidence graph corresponding to the projection of the value λ times onto a two-dimensional plane. This represents the rotation of the k-th 3D keypoint of the object around the bolt target's |θ| axis of symmetry. After multiplying by λ, the corresponding two affinity maps are projected onto a two-dimensional plane. For the bolt target, the symmetry axis weight |θ|=6, and the confidence map output from the last stage is taken as the result. The location where the confidence peak appears on this map is marked as a candidate key point;

[0106] S14, Combining the affinity map obtained in step S13, a heuristic algorithm is used to search for the candidate key points to obtain the two-dimensional key point coordinates of the bolt target;

[0107] Specifically, the process of heuristically searching for candidate key points is as follows:

[0108] Obtain the set of all m candidate points for the prediction result of the convolutional pose network for the k-th keypoint:

[0109]

[0110] And their eigenvectors at the corresponding positions in the affinity map prediction results:

[0111]

[0112] in, The eigenvalues ​​representing the coordinates of the m-th candidate point at their corresponding positions in the two affinity maps. The eigenvectors formed.

[0113] For the nth object, search for the coordinates of its kth key point in the candidate point set. The process is as follows:

[0114]

[0115] in, This represents the prediction result of the convolutional pose network for the center point of the nth object, where δ = 0.5 is a hyperparameter.

[0116] Repeat the above steps until the coordinates of k key points of all n bolt targets are obtained;

[0117] S15: Solve the 2D-3D correspondence between the 2D key point coordinates obtained in S14 and the known bolt CAD model using the direct linear transformation algorithm to obtain the 6-DOF pose of the bolt target. The 6-DOF pose of the bolt target is represented by the rotation matrix and translation matrix of the bolt target relative to the camera coordinate system, thus completing the bolt pose estimation based on monocular RGB image.

[0118] Specifically, after obtaining the coordinates of the object's two-dimensional key points, a direct transformation algorithm is used to solve for the pose transformation relationship between the object and the camera coordinate system. The mapping relationship between the three-dimensional control points on the object and their two-dimensional projection points can be expressed as:

[0119] ωP=KP c =K[R cw |t cw ]P w

[0120] Where ω is the depth of the control point in the camera coordinate system, P = (u, v, 1) T P represents the control point in the pixel-normalized coordinate system. c =(X C ,Y C Z C ) T P w =(X W ,Y W Z W ,1) T , respectively, represent the homogeneous coordinates of the control points in the camera coordinate system and the world coordinate system. K is the camera's intrinsic parameter matrix, R cw ,t cw This involves rotation and translation transformations from the world coordinate system to the camera coordinate system. During the experiment, a 3D model of the object is created using a scanning device to obtain its CAD model. The origin of the world coordinate system is then aligned with the center of the object's bounding box, resulting in nine sets of control points for the object in the world coordinate system.

[0121] For a pair of known 2D-3D control points (P1, P2) W The mapping relationship between its three-dimensional control points and two-dimensional projection points can be expressed as:

[0122] ωK -1 P1 = P1 c =[R cw |t cw P1 w

[0123] make

[0124]

[0125] A set of homogeneous constraints can be obtained:

[0126]

[0127] For multiple pairs of 2D-3D control points, the equation can be expressed as:

[0128] A 2r×12 x 12x1 =0

[0129] Where r is the number of control point pairs. When r = 6, the transformation matrix T = [R] can be uniquely determined. cw |t cw In this invention application, a total of 9 sets of 2D-3D control points can be obtained. Since the exact solution to the equation cannot be obtained at this point, the least squares solution argmin||Ax is instead used. 2 || This serves as the result of attitude prediction.

[0130] Figure 2 This is a schematic diagram of the feature extraction network structure. The feature extraction network is used to extract features from the input RGB image. The feature extraction network consists of 5 feature modules M1, 5 feature modules M2, a bottom feature module, and a top feature module; CBS + CBS - The term "channel-upper-channel convolutional module" refers to the module where SE represents the channel attention module, CONV represents the convolutional layer, BN represents the batch normalization layer, and Swish represents the activation function layer. A positive sign indicates that the operation increases the dimensionality of the feature channels, while a negative sign indicates that it decreases the dimensionality of the feature channels. Multiple positive and negative signs indicate that the dimensionality increase or decrease scale of the module's channels is greater than that of a single sign. The channel-upper-channel convolutional module is composed of a concatenated convolutional layer, a batch normalization layer, and a Swish activation function, serving as the basic structure of the feature module. The channel attention module is used to perform global pooling on the current input feature map to obtain a 1×1 feature vector with multiple channels, and then multiplies the feature vector as channel weights with the current input feature map to achieve channel filtering.

[0131] Specifically, the feature module Bottom is the first feature module in the feature extraction network, and it is connected to the first feature module M1. Five feature modules M1 are stacked vertically, and each M1 is connected horizontally to a feature module M2. Each feature module M2 outputs deep feature information, which is element-wise added to the output of the corresponding feature module M1 to serve as the input feature map for the next feature module M1. The last feature module M1 is connected to the feature module Top, which serves as the terminal feature module of the feature extraction network. Some M1 modules contain downsampling convolutional layers with a stride of 2 to reduce the feature map size. The M2 module is only used for feature extraction and propagation and is not used as a downsampling module; its channel dimensionality scaling is larger than that of the M1 module.

[0132] The feature module M1 consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 1-1, ascending and descending channel convolutional module 1-2, and ascending and descending channel convolutional module 1-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 1-1, ascending and descending channel convolutional module 1-2, channel attention module, and ascending and descending channel convolutional module 1-3.

[0133] The feature module M2 consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 2-1, ascending and descending channel convolutional module 2-2, and ascending and descending channel convolutional module 2-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 2-1, ascending and descending channel convolutional module 2-2, channel attention module, and ascending and descending channel convolutional module 2-3.

[0134] The feature module Bottom consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 3-1, ascending and descending channel convolutional module 3-2, and ascending and descending channel convolutional module 3-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 3-1, ascending and descending channel convolutional module 3-2, channel attention module, and ascending and descending channel convolutional module 3-3.

[0135] The feature module Top is an ascending and descending channel convolutional module;

[0136] The rising and falling channel convolutional layers 1-1, 2-1, and 3-1 are used to increase the number of feature channels, while the rising and falling channel convolutional layers 1-3, 2-3, and 3-3 are used to decrease the number of feature channels.

[0137] Specifically, in this embodiment, the Bottom module is the first feature module, with an input size of 400×400 and an output size of 200×200, and its hierarchical structure is as follows:

[0138] The first layer is a convolutional layer, using a 3*3 convolutional kernel, with 3 input channels, 32 output channels, a stride of 2, and a padding size of 1.

[0139] The second layer is the batch normalization and Swish activation function layer;

[0140] The third layer is a convolutional layer, using a 3*3 convolutional kernel, with 32 input channels, 32 output channels, a stride of 1, and a padding size of 1.

[0141] The fourth layer is the batch normalization and Swish activation function layer;

[0142] Layer 5 is the channel attention layer;

[0143] The 6th layer is a convolutional layer, using a 3*3 convolutional kernel, with 32 input channels, 24 output channels, a stride of 1, and a padding size of 1.

[0144] Layer 7 is the batch normalization and Swish activation function layer;

[0145] In this embodiment, the M1 modules have the same structure, but the input and output dimensions of M1 modules at different locations differ. Taking the first M1 module as an example, its input size is 200×200, its output size is 200×200, and its hierarchical structure is as follows:

[0146] The first layer is a convolutional layer with a 1*1 convolutional kernel, 24 input channels, 144 output channels, a stride of 1, and a padding size of 0.

[0147] The second layer is the batch normalization and Swish activation function layer;

[0148] The third layer is a convolutional layer, using a 5*5 convolutional kernel, with 144 input channels, 144 output channels, a stride of 1, and a padding size of 2.

[0149] The fourth layer is the batch normalization and Swish activation function layer;

[0150] Layer 5 is the channel attention layer;

[0151] The 6th layer is a convolutional layer, using a 1*1 convolutional kernel, with 144 input channels, 40 output channels, a stride of 1, and a padding size of 0.

[0152] Layer 7 is the batch normalization and Swish activation function layer;

[0153] Specifically, when the M1 module is used as a downsampling module, its third convolutional layer has a stride of 2, and the output size is reduced to half the input size;

[0154] In this embodiment, the M2 modules have the same structure, but the input and output sizes differ at different locations. Taking the first M2 module in the feature extraction network as an example, its input size is 200×200, its output size is 200×200, and its hierarchical structure is as follows:

[0155] The first layer is a convolutional layer with a 1*1 convolutional kernel, 40 input channels, 240 output channels, a stride of 1, and a padding size of 0.

[0156] The second layer is the batch normalization and Swish activation function layer;

[0157] The third layer is a convolutional layer, using a 5*5 convolutional kernel, with 240 input channels, 240 output channels, a stride of 1, and a padding size of 2.

[0158] The fourth layer is the batch normalization and Swish activation function layer;

[0159] Layer 5 is the channel attention layer;

[0160] The 6th layer is a convolutional layer, using a 1*1 convolutional kernel, with 240 input channels, 40 output channels, a stride of 1, and a padding size of 0.

[0161] Layer 7 is the batch normalization and Swish activation function layer;

[0162] In this embodiment, the Top module is the last feature module, with an input size of 50×50 and an output size of 50×50. Its hierarchical structure is as follows:

[0163] The first layer is a convolutional layer, using a 1*1 convolutional kernel, with 240 input channels, 128 output channels, a stride of 1, and a padding size of 0.

[0164] The second layer is the batch normalization and Swish activation function layer;

[0165] Figure 3This is a diagram of the convolutional pose network structure. The input to the convolutional pose network is a high-dimensional feature map of size 128×50×50 output from the feature extraction network, used to predict the confidence map and affinity map of the two-dimensional key points of the bolt target. The convolutional pose network module includes five key point prediction modules: Key Point Prediction Module 1, Key Point Prediction Module 2, Key Point Prediction Module 3, Key Point Prediction Module 4, and Key Point Prediction Module 5. Key Point Prediction Module 1 includes two prediction channels, used to predict the two-dimensional key point confidence map and affinity map respectively, and each prediction channel includes two feature modules M1. Key Point Prediction Module 2 also includes two prediction channels, used to predict the two-dimensional key point confidence map and affinity map respectively, and each prediction channel includes four feature modules M1. Key Point Prediction Modules 2, 3, 4, and 5 have the same structure.

[0166] like Figure 4 As shown, this embodiment also provides an electronic device, including a memory 401 and a processor 402. The memory 401 stores a computer program, and the processor 402 is used to run the computer program and execute any step in the above-described technical method embodiments, and implement the bolt posture detection method based on monocular RGB images in the embodiments. Furthermore, the above-described electronic device also includes a transmission device 403, which receives and sends data via a wired or wireless network; computer peripherals and image acquisition devices 404 represent devices such as displays, mice, keyboards, and cameras used for interaction between the computer and operators / environment.

[0167] Furthermore, those skilled in the art should understand that the embodiments involved in this application are merely one implementation of the present invention and should not be construed as limiting the present invention. The various technical features therein can be combined arbitrarily, and as long as the combination of these technical features does not contradict each other, they should all be considered to be within the scope of protection of this application.

Claims

1. A bolt pose estimation method based on monocular RGB image, characterized in that The steps of this method include: S11. Acquire several RGB images containing bolt targets. At the same time, use the rotation matrix, translation matrix, and corresponding two-dimensional key point coordinates of each bolt in the RGB images containing bolt targets relative to the camera coordinate system as the true pose values ​​and save them in the form of image-pose label pairs. S12, input the RGB image obtained in step S11 into the feature extraction network to obtain a high-dimensional feature map; S13, input the high-dimensional feature map obtained in step S12 into the convolutional pose network module to obtain 9 confidence maps and 16 affinity maps of the two-dimensional key points corresponding to the bolt target. The position of the peak in the confidence map is the candidate key point. S14, Combine the affinity map obtained in step S13 to search for the candidate key points and obtain the two-dimensional key point coordinates of the bolt target; S15, the coordinates of the two-dimensional key points of the bolt target found in step S14 are compared with the 2D-3D correspondence of the key points of the bolt target by solving the known CAD model of the bolt to be detected, so as to obtain the 6-DOF pose of the bolt target. The 6-DOF pose of the bolt target is represented by the rotation matrix and translation matrix of the bolt target relative to the camera coordinate system, thus completing the bolt pose estimation based on monocular RGB image. In step S14, a heuristic search is performed on the candidate key points, and the candidate key points are assigned to their corresponding bolt targets. The heuristic search method is as follows: obtain a first candidate key point set of all key point prediction results of the convolutional pose network module on the first image ​​ Obtain the feature vectors of the candidate keypoints at the corresponding positions in the affinity map prediction results: wherein, represents the feature vector composed of the feature values of the corresponding positions of the candidate point coordinates in the two affinity maps , ​​ For the first bolt target, the first keypoint coordinates are: wherein, represents a prediction result of the convolution pose network module on the i-th screw target center point, is a hyperparameter; Repeat the above steps until all key point coordinates of the screw targets are obtained; In step S13, the network loss during training is calculated from the confidence map and affinity map of the five outputs of the convolutional pose network module. The method for calculating the network loss is as follows: The network loss function is expressed as: wherein, , respectively represent the confidence map and affinity field loss on the qth convolutional pose network module, for each convolutional pose network module, and are calculated in the following manner: in, , They represent the network for the first... The predicted values ​​and actual values ​​of the confidence plots for each key point; , , respectively representing the network for the . The predicted and actual values ​​of the affinity maps corresponding to each key point. , Indicates the first The predicted values ​​of the two affinity maps corresponding to each key point. , Indicates the first The true values ​​of the two affinity maps corresponding to each key point , For the resolution of the confidence map and affinity map, Indicates the first object 3D key points around the bolt target Rotation along the axis of symmetry of The confidence graph corresponding to the projection onto the two-dimensional plane is then multiplied. Indicates the first object 3D key points around the bolt target Rotation along the axis of symmetry of After multiplication, the two affinity maps corresponding to the projection onto the two-dimensional plane are obtained. For the bolt target, the symmetry axis multiplicity is... The confidence map output from the last stage is taken as the result, and the location where the confidence peak appears on the confidence map output from the last stage is marked as a candidate key point.

2. The bolt attitude estimation method based on monocular RGB images according to claim 1, characterized in that: In step S11, 8k-10k images with a resolution of RGB images containing bolt targets, each containing no more than 10 bolt targets of random number and pose, are used for network training; In step S12, the method for inputting the acquired RGB image into the feature extraction network to obtain a high-dimensional feature map is as follows: An RGB image is input into a feature extraction network to obtain a low-dimensional feature map. When the low-dimensional feature map passes through each feature module in the feature extraction network, it undergoes channel-level dimensionality-up convolution, channel-level attention filtering, and channel-level dimensionality-down convolution. During the forward propagation process, the low-dimensional feature map is fused with the deep features output from the bypass.

3. A bolt attitude estimation method based on monocular RGB images according to claim 1 or 2, characterized in that: The feature extraction network consists of 5 feature modules M1, 5 feature modules M2, a feature module Bottom, and a feature module Top. The feature module M1 consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 1-1, ascending and descending channel convolutional module 1-2, and ascending and descending channel convolutional module 1-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 1-1, ascending and descending channel convolutional module 1-2, channel attention module, and ascending and descending channel convolutional module 1-3. The feature module M2 consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 2-1, ascending and descending channel convolutional module 2-2, and ascending and descending channel convolutional module 2-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 2-1, ascending and descending channel convolutional module 2-2, channel attention module, and ascending and descending channel convolutional module 2-3. The feature module Bottom consists of three ascending and descending channel convolutional modules connected in series with one channel attention module. The three ascending and descending channel convolutional modules are ascending and descending channel convolutional module 3-1, ascending and descending channel convolutional module 3-2, and ascending and descending channel convolutional module 3-3. The order of the three ascending and descending channel convolutional modules connected in series with the one channel attention module is ascending and descending channel convolutional module 3-1, ascending and descending channel convolutional module 3-2, channel attention module, and ascending and descending channel convolutional module 3-3. The feature module Top is an ascending and descending channel convolutional module; The method for constructing the feature extraction network is as follows: The feature module Bottom is the first feature module in the feature extraction network, and it is connected to the first feature module M1. The five feature modules M1 are stacked vertically, and each feature module M1 is connected to a feature module M2 horizontally. Each feature module M2 outputs deep feature information, and the output deep feature information is element-wise added to the output of the corresponding feature module M1 to serve as the input feature map for the next feature module M1. The last feature module M1 is connected to the feature module Top, which serves as the terminal feature module of the feature extraction network.

4. The bolt attitude estimation method based on monocular RGB images according to claim 3, characterized in that: The ascending and descending channel convolutional module consists of a concatenated convolutional layer, a batch normalization layer, and a Swish activation function, serving as the basic structure of the feature module. The channel attention module is used to perform global pooling on the current input feature map to obtain a feature vector with a size of 1×1 channels, and then multiply the feature vector as channel weights with the current input feature map to achieve channel filtering effect; The rising and falling channel convolution modules 1-1, 2-1, and 3-1 are used to increase the number of feature channels, while the rising and falling channel convolution modules 1-3, 2-3, and 3-3 are used to decrease the number of feature channels.

5. The bolt attitude estimation method based on monocular RGB images according to claim 1, characterized in that: In step S13, each of the nine key points of the bolt target corresponds to a confidence map, and each of the eight key points of the bolt target other than the center point corresponds to two affinity maps. The confidence maps and affinity maps together serve as two-dimensional key point features extracted from the RGB image. When a high-dimensional feature map is input into the convolutional pose network module, the convolutional pose network module calculates the network loss using the true values ​​of the two-dimensional keypoint confidence map generated by the pose label and the true values ​​of the affinity map. The calculated network loss is then backpropagated and the network parameters are updated. The convolutional pose network module includes five keypoint prediction modules: keypoint prediction module 1, keypoint prediction module 2, keypoint prediction module 3, keypoint prediction module 4, and keypoint prediction module 5. Keypoint prediction module 1 includes two prediction channels, used to predict the two-dimensional keypoint confidence map and affinity map respectively, and each prediction channel includes two feature modules M1. Keypoint prediction module 2 also includes two prediction channels, used to predict the two-dimensional keypoint confidence map and affinity map respectively, and each prediction channel includes four feature modules M1. Keypoint prediction modules 2, 3, 4, and 5 have the same structure. The high-dimensional feature maps are input into the five keypoint prediction modules, and each module outputs a size of [size missing]. The confidence plot and dimensions are The affinity map is obtained by adding the output of the first keypoint prediction module and the high-dimensional feature map output by the feature extraction network element-wise, and using them together as the input features of the next keypoint prediction module. The confidence map output by the last keypoint prediction module and the affinity map are used as the final prediction result. The confidence diagram represents the probability distribution of the 2D key point locations of the bolt target.

6. The bolt attitude estimation method based on monocular RGB images according to claim 5, characterized in that: The true values ​​of the confidence map of the bolt target's two-dimensional key points are generated based on the positions of the bolt target's two-dimensional key points. The generation method is as follows: Initialize the first The true values ​​of the confidence plot for each key point are: ； make Representing coordinates Place and No. The first bolt target Key points The magnitude of the correlation, Let be a hyperparameter; Traverse the key point confidence graph Elements in; Repeat the above steps until the true values ​​of the confidence plot for the 9 key points are obtained. ; The affinity map is a feature representation of a two-dimensional key point of a bolt target with respect to the two-dimensional center point of the bolt target to which the two-dimensional key point belongs. The method for constructing the true value of the affinity map is as follows: Initialize the first The true values ​​of the affinity maps for the two keypoints corresponding to each keypoint are: ， ; Obtain the first of all bolt targets A set of points consisting of key points: in, Indicates the first The first bolt target Let the coordinates of the key points be... Indicates the first The coordinates of the center point of each bolt target, and the true value of the affinity diagram are expressed as follows: Repeat the above steps to obtain a total of 16 true affinity maps. .

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the bolt attitude estimation method based on monocular RGB images as described in any one of claims 1-6.

8. A computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the bolt attitude estimation method based on monocular RGB images as described in any one of claims 1-6.

Images