Target attitude estimation method based on binocular vision platform

Abstract

The embodiment of the invention provides a target attitude estimation method based on a binocular vision platform. The method comprises the steps of completing key point detection of a target object; and completing the three-dimensional attitude estimation of the target. By applying the method, the detection capability of key point detection can be ensured under the condition that the target scale changes greatly, and real-time detection can be achieved through reasoning framework optimization. Secondly, a left-eye-right-eye key point matching and screening algorithm is designed, the detection correctness of the screened key points can be rechecked, and the key points detected correctly are selected to complete three-dimensional reconstruction.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target posture estimation method based on a binocular vision platform. Background Art

[0002] With the development of computer vision technology, the application of target pose estimation is becoming more and more extensive. In the military field, target pose estimation technology can support the recognition and intention judgment of frogman targets, thereby improving the target situation awareness capability.

[0003] Posture estimation is generally divided into two parts: key point detection and key point 3D reconstruction. Key point detection often relies on deep learning neural networks, but the existing key point detection networks mainly consider the network's feature extraction capabilities and the generalization capabilities of the network model. Few detection networks focus on the impact of target scale changes on detection results. However, in the process of target key point recognition, the scale of the target object in the video stream changes greatly under many conditions, which reduces the recognition accuracy. Therefore, it is necessary to design a network structure to effectively deal with the negative impact of the scale change of the target object on the detection accuracy.

[0004] The three-dimensional reconstruction based on binocular cameras basically uses traditional mathematical modeling to complete the three-dimensional pose estimation of key points. The binocular stereo vision system can be composed of two cameras arranged left and right. By observing the same object, two images with different perspectives can be obtained. Then, by using triangulation, the object's obvious depth sense is obtained to complete the reconstruction. The accuracy of reconstruction mainly depends on the detection accuracy and matching accuracy of the left and right eye key point detection. Existing reconstruction methods rarely consider the matching accuracy problem. Therefore, it is necessary to design a key point matching algorithm to complete the left-right eye key point matching screening task to complete the reconstruction. Summary of the invention

[0005] The purpose of the present invention is to provide a target posture estimation method based on a binocular vision platform to solve the above problems.

[0006] An embodiment of the present invention provides a method for estimating a target posture based on a binocular vision platform, the method comprising:

[0007] Step 1, complete the key point detection of the target object;

[0008] Step 2: Complete the 3D pose estimation of the target.

[0009] In some embodiments, the step 1, completing key point detection of the target object, includes:

[0010] Step 1.1, complete the data set collection and production;

[0011] Step 1.2, design the key point detection network encoder part based on multi-resolution fusion;

[0012] Step 1.3, design the key point detection network decoder part based on multi-resolution fusion;

[0013] Step 1.4, complete the key point detection model training of the target;

[0014] Step 1.5, optimize the model inference speed.

[0015] In some embodiments, the step 1.1, completing the data set collection and production, includes:

[0016] Use cameras to collect data from multiple angles and scales on objects that require key point recognition, and complete manual labeling of key points.

[0017] In some embodiments, the step 1.2, designing a key point detection network encoder part based on multi-resolution fusion, includes:

[0018] In the encoder, the input image is first converted into a 1 / 4 feature map through a convolution operation;

[0019] The first stage generates two resolution channels, 1 / 4 and 1 / 8 in size, and each channel is connected to 4 residual blocks to enhance feature extraction;

[0020] Stage 2 produces three resolution channels, adding a smaller 1 / 16 size channel to the two channels retained in stage 1;

[0021] In the third stage, a fourth channel is added based on the second stage, i.e., a channel with a resolution of 1 / 32. The convolution widths of the four resolutions are C, 2C, 4C, and 8C, respectively, where C is 1 / 4 of the width of the feature map.

[0022] Finally, the feature maps of the four channels are fused as the output of the encoder and the input of the decoder;

[0023] In the encoder process, multi-scale feature fusion is fully completed to improve the scale adaptability of the model.

[0024] In some embodiments, the step 1.3, designing a key point detection network decoder part based on multi-resolution fusion, includes:

[0025] The decoder consists of two parts, one for each of the two outputs used to calculate the final detection result;

[0026] Both outputs are upsampled to the resolution of the encoder input image using bilinear interpolation, and the heatmaps at all scales are averaged for the final keypoint prediction;

[0027] The merged heatmap is used to predict keypoint coordinates and calculate confidence;

[0028] The final output of the entire network is a matrix composed of key point coordinates and key point detection confidence.

[0029] In some embodiments, the step 1.4, completing the key point detection model training of the target, includes:

[0030] Use labeled datasets for neural network training;

[0031] Set the training parameters, including the number of iterations and learning rate. When the loss function converges successfully, the key point detection model is obtained.

[0032] Use the test set for testing. When the test evaluation value is higher than the set threshold, the model training is considered successful.

[0033] In some embodiments, the step 1.5, optimizing the model reasoning speed, includes:

[0034] Use the accelerated inference model framework to optimize training models, optimize neural network calculations, and generate a lightweight runtime engine to complete real-time inference tasks.

[0035] In some embodiments, the step 2, completing the three-dimensional posture estimation of the target, includes:

[0036] Step 2.1, design a left-eye-right-eye key point screening and matching algorithm;

[0037] Step 2.2, design a key point 3D pose estimation algorithm.

[0038] In some embodiments, the step 2.1, designing a left-eye-right-eye key point screening and matching algorithm, includes:

[0039] The left-right key point screening matching algorithm includes:

[0040] The target frame detection process completes the key point location distribution area information through the target detection algorithm;

[0041] Keypoint matching.

[0042] In some embodiments, the step 2.2, designing a key point three-dimensional pose estimation algorithm, includes:

[0043] A three-dimensional point is randomly initialized in space, and the reverse conversion is performed as before. This three-dimensional point is projected onto a two-dimensional plane. A projection point and a key point of detection appear on the two-dimensional plane. If the distance between the two can be optimized to the minimum, then the projected three-dimensional point is the optimal solution that can ensure that the two lines intersect in space. The LM method is used to complete the posture solution and obtain the three-dimensional posture of the key point.

[0044] The beneficial effects of the present invention are mainly reflected in that the detection capability of key point detection can be guaranteed when the target scale changes greatly, and after being optimized by the inference framework, real-time detection can be achieved. Secondly, a left-eye-right-eye key point matching screening algorithm is designed, which can review the correctness of the screening key point detection and select the key points with correct detection to complete the three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings illustrate generally, by way of example and not limitation, various embodiments discussed herein.

[0046] Figure 1 It is a schematic diagram of the technical implementation process of the target posture estimation method;

[0047] Figure 2 This is a diagram of the key point detection network model based on multi-resolution fusion;

[0048] Figure 3 The flowchart of the algorithm for left-eye and right-eye key point matching algorithm;

[0049] Figure 4 Schematic diagram of the three-dimensional pose estimation solution process. DETAILED DESCRIPTION

[0050] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0051] In the embodiments of the present application, it should be noted that, unless otherwise specified and limited, the term "connection" should be understood in a broad sense. For example, it can be an electrical connection or a connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0052] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0053] The following is combined with Figures 1 to 4 The embodiments of the present application are described in detail.

[0054] The present application embodiment provides a method for estimating target posture based on a binocular vision platform. The method flow is as follows: Figure 1 As shown, the following steps 1 and 2 are included.

[0055] Step 1: Complete key point detection of the target object.

[0056] Step 1.1: Complete data set collection and production.

[0057] Use cameras to collect data from multiple angles and scales on objects that need to be identified with key points, and use professional annotation software to complete manual annotation of key points. Divide the test set and training set into a ratio of 1:9.

[0058] Step 1.2: Design the encoder part of the keypoint detection network based on multi-resolution fusion.

[0059] In the encoder, the input image is first converted into a 1 / 4 feature map through a convolution operation. Stage 1 generates two resolution channels, 1 / 4 size and 1 / 8 size. Each channel is connected to 4 residual blocks to enhance feature extraction. Stage 2 generates three resolution channels, adding a smaller 1 / 16 size channel to the two channels retained in stage 1. Stage 3 adds a fourth channel on the basis of stage 2, that is, a channel with a resolution of 1 / 32. The convolution widths of the four resolutions are C, 2C, 4C, and 8C, where C is the width of the 1 / 4 feature map. Finally, the feature maps of the four channels are fused as the output of the encoder and the input of the decoder. In the process of the encoder, multi-scale feature fusion is fully completed to improve the scale adaptability of the model.

[0060] Step 1.3: Design the decoder part of the keypoint detection network based on multi-resolution fusion.

[0061] The decoder consists of two parts, one for each of the two outputs used to calculate the final detection result. The first part is a normal convolution operation. The output of the encoder part is used as the first output to undergo a convolution operation. It is connected to the output of the encoder as the input of the second part. The second part contains a deconvolution module and four residual blocks, which can generate high-quality, high-resolution feature maps. Using the feature map generated by the deconvolution module as the second output can improve the accuracy of the results, and its resolution is twice that of the decoder input. The two outputs are upsampled to the resolution of the encoder input image using bilinear interpolation, and the heat maps of all scales are averaged for final key point prediction. The merged heat map is used to predict the key point coordinates and calculate the confidence. The final output of the entire network is a matrix composed of the key point coordinates and the key point detection confidence. The specific model design ideas are as follows Figure 2 shown.

[0062] Step 1.4: Complete the key point detection model training of the target.

[0063] Use the labeled data set to train the neural network. Set reasonable training parameters, including the number of iterations, learning rate, etc. When the loss function successfully converges, the key point detection model is obtained. Use the test set for testing. When the test evaluation value is higher than the set threshold, the model training is considered successful.

[0064] Step 1.5: Optimize model inference speed.

[0065] Use the accelerated inference model framework to optimize training models, optimize neural network calculations, and generate a lightweight runtime engine to complete real-time inference tasks.

[0066] Step 2: Complete the 3D pose estimation of the target

[0067] Step 2.1: Design a left-eye-right-eye key point screening and matching algorithm.

[0068] The matching algorithm consists of two parts. The first part is the target frame detection process. The key point position distribution area information is completed through the target detection algorithm. The output format of the key point detection of the detected target is to output n sets of key point coordinates. In order to ensure the correctness of the key point detection, the detection frame and key point detection screening rules are set. The smallest horizontal coordinate and the largest horizontal coordinate of several detected key points are within the right shift Δx (horizontal error limit) of the detection frame detected by the right eye, and the smallest vertical coordinate and the largest vertical coordinate are within the height limit of the detection frame.

[0069] The target box matching process formula is as follows:

[0070]

[0071] Among them, the maximum value of the horizontal coordinate of the right target detection box Minimum value of the horizontal coordinate of the right target detection box The maximum vertical coordinate of the right target detection box The vertical coordinate of the right target detection box Minimum value of the horizontal coordinate of the left eye key point detection The maximum value of the horizontal coordinate of the left eye key point detection Minimum vertical coordinate value of left eye key point detection The maximum value of the left eye key point detection vertical coordinate Horizontal error limit Δx; vertical error limit Δy.

[0072] The second part is key point matching. The target detection model can output the coordinates of the key points and the confidence of the detection results. The higher the confidence, the higher the prediction accuracy of the key point. In order to eliminate the key points that are detected by mistake, the key point detection threshold c is set, and only the key points whose prediction confidence is above the confidence and threshold are retained. When the key point exists in both the left and right eyes, in order to further ensure the position detection accuracy of the key point, the judgment conditions are set. When the coordinate relationship of the key points with the same label for the left and right eyes can ensure that the vertical error is within Δy (vertical error limit, which is an empirical parameter) and the horizontal coordinate is within Δx (horizontal error limit, which is an empirical parameter), the pairing can be completed and the matching result can be output finally. The overall key point matching screening process is as follows: Figure 3 shown.

[0073] The key point matching process formula is as follows:

[0074]

[0075] Among them, the left and right eye key point detection results (x-coordinate, y-coordinate and confidence level respectively); vertical error limit Δy; key point confidence threshold c.

[0076] Step 2.2: Design a keypoint 3D pose estimation algorithm.

[0077] The three-dimensional posture estimation method of the binocular camera is triangulation reconstruction. For the correlation results of two or more known images, the position of the three-dimensional point is restored through mathematical calculation. In the process of three-dimensional reconstruction, there will often be a certain amount of noise, which will have a certain impact on the two-dimensional imaging, causing deviations in the two-dimensional presentation, that is, the lines connecting the left and right eyes of the camera optical center often do not intersect in space. In order to find reasonable three-dimensional points as much as possible, a three-dimensional point is randomly initialized in space, and this three-dimensional point is projected onto a two-dimensional plane. A projection point and a key point of detection appear on the two-dimensional plane. The distance between the two is optimized to be minimized, and the projected three-dimensional point is the optimal solution to ensure that the two lines intersect in space. The conversion process uses the LM method to complete the posture solution and obtain the three-dimensional posture of the key points. The algorithm solution process is as follows: Figure 4 shown.

[0078] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0079] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A target posture estimation method based on a binocular vision platform, characterized in that: The method comprises: Step 1, complete the key point detection of the target object; Step 2: Complete the 3D pose estimation of the target.

2. The target posture estimation method based on the binocular vision platform according to claim 1 is characterized in that: The step 1, completing the key point detection of the target object, includes: Step 1.1, complete the data set collection and production; Step 1.2, design the key point detection network encoder part based on multi-resolution fusion; Step 1.3, design the key point detection network decoder part based on multi-resolution fusion; Step 1.4, complete the key point detection model training of the target; Step 1.5, optimize the model inference speed.

3. The target posture estimation method based on the binocular vision platform according to claim 2 is characterized in that: The step 1.1, completing the data set collection and production, includes: Use cameras to collect data from multiple angles and scales on objects that require key point recognition, and complete manual labeling of key points.

4. The target posture estimation method based on the binocular vision platform according to claim 2 is characterized in that: The step 1.2, designing the key point detection network encoder part based on multi-resolution fusion, includes: In the encoder, the input image is first converted into a 1 / 4 feature map through a convolution operation; The first stage generates two resolution channels, 1 / 4 and 1 / 8 in size, and each channel is connected to 4 residual blocks to enhance feature extraction; Stage 2 produces three resolution channels, adding a smaller 1 / 16 size channel to the two channels retained in stage 1; In the third stage, a fourth channel is added based on the second stage, i.e., a channel with a resolution of 1 / 32. The convolution widths of the four resolutions are C, 2C, 4C, and 8C, respectively, where C is 1 / 4 of the width of the feature map. Finally, the feature maps of the four channels are fused as the output of the encoder and the input of the decoder; In the encoder process, multi-scale feature fusion is fully completed to improve the scale adaptability of the model.

5. The target posture estimation method based on the binocular vision platform according to claim 2 is characterized in that: The step 1.3, designing a key point detection network decoder part based on multi-resolution fusion, includes: The decoder consists of two parts, one for each of the two outputs used to calculate the final detection result; Both outputs are upsampled to the resolution of the encoder input image using bilinear interpolation, and the heatmaps at all scales are averaged for the final keypoint prediction; The merged heatmap is used to predict keypoint coordinates and calculate confidence; The final output of the entire network is a matrix composed of key point coordinates and key point detection confidence.

6. The target posture estimation method based on the binocular vision platform according to claim 2 is characterized in that: The step 1.4 completes the key point detection model training of the target, including: Use labeled datasets for neural network training; Set the training parameters, including the number of iterations and learning rate. When the loss function converges successfully, the key point detection model is obtained. Use the test set for testing. When the test evaluation value is higher than the set threshold, the model training is considered successful.

7. The target posture estimation method based on the binocular vision platform according to claim 2 is characterized in that: The step 1.5, optimizing the model reasoning speed, includes: Use the accelerated inference model framework to optimize training models, optimize neural network calculations, and generate a lightweight runtime engine to complete real-time inference tasks.

8. The target posture estimation method based on binocular vision platform according to claim 1, characterized in that: The step 2, completing the three-dimensional posture estimation of the target, includes: Step 2.1, design a left-eye-right-eye key point screening and matching algorithm; Step 2.2, design a key point 3D pose estimation algorithm.

9. The target posture estimation method based on the binocular vision platform according to claim 8 is characterized in that: The step 2.1, designing a left-eye-right-eye key point screening and matching algorithm, includes: The left-right key point screening matching algorithm includes: The target frame detection process completes the key point location distribution area information through the target detection algorithm; Keypoint matching.

10. The target posture estimation method based on binocular vision platform according to claim 8, characterized in that: The step 2.2, designing a key point three-dimensional posture estimation algorithm, includes: A three-dimensional point is randomly initialized in space, and the reverse conversion is performed as before. This three-dimensional point is projected onto a two-dimensional plane. A projection point and a key point of detection appear on the two-dimensional plane. If the distance between the two can be optimized to the minimum, then the projected three-dimensional point is the optimal solution that can ensure that the two lines intersect in space. The LM method is used to complete the posture solution and obtain the three-dimensional posture of the key point.

Images