A high-speed pose detection method based on vision optimization

Abstract

This invention discloses a high-speed pose detection method based on vision optimization. First, a QR code marker is affixed to the target to be detected, and a target tracking algorithm is used to locate the region of interest (ROI) where the target is located. Second, an image segmentation algorithm is used to segment the image within the ROI to extract the QR code markers. Finally, a global coordinate transformation model of the scene is constructed, and the pose information is calculated using the PnP algorithm based on the pixel coordinates of the QR code markers. The pose detection method proposed in this invention has high detection accuracy, good real-time performance, and high detection success rate and speed.

Description

Technical Field

[0001] This invention belongs to the field of visual pose detection, and particularly relates to a high-speed pose detection method optimized using visual algorithms. Background Technology

[0002] Visual sensor-based pose detection is a common method for mobile robot pose detection. Compared to other sensors, visual sensors can acquire rich image information, such as color, texture, and depth, providing diverse information for the mobile robot's environmental perception and autonomous navigation. Furthermore, visual sensors can acquire images with pixel-level resolution, allowing visual algorithms to process images with high precision, thus offering advantages in accuracy. To further improve the usability of visual detection methods, visual markers can be used to assist detection. Since the features of markers are definite, they are easier to detect and recognize. Indirectly detecting target pose through marker detection simplifies the feature extraction and matching process of the detection algorithm while improving robustness. In general, visual detection methods have advantages such as low cost, light weight, high accuracy, and rich information. However, visual methods also have some drawbacks. For example, the shape, color, and texture of objects change under different lighting conditions, affecting the accuracy and robustness of visual detection methods. Additionally, handling complex scenes requires highly complex algorithms and significant computational resources.

[0003] Traditional methods for visual pose detection typically rely on manual feature engineering for image feature extraction and matching, such as SIFT (Scale-invariant Feature Transform) and HOG (Histogram of Oriented Gradient). These methods are relatively mature, offer good interpretability, and are widely used in many scenarios. However, they are less robust to factors such as lighting, texture, and viewpoint, and are prone to errors in complex environments. In recent years, deep learning-based visual pose detection methods have gradually become a research hotspot. Deep learning-based methods generally do not require manual design of feature extraction and matching algorithms. Therefore, for new tasks and scenarios, only dataset collection and model training are needed to achieve good results. They also exhibit good robustness and generalization ability in handling complex and variable scenes. However, deep learning-based methods require a large amount of labeled data and computational resources. Therefore, in scenarios with limited data or computational resources, their performance may be inferior to traditional methods. Furthermore, deep learning models have weak interpretability, making it difficult to explain and adjust the model's decisions. In conclusion, how to optimize visual pose detection methods, especially by combining the advantages of traditional methods and deep learning methods to improve detection performance, is a topic worthy of exploration.

[0004] Research on visual marker detection optimization reveals two main approaches: one is to enhance marker recognizability by customizing their shape, and the other is to improve the algorithm's discriminative power by combining new models and algorithms. The former can significantly improve detection performance in specific scenarios, but may not be applicable outside of those scenarios, lacking universality. The second approach, however, enhances the discriminative power of the visual marker detection algorithm itself, especially visual algorithms combined with deep learning. These algorithms can automatically learn marker features through model training, enabling successful recognition of visual markers across multiple scenarios. However, deep learning algorithms require significant computational power. For real-time tasks like pose detection, a balance between accuracy and speed must be carefully considered when applying deep learning algorithms. This invention combines a target tracking algorithm based on a deep learning framework to enhance the target discrimination ability of the pose detection method. Simultaneously, it optimizes the tracker network structure to meet the requirements of high-speed detection while maintaining detection accuracy. Summary of the Invention

[0005] Purpose of the invention: To address the problems of target loss and low efficiency in traditional pose detection methods, a pose detection method based on visual algorithm optimization is proposed. By shrinking the detection ROI through the target tracking algorithm, the amount of input data of the pose detection algorithm is reduced, thus obtaining a high-speed pose detection method.

[0006] Technical Solution: To achieve the objectives of this invention, the technical solution adopted is: a high-speed pose detection method based on vision optimization, comprising the following steps:

[0007] The first step is to attach an ArUco code to the target to be detected, and then suspend a camera above the target as a global vision sensor to collect images of the target in real time.

[0008] The second step is to select the first frame image of the target as the template image. Subsequent search images are then used as a reference for feature matching. The search images and template images are used as inputs to the target tracking algorithm to locate the ROI of the target in the search images.

[0009] The third step is to use the ROI obtained by the target tracking algorithm as the detection area for pose detection, and use the image segmentation algorithm to segment the QR code marker in the ROI to obtain the pixel coordinates of the four corner points of the marker.

[0010] The fourth step is to construct a global coordinate transformation model for the pose detection scene and use the PnP algorithm to calculate the pose information using the pixel coordinates marked by the QR code.

[0011] Furthermore, in the first step, an image is acquired using a Hikvision MV-CA013-20GC camera with a resolution of 1280 pixels × 1024 pixels and a maximum frame rate of 201 FPS. The camera is suspended above the target to be detected using a gate-shaped bracket as a global vision sensor.

[0012] Further, in the second step, the target tracking algorithm segments the search image and template image into image blocks and encodes them according to pixel channels. The encoding first undergoes a trainable linear mapping operation to obtain the encoding vector of the input image block. The encoding corresponding to each image block is added to the corresponding positional encoding on its original image to obtain the final image block encoding. The encoded image is then used as input to the target tracking algorithm. The input encoding achieves simultaneous feature extraction and relationship modeling of the template image and the search image through a cascaded Transformer encoder. To fully perform feature extraction and relationship modeling, the following equation is used:

[0013]

[0014] Where Q, K, and V represent the input query matrix, key matrix, and value matrix, respectively, all of which are derived from the input, d k The dimension representing the input is used in this calculation step to assign different weights to different inputs, enabling selective focus on input information. The subscripts z and x represent the matrix originating from the template and search region, respectively, and Z represents the encoder output. This invention uses a 12-layer stacked 8-head Transformer encoder as the backbone network of the target tracking model. Background removal strategies are incorporated into layers 4, 7, and 10 of the 12-layer cascaded encoder. During the inference process of the target tracking model, features with lower weights in the image patches are gradually discarded, causing the model to gradually discard background features and focus on target features. Simultaneously, the model's data dimensionality is significantly reduced, effectively improving inference speed.

[0015] Furthermore, the background removal strategy retains only the top k candidate vectors with the highest scores, discarding the remaining candidate vectors. Simultaneously, the positions of the retained feature images need to be recorded for subsequent image reconstruction and restoration. The candidate vector retention rate is set to ρ = k / n, where n is the total number of candidate vectors; in this invention, ρ = 0.7.

[0016] Furthermore, in the target tracking algorithm of the second step, a 4-layer cascaded fully convolutional neural network is used as the output head of the target tracking model. The score of each image patch in the search image is calculated based on its matching degree with the template image. The image patch with the highest score is the location of the target. A bounding box is used to represent the position and size of the target ROI. Therefore, the output of the output head is the similarity score of each image patch in the search image, as well as the center pixel coordinates and width and height of the bounding box containing the target, expressed as:

[0017] (x,y,w,h)=(x d +O(0,x d ,y d ),y d +O(1,x d ,y d ),S(0,x d ,y d ),S(1,x d ,y d ))

[0018] Where (x) d ,y d ) = argmax (x,y) P xy , A two-dimensional distribution plot representing the similarity scores between candidate search vectors and the target, with dimensions of [size missing]. H x W x Represents the height and width of the searched image, where P is the side length of the segmented pixel block, (x d ,y d ) represents the coordinates of the candidate vector with the highest score. Representative (x) d ,y d The local offset of ) These two variables, representing the normalized regression box size, are used to correct the position and size of the regression box, respectively, and both have dimensions of 1. It has one more dimension than P to represent the horizontal coordinate x and the vertical coordinate y respectively.

[0019] Furthermore, in the third step, the ROI obtained by the target tracking algorithm is used as the detection region for pose detection. An image segmentation algorithm is then used to segment the QR code marker within the ROI, obtaining the pixel coordinates of the four corner points of the marker. The specific steps of image segmentation mainly include the following:

[0020] (1) Threshold segmentation: The original color image is converted to grayscale, and noise is removed by smoothing filter. Then, a black and white image is obtained by threshold segmentation.

[0021] (2) Contour extraction: Contour extraction is performed using a contour recognition algorithm, and then a polygon approximation algorithm is used to find polygons in the image;

[0022] (3) Contour filtering: Polygons that are not rectangles from the polygons obtained in the previous step will be excluded. At the same time, a maximum and minimum threshold will be set, and polygons with excessively large or small side lengths will also be excluded. In addition, if the shortest average distance between the four corner points of two quadrilaterals is less than the smaller of their perimeter values, they are considered similar polygons. Among similar quadrilaterals, those with smaller perimeters will also be excluded. The rectangles that are finally retained are the candidate rectangles for the target ArUco code.

[0023] (4) Perspective transformation: The candidate rectangles obtained in the previous step are first subjected to perspective transformation to obtain the front view, and then gridded to obtain the front view of the coded image;

[0024] (5) Encoding Recognition: The number of black and white pixels within each grid is counted to correspond to an encoding of 0 or 1. Grids with more black pixels correspond to 0, and those with fewer black pixels correspond to 1. The order of corner points is further determined through binary encoding information, and the encoding is matched with the dictionary. The ID value corresponding to the successfully matched encoding is returned.

[0025] (6) Corner point adjustment: In order to obtain the pixel coordinates of the four corner points of ArUco more accurately, the sub-pixel interpolation method is used to locate the corner points more precisely.

[0026] After obtaining the pixel coordinates of the four corner points of the QR code marker in the ROI, a coordinate transformation is performed to restore the pixel coordinates of the four corner points in the original image. Let the pixel coordinates of the top-left corner of the bounding box returned by the target tracking algorithm be (x...). t ,y t After image segmentation, the pixel coordinates of a corner point of the QR code marker are (x...). s ,y s If the pixel coordinates of the corner point in the original image are (x, y), then the final pixel coordinates of the corner point in the original image are (x, y). t +x s ,y t +y s ).

[0027] Furthermore, in the fourth step, a global coordinate transformation model for the pose detection scene is constructed by converting image pixel coordinates to world coordinates in the world coordinate system. The world coordinate system takes the ground plane as the starting point of the Z-axis, and the transformation relationship is as follows:

[0028]

[0029] Where K1 represents the camera intrinsic parameter, (x w ,y w ,z w ) and (x c ,y c ,z cLet (u, v) represent the coordinates of a point in space in the world coordinate system and the camera coordinate system, respectively, and (u, v) be its image pixel coordinates. Since the intrinsic parameter matrix is ​​invertible, the world coordinates of the target in the world coordinate system can be further obtained:

[0030]

[0031] Furthermore, in the fourth step, the PnP algorithm is used to calculate the pose information using the pixel coordinates marked by the QR code. After expanding the intrinsic and extrinsic parameter matrices, the transformation relationship from pixel coordinates to world coordinates can be obtained:

[0032]

[0033] The equation can be written as follows:

[0034]

[0035] Solving the extrinsic parameter matrix can be seen as solving for 12 unknowns. The rotation matrix has orthogonality, and the other 6 of the 9 parameters can be solved by using 3 of them. This invention uses the PnP algorithm to solve the pose information when the pixel coordinates of the four corner points of the QR code mark and the actual side length of the square mark are known.

[0036] The beneficial effects of the present invention are as follows: the pose detection method proposed in this invention has high detection accuracy, good real-time performance, and high detection success rate and detection speed. Attached Figure Description

[0037] Figure 1 The flowchart shows the overall process of a vision-optimized high-speed pose detection method.

[0038] Figure 2 This is a schematic diagram of the network model for a target tracking algorithm.

[0039] Figure 3 A schematic diagram for obtaining the pixel coordinates of the corner points of a QR code marker. Detailed Implementation

[0040] The invention will now be further described with reference to the accompanying drawings.

[0041] like Figure 1 As shown, the overall process of the high-speed pose detection method based on vision optimization is as follows: first, the target image is acquired; then, the target tracking algorithm is used to locate the ROI where the target is located; next, the image segmentation algorithm is used to segment the QR code markers in the image of the ROI; finally, a global coordinate transformation model of the scene is constructed, and the pose information is calculated by using the PnP algorithm through the pixel coordinates of the QR code markers.

[0042] During the image acquisition phase, a Hikvision MV-CA013-20GC camera was used for image acquisition. The camera has a resolution of 1280 pixels × 1024 pixels and a maximum frame rate of 201 FPS. It was suspended above the target to be detected using a gate-shaped bracket as a global vision sensor.

[0043] like Figure 2 As shown, in the target tracking stage, the first frame image of the target is selected as the template image. Subsequent search images use the template image as a reference for feature matching. The search image and the template image are used as inputs to the target tracking algorithm to locate the ROI of the target on the search image. The target tracking algorithm uses the search image and the template image as input image pairs, cuts the input image pairs into image blocks, and encodes them according to pixel channels. The encoding first undergoes a trainable linear mapping operation E to obtain the encoding vector of the input image block. The encoding corresponding to each image block is added to the corresponding position encoding on its original image to obtain the final image block encoding. The encoded image is used as input to the target tracking algorithm. The input encoding uses a cascaded Transformer encoder to simultaneously extract features and model relationships between the template image and the search image. This invention uses a 12-layer stacked 8-head Transformer encoder layer as the backbone network of the target tracking model. Background elimination strategy is added to layers 4, 7, and 10 of the 12-layer cascaded encoder. During the inference process of the target tracking model, the image block features with smaller weights are gradually discarded, so that the model gradually discards background features and focuses on target features. At the same time, the model data dimensionality is greatly reduced, and the inference speed is effectively improved. The background removal strategy retains only the top k candidate vectors with the highest scores, discarding the rest. It also needs to record the positions of the retained feature images for subsequent image reconstruction and restoration. The candidate vector retention rate is set to ρ = k / n, where n is the total number of candidate vectors; in this invention, ρ = 0.7. The output head of the target tracking model uses a 4-layer cascaded fully convolutional neural network. It scores each image patch in the search image according to its matching degree with the template image. The image patch with the highest score is the location of the target. A regression box is used to represent the position and size of the target ROI. Therefore, the output head outputs the similarity score of each image patch in the search image, as well as the center pixel coordinates and width and height of the regression box containing the target.

[0044] like Figure 3 As shown, the ROI obtained by the target tracking algorithm is used as the detection region for pose detection. An image segmentation algorithm is then used to segment the QR code marker within the ROI, obtaining the pixel coordinates of the four corner points of the marker. Let the pixel coordinates of the top-left corner of the bounding box returned by the target tracking algorithm be (x...). t y t After image segmentation, the pixel coordinates of a corner point of the QR code marker are (x...). s y sIf the pixel coordinates of the corner point in the original image are (x, y), then the final pixel coordinates of the corner point in the original image are (x, y). t +x s y t +y s ).

[0045] Construct a global coordinate transformation model for the pose detection scenario, including the world coordinate system O. w -X w Y w Z w To camera coordinate system O c -X c Y c Z c Transformation of camera coordinate system O c -X c Y c Z c Transformation from image physical coordinate system O-XY to image pixel coordinate system O′-UV, and transformation from image pixel coordinate system O′-UV to world coordinate system O w -X w Y w z w The transformation is performed, and the pose information is calculated using the PnP algorithm based on the pixel coordinates of the QR code marker.

[0046] The pose detection method of this invention achieves an average static position error of 0.0189m and an average static attitude error of 0.9822° in multiple sets of experiments, including static single-target, static multi-target, and dynamic target pose detection, as well as an average dynamic position error of 0.0258m and an average dynamic attitude error of 1.1772°, demonstrating good accuracy and real-time performance. Furthermore, the target tracking algorithm proposed in this invention significantly improves the efficiency of pose detection, increasing the detection speed from 28.6 FPS to 95.2 FPS, an improvement of 233%.

[0047] The above embodiments are only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. It is worth noting that any improvements made to the technical solution based on the technical concept of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A high-speed pose detection method based on vision optimization, characterized in that, Includes the following steps: The first step is to acquire images. The image acquisition method is to attach an ArUco code to the target to be detected and suspend the camera above the target as a global vision sensor to acquire target images in real time. The second step is to use a target tracking algorithm to locate the ROI where the target is located, and then shrink the pose detection area to the bounding box output by the target tracking algorithm. The target tracking algorithm uses a cascaded Transformer encoder to simultaneously extract features and model relationships between template and search images. To fully perform feature extraction and relationship modeling, a 12-layer stacked 8-head Transformer encoder is used as the backbone network of the target tracking model. The target tracking algorithm incorporates a background elimination strategy in layers 4, 7, and 10 of the 12-layer cascaded encoder. During the inference process of the target tracking model, features of image blocks with smaller weights are gradually discarded, allowing the model to gradually discard background features and focus on target features. At the same time, the model data dimensionality is significantly reduced, effectively improving the inference speed. The third step is to use the ROI obtained by the target tracking algorithm as the detection area for pose detection, and use the image segmentation algorithm to segment the QR code marker in the ROI to obtain the pixel coordinates of the four corner points of the marker. The fourth step is to construct a global coordinate transformation model for the pose detection scene and use the PnP algorithm to calculate the pose information using the pixel coordinates marked by the QR code.

2. The high-speed pose detection method based on vision optimization according to claim 1, characterized in that, In the first step, after acquiring the target image, the first frame of the target image is selected as the template image, and subsequent search images are performed using the template image as a reference for feature matching.

3. The high-speed pose detection method based on vision optimization according to claim 1, characterized in that, In the second step, the search image and template image are used as input to the target tracking algorithm to locate the ROI of the target on the search image. The target tracking algorithm cuts the search image and template image into image blocks and encodes them according to pixel channels. The encoding first goes through a trainable linear mapping operation to obtain the encoding vector of the input image block. The encoding of each image block is added to the corresponding position encoding on its original image to get the final image block encoding. The encoded image is then used as input to the target tracking algorithm.

4. The high-speed pose detection method based on vision optimization according to claim 3, characterized in that, The target tracking algorithm uses a 4-layer cascaded fully convolutional neural network as the output head of the target tracking model. It scores each image block in the search image according to its matching degree with the template image, and the image block with the highest score is the location of the target. A regression box is used to represent the position and size of the target ROI. Therefore, the output of the output head is the similarity score of each image block in the search image and the center pixel coordinates and width and height of the regression box where the target is located.

5. The high-speed pose detection method based on vision optimization according to claim 1, characterized in that, In the third step, the ROI obtained by the target tracking algorithm is used as the detection area for pose detection. The QR code marker in the ROI is segmented using an image segmentation algorithm to obtain the pixel coordinates of the four corner points of the marker.

6. The high-speed pose detection method based on vision optimization according to claim 1, characterized in that, In the fourth step, a global coordinate transformation model for the pose detection scenario is constructed, and the pose information is calculated using the PnP algorithm based on the pixel coordinates marked by the QR code.

Images