Target identification positioning and distance measuring method based on monocular camera

Abstract

The invention discloses a target identification positioning and distance measuring method based on a monocular camera, and the method comprises the steps: building a world coordinate system through fixing the position, angle and range of monitoring equipment, and setting a base point to determine the world coordinate of the base point; world coordinates are converted into image coordinates through camera parameters, feature points are recognized through a YOLOv7-ORB algorithm, base point pixel coordinates are extracted, and then the affine transformation relation between the world coordinates and the image coordinates is established. And after the target is identified, substituting the position of the target in the image coordinate into the affine function, converting the affine function into a world coordinate, and calculating the distance of the target through a camera projection model to realize the positioning and distance measurement of the target. The method has high precision and real-time performance, and is suitable for various target identification and ranging positioning scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and robotics, and in particular to a target recognition, positioning and distance measurement method based on a monocular camera, which is used to establish a mapping relationship between an image coordinate system and a world coordinate system, and realize the positioning of a target for distance measurement. Background Art

[0002] Object detection and localization technology has important applications in autonomous driving, robot navigation, augmented reality and other fields. With the advancement of deep learning, object detection based on algorithms such as YOLO has made significant progress and can quickly and accurately identify objects. However, accurately mapping the two-dimensional coordinates in the image to the three-dimensional world coordinate system is still a challenge.

[0003] At present, commonly used positioning methods include camera calibration, geometric transformation and feature point matching. Traditional camera calibration methods rely on manual calibration plates or labeled data, which are computationally complex and not suitable for dynamic environments. Feature point matching methods (such as SIFT, ORB, etc.) can improve positioning accuracy, but there is still room for improvement in computational efficiency and robustness.

[0004] Therefore, how to combine target detection and feature point extraction technology to improve positioning accuracy and computational efficiency has become a research hotspot in this field. This paper proposes a target positioning method based on the combination of YOLOv7 and ORB feature points. By optimizing the mapping from the image to the world coordinate system, efficient and accurate target positioning can be achieved in complex environments. Summary of the invention

[0005] In order to solve the problems involved in the background technology, the present invention proposes a target recognition, positioning and ranging method based on a monocular camera for the monocular camera to recognize and locate the target in the shooting picture.

[0006] The present invention adopts the following technical solution: The target recognition, positioning and ranging method based on a monocular camera comprises the following steps: Step 1: Initialize the camera and base point positions; Step 2: Establish the transformation relationship between the world coordinate system and the image coordinate system P'=K*[R|T]*P w , where P' is the pixel coordinate, K is the camera internal parameter matrix, [R|T] is the external parameter matrix, P w =( X w ,Y w , Z w ) is a point in the world coordinate system; Step 3: Base point identification and detection; Step 4: Use triangulation to derive the mapping relationship between the world coordinate system and the image coordinate system. Based on the perspective conversion, calculate the conversion function between the two as P''=A(K*[R|T]*P)+b, where A is the transformation matrix, b is the offset vector, P'' is the pixel coordinate, and P is the actual coordinate; Step 5: Target detection and recognition; Step 6: Coordinate transformation, determine the actual coordinates in the target world coordinate system; Step 7: Calculate the target distance.

[0007] Compared with the prior art, the present invention has the following beneficial effects: In the coordinate conversion stage, the triangulation method is combined with the base point position, and the affine function between the image coordinates and the world coordinates is used as the mapping formula. The matching relationship is accurately derived through the perspective conversion and the spatial position of the base point, which significantly improves the positioning accuracy and robustness of the target pixels in the two-dimensional image to the three-dimensional coordinates.

[0008] In the feature point extraction stage, the detection box constraints provided by YOLOv7 and the fine-grained features of the high-resolution feature map are integrated, and the CNN descriptor is used instead of the traditional binary descriptor (BRIEF), which significantly improves the feature matching accuracy of the target contour pixels.

[0009] In the coordinate mapping process, high-resolution and low-resolution multi-scale feature maps are combined to optimize the multi-scale aggregation effect with the Feature Pyramid Network (FPN). By limiting the feature point detection range and combining the global optimization strategy, the generated target position data is more accurate, which improves the robustness and accuracy of the object's spatial position measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 : Overall flow chart of the ranging and positioning method of the present invention; Figure 2 : The conversion principle between the world coordinate system and the image coordinate system; Figure 3 :The fusion process mechanism of YOLOv7 and ORB. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0012] like Figure 1-Figure 3As shown, the present invention proposes a target recognition, positioning and ranging method based on a monocular camera. Through a camera with fixed internal and external parameters and three known coordinate base points within the monitoring range, the camera parameters are used to convert the world coordinates into projection image coordinates. The YOLOv7-ORB algorithm is combined to identify the pixel coordinates of the base point, and the conversion function between the world coordinate system and the image coordinate system is derived through the conversion relationship between the projection coordinates and the pixel coordinates. When the target is detected, the image coordinates of the target are brought into the conversion function and converted into world coordinates to accurately determine the actual position of the target. The specific steps of this method are: Step 1: Initialize the camera and set the camera's intrinsic geometric information, i.e., the intrinsic parameter matrix K, including the camera's focal length f x ,f y and the main point c x ,c y , the matrix formula is , set the external parameter matrix [R|T] that describes the rotation and translation information of the camera, where R is the rotation matrix (3*3 matrix) and T is the translation vector (3*1 vector); set the spatial position of the surveillance camera O=(0,0,0), and set three base points P around the surveillance camera W1 , P W2 , P W3 , from the known world coordinates P W1 =(X1,Y1,Z1),P W2 =( X2, Y2, Z2),P W3 =(X3, Y3, Z3), the three base points form a triangle in space, which determines the range and direction of the monitoring view.

[0013] Step 2: Use the external parameter matrix of the monitoring camera to transform the base point P in the world coordinate system w =( X w ,Y w , Z w ) is transformed to the camera coordinate system P c =( X c ,Y c ,Z c ), the conversion formula is , so that each base point P Wi =( X wi ,Y wi , Z wi )(i=1,2,3) to get the point P in the camera coordinate system ci =( X ci ,Y ci ,Z ci );Then, through the camera's intrinsic parameter matrix K, point P in the camera coordinate system is c =( X c ,Y c,Z c ) is projected to the image coordinate system (x,y): , that is, the calculation formula of the image coordinates (x, y) is: , using the above formula, we can get the base point P of the three world coordinate systems W1 , P W2 , P W3 and their corresponding points in the image coordinate system (x1,y1), (x2,y2), (x3,y3). Using these known points, a more accurate transformation model is established through direct linear transformation and least squares method. The transformation function between the world coordinate system and the image coordinate system expressed by homogeneous coordinates and perspective transformation matrix is ​​obtained. : .

[0014] Step 3: Deeply fuse the YOLOv7 and ORB algorithms to improve recognition accuracy and efficiency. The specific fusion steps are as follows: S1, read the camera monitoring screen video frame by frame, obtain each frame image, and input RGB image, the image size is set to H*W, and then normalize the image. , X represents the normalized pixel value, X 原始 represents the initial image data, X max represents the maximum value of X, X min Represents the minimum value of X, adjusts the image size to 640*640, and the processed image I' is used for network inference.

[0015] S2. Use the backbone network (CSPDarknet53) of YOLOv7 to extract multi-scale feature maps for I'.

[0016] S3. The input feature map X is processed and divided into two parts, X1X2=split(X). X1 is directly passed to the subsequent network (Shortcut path), and X2 is processed by deep convolution to extract local features.

[0017] S4. Perform residual block processing on X2. Each residual block extracts features through Y=F(X2,Y)+X2, where F(X2,W) extracts features through a 3×3 convolution operation with a parameter of W. +X2 is a residual connection used to maintain feature integrity and alleviate gradient disappearance.

[0018] S5. Concatenate X1 and Y in the channel dimension: Z=concar(X1,Y). The output Z is the fused feature map, which contains the global information X1 and the deep information Y.

[0019] S6. Output Z is further convolved to obtain the input feature map of the next stage.

[0020] S7, extract high-resolution feature map FeatureMapP3, image I′ first passes through a 3×3 convolution layer with a stride of s=2, p is the padding size, and generates the first layer of feature map. The feature map retains more spatial detail information and is suitable for detecting small targets and key points. The extraction formula is: , , where s is the stride size in feature extraction, p is the padding size, k is the convolutional layer size, H is the image pixel height, and W is the image pixel width.

[0021] pS8, extract the medium-resolution feature map FeatureMapP4, starting from FeatureMapP3, further processed by the CSP module, and downsampled through the pooling layer. The output feature map resolution is 1 / 16 of the original image, and the extraction formula is: , .

[0022] S9, extract the low-resolution feature map FeatureMapP5, starting from FeatureMapP4, after deeper convolution and downsampling operations. The final output feature map resolution is 1 / 32 of the original image, and the extraction formula is: ,

[0023] S10. Perform multi-scale fusion of feature pyramids. The feature map output by CSPDarknet53 is passed through the feature pyramid network (FPN). The low-resolution feature map is upsampled and combined with high-resolution features to achieve further fusion. The high-resolution feature map FeatureMapP3 and low-resolution feature maps FeatureMapP4 and FeatureMapP5 are output. FeatureMapP3 is used for key point detection, and FeatureMapP4 and FeatureMapP5 are used for target detection.

[0024] S11. For the feature map extracted as described above, use the YOLOv7 detection framework based on the anchor box to predict the target bounding box and category, and pre-select the feature map; for the high-resolution feature map FeatureMapP3, limit the feature point detection range according to the detection frame, run the FAST algorithm within the limited range to detect corner points, and extract feature point descriptors on the feature map FeatureMapP3.

[0025] S12: Associate the feature points and descriptors of each detection box and output the fused real-time results.

[0026] S13. Apply L-detect in the detection box prediction stage to optimize the target box and classification performance. Apply L-keypoint in the training of the feature point detection module to optimize the feature point position and confidence. Optimize the detection loss and feature point loss at the same time in the multi-task learning framework to achieve deep fusion of YOLOv7 and ORB algorithms.

[0027] S14. Using the above fused YOLOv7-ORB algorithm, identify the position of the base point in the image and determine its pixel position Pi''(x i ',y i ').

[0028] Step 4: Get the affine relationship between world coordinates and pixel coordinates. The pixel coordinates of the base points P1'' (x1,y1) P2'' (x22,y2) P3'' (x3,y3) and P1' (x1,y1) P2' (x2,y2) P3' (x3,y3) obtained by YOLOv7-ORB recognition are affine transformed to obtain: , where A is a 2×2 matrix representing scaling and rotation, and b is the translation vector, by the least squares method , optimization process, solving the parameters to obtain the functional relationship between the image coordinates: P i '' =A*P i '+b, the transformation function between the world coordinates obtained above and the corresponding projection image coordinates P'=K*[R|T]*P w , further derive the conversion function between world coordinates and pixel coordinates: P''=A*(K*[R|T]*P)+b, and inversely deduce the function of world coordinates: P=(P''-b) / [R|T] / K / A, which is the conversion function that finally determines the target position.

[0029] Step 5: Use the above fused YOLOv7-ORB algorithm to identify the target in the image, extract feature points, and determine the pixel coordinates of the target.

[0030] Step 6: Substitute the pixel coordinates obtained by the above YOLOv7-ORB recognition into the transformation function P = (P''-b) / [R|T] / K / A to determine the actual coordinates of the target in the world coordinate system.

[0031] Step 7: Establish a mapping relationship using the focal length of the camera and the known depth of the base point: , where u g The horizontal coordinate of the target, u b is the horizontal coordinate of the base point, f is the focal length of the camera, Z knowis the known depth of the base point, and substitute the obtained world coordinates P(X, Y, Z) of the target and the camera coordinates O(X0, Y0, Z0) into the Euclidean distance formula: , and the straight-line distance d between the camera and the target is obtained. This realizes the distance measurement and positioning of the unknown target.

[0032] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A target recognition, positioning and ranging method based on a monocular camera, characterized in that: The following steps are involved: Step 1: Initialize the camera and base point positions; Step 2: Establish the transformation relationship between the world coordinate system and the image coordinate system P'=K*[R|T]*P w , where P' is the image coordinate, K is the camera internal parameter matrix, [R|T] is the external parameter matrix, P w =( X w ,Y w , Z w ) are the base point coordinates in the world coordinate system; Step 3: Base point identification and detection; Step 4: Use triangulation to derive the mapping relationship between the world coordinate system and the image coordinate system. Based on the perspective conversion, calculate the conversion function between the two as P''=A(K*[R|T]*P)+b, where A is the transformation matrix, b is the offset vector, P'' is the pixel coordinate, and P is the actual coordinate; Step 5: Target detection and recognition; Step 6: Coordinate transformation, determine the actual coordinates in the target world coordinate system; Step 7: Calculate the target distance.

2. The target recognition, positioning and ranging method based on a monocular camera according to claim 1, characterized in that: In the step 1, the camera is initialized, the camera's internal parameters K and external parameters [R|T] are set; the spatial position O=(X0, Y0, Z0) of the monitoring camera is established, and three base points P are set around the monitoring camera. W1 , P W2 , P W3 , given the known world coordinates P W1 =(X1,Y1,Z1),P W2 =( X2, Y2, Z2),P W3 =( X3, Y3, Z3), the three base points form a triangle in space, which determines the range and direction of the monitoring view.

3. The target recognition, positioning and ranging method based on a monocular camera according to claim 1, characterized in that: In step 2, the base point P in the world coordinate system is converted by monitoring the camera external parameter [R|T] and the camera internal parameter K. w Transform to the camera coordinate system P c =( X c ,Y c ,Z c ), and then projected to the image coordinate system (x, y): P'=K*[R|T]*P w , get the base point P of the three world coordinate systems W1 , P W2 , P W3 And their corresponding points in the image coordinate system P1'=(x1, y1) P2'=(x2, y2) P3'=(x3, y3), through linear transformation, using homogeneous coordinates and perspective transformation matrix, we get the transformation function between the world coordinate system and the image coordinate system: .

4. The target recognition, positioning and ranging method based on a monocular camera according to claim 1, characterized in that: In step 3, the position of the base point in the image is identified, and its pixel position P in the image coordinate system is determined. i ''( x i ' ,y i '), when identifying the base point, the ORB and YOLOv7 algorithms are integrated. The specific steps are: Step 4.1: Read the camera monitoring screen frame by frame, normalize it, and directly input it into RGB format; Step 4.2: Shared feature map, based on the convolutional neural network CNN, extract a single shared feature map, and use the multi-scale feature map FPN to generate FeatureMapP5, FeatureMapP4 and FeatureMapP3 at the low resolution, medium resolution and high resolution levels respectively; Step 4.3: The YOLOv7 detection framework processes the shared feature map to generate the bounding box and category prediction results of the target, and uses the detection box as a constraint to perform feature point detection only on the pixel area within the target detection box; Step 4.4: Perform feature point detection and descriptor extraction on the high-resolution feature map FeatureMapP3; Step 4.5: Replace the binary descriptor BRIEF of the traditional ORB with the high-dimensional semantic CNN feature vector; Step 4.6: Joint loss optimization. During the detection box prediction and feature point detection process, a joint loss function is defined, including target bounding box loss, classification loss, feature point confidence loss, and feature point position loss.

5. The target recognition, positioning and ranging method based on a monocular camera according to claim 4, characterized in that: The fourth step obtains the functional relationship between the world coordinates and the pixel coordinates, and obtains the mapping relationship between the two image coordinate systems by identifying the pixel coordinates P1''(x1',y1')P2''(x2',y2')P3''(x3',y3') of the base point and the image coordinates P1'(x1,y1)P2'(x2,y2)P3'(x3,y3), and using the similarity transformation between the two triangles: P i ''=A*P i '+b, and the world coordinate and image coordinate conversion function P'=K*[R|T]*P w , deduce the mapping function between world coordinates and pixel coordinates P''=A*(K*[R|T]*P)+b, and inversely deduce the function of world coordinates P=(P''-b) / [R|T] / K / A.

6. The target recognition, positioning and ranging method based on a monocular camera according to claim 5, characterized in that: The step five identifies the target in the image, and uses the YOLOv7-ORB algorithm fused in the above steps to perform target recognition and feature point extraction on the target to be identified in the image to determine the pixel coordinates of the target.

7. The target recognition, positioning and ranging method based on a monocular camera according to claim 6, characterized in that: In step six, the pixel coordinates obtained by recognition are introduced into the conversion function P=(P''-b) / [R|T] / K / A to determine the world coordinates of the target.

8. The target recognition, positioning and ranging method based on a monocular camera according to claim 7, characterized in that: The seventh step calculates the target distance, calculates the target depth through the pixel coordinates and the camera's intrinsic parameter matrix, and uses the camera projection model to substitute the target's world coordinates and depth coordinates to determine the target's actual position in the world coordinate system and its distance from the camera.

Images