Ranging method based on combined binocular camera and computer readable storage medium

Abstract

The invention discloses a distance measurement method based on a combined binocular camera and a computer readable storage medium, and the method comprises the following steps: calibrating and deploying the binocular camera, and obtaining camera parameters; synchronously acquiring an image, adjusting the size, inputting the image into the pre-training detection network, extracting image information, and processing to obtain a target image and target information; adjusting the size of a target image, inputting the target image into the feature extraction network to obtain a feature vector, constructing a cosine distance matrix by calculating the cosine distance of the feature vector, and constructing a geometric similar matrix by using target information; after the two matrixes are fused and calculated, a matching target is obtained by adopting a Hungary algorithm, an optimal matching point is searched in a matching target area, and distance measurement is performed on the optimal matching point by adopting triangulation in combination with camera parameters. The method supports simultaneous distance measurement of multiple types and multiple targets, further improves the matching accuracy, is not liable to be influenced by illumination and environment changes, is flexible in deployment, is high in calculation efficiency, and is low in cost of used hardware.

Description

Technical Field

[0001] This invention relates to the field of ranging, and particularly to a ranging method based on a combined binocular camera and a computer-readable storage medium. Background Technology

[0002] In short, ranging (height measurement) involves obtaining three-dimensional information about a target. This three-dimensional information can assist in tasks across many fields. For example, in autonomous driving, it helps vehicle systems determine the distance between objects ahead and oncoming vehicles. On roads with height restrictions, height measurement can detect in advance whether passing vehicles exceed the standard. In industry, combining ranging and height measurement can be used to assess the quality of components on a production line. Existing binocular ranging methods mainly rely on target detection, stereo matching, and depth estimation techniques. However, existing solutions still suffer from problems such as computational redundancy, high hardware costs, and limited applicability.

[0003] For example, the technical solution of the patent application with application number "CN114565900A", patent name "Target Detection Method Based on Improved YOLOv5 and Binocular Stereo Vision", and patent applicant "Guangzhou Software Application Technology Research Institute" is as follows: Binocular cameras simultaneously acquire image data, and ultrasonic sensors simultaneously acquire ultrasonic data; the images acquired by the left and right cameras are matched, and the image from the left camera is input into the improved YOLOv5 model for target detection; the SGBM algorithm is used for stereo matching to obtain a disparity map; the target detection information, disparity map, and ultrasonic information are fused and calculated to output the distance of the target. This approach requires expensive hardware, employing not only binocular cameras but also ultrasonic devices for object localization. SGBM calculates the parallax of the entire image, including areas of no interest, resulting in numerous unnecessary calculations. Furthermore, if the left and right cameras are out of sync, environmental factors such as lighting and dust can significantly impact SGBM performance. YOLOv5's improvement involves adding a network module, which may make the detection computation too cumbersome, potentially increasing memory usage and decreasing FPS. Currently, YOLOv5 can achieve excellent results with a lightweight network and appropriate data design. Additionally, the lack of ID information prevents the simultaneous measurement of distances to multiple different targets.

[0004] For example, the technical solution of the patent application with application number "CN114494462A", patent name "A Binocular Camera Ranging Method Based on YOLOv5 and Improved Tracking Algorithm", and patent applicant "Xi'an University of Electronic Science and Technology Wuhu Research Institute" is as follows: Binocular cameras simultaneously acquire image data, followed by parameter calibration and binocular correction of the binocular cameras; images from the left and right cameras are simultaneously input to the YOLOv5 network to obtain target boxes and track the target; binocular stereo matching and disparity calculation are performed on the images, and finally, the distance is calculated by fusing the target detection information and the disparity map from the fourth step. For this scheme, the use of a stereo camera, compared to two separate cameras, results in less freedom of movement due to the relatively fixed positions of the two cameras, and stereo cameras are also more expensive. While the method for obtaining the disparity map is not specified, the invention claims to calculate the disparity of the entire image pixel-by-pixel, leading to unnecessary computation and susceptibility to lighting and dust effects. Furthermore, the full-image calculation is easily affected by lighting and noise. The tracking module is overly redundant. In stereo ranging, a tracking module can improve the detection of shaking and range multiple targets, but in reality, achieving multi-target ranging and improved shaking detection does not require a complete tracking module; instead, it increases computational resources.

[0005] For example, the technical solution of application number "CN115797459A", patent title "A Ranging Method for a Binocular Vision System with Arbitrary Focal Length Combinations", and patent applicant "Nanjing University of Aeronautics and Astronautics" is as follows: Two separate cameras with different focal lengths are placed on the same horizontal plane, the cameras are calibrated, and their intrinsic parameters are obtained. The distance is then obtained using the triangle similarity formula. This solution does not mention a detection algorithm, and as a whole, it can only perform single-target ranging. Furthermore, it does not mention a matching algorithm; this invention only calculates the distance for matched points, while matching is extremely important and complex in binocular ranging.

[0006] For example, the technical solution of application number "CN113587895A", patent name "Binocular Ranging Method and Device", and patent applicant "Hangzhou Santan Medical Technology Co., Ltd." is as follows: The binocular camera is calibrated, and the binocular camera simultaneously acquires images. A depth estimation network processes the two images captured by the binocular camera to obtain an RGB depth map, and finally, the distance is obtained from the RGB depth map. This solution does not mention a detection algorithm, and as a whole, it can only perform single-target ranging. It also does not mention a matching algorithm; the obtained depth information is the depth information of the entire image, not the depth information of the target. To obtain the depth information of the corresponding target, matching is required between the two targets in the left and right cameras. The depth estimation network is currently not mature enough; without calibration, only relative distance is obtained. Although calibration can obtain the actual distance, depth estimation annotation requires LiDAR and point clouds. Training such a network requires expensive equipment for annotation, and the depth estimation model is easily rendered unsuitable for new scenes due to scene changes. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a ranging method based on a combined binocular camera.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] The ranging method based on a combined binocular camera includes the following steps:

[0010] S1. Calibrate and deploy the binocular camera to ensure that the target or area to be measured appears simultaneously in the field of view of the binocular camera, and obtain the binocular camera parameters, including intrinsic parameters, extrinsic parameters and basic matrix; the binocular camera includes a first camera and a second camera;

[0011] The intrinsic parameters are the intrinsic parameter matrices K of the two cameras, which describe the characteristics of the camera imaging model itself, that is, the relationship between the pixel coordinate system and the camera coordinate system; the extrinsic parameters refer to the positional relationship of one camera relative to another camera, which is described by the rotation matrix R and translation matrix t of one camera relative to the other camera. In other words, the camera can reach the position of the other camera through rotation and translation; the fundamental matrix describes the geometric relationship between the two cameras in pixel coordinates.

[0012] S2. Simultaneously acquire images using a stereo camera, adjust the image size, and input them into a pre-trained detection network to obtain image information. Process the image information to obtain target images and target information from the first and second cameras. The target information includes target category, target bounding box coordinates, and target bounding box point coordinates.

[0013] S3. Adjust the size of the target image from the binocular camera and input it into the feature extraction network for feature extraction. Complete feature matching to obtain feature vectors, and construct a cosine distance matrix by calculating the cosine distance of the feature vectors.

[0014] The L2×F×L1 matrix is ​​transposed, and then elements in L2×F×L1 with absolute values ​​less than a set value are reassigned to 0, thus obtaining the final geometric similarity matrix. Here, L1 is the coordinate matrix of the points in the target frame of the first camera, L2 is the coordinate matrix of the points in the target frame of the second camera, F is the basic matrix, each element in L2×F×L1 represents the distance from pixel p' to epipolar line Pp, epipolar line Pp is the line connecting object point P and pixel p, pixel p is the image of object point P in the first camera, and pixel p' is the image of object point P in the second camera.

[0015] Among them, the target bounding point coordinate matrix of the first and second cameras is composed of the target bounding point coordinates in the target information of the first and second cameras, respectively;

[0016] S4. The cosine distance matrix and the geometric similarity matrix are fused to obtain the similarity matrix. The similarity matrix is ​​then matched using the Hungarian algorithm to obtain the matching target of the stereo camera.

[0017] S5. Define the first camera as the ranging reference camera, and define the ranging point of the target box of the first camera as point u;

[0018] Find the point v corresponding to point u within the matching target box region of the second camera;

[0019] Traverse each point within an n×n range around point v and find the point with the shortest distance to the epipolar line Pp, which is the best matching point v';

[0020] The distance between the target and the first camera is obtained by processing the coordinates of point u and the best matching point v' using the triangulation method.

[0021] In step S2, the pre-trained detection network is a YOLO series detection network that has been pre-trained and can detect all types of targets to be tested. The images acquired by the first camera and the second camera at the same time are adjusted to a size of 640×640. The number of target boxes and the coordinate information of each target box in the images acquired by the first camera and the second camera are obtained through the pre-trained detection network. The target can be cropped from the original image using the target box coordinate information. Then, the target information of the cropped image of each target is stored in a dictionary. The target information includes the target category, the target box coordinate information, and the coordinates of the point in the target box. If the first camera has M detected targets, there are M dictionaries. Finally, these M dictionaries are combined into a list. If the second camera has N detected targets, N dictionaries are combined into a list.

[0022] Step S3 involves adjusting all target images cropped from the first and second cameras to a size of 256×128 and inputting them into the ResNet34 network for feature extraction. After feature matching is completed for each target image, a 512×1 feature vector is output. By calculating the cosine distance between the M feature vectors of the first camera and the N feature vectors of the second camera, a cosine distance matrix of size M×N is constructed.

[0023] In the image coordinate system, the top-left corner of the target image is set as the origin. The coordinates of the points in the target bounding box from the M dictionaries of the first camera are constructed into the form (x, y, 1), where x and y represent the horizontal and vertical coordinates of the points in the target bounding box, respectively, and 1 indicates that a 1 is added to the end of the coordinates of the points in the target bounding box to form homogeneous coordinates, so that translation can also be included in the linear algebra framework, and operations such as rotation and scaling are unified into matrix multiplication; it also supports seamless combination of perspective projection, infinity point, and multiple transformations. The coordinates of the points in the target bounding box are extracted to form a 3×M midpoint coordinate matrix L1, where 3 represents the dimension, that is, there are 3 elements in (x, y, 1), which represents 3 columns in the target bounding box coordinate matrix; M represents M targets, which represents M rows in the target bounding box coordinate matrix. The same processing is performed on the second camera, and an N×3 midpoint coordinate matrix L2 is constructed. The transpose of L2×F×L1 is calculated, and then the elements in L2×F×L1 with absolute values ​​less than a set value are reassigned to 0, thus obtaining the final geometric similarity matrix.

[0024] Step S4 is to fuse and calculate the cosine distance matrix and the geometric similarity matrix. Let the cosine distance matrix be D and the geometric similarity matrix be G. Then the final similarity matrix is ​​W=(D^2+G^2) / 2.

[0025] The similarity matrix is ​​used to obtain matching targets using the Hungarian algorithm. The result actually returns the indices of the M detected targets in the first camera and the N detected targets in the second camera that match.

[0026] The Hungarian algorithm can only achieve one-way matching:

[0027] If M=N, then each target in the first camera will have a matching target in the second camera;

[0028] If M≠N, then there must be some targets that did not appear in the other camera, or the pre-trained detection network made a mistake that caused some targets to be missed / over-detected. Calculate the row and column indices of the matching for the W matrix, construct the matching degree vector, and obtain the matching target when the sum of the elements of the matching degree vector is the smallest.

[0029] In step S5, the coordinates of point u and the best matching point v' are processed using triangulation to obtain the distance between the target and the first camera. Specifically:

[0030] (1) Coordinate normalization;

[0031] Convert the coordinates of points u and v' to homogeneous coordinates.

[0032] (X1,Y1,1) = K1'×(x1,y1,1);

[0033] (X2,Y2,1) = K2'×(x2,y2,1);

[0034] Where u=(x1,y1), v'=(x2,y2), x1 and y1 are the x-coordinate and y-coordinate of point u, respectively; x2 and y2 are the x-coordinate and y-coordinate of point v', respectively.

[0035] X1 and Y1 are the x and y coordinates of point u transformed from the image coordinate system to the first camera coordinate system, respectively; X2 and Y2 are the x and y coordinates of point v' transformed from the image coordinate system to the second camera coordinate system, respectively.

[0036] Using the first camera as the world coordinate system, we obtained the intrinsic parameter matrix K1 of the first camera, the intrinsic parameter matrix K2 of the second camera, the rotation matrix R of the second camera, and the translation vector t; K1' is the inverse matrix of K1, and K2' is the inverse matrix of K2;

[0037] (2) Find the projection matrices P1 and P2 of the first camera and the second camera respectively;

[0038] P1 = K1×[I|0];

[0039] P2 = K2×[R|t];

[0040] Where I is a 3×3 identity matrix, 0 is a 3×1 vector consisting entirely of zeros, R is a 3×3 rotation matrix, and t is a 3×1 translation vector; [R|t] is a 3×4 matrix formed by concatenating R and t.

[0041] (3) Calculate the homogeneous matrix A of [R|t];

[0042] A = [[X1×P1[2,:]-P1[0,:]],

[0043] [Y1×P1[2,:]-P1[1,:]],

[0044] [X2×P2[2,:]-P2[0,:]],

[0045] [Y2×P2[2,:]-P2[1,:]]];

[0046] P1[2,:] represents the third row of matrix P1, and [X1×P1[2,:]-P1[0,:]] is a shortened form that expands to:

[0047] [X1×P1[2,0] - P1[0,0], X1×P1[2,1] - P1[0,1], X1×P1[2,2] - P1[0,2],X1×P1[2,3] - P1[0,3]];

[0048] (4) Perform SVD decomposition on the aligned matrix A and obtain the distance;

[0049] The SVD algorithm is used to perform singular value decomposition on A to obtain the eigenvector (X,Y,Z,W) corresponding to the smallest eigenvalue, where W is the scale factor. Then the actual coordinates of the target relative to the first camera are (X / W,Y / W,Z / W), and the final distance is the length of the line segment formed by the coordinates (X / W,Y / W,Z / W) and (0,0,0).

[0050] In step S1, when calibrating the binocular camera, the calibration board image of the first or second camera should cover 1 / 2 of the camera's field of view.

[0051] Meanwhile, this invention provides:

[0052] A server includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the above-described ranging method based on a combined binocular camera.

[0053] A computer-readable storage medium storing at least one program that is loaded and executed by a processor to implement the above-described ranging method based on a combined binocular camera.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] 1. This invention supports simultaneous ranging of multiple types of targets. The added matching mechanism can simultaneously measure the distance of multiple targets of different types and multiple targets of the same type. Furthermore, by complementing epipolar matching and feature matching, and by searching for the best matching point in the matching target region, the matching accuracy is improved.

[0056] 2. This invention is easy to deploy. In traditional binocular cameras, the two cameras are in a parallel state and are close to each other, and the deployment is easily restricted by the environment and terrain. However, due to the introduction of the matching mechanism, this invention can deploy the two cameras face to face, or one can be deployed on the ground and the other on the ceiling.

[0057] 3. This invention has good resistance to environmental interference. Triangulation directly calculates the distance of the matching point, and the matching point is constructed through feature matching and epipolar matching. Therefore, it focuses more on the overall features and spatial information of the target, rather than the feature information of the object in optical imaging. This effectively avoids inaccurate ranging caused by environmental factors.

[0058] 4. This invention can effectively filter out some targets that do not appear in both cameras by using the Hungarian matching algorithm.

[0059] 5. This invention features low hardware cost. Since triangulation does not require calculating information of the entire image, it does not require camera synchronization and global exposure. Even the focal lengths can be different, as long as the camera resolution is consistent, which greatly reduces the hardware cost of the camera.

[0060] 6. This invention is simple to calculate and fast. The auxiliary feature extraction network for epipolar matching does not need to be very complex. Furthermore, the introduction of the matching mechanism does not require calculation of the entire image, but only the pixels of the detected target.

[0061] 7. This invention has certain scalability. Since the purpose of distance measurement is to obtain coordinates and calculate distances using coordinates, the length, width, height, and other information of an object can also be measured using coordinates. Attached Figure Description

[0062] Figure 1 This is an overall flowchart of the ranging method based on a combined binocular camera. Detailed Implementation

[0063] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0064] like Figure 1 As shown, the ranging method based on a combined binocular camera includes the following steps:

[0065] S1. Calibrate and deploy the binocular camera to ensure that the target or area to be measured appears simultaneously in the field of view of the binocular camera, and obtain the binocular camera parameters, including intrinsic parameters, extrinsic parameters and basic matrix; the binocular camera includes a first camera and a second camera;

[0066] S2. Simultaneously acquire images using a stereo camera, adjust the image size, and input them into a pre-trained detection network to obtain image information. Process the image information to obtain the target image and target information of the stereo camera; among which, the target information includes target category, target bounding box coordinates, and target bounding box point coordinates.

[0067] S3. For target images of the same target category, adjust the target image size of the binocular camera and input it into the feature extraction network for feature extraction. Complete feature matching to obtain feature vectors, and construct a cosine distance matrix by calculating the cosine distance of the feature vectors.

[0068] The L2×F×L1 matrix is ​​transposed, and then elements in L2×F×L1 with absolute values ​​less than a set value are reassigned to 0, thus obtaining the final geometric similarity matrix. Here, L1 is the coordinate matrix of the points in the target frame of the first camera, L2 is the coordinate matrix of the points in the target frame of the second camera, F is the basic matrix, each element in L2×F×L1 represents the distance from pixel p' to epipolar line Pp, epipolar line Pp is the line connecting object point P and pixel p, pixel p is the image of object point P in the first camera, and pixel p' is the image of object point P in the second camera.

[0069] S4. The cosine distance matrix and the geometric similarity matrix are fused to obtain the similarity matrix. The similarity matrix is ​​then matched using the Hungarian algorithm to obtain the matching target of the stereo camera.

[0070] S5. Define the first camera as the ranging reference camera, and define the ranging point of the target box of the first camera as point u;

[0071] Find the point v corresponding to point u within the matching target box region of the second camera;

[0072] Traverse each point within an n×n range around point v and find the point with the shortest distance to the epipolar line Pp, which is the best matching point v';

[0073] The distance between the target and the first camera is obtained by processing the coordinates of point u and the best matching point v' using the triangulation method.

[0074] In step S1, the binocular camera is calibrated and deployed. The binocular camera consists of a first camera and a second camera, ensuring that the target or area to be measured appears in the field of view of both cameras simultaneously. The accuracy of the calibration will affect the subsequent measurement accuracy. Specifically, the following principles must be followed:

[0075] 1. Select a suitable calibration plate based on the camera's distortion coefficient; generally speaking, it is better to choose a calibration plate with a smaller checkerboard size, because the camera has distortion, and the larger the checkerboard, the greater the measurement error caused by lens distortion;

[0076] 2. The image on the calibration board should cover half of the camera's field of view;

[0077] 3. The calibration board should have as many checkerboard squares as possible and the calibration images should not exceed 20. The parameter optimization in the calibration process relies on the corner points of the checkerboard squares in an image. The more checkerboard squares there are, the more features are processed in an image, which makes it easier to optimize the parameters. Multiple images are used to prevent the angle of a single image from causing the optimization to get stuck in local optimization. However, experiments have shown that the calibration error increases when the number of images exceeds 20.

[0078] After calibration, the intrinsic parameters, extrinsic parameters, and fundamental matrix F of the two cameras can be obtained. The intrinsic parameters are the intrinsic parameter matrices K of the two cameras, which describe the characteristics of the camera imaging model itself, that is, the relationship between the pixel coordinate system and the camera coordinate system. The extrinsic parameters refer to the positional relationship of one camera relative to the other camera. This positional relationship is described by the rotation matrix R and translation matrix t of one camera relative to the other camera. In other words, the camera can reach the position of the other camera through rotation and translation. The fundamental matrix describes the geometric relationship between the two cameras in pixel coordinates.

[0079] In step S2, the pre-trained detection network is a YOLO series detection network that has been pre-trained and can detect all types of targets to be tested. The images acquired by the first camera and the second camera at the same time are adjusted to a size of 640×640. The number of target boxes and the coordinate information of each box in the images acquired by the first camera and the second camera are obtained through the pre-trained detection network. The target can be cropped from the original image using the coordinate information of the box. Then, the target information of each target in the cropped image is stored in a dictionary. The target information includes the target category, the target box coordinate information, and the coordinates of the point in the target box. If the first camera has M detected targets, then there are M dictionaries. Finally, these M dictionaries are combined into a list. The same applies to the second camera, with N dictionaries forming a list.

[0080] In step S3, all target images cropped from the first and second cameras are adjusted to a size of 256×128 and input into a ResNet34 network for feature extraction. The feature extraction network used in this invention is ResNet34; if speed is critical, ResNet18 can be used. After feature matching is completed for each target image, a 512×1 feature vector is output. An M×N cosine distance matrix is ​​constructed by calculating the cosine distance between the M feature vectors from the first camera and the N feature vectors from the second camera. To effectively filter out targets that should not be matched, the cosine distance is set to the range [0-1], and cosine distances greater than a threshold c are set to 1. The threshold c is selected based on the target category and the object's pixel size. If pedestrians and their clothing vary widely, and the first and second cameras capture the same person, their similarity distance will be very small, so the threshold c can be set to 0.5. If detecting items produced on a production line, since they are essentially identical with only some angular differences, the threshold c can be set to 0.9. Similarly, if the target pixel is very small, effective feature information cannot be extracted, and two blurry objects may be considered as one target, so the threshold c can be set higher. In practice, several candidate thresholds c can be set. The size of the bounding box can be determined using the bounding box coordinate information in each target's dictionary, thereby determining the pixel size of the target in the image. Then, an appropriate threshold c can be automatically selected based on the object's pixel size and target category.

[0081] In step S3, the coordinates of the bounding box points in the M dictionaries of the first camera are constructed into a (x, y, 1) form, and the bounding box point coordinates are extracted to form a 3×M midpoint coordinate matrix L1. The same process is performed on the second camera to construct an N×3 midpoint coordinate matrix L2. An M×N geometric similarity matrix can be obtained by calculating L2×F×L1 and then transposing it. The geometric similarity matrix utilizes epipolar matching of the cameras. The principle is as follows: given an object point P, the object is imaged as pixel p in the first camera; connecting P and p forms a line. If the second camera... If the camera also captures an object P, then the line Pp in the first camera can be projected into an epipolar line in the second camera, and the image point p' of the object point P in the second camera must lie on this epipolar line; F×L1 is the mathematical expression of the epipolar line Pp in the second camera, and L2×F×L1 is the distance from point p' to the epipolar line. If point p' lies on the epipolar line, this value is 0; considering the existence of calibration error and detection error, this calculation cannot actually be 0, but should be as close to 0 as possible; for ease of calculation, all values ​​in the geometric similarity matrix with an absolute value less than 0.8 are set to 0.

[0082] In step S4, the cosine distance matrix and the geometric similarity matrix are fused. If the detected object and the camera are on the same horizontal line and the two cameras are parallel, any target in the first camera may be on the same epipolar line in the second camera. In this case, the geometric similarity matrix has little effect. If a single camera contains multiple detected targets with visually identical appearance features, the cosine distance matrix has little effect. Therefore, let the cosine distance matrix be D and the geometric similarity matrix be G. The final similarity matrix is ​​W=(D^2+G^2) / 2. Since the similar elements were previously set to 0 or the dissimilar elements were set to 1 by using a threshold, squaring can further widen the gap between similar and dissimilar elements. Finally, the cosine distance matrix and the geometric similarity matrix are added together to cancel out the limitations of using only one of the similarity matrices.

[0083] In step S4, the matching targets are obtained by applying the Hungarian algorithm to the similarity matrix. The result actually returns the indices of the matching targets among the M detected targets in the first camera and the N detected targets in the second camera. The Hungarian algorithm can only obtain one-way matching: if M=N, then each target in the first camera will have a matching target in the second camera. If M≠N, then there must be some targets that are not present in the other camera, or the detection algorithm has made a mistake that caused some targets to be missed or over-detected. In this case, in order to avoid mismatch, it is necessary to calculate the row and column indices of the matching matrix W before constructing the matching degree vector. The matching target is obtained when the sum of the elements of the matching degree vector is minimized.

[0084] In step S5, considering the existence of some errors in the detection box, it is necessary to search for the optimal matching point in the target box region of the matching target. The distance is then measured using triangulation based on the optimal matching point. When a point in the target box is used as the distance measurement point (i.e., point u), the first camera is used as the distance measurement reference camera. Then, the midpoint of the box matched by the second camera (i.e., point v) is found. Within an n×n range of this midpoint, each point is traversed to find the point with the shortest distance to the epipolar line, thus finding the optimal matching point (i.e., point v'). Similarly, this also applies when the distance measurement point is a vertex of the detection box. Finally, the actual coordinates of the point can be calculated using triangulation based on the camera's intrinsic and extrinsic parameters combined with the coordinates of the optimal matching point. The target box is also called the detection box.

[0085] The specific calculations for the triangulation method are as follows:

[0086] By calibrating the binocular cameras and using the first camera as the world coordinate system, we obtained the intrinsic parameter matrix K1 of the first camera, the intrinsic parameter matrix K2 of the second camera, and the rotation matrix R and translation matrix t of the second camera.

[0087] u=(x1,y1),v'=(x2,y2), where x1 and y1 are the x and y coordinates of the midpoint of the first camera detection box, respectively; x2 and y2 are the x and y coordinates of point v', respectively.

[0088] 1. Normalize the coordinates on the image of the point to be measured;

[0089] First, points u and v' are the coordinates on the images of the first and second cameras, respectively. They are then converted into homogeneous coordinates, where K1' is the inverse matrix of K1 and K2' is the inverse matrix of K2.

[0090] (X1,Y1,1) = K1'×(x1,y1,1);

[0091] (X2,Y2,1) = K2'×(x2,y2,1);

[0092] Where X1 and Y1 are the horizontal and vertical coordinates of the target center point (i.e., point u) in the first camera image, transformed from the image coordinate system to the first camera coordinate system, respectively; and X2 and Y2 are the horizontal and vertical coordinates of the best matching point (i.e., point v') of the target center point in the second camera image, transformed from the image coordinate system to the second camera coordinate system, respectively.

[0093] 2. Find the projection matrices of the first and second cameras;

[0094] P1 = K1×[I|0];

[0095] P2 = K2×[R|t];

[0096] Where I is a 3×3 identity matrix, 0 is a 3×1 vector consisting entirely of zeros, R is a 3×3 rotation matrix, and t is a 3×1 translation vector. [R|t] is a 3×4 matrix formed by concatenating R and t.

[0097] 3. Calculate the homogeneous matrix A of [R|t];

[0098] A = [[X1×P1[2,:]-P1[0,:]],

[0099] [Y1×P1[2,:]-P1[1,:]],

[0100] [X2×P2[2,:]-P2[0,:]],

[0101] [Y2×P2[2,:]-P2[1,:]]];

[0102] P1[2,:] represents the third row of matrix P1, and [X1×P1[2,:]-P1[0,:]] is a shortened form that expands to:

[0103] [X1×P1[2,0] - P1[0,0], X1×P1[2,1] - P1[0,1], X1×P1[2,2] - P1[0,2],X1×P1[2,3] - P1[0,3]];

[0104] 4. Perform SVD decomposition on the aligned matrix A and obtain the distance;

[0105] The SVD algorithm is used to perform singular value decomposition on A to obtain the eigenvector (X,Y,Z,W) corresponding to the smallest eigenvalue, where W is the scale factor. Then the actual coordinates of the target relative to the first camera are (X / W,Y / W,Z / W), and the final distance is the length of the line segment formed by the coordinates (X / W,Y / W,Z / W) and (0,0,0).

[0106] Meanwhile, this invention provides:

[0107] A server includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the above-described ranging method based on a combined binocular camera.

[0108] A computer-readable storage medium storing at least one program that is loaded and executed by a processor to implement the above-described ranging method based on a combined binocular camera.

[0109] This invention fuses the cosine distance matrix and the geometric similarity matrix to obtain a similarity matrix, and then uses the Hungarian algorithm to match the similarity matrix to obtain the matching target of the stereo camera. At the same time, this invention searches for the optimal matching point in the matching target box region, thereby improving the matching accuracy.

[0110] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A ranging method based on a combined binocular camera, characterized in that, Includes the following steps: S1. Calibrate and deploy the binocular camera to ensure that the target or area to be measured appears simultaneously in the field of view of the binocular camera, and obtain the binocular camera parameters, including intrinsic parameters, extrinsic parameters and basic matrix; the binocular camera includes a first camera and a second camera; S2. Use a stereo camera to synchronously acquire images, adjust the image size and input it into a pre-trained detection network to obtain image information. Process the image information to obtain the target images and target information of the first and second cameras. S3. Adjust the size of the target image from the binocular camera and input it into the feature extraction network for feature extraction. Complete feature matching to obtain feature vectors, and construct a cosine distance matrix by calculating the cosine distance of the feature vectors. The L2×F×L1 matrix is ​​transposed, and then elements in L2×F×L1 with absolute values ​​less than a set value are reassigned to 0, thus obtaining the final geometric similarity matrix. Here, L1 is the coordinate matrix of the points in the target frame of the first camera, L2 is the coordinate matrix of the points in the target frame of the second camera, F is the basic matrix, each element in L2×F×L1 represents the distance from pixel p' to epipolar line Pp, epipolar line Pp is the line connecting object point P and pixel p, pixel p is the image of object point P in the first camera, and pixel p' is the image of object point P in the second camera. Among them, the target bounding point coordinate matrix of the first and second cameras is composed of the target bounding point coordinates in the target information of the first and second cameras, respectively; S4. The cosine distance matrix and the geometric similarity matrix are fused to obtain the similarity matrix. The similarity matrix is ​​then matched using the Hungarian algorithm to obtain the matching target of the stereo camera. S5. Define the first camera as the ranging reference camera, and define the ranging point of the target box of the first camera as point u; Find the point v corresponding to point u within the matching target box region of the second camera; Traverse each point within an n×n range around point v and find the point with the shortest distance to the epipolar line Pp, which is the best matching point v'; The distance between the target and the first camera is obtained by processing the coordinates of point u and the best matching point v' using the triangulation method.

2. The ranging method based on a combined binocular camera according to claim 1, characterized in that, In step S2, the pre-trained detection network is a YOLO series detection network that has been pre-trained and can detect all types of targets to be tested. The images acquired by the first camera and the second camera at the same time are adjusted to a size of 640×640. The number of target boxes and the coordinate information of each target box in the images acquired by the first camera and the second camera are obtained through the pre-trained detection network. The targets are cropped from the original images using the target box coordinate information. Then, the target information of each target in the cropped image is stored in a dictionary. The target information includes the target category, the target box coordinate information, and the coordinates of the point in the target box. If the first camera has M targets, there are M dictionaries. Finally, the M dictionaries are combined into a list. If the second camera has N targets, the N dictionaries are combined into a list.

3. The ranging method based on a combined binocular camera according to claim 2, characterized in that, Step S3 involves adjusting all target images cropped from the first and second cameras to a size of 256×128 and inputting them into the ResNet34 network for feature extraction. After feature matching is completed for each target image, a 512×1 feature vector is output. By calculating the cosine distance between the M feature vectors of the first camera and the N feature vectors of the second camera, a cosine distance matrix of size M×N is constructed. The coordinates of the target points in the M dictionaries of the first camera are constructed into the form (x, y, 1), and the coordinates of the target points in the target frame are extracted to form a 3×M midpoint coordinate matrix L1. The same process is performed on the second camera to form an N×3 midpoint coordinate matrix L2. The transpose of L2×F×L1 is calculated, and then the elements in L2×F×L1 with absolute values ​​less than a set value are reassigned to 0. This yields the final geometric similarity matrix.

4. The ranging method based on a combined binocular camera according to claim 1, characterized in that, Step S4 is to fuse and calculate the cosine distance matrix and the geometric similarity matrix. Let the cosine distance matrix be D and the geometric similarity matrix be G. Then the final similarity matrix is ​​W=(D^2+G^2) / 2. The similarity matrix is ​​used to obtain matching targets using the Hungarian algorithm. The result actually returns the indices of the M detected targets in the first camera and the N detected targets in the second camera that match. The Hungarian algorithm can only achieve one-way matching: If M=N, then each target in the first camera will have a matching target in the second camera; If M≠N, then some targets may not appear in another camera, or the pre-trained detection network may have made a mistake, resulting in some targets being missed or over-detected. Calculate the row and column indices of the matching matrix W, construct the matching degree vector, and obtain the matching target when the sum of the elements of the matching degree vector is minimized.

5. The ranging method based on a combined binocular camera according to claim 1, characterized in that, In step S5, the coordinates of point u and the best matching point v' are processed using triangulation to obtain the distance between the target and the first camera. Specifically: (1) Coordinate normalization; Convert the coordinates of points u and v' to homogeneous coordinates. (X1,Y1,1) = K1'×(x1,y1,1); (X2,Y2,1) = K2'×(x2,y2,1); Where u=(x1,y1), v'=(x2,y2), x1 and y1 are the x-coordinate and y-coordinate of point u, respectively; x2 and y2 are the x-coordinate and y-coordinate of point v', respectively. X1 and Y1 are the x and y coordinates of point u transformed from the image coordinate system to the first camera coordinate system, respectively; X2 and Y2 are the x and y coordinates of point v' transformed from the image coordinate system to the second camera coordinate system, respectively. Using the first camera as the world coordinate system, we obtained the intrinsic parameter matrix K1 of the first camera, the intrinsic parameter matrix K2 of the second camera, the rotation matrix R of the second camera, and the translation vector t; K1' is the inverse matrix of K1, and K2' is the inverse matrix of K2; (2) Find the projection matrices P1 and P2 of the first camera and the second camera respectively; P1 = K1×[I|0]; P2 = K2×[R|t]; Where I is a 3×3 identity matrix, 0 is a 3×1 vector consisting entirely of zeros, R is a 3×3 rotation matrix, and t is a 3×1 translation vector; [R|t] is a 3×4 matrix formed by concatenating R and t. (3) Calculate the homogeneous matrix A of [R|t]; A = [[X1×P1[2,:]-P1[0,:]], [Y1×P1[2,:]-P1[1,:]], [X2×P2[2,:]-P2[0,:]], [Y2×P2[2,:]-P2[1,:]]]; P1[2,:] represents the third row of matrix P1, and [X1×P1[2,:]-P1[0,:]] is a shortened form that expands to: [X1×P1[2,0] - P1[0,0], X1×P1[2,1] - P1[0,1], X1×P1[2,2] - P1[0,2], X1×P1[2,3] - P1[0,3]]; (4) Perform SVD decomposition on the aligned matrix A and obtain the distance; The SVD algorithm is used to perform singular value decomposition on A to obtain the eigenvector (X,Y,Z,W) corresponding to the smallest eigenvalue, where W is the scale factor. Then the actual coordinates of the target relative to the first camera are (X / W,Y / W,Z / W), and the final distance is the length of the line segment formed by the coordinates (X / W,Y / W,Z / W) and (0,0,0).

6. The ranging method based on a combined binocular camera according to claim 1, characterized in that, In step S1, when calibrating the binocular camera, the calibration board image of the first camera or the second camera should cover 1 / 2 of the camera's field of view.

7. A server, the server comprising a processor and a memory, the memory storing at least one program, characterized in that, The program is loaded and executed by the processor to implement the ranging method based on a combined binocular camera as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing at least one program, characterized in that, The program is loaded and executed by a processor to implement the ranging method based on a combined binocular camera as described in any one of claims 1 to 6.

Images