A binocular camera-based intelligent recognition distance measurement method and device

Abstract

The application discloses a kind of intelligent identification distance measurement method and device based on binocular camera, the method includes: obtaining camera calibration parameter, complete the stereoscopic calibration of binocular camera and the like function;Obtain the image data provided by binocular camera for to be identified ranging;Obtain the disparity calculation and depth calculation result of input image;Using target detection model, the image content to be identified is identified, and target coordinate information is obtained;The distance between target and camera is measured on the basis of the coordinate information obtained in target detection module.This application greatly reduces the calculation parameter and model structure volume of model, reduces the model calculation amount, improves the precision and speed of detection;Binocular ranging technology and target detection technology are combined and matched, high-precision target object real-time detection identification and distance measurement effect are realized.

Description

Technical Field

[0001] This invention relates to the field of visual positioning and detection technology, and in particular to an intelligent recognition distance calculation method and device based on a binocular camera. Background Technology

[0002] With the rapid development of computer vision and image processing technologies, object detection has gradually become a core problem and an important technological component in this field. Due to the different appearances, shapes, and postures of various objects, coupled with interference from factors such as lighting and occlusion during imaging, object detection has always been one of the most challenging problems in computer vision. Correspondingly, the demand for this field is gradually increasing.

[0003] Deep learning-based object detection technology uses convolutional neural networks for feature extraction, achieving stronger adaptability and generalization capabilities through training. YOLO v4 is currently the mainstream algorithm for object detection applications in industrial settings. However, its large size and computational parameters make it unsuitable for real-time detection in devices with limited computing power. Furthermore, a single object detection model can only identify the coordinates of the object on a plane, failing to perceive the actual physical distance between the object and the camera, thus hindering further applications of object detection. Combining camera ranging technology with object detection tasks can effectively address this problem.

[0004] Mainstream camera ranging technologies include monocular camera ranging, binocular camera ranging, structured light ranging, and Time-of-Flight (TOF) ranging. These ranging solutions all have certain limitations. For example, some solutions, such as LiDAR, have high costs and limited recognition rates. Monocular ranging has lower accuracy, and other ranging solutions require the maintenance of a sample database. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] To this end, this invention proposes an intelligent recognition distance calculation method based on a binocular camera, which combines binocular ranging technology with target detection technology to achieve high-precision real-time target detection, recognition, and distance calculation.

[0007] Another objective of this invention is to provide an intelligent recognition distance measurement device based on a binocular camera.

[0008] To achieve the above objectives, this invention proposes an intelligent recognition distance calculation method based on a binocular camera, comprising:

[0009] Obtain the calibration mapping table of the calibrated binocular camera parameters, and use the calibrated binocular camera to acquire the input image data for the distance measurement to be identified;

[0010] Based on the correction mapping table, the left and right views after the input image data is segmented are remapped geometrically to obtain the corrected left and right views, and the depth disparity is calculated on the corrected left and right views using a preset matching algorithm to obtain a disparity map.

[0011] The depth map of the image pixels is calculated using three-dimensional stereo reconstruction to obtain the three-dimensional coordinates of the pixels; and the segmented left view is input into the trained target detection model to obtain the coordinates of the detection box of the target object in the left view, so as to calculate the pixel coordinates of the center point of the target object detection box.

[0012] The distance between the target object and the stereo camera is calculated by using the center point pixel coordinates and the pixel 3D coordinates to determine the pixel object's 3D coordinates in the depth map.

[0013] In addition, the intelligent recognition distance calculation method based on a binocular camera according to the above embodiments of the present invention may also have the following additional technical features:

[0014] Furthermore, in one embodiment of the present invention, obtaining the calibration mapping table of the calibrated binocular camera parameters includes:

[0015] The calibration image size of the calibration board image at different angles is obtained based on the data captured by the binocular camera. The calibration parameters of the binocular camera are obtained based on the calibration image size and the camera calibration algorithm. The calibration parameters include the intrinsic parameters, extrinsic parameters, distortion coefficients and tangential distortion coefficients of the left and right cameras of the binocular camera.

[0016] The reprojection matrix is ​​obtained by performing a stereo camera correction head transformation using the calibration parameters. Based on the reprojection matrix and the basic configuration information of the binocular camera, a correction mapping table of the binocular camera parameters is calculated through correction transformation.

[0017] Further, in one embodiment of the present invention, the step of performing remapping geometric transformation calculations on the left and right views after segmentation of the input image data based on the correction mapping table to obtain the corrected left and right views includes:

[0018] The mapping relationship is constructed based on the correction mapping table and the inverse mapping algorithm;

[0019] Based on the mapping relationship and the bilinear interpolation method, the pixel values ​​of non-integer coordinates are calculated, and the coordinate positions of any pixel in the left and right views in the mapped image are calculated to obtain the corrected coplanar left and right views.

[0020] Furthermore, in one embodiment of the present invention, the step of calculating the depth disparity of the corrected left and right views using a preset matching algorithm to obtain a disparity map includes:

[0021] The corrected left and right views are converted to grayscale to obtain the grayscale left and right views.

[0022] A disparity map is obtained by calculating the disparity of the grayscale left and right views using a semi-global region stereo matching algorithm.

[0023] Furthermore, in one embodiment of the present invention, the step of using a semi-global region stereo matching algorithm to calculate the disparity of the grayscale left and right views to obtain a disparity map includes:

[0024] Image gradient information is obtained by preprocessing the grayscale left and right views using the horizontal Sobel operator;

[0025] Based on the image gradient information and the input image data, gradient cost and SAD cost are obtained according to the sampling method, and the gradient cost and SAD cost are matched and calculated through the SAD window to obtain the cost calculation result.

[0026] Based on the cost calculation results, multi-path cost aggregation is performed to obtain the total matching cost;

[0027] Based on the total matching cost, various disparity optimization operations are used to remove erroneous disparity information in the grayscale left and right views, thereby completing the optimization of the disparity map.

[0028] Furthermore, in one embodiment of the present invention, the step of calculating the depth map of image pixels using three-dimensional stereo reconstruction to obtain the three-dimensional coordinates of pixels includes:

[0029] Based on the disparity map and the reprojection matrix, a mapping map of the same size as the disparity map is obtained;

[0030] Based on the mapping map, the disparity map is remapped to three-dimensional space so that each pixel in the disparity map has three-dimensional coordinates in the camera coordinate system.

[0031] Further, in one embodiment of the present invention, the step of inputting the segmented left view into a trained target detection model for target recognition to obtain the coordinates of the detection box of the target object in the left view, and calculating the pixel coordinates of the center point of the target object detection box, includes:

[0032] After normalizing the segmented left view, it is input into the trained target detection model to obtain the predicted output data of different grid images, and the predicted output data is transformed to obtain the relative offset of the center point pixel coordinates.

[0033] The position result of the target object detection box is obtained based on the relative offset of the coordinates of each grid point and the center point pixel coordinates of the different grid images;

[0034] The position results of the target object detection box are sorted by score and filtered by non-maximum suppression, so as to obtain the prior box target recognition result of the center point pixel coordinates of the target object detection box according to the sorting and filtering results.

[0035] Furthermore, in one embodiment of the present invention, based on the point-to-point distance formula in three-dimensional space, the three-dimensional coordinates of the pixels of the effective center point of the prior box reconstructed by the depth map are used as input, and the lens of the binocular camera is used as the origin of the three-dimensional coordinate system to calculate the spatial distance from the effective center point of the prior box to the origin.

[0036] To achieve the above objectives, another aspect of the present invention proposes an intelligent recognition distance measurement device based on a binocular camera, comprising:

[0037] The camera calibration module is used to obtain the calibration mapping table of the binocular camera parameters and to use the calibrated binocular camera to obtain the input image data for the distance measurement to be identified.

[0038] The disparity calculation module is used to perform remapping geometric transformation calculation on the left and right views after the input image data is segmented based on the correction mapping table to obtain the corrected left and right views, and to use a preset matching algorithm to perform depth disparity calculation on the corrected left and right views to obtain a disparity map.

[0039] The target detection module is used to calculate the depth map of the image pixels of the disparity map using three-dimensional stereo reconstruction to obtain the three-dimensional coordinates of the pixels; and to input the segmented left view into the trained target detection model to perform target recognition to obtain the detection box coordinates of the target object in the left view, so as to calculate the pixel coordinates of the center point of the target object detection box.

[0040] The distance calculation module is used to calculate the three-dimensional coordinates of the pixel object in the depth map based on the center point pixel coordinates and the pixel three-dimensional coordinates, so as to calculate the distance between the target object and the stereo camera.

[0041] The binocular camera-based intelligent recognition distance calculation method and device of this invention greatly reduces the model's calculation parameters and structural volume, reduces the model's computational load, and improves the accuracy and speed of detection. By combining binocular ranging technology with target detection technology, it achieves high-precision real-time target detection, recognition, and distance calculation.

[0042] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0043] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0044] Figure 1 This is a flowchart of an intelligent recognition distance calculation method based on a binocular camera according to an embodiment of the present invention;

[0045] Figure 2 This is an organizational chart of the intelligent recognition distance measurement method based on a binocular camera according to an embodiment of the present invention;

[0046] Figure 3 This is a logic flowchart of an intelligent recognition distance measurement method based on a binocular camera according to an embodiment of the present invention;

[0047] Figure 4 This is a structural diagram of a binocular ranging imaging model according to an embodiment of the present invention;

[0048] Figure 5 This is a structural diagram of the GSYOLO target detection model according to an embodiment of the present invention;

[0049] Figure 6 This is a structural diagram of an intelligent recognition distance measurement device based on a binocular camera according to an embodiment of the present invention. Detailed Implementation

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] The following description, with reference to the accompanying drawings, describes an intelligent recognition distance calculation method and apparatus based on a binocular camera according to an embodiment of the present invention.

[0053] Figure 2 This is an organizational chart of the present invention, such as Figure 2As shown, the system is divided into modules such as camera calibration and stereo correction, image acquisition and preprocessing, disparity acquisition and depth calculation, and target detection and distance measurement. The camera calibration module acquires camera calibration parameters to perform stereo calibration of the binocular camera; the data acquisition module acquires image data from the binocular camera for distance measurement; the disparity and depth calculation module acquires the disparity and depth calculation results of the input image; and the target detection and distance measurement module uses a target detection model to identify the content of the image and acquire target coordinate information, and then calculates the distance between the target and the camera based on the coordinate information acquired by the target detection module.

[0054] Figure 3 The following is a flowchart of the ranging algorithm of the present invention. Figure 3 As shown, images of the calibration board at different angles are captured using a binocular camera. The camera calibration module is used to acquire various calibration parameters for the binocular camera and perform stereo calibration. The calibrated binocular camera is activated, capturing image data for detection and localization. The left and right views of the binocular camera are segmented. After stereo calibration of the input image data using the camera's calibration parameters, SGBM (Semi-Global Block Matching) is used to obtain the disparity map of the binocular image, and a 3D coordinate system is reconstructed to obtain its 3D depth map. Using the segmented left view of the binocular camera as a reference, the GSYOLO target recognition algorithm is called to obtain the bounding box coordinates and target category of the target object in the image. The coordinates of the bounding box in the image are retrieved, the midpoint coordinates of the bounding box are obtained, and the 3D coordinates of the object in the depth map are calculated. The distance calculation module is then called to calculate the distance between the target object and the camera module.

[0055] Figure 1 This is a flowchart of an intelligent recognition distance calculation method based on a binocular camera according to an embodiment of the present invention.

[0056] like Figure 1 As shown, the method includes, but is not limited to, the following steps:

[0057] S1. Obtain the calibration mapping table of the calibrated binocular camera parameters, and use the calibrated binocular camera to obtain the input image data for the distance measurement to be identified.

[0058] Specifically, using a 9*7 checkerboard image as a calibration template, a binocular camera is used to capture images of the calibration board at different angles. The camera calibration algorithm is then invoked to obtain various calibration parameters for the binocular camera.

[0059] Preferably, the calibration parameters include the intrinsic parameters (fx, fy, cx, cy) and extrinsic parameters (rotation matrix R and translation vector T of the left camera relative to the right camera) of the binocular camera, and distortion coefficients (including radial distortion coefficients k1, k2, k3; tangential distortion coefficients p1, p2).

[0060] Furthermore, by calibrating the parameter information, a stereo camera correction head transformation is performed to obtain a new camera matrix. This matrix, combined with the basic configuration information of the binocular camera, is used to calculate the camera's correction mapping table through a correction transformation. The result of this correction mapping table is used to calculate the remapping geometric transformation.

[0061] As an embodiment of the present invention, the calibration method for a binocular camera can be:

[0062] 1) Use a binocular camera to capture images of a calibration board (9*7) with a black and white checkerboard pattern from different angles and directions, cut the images from the left and right cameras, save them separately, and obtain the size attributes of the images;

[0063] 2) Traverse the left and right folders, read the images as grayscale images, extract the coordinates of the object points in the chessboard image in 3D coordinates, extract the corner points of the image, find the precise position of the sub-pixel points based on the corner points as the image points in the real world 2D coordinate system of the image, and match and store the object points and image points as the preliminary data for obtaining the calibration parameters.

[0064] 3) Based on the principle of Zhang Zhengyou's calibration method, a camera intrinsic and extrinsic parameter estimation model is constructed to calculate the initial intrinsic parameters of the left and right cameras, estimate the initial camera attitude, and thus calculate the initial calibration parameter values ​​of the camera.

[0065] 4) Perform stereo camera calibration calculations. After minimizing the total reprojection error of points in all views of the left and right cameras, perform stereo calibration based on the epipolar correction algorithm principle using camera parameters to obtain the calibration mapping table data of the left and right cameras.

[0066] 5) The obtained calibration image size, calibration parameters, correction mapping table, reprojection matrix Q, and other data are written into the configuration file as configuration information for easy use in subsequent distance calculations.

[0067] The parameter configuration information of this invention embodiment is shown in Table 1:

[0068] Table 1

[0069]

[0070] Furthermore, a binocular image of the target to be detected is captured using a binocular camera. This image is then segmented into left and right view images, and the image format is converted to grayscale to reduce the impact of illumination on disparity calculation and improve the accuracy of disparity calculation. The left view image is then resized without distortion, converting the image size to a fixed 416*416 pixels, which serves as the input image for detection and distance calculation. This binocular ranging imaging model of the present invention, as follows... Figure 4 As shown.

[0071] S2, based on the correction mapping table, performs remapping geometric transformation calculation on the left and right views after segmentation of the input image data to obtain the corrected left and right views, and uses a preset matching algorithm to calculate the depth disparity of the corrected left and right views to obtain the disparity map.

[0072] Specifically, based on the aforementioned correction mapping table, the provided segmented left and right views undergo remapping geometric transformation calculations using bilinear interpolation. On this basis, the disparity maps of the left and right views are obtained through the semi-global stereo matching algorithm of SGBM, providing dense matching pairs for the corrected left and right views. Based on the obtained reprojection matrix Q, the image pixel depth is calculated using a 3D stereo reconstruction method, converting pixel coordinates into 3D coordinates.

[0073] Furthermore, using the obtained calibration mapping table of the left and right views as parameters, a mapping relationship is constructed based on the inverse mapping algorithm. Bilinear interpolation is then used to calculate the pixel values ​​at non-integer coordinates, thus calculating the coordinate positions of any pixel in the left and right view images within the mapped image, thereby obtaining the coplanar left and right view images after stereo calibration. The formula is:

[0074] dst(x, y) = src(f x (x, y), f y (x, y))

[0075] Understandably, the purpose of this step is to ensure that the subsequent 3D reconstruction distance and disparity calculations are based on binocular stereo matching. This requires knowing the relative positions of the two cameras and aligning the left and right planes to the same observation plane to ensure strict pixel alignment on the cameras. However, in actual binocular camera shooting, the optical centers of the cameras are not parallel. Therefore, when searching for matching points in stereo matching, a search needs to be performed across the entire image. By using stereo correction based on epipolar mapping relationships, the epipolar lines in both images are made perfectly horizontal. This ensures that the pixel positions of the same point in space are in the same row in both images, thus reducing the matching search from two dimensions to one dimension, saving computational resources and quickly eliminating false matching points.

[0076] Furthermore, the obtained corrected left and right views are converted to grayscale to reduce the impact of lighting on stereo matching. The SGBM algorithm is then used to calculate disparity, resulting in a disparity map. The purpose is to provide dense matching pairs for the corrected left and right views, thereby finding the corresponding pixels in the right view for pixels in the left view, and thus calculating the disparity information of the left and right camera images.

[0077] As an embodiment of the present invention, the calculation process of the semi-global region stereo matching algorithm can be specifically divided into several steps: (1) image preprocessing, (2) matching cost calculation, (3) cost aggregation, and (4) disparity optimization.

[0078] (1) Image preprocessing steps: The horizontal Sobel operator is used to preprocess the image to obtain the gradient information of the image.

[0079] The calculation formula is expressed as follows:

[0080] Sobel(x,y)=2[P(x+1,y)-P(x-1,y)]+P(x+1,y-1)-P(x-1,y-1)+P(x+1,y+1)-P(x-1,y+1)

[0081] After preprocessing calculations, a new image is obtained. Let the value of P represent the pixel values ​​of the input image:

[0082] P new If we represent pixel values ​​in the new image, then the following mapping relationship exists:

[0083]

[0084] (2) Matching cost calculation: The cost calculation consists of two parts: one is the gradient cost calculated based on the sampling method after preprocessing the gradient information, and the other is the SAD cost obtained by the sampling method of the original image. Then, the two costs are superimposed and summed through the SAD window.

[0085] (3) Cost Aggregation: The matching costs in each direction are added together according to the dynamic programming approach, i.e., multi-path cost aggregation is performed to obtain the total matching cost. The calculation method is as follows:

[0086]

[0087] Where L is the aggregate cost result of the current path, C(p,d) represents the initial cost value; p represents a pixel, d represents disparity, and r represents the path; P1 and P2 are the smoothing penalty coefficients for small and large disparity differences between a pixel and its neighbors; pr represents the pixel within the path of pixel p; the total matching cost is obtained by summing the matching costs of all paths of pixel p along the r direction.

[0088]

[0089] (4) Parallax optimization: By performing post-processing steps such as uniqueness detection, subpixel interpolation, left-right consistency detection, and connected component noise filtering, erroneous parallax is eliminated, thereby optimizing and refining the parallax results.

[0090] S3 uses 3D stereo reconstruction to calculate the depth map of the image pixels in the disparity map to obtain the 3D coordinates of the pixels; and inputs the segmented left view into the trained target detection model to obtain the detection box coordinates of the target object in the left view, so as to calculate the pixel coordinates of the center point of the target object detection box.

[0091] Specifically, the image pixel depth map is calculated using a three-dimensional reconstruction method based on the disparity map to obtain the three-dimensional coordinates of the pixels. The algorithm returns a 3-channel matrix that stores the three-dimensional coordinates of the pixels in the left camera coordinate system, thereby obtaining the three-dimensional coordinates (x, y, z) of all pixels in the detected image in the left camera coordinate system in actual use.

[0092] The formula for calculating pixel depth after obtaining the disparity map is as follows:

[0093]

[0094] Where f represents the pixel focal length; b is the baseline length; d represents the parallax; c xl With c xr The column coordinates are the principal points of the two cameras.

[0095] Specifically, based on the obtained disparity map and using the obtained reprojection matrix Q, a mapping map of the same size as the disparity map is obtained, with each pixel having three channels. These channels store the corresponding 3D coordinates of the pixel's position in the camera coordinate system. This remaps the disparity map into 3D space, thus obtaining the 3D coordinates of each pixel in the camera coordinate system. Specifically, the following calculations are performed on each pixel p(x,y) and its corresponding disparity d(x,y):

[0096] [XYZW] T =Q×[xyd(x,y)1] T

[0097] 3dImage(x,y)=(X / W,Y / W,Z / W)

[0098] S4 calculates the three-dimensional coordinates of the pixel object in the depth map based on the center point pixel coordinates and the pixel 3D coordinates, in order to calculate the distance between the target object and the stereo camera.

[0099] Understandably, using the aforementioned cropped left view image as a reference, a lossless resize operation is performed to convert it into a 416*416 size input image. After normalization, this image is input into the trained YOLOv4-Mobilenetv3 object detection algorithm model for prediction and recognition. The network model's prediction results are then decoded to obtain the predicted bounding box coordinates, predicted category, and confidence score in the left view. Based on the predicted bounding box coordinates, the coordinates of the points within the predicted bounding boxes of all objects are calculated.

[0100] Specifically, the GSYOLO target detection model of the present invention, such as Figure 5 As shown, the target detection algorithm model outputs prediction results from three feature layers with shapes of (N, 13, 13, 255), (M, 26, 26, 255), and (N, 52, 52, 255), respectively. These results correspond to the positions of three predicted prior boxes on grids of 13×13, 26×26, and 52×52 in each image. This result does not correspond to the final predicted box position on the image; it can only be determined after the decoding operation. In the dataset, the three prediction results of the feature layer are reshaped to obtain (N, 13, 13, 6), (M, 26, 26, 6), and (N, 52, 52, 6). Here, the last dimension 6 contains 4+1+1, representing the offset of x and y relative to the center point, the width and height of the prior box, the confidence of the prior box, and the classification result, respectively. The detection model network divides the entire image into 13×13, 26×26, and 52×52 grids. Each network point is responsible for the detection of one region. The center position of the predicted box is obtained by adding the offset of x and y relative to the center point to each grid point. Combined with other data, the position of the entire predicted box is calculated, which is the prediction result. The position results of the predicted boxes obtained here are then sorted by score and filtered by non-maximum suppression to obtain the final target prior box and the target recognition result prediction.

[0101] Furthermore, the coordinates of the obtained center point pixel are obtained, and the distance to the coordinate point is calculated by combining the coordinates of the center point pixel in three-dimensional space obtained from the depth map.

[0102] Preferably, the target distance calculation formula is based on the point-to-point distance formula in three-dimensional space. It takes the three-dimensional coordinates obtained by three-dimensional reconstruction of the effective pixels of each prior box center point through the depth map as input, and takes the lens as the origin of the three-dimensional coordinate system to calculate the spatial distance from the effective center point of the prior box to the origin.

[0103] The distance formula is expressed as follows:

[0104] The binocular camera-based intelligent distance measurement method according to embodiments of the present invention combines target detection and recognition technology with binocular ranging technology. It uses program-level computer vision image processing technology instead of relying on third-party Matlab tools for image calibration parameter acquisition and stereo matching. It replaces the original backbone network CSPDarknet53 of YOLOv4 with the lightweight network model GhostNet, which greatly reduces the model's computational parameters and structural volume, reduces the computational load, and improves the accuracy and speed of detection. By combining binocular ranging technology with target detection technology, it achieves high-precision real-time target detection, recognition, and distance measurement.

[0105] To achieve the above embodiments, such as Figure 6 As shown, this embodiment also provides an intelligent recognition distance calculation device 10 based on a binocular camera. The device 10 includes a camera calibration module 100, a parallax calculation module 200, a target detection module 300, and a distance calculation module 400.

[0106] The camera calibration module 100 is used to obtain the calibration mapping table of the binocular camera parameters and to use the calibrated binocular camera to obtain the input image data for the distance measurement to be identified.

[0107] The disparity calculation module 200 is used to perform remapping geometric transformation calculation on the left and right views after the input image data is segmented based on the correction mapping table to obtain the corrected left and right views, and to use a preset matching algorithm to perform depth disparity calculation on the corrected left and right views to obtain a disparity map.

[0108] The target detection module 300 is used to calculate the depth map of the image pixels of the disparity map using three-dimensional stereo reconstruction to obtain the three-dimensional coordinates of the pixels; and to input the segmented left view into the trained target detection model to obtain the detection box coordinates of the target object in the left view, so as to calculate the pixel coordinates of the center point of the target object detection box.

[0109] The distance calculation module 400 is used to calculate the three-dimensional coordinates of the pixel object in the depth map based on the center point pixel coordinates and the pixel three-dimensional coordinates, so as to calculate the distance between the target object and the stereo camera.

[0110] Furthermore, the aforementioned camera calibration module 100 is also used for:

[0111] The calibration image size of the calibration board image at different angles is obtained based on the data captured by the binocular camera. The calibration parameters of the binocular camera are obtained based on the calibration image size and the camera calibration algorithm. The calibration parameters include the intrinsic parameters, extrinsic parameters, distortion coefficients and tangential distortion coefficients of the left and right cameras of the binocular camera.

[0112] The reprojection matrix is ​​obtained by performing a stereo camera correction head transformation using calibration parameters. Based on the reprojection matrix and the basic configuration information of the stereo camera, a correction mapping table of the stereo camera parameters is calculated through correction transformation.

[0113] The binocular camera-based intelligent recognition distance measurement device according to embodiments of the present invention combines target detection and recognition technology with binocular ranging technology. It uses program-level computer vision image processing technology instead of relying on third-party tools like Matlab for image calibration parameter acquisition and stereo matching. By replacing the original backbone network CSPDarknet53 of YOLOv4 with the lightweight network model GhostNet, the computational parameters and model structure volume of the model are greatly reduced, the computational load is reduced, and the detection accuracy and speed are improved. By combining binocular ranging technology with target detection technology, high-precision real-time target detection, recognition, and distance measurement are achieved.

[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A binocular camera-based intelligent recognition distance measurement method, characterized in that, The method comprises the following steps: obtaining a correction mapping table of calibrated binocular camera parameters, and obtaining input image data to be identified and ranged by using the calibrated binocular camera; based on the correction mapping table, performing re-mapping geometric transformation calculation on left and right views segmented from the input image data to obtain corrected left and right views, and using a preset matching algorithm to perform depth disparity calculation on the corrected left and right views to obtain a disparity map; using three-dimensional reconstruction to perform depth map calculation on image pixels of the disparity map to obtain three-dimensional coordinates of the pixels; inputting the segmented left view into a trained target detection model to perform target identification to obtain detection frame coordinates of a target object in the left view, so as to calculate a center point pixel coordinate of the target object detection frame; based on the center point pixel coordinate and the three-dimensional coordinates of the pixels, calculating a three-dimensional coordinate of a pixel object in the depth map of the center point pixel coordinate, so as to calculate a distance between the target object and the binocular camera; the method of obtaining the correction mapping table of the calibrated binocular camera parameters comprises: obtaining the size of the calibration image of the calibration board at different angles based on the binocular camera shooting data, and obtaining the calibration parameters of the binocular camera based on the size of the calibration image and a camera calibration algorithm; wherein the calibration parameters include the intrinsic parameters, extrinsic parameters, distortion coefficients and tangential distortion coefficients of the left and right cameras of the binocular camera; using the calibration parameters to perform stereo camera correction head transformation to obtain a re-projection matrix, and based on the re-projection matrix and the basic configuration information of the binocular camera, calculating the correction mapping table of the binocular camera parameters through correction transformation; wherein the disparity map of the left and right views is obtained through calculation of the SGBM semi-global stereo matching algorithm.

2. The method of claim 1, wherein, the method of performing re-mapping geometric transformation calculation on the left and right views segmented from the input image data based on the correction mapping table to obtain corrected left and right views comprises: constructing a mapping relationship based on the correction mapping table and an inverse mapping algorithm; calculating the pixel value of a non-integer coordinate based on the mapping relationship and a bilinear interpolation method, and calculating the coordinate position of any pixel in the mapped image to obtain the corrected coplanar left and right views.

3. The method of claim 2, wherein, the method of performing depth disparity calculation on the corrected left and right views to obtain a disparity map using a preset matching algorithm comprises: performing grayscale processing on the corrected left and right views to obtain grayscale left and right views; performing disparity calculation on the grayscale left and right views using a semi-global area stereo matching algorithm to obtain a disparity map.

4. The method of claim 3, wherein, the method of performing disparity calculation on the grayscale left and right views using a semi-global area stereo matching algorithm to obtain a disparity map comprises: performing preprocessing operation on the grayscale left and right views using a horizontal Sobel operator to obtain image gradient information; based on the image gradient information and the input image data, respectively obtaining gradient cost and SAD cost according to a sampling method, and calculating the matching cost of the gradient cost and the SAD cost through a SAD window to obtain a cost calculation result; based on the cost calculation result, performing multi-path cost aggregation calculation to obtain a total matching cost; The erroneous disparity information of the left and right views after the grayscale processing is removed by using a plurality of disparity optimization operations based on the total matching cost, so as to complete the optimization of the disparity map.

5. The method of claim 3, wherein, The depth map calculation of the image pixels of the disparity map by using the three-dimensional reconstruction includes: A mapping map with the same size as the disparity map is obtained based on the disparity map and the re-projection matrix; The disparity map is re-mapped into the three-dimensional space based on the mapping map, so as to obtain the three-dimensional coordinates of each pixel point in the disparity map in the camera coordinate system.

6. The method of claim 1, wherein, The detection frame coordinates of the target object in the left view are obtained by inputting the segmented left view into the trained target detection model for target recognition, so as to calculate the center point pixel coordinates of the target object detection frame, including: The normalized segmented left view is input into the trained target detection model to obtain the prediction output data of different grid images, and the relative offset of the center point pixel coordinates is obtained by performing data transformation on the prediction output data; The position result of the target object detection frame is obtained based on the relative offset of the center point pixel coordinates and the coordinates of each grid point of the different grid images; The position result of the target object detection frame is sorted and non-maximum suppressed, so as to obtain the prior frame target recognition result of the center point pixel coordinates of the target object detection frame according to the sorting and screening result.

7. The method of claim 6, wherein, The three-dimensional coordinates of the pixel of the effective center point of the prior frame are taken as input by performing three-dimensional reconstruction on the pixel of the effective center point of the prior frame through the depth map, and the lens of the binocular camera is taken as the origin of the three-dimensional coordinate system, so as to calculate the spatial distance from the effective center point of the prior frame to the origin.

8. A binocular camera-based intelligent recognition distance measuring device, characterized in that, It includes: A camera calibration module is configured to obtain a correction mapping table of calibrated binocular camera parameters, and to obtain input image data to be recognized and measured by using the calibrated binocular camera; A disparity calculation module is configured to perform re-mapping geometric transformation on left and right views segmented from the input image data based on the correction mapping table to obtain corrected left and right views, and to perform depth disparity calculation on the corrected left and right views by using a preset matching algorithm to obtain a disparity map; A target detection module is configured to perform depth map calculation of image pixels of the disparity map by using three-dimensional reconstruction to obtain three-dimensional coordinates of pixels; and to input the segmented left view into a trained target detection model to obtain detection frame coordinates of a target object in the left view for target recognition, so as to calculate center point pixel coordinates of the target object detection frame; A distance measurement module is configured to calculate pixel object three-dimensional coordinates of the center point pixel coordinates in the depth map based on the center point pixel coordinates and the three-dimensional coordinates of the pixels, so as to calculate the distance between the target object and the binocular camera. The camera calibration module is further configured to: obtain calibration image sizes of calibration board images at different angles based on binocular camera shooting data, and obtain calibration parameters of the binocular camera based on the calibration image sizes and a camera calibration algorithm; wherein the calibration parameters include intrinsic parameters, extrinsic parameters, distortion coefficients, and tangential distortion coefficients of left and right cameras of the binocular camera. The calibration parameters are used to perform a stereo camera correction head transformation to obtain a re-projection matrix, and based on the re-projection matrix and basic configuration information of the binocular camera, a correction mapping table of the binocular camera parameters is calculated through a correction transformation. The disparity map of the left and right views is obtained through calculation of an SGBM semi-global stereo matching algorithm.

Images