Image distance acquisition method and device, equipment and storage medium

Abstract

The invention discloses an image distance acquisition method and device, equipment and a storage medium, and belongs to the technical field of image processing. The image distance acquisition method comprises the following steps: acquiring target scene point cloud data acquired by a laser radar and target scene image data acquired by a camera; wherein the target scene point cloud data comprises distance data of each point; selecting a target object area from the target scene image data; the target object area is a rectangular frame or a polygonal frame; and obtaining distance data of the target object region based on the target scene point cloud data. The distance data of the target object area is obtained based on the target scene point cloud data collected by the laser radar and the target scene image data collected by the camera, the advantages of the two data can be fully utilized, the laser radar provides accurate distance information, the problem that the distance information is unreliable in a traditional image-based object detection technology is solved, and the detection accuracy is improved. And the distance measurement precision of the target object is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image distance acquisition method, device, equipment and storage medium. Background Art

[0002] Image-based object detection technology can identify objects in images. However, the distance measurement accuracy of object detection technology is limited by factors such as ambient light, object texture, and camera parameters. Especially in long-distance and low-contrast scenes, it is often difficult to obtain reliable distance information based on image data alone. Summary of the invention

[0003] In response to the above problems, the present application provides an image distance acquisition method, device, equipment and storage medium, which solves the technical problem that image-based object detection technology is difficult to obtain reliable distance information due to various factors.

[0004] The embodiment of the present application provides an image distance acquisition method, comprising: acquiring target scene point cloud data collected by a laser radar and target scene image data collected by a camera; wherein the target scene point cloud data includes distance data of each point;

[0005] Selecting a target object region from the target scene image data; the target object region is a rectangular frame or a polygonal frame;

[0006] Based on the target scene point cloud data, the distance data of the target object area is obtained.

[0007] In some embodiments, obtaining distance data of a target object area based on the target scene point cloud data includes:

[0008] The target scene point cloud data and the target scene image data are fused to obtain target scene fusion data; the target scene fusion data includes the target scene image data and the distance data of each pixel block in the target scene image data;

[0009] Obtain each pixel block corresponding to the target object area to obtain distance data of the target object area.

[0010] In some embodiments, the target scene point cloud data and the target scene image data are fused to obtain the target scene fused data, including:

[0011] Projecting the target scene point cloud data onto the pixel points in the target scene image data;

[0012] Based on the distribution of each pixel point corresponding to each point in the target scene point cloud data in the target scene image data, the target scene image data is divided into a plurality of pixel blocks;

[0013] Based on the pixel points with distance data contained in each pixel block, the distance data of each pixel block is determined to obtain the target scene fusion data.

[0014] In some embodiments, fusing the target scene point cloud data and the target scene image data to obtain target scene fusion data further includes:

[0015] Eliminate outliers in the target scene point cloud data.

[0016] In some embodiments, projecting the target scene point cloud data onto pixel points in the target scene image data includes:

[0017] Based on the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera, the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera are converted into a unified coordinate system;

[0018] Based on the intrinsic parameter matrix corresponding to the camera, the target scene point cloud data in the unified coordinate system is projected onto the pixel points in the target scene image data.

[0019] In some embodiments, a method for determining an external parameter relationship between a coordinate system corresponding to a laser radar and a coordinate system corresponding to a camera includes:

[0020] Based on the point cloud data of the calibration plate collected by the lidar and the image data of the calibration plate collected by the camera, the external parameter relationship between the coordinate system corresponding to the lidar and the coordinate system corresponding to the camera is confirmed through the camera, lidar calibration algorithm and lidar point cloud registration algorithm. The external parameter relationship includes the rotation matrix and the translation vector.

[0021] In some embodiments, the method for determining the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera further includes:

[0022] Use optimization algorithms to optimize external parameter relationships;

[0023] The optimization algorithms include:

[0024] Determine the objective function, which is expressed as follows:

[0025]

[0026] Among them, e i represents the error corresponding to the external parameter relationship at the i-th iteration, e i =P camera -(R i *P lidar +t i ), P camera Represents the coordinate data of the image data of the calibration plate in the coordinate system corresponding to the camera, R i represents the rotation matrix corresponding to the i-th iteration, Plidar Represents the coordinate data of the point cloud data of the calibration plate in the coordinate system corresponding to the laser radar, t i represents the translation vector corresponding to the i-th iteration, and n is the total number of iterations;

[0028] The external parameter relationship is iteratively updated by the nonlinear least squares method, and the external parameter relationship corresponding to the minimization of the objective function is confirmed as the external parameter relationship between the coordinate system corresponding to the lidar and the coordinate system corresponding to the camera.

[0029] In some embodiments, selecting a target object area from the target scene image data includes:

[0030] Select the target object range from the target scene image data by using target recognition technology;

[0031] By presetting the proportional relationship, the target object range is narrowed down to obtain the target object area.

[0032] In some embodiments, obtaining distance data of a target object area based on the target scene point cloud data includes:

[0033] Based on the target object area, determining a target object space corresponding to the target object area in a laser radar coordinate system;

[0034] The point cloud data contained in the target object space is obtained, and then the distance data of the target object area is obtained.

[0035] Accordingly, the embodiment of the present application further provides an image distance acquisition device, including:

[0036] The first module is configured to obtain target scene point cloud data collected by the laser radar and target scene image data collected by the camera; wherein the target scene point cloud data includes distance data of each point;

[0037] The second module is configured to frame a target object area from the target scene image data; the target object area is a rectangular frame or a polygonal frame;

[0038] The third module is configured to obtain distance data of the target object area based on the target scene point cloud data.

[0039] Accordingly, the embodiment of the present application further provides an image distance acquisition device, including:

[0040] at least one processor;

[0041] at least one memory for storing at least one program;

[0042] When at least one program is executed by at least one processor, the at least one processor implements the image distance acquisition method in the above embodiment.

[0043] Correspondingly, the present application also provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the image distance acquisition method in the above embodiment.

[0044] The beneficial effect of the present application is that the present application provides an image distance acquisition method, which includes: acquiring target scene point cloud data collected by a laser radar and target scene image data collected by a camera; wherein the target scene point cloud data includes distance data of each point; selecting a target object area from the target scene image data; the target object area is a rectangular frame or a polygonal frame; based on the target scene point cloud data, acquiring distance data of the target object area. The present application obtains distance data of the target object area based on target scene point cloud data collected by a laser radar and target scene image data collected by a camera, and can make full use of the advantages of both. The laser radar provides accurate distance information, solves the problem of unreliable distance information in traditional image-based object detection technology, and improves the distance measurement accuracy of the target object.

[0045] The present application also provides an image distance acquisition device, which applies the image distance acquisition method in the above embodiment, and thus has all the technical features and technical effects of the above image distance acquisition method, which will not be described in detail here.

[0046] The present application also provides an image distance acquisition device, which implements the image distance acquisition method in the above embodiment, and thus has all the technical features and technical effects of the above image distance acquisition method, which will not be described in detail here.

[0047] The present application also provides a computer-readable storage medium for executing the image distance acquisition method in the above embodiment, so that the above image distance acquisition method has all the technical features and technical effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A schematic flow chart of a method for acquiring image distance provided in an embodiment of the present application;

[0050] Figure 2 A schematic flow chart of another method for acquiring image distance provided in an embodiment of the present application;

[0051] Figure 3 A schematic flowchart of another method for acquiring image distance provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0053] In the description of the present application, it should be understood that the specific embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. The terms "first" and "second" are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features.

[0054] The present application provides an image distance acquisition method, device, equipment and storage medium, which are described in detail below. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments of the present application. In the following embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0055] In the application of target detection in the blind area of ​​electric shovels in mining areas, it is particularly important to accurately obtain the distance of the target identified by the image. The mining environment is complex, and there are many potential obstacles when the electric shovel is operating, such as rocks, mine cars, personnel, etc. If these obstacles are not detected in a timely and accurate manner, serious safety accidents will occur. The blind area of ​​the electric shovel is large and changes dynamically, making it impossible to fully cover and accurately perceive all potential threats by relying solely on traditional vision or radar systems. Therefore, accurately identifying and measuring the target distance has become a key technology to improve the safety and efficiency of electric shovel operations in mining areas.

[0056] After years of development, image-based object detection technology has achieved a high recognition rate. Modern algorithms, such as YOLO, can efficiently and accurately detect targets in images in real time and provide rich category information. However, the distance measurement accuracy of these algorithms is limited by factors such as ambient light, object texture, and camera parameters. Especially in long-distance and low-contrast scenes, it is often difficult to obtain reliable distance information based on image data alone.

[0057] LiDAR measures the distance to obstacles by emitting and receiving laser beams, and has the advantages of high precision and fast response. LiDAR can provide reliable distance information under any lighting conditions and is a high-precision ranging sensor independent of ambient light. However, target recognition and detection based on LiDAR point cloud has the problems of few recognition types and low recognition rate. Although the point cloud data generated by LiDAR has precise geometric information, its resolution is relatively low and it cannot provide the rich texture information of images, which makes it difficult to accurately identify and classify targets relying solely on LiDAR data.

[0058] In view of this, the present application proposes an image distance acquisition method, aiming to solve at least one of the above technical problems.

[0059] See also Figure 1 As shown, Figure 1 A schematic flow chart of an image distance acquisition method provided in an embodiment of the present application. The present application provides an image distance acquisition method, comprising: acquiring target scene point cloud data collected by a laser radar and target scene image data collected by a camera; wherein the target scene point cloud data includes distance data of each point; selecting a target object area from the target scene image data; the target object area is a rectangular frame or a polygonal frame; based on the target scene point cloud data, acquiring distance data of the target object area.

[0060] It should be understood that the target scene point cloud data includes the distance data of each point, and the distance data is calculated from the coordinate data of the point cloud. Among them, x, y, and z are the coordinates of the point in the laser radar coordinate system respectively; from the target scene image data, the target area is selected. This area can be a rectangular frame or a polygonal frame. Image processing techniques such as target recognition, edge detection, and object detection algorithms can be used to automatically or manually select the target object area. Based on the target scene point cloud data collected by the laser radar, the distance data of the target object area is obtained. Since the point cloud data is dense, the distance data of the target object area can include the distance data of multiple points in the target object area, or it can be the distance data representing the target object area selected by the distance data. For example, the average value of the distance data of all points in the target object area can be calculated as the distance data of the target object area. This method can provide the overall distance information of the points in the target object area. The average value of the distance data from all points in the target object area to the center point of the area can also be calculated as the distance data of the target object area. This method can provide distance information of the center position of the target object area; it can also select the distance data of the farthest or closest point in the target object area as the distance data of the target object area, and this method can provide distance information of the farthest point or the closest point in the target object area; it can also perform statistics and distribution analysis on the distance data in the target object area to generate a distance histogram, which can provide the distribution of distance data in the target object area, and help further analysis and processing. It is necessary to select the appropriate form of distance data for the target object area according to the specific application scenario and algorithm requirements. Different forms of distance data can provide different information, which can be selected and processed according to needs.

[0061] For example, in the blind spot detection of electric shovels in mining areas, obstacle detection is required, and then the obstacles need to be identified, and then the distance information corresponding to the obstacles is obtained, so as to ensure that the electric shovel operator can promptly discover potential dangers in the blind spot, improve the safety and efficiency of operations, reduce the risk of accidents, and ensure the smooth progress of mining production.

[0062] Through the above technical scheme, the present application obtains the distance data of the target object area based on the target scene point cloud data collected by the laser radar and the target scene image data collected by the camera, which can make full use of the advantages of both. The laser radar provides accurate distance information, solves the problem of unreliable distance information in the traditional image-based object detection technology, and improves the ranging accuracy of the target object. Further, the combination of the laser radar point cloud and the image data can complement each other and achieve high-precision target distance acquisition. Specifically, the high-precision distance information provided by the laser radar can make up for the shortcomings of image ranging, and the high resolution and rich texture features of the image can improve the target recognition ability of the laser radar point cloud data. In the blind spot detection of the electric shovel in the mining area, the fusion of laser radar and image data technology can not only provide accurate distance measurement, but also effectively improve the types and accuracy of obstacle recognition. Through this fusion method, it can ensure that the electric shovel operator can timely discover the potential dangers in the blind spot, improve the safety and efficiency of the operation, reduce the risk of accidents, and ensure the smooth production of the mining area.

[0063] See also Figure 2 As shown, Figure 2 A schematic flow chart of another method for acquiring image distance provided in an embodiment of the present application. In some embodiments, acquiring distance data of a target object area based on target scene point cloud data includes:

[0064] The target scene point cloud data and the target scene image data are fused to obtain target scene fusion data; the target scene fusion data includes the target scene image data and the distance data of each pixel block in the target scene image data;

[0065] Obtain each pixel block corresponding to the target object area to obtain distance data of the target object area.

[0066] It should be understood that by fusing the target scene point cloud data collected by the laser radar and the target scene image data collected by the camera, the distance information can be associated with the image pixels by matching the point cloud data with the image data to form target scene fusion data, that is, each pixel block in the target scene image data corresponds to a distance data representing the distance of the pixel block. Furthermore, under the premise that the target object area is determined, each pixel block contained in the target object area can be directly obtained, thereby extracting the distance data of the pixel block corresponding to the target object area from the target scene fusion data.

[0067] Through the above technical solution, the present application fuses point cloud data and image data to associate distance information with image pixels. This fusion can provide more accurate distance data and improve the accuracy of distance data compared to using only image data for distance estimation.

[0068] In some embodiments, the target scene point cloud data and the target scene image data are fused to obtain the target scene fused data, including:

[0069] Projecting the target scene point cloud data onto the pixel points in the target scene image data;

[0070] Based on the distribution of each pixel point corresponding to each point in the target scene point cloud data in the target scene image data, the target scene image data is divided into a plurality of pixel blocks;

[0071] Based on the pixel points with distance data contained in each pixel block, the distance data of each pixel block is determined to obtain the target scene fusion data.

[0072] It should be understood that the target scene point cloud data is projected onto the pixel points in the target scene image data. That is, by matching each point in the point cloud with the corresponding pixel point in the image, the distance information of the point cloud data can be mapped to the image data. The division of pixel blocks in the target scene image data can evenly divide the image into pixel blocks of fixed size, such as dividing a 256*256 image into 256*256 pixel blocks; the image can be divided into multiple pixel blocks based on the boundary information of each area within the target object, and the target detection algorithm or edge detection algorithm can be used to obtain the boundary information of each area within the target object, and then the image can be divided into multiple blocks based on the boundary information. This method can better adapt to the shape and size changes of the target object; it can be divided based on pixel density: according to the density distribution of pixel points in the image, the image can be divided into multiple pixel blocks, which can be obtained through pixel clustering algorithms or density estimation methods. To determine the density distribution of pixels, and then divide the image into multiple blocks according to the density distribution. This method can better adapt to the density changes in different areas of the image; it can be based on superpixel division: use the superpixel segmentation algorithm to divide the image into multiple superpixel blocks, and then further divide according to the position and distribution of the superpixel blocks. The superpixel segmentation algorithm can divide the image into areas with similar distances, and then divide the image pixel blocks according to the position and distribution of the superpixel blocks, such as based on the distance difference threshold, and then divide the image pixel blocks according to the position and distance distribution of the superpixel blocks. The difference between the maximum distance and the minimum distance in each pixel block is controlled within the distance difference threshold. In each pixel block, there are pixels with distance data. By statistically analyzing the distance data of the pixels in each pixel block, the distance data of each pixel block can be determined. The average distance, maximum distance, minimum distance or other statistical methods can be selected to determine the distance data of the pixel block. It should be noted that the present application assumes that each pixel block contains at least one pixel with distance data of point cloud data projection, and rich point cloud data details can be provided through laser radar non-repetitive scanning technology.

[0073] Through the above technical solution, more accurate distance data can be obtained by projecting the distance information of the point cloud data onto the pixel points in the image data. Compared with using only image data for distance estimation, fusing point cloud data can improve the accuracy of distance data; by dividing the image data into multiple pixel blocks, and determining the distance data based on the pixel points with distance data in each pixel block, the target scene fusion data can be obtained. This fusion data includes image data and distance data of each pixel block, providing more comprehensive scene information. In addition, based on the distribution of each pixel point corresponding to each point in the target scene point cloud data in the target scene image data, the target scene image data is divided into multiple pixel blocks, and the selection of the pixel block size can ensure that the point cloud data in the pixel block fully reflects the scene details, thereby improving the accuracy of the distance information; and, it can be adaptively adjusted according to the point cloud density and image resolution of different scenes, so it has strong versatility and adaptability. In some embodiments, the target scene point cloud data and the target scene image data are fused to obtain the target scene fusion data, and it also includes:

[0074] Eliminate outliers in the target scene point cloud data.

[0075] It should be understood that outliers that may appear on pixel points when projecting the target scene point cloud data onto the target scene image data are filtered out to ensure the quality of the point cloud data and avoid interference with the image data.

[0076] For example, outliers in the target scene point cloud data can be removed by the K-Nearest Neighbors (KNN) algorithm. Specifically, the distances of the k nearest neighbor points of each point are calculated. If the average distance of the k nearest neighbor points of a point is greater than a set distance threshold, the point can be considered an outlier, which can be judged by the following formula:

[0077]

[0078] Among them, D avg represents the average distance of k nearest neighbor points, D i represents the i-th nearest neighbor point, k represents the total number of nearest neighbor points, which can be set based on actual solution requirements; furthermore, if D avg >D threshold , then the point is an outlier, D thresholdRepresents the set distance threshold, which can be set based on actual solution requirements. It should be noted that the distance between the k nearest neighbors of each point can be obtained by Euclidean distance, Manhattan distance, Chebyshev distance, etc. It should be noted that the definition and removal method of outliers may vary depending on the application scenario and specific algorithm. According to the characteristics and requirements of point cloud data, appropriate outlier detection and removal methods can be selected, such as methods based on statistical analysis, cluster analysis, distance threshold, etc.

[0079] Through the above technical solution, by removing outliers, the quality of point cloud data can be improved, the impact of noise and outliers can be reduced, and the data can be made more accurate and reliable. And the consistency of point cloud data can be ensured to make it more consistent with the actual scene and consistent with the image data. In addition, by removing outliers, the interference to the image data can be reduced, the fusion effect can be improved, and the fused data can be made more accurate and reliable.

[0080] In some embodiments, projecting the target scene point cloud data onto pixel points in the target scene image data includes:

[0081] Based on the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera, the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera are converted into a unified coordinate system;

[0082] Based on the intrinsic parameter matrix corresponding to the camera, the target scene point cloud data in the unified coordinate system is projected onto the pixel points in the target scene image data.

[0083] It should be understood that lidar and cameras usually use different coordinate systems for measurement and data collection. In order to fuse their data, it is necessary to know the external parameter relationship between them, that is, the conversion relationship between the lidar coordinate system and the camera coordinate system, and then convert the coordinate system corresponding to the lidar and the coordinate system corresponding to the camera into a unified coordinate system; it should be noted that the unified coordinate system in this application can be one of the coordinate system corresponding to the lidar and the coordinate system corresponding to the camera.

[0084] Exemplarily, the external parameter relationship estimation process can perform an initial estimation when the laser radar and the camera are roughly parallel, and calculate the rotation vector and the translation vector; then, each point of the laser radar point cloud can be transformed from the laser radar coordinate system to the camera coordinate system by the following formula:

[0085]

[0086] Among them, the coordinates of the point cloud point in the laser radar coordinate system are P lidar =(x lidar ,y lidar ,z lidar ), the point cloud point in the laser radar coordinate system is converted to the coordinates in the camera coordinate system as Pcan =(x can ,y can ,z can ), the external parameter matrix between the two includes the rotation matrix R and the translation vector t = (t x ,t y ,t z ). It should be understood that the rotation matrix R is used to adjust the direction alignment from the lidar coordinate system to the camera coordinate system, and the translation vector t is used to translate the lidar origin to the camera origin.

[0087] Then, the target scene point cloud data is projected onto the pixel points in the target scene image data; specifically, the converted point P can =(x can ,y can ,z can ) is projected onto the two-dimensional image plane captured by the camera using the following formula:

[0088]

[0089] in, is the pixel coordinate of the point cloud projected onto the image, f x 、f y is the focal length of the x-axis and y-axis in the camera coordinate system, c x 、c y is the x-axis and y-axis coordinate position of the main point in the camera coordinate system; it should be noted that when projecting, the depth information z of the laser radar point is retained can As the basis for subsequent calculation of pixel block distance. Then the pixel coordinates of the target scene image data include the corresponding depth information z can It should be noted that the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera are both three-dimensional coordinate systems; the pixel coordinates are two-dimensional coordinates.

[0090] Through the above technical solution, by projecting the point cloud data onto the pixel points in the image data, the consistency of the point cloud data and the image data can be achieved. In this way, the point cloud data can be visualized more intuitively, and compared and analyzed with the image data, providing more comprehensive scene information. In addition, by projecting the distance information of the point cloud data onto the pixel points of the image data, more accurate distance data can be obtained, which can improve the accuracy and reliability of distance measurement.

[0091] In some embodiments, a method for determining an external parameter relationship between a coordinate system corresponding to a laser radar and a coordinate system corresponding to a camera includes:

[0092] Based on the point cloud data of the calibration plate collected by the lidar and the image data of the calibration plate collected by the camera, the external parameter relationship between the coordinate system corresponding to the lidar and the coordinate system corresponding to the camera is confirmed through the camera, lidar calibration algorithm and lidar point cloud registration algorithm. The external parameter relationship includes the rotation matrix and the translation vector.

[0093] It should be understood that the calibration plate is a plate with known feature point positions, and the feature points are calibration targets arranged at known positions. The targets are usually black and white checkerboards to ensure clear identification under different lighting conditions. It should be noted that multiple calibration plates are used. Using multiple calibration plates can provide more feature points, thereby improving the accuracy of calibration. In addition, using multiple calibration plates can cover different viewing angles and distance ranges, which can better capture the imaging characteristics of the camera under different conditions and improve the global performance of calibration. The external reference relationship between the laser radar and the camera refers to their relative position and posture relationship in space. The external reference relationship can align and transform their coordinate systems so that they can perform data fusion and processing in the same coordinate system, which can achieve data complementarity between the laser radar and the camera, and improve the accuracy of target detection, ranging and positioning. It should be noted that the external reference relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera can also be estimated by using deep learning methods to learn the correspondence between point clouds and images, thereby estimating the external reference relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera.

[0094] In some embodiments, the method for determining the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera further includes:

[0095] Use optimization algorithms to optimize external parameter relationships;

[0096] The optimization algorithms include:

[0097] Determine the objective function, which is expressed as follows:

[0098]

[0099] Among them, e i represents the error corresponding to the external parameter relationship at the i-th iteration, e i =P camera -(R i *

[0100] P lidar +t i ), P camera Represents the coordinate data of the image data of the calibration plate in the coordinate system corresponding to the camera, R i represents the rotation matrix corresponding to the i-th iteration, P lidar Represents the coordinate data of the point cloud data of the calibration plate in the coordinate system corresponding to the laser radar, ti represents the translation vector corresponding to the i-th iteration, n is the total number of iterations; the total number of iterations is an empirical preset value;

[0101] The external parameter relationship is iteratively updated by the nonlinear least squares method, and the external parameter relationship corresponding to the minimization of the objective function is confirmed as the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera. Among them, the nonlinear least squares method can use the Levenberg-Marquardt algorithm.

[0102] Through the above technical solution, the external parameter relationship is optimized through the optimization algorithm, the external parameter relationship between the lidar and the camera is confirmed, the calibration accuracy is improved, the error is reduced, and the accurate correspondence between the point cloud and the image is ensured.

[0103] In some embodiments, selecting a target object area from the target scene image data includes:

[0104] Select the target object range from the target scene image data by using target recognition technology;

[0105] By presetting the proportional relationship, the target object range is narrowed down to obtain the target object area.

[0106] It should be understood that based on the target scene image data, an efficient target detection algorithm (such as YOLO, SSD, etc.) can be used to detect, identify and select the target object in the image. The target object area selection can be a rectangular frame or a polygonal frame; according to the preset proportional relationship, the bounding box of the detected target object can be scaled to reduce the scope of the target object. The proportional relationship can be set according to the size, distance and application requirements of the target object. For example, the preset proportional relationship can be 0.9, that is, the distance from each boundary line of the bounding box to the center of the target object area is reduced to a ratio of 0.9, which means that the width and height of the bounding box are reduced to 0.9 times the original, thereby reducing the scope of the target object; specifically, the width and height of the bounding box can be reduced by a certain ratio according to the size and distance of the target object, thereby reducing the scope of the target object.

[0107] Through the above technical solutions, efficient target detection algorithms (such as YOLO, SSD, etc.) can detect and identify targets in images in a shorter time. In this way, the scope of the target object can be quickly selected, improving processing speed and efficiency. By presetting the proportional relationship, the bounding box of the detected target object can be scaled, thereby reducing the scope of the target object, avoiding the influence of the surrounding environment on the target object, and improving the accuracy of the target object area. In this way, the size of the target object area can be accurately controlled according to actual needs, making it more in line with application requirements. In addition, the frame selection of the target object area can be a rectangular frame or a polygonal frame, and the rectangular frame is simple and easy to calculate. The polygonal frame can more accurately describe the shape of the target object and is suitable for targets of some special shapes. According to the application scenario and the characteristics of the target, choosing an appropriate frame selection method can improve the accuracy and adaptability of target detection.

[0108] See also Figure 3 As shown, Figure 3 A schematic flow chart of another method for acquiring image distance provided in an embodiment of the present application. In some embodiments, acquiring distance data of a target object area based on target scene point cloud data includes:

[0109] Based on the target object area, determining a target object space corresponding to the target object area in a laser radar coordinate system;

[0110] The point cloud data contained in the target object space is obtained, and then the distance data of the target object area is obtained.

[0111] It should be understood that based on the target object area, the target object space corresponding to the target object area in the laser radar coordinate system is determined; based on the bounding box of the target object area, the projection transformation of the camera can be used to establish a three-dimensional point cloud space with the camera plane detection frame as the starting point, as the target object space corresponding to the target object area in the laser radar coordinate system; then, based on the target scene point cloud data, point cloud data that satisfies the target object space is obtained, and then the distance data of the target object area is obtained. It should be noted that by analyzing the point cloud data of the target object space, the distance of each point in the target object area can be calculated. These distance data can provide the distance information between each point in the target object area and the laser radar, reflecting the depth information of the target object area, which is very useful for tasks such as distance measurement, obstacle avoidance and environmental perception of the target object.

[0112] In summary, this application realizes the rapid identification and distance measurement of target objects in the blind spot of the vehicle by mapping the laser radar point cloud information to the visual image and combining the image target recognition algorithm. This application can improve the accuracy of distance measurement. By integrating the high-precision distance information of the laser radar with the rich image data of the vision, this application can significantly improve the accuracy of the distance measurement of the target object and overcome the problem of low ranging accuracy of a single sensor. In addition, this application uses an efficient target recognition algorithm to quickly detect the target object in the image, and combines the laser radar point cloud data to obtain the target distance information in real time, greatly improving the efficiency of target detection and distance measurement. And the method of this application can maintain high-precision measurement in various complex environments, is not affected by changes in visual ambient light, and is not limited by changes in the density of the laser radar point cloud data, and has good environmental adaptability. Furthermore, this application can assist the driver in making safe decisions in operations such as changing lanes and reversing by accurately detecting and measuring the distance of target objects in the blind spot of the vehicle, and effectively reduce the occurrence of traffic accidents.

[0113] Accordingly, the embodiment of the present application further provides an image distance acquisition device, including:

[0114] The first module is configured to obtain target scene point cloud data collected by the laser radar and target scene image data collected by the camera; wherein the target scene point cloud data includes distance data of each point;

[0115] The second module is configured to frame a target object area from the target scene image data; the target object area is a rectangular frame or a polygonal frame;

[0116] The third module is configured to obtain distance data of the target object area based on the target scene point cloud data.

[0117] It should be understood that the present application provides an image distance acquisition device, which applies the image distance acquisition method in the above embodiment, and thus can have all the technical features and technical effects of the above image distance acquisition method, which will not be described in detail here.

[0118] Accordingly, the present application also provides an image distance acquisition device, comprising:

[0119] at least one processor;

[0120] at least one memory for storing at least one program;

[0121] When at least one program is executed by at least one processor, the at least one processor implements the image distance acquisition method in the above embodiment.

[0122] It should be understood that the present application provides an image distance acquisition device to implement the image distance acquisition method in the above embodiment, and thus can have all the technical features and technical effects of the above image distance acquisition method, which will not be described in detail here.

[0123] Correspondingly, the present application also provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the image distance acquisition method in the above embodiment.

[0124] It should be understood that the present application provides a computer-readable storage medium that executes the image distance acquisition method in the above embodiment, and thus can have all the technical features and technical effects of the above image distance acquisition method, which will not be repeated here.

[0125] The above is a detailed introduction to an image distance acquisition method, device, equipment and storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for acquiring image distance, characterized in that: include: Acquire target scene point cloud data collected by the laser radar and target scene image data collected by the camera; wherein the target scene point cloud data includes distance data of each point; Frame a target object region from the target scene image data; the target object region is a rectangular frame or a polygonal frame; Based on the target scene point cloud data, distance data of the target object area is acquired.

2. The image distance acquisition method according to claim 1, characterized in that: The step of acquiring distance data of the target object area based on the target scene point cloud data includes: The target scene point cloud data and the target scene image data are fused to obtain target scene fusion data; the target scene fusion data includes the target scene image data and distance data of each pixel block in the target scene image data; Obtain each pixel block corresponding to the target object area to obtain distance data of the target object area.

3. The image distance acquisition method according to claim 2, characterized in that: The step of fusing the target scene point cloud data and the target scene image data to obtain target scene fusion data includes: Projecting the target scene point cloud data onto pixel points in the target scene image data; Based on the distribution of each pixel point in the target scene image data corresponding to each point in the target scene point cloud data, dividing the target scene image data into a plurality of pixel blocks; Based on the pixel points with distance data contained in each pixel block, the distance data of each pixel block is determined to obtain the target scene fusion data.

4. The image distance acquisition method according to claim 3, characterized in that: The step of fusing the target scene point cloud data and the target scene image data to obtain target scene fusion data further includes: Eliminate outliers in the target scene point cloud data.

5. The image distance acquisition method according to claim 3, characterized in that: The projecting the target scene point cloud data onto pixel points in the target scene image data comprises: Based on the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera, the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera are converted into a unified coordinate system; Based on the intrinsic parameter matrix corresponding to the camera, the target scene point cloud data in a unified coordinate system is projected onto pixel points in the target scene image data.

6. The image distance acquisition method according to claim 5, characterized in that: The method for determining the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera includes: Based on the point cloud data of the calibration plate collected by the laser radar and the image data of the calibration plate collected by the camera, the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera is confirmed through the camera, laser radar calibration algorithm and laser radar point cloud registration algorithm, and the external parameter relationship includes a rotation matrix and a translation vector.

7. The image distance acquisition method according to claim 6, characterized in that: The method for determining the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera also includes: Optimizing the external parameter relationship using an optimization algorithm; The optimization algorithm includes: Determine the objective function, which is expressed as follows: Among them, e i represents the error corresponding to the external parameter relationship at the i-th iteration, e i =P camera -(R i *P lidar +t i ), P camera represents the coordinate data of the image data of the calibration plate in the coordinate system corresponding to the camera, R i represents the rotation matrix corresponding to the i-th iteration, P lidar represents the coordinate data of the point cloud data of the calibration plate in the coordinate system corresponding to the laser radar, t i represents the translation vector corresponding to the i-th iteration, and n is the total number of iterations; The external parameter relationship is iteratively updated by the nonlinear least squares method, and the external parameter relationship corresponding to the minimization of the objective function is confirmed as the external parameter relationship between the coordinate system corresponding to the laser radar and the coordinate system corresponding to the camera.

8. The image distance acquisition method according to claim 1, characterized in that: The step of selecting a target object area from the target scene image data includes: Selecting a target object range from the target scene image data by using target recognition technology; The target object range is narrowed by a preset proportional relationship to obtain the target object area.

9. The image distance acquisition method according to claim 1, characterized in that: The step of acquiring distance data of the target object area based on the target scene point cloud data includes: Based on the target object area, determining a target object space corresponding to the target object area in the laser radar coordinate system; The point cloud data contained in the target object space is obtained, and then the distance data of the target object area is obtained.

10. An image distance acquisition device, characterized in that: include: The first module is configured to obtain target scene point cloud data collected by the laser radar and target scene image data collected by the camera; wherein the target scene point cloud data includes distance data of each point; The second module is configured to frame a target object area from the target scene image data; the target object area is a rectangular frame or a polygonal frame; The third module is configured to obtain the distance data of the target object area based on the target scene point cloud data.

11. An image distance acquisition device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the image distance acquisition method described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program executable by a processor, and when the program executable by the processor is executed by the processor, it is used to execute the image distance acquisition method described in any one of claims 1 to 9.

Images