Obstacle detection and processing method, device, equipment and system based on binocular camera

Abstract

The present application relates to an obstacle detection and processing method, device and system based on a binocular camera. The method includes: obtaining point cloud data detected by radar and image data detected by a binocular camera; clustering the point cloud data and the image data respectively to obtain the obstacle point cloud information to be completed and the image information to be adjusted corresponding to the obstacle; optimizing the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted to obtain optimized image information; obtaining complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed; determining the obstacle contour according to the complete obstacle point cloud information. The solution provided by the present application can more accurately detect the contour of large obstacles and improve the effectiveness of path planning.

Description

Technical Field

[0001] The present application relates to the field of sensor processing technology, and in particular to an obstacle detection processing method, device, equipment and system based on a binocular camera. Background Art

[0002] In order to improve driving safety and facilitate route planning, sensors are used to detect obstacles in aircraft or automobiles. Currently, binocular cameras and laser radars can be used for roadblock detection.

[0003] In related technologies, both binocular cameras and lidar can obtain three-dimensional information of obstacles, but the depth information accuracy of binocular cameras is poor, and when lidar detects large obstacles at medium and long distances, the point cloud may not be dense enough or some points may be lost, resulting in limited detection distance.

[0004] Therefore, the data obtained by the laser radar and binocular camera in the related technologies when detecting obstacles are insufficient. Especially when encountering large obstacles, it is difficult to detect the outline of the obstacles and it is impossible to obtain accurate large obstacle information, which reduces the effectiveness of path planning. Summary of the invention

[0005] In order to solve or partially solve the problems existing in the related art, the present application provides an obstacle detection and processing method, device, equipment and system based on a binocular camera, which can more accurately detect the outline of large obstacles and improve the effectiveness of path planning.

[0006] The first aspect of the present application provides an obstacle detection and processing method based on a binocular camera, comprising:

[0007] Obtain point cloud data detected by radar and image data detected by binocular camera;

[0008] Performing clustering processing on the point cloud data and the image data respectively to obtain point cloud information of obstacles to be completed and image information to be adjusted corresponding to the obstacles;

[0009] According to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, optimizing the image information to be adjusted to obtain optimized image information;

[0010] Obtaining complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed;

[0011] The obstacle contour is determined according to the complete obstacle point cloud information.

[0012] In one embodiment, the step of optimizing the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted to obtain the optimized image information includes:

[0013] According to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, matching the obstacle point cloud information to be completed with the image information to be adjusted to obtain point cloud matching information;

[0014] Adjusting a matching portion of the image information to be adjusted according to the point cloud matching information to obtain first image information;

[0015] adjusting the unmatched portion of the image information to be adjusted with reference to the first image information to obtain second image information;

[0016] Optimized image information is obtained according to the first image information and the second image information.

[0017] In one implementation, adjusting the matching portion of the image information to be adjusted according to the point cloud matching information to obtain the first image information includes:

[0018] Determining a deviation value between the image information to be adjusted and the actual image information according to the point cloud matching information;

[0019] The matching part in the image information to be adjusted is adjusted according to the deviation value to obtain the first image information.

[0020] In one implementation, the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted is obtained according to an external parameter conversion relationship between the radar and the binocular camera.

[0021] In one implementation, the external parameter conversion relationship between the radar and the binocular camera is obtained according to a joint calibration of the radar and the binocular camera.

[0022] The second aspect of the present application provides an obstacle detection and processing device based on a binocular camera, comprising:

[0023] A data acquisition module is used to acquire point cloud data detected by the radar and image data detected by the binocular camera;

[0024] A clustering processing module, used to perform clustering processing on the point cloud data and the image data respectively, to obtain obstacle point cloud information to be completed and image information to be adjusted corresponding to the obstacle;

[0025] A data optimization module, configured to optimize the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, so as to obtain optimized image information;

[0026] A point cloud information module, which obtains complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed;

[0027] The outline determination module is used to determine the obstacle outline according to the complete obstacle point cloud information.

[0028] In one embodiment, the data optimization module includes:

[0029] a matching submodule, configured to match the obstacle point cloud information to be completed with the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, so as to obtain point cloud matching information;

[0030] A first optimization submodule, configured to adjust a matching portion of the image information to be adjusted according to the point cloud matching information to obtain first image information;

[0031] A second optimization submodule is used to adjust the unmatched part of the image information to be adjusted with reference to the first image information to obtain second image information;

[0032] The optimization fusion submodule is used to obtain optimized image information according to the first image information and the second image information.

[0033] The third aspect of the present application provides an obstacle detection and processing system based on a binocular camera, comprising:

[0034] Radar, used to send detected point cloud data to the obstacle detection and processing device;

[0035] A binocular camera, used for sending detected image data to an obstacle detection and processing device;

[0036] The obstacle detection and processing device is used to obtain point cloud data detected by a radar and image data detected by a binocular camera; perform clustering processing on the point cloud data and the image data respectively to obtain point cloud information of obstacles to be completed and image information to be adjusted corresponding to the obstacles; optimize the image information to be adjusted according to the corresponding relationship between the point cloud information of obstacles to be completed and the image information to be adjusted to obtain optimized image information; obtain complete obstacle point cloud information according to the optimized image information and the point cloud information of obstacles to be completed; and determine the obstacle contour according to the complete obstacle point cloud information.

[0037] A fourth aspect of the present application provides an electronic device, including: a processor; and

[0038] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0039] A fifth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0040] The technical solution provided by this application may have the following beneficial effects:

[0041] The solution of the present application, after clustering the point cloud data and the image data respectively to obtain the obstacle point cloud information to be completed and the image information to be adjusted corresponding to the obstacle, optimize the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted to obtain the optimized image information; and then obtain the complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed. Through the above processing, the impact of incomplete point cloud information of large obstacles can be effectively compensated, and the high resolution, long perception distance characteristics of the binocular camera and the high precision characteristics of the radar are comprehensively utilized. The advantages of the two can complement each other, and accurate complete point cloud information of large obstacles can be obtained, and the obstacle contour of large obstacles can be determined, so that the integrity of large obstacle information is guaranteed, which is conducive to designing obstacle avoidance routes according to the contours of large obstacles and improving the effectiveness of path planning.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0044] Figure 1 is a flowchart of an obstacle detection and processing method based on a binocular camera shown in an embodiment of the present application;

[0045] Figure 2 is another flow chart of an obstacle detection and processing method based on a binocular camera shown in an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of the application flow of the obstacle detection and processing method based on a binocular camera shown in an embodiment of the present application;

[0047] Figure 4 is a structural schematic diagram of an obstacle detection and processing device based on a binocular camera shown in an embodiment of the present application;

[0048] Figure 5is another structural schematic diagram of an obstacle detection and processing device based on a binocular camera shown in an embodiment of the present application;

[0049] Figure 6 is a structural schematic diagram of an obstacle detection and processing system based on a binocular camera shown in an embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0052] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0053] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0054] In the related art, when encountering a large obstacle, it is difficult to detect the outline of the obstacle and it is impossible to obtain accurate obstacle information, thereby reducing the effectiveness of path planning. In response to the above problems, the present application provides an obstacle detection and processing method based on a binocular camera, which can more accurately detect the outline of a large obstacle and improve the effectiveness of path planning.

[0055] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0056] Figure 1 It is a flowchart of an obstacle detection and processing method based on a binocular camera shown in an embodiment of the present application.

[0057] See also Figure 1 , the method comprising:

[0058] In S101, point cloud data detected by the radar and image data detected by the binocular camera are obtained.

[0059] The radar of the present application may be a laser radar. The laser radar can obtain three-dimensional point cloud information, and the binocular camera can obtain three-dimensional image information. The binocular camera has high resolution, long perception distance, and can calculate depth information, but the accuracy of the depth information at a long distance is poor. The laser radar can obtain high-quality three-dimensional information of obstacles, and its accuracy is much higher than that of the binocular camera.

[0060] In S102, clustering processing is performed on the point cloud data and the image data respectively to obtain obstacle point cloud information to be completed and image information to be adjusted corresponding to the obstacle.

[0061] For the point cloud data of the laser radar, the traditional point cloud clustering algorithm can be used to cluster the data to find obstacles. This step clusters the point cloud data to obtain the point cloud information of the obstacles to be completed. For the image data of the binocular camera, the traditional image clustering algorithm such as the Kmeans clustering algorithm can be used to cluster the image data to find obstacles. This step clusters the image data to obtain the image information to be adjusted.

[0062] In S103, according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, the image information to be adjusted is optimized to obtain optimized image information.

[0063] In step S103, the obstacle point cloud information to be completed can be matched with the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted to obtain point cloud matching information; the matching part in the image information to be adjusted can be adjusted according to the point cloud matching information to obtain first image information; the unmatched part in the image information to be adjusted can be adjusted with reference to the first image information to obtain second image information; and the optimized image information can be obtained according to the first image information and the second image information.

[0064] Among them, adjusting the matching part in the image information to be adjusted according to the point cloud matching information to obtain the first image information can include: determining the deviation value between the image information to be adjusted and the actual image information according to the point cloud matching information; adjusting the matching part in the image information to be adjusted according to the deviation value to obtain the first image information.

[0065] Among them, the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted is obtained according to the external parameter conversion relationship between the radar and the binocular camera, and the external parameter conversion relationship between the radar and the binocular camera is obtained according to the joint calibration of the radar and the binocular camera.

[0066] In S104, complete obstacle point cloud information is obtained according to the optimized image information and the obstacle point cloud information to be completed.

[0067] In this step, the optimized image information can be converted into corresponding point cloud information according to the external parameter conversion relationship between the binocular camera and the lidar. By combining the converted point cloud information with the original obstacle point cloud information to be completed, the complete obstacle point cloud information can be obtained.

[0068] In S105 , the obstacle contour is determined according to the complete obstacle point cloud information.

[0069] For obstacles at a long distance, although the point cloud initially projected on the obstacle is not uniform and dense, and the obstacle contour cannot be accurately restored, after optimization processing to obtain the complete obstacle point cloud information, these point clouds can be evenly dense, and the obstacle contour can be restored more accurately. Therefore, the obstacle contour can be determined based on the complete obstacle point cloud information for path planning and obstacle avoidance.

[0070] It can be seen from this embodiment that the solution of the present application can effectively compensate for the impact of incomplete point cloud information of large obstacles, and comprehensively utilizes the high resolution, long perception distance characteristics of the binocular camera and the high precision characteristics of the radar. The advantages of the two can complement each other, and accurate complete point cloud information of large obstacles can be obtained, and the obstacle contour of large obstacles can be determined, so that the integrity of large obstacle information is guaranteed, which is conducive to designing obstacle avoidance routes according to the obstacle contour of large obstacles and improving the effectiveness of path planning.

[0071] Figure 2 is another flowchart of an obstacle detection method based on a binocular camera shown in an embodiment of the present application. Figure 3 It is a schematic diagram of the application flow of the obstacle detection and processing method based on a binocular camera shown in an embodiment of the present application.

[0072] In the related art, the binocular camera and lidar fusion algorithm is mainly used to detect small obstacles. When performing path planning, it is only necessary to know the relative position of the obstacle and the vehicle body, and it is not necessary to obtain the outline information of the obstacle. For the application scenario of large obstacles in this application, it is necessary to obtain the outline information of the obstacle when performing path planning. This application can first jointly calibrate the lidar and binocular camera, and then let the lidar and binocular camera obtain obstacle information at the same time. Among them, the binocular camera has high resolution, long perception distance, and can calculate depth information, but its long-distance depth information accuracy is poor; the lidar can also obtain three-dimensional information of obstacles, and the accuracy is much higher than that of the binocular camera, but the detection distance is limited. The solution of this application combines the advantages of both, that is, using the long-distance perception range of the binocular camera + the high-precision three-dimensional information of the lidar, so that they can complement each other and obtain accurate obstacle information.

[0073] See also Figure 2 and Figure 3 , the method comprising:

[0074] In S201, after the laser radar and the binocular camera are jointly calibrated, point cloud data detected by the laser radar and image data detected by the binocular camera are obtained respectively.

[0075] In order to improve the robustness of the system, this application adopts a multi-sensor fusion solution. Generally speaking, fusion includes time synchronization and spatial synchronization of different sensors. The joint calibration of the laser radar and binocular camera in this application belongs to spatial synchronization.

[0076] This step jointly calibrates the laser radar and the binocular camera, and can obtain the external parameter relationship between the laser radar and the binocular camera, so that they can be converted to each other, for example, the corresponding visual information can be converted from the laser point cloud information. It should be noted that the laser radar and the binocular camera can be jointly calibrated using relevant technical methods, and this application does not limit this.

[0077] In this step, the laser radar and binocular camera are jointly calibrated so that they are executed synchronously, and the point cloud data and image data (RGB-D image) of large obstacles are collected respectively.

[0078] Among them, the external parameter relationship between the laser radar and the binocular camera can be obtained through the joint calibration process of the laser radar and the binocular camera. The external parameter conversion relationship can be as follows:

[0079]

[0080] Among them, x c ,y c is the coordinate of the corner point in the visual data, x l ,y l 、zl is the coordinate of the corner point in the point cloud. c is the parameter for normalizing visual data (image data), f x 、f y is the focal length of the binocular camera, c x 、c y is the principal point offset of the binocular camera. R is the rotation matrix determined in the multi-sensor calibration process, and t is the translation matrix determined in the multi-sensor calibration process. Both R and t are external parameters determined in the multi-sensor calibration process.

[0081] In S202, the point cloud data detected by the laser radar and the image data detected by the binocular camera are fused to obtain multi-sensor fusion information.

[0082] After the external parameter relationship is determined through joint calibration, the point cloud data detected by the lidar and the image data detected by the binocular camera can be fused to obtain multi-sensor fusion information. Information fusion can include point cloud to image fusion and image to point cloud fusion.

[0083] In S203, clustering is performed on the point cloud data to obtain point cloud information of obstacles to be completed.

[0084] For the point cloud data of the laser radar, the traditional point cloud clustering algorithm can be used to perform clustering processing to find obstacles. This step clusters the point cloud data to obtain the point cloud information of the obstacles to be completed. For example, the point cloud clustering algorithm can be used to traverse each point in the point cloud and search for its adjacent points; if the distance between them is within a certain threshold, these points can be considered to belong to the same type of obstacles. For obstacles at a close distance, the laser point cloud can be projected on it more densely, so the point cloud information after clustering processing can accurately restore the outline of the obstacle; but for obstacles at a long distance, because the resolution of the laser radar is limited, the point cloud projected on the obstacle is not uniformly dense, and the outline of the obstacle cannot be accurately restored. At this time, these clustered obstacle information can be marked as "point cloud information of obstacles to be completed."

[0085] In S204, clustering processing is performed on the image data to obtain image information to be adjusted.

[0086] Although binocular cameras have high resolution, long perception distance, and can calculate depth information, the accuracy of their depth information at long distances is poor.

[0087] For the image data of the binocular camera, traditional image clustering algorithms such as the Kmeans clustering algorithm can be used to cluster the image data to find obstacles. This step clusters the image data to obtain the image information to be adjusted. The Kmeans clustering algorithm is a very commonly used clustering algorithm. Given the number of clusters N, Kmeans will automatically find N centroids in the sample data, thereby dividing the sample data into N categories. Since the depth information accuracy of long-distance images is poor and obstacles cannot be accurately restored, these clustered image information can be marked as image information to be adjusted.

[0088] In S205, the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted is obtained according to the external parameter conversion relationship between the laser radar and the binocular camera.

[0089] Among them, the external parameter conversion relationship between the laser radar and the binocular camera can be obtained according to the aforementioned joint calibration of the laser radar and the binocular camera.

[0090] In S206, according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, the image information to be adjusted is optimized to obtain optimized image information.

[0091] In step S206, the obstacle point cloud information to be completed can be matched with the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted to obtain point cloud matching information; the matching part in the image information to be adjusted can be adjusted according to the point cloud matching information to obtain first image information; the unmatched part in the image information to be adjusted can be adjusted with reference to the first image information to obtain second image information; and the optimized image information can be obtained according to the first image information and the second image information.

[0092] Among them, adjusting the matching part in the image information to be adjusted according to the point cloud matching information to obtain the first image information can include: determining the deviation value between the image information to be adjusted and the actual image information according to the point cloud matching information; adjusting the matching part in the image information to be adjusted according to the deviation value to obtain the first image information.

[0093] As for the image information (RGB-D information) of the binocular camera, it is three-dimensional like the point cloud information, and has a wider range and can perceive more obstacle information, but its depth information needs to be improved. In the technical solution of this application, through the external parameter conversion relationship between the laser radar and the binocular camera, the correspondence between the point cloud information and the image information (RGB-D information) can be found, that is, the correspondence between the point cloud information of the obstacle to be completed and the image information to be adjusted, and then the point cloud information of the obstacle to be completed is matched with the image information to be adjusted. Because the perception range of the binocular camera is farther, only part of the RGB-D information may be able to find the corresponding point cloud information, and the accuracy of the point cloud information is more accurate. Therefore, the present application can use the more accurate point cloud information found by matching to optimize this part of the RGB-D information, for example, the deviation value between this part of the RGB-D information and the actual RGB-D information can be determined; adjust this part of the RGB-D information according to the deviation value; for the other part of the RGB-D information that cannot find the corresponding relationship with the point cloud information, the present application can use the optimized RGB-D information to infer the optimization value of the other part of the RGB-D information, that is, optimize the other part of the RGB-D information. Because the other part of the RGB-D information to be optimized has a certain geometric relationship with the optimized RGB-D information, for example, the two are close to each other, the depth information of the two is close, etc., the geometric relationship can be used to optimize the other part of the RGB-D information. If there is also semantic information to assist, the connection between the two can be further and more accurately established for optimization reference.

[0094] For example, for a plane, the laser radar can only detect and perceive half of it, while the binocular camera can perceive the whole thing, but the image data output by the binocular camera deviates from the actual image by a certain distance. According to the external parameter conversion relationship between the laser radar and the binocular camera, the deviation value between the RGB-D information and the actual RGB-D information can be determined through more accurate point cloud information, and then this part of the RGB-D information can be adjusted according to the deviation value. Even if the point cloud information and the RGB-D information are not strictly one-to-one corresponding, it is still possible to obtain more accurate and complete plane three-dimensional data.

[0095] In S207, complete obstacle point cloud information is obtained according to the optimized image information and the obstacle point cloud information to be completed.

[0096] In this step, the optimized image information can be converted into corresponding point cloud information according to the external parameter conversion relationship between the binocular camera and the lidar. By combining the converted point cloud information with the original obstacle point cloud information to be completed, the complete obstacle point cloud information can be obtained.

[0097] Because the binocular camera has a wider detection range and can perceive more obstacle information, it can detect obstacles that cannot be detected by the laser radar. After the optimized image information is converted into the corresponding point cloud information according to the external parameter conversion relationship between the binocular camera and the laser radar, the converted point cloud information includes the point cloud information of obstacles that were not detected by the laser radar before. Therefore, by combining these converted point cloud information with the original point cloud information of the obstacle to be completed, the complete obstacle point cloud information can be obtained.

[0098] In S208 , the obstacle contour is determined according to the complete obstacle point cloud information.

[0099] The complete obstacle point cloud information includes both the point cloud information of obstacles detected by the lidar itself and the point cloud information converted from image information detected by the binocular camera but not detected by the lidar itself. Therefore, the outline of large obstacles can be determined relatively completely.

[0100] At this time, for obstacles at a long distance, although the point cloud initially projected by the LiDAR on the obstacle is not uniform and dense, and the obstacle contour cannot be accurately restored, after processing to obtain the complete obstacle point cloud information, these point clouds can be evenly dense, and the obstacle contour can be restored more accurately. Therefore, the obstacle contour can be determined based on the complete obstacle point cloud information.

[0101] It should be noted that after step S208, path planning for obstacle avoidance may also be performed based on the obstacle contour.

[0102] The path planning module of a manned aircraft can generally design an obstacle avoidance route based on the obstacle contour. This application can perform obstacle avoidance path planning based on the determined obstacle contour of a large obstacle, so that the manned aircraft can accurately identify large obstacles and successfully avoid them during flight.

[0103] In summary, the present application is an optimization of the traditional multi-sensor fusion algorithm, which effectively compensates for the impact of incomplete point cloud information of large obstacles. The high resolution of the binocular camera can provide more details, perceive at a long distance, and calculate depth information; and the laser radar can also obtain high-quality three-dimensional information of obstacles, and its accuracy is much higher than that of the binocular camera; therefore, the present application can make full use of the advantages of the binocular camera and the laser radar to complement each other, obtain accurate obstacle information, ensure the integrity of the obstacle information, accurately detect the outline of obstacles such as large obstacles, design obstacle avoidance routes according to the obstacle outline, and improve the effectiveness of path planning.

[0104] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides an obstacle detection and processing device, system, electronic device and corresponding embodiments based on a binocular camera.

[0105] Figure 4 It is a structural schematic diagram of an obstacle detection and processing device based on a binocular camera shown in an embodiment of the present application.

[0106] See also Figure 4 The present application provides an obstacle detection and processing device 400 based on a binocular camera, comprising: a data acquisition module 41, a clustering processing module 42, a data optimization module 43, a point cloud information module 44, and a contour determination module 45.

[0107] The data acquisition module 41 is used to acquire the point cloud data detected by the radar and the image data detected by the binocular camera. The radar of the present application may be a laser radar. The laser radar can acquire three-dimensional point cloud information, and the binocular camera can acquire three-dimensional image information.

[0108] The clustering processing module 42 is used to perform clustering processing on the point cloud data and the image data respectively, and obtain the obstacle point cloud information to be completed and the image information to be adjusted corresponding to the obstacle. For the point cloud data of the laser radar, the clustering processing module 42 can use the traditional point cloud clustering algorithm to perform clustering processing to find obstacles. For the image data of the binocular camera, the clustering processing module 42 can use the traditional image clustering algorithm such as the Kmeans clustering algorithm to perform clustering processing on the image data to find obstacles.

[0109] The data optimization module 43 is used to optimize the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, so as to obtain optimized image information.

[0110] The point cloud information module 44 obtains the complete obstacle point cloud information based on the optimized image information and the obstacle point cloud information to be completed. The point cloud information module 44 can convert the optimized image information into corresponding point cloud information based on the external parameter conversion relationship between the binocular camera and the laser radar, and combine the converted point cloud information with the original obstacle point cloud information to be completed to obtain the complete obstacle point cloud information.

[0111] The contour determination module 45 is used to determine the obstacle contour based on the complete obstacle point cloud information. For a long-distance obstacle, although the point cloud initially projected on the obstacle is not uniform and dense, and the obstacle contour cannot be accurately restored, after the optimization process to obtain the complete obstacle point cloud information, these point clouds can be evenly dense, and the obstacle contour can be restored more accurately. Therefore, the contour determination module 45 can determine the obstacle contour based on the complete obstacle point cloud information for path planning and obstacle avoidance.

[0112] Figure 5 It is another structural schematic diagram of the obstacle detection and processing device based on the binocular camera shown in an embodiment of the present application.

[0113] See also Figure 5 The present application provides an obstacle detection and processing device 400 based on a binocular camera, comprising: a data acquisition module 41, a clustering processing module 42, a data optimization module 43, a point cloud information module 44, and a contour determination module 45.

[0114] The functions of the data acquisition module 41, the clustering processing module 42, the data optimization module 43, the point cloud information module 44, and the contour determination module 45 can be found in Figure 4 Description in .

[0115] The data optimization module 43 may include: a matching submodule 431 , a first optimization submodule 432 , a second optimization submodule 433 , and an optimization fusion submodule 434 .

[0116] The matching submodule 431 is used to match the obstacle point cloud information to be completed with the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, so as to obtain point cloud matching information;

[0117] A first optimization submodule 432 is used to adjust the matching part of the image information to be adjusted according to the point cloud matching information to obtain the first image information;

[0118] The second optimization submodule 433 is used to adjust the unmatched part of the image information to be adjusted with reference to the first image information to obtain the second image information;

[0119] The optimization fusion submodule 434 is used to obtain optimized image information according to the first image information and the second image information.

[0120] The first optimization submodule 432 may determine a deviation value between the image information to be adjusted and the actual image information according to the point cloud matching information; and adjust the matching portion in the image information to be adjusted according to the deviation value to obtain the first image information.

[0121] The corresponding relationship between the obstacle point cloud information to be completed and the image information to be adjusted is obtained according to the external parameter conversion relationship between the radar and the binocular camera. The external parameter conversion relationship between the radar and the binocular camera is obtained according to the joint calibration of the radar and the binocular camera.

[0122] It can be seen from this embodiment that the obstacle detection and processing device based on the binocular camera provided in the present application can effectively compensate for the impact of incomplete point cloud information of large obstacles, and comprehensively utilizes the high resolution, long perception distance characteristics of the binocular camera and the high precision characteristics of the radar. The advantages of the two can complement each other, and accurate complete point cloud information of large obstacles can be obtained, and the obstacle contour of large obstacles can be determined, so that the integrity of the large obstacle information is guaranteed, which is conducive to designing obstacle avoidance routes according to the contours of large obstacles and improving the effectiveness of path planning.

[0123] Figure 6 It is a structural schematic diagram of an obstacle detection and processing system based on a binocular camera shown in an embodiment of the present application.

[0124] See also Figure 6 The obstacle detection and processing system 600 based on a binocular camera provided in the present application includes: an obstacle detection and processing device 400, a radar 61, and a binocular camera 62.

[0125] Radar 61, used to send detected point cloud data to obstacle detection processing device 400;

[0126] The binocular camera 62 is used to send detected image data to the obstacle detection processing device 400;

[0127] The obstacle detection and processing device 400 is used to obtain the point cloud data detected by the radar 61 and the image data detected by the binocular camera 62; perform clustering processing on the point cloud data and the image data respectively to obtain the obstacle point cloud information to be completed and the image information to be adjusted corresponding to the obstacle; optimize the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted to obtain the optimized image information; obtain the complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed; and determine the obstacle contour according to the complete obstacle point cloud information.

[0128] The function and structure of the obstacle detection and processing device 400 can be found in Figure 4 and Figure 5 The description in will not be repeated here.

[0129] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0130] Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0131] See also Figure 7 , the electronic device 1000 includes a memory 1010 and a processor 1020 .

[0132] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0133] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0134] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0135] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0136] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0137] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An obstacle detection and processing method based on a binocular camera, characterized in that: include: Obtain point cloud data detected by radar and image data detected by binocular camera; Performing clustering processing on the point cloud data and the image data respectively to obtain point cloud information of obstacles to be completed and image information to be adjusted corresponding to the obstacles; According to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, the image information to be adjusted is optimized to obtain optimized image information, which includes: according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, the obstacle point cloud information to be completed is matched with the image information to be adjusted to obtain point cloud matching information; according to the point cloud matching information, a matching part in the image information to be adjusted is adjusted to obtain first image information; according to a geometric relationship between the matching part and the unmatched part in the image information to be adjusted in depth information, the unmatched part is adjusted with reference to the first image information to obtain second image information; and according to the first image information and the second image information, optimized optimized image information is obtained; Obtaining complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed, including: converting the optimized image information into corresponding point cloud information according to the external parameter conversion relationship between the radar and the binocular camera, wherein the converted point cloud information includes point cloud information of obstacles not detected by the radar; combining the converted point cloud information with the obstacle point cloud information to be completed to obtain complete obstacle point cloud information; The obstacle contour is determined according to the complete obstacle point cloud information.

2. The method according to claim 1, characterized in that The step of adjusting the matching portion of the image information to be adjusted according to the point cloud matching information to obtain the first image information includes: Determining a deviation value between the image information to be adjusted and the actual image information according to the point cloud matching information; The matching part in the image information to be adjusted is adjusted according to the deviation value to obtain the first image information.

3. The method according to claim 2, characterized in that: The correspondence between the obstacle point cloud information to be completed and the image information to be adjusted is obtained according to the external parameter conversion relationship between the radar and the binocular camera.

4. The method according to claim 3, characterized in that: The external parameter conversion relationship between the radar and the binocular camera is obtained according to the joint calibration of the radar and the binocular camera.

5. An obstacle detection and processing device based on a binocular camera, characterized in that: include: A data acquisition module is used to acquire point cloud data detected by the radar and image data detected by the binocular camera; A clustering processing module, used to perform clustering processing on the point cloud data and the image data respectively, to obtain obstacle point cloud information to be completed and image information to be adjusted corresponding to the obstacle; A data optimization module, configured to optimize the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, so as to obtain optimized image information; A point cloud information module, which obtains complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed, including: converting the optimized image information into corresponding point cloud information according to the external parameter conversion relationship between the radar and the binocular camera, wherein the converted point cloud information includes point cloud information of obstacles not detected by the radar; combining the converted point cloud information with the obstacle point cloud information to be completed to obtain complete obstacle point cloud information; A contour determination module, used to determine the obstacle contour according to the complete obstacle point cloud information; Wherein, the data optimization module includes: a matching submodule, configured to match the obstacle point cloud information to be completed with the image information to be adjusted according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, so as to obtain point cloud matching information; A first optimization submodule, configured to adjust a matching portion of the image information to be adjusted according to the point cloud matching information to obtain first image information; A second optimization submodule is configured to adjust the unmatched part according to the geometric relationship between the matched part and the unmatched part in the image information to be adjusted in depth information, with reference to the first image information, to obtain second image information; The optimization fusion submodule is used to obtain optimized image information according to the first image information and the second image information.

6. An obstacle detection and processing system based on a binocular camera, characterized in that: include: Radar, used to send detected point cloud data to the obstacle detection and processing device; A binocular camera, used for sending detected image data to an obstacle detection and processing device; The obstacle detection and processing device is used to obtain the point cloud data detected by the radar and the image data detected by the binocular camera; cluster the point cloud data and the image data respectively to obtain the obstacle point cloud information to be completed and the image information to be adjusted corresponding to the obstacle; According to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, the image information to be adjusted is optimized to obtain optimized image information, which includes: according to the correspondence between the obstacle point cloud information to be completed and the image information to be adjusted, the obstacle point cloud information to be completed is matched with the image information to be adjusted to obtain point cloud matching information; according to the point cloud matching information, a matching part in the image information to be adjusted is adjusted to obtain first image information; according to a geometric relationship between the matching part and the unmatched part in the image information to be adjusted in depth information, the unmatched part is adjusted with reference to the first image information part, obtaining second image information; obtaining optimized image information after optimization according to the first image information and the second image information; obtaining complete obstacle point cloud information according to the optimized image information and the obstacle point cloud information to be completed, including: according to the external parameter conversion relationship between the radar and the binocular camera, converting the optimized image information into corresponding point cloud information, wherein the converted point cloud information includes point cloud information of obstacles not detected by the radar; combining the converted point cloud information with the obstacle point cloud information to be completed to obtain complete obstacle point cloud information; determining the obstacle contour according to the complete obstacle point cloud information.

7. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 4.

Images