Dynamic obstacle recognition method and device based on multi-source data fusion and medium

Abstract

The invention discloses a dynamic obstacle recognition method and device based on multi-source data fusion and a storage medium. The method comprises the steps that laser radar point cloud data, millimeter wave radar point cloud data and a panoramic image of a vehicle in the normal driving process are acquired; performing spatial synchronization on the laser radar point cloud data and the millimeter wave radar point cloud data to obtain target point cloud data; extracting an obstacle area from the panoramic image; fusing the obstacle area image and the target point cloud data, and outputting a grid map; and through grid map clustering, obstacles are identified, and dynamic obstacles are identified. According to the method, accurate space information is obtained, so that a comprehensive and accurate grid map is output through the obstacle image data and the point cloud data, the accuracy of the grid map is improved, and extraction of dynamic obstacles is more accurate. The integrity of dynamic obstacle detection and recognition is ensured, and the omission ratio is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle recognition, and specifically to a dynamic obstacle recognition method, device and medium based on multi-source data fusion. Background Art

[0002] Obstacle detection and recognition is one of the important research topics in the field of intelligent assisted driving and robotics applications. Generally, various types of sensors can be used for obstacle detection. Obstacle detection is not only about detecting static obstacles, but also dynamic obstacles. Existing dynamic obstacle detection has the following problems:

[0003] 1. A single data type is used for dynamic obstacle detection, and the detection result is single, resulting in incomplete obstacles, full of uncertainty and insecurity;

[0004] 2. The versatility of obstacle detection is not strong and the missed detection rate is high. Summary of the invention

[0005] In order to overcome the following problems of the existing dynamic obstacle detection in the above-mentioned background technology: a single data type is used for the detection of dynamic obstacles, and the detection result is single, resulting in incomplete detected obstacles, full of uncertainty and insecurity; the obstacle detection is not universal and has a high missed detection rate. The purpose of the present invention is to provide a dynamic obstacle recognition method based on multi-source data fusion.

[0006] In order to achieve the above purpose, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention is a dynamic obstacle recognition method based on multi-source data fusion, comprising the following steps:

[0008] Acquire the laser radar point cloud data, millimeter wave radar point cloud data and panoramic images of the vehicle during normal driving;

[0009] Spatially synchronizing the laser radar point cloud data and the millimeter wave radar point cloud data to obtain target point cloud data;

[0010] Extracting an obstacle area from the panoramic image;

[0011] Fusing the obstacle area image and the target point cloud data to output a raster map;

[0012] Obstacles are identified through grid map clustering, and dynamic obstacles are identified.

[0013] In some possible implementations, spatial synchronization is performed on the laser radar point cloud data and the millimeter wave radar point cloud data, and temporal synchronization is performed on the acquisition of the panoramic image data.

[0014] Compared with the prior art, the invention has the following beneficial effects: by fusing the laser radar point cloud data and the millimeter wave radar point cloud data, accurate spatial information is obtained; by fusing the obstacle area image and the target point cloud data, a grid map is output, so that a comprehensive and accurate grid map is output through the obstacle image data and the point cloud data, the accuracy of the grid map is improved, and the extraction of dynamic obstacles is more accurate. The completeness of dynamic obstacle detection and identification is guaranteed, and the missed detection rate is reduced.

[0015] In some possible implementations, the laser radar point cloud data and the millimeter wave radar point cloud data are spatially synchronized to obtain the target point cloud data as follows: the laser radar point cloud data and the millimeter wave radar point cloud data are unified into the same coordinate system, coordinate conversion is performed, and one-to-one corresponding target point cloud data are extracted.

[0016] In some possible implementations, extracting the obstacle area from the panoramic image data specifically includes the following steps:

[0017] Dividing the panoramic image data into a plurality of sub-image areas; wherein pixels in each sub-image area have consistent image features;

[0018] Obtaining obstacle image features, and using the obstacle image features to calculate the confidence that each sub-image area belongs to an obstacle;

[0019] Determine whether the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, and obtain a first determination result;

[0020] When the first judgment result indicates that the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, the sub-image area is determined to be an obstacle area.

[0021] In some possible implementations, the obstacle image features include obstacle position features and / or obstacle projection direction features and / or obstacle area features.

[0022] A second aspect of the present invention provides a dynamic obstacle recognition device based on laser radar and millimeter wave radar, comprising:

[0023] Data acquisition module: used to obtain laser radar point cloud data, millimeter wave radar point cloud data and panoramic images of the vehicle during normal driving;

[0024] Spatial synchronization acquisition module: used to perform spatial synchronization on the laser radar point cloud data and the millimeter wave radar point cloud data to obtain target point cloud data;

[0025] Obstacle extraction module: used to extract obstacle areas from the panoramic image;

[0026] A grid map construction module is used to fuse the obstacle area image and the target point cloud data to output a grid map;

[0027] Dynamic obstacle identification module: used to identify obstacles through grid map clustering and identify dynamic obstacles.

[0028] In some possible implementations, the laser radar point cloud data and the millimeter wave radar point cloud data are spatially synchronized to obtain the target point cloud data as follows: the laser radar point cloud data and the millimeter wave radar point cloud data are unified into the same coordinate system, coordinate conversion is performed, and one-to-one corresponding target point cloud data are extracted.

[0029] In some possible implementations, extracting the obstacle area from the panoramic image data specifically includes the following steps:

[0030] Dividing the panoramic image data into a plurality of sub-image areas; wherein pixels in each sub-image area have consistent image features;

[0031] Obtaining obstacle image features, and using the obstacle image features to calculate the confidence that each sub-image area belongs to an obstacle;

[0032] Determine whether the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, and obtain a first determination result;

[0033] When the first judgment result indicates that the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, the sub-image area is determined to be an obstacle area.

[0034] In some possible implementations, the obstacle image features include obstacle position features and / or obstacle projection direction features and / or obstacle area features.

[0035] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the dynamic obstacle identification method based on multi-source data fusion are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The following is a flowchart of the overall steps of a dynamic obstacle recognition method based on multi-source data fusion according to an embodiment of the present invention;

[0037] Figure 2 A flowchart of the steps of obstacle area extraction according to an embodiment of the present invention;

[0038] Figure 3 A flowchart of the steps of constructing a grid map according to an embodiment of the present invention;

[0039] Figure 4 The figure is a schematic diagram of the structure of a dynamic obstacle recognition device based on multi-source data fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0041] See attached Figure 1 As shown, the dynamic obstacle recognition method based on multi-source data fusion in this embodiment includes the following steps:

[0042] S1: Obtain the lidar point cloud data, millimeter wave radar point cloud data and panoramic images of the vehicle during normal driving.

[0043] Specifically, the acquisition of laser radar point cloud data, millimeter wave radar point cloud data and panoramic image data is time synchronized. Multi-threaded programming is used to achieve time synchronization, and a separate thread is assigned to each millimeter wave radar, laser radar and camera. Mutex locks are used between each thread to ensure thread synchronization. This is to ensure that the laser radar point cloud data, millimeter wave radar point cloud data and panoramic image data correspond to the same actual scene and maintain consistency and relativity.

[0044] S2: spatially synchronizing the laser radar point cloud data and the millimeter wave radar point cloud data to obtain target point cloud data.

[0045] Specifically, the laser radar point cloud data and the millimeter wave radar point cloud data are unified into the same coordinate system, coordinate conversion is performed, and one-to-one corresponding target point cloud data is extracted. This is to obtain spatial information more accurately and ensure the consistency of spatial information.

[0046] S3: Extract the obstacle area from the panoramic image. Figure 2 As shown, the specific steps include:

[0047] S31: dividing the panoramic image data into a plurality of sub-image areas; wherein pixels in each sub-image area have consistent image features;

[0048] S32: Obtain obstacle image features, and use the obstacle image features to calculate the confidence that each sub-image area belongs to an obstacle;

[0049] S33: Determine whether the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, and obtain a first determination result;

[0050] S34: When the first judgment result indicates that the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, determine that the sub-image area is an obstacle area.

[0051] The above method improves the integrity of the obstacle area detected and identified. In addition, the confidence that the segmented image area belongs to an obstacle is obtained by using the obstacle image features, without the need for scene training, and is applicable to a variety of scene types, thereby improving the detection and recognition rate, and effectively overcoming the defects of the existing technology of obstacle detection, such as low versatility and high missed detection rate.

[0052] In some embodiments, the obstacle image features include obstacle position features and / or obstacle projection direction features and / or obstacle area features.

[0053] Although the types of obstacles are arbitrary, most obstacles have some common features: the location of the area corresponding to the obstacle, the projection direction of the obstacle, the area characteristics of the obstacle, etc. Therefore, in this embodiment, the confidence that each segmented image area belongs to an obstacle is calculated using the obstacle image features.

[0054] The image features of obstacles include but are not limited to position features, projection direction features and area features. Of course, those skilled in the art will appreciate that other obstacle image features may also be used for processing, and the present invention does not limit this. Other implementations obtained by those skilled in the art without creative work all fall within the protection scope of the embodiments of the present invention.

[0055] S4: Fusing the obstacle area image and the target point cloud data to output a grid map. Figure 3 As shown, the specific steps include:

[0056] S41: Performing Gaussian filtering on the obstacle area image. Gaussian filtering can eliminate noise and retain image details, making the processed image smoother and more natural.

[0057] S42: Filter and remove target point cloud data; used to remove interfering point clouds and mis-extracted point clouds in S2.

[0058] S43: Based on the SplatNet2D-3D network, multi-task learning is performed on the obstacle area image after Gaussian filtering and the target point cloud data after filtering and elimination to obtain a grid map. The accuracy of the fused grid map is improved, making it more accurate for extracting dynamic obstacles.

[0059] S5: Identify obstacles through grid map clustering to identify dynamic obstacles. Specifically, grid map clustering is used to filter out static obstacles to obtain dynamic obstacles.

[0060] This embodiment also provides a dynamic obstacle recognition device based on multi-source data fusion, and applies the above dynamic obstacle recognition method. Figure 4 As shown, the dynamic obstacle recognition device includes

[0061] Data acquisition module: used to obtain laser radar point cloud data, millimeter wave radar point cloud data and panoramic images of the vehicle during normal driving;

[0062] Spatial synchronization acquisition module: used to perform spatial synchronization on the laser radar point cloud data and the millimeter wave radar point cloud data to obtain target point cloud data;

[0063] Obstacle extraction module: used to extract obstacle areas from the panoramic image;

[0064] A grid map construction module is used to fuse the obstacle area image and the target point cloud data to output a grid map;

[0065] Dynamic obstacle identification module: used to identify obstacles through grid map clustering and identify dynamic obstacles.

[0066] By fusing the laser radar point cloud data and the millimeter wave radar point cloud data, accurate spatial information can be obtained; by fusing the obstacle area image and the target point cloud data, a grid map is output, so that a comprehensive and accurate grid map is output through the obstacle image data and the point cloud data, the accuracy of the grid map is improved, and the extraction of dynamic obstacles is more accurate. The completeness of dynamic obstacle detection and identification is guaranteed, and the missed detection rate is reduced.

[0067] In some embodiments, the laser radar point cloud data and the millimeter wave radar point cloud data are spatially synchronized to obtain the target point cloud data as follows: the laser radar point cloud data and the millimeter wave radar point cloud data are unified into the same coordinate system, coordinate conversion is performed, and one-to-one corresponding target point cloud data are extracted.

[0068] In some implementations, extracting the obstacle area from the panoramic image data specifically includes the following steps:

[0069] Dividing the panoramic image data into a plurality of sub-image areas; wherein pixels in each sub-image area have consistent image features;

[0070] Obtaining obstacle image features, and using the obstacle image features to calculate the confidence that each sub-image area belongs to an obstacle;

[0071] Determine whether the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, and obtain a first determination result;

[0072] When the first judgment result indicates that the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, the sub-image area is determined to be an obstacle area.

[0073] In some embodiments, the obstacle image features include obstacle position features and / or obstacle projection direction features and / or obstacle area features.

[0074] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned dynamic obstacle recognition method based on multi-source data fusion are performed.

[0075] The storage medium stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, or a tablet.

[0076] The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be:

[0077] DSP (Digital Signal Processor, digital signal processor, digital signal processor is a processor composed of large-scale or ultra-large-scale integrated circuit chips to complete certain signal processing tasks. It has gradually developed to meet the needs of high-speed real-time signal processing tasks. With the development of integrated circuit technology and digital signal processing algorithms, the implementation methods of digital signal processors are constantly changing, and the processing functions are constantly improving and expanding.)

[0078] ASIC (Application Specific Integrated Circuit) is an integrated circuit designed and manufactured to meet the needs of specific users and specific electronic systems.

[0079] FPGA (Field Programmable Gate Array) is a product further developed on the basis of programmable devices such as PAL (Programmable Array Logic) and GAL (generic array logic). It emerged as a semi-custom circuit in the field of application-specific integrated circuits (ASICs), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices.

[0080] The general-purpose processor may be a microprocessor or any conventional processor. The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to allow people familiar with the technology to understand the content of the present invention and implement it, and cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. Dynamic obstacle recognition method based on multi-source data fusion, It is characterized in that The steps include: Acquire the laser radar point cloud data, millimeter wave radar point cloud data and panoramic images of the vehicle during normal driving; Spatially synchronizing the laser radar point cloud data and the millimeter wave radar point cloud data to obtain target point cloud data; Extracting an obstacle area from the panoramic image; Fusing the obstacle area image and the target point cloud data to output a raster map; Obstacles are identified through grid map clustering, and dynamic obstacles are identified.

2. The dynamic obstacle recognition method based on multi-source data fusion according to claim 1, Features: Time synchronization is performed on the acquisition of laser radar point cloud data, millimeter wave radar point cloud data and panoramic image data. Multi-threaded programming is used to achieve time synchronization, and a separate thread is assigned to each millimeter wave radar and laser radar. Mutex locks are used between threads to ensure thread synchronization.

3. The dynamic obstacle recognition method based on multi-source data fusion according to claim 1, Features: The laser radar point cloud data and the millimeter wave radar point cloud data are spatially synchronized to obtain the target point cloud data as follows: the laser radar point cloud data and the millimeter wave radar point cloud data are unified into the same coordinate system, coordinate conversion is performed, and one-to-one corresponding target point cloud data are extracted.

4. The dynamic obstacle recognition method based on multi-source data fusion according to claim 1, Features: Extracting the obstacle area from the panoramic image data specifically includes the following steps: Dividing the panoramic image data into a plurality of sub-image areas; wherein pixels in each sub-image area have consistent image features; Obtaining obstacle image features, and using the obstacle image features to calculate the confidence that each sub-image area belongs to an obstacle; Determine whether the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, and obtain a first determination result; When the first judgment result indicates that the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, the sub-image area is determined to be an obstacle area.

5. The dynamic obstacle recognition method based on multi-source data fusion according to claim 1, Features: The obstacle image features include obstacle position features and / or obstacle projection direction features and / or obstacle area features.

6. Dynamic obstacle recognition device based on multi-source data fusion, It is characterized in that include Data acquisition module: used to obtain laser radar point cloud data, millimeter wave radar point cloud data and panoramic images of the vehicle during normal driving; Spatial synchronization acquisition module: used to perform spatial synchronization on the laser radar point cloud data and the millimeter wave radar point cloud data to obtain target point cloud data; Obstacle extraction module: used to extract obstacle areas from the panoramic image; A grid map construction module: used to fuse the obstacle area image and the target point cloud data to output a grid map; Dynamic obstacle identification module: used to identify obstacles through grid map clustering, and identify dynamic obstacles.

7. The dynamic obstacle recognition device based on multi-source data fusion according to claim 5, Features: The laser radar point cloud data and the millimeter wave radar point cloud data are spatially synchronized to obtain the target point cloud data as follows: the laser radar point cloud data and the millimeter wave radar point cloud data are unified into the same coordinate system, coordinate conversion is performed, and one-to-one corresponding target point cloud data are extracted.

8. The dynamic obstacle recognition device based on multi-source data fusion according to claim 5, Features: Extracting the obstacle area from the panoramic image data specifically includes the following steps: Dividing the panoramic image data into a plurality of sub-image areas; wherein pixels in each sub-image area have consistent image features; Obtaining obstacle image features, and using the obstacle image features to calculate the confidence that each sub-image area belongs to an obstacle; Determine whether the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, and obtain a first determination result; When the first judgment result indicates that the confidence level that the sub-image area belongs to an obstacle is greater than a first set confidence level, the sub-image area is determined to be an obstacle area.

9. The dynamic obstacle recognition device based on multi-source data fusion according to claim 8, Features: The obstacle image features include obstacle position features and / or obstacle projection direction features and / or obstacle area features.

10. A computer-readable storage medium, Features: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the dynamic obstacle identification method based on multi-source data fusion according to any one of claims 1 to 5 are implemented.

Images