Target detection method, system and equipment based on laser radar and camera fusion

Abstract

The present invention discloses a target detection method, system, and device based on the fusion of laser radar and camera. The method includes: synchronizing the point cloud data of the laser radar and the image data captured by the camera frame by frame; generating a sparse depth image based on the point cloud data; determining a dense depth image based on the sparse depth image and the image data, and projecting the dense depth image into a dense point cloud in combination with the intrinsic parameter matrix of the camera; generating a preliminary recognition result of a first target based on the dense point cloud; using an offline trained road target detection model to recognize a preliminary recognition result of a second target in the image data; fusing the preliminary recognition result of the first target and the preliminary recognition result of the second target to generate a bounding box of the target in the driving scene based on the laser radar and camera; projecting the bounding box into the dense point cloud to generate the size information of the target and the position information of the target relative to the vehicle platform. The present invention improves the target detection and recognition effect.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a target detection method, system and device based on the fusion of laser radar and camera. Background Art

[0002] Target detection is a key task in the autonomous driving environment perception module. Back-end decision-making, planning, motion control and other modules all rely on the perception system to reliably and accurately detect targets around the vehicle platform. Targets in driving scenarios include large-scale targets such as vehicles and pedestrians around the vehicle during actual driving, as well as small-scale targets such as cones, vehicle debris, and typical road defects (potholes and bulges). The difficulties of target detection in the above driving scenarios are: (1) there are many types of objects involved, and the target size, color, shape, reflectivity and other characteristics are different and vary greatly; (2) environmental factors such as lighting conditions and weather conditions are constantly changing during the driving process of the vehicle platform.

[0003] LiDAR and cameras are commonly used sensors in the field of target detection. The point cloud information obtained by LiDAR can accurately provide the reflectivity, distance, and three-dimensional size information of targets in the surrounding environment and is unaffected by changes in lighting. However, point cloud data is sparse and cannot provide texture information, making it difficult to detect small targets. The image data obtained by the camera is two-dimensional and dense data, which can provide rich color, texture, and shape information of the surrounding environment. Image data is suitable for deep learning-based target detection algorithms. However, deep learning-based detection methods can only recognize target categories included in the training set. At the same time, the images captured by the camera are sensitive to changes in lighting and have difficulty coping with the complex and changing real-world driving environment. Therefore, using LiDAR or cameras alone is difficult to handle target detection tasks in real-world complex driving scenarios.

[0004] The use of a method based on the fusion of lidar and camera can comprehensively utilize the advantages of each sensor information, improve the accuracy of target recognition and target positioning. Commonly used fusion methods based on lidar and camera can be divided into pre-fusion of raw data, post-fusion of recognition results, and partial fusion according to the different fusion stages.

[0005] One common fusion framework is the front-end fusion of lidar point cloud and camera image raw data (e.g., CN115015954A). This method is relatively straightforward: the RGB image captured by the camera is cascaded with the depth map generated by projecting the lidar point cloud data onto the camera imaging plane. This cascade fusion generates an RGBD image, which is then used for object detection using a convolutional neural network. This simple and direct approach fails to account for the heterogeneous data characteristics of multimodal sensors. Furthermore, due to the different fields of view of the lidar and camera, some original information may be lost during the fusion process.

[0006] 2. Post-fusion of LiDAR and camera recognition results is a common fusion framework (e.g., CN111340797A, CN114463303A, and CN113139602A). This framework utilizes LiDAR point cloud data and camera image data for independent detection, followed by post-fusion of the detection results. Because LiDAR and cameras function as independent sensors during their respective detection processes, data exchange is not performed, and multimodal data is not fully integrated.

[0007] 3. One of the common fusion frameworks based on the partial fusion of lidar and camera, the target detection task includes two parts: target recognition and target positioning. The partial fusion method uses image data and point cloud data for target detection and target positioning respectively (such as: CN114120075A, CN113985445A, etc.). The above patent uses a target detection algorithm based on a convolutional neural network to detect the two-dimensional envelope rectangle of the target in the camera image, and uses the cone projection method to project the recognition result on the image into the lidar point cloud to locate the target. Detecting image data only through deep neural network methods can only detect the types contained in the model training data set, and cannot detect unknown types of targets. In addition, the images taken by the camera are sensitive to changes in lighting, which can easily affect the recognition effect. Summary of the Invention

[0008] The purpose of the present invention is to provide a target detection method, system and device based on the fusion of laser radar and camera to solve the problem of poor target detection and recognition effect.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A target detection method based on laser radar and camera fusion, comprising:

[0011] Jointly calibrate the LiDAR, camera, and vehicle platform, align the LiDAR and camera timestamps, and synchronize the LiDAR point cloud data and camera image data frame by frame.

[0012] generating a sparse depth image based on the point cloud data;

[0013] Determine a dense depth image based on the sparse depth image and the image data, and project the dense depth image into a dense point cloud in combination with an intrinsic parameter matrix of the camera;

[0014] Generating a first target preliminary recognition result based on the dense point cloud; the first target preliminary recognition result is a target preliminary recognition result in a driving scene based on the lidar point cloud; the targets in the driving scene include common targets and abnormal targets;

[0015] Identifying a preliminary recognition result of a second target in the image data using an offline trained road target detection model; wherein the preliminary recognition result of the second target is a preliminary recognition result of a target in a driving scene based on the camera image;

[0016] Fusing the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in a driving scene based on a lidar and a camera;

[0017] The bounding box is projected into the dense point cloud to generate size information of the object and position information of the object relative to the vehicle platform.

[0018] Optionally, generating a sparse depth image according to the point cloud data specifically includes:

[0019] Unifying the point cloud data into the camera coordinate system through coordinate transformation;

[0020] The point cloud data in the camera coordinate system is projected onto the image coordinate system of the camera to generate a sparse depth image.

[0021] Optionally, determining a dense depth image according to the sparse depth image and the image data specifically includes:

[0022] Using the image data as a guide, affinity analysis is performed on the image data, and the sparse depth image is densified through a convolutional spatial propagation network to generate a dense depth image.

[0023] Optionally, generating a preliminary recognition result of a first target according to the dense point cloud specifically includes:

[0024] Detecting the road surface area in the dense point cloud using a random sampling consistency algorithm to extract the road surface area point cloud;

[0025] Eliminate road surface points according to the point cloud of the road sign area to generate a dense point cloud after elimination;

[0026] Performing clustering processing on the dense point cloud after elimination using a density-based clustering method to generate clustered point cloud clusters;

[0027] The clustered point cloud cluster is projected into the image coordinate system of the camera, and a minimum two-dimensional envelope rectangle of the clustered point cloud cluster is calculated; the minimum two-dimensional envelope rectangle of the clustered point cloud cluster is the preliminary recognition result of the first target.

[0028] Optionally, the identifying a preliminary recognition result of the second target in the image data using an offline trained road target detection model specifically includes:

[0029] Obtain the target in the driving scenario;

[0030] Training a deep convolutional neural network based on the targets in the driving scenario to generate an offline trained road object detection model;

[0031] The offline trained road target detection model is used to identify targets in the driving scene in the image data, and a minimum two-dimensional envelope rectangle of the target in the driving scene is output; the minimum two-dimensional envelope rectangle of the target in the driving scene is the preliminary recognition result of the second target.

[0032] Optionally, the fusing of the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in a driving scene based on a lidar and a camera specifically includes:

[0033] Using the formula Generate bounding boxes for objects in driving scenarios based on lidar and cameras;

[0034] Wherein, m is the number of the first target preliminary recognition results at the current moment; n is the number of the second target preliminary recognition results at the current moment; is the preliminary recognition result of the first target i, is the preliminary recognition result of the jth second target; For and Find the intersection and comparison.

[0035] A target detection system based on laser radar and camera fusion, comprising:

[0036] A spatiotemporal synchronization module, which is used to jointly calibrate the LiDAR, camera, and vehicle platform, align the timestamps of the LiDAR and camera, and synchronize the point cloud data of the LiDAR and the image data captured by the camera frame by frame;

[0037] A sparse depth image generation module, configured to generate a sparse depth image based on the point cloud data;

[0038] a dense point cloud generation module, configured to determine a dense depth image based on the sparse depth image and the image data, and project the dense depth image into a dense point cloud in combination with an intrinsic parameter matrix of the camera;

[0039] a first target preliminary recognition result generating module, configured to generate a first target preliminary recognition result based on the dense point cloud; the first target preliminary recognition result is a target preliminary recognition result in a driving scenario based on the lidar point cloud; the targets in the driving scenario include common targets and abnormal targets;

[0040] a second target preliminary recognition result generating module, configured to use an offline trained road target detection model to identify a second target preliminary recognition result in the image data; the second target preliminary recognition result is a target preliminary recognition result in a driving scenario based on the camera image;

[0041] a fusion module, configured to fuse the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in a driving scene based on the lidar and the camera;

[0042] A size information and position information generation module is used to project the bounding box into the dense point cloud to generate size information of the target and position information of the target relative to the vehicle platform.

[0043] Optionally, the sparse depth image generation module specifically includes:

[0044] A coordinate transformation unit, configured to transform the point cloud data into a camera coordinate system;

[0045] The sparse depth image generating unit is used to project the point cloud data in the camera coordinate system onto the image coordinate system of the camera to generate a sparse depth image.

[0046] An electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-mentioned target detection method based on the fusion of laser radar and camera.

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned target detection method based on laser radar and camera fusion.

[0048] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention provides a target detection method, system and equipment based on the fusion of laser radar and camera, which comprehensively utilizes the data of laser radar and camera, adopts a detection process combining data fusion and result fusion, performs pre-fusion on the raw data of laser radar and camera, densifies the sparse point cloud information of laser radar, projects the bounding box into the dense point cloud, and extracts the size information and position information of the point cloud within the recognition box in combination with the previously obtained dense point cloud data, effectively improving the system's detection ability for small-sized targets; performs post-fusion on the preliminary detection results of laser radar and camera, enriches the data types, improves the redundancy and robustness of the system, effectively improves the detection accuracy of the system, and further improves the target detection and recognition effect. In addition, the target detection module based on laser radar and the detection module based on camera are independent of each other. When a sensor fails, it does not affect the operation of the other sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of the target detection method based on laser radar and camera fusion provided by the present invention;

[0051] Figure 2 This is a simplified flowchart of the target detection method based on laser radar and camera fusion provided by the present invention;

[0052] Figure 3 This is an overall flow chart of the target detection method based on lidar and camera fusion provided by the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The purpose of the present invention is to provide a target detection method, system and equipment based on the fusion of laser radar and camera, which improves the target detection and recognition effect.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Example 1

[0057] Figure 1 This is a flow chart of the target detection method based on laser radar and camera fusion provided by the present invention, such as Figure 1 As shown, the present invention provides a target detection method based on the fusion of laser radar and camera, comprising:

[0058] Step 101: Jointly calibrate the lidar, camera, and vehicle platform, and perform timestamp alignment on the lidar and camera, synchronizing the point cloud data of the lidar and the image data captured by the camera frame by frame.

[0059] In practical applications, a calibration rod and a chess-grid calibration plate are used for joint calibration to solve the rotation and translation matrices between the LiDAR coordinate system, the camera coordinate system, and the vehicle platform coordinate system. Timestamps are aligned between the LiDAR and camera to ensure frame-by-frame synchronization of point cloud data and image data.

[0060] Step 102: Generate a sparse depth image based on the point cloud data.

[0061] In practical applications, step 102 specifically includes: unifying the point cloud data into a camera coordinate system through coordinate transformation; and projecting the point cloud data in the camera coordinate system onto the image coordinate system of the camera to generate a sparse depth image.

[0062] In practical applications, the point cloud data obtained by the lidar is unified into the camera coordinate system through coordinate transformation, where R LC With t LC Represents the rotation matrix and translation matrix between the laser radar coordinate system and the camera coordinate system, [X l Y l Z l ] T With [X c Y c Z c ] T Represent the coordinates of the point cloud data in the lidar coordinate system and the camera coordinate system respectively.

[0063]

[0064] The point cloud data in the camera coordinate system is projected onto the camera's image coordinate system to obtain a sparse depth image.

[0065]

[0066] Where u and v represent the coordinates of the point cloud projected into the pixel coordinate system, and d represents the grayscale value corresponding to the (u, v) coordinate in the sparse depth image; f x 、f v 、c x 、c y is the internal parameter of the camera, f x 、f v Respectively represent the camera focal length in pixels in the horizontal and vertical directions on the camera imaging plane; c x 、c y Respectively represent the horizontal and vertical offset distances of the camera optical axis on the camera imaging plane in pixels; 0 3Xl is

[000] .

[0067] Step 103: Determine a dense depth image based on the sparse depth image and the image data, and project the dense depth image into a dense point cloud in combination with an intrinsic parameter matrix of the camera.

[0068] In practical applications, step 103 specifically includes: taking the sparse depth image and the RGB image (i.e., image data) taken by the camera as input, using the image data as a guide, performing affinity analysis on the image data, and densifying the sparse depth image through a convolutional spatial propagation network (CSPN) to generate a dense depth image.

[0069] Step 104: Generate a first preliminary target recognition result based on the dense point cloud. This first preliminary target recognition result is a preliminary target recognition result for a driving scenario based on the LiDAR point cloud. Targets in a driving scenario include common targets and abnormal targets. Common targets include large-sized targets such as vehicles and pedestrians around the vehicle during actual driving, while abnormal targets include small-sized targets such as cones, vehicle debris, and typical road defects.

[0070] In practical applications, step 104 specifically includes: using a random sampling consensus algorithm (RANSAC) to detect the road surface area in the dense point cloud and extract the road surface area point cloud; eliminating road surface points based on the road sign area point cloud to generate a dense point cloud after elimination; using a density-based spatial clustering method (DBSCAN) to cluster the dense point cloud after elimination to generate clustered point cloud clusters; projecting the clustered point cloud clusters to the image coordinate system of the camera and calculating the minimum two-dimensional envelope rectangle of the clustered point cloud clusters; the minimum two-dimensional envelope rectangle of the clustered point cloud clusters is the preliminary recognition result of the first target.

[0071] Step 105: using the offline trained road target detection model to identify a second target preliminary recognition result in the image data; the second target preliminary recognition result is a target preliminary recognition result in a driving scene based on the camera image.

[0072] In practical applications, step 105 specifically includes: acquiring a target in a driving scene; training a deep convolutional neural network based on the target in the driving scene to generate an offline-trained road target detection model; using the offline-trained road target detection model to identify the target in the driving scene in the image data, and outputting the minimum two-dimensional envelope rectangle of the target in the driving scene; the minimum two-dimensional envelope rectangle of the target in the driving scene is the preliminary recognition result of the second target.

[0073] Step 106: Fusing the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in the driving scene based on the lidar and camera.

[0074] In practical applications, the step 106 specifically includes: using the formula Generate a bounding box of a target in a driving scene based on a lidar and a camera; where m is the number of preliminary recognition results of the first target at the current moment; n is the number of preliminary recognition results of the second target at the current moment; is the preliminary recognition result of the first target i, is the preliminary recognition result of the jth second target; For and Calculate the intersection and union ratio; Threshold is the threshold.

[0075] In practical applications, the preliminary target recognition results in driving scenarios based on lidar point clouds (i.e., the first target preliminary recognition results) and the preliminary target recognition results in driving scenarios based on camera images (i.e., the second target preliminary recognition results) are post-fused through threshold removal to remove redundant recognition results and output the bounding box of the target in driving scenarios based on lidar and camera.

[0076] Step 107: Project the bounding box into the dense point cloud to generate size information of the target and position information of the target relative to the vehicle platform.

[0077] Figure 2 This is a simplified flow chart of the target detection method based on laser radar and camera fusion provided by the present invention, as shown in Figure 2As shown, the present invention addresses the problem of targets in driving scenarios having large size variations, irregular shapes, and no specific features. It adopts a target detection method based on the fusion of LiDAR and cameras, performing multi-stage fusion of the LiDAR and camera through methods such as raw data fusion and post-fusion of detection results. First, the sparse point cloud generated by the LiDAR is densified using images captured by the camera, achieving data-level fusion of multimodal sensor information and improving the LiDAR's detection capability for small road targets. Then, point cloud clustering is used to enable LiDAR detection of road targets. Secondly, a deep convolutional neural network algorithm, trained offline, is used to enable camera detection of targets in the vehicle platform's surrounding environment. This approach combines traditional and deep learning methods to improve system robustness. The LiDAR-based target detection module and the camera-based detection module are independent of each other, ensuring that a sensor failure does not affect the operation of the other sensor. Finally, a threshold removal method is used to remove redundant detection results, achieving post-fusion of target detection results in driving scenarios. Combined with the previously generated dense point cloud data, the size and position information of the point cloud within the recognition frame is extracted.

[0078] Example 2

[0079] Based on Example 1, the target detection method based on lidar and camera fusion provided by the present invention mainly consists of five parts: multimodal sensor joint calibration, multimodal data fusion, lidar point cloud-based detection, camera image-based detection, and multimodal detection result post-fusion. Figure 3 This is the overall flow chart of the target detection method based on laser radar and camera fusion provided by the present invention, as shown in Figure 3 shown.

[0080] The multimodal sensor joint calibration part includes the joint calibration of lidar, camera and vehicle platform, as well as the time synchronization of lidar and camera.

[0081] The densification of LiDAR point clouds involves point cloud projection and completion. The LiDAR point cloud is projected onto the camera's imaging plane to generate a sparse depth image. Image affinity analysis guides densification of the sparse depth image, and the camera's intrinsic parameter matrix is ​​used to project the dense depth map into a dense point cloud.

[0082] The LiDAR point cloud detection process includes road surface filtering, point cloud clustering, and recognition box calculation. The filtering algorithm detects road surface areas, removes ground points based on the detected road surface area, and clusters the point cloud in the field of view. The clustered point cloud cluster data points are projected onto the camera's imaging plane, and the minimum enveloping rectangle of the point cloud cluster in the pixel coordinate system is calculated as the initial target recognition result for the LiDAR point cloud driving scenario.

[0083] The camera image-based detection component includes both offline training of a road surface object detection model and online detection of road surface objects. A dataset of objects found in driving scenarios is constructed (including large objects commonly encountered during actual driving, such as vehicles and pedestrians, as well as smaller objects such as cones, vehicle debris, and typical road surface defects (potholes and bumps)). A convolutional neural network-based object detection model is trained offline. This object detection model performs real-time detection on RGB images captured by the camera, outputting the minimum enveloping rectangle of the objects in the driving scenario in the image coordinate system as the initial recognition result for the camera-based driving scenario.

[0084] The detection part based on the lidar point cloud and the detection part based on the camera image run in parallel and independently of each other. When one sensor fails, it does not affect the target detection process of the other sensor.

[0085] The post-fusion of multimodal detection results includes the fusion of multimodal recognition results and the output of driving scene target information. This involves fusing the initial recognition results based on the LiDAR point cloud and the camera image, removing redundant recognition results, outputting the target's minimum enveloping rectangle in the image coordinate system, and projecting it onto the point cloud data. The target's size and position are then calculated.

[0086] Example 3

[0087] In order to execute the method corresponding to the above-mentioned embodiment 1 and achieve the corresponding functions and technical effects, a target detection system based on the fusion of lidar and camera is provided below.

[0088] A target detection system based on laser radar and camera fusion, comprising:

[0089] The spatiotemporal synchronization module is used to jointly calibrate the lidar, camera, and vehicle platform, align the timestamps of the lidar and camera, and synchronize the point cloud data of the lidar and the image data captured by the camera frame by frame.

[0090] A sparse depth image generation module is used to generate a sparse depth image according to the point cloud data.

[0091] In practical applications, the sparse depth image generation module specifically includes: a coordinate transformation unit, which is used to unify the point cloud data into a camera coordinate system through coordinate transformation; and a sparse depth image generation unit, which is used to project the point cloud data in the camera coordinate system onto the image coordinate system of the camera to generate a sparse depth image.

[0092] A dense point cloud generation module is used to determine a dense depth image based on the sparse depth image and the image data, and project the dense depth image into a dense point cloud in combination with an intrinsic parameter matrix of the camera.

[0093] The first target preliminary recognition result generation module is used to generate a first target preliminary recognition result based on the dense point cloud; the first target preliminary recognition result is a preliminary recognition result of the target in a driving scene based on the lidar point cloud; the targets in the driving scene include common targets and abnormal targets.

[0094] The second target preliminary recognition result generation module is used to use the offline trained road target detection model to identify the second target preliminary recognition result in the image data; the second target preliminary recognition result is the target preliminary recognition result in the driving scene based on the camera image.

[0095] A fusion module is used to fuse the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in a driving scene based on the lidar and the camera.

[0096] A size information and position information generation module is used to project the bounding box into the dense point cloud to generate size information of the target and position information of the target relative to the vehicle platform.

[0097] Example 4

[0098] An embodiment of the present invention provides an electronic device including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the target detection method based on laser radar and camera fusion provided in Example 1.

[0099] In practical applications, the above-mentioned electronic device may be a server.

[0100] In practical applications, an electronic device includes at least one processor, a memory, a bus, and a communications interface.

[0101] Wherein: the processor, the communication interface, and the memory communicate with each other via a communication bus.

[0102] Communication interface, used to communicate with other devices.

[0103] The processor is used to execute the program, and specifically can execute the method described in the above embodiment.

[0104] Specifically, the program may include program codes including computer operation instructions.

[0105] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0106] Memory is used to store programs. The memory may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk storage.

[0107] Based on the description of the above embodiments, the present invention provides a storage medium on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method described in any embodiment.

[0108] The target detection system based on the fusion of LiDAR and camera provided in the embodiments of the present application exists in various forms, including but not limited to:

[0109] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.

[0110] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access capabilities. These terminals include PDAs, MIDs, and UMPC devices, such as the iPad.

[0111] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0112] (4) Other electronic devices with data interaction functions.

[0113] Thus far, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0114] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0115] For the convenience of description, the above devices are described in terms of their functions and are divided into various units and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in the same one or more software and / or hardware. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0119] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0120] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0121] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM),

[0122] Digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices

[0123] Or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0125] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0126] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0127] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A target detection method based on laser radar and camera fusion, characterized in that: include: Jointly calibrate the LiDAR, camera, and vehicle platform, align the LiDAR and camera timestamps, and synchronize the LiDAR point cloud data and camera image data frame by frame. generating a sparse depth image based on the point cloud data; Determine a dense depth image based on the sparse depth image and the image data, and project the dense depth image into a dense point cloud in combination with an intrinsic parameter matrix of the camera; Generating a first target preliminary recognition result according to the dense point cloud; the first target preliminary recognition result is a target preliminary recognition result in a driving scenario based on the lidar point cloud; The targets in the driving scenario include common targets and abnormal targets; Identifying a preliminary recognition result of a second target in the image data using an offline trained road target detection model; The second target preliminary recognition result is a target preliminary recognition result in a driving scene based on a camera image; Fusing the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in a driving scene based on a lidar and a camera; The bounding box is projected into the dense point cloud to generate size information of the object and position information of the object relative to the vehicle platform.

2. The target detection method based on laser radar and camera fusion according to claim 1, characterized in that: Generating a sparse depth image according to the point cloud data specifically includes: Unifying the point cloud data into the camera coordinate system through coordinate transformation; The point cloud data in the camera coordinate system is projected onto the image coordinate system of the camera to generate a sparse depth image.

3. The target detection method based on laser radar and camera fusion according to claim 1, characterized in that: The determining of a dense depth image according to the sparse depth image and the image data specifically includes: Using the image data as a guide, affinity analysis is performed on the image data, and the sparse depth image is densified through a convolutional spatial propagation network to generate a dense depth image.

4. The target detection method based on laser radar and camera fusion according to claim 1, characterized in that: Generating a preliminary recognition result of the first target according to the dense point cloud specifically includes: Detecting the road surface area in the dense point cloud using a random sampling consistency algorithm to extract the road surface area point cloud; Eliminate road surface points according to the road area point cloud to generate a dense point cloud after elimination; Performing clustering processing on the dense point cloud after elimination using a density-based clustering method to generate clustered point cloud clusters; The clustered point cloud cluster is projected into the image coordinate system of the camera, and a minimum two-dimensional envelope rectangle of the clustered point cloud cluster is calculated; the minimum two-dimensional envelope rectangle of the clustered point cloud cluster is the preliminary recognition result of the first target.

5. The target detection method based on laser radar and camera fusion according to claim 1, characterized in that: The identifying of the preliminary recognition result of the second target in the image data by using the offline trained road target detection model specifically includes: Obtain the target in the driving scenario; Training a deep convolutional neural network based on the targets in the driving scenario to generate an offline trained road object detection model; The offline trained road target detection model is used to identify targets in the driving scene in the image data, and a minimum two-dimensional envelope rectangle of the target in the driving scene is output; the minimum two-dimensional envelope rectangle of the target in the driving scene is the preliminary recognition result of the second target.

6. The target detection method based on laser radar and camera fusion according to claim 1, characterized in that: The fusing of the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in the driving scene based on the lidar and the camera specifically includes: Using the formula Generate bounding boxes for objects in driving scenarios based on lidar and cameras; Wherein, m is the number of the first target preliminary recognition results at the current moment; n is the number of the second target preliminary recognition results at the current moment; is the preliminary recognition result of the first target i, is the preliminary recognition result of the jth second target; For and Find the intersection and comparison.

7. A target detection system based on laser radar and camera fusion, characterized in that: include: A spatiotemporal synchronization module, which is used to jointly calibrate the LiDAR, camera, and vehicle platform, align the timestamps of the LiDAR and camera, and synchronize the point cloud data of the LiDAR and the image data captured by the camera frame by frame; A sparse depth image generation module, configured to generate a sparse depth image based on the point cloud data; a dense point cloud generation module, configured to determine a dense depth image based on the sparse depth image and the image data, and project the dense depth image into a dense point cloud in combination with an intrinsic parameter matrix of the camera; a first target preliminary recognition result generating module, configured to generate a first target preliminary recognition result based on the dense point cloud; the first target preliminary recognition result is a target preliminary recognition result in a driving scenario based on the lidar point cloud; the targets in the driving scenario include common targets and abnormal targets; A second target preliminary recognition result generating module, configured to recognize a second target preliminary recognition result in the image data using an offline trained road target detection model; The second target preliminary recognition result is a target preliminary recognition result in a driving scene based on a camera image; a fusion module, configured to fuse the first target preliminary recognition result and the second target preliminary recognition result to generate a bounding box of the target in a driving scene based on the lidar and the camera; A size information and position information generation module is used to project the bounding box into the dense point cloud to generate size information of the target and position information of the target relative to the vehicle platform.

8. The target detection system based on laser radar and camera fusion according to claim 7, characterized in that: The sparse depth image generation module specifically includes: A coordinate transformation unit, configured to transform the point cloud data into a camera coordinate system; The sparse depth image generating unit is used to project the point cloud data in the camera coordinate system onto the image coordinate system of the camera to generate a sparse depth image.

9. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the target detection method based on laser radar and camera fusion as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the target detection method based on laser radar and camera fusion as described in any one of claims 1 to 6.

Images