Target detection and early warning method, device, system and medium based on data fusion

Abstract

The application provides a target detection and early warning method, device, system and medium based on data fusion. The method comprises the following steps: acquiring point cloud data and infrared images of a target detection object synchronously, fusing the point cloud data and the infrared images of the target object, obtaining point cloud image fusion data, and then using a three-dimensional neural network model to detect the point cloud image fusion data to obtain type information and three-dimensional information of the target detection object. The target detection method provided by the application can realize accurate three-dimensional detection of the target detection object in various application scenarios, has relatively low dependence on the sensing performance of an infrared acquisition device and a laser radar, has a certain error correction capability for target mis-detection and missed detection, has relatively small data processing amount, and has low demand for the computing power of a target detection device.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a target detection and early warning method, device, system and medium based on data fusion. Background Technology

[0002] With the rapid development of the autonomous driving industry, the demand for related technologies is becoming increasingly urgent. In autonomous vehicle systems, it is necessary not only to identify the categories of targets and obstacles, but also to identify their position, size, orientation, and distance information, providing crucial assurance for safe and reliable autonomous driving. Ordinary 2D (Two-Dimensional) object detection cannot provide all the information needed for perception; it only provides the position of the target object in a two-dimensional image and the confidence level of its corresponding category. In contrast, 3D (Three-Dimensional) object detection combines the depth information of the target object, providing spatial scene information such as its position, size, and orientation. As a crucial task in the autonomous driving perception subsystem, the reliability of 3D object detection results provides vital assurance for the backend decision-making and planning layers of autonomous driving. 3D object detection aims to enable autonomous vehicles to detect targets such as vehicles, pedestrians, and obstacles through multi-sensor data such as LiDAR, cameras, and millimeter-wave radar, ensuring safe autonomous driving.

[0003] Currently, 3D target detection technology is experiencing rapid development. Based on different sensors and input information, it is mainly divided into three categories: point cloud 3D target detection using LiDAR; monocular or stereo image 3D target detection using visible light cameras; and multimodal fusion 3D detection using LiDAR and visible light images. Monocular or stereo image 3D target detection using cameras is relatively inexpensive, but monocular cameras cannot provide accurate information, and monocular ranging methods based on similar triangles are easily affected by the size of the target object. Binocular 3D target detection can utilize binocular disparity maps, but it is easily affected by the environment, and the distance obtained by binocular vision also has some error compared to LiDAR. Point cloud 3D target detection using LiDAR, while providing accurate distance perception and 3D information, suffers from the problems of sparse point clouds and lack of color information, making it prone to false positives and false negatives for small targets at long distances. Multimodal fusion 3D detection using LiDAR and images can effectively utilize the 3D perception capabilities of LiDAR and the rich semantic information of visible light. However, visible light cameras and LiDAR sensors have weak anti-interference capabilities and poor adaptability to adverse weather conditions such as smoke, snow, rain, fog, and haze, making it difficult to meet the safety and reliability requirements of autonomous driving perception systems. Therefore, research on multimodal fusion 3D target detection technology is particularly important for the safety and reliability performance of autonomous driving perception.

[0004] Currently, multimodal fusion 3D object detection mainly falls into two categories: First, soft correlation fusion, which uses attention to integrate image and point cloud representations and learn the relationships between features from multiple sensors. However, it requires a large amount of data, needing hundreds of millions of data points to achieve good results. Second, hard correlation fusion detection based on calibration extrinsic parameters. Calibration extrinsic parameters can be obtained through multimodal data calibration using specific calibration boards or target-free methods. Furthermore, hard correlation fusion is mainly divided into data-layer and decision-layer fusion. Decision-layer fusion, also known as target-level fusion, fuses the prediction results from both visual and point cloud modalities. However, it heavily relies on the perception performance of each sensor and has low error correction capabilities for false detections and missed detections. Data-layer fusion can retain richer original data, but it requires high temporal synchronization and spatial registration between the camera and point cloud, and involves a large amount of data processing, placing high demands on the platform. Summary of the Invention

[0005] To address the existing technical problems, this application provides a data fusion-based target detection method, intelligent assisted driving warning method, target detection device, intelligent assisted driving warning system, and computer-readable storage medium, which have low dependence on the perception performance of each sensor, high error correction capability for false detection and missed detection of targets, and relatively small data processing volume.

[0006] A data fusion-based target detection method includes:

[0007] Acquire synchronized point cloud data and infrared images of the target object;

[0008] The point cloud data and the infrared image are fused to obtain point cloud image fusion data;

[0009] Target detection is performed on the point cloud image fusion data using a three-dimensional neural network model to obtain the category information and three-dimensional information of the target object.

[0010] The three-dimensional information includes at least one of the following: the size information, position information, distance information, and direction of motion information of the target object being detected.

[0011] A method for intelligent assisted driving warning, comprising:

[0012] The system acquires infrared images of the target objects in the target driving scene collected by an infrared acquisition device and simultaneously acquires point cloud data of the target objects in the target driving scene collected by a lidar. Both the infrared acquisition device and the lidar are mounted on the vehicle body.

[0013] Based on the point cloud data and the infrared image, target detection is performed using the target detection method described above to obtain the three-dimensional information of the target object.

[0014] Based on the three-dimensional information, it is determined whether there is a collision risk between the driving vehicle and the target detection object, and a corresponding prompt is given.

[0015] A target detection device includes a memory and a processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed by the processor, it implements the target detection method as described above.

[0016] An intelligent assisted driving warning system includes an infrared acquisition device, a lidar, a processor, and an alarm device;

[0017] The infrared acquisition device is used to acquire infrared images of the target object being detected.

[0018] The lidar is used to collect point cloud data of the target object;

[0019] The processor implements the intelligent assisted driving warning method as described above when executing a computer program;

[0020] The alarm device is used to issue an alarm based on the prompt information generated by the processor.

[0021] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the target detection method or the intelligent assisted driving warning method as described above.

[0022] As can be seen from the above, this application acquires synchronized point cloud data and infrared images of the target object, fuses the point cloud data and infrared images of the target object to obtain point cloud image fusion data, and then uses a three-dimensional neural network model to perform target detection on the point cloud image fusion data to obtain the type information and three-dimensional information of the target object. The target detection method provided by this application addresses the problems of lack of depth information in infrared images and low resolution of point cloud data by fusing and complementing point cloud data and infrared images. It can achieve accurate three-dimensional detection of target objects in various application scenarios. Furthermore, the target detection method provided by this application performs three-dimensional detection of target objects based on point cloud image fusion data, which has relatively low dependence on the perception performance of infrared acquisition equipment and lidar, and has a certain degree of error correction capability for false detection and missed detection, as well as relatively small data processing volume and low computational requirements for target detection equipment. Attached Figure Description

[0023] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0024] Figure 1This is a schematic diagram of the target detection method based on data fusion provided in the embodiments of this application;

[0025] Figure 2 This is an infrared image showing the result of detecting a target object in a nighttime application scenario using the target detection method provided in the embodiments of this application;

[0026] Figure 3 An infrared image showing the result of the target detection method provided in the embodiments of this application for detecting a target object in a dimly lit, cloudy day application scenario;

[0027] Figure 4 An infrared image showing the result of the target detection method provided in the embodiments of this application for detecting target objects in a haze application scenario;

[0028] Figure 5 This refers to point cloud data maps that are labeled with category information and 3D information of target detection objects in some application scenarios;

[0029] Figure 6 This refers to point cloud data maps that are labeled with category information and 3D information of target detection objects in some application scenarios;

[0030] Figure 7 This is a schematic diagram illustrating the process of fusing point cloud data with infrared image data in the target detection method based on data fusion provided in the embodiments of this application;

[0031] Figure 8 This is a schematic diagram illustrating the process of obtaining point cloud image fusion data in the target detection method based on the data fusion provided in the embodiments of this application;

[0032] Figure 9 This is a schematic diagram illustrating the process of feature extraction and recognition of point cloud image fusion data in the target detection method based on data fusion provided in the embodiments of this application;

[0033] Figure 10 This is a schematic diagram of the data-based target detection method provided in the embodiments of this application;

[0034] Figure 11 This is a schematic diagram of the process for obtaining the coordinate mapping relationship between point clouds and pixels in the target detection method based on data fusion provided in the embodiments of this application;

[0035] Figure 12 This is a flowchart illustrating the process of obtaining a training sample dataset for a three-dimensional neural network model in the target detection method provided in the embodiments of this application.

[0036] Figure 13 This is a schematic diagram illustrating the process of training a two-dimensional neural network model in the target detection method provided according to the embodiments of this application;

[0037] Figure 14 This is a schematic diagram of the intelligent assisted driving warning method provided according to the embodiments of this application;

[0038] Figure 15 This is a schematic diagram of the target detection device provided according to the embodiments of this application;

[0039] Figure 16 This is a schematic diagram of the structure of an intelligent assisted driving warning system provided according to an embodiment of this application;

[0040] Figure 17 This is a schematic diagram of the workflow of the intelligent assisted driving warning system provided in the embodiments of this application. Detailed Implementation

[0041] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the ways in which this application may be implemented. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0043] In the following description, the expression “some embodiments” is used, which describes a subset of possible embodiments. However, it should be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0044] This application addresses the problems of existing multimodal fusion 3D target detection methods that employ data-layer fusion or decision-layer fusion, including high requirements for sensor performance, low error correction capabilities for false and missed detections, and high platform computing power demands. It also addresses the poor adaptability of visible light image acquisition devices in scenarios such as smoke, rain, fog, and haze, making it difficult to meet the safety and reliability requirements of autonomous driving perception systems. The invention provides a multi-sensor fusion target detection method, early warning method, and corresponding equipment, systems, and media based on lidar and infrared acquisition devices. The following will combine... Figures 1 to 17 The target detection method, early warning method, target detection equipment, early warning system, and computer-readable storage medium provided in the embodiments of this application will be further described in detail.

[0045] Please see Figure 1The diagram shown is a flowchart illustrating a multi-sensor fusion target detection method provided according to an embodiment of this application. The target detection method provided in this embodiment is applied to... Figure 15 In the target detection equipment shown or such Figure 16 The intelligent assisted driving warning system shown in this embodiment includes S01, S03, and S05, which are described in detail below. If the target detection method provided in this embodiment is applied to, for example... Figure 15 When the target detection device shown is used, the memory 212 stores computer program instructions corresponding to each execution step, and the processor 211 executes the computer program instructions to implement S01, S03, and S05. For example, the target detection method provided in this embodiment is applied to, for example... Figure 16 In the intelligent assisted driving warning system shown, infrared images of the target object are acquired by infrared acquisition device 1, and point clouds of the target object are acquired by lidar 2. The infrared images and point cloud data are then sent to processor 3, which acquires the infrared images and point cloud data and processes them accordingly.

[0046] S01: Acquire the point cloud data and infrared image of the target detection object in sync.

[0047] The target detection object refers to the object that needs to be detected. Taking autonomous driving application scenarios as an example, the target detection method provided in this application embodiment refers to detecting vehicles, pedestrians, and obstacles on the driving road where the driving body is located, which may affect the driving safety of the driving body. The driving body refers to the main body carrying the target detection device or intelligent assisted driving warning system that applies the target detection method provided in this application embodiment, such as a car or other means of transportation. In some application scenarios, the target detection object includes multiple target detection objects of different scales. Furthermore, the target detection object also includes small target detection objects with a scale smaller than a threshold scale and / or partially occluded target detection objects.

[0048] Point cloud data refers to a set of vectors in a three-dimensional coordinate system; that is, point cloud data is a geometry composed of point cloud feature data from multiple point clouds. Each point cloud is distributed within the three-dimensional coordinate system corresponding to the point cloud data, and the point cloud feature data includes the coordinate data of the point cloud in that three-dimensional coordinate system. Point cloud data contains depth information of the target object, i.e., it includes the distance information of the target object. In the target detection method provided according to the embodiments of this application, the point cloud data is composed of… Figure 16 The lidar 2 shown in the image collects three-dimensional data containing distance information of the target detection object.

[0049] An infrared image is an image containing the infrared information of a target object, acquired by an infrared acquisition device. This acquired infrared information is then transformed into grayscale, resulting in an image containing the target object. Therefore, an infrared image contains the contour information of the target object. Infrared acquisition devices can be monocular or binocular. Compared to visible light image acquisition devices, infrared acquisition devices have stronger anti-interference capabilities and are more adaptable to adverse weather conditions such as smoke, snow, rain, fog, and haze. They can still obtain the contour information of the target object in these harsh environments, facilitating its detection.

[0050] Synchronized point cloud data and infrared images refer to the simultaneous acquisition of data by LiDAR and infrared acquisition devices from the same acquisition perspective. Synchronous acquisition here refers to temporal synchronization, but is not limited to absolute temporal synchronization; a certain time difference in acquisition is permissible.

[0051] S03: Fusion of point cloud data and infrared image to obtain point cloud image fusion data.

[0052] The fusion in S03 can include directly fusing the corresponding pixels in the point cloud data and the infrared image. Alternatively, the fusion in S03 can involve extracting features from the point cloud data and then fusing the extracted features with the corresponding pixels in the infrared image. Furthermore, the fusion in S03 can first extract features from the corresponding pixels in the infrared image and then fuse the extracted features with the corresponding points in the point cloud data. The fusion in S03 can even involve first extracting features from both the point cloud data and the infrared image separately, and then fusing the extracted features. The fused point cloud image data contains both the point cloud data information of each point cloud and the corresponding infrared information of each point cloud.

[0053] S05: Feature extraction and recognition are performed on the point cloud image fusion data using a 3D neural network model to obtain the category information and 3D information of the target object. The 3D information includes at least one of the following: the target object's size, position, distance, and direction of motion.

[0054] A 3D neural network model is a 3D target detection model trained on fused point cloud image data carrying labeled information. During training, it learns the labeled information carried in the fused point cloud image data and detects target objects within the data based on the learned results. The input data to the 3D neural network model is essentially still point cloud data, but each point cloud carries infrared image information; that is, each point cloud carries corresponding infrared feature data. Therefore, the 3D neural network model is a neural network model that performs target detection on point cloud data carrying infrared feature data (i.e., fused point cloud image data), obtaining the category and 3D information of the target objects within the point cloud data.

[0055] The category information of the target object can refer to the attribute category of the target object, such as car, truck, pedestrian, and other objects. The three-dimensional information of the target object can be used to characterize its size. In some embodiments, the three-dimensional information of the target object includes at least one of the following: size information, position information, distance information, and direction of motion information. The size information of the target object can be characterized by a three-dimensional bounding box used to frame the target object in infrared images or point cloud data. The length, width, and height of the target object in the world coordinate system can be determined using the three-dimensional bounding box information. Specifically, the length information refers to the dimension corresponding to the first coordinate axis direction in the world coordinate system, the width information refers to the dimension corresponding to the second coordinate axis direction, and the height information refers to the dimension corresponding to the third coordinate axis direction. Furthermore, the 3D detection bounding box information of the target object also includes center point information representing the position of the target object's center point in the world coordinate system. Based on the coordinates of the center point of the 3D detection bounding box used to frame the target object in the infrared image, the position of the target object's center point in the world coordinate system can be determined; that is, the center point information in the 3D detection bounding box information can determine the target object's position information. The distance information of the target object refers to the distance between the target object and the infrared acquisition device or the lidar (the infrared acquisition device and lidar are installed in close proximity on the vehicle body). For example, the target detection method provided in the embodiments of this application is applied to... Figure 16In the intelligent assisted driving warning system shown, since both the infrared acquisition device and the lidar are installed at the front of the vehicle (such as a car, ship, or other vehicle), the distance information of the target detection object is the distance between the target detection object and the vehicle. This distance is determined based on the position of the target detection object's center point in the world coordinate system corresponding to the monocular infrared acquisition device, the size of the target detection object (e.g., length and width), and the size of the vehicle (e.g., length and width). In some embodiments of the intelligent assisted driving warning system, the distance information of the target detection object refers to the shortest distance between the target detection object and the vehicle. The motion direction information of the target detection object can be obtained from the point cloud feature data of the corresponding point cloud data. The point cloud feature data includes the yaw angle information of the target detection object, which characterizes the motion direction information of the target detection object relative to the vehicle.

[0056] As can be seen from the above, the embodiments of this application acquire synchronous point cloud data and infrared images of the target object, fuse the point cloud data and infrared images of the target object to obtain point cloud image fusion data, and then use a three-dimensional neural network model to perform target detection on the point cloud image fusion data to obtain the type information and three-dimensional information of the target object. The target detection method provided by this application addresses the problems of lack of depth information in infrared images and low resolution of point cloud data by fusing and complementing point cloud data and infrared images. It can achieve accurate three-dimensional detection of target objects in various application scenarios. Moreover, the target detection method provided by this application performs three-dimensional detection of target objects based on point cloud image fusion data, which has relatively low dependence on the perception performance of infrared acquisition equipment and lidar, and has a certain degree of error correction capability for false detection and missed detection, as well as relatively small data processing volume and low computational requirements for target detection equipment.

[0057] The target detection method provided in this application can perform 3D detection of target objects of different scales and types in various application scenarios. The category and 3D information of the target object obtained by the target detection method provided in this application can be displayed in the infrared image of the target object. After extracting and recognizing features from the point cloud image fusion data using a 3D neural network model to obtain the category and 3D information of the target object, the detection result is obtained. Based on the detection result, the category and 3D information of the target object are then marked in the infrared image. The 3D information of the target object can be represented by a 3D detection bounding box enclosing the target object.

[0058] like Figure 2As shown, this is an infrared image displaying the target detection results obtained by detecting a target object in a nighttime application scenario using the target detection method provided in this application embodiment. In this image, the category of the target object is marked, such as "Car," and the distance information of the target object is marked, such as 36.5m and 29.5m. Furthermore, the corresponding 3D detection bounding box is marked. The 3D detection bounding box also carries the motion direction information of the target object; for example, the shaded area of ​​the 3D detection bounding box in the figure represents the motion direction of the target object.

[0059] like Figure 3 The image shown is an infrared image displaying the target detection results obtained by detecting a target object in a dimly lit, overcast environment according to the target detection method provided in this application embodiment. In this infrared image, the category of the target object is marked, such as "Car" or "Cyclist," and the distance information of the target object is marked, such as 28.7m, 26.3m, 106.7m, and 74.5m. The corresponding 3D detection bounding box is also marked. The 3D detection bounding box also carries the motion direction information of the target object; for example, the shaded area of ​​the 3D detection bounding box in the image represents the motion direction of the target object. Clearly, the target detection method provided in this application embodiment can accurately detect the category and 3D information of small, distant target objects even in poorly lit environments.

[0060] like Figure 4 As shown, it is an infrared image display of the result of the target detection method provided in the embodiments of this application, used to detect target objects in a haze application scenario. Because... Figure 2 and Figure 3 The corresponding infrared images of the results have been described. Figure 4 Similarly, the content marked therein will not be described in detail here. Figure 4 As shown, the target detection method provided in the embodiments of this application can accurately detect the category information and three-dimensional information of small target objects at a distance in hazy weather application scenarios.

[0061] Furthermore, the detection results obtained by detecting the target object according to the target detection method provided in this application, which include the target object's category information and three-dimensional information, can also be displayed in the point cloud data corresponding to the target object. Specifically, after extracting and recognizing features from the point cloud image fusion data using a three-dimensional neural network model to obtain the target object's category information and three-dimensional information, the target object's category information and three-dimensional information are then labeled in the point cloud data based on the detection results. The three-dimensional information of the target object can be represented by a three-dimensional detection box that encloses the target object. Figure 5 and Figure 6 These are point cloud data maps, each labeled with the category and 3D information of the target objects, obtained under different application scenarios. The detection results of the marked target objects in each point cloud data map can be referenced from the infrared images shown above, and will not be repeated here.

[0062] Please see Figure 7 The diagram illustrates the process of fusing point cloud data with infrared image data in a target detection method based on data fusion according to an embodiment of this application. In this embodiment, S03 further includes S031, S033, and S035, which are described in detail below.

[0063] S031: The infrared image is subjected to feature extraction using a two-dimensional neural network model to obtain the infrared feature data of each pixel in the infrared image.

[0064] The two-dimensional neural network model can be simply a feature extraction network model used to extract infrared feature data from each pixel in an infrared image. In some embodiments, the two-dimensional neural network model can also be a feature extraction and recognition model, that is, the two-dimensional neural network model includes a feature extraction network for extracting infrared image features and a classification and detection network for recognizing the extracted features to identify target objects in the infrared image. Infrared feature data refers to the feature data of the infrared information corresponding to each pixel. An infrared image is an image obtained by performing corresponding grayscale transformations on the infrared information of each pixel; that is, the pixel value of each pixel in the infrared image can represent the infrared information corresponding to that pixel. Therefore, in some embodiments, each infrared feature data is the grayscale value (grayscale value) of each corresponding pixel in the infrared image.

[0065] S033: Determine the infrared feature data corresponding to each point cloud based on the coordinate mapping relationship between each point cloud in the point cloud data and each pixel in the infrared image.

[0066] The coordinate mapping relationship between point clouds in point cloud data and pixels in infrared images can be determined by transforming the coordinate systems based on an intrinsic parameter calibration matrix and an extrinsic parameter transformation matrix. The intrinsic parameter calibration matrix consists of parameters calibrated by the infrared acquisition device acquiring the infrared image, while the extrinsic parameter transformation matrix consists of parameters determined by joint calibration between the infrared acquisition device and the lidar used to acquire the point cloud data. After determining this coordinate mapping relationship, any point cloud in the point cloud data can be mapped to the pixel coordinate system of the infrared image, thus determining the corresponding pixel in that pixel coordinate system. Once the corresponding pixel is determined, its infrared feature data is the infrared feature data of the point cloud.

[0067] S035: The point cloud feature data and the infrared feature data of the corresponding pixels of each point cloud are fused to obtain the point cloud image fusion data corresponding to each point cloud.

[0068] Point cloud feature data is data containing feature information of points in point cloud data, such as the position data of the point cloud in the point cloud data, the corresponding depth information (distance information) data of the point cloud, and the corresponding yaw angle information data of the point cloud. Point cloud image fusion data refers to the data obtained by combining, correlating, and integrating the point cloud feature data of each point cloud in the point cloud data with the infrared feature data of the corresponding pixels in the infrared image. Each point cloud in the point cloud image fusion data has both corresponding point cloud feature data and corresponding infrared feature data. It should be noted that the point cloud feature data is the raw data collected by the lidar, while the infrared feature data is the feature data obtained by feature extraction of the infrared image based on the above-mentioned two-dimensional neural network model. Therefore, the fusion of point cloud data and infrared data in the target detection method provided in this application embodiment refers to the fusion between the data layer (point cloud feature data of the point cloud) and the feature layer (infrared feature data). This fusion method has low requirements for time synchronization and spatial registration of point cloud data and infrared images, and the computational requirements are not too high. It also has a certain error correction capability for false detection and false detection of targets. Furthermore, by fusing the infrared feature data of the infrared image with the corresponding point cloud feature data of the point cloud data, the resulting point cloud image fusion data contains both the depth information and the infrared information of the target object. Therefore, detecting the three-dimensional information of the target object based on the point cloud image fusion data can overcome the problem of inaccurate detection caused by the low resolution of point cloud data, and may also overcome the problem of infrared images lacking depth information and thus unable to obtain the three-dimensional information of the target object.

[0069] like Figure 7As shown, this embodiment of the application acquires synchronized point cloud data and infrared images of the target object, and uses a two-dimensional neural network model to extract features from each pixel in the infrared image to obtain infrared feature data corresponding to each pixel. Then, based on the coordinate mapping relationship between the point cloud data and the corresponding pixels in the infrared image, the infrared feature data corresponding to each point in the point cloud data is determined. Next, data fusion is performed based on the point cloud feature data and the corresponding infrared feature data to obtain fused point cloud image data. Finally, a three-dimensional neural network model is used to extract features and identify the fused point cloud image data to obtain the category information and three-dimensional information of the target object. The target detection method provided by this embodiment addresses the problems of insufficient depth information in infrared images and low resolution in point cloud data by fusing and complementing the infrared feature data of the infrared image and the point cloud feature data of the point cloud data, enabling accurate three-dimensional detection of target objects in various application scenarios. Furthermore, by adopting a fusion approach of data layer and feature layer, the dependence on the perception performance of infrared acquisition equipment and lidar is relatively low. It also has a certain ability to correct false detection and false detection of targets, and has a relatively small amount of data processing, so the computing power requirement of target detection equipment is not high.

[0070] In some embodiments, S01 further includes S011, S013 and S015, and the specific description of each step is as follows.

[0071] S011: Acquire the infrared image of the target object.

[0072] S013: Acquire the original point cloud data of the target object within a preset time difference range from the time difference of infrared image acquisition.

[0073] S015: Filter the raw point cloud data according to the field of view of the infrared image to obtain point cloud data synchronized with the infrared image.

[0074] Specifically, infrared images of the target object are acquired from an infrared acquisition device, which can be an infrared camera. The frame rate of an infrared camera acquiring infrared images is typically 50Hz. In addition to acquiring the infrared image containing the target object from the infrared camera, the frame rate of the infrared image also needs to be obtained. The raw point cloud data is the data output by a lidar system installed at the corresponding location and sharing a common field of view with the infrared camera, which acquires depth information of the target object. The frame rate of the lidar acquiring the raw point cloud data is typically 10Hz. In addition to acquiring the raw point cloud data acquired by the lidar, the frame rate of the lidar also needs to be obtained. Then, based on the frame rates of the infrared camera and lidar, the raw point cloud data of the target object is acquired within a preset time difference range from the infrared image acquisition time. For example, a soft synchronization method can be used to acquire data where the time difference between the lidar and infrared camera acquisition is less than 10ms, which is considered the same frame data, i.e., synchronized raw point cloud data and infrared image. Furthermore, for the lidar and infrared camera mounted on the vehicle body, the lidar has a larger horizontal field of view than the infrared camera, and there are some areas that are not of interest. Therefore, after acquiring the raw point cloud data synchronized with the infrared image, it is necessary to perform data filtering to select the regions of interest that belong to the field of view of the infrared camera from the raw point cloud data, which will be used as the point cloud data in S01 for subsequent processing. By filtering the raw point cloud data, only the data of the region of interest that is in the common field of view of the lidar and infrared camera is retained, which reduces the amount of data for subsequent processing and improves the real-time performance of the 3D neural network model in target detection.

[0075] Please see Figure 8 The diagram illustrates the process of fusing point cloud data and infrared feature data in a target detection method based on data fusion according to an embodiment of this application. In this embodiment, S035 specifically includes S0351, S0353, and S0355, which are described in detail below.

[0076] S0351: Based on the point cloud feature data of each point cloud, determine the neighborhood centered on each pixel corresponding to each point cloud.

[0077] S0353: The infrared feature data corresponding to each pixel in each neighborhood of the infrared image are weighted and averaged to obtain the weighted average infrared feature data corresponding to each point cloud.

[0078] S0355: The point cloud feature data, pixel feature data and weighted average pixel feature data corresponding to each point cloud are fused to obtain the point cloud image fusion data corresponding to each point cloud.

[0079] The point cloud feature data contains distance information for the corresponding target detection object. Therefore, based on the point cloud feature data, the distance between the target detection object and the driving vehicle can be determined. The greater the distance, the larger the neighborhood radius in S0351, and vice versa. The neighborhood radius refers to the distance from the edge of the neighborhood to its center point.

[0080] In the target detection method provided in this application embodiment, infrared feature data containing more semantic and texture information is fused with point cloud feature data (raw data directly output by the LiDAR) from point cloud data. The coordinate mapping relationship between the point cloud and the corresponding pixel in the infrared image is determined by the intrinsic parameter calibration matrix and extrinsic parameter transformation matrix of the infrared camera. Then, each point cloud is projected into the pixel coordinate system of the infrared image according to the coordinate mapping relationship to determine the pixel corresponding to the point cloud in the pixel coordinate system, and the infrared feature data of the corresponding pixel is added to the point cloud. Since the point cloud data is relatively sparse and the distance of the target decreases, for example, in a 128-line LiDAR, there are only 10 points for trucks, cars, and electric vehicles at 200m, 160m, and 110m, and they are basically all in the center of the object, but there may be hundreds of points at close range. Considering that acquiring only the infrared feature data of a single pixel in the infrared image for each point cloud would result in the loss of some image information, this embodiment, in addition to adding the infrared feature data of the corresponding infrared image to each point cloud, also determines the neighborhood radius of the corresponding pixel in the infrared image based on the distance represented by the point cloud. Then, the infrared feature data within this neighborhood is weighted and averaged to obtain the weighted average infrared feature data f. This weighted average infrared feature data f, along with the infrared feature data, is then assigned to the corresponding point cloud, so that the corresponding point cloud possesses both point cloud feature data and infrared feature data, as well as the weighted average infrared feature data. For example, if the point cloud is relatively dense within 10m, the weighted average infrared feature data f of the pixel infrared feature data f(i,j) within a neighborhood with a radius of 1 is taken. As the distance increases, the neighborhood radius also increases accordingly.

[0081]

[0082] In the above formula, r is the neighborhood radius, (i,j) are the pixel coordinates, and N is the number of pixels in the neighborhood. The two-dimensional neural network model used to obtain the infrared feature data is a Convolutional Neural Network (CNN) model; therefore, the infrared feature data extracted based on it can also be called CNN feature data.

[0083] In this embodiment, the point cloud feature data, infrared feature data, and weighted average infrared feature data corresponding to the point cloud data are fused. Each point cloud in the obtained point cloud image fusion data includes its own point cloud feature data, as well as the infrared feature data and weighted average infrared feature data from the infrared image. When using a three-dimensional neural network model to extract and recognize features from such point cloud image fusion data, the receptive field of distant small targets can be increased to obtain more semantic information. At the same time, the interference of infrared feature data of pixels in the neighborhood of nearby targets can be reduced, which can improve the accuracy of target detection and facilitate the accurate detection of information of distant small-scale target objects.

[0084] Please see Figure 9 The diagram illustrates the process of target detection using point cloud image fusion data in a target detection method based on data fusion provided in this application. In this embodiment, S05 specifically includes S051, S053, and S055, which are described in detail below.

[0085] S051: Voxelize the point cloud image fusion data to obtain multiple columnar voxels.

[0086] S053: Perform feature extraction and mapping on multiple columnar voxels to obtain voxel features of point cloud image fusion data, and map the voxel features to a bird's-eye view to obtain a bird's-eye view feature map corresponding to the point cloud image fusion data. The voxel features include point cloud 3D coordinate data, pixel feature data, weighted average pixel feature data, geometric center data, and geometric center offset data corresponding to the point cloud image fusion data.

[0087] S055: Input the bird's-eye view feature map into the 3D neural network model for feature extraction and recognition to obtain the category signal and 3D information of the target object.

[0088] In autonomous driving, besides ensuring the accuracy of the 3D neural network model, it is also necessary to consider the platform computing power of the target detection equipment, so that the 3D neural network model can achieve both accuracy and efficiency. Currently, the point cloud detection network widely used in industry is PointPillars, which converts point clouds into pseudo-images and then performs detection on a 2D backbone network, achieving an inference speed of 62Hz. The input vector in the original PointPillars includes 3D coordinates, reflection intensity, geometric center, and offset from the geometric center. Among them, the reflection intensity varies greatly due to the different types of LiDAR equipment produced by different manufacturers (such as MEMS and mechanical types). Inputting the reflection intensity into the detection network for training will lead to poor model generalization. Therefore, the target detection method provided in this application does not use reflection intensity information, that is, the input data to the 3D neural network model does not include reflection intensity information.

[0089] Currently, a common method for multimodal fusion detection of infrared images and point cloud data is PointPainting. This method performs semantic segmentation on the image, calculates the semantic segmentation score for each pixel, and assigns the pixel's semantic segmentation score to the point cloud using an extrinsic transformation matrix. PointPainting is simple and intuitive, and can append semantic segmentation results to any 3D point cloud target detection framework. However, it relies on data alignment results and is limited by the semantic segmentation outcome. The target detection method provided in this application determines the coordinate mapping relationship between the infrared acquisition device and the lidar sensor using the intrinsic calibration matrix and extrinsic transformation matrix of the infrared acquisition device. This means determining the coordinate mapping relationship between the corresponding pixels in the point cloud data and the infrared image. Based on this coordinate mapping relationship, each point cloud is projected onto the pixel coordinates of the infrared image to determine the corresponding pixels in each point cloud. Then, the infrared feature data and average weighted infrared feature data of each corresponding pixel are added to the point cloud to fuse its own point cloud feature data, obtaining point cloud image fusion data. Then, the point cloud corresponding to the point cloud image fusion data is voxelized to obtain corresponding columnar voxels. These voxelized columnar voxels are then converted into a Bird's Eye View (BEV). The input vector for this BEV includes the 3D coordinate data of the point cloud, infrared feature data, weighted average infrared feature data, geometric center data, and geometric center offset data. The geometric center data represents the position of the center point of each columnar voxel, and the geometric center offset data represents the distance from the point cloud to the center of the corresponding voxel. After obtaining the BEV, this feature map is input into a 3D neural network model to extract the feature information of the target detection object. Based on the extracted feature information, the center point of the target detection object is detected, and the 3D detection box information and the category information of the target detection object are output.

[0090] Please see Figure 10 The diagram shown is a flowchart of a target detection method based on data fusion provided in an embodiment of this application. The target detection method provided in this embodiment is applied to... Figure 10 In the target detection equipment shown, besides such as Figure 1 In addition to S01, S03 and S05 shown in the figure, the target detection method in this embodiment further includes S02, which is executed before S03, and is described in detail below.

[0091] S02: Perform detail enhancement processing on the infrared image.

[0092] For vehicle-mounted infrared images, since the thermal images obtained solely from the surface temperature of objects lack rich color information, registration and fusion detection are more challenging. During detection, image processing algorithms such as detail enhancement and contrast adjustment are applied to the infrared images to obtain infrared images with richer details and more prominent targets.

[0093] In the target detection method provided in this embodiment, before inputting the infrared image into the two-dimensional neural network model for feature extraction, the infrared image is preprocessed to enhance the information of the target object in the infrared image. Specifically, detail enhancement processing is performed on the infrared image to enhance details and adjust contrast, obtaining rich detail information of the target object and making the target object more prominent in the infrared image. After detail enhancement processing, when the infrared image is used by the two-dimensional neural network model for feature extraction, richer feature information can be extracted, thus making the point cloud image fusion data contain richer and more detailed feature information. This makes it easier for the three-dimensional neural network model to identify the information of the target object from the point cloud image fusion data, realizing all-day, all-weather, and all-scene obstacle detection, which is beneficial to enhancing the warning performance of the intelligent assisted driving warning system.

[0094] Please see Figure 11 The diagram illustrates the process of obtaining the coordinate mapping relationship between point clouds and pixels in a target detection method based on data fusion according to an embodiment of this application. The target detection method in this embodiment further includes, before S03, obtaining the coordinate mapping relationship between each point cloud in the point cloud data and each pixel in the infrared image. Specifically, obtaining the coordinate mapping relationship between each point cloud in the point cloud data and each pixel in the infrared image includes S021, S023, S025, and S027, which are described in detail below.

[0095] S021: By jointly calibrating the lidar used to collect point cloud data and the infrared acquisition device used to collect infrared images, the external parameter transformation matrix from lidar to infrared acquisition device is determined.

[0096] S023: Determine the first transformation relationship between the coordinate system of the infrared acquisition device and the point cloud coordinate system of the lidar based on the external parameter transformation matrix.

[0097] S025: Based on the intrinsic parameter calibration matrix of the infrared acquisition device, determine the second transformation relationship between the coordinate system of the infrared acquisition device and the pixel coordinate system of the infrared image.

[0098] S027: Determine the coordinate mapping relationship between each point cloud in the point cloud data and each pixel in the infrared image based on the first transformation relationship and the second transformation relationship.

[0099] LiDAR is a sensing device that can obtain three-dimensional information of a target object, including its distance and direction of motion. By jointly calibrating the infrared image acquired by the infrared acquisition device using LiDAR, the extrinsic parameter transformation matrix between the LiDAR and the infrared acquisition device can be obtained. The process of determining the extrinsic parameter transformation matrix of the infrared acquisition device based on the joint calibration of LiDAR and the infrared acquisition device is as follows.

[0100] A checkerboard calibration method can be used to jointly calibrate the lidar and infrared acquisition device. Specifically, a heated checkerboard is used as the calibration board, and the black and white squares are heated to different temperatures to enhance their contrast in the infrared image. A coating is applied to the surface of the calibration board to ensure inconsistent reflectivity of the lidar on the black and white squares. A calibration toolkit is then used to jointly calibrate the lidar and infrared acquisition device, resulting in the extrinsic parameter transformation matrix [R,T] from the lidar to the infrared acquisition device, where R is the rotation extrinsic parameter and T is the translation extrinsic parameter.

[0101] Assume that point PL(X) is in the lidar coordinate system (the coordinate system where the point cloud data is located). L ,Y L Z L The point PL(X) in the lidar coordinate system is determined by the jointly calibrated extrinsic parameter matrix [R,T]. L ,Y L Z L Transform to point P in the coordinate system of the infrared acquisition device. C (X C ,Y C Z C The first conversion formula between () is shown below:

[0102]

[0103] Where [R,T] is the extrinsic transformation matrix.

[0104] Point P in the coordinate system of the infrared acquisition device C (X C ,Y C Z C The third transformation formula for converting a point to its corresponding image coordinate system P(x,y) is as follows:

[0105]

[0106] Z C The scaling factor from the infrared acquisition device coordinate system to the corresponding image coordinate system.

[0107] The fourth transformation formula for transforming point P(x,y) in the image coordinate system to point P(u,v) in the pixel coordinate system is as follows:

[0108]

[0109] Based on the third and fourth transformation relationships mentioned above, the midpoint P of the infrared acquisition device coordinate system can be obtained. C (X C ,Y C Z C The second transformation formula corresponding to the pixel coordinate system point P(u,v) is as follows:

[0110]

[0111] Where K is the intrinsic parameter calibration matrix, f x f y Let (u0, v0) be the focal lengths of the infrared acquisition device along the x and y axes, respectively, and (u0, v0) be the coordinates of the principal point corresponding to the infrared acquisition device. Therefore, point P in the final radar coordinate system... L (X L ,Y L Z L The coordinate mapping relationship from point U to point P(u,v) in the pixel coordinate system is as follows:

[0112]

[0113] When jointly calibrating the lidar and infrared acquisition equipment, multiple pairs of calibration points in the lidar coordinate system and their corresponding calibration points in the pixel coordinate system can be obtained. These points can then be substituted into the aforementioned coordinate mapping relationship to determine the extrinsic parameter matrix [R,T]. After determining the intrinsic parameter matrix K and the extrinsic parameter matrix [R,T], any point in the lidar coordinate system can be transformed into the pixel coordinate system according to the aforementioned coordinate mapping relationship.

[0114] After obtaining the aforementioned extrinsic parameter transformation matrix, the first transformation relationship can be determined. After determining the first transformation relationship, the second transformation relationship can be determined further based on the intrinsic parameter calibration matrix. Based on the first and second transformation relationships, the coordinate mapping relationship between the infrared acquisition device and the lidar coordinate system can be determined.

[0115] In some embodiments, the 3D neural network model in S05 is a pre-trained neural network model. Therefore, prior to S05, the target detection method provided according to some embodiments of this application further includes: constructing a training sample dataset based on sample point cloud image fusion data carrying annotation information, and then using the training sample dataset to iteratively train the 3D neural network model until a preset training condition is reached. The annotation information includes the category annotation information and 3D bounding box information of the target detection object. Specifically, the 3D bounding box information includes the center point information, length, width, and height information, and yaw angle information of the 3D bounding box.

[0116] In the target detection method provided in this application embodiment, the three-dimensional neural network model is set as follows: Figure 15 The processor 211 of the target detection device shown is and / or located in, as in, Figure 16 In the processor 3 of the intelligent assisted driving warning system shown, the trained 3D neural network model performs target detection on the point cloud image fusion data to obtain the category information and 3D detection box information of the target object in the point cloud image fusion data. Figure 16 The display device 5 displays an infrared image or point cloud data map containing category information and 3D bounding box information output by the 3D neural network model. The data processing module in processor 211 and / or processor 3 further determines the distance information of the target object based on the position, length, and width of the center point of the 3D bounding box in the world coordinate system corresponding to the monocular infrared acquisition device. It also further determines the distance information of the target object relative to the driving body based on the length and width dimensions of the driving body and the installation position of the monocular infrared acquisition device on the driving body. The center point of the 3D bounding box and the corresponding center point of the target object represent their respective positions, and the length, width, and height of the 3D bounding box represent the length, width, and height of the corresponding target object, respectively.

[0117] Furthermore, in the target detection method provided according to some embodiments of this application, the annotation information carried in the training sample dataset for training the 3D neural network model also includes annotation information on the occlusion degree of the target detection object in the sample point cloud image fusion data. Annotating the occlusion degree information of the target detection object in the sample point cloud image fusion data enables the first neural network to extract the infrared feature data of the occluded target detection object, and enables the 3D neural network model to learn to recognize the occluded target detection object and learn the corresponding occlusion degree during training. Thus, when performing target detection on the infrared image, the category information, 3D dimensions, and occlusion degree information of the partially occluded target detection object can be obtained.

[0118] Please see Figure 12The diagram illustrates the process of constructing a training sample dataset for a 3D neural network model in the target detection method provided in this application. In this embodiment, before iteratively training the neural network model using the training sample dataset, the target detection method further includes constructing a training sample dataset based on fused point cloud images carrying labeled information. Specifically, constructing the training sample dataset based on fused point cloud images carrying labeled information includes the following steps: S041, S043, and S045.

[0119] S041: Label the sample point cloud data corresponding to the sample point cloud image fusion data to obtain 3D annotation box information.

[0120] S043: Based on the mapping relationship between the sample infrared image and the sample point cloud data corresponding to the sample point cloud image fusion data, project the 3D annotation box corresponding to the sample point cloud data onto the sample infrared image.

[0121] S045: When the projection area of ​​the 3D bounding box corresponding to the sample point cloud data onto the sample infrared image meets the preset conditions, the fused data of the sample point cloud image carrying the 3D bounding box information is used as a training sample to form a training sample dataset.

[0122] For point cloud data from vehicle-mounted infrared images and LiDAR, the point cloud data is labeled, and the 3D bounding box information in the infrared image is obtained through the aforementioned coordinate mapping relationship. The labeling information includes the category information of the target detection object, the center point (x, y, z), length, width, and height (l, w, h), and yaw angle (θ) of the target detection object's 3D bounding box. Because the horizontal field of view of the LiDAR is larger than that of the infrared acquisition device acquiring the infrared image, if 50% of the 3D bounding box in the point cloud data is projected onto the infrared image (one of the preset conditions) during the process of converting the 3D bounding box in the point cloud data into a 3D bounding box in the infrared image, the labeling of that target detection object is discarded.

[0123] Please see Figure 13 This is a schematic diagram illustrating the process of training a two-dimensional neural network model in the target detection method provided in this application embodiment. In this embodiment, the two-dimensional neural network model in S01 is the trained two-dimensional target detection neural network model. Before executing S01, the target detection method provided in this application embodiment further includes training the two-dimensional neural network model, and the training process includes S001 and S002, which are described in detail below.

[0124] S001: Determine the two-dimensional bounding box information corresponding to the target detection object in the sample infrared image based on the three-dimensional bounding box information. The two-dimensional bounding box information includes the center point information and width and height information of the two-dimensional bounding box.

[0125] S002: Train a two-dimensional neural network model using sample infrared images carrying two-dimensional bounding box information.

[0126] PointPillars uses pre-set anchors for prediction, where the anchors include target size, center, and orientation information. First, because the installation height and field of view of LiDAR sensors vary, the pre-set anchor centers have low adaptability, resulting in poor model generalization. Second, unlike 2D boxes, 3D boxes do not follow any specific orientation, making it difficult for anchor-based detectors to enumerate all orientations or fit an axis-aligned detection box for rotating objects. Therefore, the target detection method provided in this application borrows from CenterPoint, using points to represent targets and predicting the target's center position, size, and orientation information. The PointPillars backbone network uses a top-down network to generate features with decreasing spatial resolution, and a second network performs upsampling and concatenated top-down functions. Its feature extraction network is not robust; many features are weak features obtained from shallower layers, and direct 2x or 4x upsampling easily leads to information loss, making it unsuitable for distant, small targets. Therefore, the target detection method provided in this application adds a 2D detection auxiliary network, namely a first neural network, to the CenterPoint model. During the training of the 3D neural network model, 2D bounding box information is annotated on the sample infrared images based on the 3D bounding box information of the sample point cloud image fusion data to obtain training samples for the 2D neural network model. In this embodiment, the 2D neural network model includes a feature extraction network and a target detection network. The feature extraction network is used to extract infrared feature data of each pixel in the infrared image, while the target detection network is used to detect the 2D bounding box information of the target object in the infrared image. By annotating the 2D bounding box information of the target object in the training samples of the 2D neural network model, the 2D neural network model can learn the feature information of small targets at a distance during training. Therefore, the infrared feature data extracted by the feature extraction network model of the trained 2D neural network model can contain richer feature information of small targets at a distance; that is, the point cloud image fusion data contains feature information of small targets at a distance, which is beneficial for the 3D neural network model to identify the 3D information of small targets at a distance from the point cloud image fusion data. Therefore, this application utilizes a two-dimensional neural network model to assist a three-dimensional neural network model in detecting small target objects at a distance, reducing false detections and missed detections of distant objects. Furthermore, the detection of small target objects by the three-dimensional neural network model is only applied during the training phase and does not increase additional inference costs.Based on CenterPoint, the following two operations were performed on the structure of the two-dimensional neural network model: feature pyramid concatenation was performed on the backbone network, and the concatenated features were upsampled and spliced. During the training of the two-dimensional neural network model, the prediction of the center point and width and height of the 2D target was performed to guide the feature extraction network of the two-dimensional neural network model to extract and learn the feature information of distant small targets, thereby reducing false detections and false negatives of distant small targets.

[0127] In the target detection method provided according to some embodiments of this application, the step of obtaining infrared feature data in S01 further includes: extracting features from the infrared image by using the feature extraction network in the two-dimensional neural network model trained based on the training steps provided in the above embodiments, and obtaining infrared feature data of each pixel in the infrared image.

[0128] Please see Figure 14 As shown, it is a schematic flowchart of the intelligent assisted driving warning method provided according to the embodiments of this application, which includes S22, S24 and S26, and is described in detail below.

[0129] S22: Acquire infrared images of the target detection object in the target driving scene collected by the infrared acquisition device and simultaneously acquire point cloud data of the target detection object in the target driving scene collected by the lidar. Both the infrared acquisition device and the lidar are installed on the driving body.

[0130] The target driving scenario refers to the scenario that the driving unit equipped with the infrared acquisition device needs to monitor during driving.

[0131] S24: Target detection is performed on the infrared image using the target detection method provided in any embodiment of this application to obtain the three-dimensional information of the target object.

[0132] Since the target detection method provided in the embodiments of this application has been described in detail above, it will not be repeated here.

[0133] S26: Based on the three-dimensional information, determine whether there is a collision risk between the driving vehicle and the target detection object, and provide corresponding prompts.

[0134] By comparing the distance information in the three-dimensional data with the corresponding warning distance threshold, it can be determined whether there is a risk of collision between the driving subject and the target detection object. If so, an alarm message indicating that the distance is too close will be generated.

[0135] Furthermore, to improve the accuracy of the close proximity warning, S26 can specifically determine whether there is a collision risk between the driving vehicle and the target detection object based on the distance information and motion direction information in the three-dimensional information, and generate corresponding prompt information. The distance information includes the lateral distance and longitudinal distance between the target detection object and the driving vehicle. Further, when the motion directions are in the same direction, if the lateral distance is less than a first lateral threshold and the longitudinal distance is less than a first longitudinal threshold, then it is determined that there is a collision risk between the driving vehicle and the target detection object, and a warning prompt is generated.

[0136] In some embodiments, based on distance information and direction of movement information, it is determined whether there is a collision risk between the driving vehicle and the target detection object, and corresponding prompt information is generated. It may further include: when the direction of movement is the same, if the lateral distance is greater than a first lateral threshold and less than a second lateral threshold, and the lateral distance increases and decreases with driving time, and the longitudinal distance is less than a second longitudinal threshold, then it is determined that there is a collision risk between the driving vehicle and the target detection object, and a warning prompt information is generated.

[0137] In some embodiments, based on distance information and direction of motion information, it is determined whether there is a collision risk between the driving body and the target detection object, and corresponding prompt information is generated. It may further include: when the direction of motion is opposite, if the lateral distance is less than a third lateral threshold, the longitudinal distance is less than a third longitudinal threshold, and the longitudinal distance decreases with the extension of driving time, then it is determined that there is a collision risk between the driving body and the target detection object, and a warning prompt information is generated.

[0138] The intelligent assisted driving warning method provided in the above embodiments achieves the same technical effect as the target detection method provided in the embodiments of this application during the warning process. To avoid repetition, it will not be described again here.

[0139] Please see Figure 15 The diagram shown is a schematic representation of the target detection device provided according to an embodiment of this application. In this embodiment, the target detection device includes a processor 211 and a memory 212. The memory 212 stores a computer program that can be executed by the processor. When the computer program is executed by the processor, it implements the target detection method of any embodiment of this application. The target detection device and the target detection method provided in the foregoing embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0140] Please see Figure 16The diagram shows a schematic of an intelligent assisted driving warning system according to an embodiment of this application. It includes an infrared acquisition device 1, a lidar 2, a processor 3, and an alarm device 4. The infrared acquisition device 1 is used to acquire infrared images of the target object. The lidar 2 is used to acquire point cloud data of the target object. The processor 3, when executing a computer program, implements the intelligent assisted driving warning method provided in any embodiment of this application. The alarm device 4 is used to issue an alarm based on the prompt information generated by the processor 3. The intelligent assisted driving warning system provided in this application is installed on a vehicle, wherein the vehicle includes a drivable carrier, such as a car, that is either stationary or in motion.

[0141] Please continue reading. Figure 16 As shown, in some embodiments, the intelligent assisted driving warning system further includes a display device 5. The display device 5 is connected to the processor 3 to display an infrared image or point cloud data map containing the target detection object, based on the category information and three-dimensional information of the target detection object obtained by the processor 3. The infrared image or point cloud data map displays the category of the target detection object, the three-dimensional detection frame enclosing the target detection object, and the distance between the target detection object and the driving vehicle. Specifically, the infrared image or point cloud data map displayed by the display device 5 is as follows: Figures 2 to 6 As shown, the results display a 3D detection box, category, and motion direction information of the target object relative to the infrared acquisition device in the infrared image or point cloud data map.

[0142] Please see Figure 17 As shown, it is a schematic diagram of the working process of the intelligent assisted driving warning system provided according to the embodiments of this application, as detailed below.

[0143] S1: Point cloud data and infrared images are acquired simultaneously using an infrared sensor and a lidar sensor. The infrared sensor is the infrared acquisition device mentioned above, and the lidar sensor is the lidar mentioned above.

[0144] S2: Perform detail and contrast enhancement processing on the infrared image, rotate the ROI region of the point cloud data, and crop the field of view according to the infrared sensor's field of view. This step is the same as the detail enhancement processing of the infrared image and the data filtering of the raw point cloud data obtained by the LiDAR mentioned above.

[0145] S3: Input the infrared image into the trained two-dimensional neural network model to extract the pixel feature information of each pixel in the infrared image and obtain pixel feature data.

[0146] S4: Project each point cloud in the point cloud data onto the infrared image, and determine the pixel feature data and neighborhood weighted average pixel feature data corresponding to each point cloud by using the coordinate mapping relationship between the point cloud and the pixel points determined by the intrinsic parameter matrix of the infrared sensor and the joint calibration outer matrix between the infrared sensor and the lidar sensor.

[0147] S5: Input the point cloud image fusion data obtained by fusing the point cloud feature data, pixel feature data and weighted average pixel feature data corresponding to each point cloud into the three-dimensional neural network model to output the size, orientation and distance information of the target detection object.

[0148] S6: Determine whether to issue a warning for the target object based on distance and orientation information. The warning here refers to generating a "too close" alert.

[0149] S7: The in-vehicle display device displays 3D detection information of the target object and provides intelligent warnings for targets that are too close. These warnings specifically involve displaying a proximity alert to the user.

[0150] The intelligent assisted driving warning system provided in this application integrates infrared images acquired by an infrared acquisition device and point cloud data acquired by a lidar, and performs 3D target detection and warning based on the fused data. Based on the above analysis and description, the intelligent assisted driving warning system provided in this application can be used normally at night and in adverse weather conditions such as rain, snow, and fog. It performs 3D target detection and outputs target size, orientation, and distance information, and provides warnings for targets that are too close, helping drivers achieve safe driving. The target detection method and device based on data fusion, as well as the intelligent assisted driving warning method and system provided in this application, can bring at least one of the following technical effects.

[0151] 1. The target detection method, device, and early warning method and system provided in this application realize 3D information target detection by fusing infrared images and point cloud data of lidar, and can realize obstacle detection and early warning functions in all time, all weather, and all scenarios.

[0152] 2. The target detection method, device, and early warning method and system provided in this application take into account the sparse nature of point cloud data. When fusing infrared image information, not only are the features of the corresponding infrared image pixels in the point cloud added, but also the weighted feature information of the pixel's adaptive neighborhood is added. This can increase the receptive field of distant targets and obtain more semantic information, while also reducing the interference of CNN features of pixels in the neighborhood of nearby targets.

[0153] 3. The target detection method, device, and early warning method and system provided in this application add a 2D detection auxiliary network (two-dimensional neural network model) to the three-dimensional neural network model used to realize 3D target detection, which reduces the false detection and missed detection of distant objects by the point cloud network, and the auxiliary network is only applied to the training stage and will not increase the additional inference cost.

[0154] 4. In view of the richness and complexity of vehicle-mounted scenarios, the target detection equipment provided in this application embodiment can be upgraded online in the future until it meets the needs of all vehicle-mounted scenarios.

[0155] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described target detection method and / or intelligent assisted driving warning method embodiments, and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium includes, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0156] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target detection method based on data fusion, characterized in that, include: Acquire synchronized point cloud data and infrared images of the target object; The point cloud data and the infrared image are fused to obtain point cloud image fusion data. This process includes: The infrared image is subjected to feature extraction using a two-dimensional neural network model to obtain infrared feature data of each pixel in the infrared image; the infrared feature data corresponding to each point cloud is determined based on the coordinate mapping relationship between each point cloud in the point cloud data and each pixel in the infrared image, as well as the infrared feature data of each pixel in the infrared image; the point cloud feature data and the corresponding infrared feature data of each point cloud are fused to obtain point cloud image fusion data corresponding to each point cloud. The step of fusing the point cloud feature data and the corresponding infrared feature data of each point cloud to obtain point cloud image fusion data corresponding to each point cloud includes: Based on the point cloud feature data of each point cloud, a neighborhood centered on each pixel of each point cloud is determined; the infrared feature data corresponding to each pixel in each neighborhood in the infrared image is weighted and averaged to obtain the weighted average infrared feature data corresponding to each point cloud; the point cloud feature data, the infrared feature data, and the weighted average infrared feature data corresponding to each point cloud are fused to obtain the point cloud image fusion data corresponding to each point cloud. Target detection is performed on the point cloud image fusion data using a three-dimensional neural network model to obtain the category information and three-dimensional information of the target object. The three-dimensional information includes at least one of the following: the size information, position information, distance information, and direction of motion information of the target object being detected.

2. The target detection method according to claim 1, characterized in that, The step of extracting and recognizing features from the point cloud image fusion data using a three-dimensional neural network model to obtain the category information and three-dimensional information of the target detection object includes: The point cloud image fusion data is voxelized to obtain multiple columnar voxels; Feature extraction and mapping are performed on multiple columnar voxels to obtain voxel features of the point cloud image fusion data, and the voxel features are mapped to a bird's-eye view to obtain a bird's-eye view feature map corresponding to the point cloud image fusion data. The voxel features include the point cloud three-dimensional coordinate data, the infrared feature data, the weighted average infrared feature data, the geometric center data, and the geometric center offset data corresponding to the point cloud image fusion data. The bird's-eye view feature map is input into the three-dimensional neural network model for feature extraction and recognition to obtain the category signal and three-dimensional information of the target detection object.

3. The target detection method according to claim 1, characterized in that, Before performing feature extraction and recognition on the point cloud image fusion data using a three-dimensional neural network model to obtain the category information and three-dimensional information of the target detection object, the target detection method further includes: A training sample dataset is constructed based on sample point cloud image fusion data carrying annotation information; wherein, the annotation information includes the category annotation information and 3D bounding box information of the target detection object; the 3D bounding box information includes the center point information, length, width and height information and yaw angle information of the 3D bounding box. The three-dimensional neural network model is iteratively trained using the training sample dataset until the preset training conditions are met.

4. The target detection method according to claim 3, characterized in that, The training sample dataset constructed based on the sample point cloud image fusion data carrying annotation information includes: The sample point cloud data corresponding to the fused sample point cloud image data is labeled to obtain the three-dimensional annotation box information; Based on the mapping relationship between the sample infrared image corresponding to the sample point cloud image fusion data and the sample point cloud data, the three-dimensional annotation box corresponding to the sample point cloud data is projected onto the sample infrared image; When the projection area of ​​the three-dimensional bounding box corresponding to the sample point cloud data onto the sample infrared image meets a preset condition, the fused data of the corresponding sample point cloud image carrying the three-dimensional bounding box information is used as a training sample to form the training sample dataset.

5. The target detection method according to claim 4, characterized in that, After annotating the sample point cloud data corresponding to the fused sample point cloud image to obtain the three-dimensional annotation box information, and before extracting features from the infrared image using a two-dimensional neural network model to obtain the infrared feature data of each pixel in the infrared image, the target detection method further includes: Based on the three-dimensional bounding box information, the two-dimensional bounding box information corresponding to the target detection object in the sample infrared image is determined. The two-dimensional bounding box information includes the center point information and width and height information of the two-dimensional bounding box. The two-dimensional neural network model is trained using the sample infrared image carrying the two-dimensional bounding box information; The step of extracting features from the infrared image using a two-dimensional neural network model to obtain infrared feature data for each pixel in the infrared image includes: The infrared image is used to extract features by the feature extraction network in the trained two-dimensional neural network model, thereby obtaining the infrared feature data of each pixel in the infrared image.

6. A method for intelligent assisted driving warning, characterized in that, include: The system acquires infrared images of the target objects in the target driving scene collected by an infrared acquisition device and simultaneously acquires point cloud data of the target objects in the target driving scene collected by a lidar. Both the infrared acquisition device and the lidar are mounted on the vehicle body. Based on the point cloud data and the infrared image, target detection is performed using the target detection method as described in any one of claims 1 to 5 to obtain the three-dimensional information of the target object; Based on the three-dimensional information, it is determined whether there is a collision risk between the driving vehicle and the target detection object, and a corresponding prompt is given.

7. The intelligent assisted driving warning method according to claim 6, characterized in that, The three-dimensional information includes distance information and motion direction information. The distance information includes the lateral distance and longitudinal distance between the target detection object and the driving body. Based on the three-dimensional information, the determination of whether there is a collision risk between the driving body and the target detection object, and the provision of corresponding prompts, includes: When the motion direction information indicates unidirectional motion, if the lateral distance is less than a first lateral threshold and the longitudinal distance is less than a first longitudinal threshold, then it is determined that there is a collision risk between the driving vehicle and the target detection object, and a warning message is generated; and / or, When the motion direction information indicates unidirectional motion, if the lateral distance is greater than the first lateral threshold and less than the second lateral threshold, and the lateral distance increases and decreases with driving time, and the longitudinal distance is less than the second longitudinal threshold, then it is determined that there is a collision risk between the driving vehicle and the target detection object, and a warning message is generated; and / or, When the direction of motion is reversed, if the lateral distance is less than the third lateral threshold, the longitudinal distance is less than the third longitudinal threshold, and the longitudinal distance decreases with the extension of driving time, then it is determined that there is a collision risk between the driving body and the target detection object, and a warning message is generated.

8. A target detection device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program that can be executed by the processor, and the computer program, when executed by the processor, implements the target detection method as described in any one of claims 1 to 5.

9. An intelligent assisted driving warning system, characterized in that, This includes infrared data acquisition equipment, lidar, processors, and alarm devices; The infrared acquisition device is used to acquire infrared images of the target object being detected. The lidar is used to collect point cloud data of the target object; The processor implements the intelligent assisted driving warning method as described in any one of claims 6 or 7 when executing a computer program; The alarm device is used to issue an alarm based on the prompt information generated by the processor.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the target detection method as described in any one of claims 1 to 5 or the intelligent assisted driving warning method as described in any one of claims 6 or 7.

Images