Camera and millimeter wave radar information fusion target detection method and detection device

Abstract

The invention discloses a target detection method and device based on camera and millimeter wave radar information fusion. The method comprises the following steps: acquiring target camera data, target radar data and vehicle driving data in a current scene; generating a first fusion track based on the target camera data and the driving data; in the first fusion domain, performing fusion based on the target radar data and the first fusion trajectory to obtain a second fusion trajectory; and in the second fusion domain, determining a target radar detection point based on the target radar data, updating the second fusion track based on the target radar detection point, and outputting the updated second fusion track under the condition that the target radar data and the target camera data are successfully associated. According to the method provided by the invention, the recognized target can be understood from different angles and dimensions, so that the accuracy of target detection is improved, and a more reliable environmental perception capability is provided for applications such as automatic driving.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a target detection method that fuses camera and millimeter-wave radar information, a computer-readable storage medium, a vehicle, and a target detection device that fuses camera and millimeter-wave radar information. Background Technology

[0002] Currently, in the field of L0-L2 level assisted driving, functions such as dual warning, AEB (Automatic Emergency Braking), and ACC (Adaptive Cruise Control) generally use two types of sensors, cameras and millimeter-wave radar, to fuse perception information and output the obstacle target with higher confidence.

[0003] However, in terms of specific fusion methods, some methods only perform information fusion at the single image level. When analyzing and matching multi-dimensional information of the captured target, the fusion accuracy is relatively low and the reliability is insufficient. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in related technologies. To this end, the first objective of this application is to propose a target detection method that fuses camera and millimeter-wave radar information, enabling the understanding and identification of targets from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0005] The second objective of this application is to provide a computer-readable storage medium.

[0006] The third objective of this application is to propose a vehicle.

[0007] The fourth objective of this application is to propose a target detection device that fuses camera and millimeter-wave radar information.

[0008] To achieve the above objectives, a first aspect of this application proposes a target detection method that fuses camera and millimeter-wave radar information. The method includes: acquiring target camera data, target radar data, and vehicle driving data in a current scene; generating a first fused trajectory based on the target camera data and the driving data; fusing the target radar data with the first fused trajectory in a first fused domain to obtain a second fused trajectory; determining target radar detection points based on the target radar data in the second fused domain, updating the second fused trajectory based on the target radar detection points, and outputting the updated second fused trajectory when the target radar data and the target camera data are successfully associated.

[0009] According to the target detection method based on the fusion of camera and millimeter-wave radar information in this application, target camera data, target radar data, and vehicle driving data in the current scene are acquired. A first fused trajectory is generated based on the target camera data and driving data. In the first fusion domain, the target radar data and the first fused trajectory are fused to obtain a second fused trajectory. In the second fusion domain, target radar detection points are determined based on the target radar data, and the second fused trajectory is updated based on the target radar detection points. If the target radar data and target camera data are successfully associated, the updated second fused trajectory is output. Therefore, this method can understand and identify targets from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0010] In addition, the target detection method based on the camera and millimeter-wave radar information fusion of the above embodiments of this application may also have the following additional technical features: According to one embodiment of this application, generating a first fused trajectory based on the target camera data and the driving data includes: performing image preprocessing on images captured by the camera to obtain the target camera data, wherein the image preprocessing includes noise reduction and contrast enhancement; identifying objects corresponding to the current camera frame in the target camera data based on a target detection algorithm, and assigning an identity identifier to each object; determining the target position of the object in the next camera frame based on a machine learning algorithm and the driving data, so as to update the target tracking trajectory based on the target position; matching the object at the target position with the object actually detected in the next camera frame, and if the matching fails, initializing a new object and assigning a new identity identifier, and adding the new tracking target to the target tracking trajectory to obtain the first fused trajectory.

[0011] According to one embodiment of this application, in a first fusion domain, a second fusion trajectory is obtained by fusing the target radar data with the first fusion trajectory, including: preprocessing the radar detection data to obtain the target radar data, wherein the preprocessing includes denoising and signal amplification; assigning an identity identifier to each object detected by the radar, and determining the target position of the object in the next radar frame based on a target prediction algorithm; matching the object position in the radar frame with the object position in the camera frame of the first fusion trajectory in the first fusion domain; if the matching is successful, updating the first fusion trajectory according to the parameter information corresponding to the object position in the radar frame to obtain the second fusion trajectory, wherein updating the first fusion trajectory includes updating at least one of the object's position, velocity, and direction.

[0012] According to one embodiment of this application, the camera data includes lane lines, which are used to identify the positional relationship between the object and the lane lines. Matching the object position in the radar frame with the object position in the camera frame of the first fused trajectory includes: determining a first positional relationship between the object position in the radar frame and the lane lines, and determining a second positional relationship between the object position in the camera frame of the first fused trajectory and the lane lines; if the first positional relationship and the second positional relationship are the same, matching the object position in the radar frame with the object position in the camera frame of the first fused trajectory based on a nearest neighbor algorithm.

[0013] According to one embodiment of this application, determining the target radar detection point in the target radar data includes: when there are multiple radar detection points corresponding to the object being detected by radar, selecting the radar detection point corresponding to the minimum distance to the center of the second fused trajectory as the target radar detection point.

[0014] According to one embodiment of this application, the method further includes: determining the number of matchings between the target camera data and the target radar data based on the second fused trajectory; and determining a target radar detection point based on the target radar data if the number of matchings is greater than a preset number of matchings threshold.

[0015] According to one embodiment of this application, the method further includes: labeling the object data source of the updated second fused trajectory, the data source including at least a camera source, a millimeter-wave radar source, or a camera and millimeter-wave radar fusion source.

[0016] To achieve the above objectives, a second aspect of this application provides a computer-readable storage medium storing a program that, when executed by a processor, implements the target detection method described above, which fuses camera and millimeter-wave radar information.

[0017] The computer-readable storage medium according to the embodiments of this application implements the target detection method of camera and millimeter-wave radar information fusion described above during execution, which can understand the identified target from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0018] To achieve the above objectives, a vehicle is proposed in the third aspect of this application, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described target detection method that fuses camera and millimeter-wave radar information.

[0019] The vehicle according to the embodiments of this application, by executing the target detection method of camera and millimeter-wave radar information fusion described above, can understand and identify the target from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0020] To achieve the above objectives, a fourth aspect of this application provides a target detection device that fuses camera and millimeter-wave radar information. The device includes: an acquisition module for acquiring target camera data, target radar data, and vehicle driving data in a current scene; a generation module for generating a first fused trajectory based on the target camera data and the driving data; a first fusion module for fusing the target radar data and the first fused trajectory in a first fusion domain to obtain a second fused trajectory; and a second fusion module for determining target radar detection points based on the target radar data in the second fusion domain, updating the second fused trajectory based on the target radar detection points, and outputting the updated second fused trajectory when the target radar data and the target camera data are successfully associated.

[0021] According to the target detection device based on the camera and millimeter-wave radar information fusion embodiments of this application, the acquisition module is used to acquire target camera data, target radar data, and vehicle driving data in the current scene; the generation module is used to generate a first fused trajectory based on the target camera data and driving data; the first fusion module is used to fuse the target radar data and the first fused trajectory in a first fusion domain to obtain a second fused trajectory; the second fusion module is used to determine target radar detection points based on the target radar data in the second fusion domain, update the second fused trajectory based on the target radar detection points, and output the updated second fused trajectory when the target radar data and target camera data are successfully associated. Therefore, this device can understand and identify targets from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] Figure 1 This is a flowchart of a target detection method based on camera and millimeter-wave radar information fusion according to an embodiment of this application.

[0024] Figure 2 This is a flowchart illustrating a target detection method that fuses camera and millimeter-wave radar information, according to a specific example of this application.

[0025] Figure 3This is a block diagram of a vehicle according to an embodiment of this application.

[0026] Figure 4 This is a block diagram of a target detection device that fuses camera and millimeter-wave radar information according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0028] The target detection method, computer-readable storage medium, vehicle, and target detection device based on camera and millimeter-wave radar information fusion proposed in this application are described below with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart of a target detection method based on camera and millimeter-wave radar information fusion according to an embodiment of this application.

[0030] like Figure 1 As shown, the target detection method based on the fusion of camera and millimeter-wave radar information in this application embodiment may include the following steps: S1: Acquire target camera data, target radar data, and vehicle driving data in the current scene.

[0031] S2 generates the first fused trajectory based on target camera data and driving data.

[0032] S3, in the first fusion domain, the target radar data and the first fusion trajectory are fused to obtain the second fusion trajectory.

[0033] S4. In the second fusion domain, the target radar detection point is determined based on the target radar data, and the second fusion trajectory is updated based on the target radar detection point. If the target radar data and the target camera data are successfully associated, the updated second fusion trajectory is output.

[0034] Specifically, during vehicle operation, target camera data, target radar data, and vehicle driving data are acquired in the current scene. Target camera data consists of image data acquired from the vehicle's onboard cameras, which may include visual information about the area in front of the vehicle. Target radar data is acquired from millimeter-wave radar, including information such as the distance, speed, and angle of the identified object. Vehicle driving data can be obtained from the vehicle's chassis system, such as vehicle speed, acceleration, and steering angular velocity, to understand the vehicle's own motion state. By simultaneously acquiring data from multiple sensors, a comprehensive information foundation is provided for subsequent target detection and tracking.

[0035] After acquiring target camera data and driving data, a first fused trajectory can be generated based on these data. For example, target detection algorithms (such as YOLO (You Only Look Once, a real-time target detection system) and SSD (Single Shot MultiBox Detector)) can be used to identify and locate objects (also known as localized targets, such as vehicles and pedestrians) in the image. Features such as shape, size, and color can be extracted from the detected targets, and these features will be used for subsequent tracking and matching. Then, tracking algorithms (such as KCF (Kernelized Correlation Filters) and DeepSORT (Simple Online and Realtime Tracking)) can be applied to track the target in consecutive image frames, estimate the target's trajectory, and estimate the target's velocity and acceleration based on the target's position changes in consecutive frames, combined with the current vehicle's driving data. Monocular ranging techniques (such as using lane vanishing points or known object sizes) can be used to estimate the distance between the target and the camera. This allows the integration of the target's position, velocity, and depth information detected by the camera to generate a first fused trajectory. Furthermore, filters (such as Kalman filters) can be used to optimize the trajectory, making it smoother and more accurate. Through these steps, an accurate first fused trajectory can be generated from the raw camera image data and vehicle driving data. This trajectory not only includes the object's position and velocity information but also adds target distance information through depth estimation, providing a foundation for subsequent multi-sensor fusion.

[0036] After acquiring target radar data, a second fused trajectory can be obtained by fusing the target radar data with a first fused trajectory within a first fusion domain. For example, radar signal processing techniques (such as the CFAR (Constant False Alarm Rate) algorithm) can be used to identify objects within the radar field of view, track the detected objects, estimate their motion states in consecutive radar frames, and generate radar trajectories. The radar data can be transformed from the radar coordinate system to the vehicle coordinate system to facilitate comparison and fusion with the first fused trajectory in the first fusion domain (BEV (Bird's Eye View) fusion domain), i.e., switching to a bird's eye view to allow for comparison and fusion with the BEV fused trajectory from a unified perspective. For example, features helpful for object identification, such as position, speed, and angle, can be extracted from the radar trajectory, and feature matching algorithms (such as nearest neighbor, probabilistic data association, etc.) can be used to match radar features with objects in the first fused trajectory. Based on the results of radar data association, the information in the first fused trajectory, including the target's position and speed, is updated to obtain the second fused trajectory. Therefore, radar data enhances the accuracy of camera data, thereby improving the accuracy and robustness of target detection and tracking.

[0037] In the second fusion domain, target radar detection points can be determined based on target radar data, and the second fused trajectory can be updated based on these detection points. After the update, the data association between the target radar data and the target camera data is assessed. If the association is successful, the updated second fused trajectory is output. The second fusion domain is the image domain, referring to the stage of data fusion and processing on a two-dimensional image plane. Target radar detection points are the closest points of the detected objects to the vehicle in the radar data, including information such as distance, speed, and angle. Target camera data is object information extracted from images captured by the camera, including the object's position, size, and shape. Radar data can be transformed from the radar coordinate system to the vehicle coordinate system, or directly to the image coordinate system, projecting the radar detection points onto the image plane for comparison and fusion with the camera data. Target features, such as edges, corners, and textures, are extracted from the image, and feature matching algorithms are used to associate the radar detection points with objects in the image. Combining radar and camera data, the second fused trajectory is updated, and filters (such as Kalman filtering) are used to optimize the trajectory, ensuring its smoothness and consistency.

[0038] Therefore, after verifying the accuracy and reliability of the fused trajectory, the updated second fused trajectory can be output for use by downstream modules, such as path planning and decision-making. The BEV fusion domain provides a top-down perspective, simplifying geometric relationships and making it easier to process and analyze the target's position and motion. It is suitable for scenarios requiring a global perspective and simplified data processing. Fusion in the image domain provides rich visual information, such as color, texture, and shape, which is crucial for target recognition and classification. Image domain processing can utilize this information for more refined target analysis and understanding. In short, combining both improves overall perception performance; BEV provides global motion information, while the image domain provides detailed visual features, and their combination provides a more comprehensive and accurate environmental understanding.

[0039] Therefore, it can effectively integrate data from different sensors to generate accurate and reliable fusion trajectories, and understand the identified targets from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0040] According to one embodiment of this application, generating a first fused trajectory based on target camera data and driving data includes: performing image preprocessing on images captured by the camera to obtain target camera data, wherein the image preprocessing includes noise reduction and contrast enhancement; identifying objects corresponding to the current camera frame in the target camera data based on a target detection algorithm, and assigning an identity identifier to each object; determining the target position of the object in the next camera frame based on a machine learning algorithm and driving data, so as to update the target tracking trajectory based on the target position; matching the object at the target position with the object actually detected in the next camera frame, and if the matching fails, initializing a new object and assigning a new identity identifier, and adding the new tracking target to the target tracking trajectory to obtain the first fused trajectory.

[0041] Specifically, when generating the first fused trajectory based on target camera data and driving data, the images captured by the camera are first preprocessed to obtain target camera data. Image preprocessing includes denoising and contrast enhancement. Denoising involves using filters (such as Gaussian filters and median filters) to reduce random noise in the image and improve image quality. Contrast enhancement involves adjusting the brightness and contrast of the image to make target features more prominent, facilitating subsequent processing. In other words, image preprocessing improves image quality, enabling target detection and tracking algorithms to more accurately identify and distinguish targets.

[0042] Then, based on object detection algorithms, the objects corresponding to the current camera frame can be identified in the target camera data, and each object can be assigned an identity identifier. For example, object detection algorithms (such as the YOLO algorithm) can be applied to the preprocessed image to identify objects (such as vehicles, pedestrians, etc.) in the image, that is, to locate and identify the objects of interest in the image, providing a basis for subsequent tracking, and assigning a unique identity identifier (ID) to each detected object so that the same target can be tracked in consecutive frames, ensuring that each object can be uniquely identified throughout the tracking process and avoiding object confusion.

[0043] Next, the target position in the next camera frame can be determined based on machine learning algorithms and driving data, and the target tracking trajectory can be updated based on the target position. For example, by using machine learning models (such as deep neural networks) combined with vehicle driving data (speed, acceleration, etc.) to predict the target's position in the next camera frame, the future position of the target can be estimated in advance, which helps to improve the continuity and accuracy of tracking. Thus, the target's tracking trajectory, including position, speed, and other information, is updated according to the predicted position.

[0044] The target location is matched with the actual detected object in the next camera frame. If the match fails, a new object is initialized and assigned a new identifier, and the new tracking target is added to the tracking trajectory to obtain the first fused trajectory. For example, feature matching algorithms (such as IOU (Intersection over Union), Mahalanobis distance, etc.) are used to match the predicted target location with the actual detected target in the next frame to ensure that the fused trajectory accurately reflects the actual motion of each object. If the match fails (i.e., no matching target is found), a new object is initialized. That is, there is a new object that was not previously detected, so a new identifier can be assigned to the new object, and the new tracking target can be added to the tracking trajectory, so that newly appearing targets can be included in the tracking in a timely manner, maintaining the integrity of the tracking. Finally, a first fused trajectory containing all target information can be generated, that is, all updated target trajectories are integrated to form the first fused trajectory, including the target's position, velocity, identifier, etc., thus generating a target trajectory that integrates visual and dynamic information to support subsequent decision-making and control.

[0045] Therefore, an accurate first fusion trajectory can be generated from the original camera image data and vehicle driving data. This trajectory not only contains the target's position and velocity information, but also adds target distance information through depth estimation, providing a foundation for subsequent multi-sensor fusion. It can be used in high-precision environmental perception applications such as autonomous driving, and can significantly improve the accuracy and robustness of target detection.

[0046] According to one embodiment of this application, in a first fusion domain, a second fusion trajectory is obtained by fusing target radar data with a first fusion trajectory, including: preprocessing radar detection data to obtain target radar data, wherein the preprocessing includes denoising and signal amplification; assigning an identity identifier to each object detected by the radar, and determining the target position of the object in the next radar frame based on a target prediction algorithm; matching the object position in the radar frame with the object position in the camera frame of the first fusion trajectory in the first fusion domain; and updating the first fusion trajectory according to the parameter information corresponding to the object position in the radar frame to obtain the second fusion trajectory, wherein updating the first fusion trajectory includes updating at least one of the object's position, velocity, and orientation.

[0047] Specifically, in the first fusion domain, when fusing the target radar data with the first fusion trajectory to obtain the second fusion trajectory, the radar detection data can first be preprocessed to obtain the target radar data. This preprocessing includes denoising and signal amplification. For example, filters (such as Kalman filters) can be used to remove random noise from the radar signal, reducing false alarms. Low-noise amplifiers can amplify the signal while minimizing introduced noise. Therefore, preprocessing the radar detection data improves its quality, making it more suitable for target detection and tracking. Then, information about the target can be extracted from the preprocessed radar data. Information such as the target's range, velocity, and angle can be extracted from the radar data; this information will be used for subsequent target identification and tracking.

[0048] Each object detected by the radar can then be assigned an identifier, and its target position in the next radar frame can be determined based on a target prediction algorithm. For example, an algorithm (such as the nearest neighbor algorithm) can be used to assign a unique ID to each detected object, ensuring that each object can be uniquely identified during tracking and avoiding target confusion. Using machine learning algorithms (such as deep learning models) in conjunction with vehicle driving data to predict the object's position in the next frame, i.e., estimating the target's future position in advance, helps improve the continuity and accuracy of tracking.

[0049] In the first fusion domain, the object positions in the radar frames can be matched with the object positions in the camera frames within the first fused trajectory. Specifically, in the BEV domain, a first fused trajectory is generated based on camera data, containing information such as the target's position, velocity, and orientation. Features helpful for target identification, such as position, velocity, and angle, are extracted from the radar trajectory. Feature matching algorithms (such as nearest neighbor and probabilistic data association) are used to match the radar features with the objects in the first fused trajectory. The matching result is then determined. If a match is successful, the first fused trajectory is updated based on the parameter information corresponding to the object positions in the radar frames to obtain a second fused trajectory. Updating the first fused trajectory includes updating at least one of the object's position, velocity, and orientation.

[0050] For example, feature matching algorithms (such as nearest neighbor, Kalman filter matching, etc.) are used to match targets in radar frames with targets in the first fused trajectory. Matching may be based on position, velocity, or other features. A threshold (such as a distance threshold) is set to determine if a match is successful. If the distance between the radar target and the camera target is less than this threshold, the match is considered successful. If a position match is successful, the position of the corresponding object in the first fused trajectory is updated using the object position in the radar frame. Similarly, if a velocity match is successful, the target's velocity information is updated. If the radar data provides direction information, the target's direction information is also updated. Additionally, filters (such as Kalman filters) can be used to smooth the trajectory and reduce trajectory fluctuations caused by noise or temporary obstructions. Thus, radar data and the first fused trajectory can be effectively fused in the BEV domain to obtain the second fused trajectory, thereby improving the accuracy and robustness of target detection and tracking.

[0051] According to one embodiment of this application, camera data includes lane lines, which are used to identify the positional relationship between an object and the lane lines. Matching the object position in the radar frame with the object position in the camera frame of the first fused trajectory includes: determining a first positional relationship between the object position in the radar frame and the lane lines, and determining a second positional relationship between the object position in the camera frame of the first fused trajectory and the lane lines; if the first positional relationship and the second positional relationship are the same, matching the object position in the radar frame with the object position in the camera frame of the first fused trajectory based on a nearest neighbor algorithm.

[0052] Specifically, camera data includes lane lines, which are used to identify the positional relationship between objects and lane lines. For example, image processing techniques (such as edge detection and machine learning) are used to identify lane lines in an image and determine the position of objects (such as vehicles and pedestrians) relative to the lane lines. Each detected object can be assigned a lane-line related label, such as "inside the lane," "left side of the lane," or "right side of the lane." In other words, by capturing image data from the camera and extracting lane line information, obtaining the lane line positions on the road ahead of the vehicle helps determine the travel paths of vehicles and targets.

[0053] When matching the object position in the radar frame with the object position in the camera frame of the first fused trajectory, a first positional relationship between the object position in the radar frame and the lane line can be determined first, and a second positional relationship between the object position in the camera frame of the first fused trajectory and the lane line can be determined second. For example, information such as the target's distance, speed, and angle can be obtained from millimeter-wave radar. The radar data can be converted to the same coordinate system as the lane line to determine the position of the target detected by the radar relative to the lane line. Each target detected by the radar can be assigned a lane-line related identifier, such as the first positional relationship being "inside the lane," "left side of the lane," or "right side of the lane." The position of the object in the camera frame of the first fused trajectory relative to the lane line can also be determined, providing the object in the first fused trajectory with relative positional information to the lane line, such as the second positional relationship being "inside the lane," "left side of the lane," or "right side of the lane," for more accurate matching. The first and second positional relationships are compared. If the first and second positional relationships are the same, the object position in the radar frame is matched with the object position in the camera frame of the first fused trajectory based on the nearest neighbor algorithm. In other words, matching is only performed when the first positional relationship between the object position in the radar frame and the lane line is satisfied, and the second positional relationship between the object position in the camera frame and the lane line in the first fused trajectory is satisfied. Furthermore, the nearest neighbor algorithm or other matching algorithms can be used to match the target position in the radar frame with the target position in the camera frame in the first fused trajectory, so as to ensure that the radar target and the camera target are only considered as the same target and matched when they are both located on the same or similar lane lines.

[0054] Therefore, the target position, speed and direction in the first fused trajectory can be updated based on the successfully matched radar data, that is, the data of radar and camera are integrated, and lane line information is used to quickly filter, thereby improving the accuracy and timeliness of object detection and tracking.

[0055] According to one embodiment of this application, determining a target radar detection point in target radar data includes: when multiple radar detection points exist in the object being detected by radar, selecting the radar detection point corresponding to the minimum distance from the center of the second fused trajectory as the target radar detection point.

[0056] Specifically, when determining the target radar detection point in the target radar data, if multiple radar detection points exist for the object being detected, the radar detection point with the minimum distance to the center of the second fused trajectory is selected as the target radar detection point. In other words, there may be multiple radar detection points for the object being detected, representing different parts of the same target or detection results at different times. For example, if the detected object is long and segmented (such as the distance between the cab and cargo box of a large truck), multiple object detection points may be detected by the radar. Therefore, the distance between each radar detection point and the center of the second fused trajectory can be calculated, and the radar detection point with the minimum distance to the center of the second fused trajectory is selected as the target radar detection point. This step ensures that the selected radar point is closest to the predicted position of the fused trajectory, thereby improving the accuracy of the matching.

[0057] Therefore, it is possible to identify the target radar detection point in the radar data and effectively match it with the fused trajectory, thereby improving the accuracy and robustness of target detection and tracking.

[0058] According to one embodiment of this application, the target detection method based on the fusion of camera and millimeter-wave radar information further includes: determining the number of matches between target camera data and target radar data based on a second fusion trajectory; and determining a target radar detection point based on the target radar data if the number of matches exceeds a preset matching number threshold. The preset matching number threshold can be determined according to actual conditions.

[0059] Specifically, a second fused trajectory is generated by combining camera and radar data. This trajectory includes information such as object position and velocity from both sensors. The number of matches of the same target in the camera and radar data across consecutive time frames is counted to evaluate the consistency of detection of the same target by both sensors. A threshold can be set to judge the stability of the matches, based on the needs of the actual application and the performance of the sensors; for example, a preset threshold of 3 matches is used. The number of matches for the current target is compared with the preset threshold to determine whether there are enough matches to confirm the reliability of target detection. When the number of matches exceeds the preset threshold, the target detection is considered stable. Only then is the target radar detection point determined based on the radar data, selecting the point in the radar frame closest to the fused trajectory. In other words, after confirming the reliability of object detection, the object's position is accurately determined for more precise tracking and analysis.

[0060] Therefore, by statistically analyzing the number of matches and setting thresholds, the confidence level of object detection can be improved, thereby providing more reliable object detection results.

[0061] According to one embodiment of this application, the target detection method based on camera and millimeter-wave radar information fusion further includes: labeling the object source of the updated second fused trajectory, wherein the data source includes at least a camera source, a millimeter-wave radar source, or a camera and millimeter-wave radar fusion source.

[0062] Specifically, after matching and fusing the radar frame with the first fused trajectory, the second fused trajectory is updated, including information such as the target's position, velocity, and orientation. The updated second fused trajectory is then labeled with the data source, which must include at least a camera source, a millimeter-wave radar source, or a camera-and-millimeter-wave radar fusion source. This clarifies which sensor or sensor combination provides the information for each target, aiding in data reliability assessment and subsequent processing. For example, if the object's information comes entirely from camera data, it is labeled "camera source." If the object's information comes entirely from millimeter-wave radar data, it is labeled "millimeter-wave radar source." If the object's information is provided by both camera and millimeter-wave radar data, it is labeled "camera-and-millimeter-wave radar fusion source."

[0063] Therefore, by fusing data from multiple sensors, the accuracy of target detection can be improved, and by labeling the data sources, the confidence level of target detection can be increased, providing more reliable information for subsequent processing.

[0064] The following is combined Figure 2 The method described in this application is used to describe the method.

[0065] As a specific example, the target detection method of this application, which fuses camera and millimeter-wave radar information, may include the following steps: S101: Acquire target camera data, target radar data, and vehicle driving data in the current scene.

[0066] S102, perform image preprocessing on the images captured by the camera to obtain target camera data. The image preprocessing includes noise reduction and contrast enhancement. Based on the target detection algorithm, identify the object corresponding to the current camera frame in the target camera data and assign an identity identifier to each object.

[0067] S103, determine the target position of the object in the next camera frame based on machine learning algorithm and driving data, and update the target tracking trajectory based on the target position; match the object at the target position with the object actually detected in the next camera frame, and if the matching fails, initialize a new object and assign a new identity, and add the new tracking target to the tracking trajectory to obtain the first fused trajectory.

[0068] S104, preprocess the radar detection data to obtain target radar data. The preprocessing includes denoising and signal amplification. Assign an identity to each object detected by the radar and determine the target position of the object in the next radar frame based on the target prediction algorithm.

[0069] S105, in the first fusion domain, the object position in the radar frame is matched with the object position in the camera frame in the first fusion trajectory. If the match is successful, the first fusion trajectory is updated according to the parameter information corresponding to the object position in the radar frame to obtain the second fusion trajectory. Updating the first fusion trajectory includes updating at least one of the object's position, velocity, and orientation.

[0070] S106, in the first fusion domain, the object position in the radar frame is matched with the object position in the camera frame in the first fusion trajectory. If the match is successful, the first fusion trajectory is updated according to the parameter information corresponding to the object position in the radar frame to obtain the second fusion trajectory. Updating the first fusion trajectory includes updating at least one of the object's position, velocity, and direction.

[0071] S107, determine the number of times the target camera data and target radar data are matched based on the second fused trajectory.

[0072] S108. Determine whether the number of matches exceeds the preset matching threshold. If yes, proceed to step S109; otherwise, proceed to step S107.

[0073] S109, when there are multiple radar detection points in the radar detection object, select the radar detection point corresponding to the minimum distance from the center of the second fused trajectory as the target radar detection point.

[0074] S110, in the second fusion domain, the second fusion trajectory is updated based on the target radar detection points.

[0075] S111, determine whether the target radar data and target camera data are successfully correlated. If yes, proceed to step S112; if no, proceed to step S110.

[0076] S112, output the updated second fused trajectory, and label the data source of the objects in the updated second fused trajectory. The data source includes at least a camera source, a millimeter-wave radar source, or a camera and millimeter-wave radar fusion source.

[0077] In summary, the target detection method based on the camera and millimeter-wave radar information fusion embodiment of this application acquires target camera data, target radar data, and vehicle driving data in the current scene. A first fused trajectory is generated based on the target camera data and driving data. Within the first fusion domain, the target radar data is fused with the first fused trajectory to obtain a second fused trajectory. Within the second fusion domain, target radar detection points are determined based on the target radar data, and the second fused trajectory is updated based on these target radar detection points. Finally, if the target radar data and target camera data are successfully associated, the updated second fused trajectory is output. Therefore, this method can understand and identify targets from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0078] Corresponding to the above embodiments, this application also proposes a computer-readable storage medium.

[0079] The computer-readable storage medium of this application embodiment stores a program that, when executed by a processor, implements the target detection method described above, which fuses camera and millimeter-wave radar information.

[0080] According to the computer-readable storage medium of the embodiments of this application, by executing the target detection method of camera and millimeter-wave radar information fusion described above, the identified target can be understood from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0081] Corresponding to the above embodiments, this application also proposes a vehicle.

[0082] like Figure 3 As shown, the vehicle 200 in this embodiment may include: a memory 210, a processor 220, and a program stored in the memory 210 and executable on the processor 220. When the processor 220 executes the program, it implements the target detection method described above, which fuses camera and millimeter-wave radar information.

[0083] The vehicle according to the embodiments of this application, by executing the target detection method of camera and millimeter-wave radar information fusion described above, can understand and identify the target from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0084] Corresponding to the above embodiments, this application also proposes a target detection device that fuses camera and millimeter-wave radar information.

[0085] like Figure 4As shown, the target detection device 100 that fuses camera and millimeter-wave radar information according to an embodiment of this application includes: an acquisition module 110, a generation module 120, a first fusion module 130, and a second fusion module 140.

[0086] The acquisition module 110 acquires target camera data, target radar data, and vehicle driving data in the current scene. The generation module 120 generates a first fused trajectory based on the target camera data and driving data. The first fusion module 130 fuses the target radar data with the first fused trajectory in the first fusion domain to obtain a second fused trajectory. The second fusion module 140 determines the target radar detection points based on the target radar data in the second fusion domain, updates the second fused trajectory based on the target radar detection points, and outputs the updated second fused trajectory if the target radar data and target camera data are successfully associated.

[0087] According to one embodiment of this application, the generation module 120 generates a first fused trajectory based on target camera data and driving data, specifically used for: performing image preprocessing on images captured by the camera to obtain target camera data, wherein the image preprocessing includes noise reduction and contrast enhancement; identifying objects corresponding to the current camera frame in the target camera data based on a target detection algorithm, and assigning an identity identifier to each object; determining the target position of the object in the next camera frame based on a machine learning algorithm and driving data, so as to update the target tracking trajectory based on the target position; matching the object at the target position with the object actually detected in the next camera frame, and in the case of a matching failure, initializing a new object and assigning a new identity identifier, and adding the new tracking target to the target tracking trajectory to obtain the first fused trajectory.

[0088] According to one embodiment of this application, in a first fusion domain, a first fusion module 130 fuses target radar data with a first fusion trajectory to obtain a second fusion trajectory. Specifically, it is used to: preprocess radar detection data to obtain target radar data, wherein the preprocessing includes denoising and signal amplification; assign an identity identifier to each object detected by the radar and determine the target position of the object in the next radar frame based on a target prediction algorithm; in the first fusion domain, match the object position in the radar frame with the object position in the camera frame of the first fusion trajectory; if the matching is successful, update the first fusion trajectory according to the parameter information corresponding to the object position in the radar frame to obtain the second fusion trajectory, wherein updating the first fusion trajectory includes updating at least one of the object's position, velocity, and direction.

[0089] According to one embodiment of this application, the camera data includes lane lines, which are used to identify the positional relationship between an object and the lane lines. The first fusion module 130 matches the object position in the radar frame with the object position in the camera frame of the first fusion trajectory. Specifically, it is used to: determine a first positional relationship between the object position in the radar frame and the lane lines, and determine a second positional relationship between the object position in the camera frame of the first fusion trajectory and the lane lines; if the first positional relationship and the second positional relationship are the same, match the object position in the radar frame with the object position in the camera frame of the first fusion trajectory based on the nearest neighbor algorithm.

[0090] According to one embodiment of this application, the second fusion module 140 determines the target radar detection point in the target radar data, specifically used for: when there are multiple radar detection points in the object being detected by radar, selecting the radar detection point corresponding to the minimum distance from the center of the second fusion trajectory as the target radar detection point.

[0091] According to one embodiment of this application, the second fusion module 140 is further configured to: determine the number of times the target camera data and the target radar data are matched based on the second fusion trajectory; and determine the target radar detection point based on the target radar data if the number of matches is greater than a preset number of matches threshold.

[0092] According to one embodiment of this application, the second fusion module 140 is further configured to: mark the data source of the object in the updated second fusion trajectory, wherein the data source includes at least a camera source, a millimeter-wave radar source, or a camera and millimeter-wave radar fusion source.

[0093] It should be noted that for details not disclosed in the target detection device for camera and millimeter-wave radar information fusion in the embodiments of this application, please refer to the details disclosed in the target detection method for camera and millimeter-wave radar information fusion in the embodiments of this application, which will not be repeated here.

[0094] According to the target detection device based on the camera and millimeter-wave radar information fusion embodiments of this application, the acquisition module is used to acquire target camera data, target radar data, and vehicle driving data in the current scene; the generation module is used to generate a first fused trajectory based on the target camera data and driving data; the first fusion module is used to fuse the target radar data and the first fused trajectory in a first fusion domain to obtain a second fused trajectory; the second fusion module is used to determine target radar detection points based on the target radar data in the second fusion domain, update the second fused trajectory based on the target radar detection points, and output the updated second fused trajectory when the target radar data and target camera data are successfully associated. Therefore, this device can understand and identify targets from different angles and dimensions, thereby improving the accuracy of target detection and providing more reliable environmental perception capabilities for applications such as autonomous driving.

[0095] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0096] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0097] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0099] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0100] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A target detection method that fuses camera and millimeter-wave radar information, characterized in that, The method includes: Acquire target camera data, target radar data, and vehicle driving data in the current scene; A first fused trajectory is generated based on the target camera data and the driving data; In the first fusion domain, the target radar data and the first fusion trajectory are fused to obtain the second fusion trajectory; In the second fusion domain, target radar detection points are determined based on the target radar data, and the second fusion trajectory is updated based on the target radar detection points. If the target radar data and the target camera data are successfully associated, the updated second fusion trajectory is output.

2. The target detection method based on the fusion of camera and millimeter-wave radar information according to claim 1, characterized in that, A first fused trajectory is generated based on the target camera data and the driving data, including: Image preprocessing is performed on images captured by the camera to obtain target camera data, wherein the image preprocessing includes noise reduction and contrast enhancement; Based on the target detection algorithm, the object corresponding to the current camera frame is identified in the target camera data, and an identity identifier is assigned to each object; The target position of the object in the next camera frame is determined based on the machine learning algorithm and the driving data, so as to update the target tracking trajectory based on the target position; The object at the target location is matched with the object actually detected in the next camera frame. If the match fails, a new object is initialized and a new identity is assigned. The new tracking target is then added to the target tracking trajectory to obtain the first fused trajectory.

3. The target detection method based on the fusion of camera and millimeter-wave radar information according to claim 1, characterized in that, In the first fusion domain, the target radar data and the first fusion trajectory are fused to obtain the second fusion trajectory, including: The radar detection data is preprocessed to obtain the target radar data, wherein the preprocessing includes noise reduction and signal amplification; Each object detected by the radar is assigned an identity identifier, and the target position of the object in the next radar frame is determined based on a target prediction algorithm; In the first fusion domain, the object position in the radar frame is matched with the object position in the camera frame in the first fusion trajectory; If a match is successful, the first fused trajectory is updated according to the parameter information corresponding to the object position in the radar frame to obtain the second fused trajectory. Updating the first fused trajectory includes updating at least one of the object's position, velocity, and orientation.

4. The target detection method based on camera and millimeter-wave radar information fusion according to claim 3, wherein the camera data includes lane lines, the lane lines are used to identify the positional relationship between the object and the lane lines, and the step of matching the object position in the radar frame with the object position in the camera frame of the first fused trajectory includes: Determine the first positional relationship between the object position in the radar frame and the lane line, and determine the second positional relationship between the object position in the camera frame in the first fused trajectory and the lane line; When the first positional relationship and the second positional relationship are the same, the object position in the radar frame is matched with the object position in the camera frame in the first fused trajectory based on the nearest neighbor algorithm.

5. The target detection method based on camera and millimeter-wave radar information fusion according to claim 1, wherein determining the target radar detection point in the target radar data includes: When there are multiple radar detection points corresponding to the target object detected by radar, the radar detection point corresponding to the minimum distance from the center of the second fused trajectory is selected as the target radar detection point.

6. The target detection method based on camera and millimeter-wave radar information fusion according to claim 1, further comprising: The number of matching times between the target camera data and the target radar data is determined based on the second fused trajectory; If the number of matching attempts exceeds a preset threshold, the target radar detection point is determined based on the target radar data.

7. The target detection method based on camera and millimeter-wave radar information fusion according to claim 1, further comprising: The updated second fused trajectory is labeled with the data source of the object, which includes at least a camera source, a millimeter-wave radar source, or a camera and millimeter-wave radar fusion source.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the target detection method based on the fusion of camera and millimeter-wave radar information according to any one of claims 1-7.

9. A vehicle, characterized in that, include: The memory, the processor, and the program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the target detection method based on camera and millimeter-wave radar information fusion according to any one of claims 1-7.

10. A target detection device that fuses camera and millimeter-wave radar information, characterized in that, The device includes: The acquisition module is used to acquire target camera data, target radar data, and vehicle driving data in the current scene; The generation module is used to generate a first fused trajectory based on the target camera data and the driving data; The first fusion module is used to fuse the target radar data and the first fusion trajectory in the first fusion domain to obtain the second fusion trajectory. The second fusion module is used to determine the target radar detection point based on the target radar data in the second fusion domain, update the second fused trajectory based on the target radar detection point, and output the updated second fused trajectory when the target radar data and the target camera data are successfully associated.

Images