A method and device for fusing vehicle speed-constrained vision with millimeter-wave targets

Abstract

This application relates to a method and apparatus for fusing visual and millimeter-wave target information based on vehicle speed constraints, belonging to the field of intelligent driving technology. The method includes the following steps: spatial calibration and alignment, and temporal synchronization processing, of vehicle camera-acquired information and millimeter-wave radar-acquired information; target perception based on the spatially calibrated and temporally synchronized vehicle camera-acquired information to obtain corresponding visual target information; target perception based on the spatially calibrated and temporally synchronized millimeter-wave radar-acquired information to obtain corresponding millimeter-wave target information; and target fusion based on the visual target information and millimeter-wave target information to obtain fused target information. This application uses data information acquired by vehicle cameras and millimeter-wave radar to perform fusion correction of target information, thereby achieving target recognition simply and efficiently, with good stability and reliability.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, specifically to a method and device for fusing vehicle speed-constrained vision with millimeter-wave targets. Background Technology

[0002] Environmental perception is one of the key technologies for developing intelligent autonomous driving. Currently, in the field of intelligent driving environmental perception technology, the most crucial task is how to detect surrounding environmental information in real time to ensure the safe driving of intelligent vehicles. However, there is still no consensus on how to configure different sensors to achieve the corresponding autonomous driving functions for different levels of autonomous driving. Currently, autonomous driving modules below Level 3 mainly adopt a fusion method of vision and millimeter-wave perception, involving primarily cameras and millimeter-wave radar sensors.

[0003] Currently, emerging domestic automakers primarily employ two technical approaches to the fusion of visual perception and millimeter-wave perception: ① Separately perform target perception on data output from cameras and millimeter-wave radars, then fuse them under spatiotemporal synchronization. This fusion method mainly utilizes data association and state estimation methods to improve the stability of the perceived target. ② Fusion is performed at the camera and millimeter-wave radar data points, and then the fused data is directly fed into a neural network for learning, ultimately outputting the perceived target. Both of these mainstream visual and millimeter-wave target fusion technologies have some drawbacks. For example, inaccurate target vehicle speed can interfere with the stability of autonomous driving functions (e.g., adaptive cruise control, following other vehicles).

[0004] To meet current usage requirements, a technology based on vehicle speed-constrained vision and millimeter-wave target fusion is provided. Summary of the Invention

[0005] This application provides a method and apparatus for fusing vehicle speed-constrained vision and millimeter-wave targets. Based on data information acquired by vehicle cameras and millimeter-wave radar, the target information is fused and corrected, thereby achieving target recognition simply and efficiently, with good stability and reliability.

[0006] To achieve the above objectives, this application provides the following solution.

[0007] In a first aspect, this application provides a method for fusing vehicle speed-constrained vision with millimeter-wave targets, the method comprising the following steps:

[0008] Spatial calibration and alignment, as well as temporal synchronization processing, are performed on information acquired by vehicle cameras and millimeter-wave radar.

[0009] Based on the vehicle camera information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding visual target information.

[0010] Based on the millimeter-wave radar information acquired after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding millimeter-wave target information.

[0011] Target fusion is performed based on the visual target information and the millimeter-wave target information to obtain fused target information.

[0012] Furthermore, based on the vehicle camera's acquired information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding visual target information, including the following steps:

[0013] The millimeter-wave radar point cloud data in the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing is converted to a preset custom image plane.

[0014] The vehicle camera information, after spatial calibration and alignment and time synchronization processing, is used for target perception to obtain 2D visual targets and convert them onto the custom image plane.

[0015] Based on the ground plane assumption theory and the camera's intrinsic and extrinsic parameters, the 2D visual target is processed to obtain the corresponding 3D visual target;

[0016] Based on the 3D visual target, a 2D top-view visual target is obtained according to the spatial relationship transformation;

[0017] Based on the distance between the 2D visual target on the custom image plane and the corresponding millimeter-wave radar point cloud data, the relative velocity of the 2D top-view visual target is constrained to obtain the corresponding visual target information.

[0018] Furthermore, based on a preset first formula, the relative velocity of the 2D top-view target is constrained, and the preset first formula is:

[0019] in,

[0020] Assuming the vehicle travels at a constant speed, its speed is V1 at time t1 and still V1 at time t2;

[0021] Within time segment [t1, t2], the visually perceived distances between the target ahead and the vehicle are [d1, d2] for each time interval.

[0022] At time t1, the visual perception estimates the velocity of the target object as V. 1观测 At time t2, the visual perception estimates the velocity of the target object as V. 2观测 .

[0023] Furthermore, based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding millimeter-wave target information, including the following steps:

[0024] Based on the information acquired by the millimeter-wave radar after spatial calibration alignment and time synchronization processing, millimeter-wave targets are obtained.

[0025] Based on the millimeter-wave target, a 2D top-view millimeter-wave target is obtained according to spatial relationship transformation;

[0026] Based on the 2D top-view millimeter-wave target, the corresponding visual target information is obtained.

[0027] Furthermore, the process of fusing the visual target information and the millimeter-wave target information to obtain fused target information includes the following steps:

[0028] Based on the visual target information and the millimeter-wave target information, target association matching will be performed to obtain the fused target;

[0029] Based on the visual target information and the millimeter-wave target information, the fusion target bounding box and the fusion target velocity are obtained, and the fusion target information is obtained by integration.

[0030] Secondly, this application provides a device for fusing vehicle speed-constrained vision with millimeter-wave targets, the device comprising:

[0031] The preprocessing module is used to perform spatial calibration and alignment, as well as time synchronization, on the information acquired by the vehicle camera and the millimeter-wave radar.

[0032] The visual target acquisition module is used to perform target perception and obtain corresponding visual target information based on the vehicle camera acquisition information after spatial calibration alignment and time synchronization processing.

[0033] The millimeter-wave target acquisition module is used to perform target perception and obtain corresponding millimeter-wave target information based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing.

[0034] The target fusion acquisition module is used to perform target fusion based on the visual target information and the millimeter-wave target information to obtain fused target information.

[0035] Furthermore, the visual target acquisition module is also used to convert the millimeter-wave radar point cloud data in the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing to a preset custom image plane.

[0036] The visual target acquisition module is also used to collect information from the vehicle camera after spatial calibration and alignment and time synchronization processing, perform target perception, obtain 2D visual targets, and convert them onto the custom image plane;

[0037] The visual target acquisition module is also used to process the 2D visual target based on the ground plane assumption theory and the camera's intrinsic and extrinsic parameters to obtain the corresponding 3D visual target;

[0038] The visual target acquisition module is also used to obtain a 2D top-view visual target based on the 3D visual target and according to spatial relationship transformation;

[0039] The visual target acquisition module is also used to constrain the relative velocity of the 2D top-view visual target based on the distance between the 2D visual target on the custom image plane and the corresponding millimeter-wave radar point cloud data, so as to obtain the corresponding visual target information.

[0040] Furthermore, the visual target acquisition module constrains the relative velocity of the 2D top-view target based on a preset first formula, which is:

[0041] in,

[0042] Assuming the vehicle travels at a constant speed, its speed is V1 at time t1 and still V1 at time t2;

[0043] Within time segment [t1, t2], the visually perceived distances between the target ahead and the vehicle are [d1, d2] for each time interval.

[0044] At time t1, the visual perception estimates the velocity of the target object as V. 1观测 At time t2, the visual perception estimates the velocity of the target object as V. 2观测 .

[0045] Furthermore, the millimeter-wave target acquisition module is also used to obtain millimeter-wave targets based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing;

[0046] The millimeter-wave target acquisition module is also used to obtain a 2D top-view millimeter-wave target based on the millimeter-wave target and according to spatial relationship transformation;

[0047] The millimeter-wave target acquisition module is also used to obtain the corresponding visual target information based on the 2D top-view millimeter-wave target.

[0048] Furthermore, the fusion target acquisition module is also used to perform target association matching based on the visual target information and the millimeter-wave target information to obtain the fusion target;

[0049] The fusion target acquisition module is also used to obtain the fusion target bounding box and fusion target velocity of the fusion target based on the visual target information and the millimeter-wave target information, and integrate them to obtain the obtained fusion target information.

[0050] The beneficial effects of the technical solution provided in this application include:

[0051] This application uses data acquired from vehicle cameras and millimeter-wave radar to fuse and correct target information, thereby achieving target recognition simply and efficiently, with good stability and reliability. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the steps of the vehicle speed-constrained vision and millimeter-wave target fusion method provided in the embodiments of this application;

[0054] Figure 2 This is an overall fusion flowchart of the vehicle speed-constrained vision and millimeter-wave target fusion method provided in the embodiments of this application;

[0055] Figure 3 This is a schematic diagram of vehicle speed constraint in the vehicle speed constraint-based vision and millimeter-wave target fusion method provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the fusion output of different distance constraints in the vehicle speed-constrained vision and millimeter-wave target fusion method provided in the embodiments of this application;

[0057] Figure 5 This is a structural block diagram of the vehicle speed-constrained vision and millimeter-wave target fusion device provided in the embodiments of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0060] This application provides a method and apparatus for fusing vehicle speed-constrained vision and millimeter-wave targets. Based on data information acquired by vehicle cameras and millimeter-wave radar, the target information is fused and corrected, thereby achieving target recognition simply and efficiently, with good stability and reliability.

[0061] To achieve the aforementioned technical effects, the overall concept of this application is as follows:

[0062] A method for fusing vehicle speed-constrained vision with millimeter-wave targets includes the following steps:

[0063] S1. Perform spatial calibration and alignment, as well as time synchronization, on the information collected by the vehicle camera and the millimeter-wave radar.

[0064] S2. Based on the vehicle camera information collected after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding visual target information.

[0065] S3. Based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding millimeter-wave target information.

[0066] S4. Target fusion is performed based on visual target information and millimeter-wave target information to obtain fused target information.

[0067] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0068] See Figures 1-4 As shown in the figure, this application provides a method for fusing vehicle speed-constrained vision with millimeter-wave targets, which includes the following steps:

[0069] S1. Perform spatial calibration and alignment, as well as time synchronization, on the information collected by the vehicle camera and the millimeter-wave radar.

[0070] S2. Based on the vehicle camera information collected after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding visual target information.

[0071] S3. Based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding millimeter-wave target information.

[0072] S4. Target fusion is performed based on visual target information and millimeter-wave target information to obtain fused target information.

[0073] It should be noted that in existing target recognition technologies:

[0074] The fusion of visual and millimeter-wave raw data through an end-to-end neural network learning approach is highly dependent on spatiotemporal synchronization and computing power, and also requires high stability of the output of each sensor, making it unsuitable for commercial use at this stage.

[0075] The method relies on both vision and millimeter-wave sensors to first perceive the target, and then perform a fusion process. Currently, the main method used is fusion based on target distance and target bounding box. However, this method is prone to fusion failures and inaccurate fused distances, leading to inaccurate fused target attributes.

[0076] In this embodiment, target information is fused and corrected based on data acquired by vehicle camera and millimeter-wave radar, thereby achieving target recognition simply and efficiently with good stability and reliability.

[0077] The technical solution of this application embodiment still adopts the fusion method of mainstream vision as the main approach and millimeter wave as the auxiliary approach;

[0078] However, the fusion of vision and millimeter-wave targets is improved and optimized in the following aspects:

[0079] First, to address the issue of millimeter-wave radar's inaccuracy in detecting stationary targets, the false detection rate of stationary targets is reduced by using the combined constraint of the millimeter-wave target distance and the vehicle's speed.

[0080] Second, improve the ranging performance of visually perceived targets by tightly coupling the original data projection fusion strategy;

[0081] Third, a multi-constraint target association strategy is employed to improve the accuracy of vision-millimeter wave fusion. Simultaneously, vehicle speed constraint association criteria at different distance levels are set to enhance the accuracy of fused targets while also improving comfort and safety.

[0082] Fourth, constraining the association of segments at different distances improves the efficiency of the association algorithm and reduces noise interference, thereby improving the accuracy of the fused target output.

[0083] Furthermore, based on the vehicle camera's acquired information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding visual target information, including the following steps:

[0084] The millimeter-wave radar point cloud data in the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing is converted to a preset custom image plane.

[0085] The vehicle camera information, after spatial calibration and alignment and time synchronization processing, is used for target perception to obtain 2D visual targets and convert them onto the custom image plane.

[0086] Based on the ground plane assumption theory and the camera's intrinsic and extrinsic parameters, the 2D visual target is processed to obtain the corresponding 3D visual target;

[0087] Based on the 3D visual target, a 2D top-view visual target is obtained according to the spatial relationship transformation;

[0088] Based on the distance between the 2D visual target on the custom image plane and the corresponding millimeter-wave radar point cloud data, the relative velocity of the 2D top-view visual target is constrained to obtain the corresponding visual target information.

[0089] Furthermore, based on a preset first formula, the relative velocity of the 2D top-view target is constrained, and the preset first formula is:

[0090] in,

[0091] Assuming the vehicle travels at a constant speed, its speed is V1 at time t1 and still V1 at time t2;

[0092] Within time segment [t1, t2], the visually perceived distances between the target ahead and the vehicle are [d1, d2] for each time interval.

[0093] At time t1, the visual perception estimates the velocity of the target object as V. 1观测 At time t2, the visual perception estimates the velocity of the target object as V. 2观测 .

[0094] Furthermore, based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding millimeter-wave target information, including the following steps:

[0095] Based on the information acquired by the millimeter-wave radar after spatial calibration alignment and time synchronization processing, millimeter-wave targets are obtained.

[0096] Based on the millimeter-wave target, a 2D top-view millimeter-wave target is obtained according to spatial relationship transformation;

[0097] Based on the 2D top-view millimeter-wave target, the corresponding visual target information is obtained.

[0098] Furthermore, the process of fusing the visual target information and the millimeter-wave target information to obtain fused target information includes the following steps:

[0099] Based on the visual target information and the millimeter-wave target information, target association matching will be performed to obtain the fused target;

[0100] Based on the visual target information and the millimeter-wave target information, the fusion target bounding box and the fusion target velocity are obtained, and the fusion target information is obtained by integration.

[0101] Based on the technical solutions of the embodiments of this application,

[0102] Step 1: The cameras and millimeter-wave radar sensors of the autonomous vehicle are first spatially calibrated and aligned to ensure the consistency of spatial mapping between the sensors;

[0103] Subsequently, target perception was performed on both the camera and the millimeter-wave radar.

[0104] Step 2: After confirming spatial alignment and temporal synchronization between the millimeter-wave radar and the camera, perform visual target perception on the camera, as shown in the attached diagram in the instruction manual. Figure 2 The tightly coupled phase flow shown mainly includes the following related operations:

[0105] Step 2.1: After acquiring the current millimeter-wave radar point cloud data and image data, the point cloud data is projected onto the image plane through spatial alignment.

[0106] Step 2.2: Perceive the target content in the image using deep learning methods and output the 2D target; this step uses the original image data, and the projected point cloud data is used for subsequent distance constraints.

[0107] Step 2.3: After obtaining the 2D target of the current image, the 3D target in the camera coordinate system is calculated and output using the ground plane assumption theory and camera intrinsic and extrinsic parameters. The mainstream algorithms used are IPM and its related improvements.

[0108] Step 2.4: After the 3D target is geometrically calculated from the current image, the millimeter-wave radar point cloud (hereinafter referred to as point cloud) is projected onto the image plane;

[0109] By counting the number of points within the current 2D bounding box, the average distance of the points within the 2D bounding box is finally calculated.

[0110] The calculated average value is used as a constraint value for the visual 3D target distance estimation;

[0111] If the average distance of the correlated point cloud is within a certain error range compared to the distance estimated by visual 3D, then the alignment is smoothed.

[0112] If it exceeds a certain range, the visual 3D recovery distance will be used as the primary factor.

[0113] Step 2.5: After step 2.4 above, based on the spatial relationship transformation, the visual 3D target is transformed to the vehicle body coordinate system, so the 2D top view target can be directly obtained;

[0114] That is, the 3D target is transformed to the vehicle body coordinate system through extrinsic parameters, and then the top-down view is saved. Figure 2 D. Related information.

[0115] Step 2.6: After obtaining the top view target in the visual 2D vehicle coordinate system, the state of the visual 2D target is estimated. At the same time, the relative speed of the target is dynamically constrained according to the speed of the autonomous vehicle and the relative distance, and more accurate smoothing is performed.

[0116] Step 2.7: Apply vehicle speed constraints and smooth the visual target state estimation:

[0117] As shown in the attached diagram of the instruction manual. Figure 3 As shown, assuming the car is traveling at a constant speed, its speed is v1 at time t1 and v1 at time t2. Within the time segment [t1, t2], the visually perceived distances of the target ahead from the car are [d1, d2] for each time interval.

[0118] So, assuming the visual perception estimates the target's velocity as v1 at time t1, and the predicted velocity at time t2 is v... 2预测 The speed of the target vehicle can be calculated as follows:

[0119]

[0120] via v 2观测 To conduct a test on v 2预测 State estimation constrains velocity correction, improving the accuracy of visual perception of target object velocity.

[0121] Step 3: The millimeter-wave radar outputs point cloud data, and then the target is perceived in a 2D top view by clustering and identifying the point cloud.

[0122] Similarly, by estimating the state of the vehicle based on its speed and the 2D target output from the millimeter-wave sensor, dynamic constraints are applied within a short time interval to make the millimeter-wave target output smoother and reduce the false detection rate.

[0123] Step 3.1: Apply vehicle speed constraints and smooth the millimeter-wave target state estimation;

[0124] As shown in the attached diagram of the instruction manual. Figure 3 As shown, the speed constraint smoothing of millimeter-wave targets is consistent with the smoothing strategy for visual targets in the state estimation stage. For details, please refer to the speed constraint smoothing steps for visual targets.

[0125] Step 4: After obtaining the visual 2D target and the millimeter-wave 2D target after state estimation, the next step is to perform a multi-constraint target association and filtering algorithm;

[0126] Its core mainly consists of the following points: dynamic constraints based on vehicle speed, distance variation, target speed, and IoU correlation, in order to improve the stability of the fused target and the accuracy of target attributes.

[0127] Step 4.1: In the above steps 2 and 3, the target state is initially improved by constraining the vehicle speed. Next, multi-constraint target association and matching are performed on the visual target and the millimeter-wave target.

[0128] The intersection-over-union ratio (IoU) is calculated using distance correlation and IoU to determine the targets that have been successfully correlated with the visual and millimeter-wave targets;

[0129] Here, distance variation mainly refers to improving the accuracy of distance variation by constraining the vehicle's speed, and further improving the accuracy and confidence of target association by associating under IoU.

[0130] Step 4.2: For successfully associated targets, the weighted average calculation of the bounding boxes of the fused targets is based primarily on visual targets, while the velocity of the fused targets is based primarily on millimeter-wave targets;

[0131] When the difference between the size of the visual target box and the size of the millimeter-wave target box is within a certain threshold range, then optimization is achieved by weighted averaging.

[0132] If the visual target and the millimeter-wave target box are outside a certain range, then the visual target should be used as the primary reference.

[0133] When the difference between the velocity of the visual target and the velocity of the millimeter-wave target is within a certain range, optimization is achieved through weighted averaging.

[0134] If the speed of visual and millimeter-wave targets exceeds a certain range, then the speed of millimeter-wave targets will take precedence.

[0135] Step 4.3: After associating the visual target and the millimeter-wave target data, the following conclusions are reached: successfully associated target, unassociated visual target, and unassociated millimeter-wave target.

[0136] As shown in the attached diagram of the instruction manual. Figure 2 As shown, targets that are successfully associated are directly added to the output list; targets that are not successfully associated need to be checked to see if they meet the constraints before they can be added to the output list.

[0137] Therefore, for targets that have not been successfully associated, the current approach is to further constrain the judgment based on different distances:

[0138] Currently, the criteria for dividing and merging distances are based on different distance methods. The distance is divided into four distance segments: [0,50], [50,100], [100,150], and [150,+]. The unit is meters.

[0139] The merged output list is a subset of the trace list. Figure 1 The criteria for determining constraints in the process are described in the following steps:

[0140] First, the visually perceived target and the millimeter-wave target are divided into their respective intervals according to the aforementioned distance ranges. Then, a multi-constraint data association matching and filtering algorithm is performed on each interval.

[0141] Second, for visual targets and millimeter-wave targets in the [0, 50] interval, multiple constraint associations are performed, and the association results are as follows:

[0142] Successfully associated targets are directly added to the fusion output list.

[0143] If a visual target that has not been successfully associated has been tracked for more than 2 consecutive frames, it will be added to the fusion output list; otherwise, it will remain in the tracking list and not be output.

[0144] If a millimeter-wave target that has not been successfully associated has been tracked for more than 5 consecutive frames, it will be added to the fusion list; otherwise, it will be considered a false detection and deleted.

[0145] Third, for visual targets and millimeter-wave targets in the [50, 100] interval, multiple constraint associations are performed. Similarly, the association results are as follows:

[0146] Targets that are successfully associated are directly added to the fusion output list.

[0147] If a visual target that has not been successfully associated has been tracked for more than 3 consecutive frames, it will be added to the fusion output list; otherwise, it will remain in the tracking list and not be output.

[0148] If a millimeter-wave target that has not been successfully associated has been tracked for more than 8 consecutive frames, it will be added to the fusion output list; if it has been tracked for more than 5 consecutive frames, it will be added to the tracking list.

[0149] Fourth, for visual targets and millimeter-wave targets within the range [100, 150], multi-constraint association is performed. Similarly, the association results are as follows:

[0150] Targets that are successfully associated are directly added to the fusion output list.

[0151] If a visual target that has not been successfully associated has been tracked for more than 5 consecutive frames, it will be added to the fusion output list; otherwise, it will remain in the tracking list and not be output.

[0152] If a millimeter-wave target that has not been successfully associated has been tracked for more than 5 consecutive frames, it will be added to the fusion output list; if it has been tracked for more than 3 consecutive frames, it will be added to the tracking list.

[0153] Fifth, for visual targets and millimeter-wave targets within the [150, +] interval, multi-constraint correlation is performed. Similarly, the correlation results are as follows:

[0154] Targets that are successfully associated are directly added to the fusion list.

[0155] Visual targets that have not been successfully associated are added to the fusion output list if they have been tracked for more than 3 consecutive frames; otherwise, they are deleted.

[0156] If a millimeter-wave target that has not been successfully associated has been tracked for more than 3 consecutive frames, it will be added directly to the fusion output list; otherwise, it will be deleted.

[0157] Sixth, if a target in the tracking list loses more than 3 frames, it will be automatically removed from the tracking list;

[0158] For targets that have been successfully associated, remove the successfully associated visual and millimeter-wave targets from their tracking list.

[0159] Seventh, for targets in the tracking list, incorporate association and state estimation optimization in the next multi-constraint association process;

[0160] This process is repeated iteratively to improve the stability of target fusion.

[0161] The embodiments of this application have the following advantages:

[0162] First, a method of tightly coupling raw data is adopted to improve the stability and accuracy of visually perceived target distance;

[0163] Second, the vehicle speed is used to constrain visual and millimeter-wave targets respectively, thereby improving the stability of the initial state estimation of visual and millimeter-wave targets.

[0164] Third, in the fusion stage, a multi-constraint association and filtering method for vision and millimeter-wave targets is adopted under different distance segments. By using vehicle speed constraints to filter stationary millimeter-wave targets within a dynamic time frame, false detections are reduced. At the same time, the weighting of multi-constraint conditions for fused targets is improved to enhance target stability and the accuracy of target attributes; data association operations are performed on different distance segments to improve algorithm efficiency while reducing false detection rates and noise interference.

[0165] See Figure 5 As shown, based on the same inventive concept as the method embodiment, this application provides a device for fusing vehicle speed-constrained vision and millimeter-wave targets, the device comprising:

[0166] The preprocessing module is used to perform spatial calibration and alignment, as well as time synchronization, on the information acquired by the vehicle camera and the millimeter-wave radar.

[0167] The visual target acquisition module is used to perform target perception and obtain corresponding visual target information based on the vehicle camera acquisition information after spatial calibration alignment and time synchronization processing.

[0168] The millimeter-wave target acquisition module is used to perform target perception and obtain corresponding millimeter-wave target information based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing.

[0169] The target fusion acquisition module is used to perform target fusion based on the visual target information and the millimeter-wave target information to obtain fused target information.

[0170] In this embodiment, target information is fused and corrected based on data acquired by vehicle camera and millimeter-wave radar, thereby achieving target recognition simply and efficiently with good stability and reliability.

[0171] The technical solution of this application embodiment still adopts the fusion method of mainstream vision as the main approach and millimeter wave as the auxiliary approach;

[0172] However, the fusion of vision and millimeter-wave targets is improved and optimized in the following aspects:

[0173] First, to address the issue of millimeter-wave radar's inaccuracy in detecting stationary targets, the false detection rate of stationary targets is reduced by using the combined constraint of the millimeter-wave target distance and the vehicle's speed.

[0174] Second, improve the ranging performance of visually perceived targets by tightly coupling the original data projection fusion strategy;

[0175] Third, a multi-constraint target association strategy is employed to improve the accuracy of vision-millimeter wave fusion. Simultaneously, vehicle speed constraint association criteria at different distance levels are set to enhance the accuracy of fused targets while also improving comfort and safety.

[0176] Fourth, constraining the association of segments at different distances improves the efficiency of the association algorithm and reduces noise interference, thereby improving the accuracy of the fused target output.

[0177] Furthermore, the visual target acquisition module is also used to convert the millimeter-wave radar point cloud data in the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing to a preset custom image plane.

[0178] The visual target acquisition module is also used to collect information from the vehicle camera after spatial calibration and alignment and time synchronization processing, perform target perception, obtain 2D visual targets, and convert them onto the custom image plane;

[0179] The visual target acquisition module is also used to process the 2D visual target based on the ground plane assumption theory and the camera's intrinsic and extrinsic parameters to obtain the corresponding 3D visual target;

[0180] The visual target acquisition module is also used to obtain a 2D top-view visual target based on the 3D visual target and according to spatial relationship transformation;

[0181] The visual target acquisition module is also used to constrain the relative velocity of the 2D top-view visual target based on the distance between the 2D visual target on the custom image plane and the corresponding millimeter-wave radar point cloud data, so as to obtain the corresponding visual target information.

[0182] Furthermore, the visual target acquisition module constrains the relative velocity of the 2D top-view target based on a preset first formula, which is:

[0183] in,

[0184] Assuming the vehicle travels at a constant speed, its speed is V1 at time t1 and still V1 at time t2;

[0185] Within time segment [t1, t2], the visually perceived distances between the target ahead and the vehicle are [d1, d2] for each time interval.

[0186] At time t1, the visual perception estimates the velocity of the target object as V. 1观测 At time t2, the visual perception estimates the velocity of the target object as V. 2观测 .

[0187] Furthermore, the millimeter-wave target acquisition module is also used to obtain millimeter-wave targets based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing;

[0188] The millimeter-wave target acquisition module is also used to obtain a 2D top-view millimeter-wave target based on the millimeter-wave target and according to spatial relationship transformation;

[0189] The millimeter-wave target acquisition module is also used to obtain the corresponding visual target information based on the 2D top-view millimeter-wave target.

[0190] Furthermore, the fusion target acquisition module is also used to perform target association matching based on the visual target information and the millimeter-wave target information to obtain the fusion target;

[0191] The fusion target acquisition module is further configured to obtain the fusion target bounding box and fusion target velocity based on the visual target information and the millimeter-wave target information, and integrate them to obtain the obtained fusion target information.

[0192] The embodiments of this application have the following advantages:

[0193] First, a method of tightly coupling raw data is adopted to improve the stability and accuracy of visually perceived target distance;

[0194] Second, the vehicle speed is used to constrain visual and millimeter-wave targets respectively, thereby improving the stability of the initial state estimation of visual and millimeter-wave targets.

[0195] Third, in the fusion stage, a multi-constraint association and filtering method for vision and millimeter-wave targets is adopted under different distance segments. By using vehicle speed constraints to filter stationary millimeter-wave targets within a dynamic time frame, false detections are reduced. At the same time, the weighting of multi-constraint conditions for fused targets is improved to enhance target stability and the accuracy of target attributes; data association operations are performed on different distance segments to improve algorithm efficiency while reducing false detection rates and noise interference.

[0196] It should be noted that the technical problems, technical means and technical effects of the vehicle speed constrained vision and millimeter wave target fusion device provided in this application are similar to those of the vehicle speed constrained vision and millimeter wave target fusion method in principle.

[0197] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0198] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for fusing vehicle speed-constrained vision with millimeter-wave targets, characterized in that, The method includes the following steps: Spatial calibration and alignment, as well as temporal synchronization, are performed on information acquired by vehicle cameras and millimeter-wave radar. Based on the vehicle camera information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding visual target information. Based on the millimeter-wave radar information acquired after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding millimeter-wave target information. Target fusion is performed based on the visual target information and the millimeter-wave target information to obtain fused target information; Based on the vehicle camera information acquired after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding visual target information, including the following steps: The millimeter-wave radar point cloud data in the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing is converted to a preset custom image plane. The vehicle camera information, after spatial calibration and alignment and time synchronization processing, is used for target perception to obtain 2D visual targets and convert them onto the custom image plane. Based on the ground plane assumption theory and the camera's intrinsic and extrinsic parameters, the 2D visual target is processed to obtain the corresponding 3D visual target; Based on the 3D visual target, a 2D top-view visual target is obtained according to the spatial relationship transformation; Based on the distance between the 2D visual target on the custom image plane and the corresponding millimeter-wave radar point cloud data, the relative velocity of the 2D top-view visual target is constrained to obtain the corresponding visual target information; The relative velocity of the 2D top-view target is constrained based on a preset first formula, which is: ;in, Assuming the car is traveling at a constant speed, in The speed at time is ,exist The vehicle speed remains constant. ; In time segment [ , The distances between the vehicle and the target ahead perceived by the internal visual system at different times are respectively [ , ]; At any moment When visual perception estimates the velocity of the target object, At any moment When visual perception estimates the velocity of the target object, .

2. The method for fusing vehicle speed-constrained vision and millimeter-wave targets as described in claim 1, characterized in that, Based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing, target perception is performed to obtain the corresponding millimeter-wave target information, including the following steps: Based on the information acquired by the millimeter-wave radar after spatial calibration alignment and time synchronization processing, millimeter-wave targets are obtained. Based on the millimeter-wave target, a 2D top-view millimeter-wave target is obtained according to spatial relationship transformation; Based on the 2D top-view millimeter-wave target, the corresponding visual target information is obtained.

3. The method for fusing vehicle speed-constrained vision and millimeter-wave targets as described in claim 1, characterized in that, The process of fusing the visual target information and the millimeter-wave target information to obtain fused target information includes the following steps: Target association matching is performed based on the visual target information and the millimeter-wave target information to obtain the fused target; Based on the visual target information and the millimeter-wave target information, the fusion target bounding box and the fusion target velocity are obtained, and the fusion target information is obtained by integration.

4. A device for fusing vehicle speed-constrained vision with millimeter-wave targets, characterized in that, The device includes: The preprocessing module is used to perform spatial calibration and alignment, as well as time synchronization, on the information acquired by the vehicle camera and the millimeter-wave radar. The visual target acquisition module is used to perform target perception and obtain corresponding visual target information based on the vehicle camera acquisition information after spatial calibration alignment and time synchronization processing. The millimeter-wave target acquisition module is used to perform target perception and obtain corresponding millimeter-wave target information based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing. The target acquisition module is used to perform target fusion based on the visual target information and the millimeter-wave target information to obtain fused target information. The visual target acquisition module is also used to convert the millimeter-wave radar point cloud data in the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing to a preset custom image plane. The visual target acquisition module is also used to collect information from the vehicle camera after spatial calibration and alignment and time synchronization processing, perform target perception, obtain 2D visual targets, and convert them onto the custom image plane. The visual target acquisition module is also used to process the 2D visual target based on the ground plane assumption theory and the camera's intrinsic and extrinsic parameters to obtain the corresponding 3D visual target; The visual target acquisition module is also used to obtain a 2D top-view visual target based on the 3D visual target and according to spatial relationship transformation; The visual target acquisition module is also used to constrain the relative velocity of the 2D top-view visual target based on the distance between the 2D visual target on the custom image plane and the corresponding millimeter-wave radar point cloud data, so as to obtain the corresponding visual target information. The visual target acquisition module constrains the relative velocity of the 2D top-view target based on a preset first formula, which is: ;in, Assuming the car is traveling at a constant speed, in The speed at time is ,exist The vehicle speed remains constant. ; In time segment [ , The distances between the vehicle and the target ahead perceived by the internal visual system at different times are respectively [ , ]; At any moment When visual perception estimates the velocity of the target object, At any moment When visual perception estimates the velocity of the target object, .

5. The device for fusing vehicle speed-constrained vision and millimeter-wave target as described in claim 4, characterized in that: The millimeter-wave target acquisition module is also used to obtain millimeter-wave targets based on the millimeter-wave radar acquisition information after spatial calibration alignment and time synchronization processing. The millimeter-wave target acquisition module is also used to obtain a 2D top-view millimeter-wave target based on the millimeter-wave target and according to spatial relationship transformation; The millimeter-wave target acquisition module is also used to obtain the corresponding visual target information based on the 2D top-view millimeter-wave target.

6. The device for fusing vehicle speed-constrained vision and millimeter-wave targets as described in claim 4, characterized in that: The fusion target acquisition module is also used to perform target association matching based on the visual target information and the millimeter-wave target information to obtain the fusion target; The fusion target acquisition module is also used to obtain the fusion target bounding box and fusion target velocity of the fusion target based on the visual target information and the millimeter-wave target information, and integrate them to obtain the obtained fusion target information.

Images