A millimeter wave radar-based visual perception data system deviation detection and fusion compensation method, system and device

Abstract

The application relates to the technical field of vehicle environment sensing, and discloses a vision sensing data system deviation detection and fusion compensation method, system and device based on a millimeter wave radar, which has the technical scheme as follows: continuously acquiring vision sensing data, millimeter wave radar data and a host vehicle motion state; when the host vehicle motion state reaches a preset deviation detection condition, the overall spatial relationship between a vision sensing scene and a millimeter wave radar environment scene is analyzed, whether the vision sensing scene has a unified systematic deviation is detected, if the vision sensing scene has the unified systematic deviation, candidate unified compensation parameters are calculated; the candidate unified compensation parameters are tried to be applied, when the overall error of the compensated scene is smaller than that of the scene before compensation, and the candidate unified compensation parameters are located in a preset allowable range, the vision sensing data is compensated according to the unified compensation parameters; and the compensated vision sensing data is associated and fused with the millimeter wave radar data.

Description

Technical Field

[0001] This invention relates to the field of vehicle environmental perception technology, and more specifically, to a method, system, and apparatus for deviation detection and fusion compensation of visual perception data systems based on millimeter-wave radar. Background Technology

[0002] With the development of vehicle-assisted driving and environmental perception technologies, vehicles typically employ visual sensors and millimeter-wave radar to acquire information about surrounding targets and the road environment. The visual perception module can output information such as target category, target location, target speed, target direction, lane line position, and the relationship between the target and lane lines; millimeter-wave radar can output information such as target distance, orientation, relative speed, and environmental reflection points. By correlating and fusing data from different sensors, the completeness and stability of the vehicle's environmental perception results can be improved.

[0003] In some vehicle perception systems, the visual perception module and the subsequent data fusion module are provided by different devices or suppliers. The data fusion module can only obtain the structured data output by the visual perception module through a preset communication interface. It cannot access the image processing procedures, camera calibration parameters, coordinate transformation parameters, and related perception algorithms inside the visual perception module, nor does it have an interface to write the corrected parameters back to the visual perception module.

[0004] After a period of vehicle use, the camera may experience an angle shift due to vehicle vibration, external impact, loose mounting brackets, or changes in the mounting structure. When the camera mounting parameters used by the vision perception module are inconsistent with the actual camera mounting state, the data output by the vision perception module, such as target position, target speed direction, target heading, lane line position, and the relationship between the target and the lane line, may simultaneously exhibit a consistent and systematic deviation.

[0005] This deviation is not limited to a single target, but may manifest as an abnormal rotation, translation, or scale relationship of the entire visual perception scene in the vehicle coordinate system or fused coordinate system. This results in a continuous mismatch between the visual perception scene and the environmental scene established by the millimeter-wave radar, affecting subsequent target association and multi-sensor fusion results.

[0006] For systems capable of modifying the internal parameters of the visual perception module, this can be achieved by recalibrating the camera and updating the internal parameters of the visual module. However, for independent visual perception modules that can only obtain structured visual output data, subsequent fusion modules cannot directly perform internal calibration and parameter updates.

[0007] Furthermore, using millimeter-wave radar data to replace or correct the position, distance, or velocity of visual targets one by one can only handle individual targets with established correspondences. It cannot resolve the systematic biases that exist in the entire visual output coordinate system, nor can it simultaneously correct the target, lane lines, road markings, and related spatial relationship information.

[0008] Therefore, a data processing method is needed for vehicle perception systems where the internal parameters of the visual perception module cannot be accessed or modified. Summary of the Invention

[0009] The purpose of this invention is to provide a method, system, and device for detecting and compensating deviations in visual perception data systems based on millimeter-wave radar. This method can identify uniform systematic deviations in visual perception output at the overall scene level and perform uniform compensation on visual perception data before data fusion without modifying the internal parameters of the visual perception module.

[0010] The above-mentioned technical objective of the present invention is achieved through the following technical solution: a method for deviation detection and fusion compensation of a visual perception data system based on millimeter-wave radar, comprising the following steps: S1. Continuously acquire visual perception data, millimeter-wave radar data, and the vehicle's motion status; S2. When the vehicle's motion reaches the preset deviation detection conditions, analyze the overall spatial relationship between the visual perception scene constructed from visual perception data and the millimeter-wave radar environment scene constructed from millimeter-wave radar environment data to detect whether there is a unified systematic deviation in the visual perception scene. S3. If a unified systematic deviation is detected in the visual perception scene, then candidate unified compensation parameters for the visual perception data are calculated based on the overall spatial difference between the visual perception scene and the millimeter-wave radar environment scene and the driving direction of the main vehicle. S4. Apply the candidate unified compensation parameters to the current visual perception scene and calculate the overall scene error before and after the trial compensation. When the overall scene error after the trial compensation is less than the overall scene error before the trial compensation, and the candidate unified compensation parameters are within the preset allowable range, the candidate unified compensation parameters are used as the formal unified compensation parameters. S5. Based on the formal unified compensation parameters, perform unified compensation on the visual perception data to obtain the compensated visual perception data. S6. The compensated visual perception data and millimeter-wave radar data are correlated and fused as independent information sources.

[0011] As a preferred embodiment of the present invention, the preset deviation detection condition is: the main vehicle is in a straight-line state.

[0012] As a preferred embodiment of the present invention, the analysis process of the overall spatial relationship between the visual perception scene and the millimeter-wave radar environment scene includes: Based on the location distribution, motion distribution, directional features and multi-frame change trends of the two types of scenes within a preset time range, the overall differences between the visual perception scene and the millimeter-wave radar environment scene are statistically analyzed. If, at least three consecutive moments, the visually perceived scene exhibits a consistent overall rotation, translation, or scale anomaly relative to the millimeter-wave radar environment, it is determined that there is a unified systematic bias in the visually perceived data.

[0013] As a preferred technical solution of the present invention, the visual perception data is also used to assist in judging the type of deviation. When multiple frames of lane lines or road markings cannot maintain the expected parallel or scale relationship in the bird's-eye view coordinate system, it is judged that the visual perception data has a pitch direction deviation; when the overall extension direction of lane lines, road markings or visual target groups is inconsistent with the straight-line direction of the main vehicle, it is judged that the visual perception data has a yaw direction deviation.

[0014] As a preferred technical solution of the present invention, the process of calculating candidate unified compensation parameters includes: determining statistical association weights based on at least one of the following: positional proximity, velocity similarity, motion direction similarity, multi-frame change trend, and stability between visual targets or visual environment features and millimeter-wave radar targets or radar environment features; wherein, a visual target is allowed to form statistical associations with multiple millimeter-wave radar targets or radar environment features with different weights, and some visual targets, millimeter-wave radar targets, or environmental features are allowed not to participate in the determination of the candidate unified compensation parameters.

[0015] As a preferred embodiment of the present invention, the unified compensation parameters include at least one of the following: yaw angle compensation, pitch angle compensation, lateral or longitudinal position offset, scale compensation, velocity direction compensation corresponding to spatial compensation, and optional velocity scale compensation.

[0016] As a preferred technical solution of the present invention, the unified compensation for visual perception data is as follows: according to the same set of unified compensation parameters, corresponding transformations are performed on different types of visual perception data.

[0017] A deviation detection and fusion compensation system for visual perception data systems based on millimeter-wave radar, comprising: The visual perception module is used to output visual perception data. Millimeter-wave radar data processing module, used to output millimeter-wave radar data; The main vehicle motion status acquisition module is used to output the main vehicle motion status. The visual perception data deviation detection module is used to acquire visual perception data, millimeter-wave radar data, and the vehicle's motion state. It constructs a visual perception scene based on the visual perception data and a millimeter-wave radar environment scene based on the millimeter-wave radar data. When the vehicle's motion state reaches a preset deviation detection condition, it analyzes the overall spatial relationship between the visual perception scene and the millimeter-wave radar environment scene to detect whether there is a unified systematic deviation in the visual perception scene. If a deviation is determined, it calculates candidate unified compensation parameters based on the overall spatial difference between the visual perception scene and the millimeter-wave radar environment scene and the vehicle's driving direction. The candidate unified compensation parameters are then applied to the current visual perception scene, and the overall scene error before and after the trial compensation is calculated. When the overall scene error after the trial compensation is less than the overall scene error before the trial compensation, and the candidate unified compensation parameters are within a preset allowable range, the candidate unified compensation parameters are used as the official unified compensation parameters. The unified compensation module is used to perform unified compensation on visual perception data according to the formal unified compensation parameters to obtain compensated visual perception data. The multi-sensor fusion module is used to associate and fuse compensated visual perception data and millimeter-wave radar data as independent information sources.

[0018] As a preferred embodiment of the present invention, the visual perception module outputs structured visual perception data through a preset data interface. The processing device executing the method does not access or modify the image processing process, camera calibration parameters, coordinate transformation parameters and perception algorithm inside the visual perception module, and does not write the unified compensation parameters to the visual perception module.

[0019] A deviation detection and fusion compensation device for a visual perception data system based on millimeter-wave radar includes: a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor implements the above-described method when executing the computer program.

[0020] In summary, the present invention has the following beneficial effects: The present invention is applicable to situations where the calibration parameters, coordinate transformation parameters and perception algorithms inside the visual perception module cannot be accessed or modified. Without disassembling, reconfiguring or writing back the parameters of the visual perception module, the deviation compensation of visual perception data can be completed on the fusion side.

[0021] This invention identifies systematic deviations at the overall level of the visual scene and uses unified compensation parameters to simultaneously correct target position, target speed direction, target heading, lane lines, road markings and other spatial relationship information, avoiding the need to correct only a few matched targets one by one.

[0022] This invention does not use millimeter-wave radar data to directly replace visual measurements. Instead, it performs a unified transformation on the visual perception data before data fusion, so that the compensated visual data and millimeter-wave radar data still participate in subsequent association and fusion as different information sources, preserving the complementary relationship of multi-sensor information.

[0023] This invention determines compensation parameters by considering overall differences at the scene level, without requiring all visual targets to establish an accurate one-to-one correspondence with millimeter-wave radar targets. Therefore, even when there is a significant overall misalignment in the visual scene, it can still identify and compensate for the systematic deviation.

[0024] This invention can improve the spatial consistency between the visual perception scene and the millimeter-wave radar environment scene, and reduce problems such as target association failure, lane relationship error and unstable fusion results caused by changes in camera installation status. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the visual scene unification system deviation and compensation of the present invention; Figure 3 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0026] It is readily understood that, based on the technical solution of this invention, various embodiments of the invention can be conceived by those skilled in the art without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention. Rather, these embodiments are provided to enable those skilled in the art to gain a more thorough understanding of the invention. Preferred embodiments of the invention are described below in conjunction with the accompanying drawings, which form part of this application and, together with the embodiments of the invention, serve to illustrate the innovative concept of the invention.

[0027] like Figures 1-3 As shown, the present invention provides a method, system and device for deviation detection and fusion compensation of visual perception data system based on millimeter-wave radar. It is used to detect systematic deviations in visual perception data before fusing millimeter-wave radar data and visual perception data, and to uniformly compensate for target information, lane line information and related motion information output by the visual perception module.

[0028] The system provided by this invention includes a visual perception module, a millimeter-wave radar data processing module, a vehicle motion state acquisition module, a visual perception data deviation detection module, a unified compensation module, and a multi-sensor fusion module.

[0029] The method of the present invention, such as Figure 1 and Figure 3 As shown, the following steps are performed through the system module: S1, Data Acquisition.

[0030] The vehicle-mounted visual perception module outputs structured visual perception data to this system through a data interface; Visual perception data includes target information and road environment information. Target information includes at least the target location, as well as the target speed, target direction, target category, and target tracking markers. Road environment information includes at least the lane line spatial coordinates, as well as road markings, road boundaries, and the relationship between the target and lane lines.

[0031] The millimeter-wave radar data processing module acquires millimeter-wave radar data, namely, millimeter-wave radar detection data, and outputs radar target information, radar reflection point information, or environmental scene information established from millimeter-wave radar detection data.

[0032] The main vehicle motion state acquisition module acquires the main vehicle motion state, namely, the main vehicle speed, steering state, yaw motion state or vehicle trajectory information, which is used to determine the main vehicle's driving direction and to determine whether the vehicle is in a motion state suitable for deviation detection. S2. The visual perception data deviation detection module determines whether there is a uniform systematic deviation in the data output by the visual perception module based on the visual perception scene, the millimeter-wave radar environment scene, and the main vehicle's motion state.

[0033] Visual perception scenes are constructed based on visual perception data, and millimeter-wave radar environment scenes are constructed based on millimeter-wave radar data. Scene construction refers to organizing information such as the position, speed and direction of movement of visual targets, lane lines, road markings and road boundaries into a visual scene feature set within a preset time range, and organizing information such as the position, speed, reflection point and static environment features of radar targets into a millimeter-wave radar scene feature set. The visual perception data and millimeter-wave radar data are already located in a preset unified coordinate system, and are subsequently compared as a whole according to the same or corresponding data types. Scene construction does not include re-performing coordinate transformation.

[0034] The visual perception scene includes one or more of the following: the location and distribution of visual targets, the speed and direction of movement of visual targets, lane lines, road markings, road boundaries, and the relationship between targets and lane lines.

[0035] The millimeter-wave radar environment scenario includes one or more of the following: the location and distribution of radar targets, the speed and direction of movement of radar targets, static reflection points, parked vehicles, guardrails, walls, road signs, or other environmental reflection characteristics.

[0036] Specifically, when the main vehicle's motion state reaches the preset deviation detection conditions, the overall spatial relationship between the visual perception scene and the millimeter-wave radar environment scene is analyzed to detect whether there is a unified systematic deviation in the visual perception scene. The preset deviation detection condition is: the main vehicle is in a straight-line state.

[0037] The analysis process of the overall spatial relationship between the visual perception scene and the millimeter-wave radar environment scene includes: Based on the position distribution, motion distribution, directional characteristics, and multi-frame change trends of the two types of scenes within a preset time range, the overall differences between the visual perception scene and the millimeter-wave radar environment scene are statistically analyzed; during the analysis, it is not required to establish a one-to-one correspondence between millimeter-wave radar targets and visual targets.

[0038] If, at least three consecutive moments, the visually perceived scene exhibits consistent overall rotation, translation, or scale anomalies relative to the millimeter-wave radar environment, a unified systematic bias is identified in the visually perceived data. Specifically, when the overall statistical bias of the scene consistently meets preset conditions, it is determined that the data output by the visually perceived module has a unified systematic bias, and the process proceeds to the next step: determining the unified compensation parameters.

[0039] Visual perception data is also used to help determine the type of deviation. When lane lines or road markings in multiple frames cannot maintain the expected parallel or scale relationship in the bird's-eye view coordinate system, it is determined that there is a pitch deviation in the visual perception data. When the overall extension direction of lane lines, road markings, or visual target groups is consistently inconsistent with the straight-line direction of the main vehicle, it is determined that there is a yaw deviation in the visual perception data.

[0040] S3, such as Figure 2 As shown, when a unified systematic deviation is determined in the visual perception scene, the visual perception data deviation detection module determines candidate unified compensation parameters for the visual perception data based on the overall spatial difference between the visual perception scene and the millimeter-wave radar environment scene and the driving direction of the main vehicle.

[0041] The unified compensation parameters are used to describe the overall deviation of the visual perception output coordinate system from the preset fusion coordinate system, including at least one of the following: yaw angle compensation, pitch angle compensation, lateral or longitudinal position offset, scale compensation, velocity direction compensation corresponding to spatial compensation, and optional velocity scale compensation.

[0042] The unified compensation parameters are determined by the overall scene information and data from multiple time points, rather than generating compensation values ​​for each individual target. When the detected systematic deviation corresponds only to some parameters, only the corresponding compensation parameters need to be determined.

[0043] Within a preset time range, a statistical overall correlation is established based on the target position, velocity, direction of motion, and environmental features in the visual perception scene and the millimeter-wave radar environment scene. For candidate unified compensation parameters, candidate scene transformation is performed on data related to spatial position, scale, or direction of motion in the visual perception scene, and at least one of the following is calculated between the transformed visual perception scene and the millimeter-wave radar environment scene: position distribution error, velocity or direction of motion distribution error, road feature geometric relationship error, and direction error between the visual road direction and the vehicle's driving direction. The overall scene error is determined based on the errors at multiple time points, and candidate unified compensation parameters that reduce the overall scene error are determined; in one embodiment, the candidate unified compensation parameters can be determined by minimizing the overall scene error at multiple time points.

[0044] Determining candidate unified compensation parameters includes: determining statistical association weights based on at least one of the following: positional proximity, velocity similarity, motion direction similarity, multi-frame change trend, and stability between visual targets or visual environment features and millimeter-wave radar targets or radar environment features; wherein, a visual target is allowed to form statistical associations with multiple millimeter-wave radar targets or radar environment features with different weights, and some visual targets, millimeter-wave radar targets, or environmental features are allowed not to participate in the determination of the candidate unified compensation parameters.

[0045] S4. After obtaining the candidate unified compensation parameter, select data within the same preset time range used to determine the candidate unified compensation parameter, and calculate the overall scene error without applying the candidate unified compensation parameter. Without changing the formal output data, apply the candidate unified compensation parameter to the visual perception scene within the same preset time range, and calculate the overall scene error after trial compensation using the same error term, statistical correlation method, and error weight. This trial calculation process does not require re-executing the complete unified systematic deviation detection. When the overall scene error after trial compensation is less than the overall scene error before trial compensation, and the candidate unified compensation parameter is within the preset allowable range, proceed to S5, and use the unified compensation parameter to perform pre-fusion compensation on subsequent visual perception data.

[0046] The preset allowable range is determined based on at least one of the following: the installation position, installation angle, installation tolerance, allowable offset range of the bracket structure, and reasonable installation variations that may occur during normal vehicle use. This range is used to define the reasonable value ranges for rotation compensation, position compensation, spatial scale compensation, and optional velocity scale compensation. If the overall scene error does not decrease after trial compensation, or if the candidate unified compensation parameter exceeds the preset allowable range, the candidate unified compensation parameter is not activated, and the visual perception data remains uncompensated.

[0047] S5. The unified compensation module is located between the data output end of the visual perception module and the multi-sensor fusion module. It performs unified compensation on the visual perception data according to the unified compensation parameters to obtain the compensated visual perception data. Specifically, through the unified compensation module, without modifying the internal calibration parameters, coordinate transformation parameters, and perception algorithm of the visual perception module, the spatial position and motion direction related data output by the visual perception module are uniformly transformed according to the unified compensation parameters to form compensated visual perception data.

[0048] The unified compensation parameter is applied to visual perception data related to spatial location or direction of motion, including at least one of the following: target location, target speed, target direction, lane line coordinates, road marking coordinates, road boundary, and spatial relationship between the target and lane lines.

[0049] The unified compensation for visual perception data involves performing corresponding transformations on different types of visual perception data based on the same set of unified compensation parameters. Specifically, target position, lane lines, road markings, and road boundaries are transformed according to a unified rotation, translation, or scaling relationship; target speed and target direction are corrected according to the directional relationship corresponding to the spatial transformation; when a systematic deviation in speed scale is detected, speed scale transformation can also be performed based on the speed scale compensation coefficient; position translation does not affect the speed vector; information unaffected by spatial coordinate deviations, such as target category and tracking markers, remains unchanged.

[0050] The unified compensation is applied to the entire visual scene output by the visual perception module, rather than replacing or correcting individual visual targets one by one based on millimeter-wave radar data. For visual targets and road environment information that do not have a direct correspondence with millimeter-wave radar targets, the same scene-level transformation is also performed according to the unified compensation parameters.

[0051] After unified compensation is completed, the compensated visual perception data is generated and input into the multi-sensor fusion module as an independent visual information source.

[0052] S6. Through the multi-sensor fusion module, the compensated visual perception data and millimeter-wave radar data are associated and fused as independent information sources.

[0053] The multi-sensor fusion module can use existing target association, target tracking and fusion processing methods to generate fused target information and road environment perception results.

[0054] This invention does not limit the specific multi-sensor fusion algorithm, but improves the spatial consistency between the visual perception scene and the millimeter-wave radar environment scene through scene-level unified compensation before the visual perception data enters the fusion module, thereby reducing the target association failure and fusion result instability caused by the overall deflection, offset or scale abnormality of the visual scene.

[0055] S1 to S6 are executed in a cyclical manner. When the main vehicle meets the preset deviation detection conditions, the scenario construction, unified system deviation detection, candidate unified compensation parameter determination and verification are continuously performed; when a persistent unified system deviation is detected again, the candidate unified compensation parameters are regenerated and verified.

[0056] As an embodiment of the present invention: The vehicle is equipped with an independent visual perception module, as well as millimeter-wave radar and a fusion processing device. The fusion processing device is the visual perception data deviation detection module and the unified compensation module in this invention.

[0057] The visual perception module outputs structured visual perception data to the fusion processing device through a preset communication interface. The structured visual perception data may include target position, target speed, target direction, target category, target tracking markers, lane coordinates, road markings, and information on the relationship between the target and lane lines.

[0058] The fusion processing device only receives structured data output by the visual perception module, and does not access the image data, camera calibration parameters, coordinate transformation parameters and perception algorithms inside the visual perception module, nor does it write compensation parameters to the visual perception module.

[0059] The millimeter-wave radar outputs radar target data and environmental reflection point data to the fusion processing unit. The fusion processing unit also acquires vehicle motion information such as vehicle speed, steering status, or vehicle direction.

[0060] The fusion processing unit detects whether there is a unified systematic deviation in the visual perception scene based on visual perception data, millimeter-wave radar data, and vehicle motion information. When a deviation is determined to exist, the fusion processing unit generates unified compensation parameters and completes scene-level unified compensation before the visual perception data enters the multi-sensor fusion processing.

[0061] The process of detecting systematic deviations in visual scene uniformity: When the main vehicle is traveling straight, the fusion processing device acquires visual perception scene data and millimeter-wave radar environmental scene data within a preset time range, and analyzes the overall spatial relationship between the two types of scenes.

[0062] When performing scene analysis, it is not required that a one-to-one correspondence be established between all visual targets and radar targets. The fusion processing device can calculate the overall difference between the visually perceived scene and the millimeter-wave radar environment scene based on the positional distribution, directional distribution, motion trend, and multi-frame changes of target or environmental features in the two types of scenes.

[0063] When the analysis results at multiple time points all indicate that the visually perceived scene has an overall rotation, overall translation, or scale anomaly with basically consistent direction and change patterns, it is determined that the difference belongs to a unified systematic bias of the visually perceived data, rather than an accidental detection error of a single target.

[0064] The system can also assist in judging the type of deviation based on the lane lines or road markings output by the visual perception module. When the lane lines or road markings in the bird's-eye view coordinate system cannot maintain the expected parallelism or scale relationship, it can be judged that there is a deviation in the pitch or scale direction; when the overall direction of the lane lines, road markings, or visual scene is consistently inconsistent with the straight-line direction of the main vehicle, it can be judged that there is a deviation in the yaw direction.

[0065] When the overall difference in the scene continues to meet the preset conditions, the fusion processing device determines that there is a unified systematic deviation in the visually perceived scene and starts the calculation process of unified compensation parameters.

[0066] The calculation process for unified compensation parameters: After determining that there is a unified systematic deviation in the visual perception scene, the fusion processing device establishes a statistical overall correlation based on the target position, speed, motion direction and environmental features in the visual perception scene and the millimeter-wave radar environment scene within a preset time range; calculates the scene transformation corresponding to the candidate unified compensation parameters, and determines the overall scene error at multiple times based on at least one of the position distribution error, speed or motion direction distribution error, road feature geometric relationship error and direction error between the transformed visual perception scene and the millimeter-wave radar environment scene, and the direction error between the visual road direction and the main vehicle driving direction, and then determines the candidate unified compensation parameters that reduce the overall scene error.

[0067] The unified compensation parameter can be expressed as: ; in, Indicates unified compensation parameters; R represents the uniform rotation relationship, used to describe at least one angle compensation in the yaw, pitch, and roll directions; t represents a uniform translation vector, used to describe at least one of the horizontal, vertical, and longitudinal positional offsets; Indicates the spatial scale compensation coefficient; This indicates the optional velocity scale compensation coefficient.

[0068] When the detected uniform systematic deviation corresponds to only some parameters, only the corresponding rotational component, translational component, or scale component needs to be determined; when there is no velocity scale systematic deviation, it is not necessary to set it. Or make The value is 1.

[0069] The overall scene error can be expressed as: ; ; in, This represents the overall scene error when the uniform compensation parameter Θ is applied. K represents the number of moments included in the statistics; k represents the time number; This represents the position distribution error at time k; Indicates the error in the distribution of velocity or direction of motion; Indicates geometric relationship errors of lane lines, road markings, or road boundaries; This indicates the directional error between the visual road direction and the vehicle's driving direction. , , and This indicates the weight corresponding to each error term; This represents the candidate unified compensation parameter.

[0070] Each error term can be selected from one or more based on available data. The above is described as "determining candidate unified compensation parameters that reduce the overall scene error"; in one specific implementation, this can be determined by minimizing the overall scene error at multiple time points. .

[0071] In statistical overall correlation, a precise one-to-one correspondence between visual targets and millimeter-wave radar targets is not required. Statistical correlation weights can be determined based on at least one of the following: proximity of position, similarity of velocity, similarity of motion direction, multi-frame variation trend, and target stability. A visual target is allowed to form statistical correlations with multiple millimeter-wave radar targets or radar environmental features with different weights. Unstable targets, anomalous targets, or visual targets, millimeter-wave radar targets, or environmental features lacking effective correlation relationships are allowed to be excluded from the determination of the current candidate unified compensation parameters.

[0072] A unified compensation process for different types of visual structured data: After obtaining and activating the unified compensation parameters, the unified compensation module performs corresponding unified transformations on data related to spatial location or direction of motion in the visual perception scene based on the type of data output by the visual perception module. Visual targets, lane lines, road markings, road boundaries, and other spatial information at the same time use the same set of unified compensation parameters, without setting independent compensation amounts for different targets.

[0073] The target position, lane line coordinates, road marking coordinates, and road boundary coordinates undergo a unified spatial transformation based on the rotation, translation, and scale relationships in the unified compensation parameters. The target velocity vector, target heading, target motion direction, lane line extension direction, and road boundary direction undergo a directional transformation based on the unified rotation relationship. A velocity scale transformation can also be performed when a systematic deviation in the velocity scale is detected. The position translation does not affect the velocity vector. The same scene-level unified transformation is also performed on visual targets and road environment information that do not have a direct correspondence with millimeter-wave radar targets.

[0074] The spatial relationship between the target and the lane line is not directly based on the visual judgment result before compensation, but is recalculated based on the compensated target position and the compensated lane line; the target category, target tracking mark, target confidence and other information not directly affected by the deviation of the spatial coordinate system remain unchanged in the original output value of the visual perception module.

[0075] Verification and application of candidate compensation parameters: After obtaining candidate unified compensation parameters, data within the same preset time range used to determine these parameters are selected. Using predetermined error terms, statistical correlation methods, and error weights, the overall scene error without applying the candidate unified compensation parameters is calculated. Subsequently, without altering the final output data, the candidate unified compensation parameters are applied to a visual perception scene within the same preset time range, and the same error calculation method is used to obtain the overall scene error after trial compensation. This trial calculation process only recalculates the overall scene error and does not require re-executing the complete unified systematic deviation detection.

[0076] , ; when ,and When this happens, candidate unified compensation parameters are enabled.

[0077] in, This indicates the unit transformation parameter when compensation is not performed; This indicates the overall error of the scene before the trial compensation; This indicates the overall error of the scene after applying candidate unified compensation parameters to data within the same preset time range. This indicates the preset allowable range of candidate unified compensation parameters.

[0078] The preset allowable range is determined based on at least one of the following: the installation position of the vision sensor, the installation angle, the installation tolerance, the allowable offset range of the bracket structure, and the reasonable installation variation range that may occur during normal vehicle use. It is used to limit the reasonable value range of rotation compensation, position compensation, spatial scale compensation, and optional speed scale compensation, respectively.

[0079] when Not less than If the candidate unified compensation parameter exceeds the preset allowable range, the candidate unified compensation parameter will not be enabled, and the visual perception data will remain uncompensated.

[0080] Fusion and output of compensated data: Once the unified compensation parameters meet the application conditions, the unified compensation module uses these parameters to perform unified compensation on the target position, target speed, target direction, lane lines, road markings, road boundaries, and related spatial relationship information subsequently output by the visual perception module before fusion.

[0081] The compensated visual perception data and millimeter-wave radar data are input into the multi-sensor fusion module as independent information sources.

[0082] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the protection scope of this invention.

[0083] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as implying that all features included in the exemplary embodiments are essential technical features of the claims of the present invention.

[0084] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0085] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

[0086] The modules, units, or components in the embodiments of the present invention can be implemented in hardware, in software running on one or more processors, or in a combination thereof. Those skilled in the art should understand that... In practice, microprocessors or digital signal processors (DSPs) can be used to implement embodiments of the invention. The invention can also be implemented on computer program products or computer-readable media for performing some or all of the methods described herein.

Claims

1. A method for deviation detection and fusion compensation in a visual perception data system based on millimeter-wave radar, characterized in that, Includes the following steps: S1. Continuously acquire visual perception data, millimeter-wave radar data, and the vehicle's motion status; S2. When the vehicle's motion reaches the preset deviation detection conditions, analyze the overall spatial relationship between the visual perception scene constructed from visual perception data and the millimeter-wave radar environment scene constructed from millimeter-wave radar environment data to detect whether there is a unified systematic deviation in the visual perception scene. S3. If a unified systematic deviation is detected in the visual perception scene, then candidate unified compensation parameters for the visual perception data are calculated based on the overall spatial difference between the visual perception scene and the millimeter-wave radar environment scene and the driving direction of the main vehicle. S4. Apply the candidate unified compensation parameters to the current visual perception scene and calculate the overall scene error before and after the trial compensation. When the overall scene error after trial compensation is less than the overall scene error before trial compensation, and the candidate unified compensation parameter is within the preset allowable range, the candidate unified compensation parameter is used as the formal unified compensation parameter. S5. Based on the formal unified compensation parameters, perform unified compensation on the visual perception data to obtain the compensated visual perception data. S6. The compensated visual perception data and millimeter-wave radar data are correlated and fused as independent information sources.

2. The method for deviation detection and fusion compensation of a visual perception data system based on millimeter-wave radar according to claim 1, characterized in that, The preset deviation detection condition is: the main vehicle is in a straight-line state.

3. The method for deviation detection and fusion compensation of a visual perception data system based on millimeter-wave radar according to claim 1, characterized in that, The analysis process of the overall spatial relationship between the visual perception scene and the millimeter-wave radar environment scene includes: Based on the location distribution, motion distribution, directional features and multi-frame change trends of the two types of scenes within a preset time range, the overall differences between the visual perception scene and the millimeter-wave radar environment scene are statistically analyzed. If, at least three consecutive moments, the visually perceived scene exhibits a consistent overall rotation, translation, or scale anomaly relative to the millimeter-wave radar environment, it is determined that there is a unified systematic bias in the visually perceived data.

4. The method for deviation detection and fusion compensation of a visual perception data system based on millimeter-wave radar according to claim 1, characterized in that, Visual perception data is also used to help determine the type of deviation. When lane lines or road markings in multiple frames cannot maintain the expected parallel or scale relationship in the bird's-eye view coordinate system, it is determined that there is a pitch deviation in the visual perception data. When the overall extension direction of lane lines, road markings, or visual target groups is consistently inconsistent with the straight-line direction of the main vehicle, it is determined that there is a yaw deviation in the visual perception data.

5. The method for deviation detection and fusion compensation of a visual perception data system based on millimeter-wave radar according to claim 1, characterized in that, The process of calculating candidate unified compensation parameters includes: determining statistical association weights based on at least one of the following factors: positional proximity, velocity similarity, motion direction similarity, multi-frame change trend, and stability between visual targets or visual environment features and millimeter-wave radar targets or radar environment features; wherein, a visual target is allowed to form statistical associations with multiple millimeter-wave radar targets or radar environment features with different weights, and some visual targets, millimeter-wave radar targets, or environmental features are allowed not to participate in the determination of candidate unified compensation parameters.

6. The method for deviation detection and fusion compensation of a visual perception data system based on millimeter-wave radar according to claim 1, characterized in that, The unified compensation parameters include at least one of the following: yaw angle compensation, pitch angle compensation, lateral or longitudinal position offset, scale compensation, velocity direction compensation corresponding to spatial compensation, and optional velocity scale compensation.

7. The method for deviation detection and fusion compensation of a visual perception data system based on millimeter-wave radar according to claim 1, characterized in that, The unified compensation for visual perception data is as follows: based on the same set of unified compensation parameters, corresponding transformations are performed on different types of visual perception data.

8. A deviation detection and fusion compensation system for a visual perception data system based on millimeter-wave radar, characterized in that, include: The visual perception module is used to output visual perception data. Millimeter-wave radar data processing module, used to output millimeter-wave radar data; The main vehicle motion status acquisition module is used to output the main vehicle motion status. The visual perception data deviation detection module is used to acquire visual perception data, millimeter-wave radar data, and the vehicle's motion state. It constructs a visual perception scene based on the visual perception data and a millimeter-wave radar environment scene based on the millimeter-wave radar data. When the vehicle's motion state reaches a preset deviation detection condition, it analyzes the overall spatial relationship between the visual perception scene and the millimeter-wave radar environment scene to detect whether there is a unified systematic deviation in the visual perception scene. If a deviation is determined, it calculates candidate unified compensation parameters based on the overall spatial difference between the visual perception scene and the millimeter-wave radar environment scene and the vehicle's driving direction. The candidate unified compensation parameters are then applied to the current visual perception scene, and the overall scene error before and after the trial compensation is calculated. When the overall scene error after trial compensation is less than the overall scene error before trial compensation, and the candidate unified compensation parameter is within the preset allowable range, the candidate unified compensation parameter is used as the formal unified compensation parameter. The unified compensation module is used to perform unified compensation on visual perception data according to the formal unified compensation parameters to obtain compensated visual perception data. The multi-sensor fusion module is used to associate and fuse compensated visual perception data and millimeter-wave radar data as independent information sources.

9. A deviation detection and fusion compensation system for a visual perception data system based on millimeter-wave radar according to claim 8, characterized in that, The visual perception module outputs structured visual perception data through a preset data interface. The apparatus for executing the method of any one of claims 1-7 does not access or modify the image processing process, camera calibration parameters, coordinate transformation parameters and perception algorithm inside the visual perception module, and does not write unified compensation parameters to the visual perception module.

10. A device for deviation detection and fusion compensation in a visual perception data system based on millimeter-wave radar, characterized in that, include: A processor and a memory, the memory storing a computer program executable by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

Images