Data fusion method and system of vision and millimeter wave radar

Abstract

The invention provides a visual and millimeter wave radar data fusion method and system, and the method guarantees the data time synchronization through a synchronization strategy based on event triggering, achieves the precise spatial alignment through a multi-stage joint calibration technology, and completes the efficient feature fusion and target perception through a feature fusion network structure based on deep learning. The three parts work cooperatively to form a set of complete visual and millimeter wave radar data fusion method, and the target detection, classification and tracking capabilities of the system in a target identification scene are significantly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data fusion, and in particular relates to a method and system for fusing vision and millimeter-wave radar data. Background Art

[0002] In target recognition technology, single-sensor technology has many limitations, making it difficult to meet the demands for accurate target identification and reliable perception in complex and ever-changing scenarios. While visual sensors can provide rich texture, color, and semantic information, facilitating target classification and detailed identification, image quality can be severely degraded in adverse weather (such as heavy rain, dense fog, and dust), low light, or occlusion, resulting in blurred or even lost target features, which in turn affects recognition accuracy. Millimeter-wave radar, on the other hand, operates in all weather conditions and around the clock, effectively detecting the distance, speed, and angle of a target, unaffected by lighting conditions or partial occlusion. However, its data is relatively sparse, lacking the intuitive semantic and texture features of visual images, making it difficult to accurately determine the specific category and form of a target. Summary of the Invention

[0003] Based on this, an embodiment of the present invention provides a method and system for the fusion of vision and millimeter-wave radar data, aiming to make up for the lack of semantic information of millimeter-wave radar through visual data. Millimeter-wave radar data enhances the reliability of vision in harsh environments, thereby achieving more comprehensive, accurate and stable recognition of targets, and improving the adaptability and robustness of the target recognition system in complex scenarios.

[0004] A first aspect of an embodiment of the present invention provides a method for fusing visual and millimeter-wave radar data, the method comprising: Acquire historical visual data and historical millimeter-wave radar data, determine recognition conditions, and determine whether the recognition conditions are triggering events; If so, the control simultaneously collects the target visual data and the target millimeter-wave radar data and timestamps them to ensure the consistency of the time dimension; Based on the installation parameters of the visual sensor and millimeter-wave radar in the target recognition scenario and the known geometric relationship, preliminary coordinate transformation is performed to obtain the coordinate correspondence; Comparing and identifying position differences of the target based on the target visual data and the target millimeter-wave radar data having consistent time dimensions, and optimizing the coordinate correspondence based on the position differences; During the actual target recognition process, the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence is continuously evaluated, and the coordinate correspondence is further optimized to ensure the spatial alignment of visual and millimeter-wave radar data; The aligned visual and millimeter-wave radar data are input into a trained deep learning-based feature fusion network model to output a comprehensive perception result of the target object.

[0005] Furthermore, in the steps of acquiring historical visual data and historical millimeter-wave radar data, determining the recognition situation, and judging whether the recognition situation is a trigger event, for the visual sensor, the boundary of the monitoring area is delineated, and when a target object is detected entering the monitoring area, or the light in the scene changes, or the shape and size of the target object changes, it is determined that a trigger event has occurred; for the millimeter-wave radar, the target distance and speed change thresholds are set, and when the detected distance change to the target exceeds the target distance, or the speed change rate exceeds the speed change threshold, it is regarded as a trigger event.

[0006] Furthermore, in the step of performing preliminary coordinate transformation based on the installation parameters of the visual sensor and the millimeter-wave radar in the target recognition scene and the known geometric relationship to obtain the coordinate correspondence, the target coordinates in the millimeter-wave radar coordinate system are first converted to the world coordinate system and then further converted to the image pixel coordinate system using traditional coordinate transformation formulas and transformation matrices.

[0007] Furthermore, the step of comparing and identifying the position difference of the target based on the target visual data and the target millimeter-wave radar data having the same time dimension, and optimizing the coordinate correspondence according to the position difference includes: Selecting target feature points in a visual image according to a feature extraction algorithm, wherein the target visual data is represented by the visual image; According to the target tracking algorithm, the identified target is continuously tracked; At different times, the coordinates of the target feature points in the visual image are compared with the coordinates of the corresponding target after the millimeter-wave radar is converted to the image pixel coordinate system, and the position difference between the two is calculated; According to the position difference, the elements in the transformation matrix are adjusted using the least square method to optimize the coordinate correspondence.

[0008] Furthermore, during the actual target recognition process, the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence relationship is continuously evaluated, and the step of further optimizing the coordinate correspondence relationship includes: Counting the missed detection rate, false detection rate, and trajectory deviation of target detection, and determining whether the missed detection rate, false detection rate, and trajectory deviation meet preset values; If not, the current coordinate correspondence is evaluated, and the elements in the transformation matrix are adjusted again according to the evaluation result.

[0009] Further, the deep learning-based feature fusion network model comprises a front end, an intermediate layer and a back end, in the front end, for visual data, an improved convolutional neural network architecture is adopted, wherein the number of convolutional layers is increased in the shallow layer of the network, and the convolution kernel design is optimized in the deep layer of the network, for millimeter wave radar data, the spatial distribution characteristics of the point cloud are utilized, and the feature information is extracted through a multilayer perceptron; In the intermediate layer, a cross-modal attention mechanism is introduced, wherein the visual features and the millimeter wave radar features are first encoded, then the similarity and the correlation between the two are calculated, the importance weights of the two features in different scenes are learned, the two features are weighted and fused according to the weight adjustment; In the back end, a multi-task learning method is adopted to simultaneously perform target detection, target classification and target trajectory prediction tasks, wherein based on the fused feature information, the target detection task determines the position and category of the target in the image through boundary box regression and category prediction; the target classification task further refines the target category; the target trajectory prediction task predicts the future motion trajectory of the target by using historical data and current features; finally, the comprehensive perception result of the target object is output.

[0010] The second aspect of the embodiment of the application provides a visual and millimeter wave radar data fusion system for realizing the visual and millimeter wave radar data fusion method provided by the first aspect of the embodiment of the application, and the system comprises: A judgment module is configured to acquire historical visual data and historical millimeter wave radar data, determine an identification condition, and judge whether the identification condition is a trigger event; A control module is configured to control the acquisition of target visual data and target millimeter wave radar data at the same time when the identification condition is determined as the trigger event, and to timestamp to ensure the consistency of the time dimension; A conversion module is configured to perform preliminary coordinate conversion according to the installation parameters of the visual sensor and the millimeter wave radar in the target identification scene and the known geometric relationship, and obtain a coordinate correspondence; A first optimization module is configured to compare the position difference of the identified target according to the target visual data and the target millimeter wave radar data consistent in the time dimension, and optimize the coordinate correspondence according to the position difference; A second optimization module is configured to continuously evaluate the accuracy of the data fused according to the optimized coordinate correspondence for target detection and tracking in the actual target identification process, and optimize the coordinate correspondence again to ensure the alignment of the visual and millimeter wave radar data in space; An input module is configured to input the aligned visual and millimeter wave radar data into the trained deep learning-based feature fusion network model, and output the comprehensive perception result of the target object.

[0011] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, including: The readable storage medium stores one or more programs, which, when executed by the processor, implement the data fusion method of vision and millimeter-wave radar as described in the first aspect.

[0012] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the data fusion method of vision and millimeter-wave radar as described in the first aspect.

[0013] The embodiments of the present invention provide a method and system for fusion of visual and millimeter-wave radar data. These methods ensure data time synchronization through an event-triggered synchronization strategy, achieve precise spatial alignment through multi-stage joint calibration technology, and complete efficient feature fusion and target perception through a deep learning-based feature fusion network structure. The three work together to form a complete method for fusion of visual and millimeter-wave radar data, significantly improving the system's ability to detect, classify, and track targets in target recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flowchart of a method for fusing visual and millimeter-wave radar data according to the first embodiment of the present invention; Figure 2 This is a structural block diagram of a data fusion system for vision and millimeter-wave radar provided in the second embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0015] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0016] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0018] Example 1 The first embodiment of the present invention provides a method for data fusion of vision and millimeter wave radar, see Figure 1 , is a flow chart for implementing a method for fusion of vision and millimeter-wave radar data, specifically including steps S01 to S06.

[0019] Step S01, obtain historical visual data and historical millimeter wave radar data, determine the recognition situation, and judge whether the recognition situation is a trigger event. If so, execute step S02.

[0020] In target recognition scenarios, trigger event conditions for visual sensors and millimeter-wave radars are pre-defined. For visual sensors, the monitoring area boundary is defined. A trigger event is detected when a target object enters the monitoring area, or when the scene's lighting changes, or when the target object's shape or size changes. For millimeter-wave radars, target distance and speed change thresholds are set. A trigger event is detected when the detected distance exceeds the target distance, or when the speed change rate exceeds the speed change threshold.

[0021] In step S02, the target visual data and the target millimeter-wave radar data are collected simultaneously and timestamped to ensure the consistency of the time dimension.

[0022] It can be understood that once a trigger event is detected, the synchronization mechanism is immediately activated to send synchronous acquisition instructions to the visual sensor and millimeter-wave radar, so that both can collect data at the same time and accurately timestamp the collected data to ensure that the visual image data and the millimeter-wave radar detection data are consistent in the time dimension, providing an accurate time reference for subsequent fusion processing.

[0023] Step S03: Perform preliminary coordinate conversion based on the installation parameters of the visual sensor and the millimeter-wave radar in the target recognition scene and the known geometric relationship to obtain a coordinate correspondence.

[0024] Specifically, based on the installation parameters of the visual sensor and millimeter-wave radar in the target recognition scenario (such as relative position and installation angle), as well as the known geometric relationship, traditional coordinate transformation formulas and transformation matrices are used to transform the target coordinates in the millimeter-wave radar coordinate system into the world coordinate system, and then further into the image pixel coordinate system. This phase results in a rough coordinate correspondence. Although there is a certain degree of error, it lays the foundation for subsequent precise calibration.

[0025] More specifically, for the target coordinates in the millimeter wave radar coordinate system , through the formula Calculate its coordinates in the world coordinate system , the coordinates in the world coordinate system Substitute into the formula Get the coordinates in the camera coordinate system , then use the camera internal parameter matrix K, through the formula Convert the coordinates in the camera coordinate system to the image coordinate system to get the coordinates Finally, according to the scaling factor and translation amount, the formula Convert the coordinates in the image coordinate system to the image pixel coordinate system to obtain the final pixel coordinates .

[0026] Among them, the position coordinates of the millimeter wave radar in the world coordinate system are obtained through high-precision measurement equipment (such as laser rangefinder, total station, etc.) And the rotation angles of the three coordinate axes relative to the world coordinate system (pitch angle θ, yaw angle φ, roll angle γ). Calculate the rotation matrix based on the rotation angles , the translation vector .

[0027] Also use the measuring device to obtain the position and attitude parameters of the camera in the world coordinate system and calculate the rotation matrix and translation vectors In addition, the camera intrinsic parameter matrix can be obtained through camera calibration methods such as Zhang's calibration method, which is used for the subsequent conversion from the camera coordinate system to the image coordinate system.

[0028] The camera's intrinsic parameter matrix K contains the camera's focal length 、 , principal point coordinates 、 These parameters can be obtained through camera calibration experiments. When the camera intrinsic parameter matrix is ​​known, the coordinates in the camera coordinate system can be converted to the image coordinate system according to the perspective projection principle.

[0029] Scaling Factor The principal point coordinates can be calculated based on the pixel size of the image sensor. Determined during camera calibration, these parameters are used to transform the image coordinates into the pixel coordinates.

[0030] Step S04 : comparing and identifying the position difference of the target based on the target visual data and the target millimeter-wave radar data having the same time dimension, and optimizing the coordinate correspondence based on the position difference.

[0031] It should be noted that, according to a feature extraction algorithm, target feature points in a visual image are selected, wherein the target visual data is represented by the visual image. In the embodiment of the present invention, the feature extraction algorithm may adopt SIFT, SURF, etc., and the target feature points may be corner points, edge feature points, etc. of the target object; Continuously track the identified target according to the target tracking algorithm, where the target tracking algorithm is Kalman filter, particle filter, etc. At different times, the coordinates of the target feature points in the visual image are compared with the coordinates of the corresponding target after the millimeter-wave radar is converted to the image pixel coordinate system, and the position difference between the two is calculated. Among them, the Euclidean distance error between the two is calculated. , is the coordinate of the target feature point in the visual image, The coordinates of the target after the millimeter-wave radar is converted to the image pixel coordinate system, and the error is used as the optimization target; According to the position difference, the least squares method is used to adjust the elements in the transformation matrix to optimize the coordinate correspondence, gradually reduce the error, and achieve more accurate coordinate calibration. For example, the angle parameters and translation vector values ​​in the rotation matrix are fine-tuned so that the error E gradually decreases, thereby achieving more accurate coordinate calibration and significantly improving the spatial matching accuracy of the two sensor data.

[0032] Step S05: During the actual target recognition process, the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence is continuously evaluated, and the coordinate correspondence is optimized again to ensure spatial alignment of the visual and millimeter-wave radar data.

[0033] It is understandable that the missed detection rate, the false detection rate, and the trajectory deviation of the target tracking are statistically calculated, and it is determined whether the missed detection rate, the false detection rate, and the trajectory deviation meet the preset values; If not, the current coordinate correspondence is evaluated. Based on the evaluation results, the elements in the transformation matrix are adjusted again, forming a closed-loop optimization process to continuously improve the accuracy and reliability of the calibration and ensure accurate spatial matching between visual and millimeter-wave radar data.

[0034] It's important to note that the purpose of evaluating the current coordinate correspondence is to determine whether performance degradation is due to inaccurate coordinate transformation. This can be determined by comparing the distribution of detection and tracking errors for different scenarios and targets, combined with historical calibration data and sensor status information. If a coordinate transformation issue is identified, an optimization algorithm (such as a genetic algorithm, simulated annealing, or other global optimization algorithm) is used based on the evaluation results to further optimize the coordinate transformation and calibration parameters.

[0035] Step S06: Input the aligned visual and millimeter-wave radar data into a trained deep learning-based feature fusion network model to output a comprehensive perception result of the target object.

[0036] In an embodiment of the present invention, a feature fusion network model based on deep learning includes a front-end, an intermediate layer, and a back-end. In the front-end, an improved convolutional neural network (CNN) architecture is adopted for visual data, wherein the number of convolutional layers is increased in the shallow layer of the network to improve the ability to extract image detail features. Specifically, the number of convolutional layers is increased to 3-5 layers, and the convolution kernel size of each layer adopts a small size design of 3×3, the step size is set to 1, and the filling method is the same.

[0037] Convolution kernel design is optimized deep within the network to enhance the extraction of semantic features (such as target category and shape structure) of target objects in complex scenarios. Specifically, grouped convolution or depthwise separable convolution techniques are used to reduce the number of model parameters while enhancing the network's ability to extract complex semantic features. For example, by introducing an attention mechanism module (such as the channel attention mechanism in SE-Net), the importance weights of different channel features are automatically learned during training, strengthening the extraction of semantic features such as target category and shape structure.

[0038] For millimeter-wave radar data, the spatial distribution characteristics of the point cloud are utilized to extract feature information through a multi-layer perceptron (MLP). Feature information includes at least the distance, angle, and speed of the target. In some other embodiments, feature information can also be extracted through a graph neural network (GNN). In the middle layer, a cross-modal attention mechanism is introduced. First, visual features and mmWave radar features are encoded separately. Specifically, for visual features, global average pooling (GAP) is used to compress the feature map into a one-dimensional vector. This is then mapped to the same dimension as the mmWave radar features through a fully connected layer to facilitate subsequent calculations. The mmWave radar features are also resized through the fully connected layer to align with the visual features.

[0039] Then, a multi-head attention mechanism is used to calculate the similarity and correlation between the two features, and learn the importance weights of the two features in different scenarios. For example, in long-range target recognition scenarios, the distance and angle features of the millimeter-wave radar are automatically given higher weights; in close-range scenarios with rich target details, the weight of the visual semantic features is increased. Based on the weight adjustment, the two features are weightedly fused. Specifically, the output of each attention head is spliced, and then the dimension is reduced through a fully connected layer to obtain the fused feature vector, achieving more flexible and efficient feature fusion and giving full play to the complementary advantages of the two sensor data. On the back end, a multi-task learning approach is adopted to simultaneously perform target detection, target classification, and target trajectory prediction tasks. Based on the fused feature information, the target detection task determines the position and category of the target in the image through bounding box regression and category prediction; the target classification task further refines the target category; the target trajectory prediction task uses historical data and current features, inputs them into a recurrent neural network (RNN) or its variants (such as GRU, LSTM), and predicts the future motion trajectory of the target; finally, the comprehensive perception result of the target object is output, providing accurate and reliable decision-making basis for target recognition scenarios.

[0040] In summary, the embodiments of the present invention propose a method for data fusion of vision and millimeter-wave radar. This method ensures data time synchronization through an event-triggered synchronization strategy, achieves precise spatial alignment through a multi-stage joint calibration technology, and completes efficient feature fusion and target perception through a deep learning-based feature fusion network structure. The three work together to form a complete set of vision and millimeter-wave radar data fusion methods, significantly improving the system's detection, classification, and tracking capabilities in target recognition scenarios.

[0041] Example 2 The second embodiment of the present invention provides a data fusion system 200 of vision and millimeter wave radar, see Figure 2 , is a structural block diagram of a method system for fusion of vision and millimeter-wave radar data. The vision and millimeter-wave radar data fusion system 200 includes: The judgment module 21 is used to obtain historical visual data and historical millimeter-wave radar data, determine the recognition status, and determine whether the recognition status is a trigger event. For visual sensors, the monitoring area boundary is delineated. When a target object enters the monitoring area, or when the light in the scene changes, or the shape and size of the target object changes, it is determined to be a trigger event. For millimeter-wave radars, target distance and speed change thresholds are set. When the detected distance to the target changes by more than the target distance, or the speed change rate exceeds the speed change threshold, it is considered a trigger event. The control module 22 is configured to control the simultaneous acquisition of target visual data and target millimeter-wave radar data and add time stamps to ensure consistency in time dimensions when the recognition situation is determined to be a trigger event; The conversion module 23 is used to perform preliminary coordinate conversion based on the installation parameters of the visual sensor and the millimeter-wave radar in the target recognition scene and the known geometric relationship to obtain a coordinate correspondence. The target coordinates in the millimeter-wave radar coordinate system are first converted to the world coordinate system and then further converted to the image pixel coordinate system using traditional coordinate conversion formulas and conversion matrices. A first optimization module 24 is configured to compare and identify target position differences based on target visual data and target millimeter-wave radar data having consistent time dimensions, and optimize the coordinate correspondence based on the position differences; A second optimization module 25 is used to continuously evaluate the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence during the actual target recognition process, and to further optimize the coordinate correspondence to ensure spatial alignment of the visual and millimeter-wave radar data; An input module 26 is configured to input the aligned visual and millimeter-wave radar data into a trained deep learning-based feature fusion network model, and output a comprehensive perception result of the target object. The deep learning-based feature fusion network model includes a front-end, an intermediate layer, and a back-end. In the front-end, an improved convolutional neural network architecture is used for visual data. The number of convolutional layers is increased in the shallow layers of the network, and the convolution kernel design is optimized in the deep layers of the network. For millimeter-wave radar data, the spatial distribution characteristics of the point cloud are utilized to extract feature information through a multi-layer perceptron. In the middle layer, a cross-modal attention mechanism is introduced. First, the visual features and millimeter-wave radar features are encoded separately. Then, by calculating the similarity and correlation between the two, the importance weights of the two features in different scenarios are learned. The two features are weightedly fused based on the weight adjustment. On the back end, a multi-task learning approach is adopted to simultaneously perform target detection, target classification, and target trajectory prediction tasks. Based on the fused feature information, the target detection task determines the position and category of the target in the image through bounding box regression and category prediction; the target classification task further refines the target category; the target trajectory prediction task uses historical data and current features to predict the future motion trajectory of the target; and finally, outputs a comprehensive perception result of the target object.

[0042] Furthermore, in some other embodiments of the present invention, the first optimization module 24 includes: a selection unit, configured to select target feature points in a visual image according to a feature extraction algorithm, wherein the target visual data is represented by the visual image; A tracking unit, configured to continuously track the identified target according to a target tracking algorithm; A calculation unit is used to compare the coordinates of the target feature points in the visual image with the coordinates of the corresponding target after the millimeter wave radar is converted to the image pixel coordinate system at different times, and calculate the position difference between the two; An adjustment unit is used to adjust the elements in the transformation matrix using a least square method according to the position difference, so as to optimize the coordinate correspondence.

[0043] Furthermore, in some other embodiments of the present invention, the second optimization module 25 includes: a judgment unit, configured to calculate a missed detection rate, a false detection rate, and a trajectory deviation of target tracking, and to judge whether the missed detection rate, the false detection rate, and the trajectory deviation meet preset values; The evaluation unit is used to evaluate the current coordinate correspondence when it is determined that the missed detection rate, the false detection rate and the trajectory deviation do not meet the preset values, and adjust the elements in the conversion matrix again according to the evaluation result.

[0044] Example 3 The third embodiment of the present invention provides an electronic device, see Figure 3 , is a structural block diagram of an electronic device, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the data fusion method of vision and millimeter-wave radar as described above is implemented.

[0045] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0046] The memory 20 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of the electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 20 may include both an internal storage unit of the electronic device and an external storage device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is about to be output.

[0047] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for fusion of vision and millimeter-wave radar data as described above is implemented.

[0048] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0049] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0050] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0051] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0052] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A data fusion method for vision and millimeter wave radar, characterized in that: The method comprises: Acquire historical visual data and historical millimeter-wave radar data, determine recognition conditions, and determine whether the recognition conditions are triggering events; If so, the control simultaneously collects the target visual data and the target millimeter-wave radar data and timestamps them to ensure the consistency of the time dimension; Based on the installation parameters of the visual sensor and millimeter-wave radar in the target recognition scenario and the known geometric relationship, preliminary coordinate transformation is performed to obtain the coordinate correspondence; Comparing and identifying position differences of the target based on the target visual data and the target millimeter-wave radar data having consistent time dimensions, and optimizing the coordinate correspondence based on the position differences; During the actual target recognition process, the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence is continuously evaluated, and the coordinate correspondence is further optimized to ensure the spatial alignment of visual and millimeter-wave radar data; The aligned visual and millimeter-wave radar data are input into a trained deep learning-based feature fusion network model to output a comprehensive perception result of the target object.

2. The method for data fusion of vision and millimeter-wave radar according to claim 1, characterized in that: In the steps of acquiring historical visual data and historical millimeter-wave radar data, determining the recognition situation, and judging whether the recognition situation is a trigger event, for the visual sensor, the boundary of the monitoring area is delineated, and when a target object is detected to enter the monitoring area, or the light in the scene changes, or the shape and size of the target object changes, it is determined that a trigger event has occurred; for the millimeter-wave radar, the target distance and speed change thresholds are set, and when the detected distance change of the target exceeds the target distance, or the speed change rate exceeds the speed change threshold, it is considered a trigger event.

3. The method for data fusion of vision and millimeter-wave radar according to claim 2, characterized in that: In the step of performing preliminary coordinate transformation based on the installation parameters of the visual sensor and millimeter-wave radar in the target recognition scenario and the known geometric relationship to obtain the coordinate correspondence, the target coordinates in the millimeter-wave radar coordinate system are first converted to the world coordinate system and then further converted to the image pixel coordinate system using traditional coordinate transformation formulas and transformation matrices.

4. The method for data fusion of vision and millimeter-wave radar according to claim 3, characterized in that: The step of comparing and identifying the position difference of the target based on the target visual data and the target millimeter-wave radar data that are consistent in the time dimension, and optimizing the coordinate correspondence according to the position difference includes: Selecting target feature points in a visual image according to a feature extraction algorithm, wherein the target visual data is represented by the visual image; According to the target tracking algorithm, the identified target is continuously tracked; At different times, the coordinates of the target feature points in the visual image are compared with the coordinates of the corresponding target after the millimeter-wave radar is converted to the image pixel coordinate system, and the position difference between the two is calculated; According to the position difference, the elements in the transformation matrix are adjusted using the least square method to optimize the coordinate correspondence.

5. The method for data fusion of vision and millimeter-wave radar according to claim 4, characterized in that: The step of continuously evaluating the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence during the actual target recognition process and optimizing the coordinate correspondence again comprises: Counting the missed detection rate, false detection rate, and trajectory deviation of target detection, and determining whether the missed detection rate, false detection rate, and trajectory deviation meet preset values; If not, the current coordinate correspondence is evaluated, and the elements in the transformation matrix are adjusted again according to the evaluation result.

6. The method for data fusion of vision and millimeter-wave radar according to claim 5, characterized in that: The deep learning-based feature fusion network model includes a front-end, middle layers, and a back-end. In the front-end, an improved convolutional neural network architecture is used for visual data. The number of convolutional layers is increased in the shallow layers of the network, and the convolution kernel design is optimized in the deep layers of the network. For millimeter-wave radar data, the spatial distribution characteristics of the point cloud are utilized to extract feature information through a multi-layer perceptron. In the middle layer, a cross-modal attention mechanism is introduced. First, the visual features and millimeter-wave radar features are encoded separately. Then, by calculating the similarity and correlation between the two, the importance weights of the two features in different scenarios are learned. The two features are weightedly fused based on the weight adjustment. On the back end, a multi-task learning approach is adopted to simultaneously perform target detection, target classification, and target trajectory prediction tasks. Based on the fused feature information, the target detection task determines the position and category of the target in the image through bounding box regression and category prediction; the target classification task further refines the target category; the target trajectory prediction task uses historical data and current features to predict the future motion trajectory of the target; and finally, outputs a comprehensive perception result of the target object.

7. A data fusion system of vision and millimeter wave radar, characterized in that: A method for implementing the data fusion of vision and millimeter-wave radar according to any one of claims 1 to 6, wherein the system comprises: A judgment module is used to obtain historical visual data and historical millimeter-wave radar data, determine the recognition situation, and determine whether the recognition situation is a trigger event; A control module, configured to control the simultaneous acquisition of target visual data and target millimeter-wave radar data and timestamp the data to ensure consistency in time dimensions when the recognition situation is determined to be a trigger event; The conversion module is used to perform preliminary coordinate conversion based on the installation parameters of the visual sensor and millimeter-wave radar in the target recognition scene and the known geometric relationship to obtain the coordinate correspondence; A first optimization module is configured to compare and identify position differences of the target based on the target visual data and the target millimeter-wave radar data having the same time dimension, and optimize the coordinate correspondence according to the position differences; A second optimization module is used to continuously evaluate the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence during the actual target recognition process, and further optimize the coordinate correspondence to ensure spatial alignment of the visual and millimeter-wave radar data; The input module is used to input the aligned visual and millimeter-wave radar data into the trained deep learning-based feature fusion network model to output the comprehensive perception results of the target object.

8. A computer-readable storage medium, characterized in that include: The readable storage medium stores one or more programs, which, when executed by a processor, implement the data fusion method of vision and millimeter-wave radar according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the data fusion method of vision and millimeter wave radar according to any one of claims 1 to 6.

Images