A data fusion method and system of vision and millimeter wave radar

Abstract

The application provides a data fusion method and system of vision and millimeter wave radar, time synchronization of data is ensured through a synchronous strategy based on event triggering, spatial precise alignment is realized through a multi-stage joint calibration technology, and efficient feature fusion and target perception are completed through a feature fusion network structure based on deep learning, the three work collaboratively to form a complete data fusion method of vision and millimeter wave radar, and the detection, classification and tracking ability of the system to targets in a target recognition scene is significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of data fusion technology, specifically relating to a data fusion method and system for vision and millimeter-wave radar. Background Technology

[0002] In target recognition technology, single-sensor technology has many limitations and cannot meet the needs for accurate target identification and reliable perception in complex and ever-changing scenarios. While visual sensors can provide rich texture, color, and semantic information, facilitating target classification and detail recognition, their imaging quality deteriorates significantly in adverse weather conditions (such as heavy rain, dense fog, and sandstorms), low light, or occlusion, leading to blurred or even lost target features and affecting recognition accuracy. Millimeter-wave radar, on the other hand, has the ability to operate in all weather and at all times, effectively detecting target distance, speed, and angle information, and is unaffected by lighting conditions or partial occlusion. However, its data is relatively sparse and lacks the intuitive semantic and texture features of visual images, making it difficult to accurately determine the specific category and shape of the target. Summary of the Invention

[0003] Based on this, the present invention provides a data fusion method and system for vision and millimeter-wave radar, which aims to make up for the lack of semantic information in millimeter-wave radar by using visual data and enhance the reliability of vision in harsh environments by using millimeter-wave radar data, thereby achieving more comprehensive, accurate and stable target recognition and improving the adaptability and robustness of the target recognition system in complex scenarios.

[0004] A first aspect of this invention provides a data fusion method for visual and millimeter-wave radar, the method comprising:

[0005] Acquire historical visual data and historical millimeter-wave radar data, determine the identification status, and determine whether the identification status is a triggering event;

[0006] If so, control the simultaneous acquisition of target visual data and target millimeter-wave radar data, and timestamp them to ensure consistency in the time dimension;

[0007] Based on the installation parameters of the visual sensor and millimeter-wave radar in the target recognition scenario, as well as the known geometric relationships, a preliminary coordinate transformation is performed to obtain the coordinate correspondence.

[0008] Based on the target visual data and target millimeter-wave radar data that are consistent in the time dimension, the positional differences of the target are compared and identified, and the coordinate correspondence is optimized based on the positional differences;

[0009] In the actual target recognition process, the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence is continuously evaluated, and the coordinate correspondence is optimized again to ensure spatial alignment between visual and millimeter-wave radar data.

[0010] Aligned visual and millimeter-wave radar data are input into a trained deep learning-based feature fusion network model, which outputs a comprehensive perception result of the target object.

[0011] Furthermore, in the steps of acquiring historical visual data and historical millimeter-wave radar data, determining the identification status, and judging whether the identification status is a triggering event, for visual sensors, the boundary of the monitoring area is defined, and when a target object is detected entering the monitoring area, or when the light in the scene changes or the shape and size of the target object changes, it is determined that a triggering event has occurred; for millimeter-wave radar, thresholds for target distance and speed change are set, and when the detected change in target distance exceeds the target distance, or the rate of change in speed exceeds the speed change threshold, it is considered a triggering event.

[0012] Furthermore, in the step of performing preliminary coordinate transformation based on the installation parameters of the visual sensor and the millimeter-wave radar in the target recognition scenario, as well as the known geometric relationship, to obtain the coordinate correspondence, traditional coordinate transformation formulas and transformation matrices are used to first transform the target coordinates in the millimeter-wave radar coordinate system to the world coordinate system, and then further transform them to the image pixel coordinate system.

[0013] Furthermore, the step of comparing and identifying the positional differences of the target based on the time-consistent target visual data and target millimeter-wave radar data, and optimizing the coordinate correspondence based on the positional differences, includes:

[0014] According to the feature extraction algorithm, target feature points in the visual image are selected, wherein the target visual data is represented by the visual image;

[0015] Based on the target tracking algorithm, the identified target is continuously tracked;

[0016] At different times, the coordinates of target feature points in the visual image are compared with the corresponding target coordinates after the millimeter-wave radar is transformed into the image pixel coordinate system, and the positional difference between the two is calculated.

[0017] Based on the positional differences, the elements in the transformation matrix are adjusted using the least squares method to optimize the coordinate correspondence.

[0018] Furthermore, the step of continuously evaluating the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence during the actual target recognition process, and then optimizing the coordinate correspondence again, includes:

[0019] The system calculates the false negative rate and false positive rate of target detection, as well as the trajectory deviation of target tracking, and determines whether the false negative rate, the false positive rate, and the trajectory deviation meet preset values.

[0020] If not, the current coordinate correspondence is evaluated, and the elements in the transformation matrix are adjusted again based on the evaluation results.

[0021] Furthermore, the deep learning-based feature fusion network model includes a front-end, a middle layer, and a back-end. In the front-end, for visual data, an improved convolutional neural network architecture is adopted, wherein the number of convolutional layers is increased in the shallow layers of the network, and the convolutional kernel design is optimized in the deep layers of the network. For millimeter-wave radar data, feature information is extracted through a multilayer perceptron by utilizing the spatial distribution characteristics of point clouds.

[0022] In the intermediate layer, a cross-modal attention mechanism is introduced. First, visual features and millimeter-wave radar features are encoded separately. Then, by calculating the similarity and correlation between the two, the importance weights of the two features in different scenarios are learned. Based on the weight adjustment, the two features are weighted and fused.

[0023] On the backend, a multi-task learning approach is adopted, simultaneously performing target detection, target classification, and target trajectory prediction tasks. Specifically, based on the fused feature information, the target detection task determines the position and category of the target in the image through bounding box regression and category prediction; the target classification task further refines the target category; and the target trajectory prediction task uses historical data and current features to predict the future motion trajectory of the target. Finally, a comprehensive perception result of the target object is output.

[0024] A second aspect of the present invention provides a data fusion system for visual and millimeter-wave radar, used to implement the data fusion method for visual and millimeter-wave radar provided in the first aspect of the present invention, the system comprising:

[0025] The judgment module is used to acquire historical visual data and historical millimeter-wave radar data, determine the recognition status, and determine whether the recognition status is a triggering event;

[0026] The control module is used to simultaneously collect target visual data and target millimeter-wave radar data and timestamp them when the identification situation is determined to be a trigger event, so as to ensure consistency in the time dimension.

[0027] The conversion module is used to perform preliminary coordinate transformation based on the installation parameters of the visual sensor and the millimeter-wave radar in the target recognition scenario, as well as the known geometric relationships, to obtain the coordinate correspondence.

[0028] The first optimization module is used to compare and identify the positional differences of the target based on the target visual data and the target millimeter-wave radar data that are consistent in the time dimension, and to optimize the coordinate correspondence based on the positional differences.

[0029] The second optimization module is used to continuously evaluate the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence during the actual target recognition process, and to optimize the coordinate correspondence again to ensure spatial alignment between visual and millimeter-wave radar data.

[0030] The input module is used to input aligned visual and millimeter-wave radar data into a trained deep learning-based feature fusion network model, and output a comprehensive perception result of the target object.

[0031] A third aspect of the present invention provides a computer-readable storage medium, comprising:

[0032] The readable storage medium stores one or more programs that, when executed by a processor, implement the data fusion method for visual and millimeter-wave radar as described in the first aspect.

[0033] A fourth aspect of the present invention provides an electronic device, the electronic device including a memory and a processor, wherein:

[0034] The memory is used to store computer programs;

[0035] When the processor executes the computer program stored in the memory, it implements the data fusion method for visual and millimeter-wave radar as described in the first aspect.

[0036] The present invention provides a data fusion method and system for vision and millimeter-wave radar. It ensures data time synchronization through an event-triggered synchronization strategy, achieves precise spatial alignment through multi-stage joint calibration technology, and completes efficient feature fusion and target perception through a feature fusion network structure based on deep learning. The three work together to form a complete data fusion method for vision and millimeter-wave radar, which significantly improves the system's ability to detect, classify and track targets in target recognition scenarios. Attached Figure Description

[0037] Figure 1 A flowchart illustrating the implementation of a data fusion method for visual and millimeter-wave radar provided in Embodiment 1 of the present invention;

[0038] Figure 2 This is a structural block diagram of a data fusion system for vision and millimeter-wave radar provided in Embodiment 2 of the present invention;

[0039] Figure 3This is a structural block diagram of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0040] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0041] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0043] Example 1

[0044] Embodiment 1 of this invention provides a data fusion method for visual and millimeter-wave radar. Please refer to [link / reference]. Figure 1 This is a flowchart of a data fusion method for vision and millimeter-wave radar, specifically including steps S01 to S06.

[0045] Step S01: Obtain historical visual data and historical millimeter-wave radar data, determine the identification status, and determine whether the identification status is a triggering event. If so, proceed to step S02.

[0046] In target recognition scenarios, trigger event conditions for visual sensors and millimeter-wave radar are pre-defined. For visual sensors, the boundaries of the monitoring area are defined. When a target object is detected entering the monitoring area, or when there is a change in the lighting or the shape and size of the target object, it is determined that a trigger event has occurred. For millimeter-wave radar, thresholds for target distance and velocity changes are set. When the detected change in target distance exceeds the target distance, or the rate of velocity change exceeds the velocity change threshold, it is considered a trigger event.

[0047] Step S02 involves simultaneously acquiring target visual data and target millimeter-wave radar data, and timestamping them to ensure consistency in the time dimension.

[0048] Understandably, once a trigger event is detected, a synchronization mechanism is immediately activated, sending synchronous acquisition commands to the visual sensor and millimeter-wave radar, enabling both to acquire data at the same time and assigning precise timestamps to the acquired data. This ensures that the visual image data and the millimeter-wave radar detection data are consistent in the time dimension, providing an accurate time reference for subsequent fusion processing.

[0049] Step S03: Based on the installation parameters of the visual sensor and millimeter-wave radar in the target recognition scenario, as well as the known geometric relationships, perform preliminary coordinate transformation to obtain the coordinate correspondence.

[0050] Specifically, based on the installation parameters (such as relative position and installation angle) of the visual sensor and millimeter-wave radar in the target recognition scenario, as well as the known geometric relationships, traditional coordinate transformation formulas and transformation matrices are used to first transform the target coordinates in the millimeter-wave radar coordinate system to the world coordinate system, and then further transform them to the image pixel coordinate system. After this stage, a rough coordinate correspondence is obtained. Although there is some error, it lays the foundation for subsequent accurate calibration.

[0051] More specifically, for the target coordinates in the millimeter-wave radar coordinate system Through formula Its coordinates in the world coordinate system were calculated. coordinates in the world coordinate system Substitute into the formula Obtain the coordinates in the camera coordinate system Then, using the camera intrinsic parameter matrix K, through the formula... Transform the coordinates in the camera coordinate system to the image coordinate system to obtain the coordinates. Finally, based on the scaling factor and translation, the formula is used. Transform the coordinates in the image coordinate system to the image pixel coordinate system to obtain the final pixel coordinates. .

[0052] Among these methods, the position coordinates of the millimeter-wave radar in the world coordinate system are obtained using high-precision measuring equipment (such as laser rangefinders, total stations, etc.). And the rotation angles of the three coordinate axes relative to the world coordinate system (pitch angle θ, yaw angle φ, roll angle γ). Calculate the rotation matrix based on the rotation angles. Translation vector .

[0053] Similarly, the position and attitude parameters of the camera in the world coordinate system are obtained using measuring equipment, and the rotation matrix is ​​calculated. Translation vector In addition, camera intrinsic parameter matrices can be obtained through camera calibration methods such as Zhang's calibration method, which can be used for subsequent transformations from the camera coordinate system to the image coordinate system.

[0054] The camera's intrinsic parameter matrix K contains the camera's focal length. , Principal point coordinates , Information such as these parameters can be obtained through camera calibration experiments. Given the camera's intrinsic parameter matrix, the coordinates in the camera coordinate system can be transformed to the image coordinate system based on the principle of perspective projection.

[0055] scaling factor The principal point coordinates can be calculated based on the pixel size of the image sensor. These parameters have been determined during camera calibration. They are used to transform the coordinates from the image coordinate system to the image pixel coordinate system.

[0056] Step S04: Based on the target visual data and target millimeter-wave radar data that are consistent in the time dimension, compare and identify the positional differences of the target, and optimize the coordinate correspondence based on the positional differences.

[0057] It should be noted that, according to the feature extraction algorithm, target feature points in the visual image are selected, wherein the target visual data is represented by the visual image. In the embodiments of the present invention, the feature extraction algorithm may be SIFT, SURF, etc., and the target feature points may be corner points, edge feature points, etc. of the target object.

[0058] Based on the target tracking algorithm, the identified target is continuously tracked. The target tracking algorithm includes Kalman filtering, particle filtering, etc.

[0059] At different times, the coordinates of target feature points in the visual image are compared with the corresponding target coordinates after transformation to the image pixel coordinate system by the millimeter-wave radar, and the positional difference between the two is calculated, including the Euclidean distance error between them. , These are the coordinates of the target feature points in the visual image. The coordinates of the target after the millimeter-wave radar is transformed into the image pixel coordinate system are used as the optimization target.

[0060] Based on the positional differences, the least squares method is used to adjust the elements in the transformation matrix to optimize the coordinate correspondence, gradually reduce the error, and achieve more accurate coordinate calibration. For example, the angle parameters and translation vector values ​​in the rotation matrix are fine-tuned so that the error E gradually decreases, thereby achieving more accurate coordinate calibration and significantly improving the spatial matching accuracy of the two sensor data.

[0061] Step S05: During the actual target recognition process, continuously evaluate the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence, and optimize the coordinate correspondence again to ensure spatial alignment between visual and millimeter-wave radar data.

[0062] It is understandable that the false negative rate and false positive rate of target detection are statistically analyzed, as well as the trajectory deviation of target tracking, and it is determined whether the false negative rate, the false positive rate, and the trajectory deviation meet the preset values.

[0063] If not, the current coordinate correspondence is evaluated, and the elements in the transformation matrix are adjusted again based on the evaluation results, forming a closed-loop optimization process to continuously improve the accuracy and reliability of calibration and ensure accurate spatial matching between visual and millimeter-wave radar data.

[0064] It should be noted that the purpose of evaluating the current coordinate correspondence is to determine whether the performance degradation is due to inaccurate coordinate transformation. This can be achieved by comparing the distribution of detection and tracking errors for different targets in different scenarios, combined with historical calibration data and sensor status information. If the problem is determined to be a coordinate transformation issue, then based on the evaluation results, optimization algorithms (such as genetic algorithms, simulated annealing algorithms, and other global optimization algorithms) are used to further optimize and adjust the coordinate transformation and calibration parameters.

[0065] Step S06: Input the aligned visual and millimeter-wave radar data into the trained deep learning-based feature fusion network model to output a comprehensive perception result of the target object.

[0066] In this embodiment of the invention, the feature fusion network model based on deep learning includes a front end, a middle layer, and a back end. In the front end, for visual data, an improved convolutional neural network (CNN) architecture is adopted. Specifically, the number of convolutional layers is increased in the shallow layers of the network to improve the ability to extract detailed features of the image. Specifically, the number of convolutional layers is increased to 3-5 layers, and the kernel size of each convolutional layer adopts a small size design of 3×3, the stride is set to 1, and the padding method is same.

[0067] To enhance the extraction of semantic features (such as object category, shape, and structure) of target objects in complex scenes, the design of convolutional kernels is optimized at a deeper level. Specifically, grouped convolution or depthwise separable convolution techniques are employed to reduce the number of model parameters while improving the network's ability to extract complex semantic features. For example, by introducing an attention mechanism module (such as the channel attention mechanism in SE-Net), the importance weights of different channel features are automatically learned during training, thereby strengthening the extraction of semantic features such as object category and shape.

[0068] For millimeter-wave radar data, feature information is extracted using the spatial distribution characteristics of point clouds through a multilayer perceptron (MLP). The feature information includes at least the target's distance, angle, and velocity. In some other embodiments, feature information can also be extracted using a graph neural network (GNN).

[0069] In the intermediate layer, a cross-modal attention mechanism is introduced. First, visual features and millimeter-wave radar features are encoded separately. Specifically, for visual features, Global Average Pooling (GAP) is used to compress the feature map into a one-dimensional vector, which is then mapped to the same dimension as the millimeter-wave radar features through a fully connected layer for easier subsequent computation. For millimeter-wave radar features, the dimension is also adjusted through a fully connected layer to match the dimension of the visual features.

[0070] Then, the similarity and correlation between the two are calculated through a multi-head attention mechanism, and the importance weights of the two features are learned in different scenarios. For example, in the scenario of long-distance target recognition, the distance and angle features of millimeter-wave radar are automatically given higher weights; in the scenario of close distance and rich target details, the weight of visual semantic features is increased. Based on the weight adjustment, the two features are weighted and fused. Specifically, the outputs of each attention head are concatenated, and then the dimensionality is reduced through a fully connected layer to obtain the fused feature vector, so as to achieve more flexible and efficient feature fusion and give full play to the complementary advantages of the two sensor data.

[0071] In the backend, a multi-task learning approach is adopted, simultaneously performing target detection, target classification, and target trajectory prediction tasks. Specifically, based on the fused feature information, the target detection task determines the position and category of the target in the image through bounding box regression and category prediction; the target classification task further refines the target category; the target trajectory prediction task uses historical data and current features, inputting them into a recurrent neural network (RNN) or its variants (such as GRU, LSTM) to predict the future motion trajectory of the target; finally, the output provides a comprehensive perception result of the target object, providing an accurate and reliable decision-making basis for target recognition scenarios.

[0072] In summary, the data fusion method for vision and millimeter-wave radar proposed in this invention ensures data time synchronization through an event-triggered synchronization strategy, achieves precise spatial alignment through multi-stage joint calibration technology, and completes efficient feature fusion and target perception through a feature fusion network structure based on deep learning. The three work together to form a complete data fusion method for vision and millimeter-wave radar, which significantly improves the system's ability to detect, classify, and track targets in target recognition scenarios.

[0073] Example 2

[0074] Embodiment 2 of the present invention provides a data fusion system 200 for vision and millimeter-wave radar. Please refer to [link / reference]. Figure 2 Here is a structural block diagram of a data fusion method system for vision and millimeter-wave radar. The data fusion system 200 for vision and millimeter-wave radar includes:

[0075] The judgment module 21 is used to acquire historical visual data and historical millimeter-wave radar data, determine the recognition status, and determine whether the recognition status is a triggering event. For visual sensors, the boundary of the monitoring area is defined. When a target object is detected entering the monitoring area, or when the light in the scene changes or the shape and size of the target object changes, it is determined that a triggering event has occurred. For millimeter-wave radar, thresholds for target distance and speed change are set. When the detected change in target distance exceeds the target distance, or the rate of change in speed exceeds the speed change threshold, it is considered a triggering event.

[0076] The control module 22 is used to control the simultaneous acquisition of target visual data and target millimeter-wave radar data when the identification situation is determined to be a trigger event, and to timestamp the data to ensure consistency in the time dimension.

[0077] The conversion module 23 is used to perform preliminary coordinate transformation based on the installation parameters of the visual sensor and the millimeter-wave radar in the target recognition scene, as well as the known geometric relationship, to obtain the coordinate correspondence. In this process, the target coordinates in the millimeter-wave radar coordinate system are first transformed to the world coordinate system using traditional coordinate transformation formulas and transformation matrices, and then further transformed to the image pixel coordinate system.

[0078] The first optimization module 24 is used to compare and identify the positional differences of the target based on the target visual data and the target millimeter-wave radar data that are consistent in the time dimension, and to optimize the coordinate correspondence based on the positional differences.

[0079] The second optimization module 25 is used to continuously evaluate the accuracy of target detection and tracking based on the data fused according to the optimized coordinate correspondence during the actual target recognition process, and to optimize the coordinate correspondence again to ensure spatial alignment between visual and millimeter-wave radar data.

[0080] Input module 26 is used to input aligned visual and millimeter-wave radar data into a trained deep learning-based feature fusion network model and output a comprehensive perception result of the target object. The deep learning-based feature fusion network model includes a front-end, a middle layer, and a back-end. In the front-end, for visual data, an improved convolutional neural network architecture is adopted, wherein the number of convolutional layers is increased in the shallow layers of the network and the convolutional kernel design is optimized in the deep layers of the network. For millimeter-wave radar data, feature information is extracted through a multilayer perceptron by utilizing the spatial distribution characteristics of point clouds.

[0081] In the intermediate layer, a cross-modal attention mechanism is introduced. First, visual features and millimeter-wave radar features are encoded separately. Then, by calculating the similarity and correlation between the two, the importance weights of the two features in different scenarios are learned. Based on the weight adjustment, the two features are weighted and fused.

[0082] On the backend, a multi-task learning approach is adopted, simultaneously performing target detection, target classification, and target trajectory prediction tasks. Specifically, based on the fused feature information, the target detection task determines the position and category of the target in the image through bounding box regression and category prediction; the target classification task further refines the target category; and the target trajectory prediction task uses historical data and current features to predict the future motion trajectory of the target. Finally, a comprehensive perception result of the target object is output.

[0083] Furthermore, in some other embodiments of the present invention, the first optimization module 24 includes:

[0084] The selection unit is used to select target feature points in a visual image according to a feature extraction algorithm, wherein the target visual data is represented by the visual image;

[0085] The tracking unit is used to continuously track the identified target according to the target tracking algorithm;

[0086] The computing unit is used to compare the coordinates of target feature points in the visual image with the corresponding target coordinates after the millimeter-wave radar is transformed into the image pixel coordinate system at different times, and calculate the positional difference between the two.

[0087] The adjustment unit is used to adjust the elements in the transformation matrix using the least squares method according to the position difference, so as to optimize the coordinate correspondence.

[0088] Furthermore, in some other embodiments of the present invention, the second optimization module 25 includes:

[0089] The judgment unit is used to statistically analyze the false negative rate and false positive rate of target detection, as well as the trajectory deviation of target tracking, and to determine whether the false negative rate, the false positive rate, and the trajectory deviation meet preset values.

[0090] The evaluation unit is used to evaluate the current coordinate correspondence when it is determined that the false negative rate, the false positive rate, and the trajectory deviation do not meet the preset values, and then adjust the elements in the transformation matrix again based on the evaluation results.

[0091] Example 3

[0092] Embodiment 3 of the present invention proposes an electronic device, please refer to [link / reference]. Figure 3This is a structural block diagram of an electronic device, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, it implements the data fusion method of vision and millimeter-wave radar as described above.

[0093] In some embodiments, the processor 10 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip, used to run program code stored in memory 20 or process data, such as executing access restriction programs.

[0094] The memory 20 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 can also be an external storage device of the electronic device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, FlashCard, etc., equipped on the electronic device. Furthermore, the memory 20 can include both internal and external storage units of the electronic device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or will be output.

[0095] This invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data fusion method for visual and millimeter-wave radar as described above.

[0096] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0097] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0098] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0099] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0100] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method of data fusion of vision and millimeter wave radar, characterized by, The method comprises: acquiring historical visual data and historical millimeter wave radar data, determining an identification condition, and judging whether the identification condition is a triggering event, wherein, for a visual sensor, a monitoring area boundary is demarcated, and when a target object is detected to enter the monitoring area or light changes or the size of the target object changes in a scene, it is determined that a triggering event occurs; for a millimeter wave radar, a target distance and a speed change threshold are set, and when the distance of the target is detected to change by more than the target distance or the speed change rate exceeds the speed change threshold, it is considered that a triggering event occurs; if yes, then target visual data and target millimeter wave radar data are simultaneously collected and time stamped to ensure consistency in the time dimension; according to installation parameters of the visual sensor and the millimeter wave radar in a target identification scene and a known geometric relationship, preliminary coordinate conversion is performed to obtain a coordinate correspondence, wherein a conventional coordinate conversion formula and a conversion matrix are used to convert target coordinates in a millimeter wave radar coordinate system to a world coordinate system and then to an image pixel coordinate system; according to target visual data and target millimeter wave radar data that are consistent in the time dimension, a position difference of an identified target is compared, and the coordinate correspondence is optimized according to the position difference; in an actual target identification process, the accuracy of data fused according to the optimized coordinate correspondence for target detection and tracking is continuously evaluated, and the coordinate correspondence is optimized again to ensure alignment of visual and millimeter wave radar data in space; the aligned visual and millimeter wave radar data are input into a trained deep learning-based feature fusion network model, and a comprehensive perception result of a target object is output.

2. The method of data fusion of vision and millimeter wave radar according to claim 1, characterized in that, The step of comparing a position difference of an identified target according to target visual data and target millimeter wave radar data that are consistent in the time dimension and optimizing the coordinate correspondence according to the position difference comprises: target feature points in a visual image are selected according to a feature extraction algorithm, wherein the target visual data are embodied by the visual image; an identified target is continuously tracked according to a target tracking algorithm; at different times, coordinates of target feature points in a visual image and coordinates of a corresponding target after conversion of millimeter wave radar data to an image pixel coordinate system are compared, and a position difference between the two is calculated; according to the position difference, elements in a conversion matrix are adjusted by a least squares method to optimize the coordinate correspondence.

3. The method of data fusion of vision and millimeter wave radar according to claim 2, characterized in that, The step of continuously evaluating the accuracy of data fused according to the optimized coordinate correspondence for target detection and tracking in an actual target identification process and optimizing the coordinate correspondence again comprises: a missed detection rate, a false detection rate of target detection, and a trajectory deviation of target tracking are counted, and it is judged whether the missed detection rate, the false detection rate, and the trajectory deviation meet preset values; if no, then the current coordinate correspondence is evaluated, and elements in a conversion matrix are adjusted again according to an evaluation result.

4. The method of data fusion of vision and millimeter wave radar according to claim 3, characterized in that, The deep learning-based feature fusion network model comprises a front end, an intermediate layer and a back end. In the front end, for visual data, an improved convolutional neural network architecture is adopted, wherein the number of convolutional layers is increased in the shallow layer of the network, and the convolution kernel design is optimized in the deep layer of the network. For millimeter wave radar data, the spatial distribution characteristics of point clouds are utilized to extract feature information through a multi-layer perception, wherein in the optimization of the convolution kernel design in the deep layer of the network, a grouped convolution or a depthwise separable convolution technology is adopted to reduce the number of model parameters and enhance the extraction capability of the network for complex semantic features.

5. A data fusion system of vision and millimeter wave radar, characterized by, In the intermediate layer, a cross-modal attention mechanism is introduced, wherein the visual features and the millimeter wave radar features are first encoded, then the similarity and the correlation between the two are calculated, the importance weights of the two features in different scenes are learned, the two features are weighted and fused according to the weight adjustment, and in the back end, a multi-task learning mode is adopted to simultaneously perform target detection, target classification and target trajectory prediction tasks. Based on the fused feature information, the target detection task determines the position and the category of the target in the image through boundary box regression and category prediction; the target classification task further refines the target category; and the target trajectory prediction task predicts the future motion trajectory of the target by using the historical data and the current features. The system comprises: a judgment module for acquiring historical visual data and historical millimeter wave radar data, determining an identification condition and judging whether the identification condition is a trigger event; a control module for controlling the acquisition of target visual data and target millimeter wave radar data at the same time when the identification condition is determined as a trigger event, and time stamping to ensure the consistency of the time dimension; a conversion module for performing preliminary coordinate conversion according to the installation parameters of the visual sensor and the millimeter wave radar in the target identification scene and the known geometric relationship to obtain a coordinate correspondence relationship; a first optimization module for comparing the position difference of the identified target according to the target visual data and the target millimeter wave radar data consistent in the time dimension, and optimizing the coordinate correspondence relationship according to the position difference; 6. A computer-readable storage medium, characterized in that, a second optimization module for continuously evaluating the accuracy of the data fused according to the optimized coordinate correspondence relationship for target detection and tracking in the actual target identification process, and optimizing the coordinate correspondence relationship again to ensure the alignment of the visual and millimeter wave radar data in space; an input module for inputting the aligned visual and millimeter wave radar data into the trained deep learning-based feature fusion network model to output a comprehensive perception result of the target object.

7. An electronic device, comprising: The readable storage medium stores one or more programs which are executed by the processor to implement the data fusion method of the visual and millimeter wave radar according to any one of claims 1-4. The electronic device comprises a memory and a processor, wherein: the memory is used to store computer programs; the processor is used to execute the computer programs. The processor is configured to implement the data fusion method of the visual and millimeter wave radar according to any one of claims 1-4 when executing the computer program stored in the memory.

Images