Collision early warning method based on 4D millimeter wave radar and image fusion technology

Abstract

The invention relates to a collision early warning method and device based on a 4D millimeter wave radar and an image fusion technology, electronic equipment and a storage medium. The method comprises the following steps: acquiring four-dimensional information of a target object based on a 4D millimeter wave radar module, and acquiring visual information of the target object based on an image capture module; performing data preprocessing of filtering and standardization processing on the four-dimensional information and the visual information; performing feature extraction on the four-dimensional information and the visual information after data preprocessing, and performing data fusion on the four-dimensional information and the visual information based on the extracted features; and target object detection and tracking are carried out based on the fused data, and collision early warning detection is completed. According to the invention, collision early warning is realized based on the 4D millimeter wave radar and the image fusion technology, multi-target processing can be carried out, the safety is enhanced, the detection precision and robustness are improved, and the expandability and adaptability are high.

Description

Background Art

[0002] With the rapid development of fields such as autonomous driving, intelligent transportation systems and industrial automation, the demand for collision detection and warning systems is increasing. Traditional collision detection methods mostly rely on visual cameras, ultrasonic sensors or simple radar systems. However, these methods have a series of limitations:

[0003] 1. Visual camera: Affected by environmental factors such as lighting and weather, its accuracy and reliability are not high.

[0004] 2. Ultrasonic sensor: The distance detection range is limited and it is easily affected by environmental noise.

[0005] 3. Simple radar system: Although it can provide certain distance and speed information, its spatial resolution and target recognition capabilities are limited.

[0006] Therefore, one or more methods are needed to solve the above problems.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] The purpose of the present disclosure is to provide a collision warning method, device, electronic device and computer-readable storage medium based on 4D millimeter-wave radar and image fusion technology, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0009] According to one aspect of the present disclosure, a collision warning method based on 4D millimeter wave radar and image fusion technology is provided, comprising:

[0010] The four-dimensional information of the target object is collected based on the 4D millimeter-wave radar module, and the visual information of the target object is collected based on the image capture module;

[0011] Data preprocessing of filtering and standardizing the four-dimensional information and the visual information respectively;

[0012] Extracting features from the four-dimensional information and visual information that have completed data preprocessing, and fusing the four-dimensional information and visual information based on the extracted features;

[0013] Target object detection and tracking are performed based on the fused data to complete collision warning detection.

[0014] In an exemplary embodiment of the present disclosure, the method further includes:

[0015] Collecting four-dimensional information of the target object based on the 4D millimeter-wave radar module, wherein the four-dimensional information includes the three-dimensional spatial position and speed information of the target object;

[0016] The visual information of the target object and the environment image of the target object are collected based on a high-resolution camera.

[0017] In an exemplary embodiment of the present disclosure, the method further includes:

[0018] The 4D millimeter wave radar module is synchronized with the timestamp of the high-resolution camera.

[0019] In an exemplary embodiment of the present disclosure, the method further includes:

[0020] Performing noise filtering on the four-dimensional information and the visual information based on a Kalman filter respectively;

[0021] The four-dimensional information and the visual information are subjected to standardized preprocessing.

[0022] In an exemplary embodiment of the present disclosure, the method further includes:

[0023] Based on the modified CenterFusion deep learning model, feature extraction is performed on the four-dimensional information and visual information that have completed data preprocessing, and data fusion is performed on the four-dimensional information and visual information based on the extracted features.

[0024] In an exemplary embodiment of the present disclosure, the modified CenterFusion deep learning model in the method further includes:

[0025] The network architecture of the model is adjusted by adding convolutional layers and fully connected layers for processing 4D millimeter wave radar data and introducing an attention mechanism;

[0026] Automatically optimizing network parameters of the model based on a genetic algorithm, and determining network hyperparameters through cross-validation;

[0027] ReLU or Leaky ReLU is used as the activation function for the model, and Dropout and L2 regularization are used to prevent overfitting;

[0028] Perform semantic segmentation on the image processing in the center point detection module of the model to identify the center point of each target object, and remove redundant detection results based on non-maximum suppression (NMS);

[0029] Based on the method of the frustum, the center points of the objects in the four-dimensional information and the visual information are associated with each other through the azimuth and distance information in the radar data, and the optimal matching is performed based on the Hungarian algorithm to ensure the uniqueness of the center point association;

[0030] For the associated radar detection data, a series of convolution and pooling operations are performed to generate radar-based feature maps and normalize them;

[0031] Based on the convolutional neural network, the original image of visual information is extracted, and the feature map is post-processed by BatchNormalization and activation function to generate the image feature map;

[0032] Based on the preset neural network, the radar feature map and the image feature map are dynamically fused element by element in a weighted manner, and the fused feature map is used as the input for 3D object detection and attribute regression;

[0033] Multi-scale features are fused based on feature pyramid network FPN.

[0034] In an exemplary embodiment of the present disclosure, the method further includes:

[0035] Based on the fused feature maps, the depth, rotation angle, and speed of the target object are predicted respectively;

[0036] The target is tracked using the multi-target tracking algorithm deepsort based on the predicted depth, rotation angle, and speed of the target object;

[0037] According to the position, velocity and predicted trajectory of the target object, the collision probability with the own vehicle is calculated based on the Bayesian network.

[0038] In one aspect of the present disclosure, a collision warning device based on 4D millimeter wave radar and image fusion technology is provided, comprising:

[0039] A data acquisition module, used to collect four-dimensional information of the target object based on the 4D millimeter wave radar module, and to collect visual information of the target object based on the image capture module;

[0040] A preprocessing module, for data preprocessing of filtering and standardizing the four-dimensional information and the visual information respectively;

[0041] A data fusion module, used to extract features from the four-dimensional information and visual information that have completed data preprocessing, and to fuse the four-dimensional information and visual information based on the extracted features;

[0042] The collision warning module is used to detect and track target objects based on the fused data and complete collision warning detection.

[0043] In one aspect of the present disclosure, there is provided an electronic device, comprising:

[0044] Processor; and

[0045] A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of the above items.

[0046] In one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the above items is implemented.

[0047] In an exemplary embodiment of the present disclosure, a collision warning method based on 4D millimeter wave radar and image fusion technology is disclosed, the method comprising: collecting four-dimensional information of the target object based on the 4D millimeter wave radar module, and collecting visual information of the target object based on the image capture module; data preprocessing of filtering and standardizing the four-dimensional information and visual information respectively; feature extraction of the four-dimensional information and visual information after data preprocessing respectively, and data fusion of the four-dimensional information and visual information based on the extracted features; target object detection and tracking based on the fused data to complete collision warning detection. The present disclosure implements collision warning based on 4D millimeter wave radar and image fusion technology, can perform multi-target processing, enhances safety, improves detection accuracy and robustness, and has strong scalability and adaptability.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other features and advantages of the present disclosure will become more apparent by describing in detail example embodiments thereof with reference to the attached drawings.

[0050] Figure 1 A flowchart of a collision warning method based on 4D millimeter wave radar and image fusion technology according to an exemplary embodiment of the present disclosure is shown;

[0051] Figure 2 A main algorithm logic diagram of a collision warning method based on 4D millimeter wave radar and image fusion technology according to an exemplary embodiment of the present disclosure is shown;

[0052] Figure 3 A structural block diagram of a collision warning device based on 4D millimeter wave radar and image fusion technology according to an exemplary embodiment of the present disclosure is shown;

[0053] Figure 4 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure;

[0054] Figure 5 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0056] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, materials, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0057] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0058] In this example embodiment, a collision warning method based on 4D millimeter wave radar and image fusion technology is first provided; Figure 1 As shown in , the collision warning method based on 4D millimeter wave radar and image fusion technology may include the following steps:

[0059] Step S110, collecting four-dimensional information of the target object based on the 4D millimeter wave radar module, and collecting visual information of the target object based on the image capture module;

[0060] Step S120, data preprocessing of filtering and standardizing the four-dimensional information and the visual information respectively;

[0061] Step S130, extracting features from the four-dimensional information and the visual information that have completed data preprocessing, and fusing the four-dimensional information and the visual information based on the extracted features;

[0062] Step S140: Detect and track the target object based on the fused data to complete collision warning detection.

[0063] In an exemplary embodiment of the present disclosure, a collision warning method based on 4D millimeter wave radar and image fusion technology is disclosed, the method comprising: collecting four-dimensional information of the target object based on the 4D millimeter wave radar module, and collecting visual information of the target object based on the image capture module; data preprocessing of filtering and standardizing the four-dimensional information and visual information respectively; feature extraction of the four-dimensional information and visual information after data preprocessing respectively, and data fusion of the four-dimensional information and visual information based on the extracted features; target object detection and tracking based on the fused data to complete collision warning detection. The present disclosure implements collision warning based on 4D millimeter wave radar and image fusion technology, can perform multi-target processing, enhances safety, improves detection accuracy and robustness, and has strong scalability and adaptability.

[0064] Next, a collision warning method based on 4D millimeter wave radar and image fusion technology in this example embodiment will be further described.

[0065] Embodiment 1:

[0066] like Figure 2 The main algorithm logic diagram of the collision warning method is shown. In the embodiment of this example, the present disclosure mainly solves the following problems:

[0067] 1. Improve detection accuracy: Existing collision warning detection methods are often affected by environmental conditions (such as light, weather, etc.) and have low accuracy. The present invention can provide high-accuracy detection under various environmental conditions by using 4D millimeter-wave radar and image fusion technology.

[0068] 2. Enhanced target recognition capability: Traditional radar systems often have limitations in target recognition. 4D millimeter-wave radar can provide richer spatial and speed information, which can be fused with image data to more accurately identify different types of targets.

[0069] 3. Real-time performance and response speed: Since the collision warning system needs to work in real time, the response time of the system is very critical. The present invention optimizes the data processing algorithm to ensure that data analysis can be completed and warnings can be triggered in a very short time.

[0070] 4. Wide applicability: In addition to being used in vehicle safety and autonomous driving systems, this method can also be widely used in many fields such as drone navigation, industrial automation, and intelligent transportation systems.

[0071] Through the present disclosure, the accuracy, reliability and real-time performance of collision warning detection can be significantly improved, thereby more effectively avoiding or reducing the occurrence of collision events.

[0072] In step S110, four-dimensional information of the target object may be collected based on the 4D millimeter wave radar module, and visual information of the target object may be collected based on the image capture module.

[0073] In the embodiment of this example, the method further includes:

[0074] Collecting four-dimensional information of the target object based on the 4D millimeter-wave radar module, wherein the four-dimensional information includes the three-dimensional spatial position and speed information of the target object;

[0075] The visual information of the target object and the environment image of the target object are collected based on a high-resolution camera.

[0076] In the embodiment of this example, the method further includes:

[0077] The 4D millimeter wave radar module is synchronized with the timestamp of the high-resolution camera.

[0078] In step S120, the four-dimensional information and the visual information may be subjected to data preprocessing by filtering and standardizing, respectively.

[0079] In the embodiment of this example, the method further includes:

[0080] Performing noise filtering on the four-dimensional information and the visual information based on a Kalman filter respectively;

[0081] The four-dimensional information and the visual information are subjected to standardized preprocessing.

[0082] In step S130, feature extraction may be performed on the four-dimensional information and visual information that have completed data preprocessing, and data fusion may be performed on the four-dimensional information and visual information based on the extracted features.

[0083] In the embodiment of this example, the method further includes:

[0084] Based on the modified CenterFusion deep learning model, feature extraction is performed on the four-dimensional information and visual information that have completed data preprocessing, and data fusion is performed on the four-dimensional information and visual information based on the extracted features.

[0085] In the embodiment of this example, the modified CenterFusion deep learning model in the method further includes:

[0086] The network architecture of the model is adjusted by adding convolutional layers and fully connected layers for processing 4D millimeter wave radar data and introducing an attention mechanism;

[0087] Automatically optimizing network parameters of the model based on a genetic algorithm, and determining network hyperparameters through cross-validation;

[0088] ReLU or Leaky ReLU is used as the activation function for the model, and Dropout and L2 regularization are used to prevent overfitting;

[0089] Perform semantic segmentation on the image processing in the center point detection module of the model to identify the center point of each target object, and remove redundant detection results based on non-maximum suppression (NMS);

[0090] Based on the method of the frustum, the center points of the objects in the four-dimensional information and the visual information are associated with each other through the azimuth and distance information in the radar data, and the optimal matching is performed based on the Hungarian algorithm to ensure the uniqueness of the center point association;

[0091] For the associated radar detection data, a series of convolution and pooling operations are performed to generate radar-based feature maps and normalize them;

[0092] Based on the convolutional neural network, the original image of visual information is extracted, and the feature map is post-processed by BatchNormalization and activation function to generate the image feature map;

[0093] Based on the preset neural network, the radar feature map and the image feature map are dynamically fused element by element in a weighted manner, and the fused feature map is used as the input for 3D object detection and attribute regression;

[0094] Multi-scale features are fused based on feature pyramid network FPN.

[0095] In step S140, target object detection and tracking may be performed based on the fused data to complete collision warning detection.

[0096] In the embodiment of this example, the method further includes:

[0097] Based on the fused feature maps, the depth, rotation angle, and speed of the target object are predicted respectively;

[0098] The target is tracked using the multi-target tracking algorithm deepsort based on the predicted depth, rotation angle, and speed of the target object;

[0099] According to the position, velocity and predicted trajectory of the target object, the collision probability with the own vehicle is calculated based on the Bayesian network.

[0100] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0101] In addition, in this exemplary embodiment, a collision warning device based on 4D millimeter wave radar and image fusion technology is also provided. Figure 3 As shown, the collision warning device 300 based on 4D millimeter wave radar and image fusion technology may include: a data acquisition module 310, a preprocessing module 320, a data fusion module 330 and a collision warning module 340. Among them:

[0102] A data acquisition module 310, used to acquire four-dimensional information of the target object based on the 4D millimeter wave radar module, and to acquire visual information of the target object based on the image capture module;

[0103] A preprocessing module 320, for data preprocessing of filtering and standardizing the four-dimensional information and the visual information respectively;

[0104] A data fusion module 330 is used to extract features from the four-dimensional information and the visual information after data preprocessing, and to fuse the four-dimensional information and the visual information based on the extracted features;

[0105] The collision warning module 340 is used to detect and track the target object based on the fused data to complete the collision warning detection.

[0106] Embodiment 2:

[0107] In the embodiment of this example, the method disclosed herein is mainly completed by a collision warning detection system, which mainly includes the following three core components:

[0108] 4D millimeter-wave radar module: responsible for obtaining four-dimensional information of the target object: three-dimensional spatial position (x, y, z) and speed.

[0109] Image capture module: uses a high-resolution camera to capture real-time visual information.

[0110] Data processing and analysis unit: Uses a modified CenterFusion deep learning model for data fusion and collision warning analysis.

[0111] In the embodiment of this example, the data acquisition and preprocessing steps include:

[0112] 1.4D millimeter wave radar data acquisition, hardware configuration: select a 4D millimeter wave radar with high resolution and large field of view. Data acquisition: the radar will obtain the distance, direction, speed and altitude information of the target. Time synchronization: ensure that the timestamps of the radar data and the camera data are aligned.

[0113] 2. Image data acquisition,Hardware configuration: Use a high-resolution camera.,Data acquisition: The camera will capture the environment image.

[0114] 3. Data preprocessing, noise filtering: noise filtering of radar and image data, using Kalman filter. Data normalization: normalize radar and image data to the same range.

[0115] In the embodiment of this example, in feature extraction and data fusion, the modification and optimization of the CenterFusion network includes:

[0116] 1. Network architecture adjustment: In the basic network architecture of CenterFusion, convolutional layers and fully connected layers specifically for processing 4D millimeter-wave radar data were added. An attention mechanism was introduced so that the network can focus more on key features.

[0117] 2. Parameter optimization: Use genetic algorithms to automatically optimize network parameters. Determine the optimal network hyperparameters through cross-validation.

[0118] 3. Activation function and regularization: Use ReLU or Leaky ReLU as the activation function. Use Dropout and L2 regularization to prevent overfitting.

[0119] 4. Center point detection algorithm: Use the center point detection module in the modified CenterFusion network to perform semantic segmentation on the image to identify the center point of each target object. Apply non-maximum suppression (NMS) to remove redundant detection results.

[0120] 5. Data association algorithm: A frustum-based method is used to associate radar detections with the center points of objects in the image using the azimuth and distance information in the radar data. The Hungarian algorithm is used for optimal matching to ensure that each radar point is associated with only one image center point.

[0121] 6. Radar feature map generation: Use the associated radar detection data to generate radar-based feature maps through a series of convolution and pooling operations. The generated radar feature maps are normalized.

[0122] 7. Image feature map generation: Use convolutional neural network (CNN) to extract features from the original image. Apply batch normalization and activation function to post-process the feature map.

[0123] 8. Feature map fusion: The radar feature map is fused with the image feature map element by element, with the weights dynamically determined by a small neural network. The fused feature map is used as the input for subsequent 3D object detection and attribute regression.

[0124] 9. Multi-scale feature fusion:

[0125] In order to capture objects of different sizes and distances, multi-scale feature maps are fused. The fusion of multi-scale features is achieved through the Feature Pyramid Network (FPN).

[0126] In the embodiment of this example, target object detection and attribute regression include:

[0127] 1.3D object detection, fused feature map: Fuse the radar and image feature maps. Object detection: Use the fused features for 3D object detection.

[0128] 2. Attribute regression, Depth regression: predict the depth of the target object. Rotation regression: predict the rotation angle of the target object. Speed ​​regression: predict the speed of the target object.

[0129] In the embodiment of this example, the collision warning includes:

[0130] Target tracking: Use the multi-target tracking algorithm deepsort for target tracking.

[0131] Collision probability calculation: Based on the target’s position, velocity, and predicted trajectory, a Bayesian network is used to calculate the collision probability with the ego vehicle.

[0132] In the embodiment of this example, the present disclosure can achieve the following beneficial effects:

[0133] 1. Improve detection accuracy

[0134] Multi-dimensional information fusion: Through the fusion of 4D millimeter-wave radar and image data, the system can obtain information from multiple dimensions (distance, direction, speed, altitude and image features), thereby improving the accuracy of target detection.

[0135] Reduce false detection and missed detection: Using the center point detection and data association algorithm of the CenterFusion network can effectively reduce false detection and missed detection.

[0136] 2. Improve robustness

[0137] Adapt to complex environments: 4D millimeter-wave radar can work in various weather and lighting conditions. After being fused with image data, the robustness of the system will be significantly improved.

[0138] Dynamic Adjustment: Dynamic weight allocation in the feature map fusion stage enables the system to dynamically adjust according to different scenarios and target characteristics.

[0139] 3. Real-time performance

[0140] Fast processing: Through the optimized CenterFusion network and efficient data association algorithm, the entire system can run in real time or near real time.

[0141] Low latency: The efficient algorithm design of feature extraction and data fusion ensures the low latency performance of the system, which is especially important for collision warning.

[0142] 4. Scalability and adaptability

[0143] Modular design: The entire system adopts modular design, which is easy to expand and upgrade.

[0144] Multi-scenario adaptability: Through parameter adjustment and network training, the system can adapt to a variety of different application scenarios, such as urban driving, highway driving, etc.

[0145] 5. Enhanced security

[0146] Collision Warning: Through precise target detection and attribute regression, the system can accurately predict the possibility of a collision, thereby issuing a warning in advance and increasing driving safety.

[0147] 6. Multi-target processing: The system can track and analyze multiple targets simultaneously, and has high application value in complex traffic environments.

[0148] Embodiment three:

[0149] In the embodiment of this example, the fusion of 4D millimeter wave radar and image data includes:

[0150] The first step of data fusion is to train and optimize the CenterFusion network. In this process, we use a large amount of actual driving scene data (including radar and image data) for pre-training. In particular, we use a series of public and self-collected data sets (such as nuScenes, Waymo Open Dataset, etc.) for model training. The loss function adopts a multi-task learning approach, including cross entropy loss for target detection and mean square error loss for attribute regression. After training, the model weights are saved in the CenterFusion network. When new radar and image data are input, the fused feature representation can be obtained by reasoning through the model. The similarity between these feature representations or target attributes can be calculated and evaluated through subsequent algorithms.

[0151] In the embodiment of this example, collision warning and target tracking include:

[0152] Since the CenterFusion network has a large number of parameters, real-time reasoning may require more computing resources. Therefore, in practical applications, we pre-process the radar and image data of some common driving scenarios (such as dense traffic, frequent pedestrians, etc.) and store their fusion features in the database. When new radar and image data are input into the system, the fusion feature representation of the data is first obtained through CenterFusion model reasoning. Then, through an efficient multi-target tracking algorithm DeepSORT and Bayesian network, we can quickly find the preset scene that is most similar to the current scene from the database and calculate the collision probability based on it.

[0153] In this way, through these two main components, our collision warning detection system can not only achieve high-precision target detection and attribute regression, but also effectively perform collision warning, thereby improving road traffic safety.

[0154] The specific details of each of the above-mentioned collision warning device modules based on 4D millimeter-wave radar and image fusion technology have been described in detail in the corresponding collision warning method based on 4D millimeter-wave radar and image fusion technology, so they will not be repeated here.

[0155] It should be noted that although several modules or units of a collision warning device 300 based on 4D millimeter wave radar and image fusion technology are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0156] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0157] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0158] Refer to the following Figure 4 An electronic device 400 according to such an embodiment of the present invention is described. Figure 4 The electronic device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0159] like Figure 4 As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include but are not limited to: the at least one processing unit 410, the at least one storage unit 420, a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410), and a display unit 440.

[0160] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 410 can perform the following steps: Figure 1 Steps S110 to S140 shown in FIG.

[0161] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 4201 and / or a cache storage unit 4202 , and may further include a read-only storage unit (ROM) 4203 .

[0162] The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0163] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0164] The electronic device 400 may also communicate with one or more external devices 470 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 450. In addition, the electronic device 400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0165] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0166] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0167] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0168] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0169] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0170] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0171] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0172] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0173] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0174] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A collision warning method based on 4D millimeter wave radar and image fusion technology, It is characterized in that The method comprises: The four-dimensional information of the target object is collected based on the 4D millimeter-wave radar module, and the visual information of the target object is collected based on the image capture module; Data preprocessing of filtering and standardizing the four-dimensional information and the visual information respectively; Extracting features from the four-dimensional information and visual information that have completed data preprocessing, and fusing the four-dimensional information and visual information based on the extracted features; Target object detection and tracking are performed based on the fused data to complete collision warning detection.

2. The method according to claim 1, It is characterized in that The method further comprises: Collecting four-dimensional information of the target object based on the 4D millimeter-wave radar module, wherein the four-dimensional information includes the three-dimensional spatial position and speed information of the target object; The visual information of the target object and the environment image of the target object are collected based on a high-resolution camera.

3. The method according to claim 2, It is characterized in that The method further comprises: The 4D millimeter wave radar module is synchronized with the timestamp of the high-resolution camera.

4. The method according to claim 1, It is characterized in that The method further comprises: Performing noise filtering on the four-dimensional information and the visual information based on a Kalman filter respectively; The four-dimensional information and the visual information are subjected to standardized preprocessing.

5. The method according to claim 1, It is characterized in that The method further comprises: Based on the modified CenterFusion deep learning model, feature extraction is performed on the four-dimensional information and visual information that have completed data preprocessing, and data fusion is performed on the four-dimensional information and visual information based on the extracted features.

6. The method according to claim 5, It is characterized in that The modified CenterFusion deep learning model in the method also includes: The network architecture of the model is adjusted by adding convolutional layers and fully connected layers for processing 4D millimeter wave radar data and introducing an attention mechanism; Automatically optimizing network parameters of the model based on a genetic algorithm, and determining network hyperparameters through cross-validation; ReLU or Leaky ReLU is used as the activation function for the model, and Dropout and L2 regularization are used to prevent overfitting; Perform semantic segmentation on the image processing in the center point detection module of the model to identify the center point of each target object, and remove redundant detection results based on non-maximum suppression (NMS); Based on the method of the frustum, the center points of the objects in the four-dimensional information and the visual information are associated with each other through the azimuth and distance information in the radar data, and the optimal matching is performed based on the Hungarian algorithm to ensure the uniqueness of the center point association; For the associated radar detection data, a series of convolution and pooling operations are performed to generate radar-based feature maps and normalize them; Based on the convolutional neural network, the original image of visual information is extracted, and the feature map is post-processed by Batch Normalization and activation function to generate an image feature map; Based on the preset neural network, the radar feature map and the image feature map are dynamically fused element by element in a weighted manner, and the fused feature map is used as the input for 3D object detection and attribute regression; Multi-scale features are fused based on feature pyramid network FPN.

7. The method according to claim 5, It is characterized in that The method further comprises: Based on the fused feature maps, the depth, rotation angle, and speed of the target object are predicted respectively; The target is tracked using the multi-target tracking algorithm deepsort based on the predicted depth, rotation angle, and speed of the target object; According to the position, velocity and predicted trajectory of the target object, the collision probability with the own vehicle is calculated based on the Bayesian network.

8. A collision warning device based on 4D millimeter wave radar and image fusion technology, It is characterized in that The device comprises: A data acquisition module, used to collect four-dimensional information of the target object based on the 4D millimeter wave radar module, and to collect visual information of the target object based on the image capture module; A preprocessing module, for data preprocessing of filtering and standardizing the four-dimensional information and the visual information respectively; A data fusion module, used to extract features from the four-dimensional information and visual information that have completed data preprocessing, and to fuse the four-dimensional information and visual information based on the extracted features; The collision warning module is used to detect and track target objects based on the fused data and complete collision warning detection.

9. An electronic device, It is characterized in that include Processor; and A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images