Single-machine obstacle avoidance method and system based on millimeter wave radar and camera

Abstract

The invention relates to a single-machine obstacle avoidance method and system based on a millimeter wave radar and a camera, and the method comprises the steps: transmitting a frequency modulation continuous wave signal through the millimeter wave radar, collecting and preprocessing environment original data, and outputting a target list; based on the target list, when the target meets a preset signal-to-noise ratio condition and the speed exceeds a preset speed threshold value, triggering a camera to collect an image, performing real-time classification on the image through a lightweight convolutional neural network model, and outputting a classification result and a bounding box; performing space-time alignment on a target list and a classification result, dynamically allocating weights of radar data and visual data by adopting a confidence weighted fusion algorithm, associating the same target through distance calculation, and outputting a fused target attribute; based on the fused target attributes, constructing a three-dimensional decision matrix, and generating an obstacle avoidance strategy according to a preset priority rule; and converting the obstacle avoidance strategy into a bus control instruction, and controlling the vehicle to execute a corresponding alarm or brake response action.

Description

Technical Field

[0001] This application relates to the field of millimeter-wave radar technology, and in particular to a single-machine obstacle avoidance method and system based on millimeter-wave radar and camera. Background Technology

[0002] Obstacle avoidance systems typically incorporate multiple sensors to enhance system reliability. Currently, commonly used sensors include cameras, LiDAR, and millimeter-wave radar. Each of these sensors has its own advantages and disadvantages, and they complement each other. Therefore, efficiently fusing data from multiple sensors has naturally become a hot topic in perception algorithm research.

[0003] Research on millimeter-wave radar perception algorithms started relatively late, and there are not many publicly available databases. Therefore, current research on multi-sensor fusion mainly focuses on fusing data from cameras (images) and lidar (point clouds). With the increasing application of millimeter-wave radar in autonomous vehicles, how to fuse its data with images has become a problem that needs to be solved.

[0004] Based on this, this application provides a single-machine obstacle avoidance method and system based on millimeter-wave radar and camera. Summary of the Invention

[0005] To address the growing application of millimeter-wave radar in autonomous vehicles and the challenge of fusing its data with images, this application provides a standalone obstacle avoidance method and system based on millimeter-wave radar and a camera.

[0006] Firstly, this application provides a single-machine obstacle avoidance method based on millimeter-wave radar and a camera, employing the following technical solution: including:

[0007] The system transmits frequency-modulated continuous wave signals using millimeter-wave radar, collects raw environmental data, preprocesses it, and outputs a target list containing location, velocity, and signal-to-noise ratio.

[0008] Based on the target list, when a target meets the preset signal-to-noise ratio condition and its speed exceeds the preset speed threshold, the camera is triggered to capture an image, and the image is classified in real time through a lightweight convolutional neural network model, outputting the classification result and bounding box.

[0009] The target list and the classification results are spatiotemporally aligned. A confidence-weighted fusion algorithm is used to dynamically allocate the weights of radar data and visual data. The same target is associated through distance calculation, and the fused target attributes are output.

[0010] Based on the fused target attributes, a three-dimensional decision matrix is ​​constructed according to obstacle type, distance level and collision time, and an obstacle avoidance strategy is generated according to a preset priority rule.

[0011] The obstacle avoidance strategy is converted into bus control commands, which include the obstacle type and the distance level, and control the vehicle to perform corresponding alarm or braking response actions.

[0012] Preferably, the step of transmitting frequency-modulated continuous wave signals via millimeter-wave radar, collecting raw environmental data and preprocessing it to output a target list including position, velocity, and signal-to-noise ratio includes:

[0013] The system transmits frequency-modulated continuous wave signals using millimeter-wave radar, collects raw environmental data, and analyzes it into a range-Doppler matrix.

[0014] Different noise estimation window sizes are set for near-field and far-field targets, and the mean background noise of the range-Doppler matrix is ​​calculated by sliding window.

[0015] Apply a dynamic threshold to each distance cell to retain target points with a signal-to-noise ratio exceeding the threshold and valid velocity.

[0016] Neighboring points are selected by Euclidean distance and velocity consistency, adjacent point clouds are merged to form target clusters, and the target list is output.

[0017] Preferably, the triggering of the camera to capture images includes:

[0018] The camera power supply is controlled by a switching device connected to the GPIO pin of the millimeter-wave radar. When the radar detects a valid target, the switching device is triggered to turn on and wake up the camera.

[0019] When the radar does not detect a valid target, the camera power is cut off to reduce power consumption.

[0020] Preferably, the lightweight convolutional neural network model is a MobileNet series model, which reduces the amount of computation by lowering the resolution of the input image, removes redundant convolutional channels by channel pruning to compress the number of model parameters, uses low-precision quantization to convert the floating-point model into an integer model, and utilizes the SIMD instructions of the embedded processor to accelerate inference operations.

[0021] 5. The single-machine obstacle avoidance method based on millimeter-wave radar and camera according to claim 1, characterized in that, after performing spatiotemporal alignment of the target list and the classification result, employing a confidence-weighted fusion algorithm to dynamically allocate the weights of radar data and visual data, associating the same target through distance calculation, and outputting the fused target attributes, it further includes:

[0022] The Kalman filter algorithm is used to predict the fused target trajectory. A state vector is constructed based on the target position and velocity. The target position at the current moment is predicted through the state transition matrix. The prediction result is corrected by combining the observation values ​​to obtain the predicted trajectory.

[0023] Based on the target attributes and the predicted trajectory, a three-dimensional decision matrix is ​​constructed according to the obstacle type, distance level and collision time, and an obstacle avoidance strategy is generated according to a preset priority rule.

[0024] Preferably, the step of employing a confidence-weighted fusion algorithm to dynamically allocate weights between radar data and visual data includes:

[0025] The radar data weights and visual data weights are dynamically allocated, wherein the radar data weights are positively correlated with the target signal-to-noise ratio and negatively correlated with the ambient light intensity, and the visual data weights are positively correlated with the classification confidence and the ambient light intensity.

[0026] After normalization, radar data and visual data are fused, and the Euclidean distance between the target in the radar data and the center of the visual detection box is calculated. Targets with a distance less than a preset threshold and the same category are identified as the same object.

[0027] Preferably, the decision dimensions of the three-dimensional decision matrix include: the obstacle type, the distance level, and the collision time;

[0028] The obstacle types include pedestrians, vehicles, and static obstacles. The distance level quantifies the target distance into multiple levels. The collision time is calculated based on the target distance and relative speed and divided into multiple threshold levels. The priority rule is that pedestrians are prioritized over dynamic vehicles, dynamic vehicles are prioritized over static obstacles, and the collision time threshold is dynamically adjusted according to environmental conditions.

[0029] Secondly, this application discloses a single-unit obstacle avoidance device based on millimeter-wave radar and a camera, which adopts the following technical solution, including:

[0030] The radar signal module is used to transmit frequency-modulated continuous wave signals through millimeter-wave radar, collect raw environmental data and preprocess it, and output a target list including position, velocity and signal-to-noise ratio.

[0031] The image data module is used to trigger the camera to capture images based on the target list when the target meets the preset signal-to-noise ratio condition and the speed exceeds the preset speed threshold, and to perform real-time classification of the images through a lightweight convolutional neural network model, and output the classification results and bounding boxes.

[0032] The spatiotemporal alignment module is used to perform spatiotemporal alignment between the target list and the classification results. It adopts a confidence-weighted fusion algorithm to dynamically allocate the weights of radar data and visual data, and associates the same target through distance calculation to output the fused target attributes.

[0033] The obstacle avoidance strategy module is used to construct a three-dimensional decision matrix based on the fused target attributes, obstacle type, distance level and collision time, and generate an obstacle avoidance strategy according to a preset priority rule.

[0034] The action response module is used to convert the obstacle avoidance strategy into bus control commands, which include the obstacle type and the distance level, and control the vehicle to perform corresponding alarm or braking response actions.

[0035] Thirdly, this application also provides a control device, the device comprising:

[0036] It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the single-machine obstacle avoidance method based on millimeter-wave radar and camera described above.

[0037] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above for a stand-alone obstacle avoidance method based on millimeter-wave radar and a camera.

[0038] In summary, this application achieves time synchronization between radar and visual data through hardware synchronization signals, and establishes coordinate transformation relationships based on sensor extrinsic parameter calibration to complete spatial alignment. The weights of both are dynamically allocated: radar weights are positively correlated with the target signal-to-noise ratio and negatively correlated with ambient light intensity, while visual weights are positively correlated with classification confidence and ambient light intensity. After normalization, the data are fused, and the same target is associated using Euclidean distance calculation. This fusion method fully leverages the advantages of radar in distance and velocity detection under adverse weather conditions and the target classification advantages of vision. The dynamic weight allocation can adapt to complex environments such as changes in lighting, improving the accuracy and robustness of target attribute detection. This provides reliable fused target attributes for the subsequent 3D decision matrix, thereby optimizing obstacle avoidance strategy generation and enhancing the environmental adaptability and decision reliability of the vehicle's single-unit obstacle avoidance system. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a single-machine obstacle avoidance method based on millimeter-wave radar and a camera.

[0040] Figure 2 This is a structural block diagram of a standalone obstacle avoidance device based on millimeter-wave radar and a camera. Detailed Implementation

[0041] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0042] Current technological limitations include: Single sensors have limitations: millimeter-wave radar is accurate in ranging but cannot identify obstacle types (such as pedestrians, vehicles, and roadblocks); cameras can classify and identify obstacles, but are greatly affected by lighting and weather conditions (e.g., they fail at night or in foggy weather). Multi-sensor fusion is costly: existing solutions rely on high-performance processors (such as GPUs) for data fusion, resulting in high hardware costs; the algorithms are complex (such as deep learning) and difficult to run in real-time on embedded devices. Obstacle avoidance strategies are limited: traditional methods only provide alarms or emergency braking, lacking dynamic path planning (e.g., avoiding low obstacles).

[0043] The core of this application is to overcome the limitations of a single sensor through multi-sensor data fusion, combining the precise ranging capability of millimeter-wave radar with the semantic recognition capability of vision to achieve a highly robust and low-cost obstacle avoidance function. It primarily involves the millimeter-wave radar emitting frequency-modulated continuous wave signals and preprocessing them to output a target list. When preset conditions are met, the camera is triggered to acquire images. After classification by a lightweight convolutional neural network, the radar and visual data are spatiotemporally aligned, and weights are dynamically assigned using a confidence-weighted fusion algorithm. The fusion attributes are associated with the same target, and the target trajectory is predicted using Kalman filtering. Based on obstacle type, distance level, and collision time, a three-dimensional decision matrix is ​​constructed to generate an obstacle avoidance strategy, which is ultimately converted into bus commands to control the vehicle's response. This technology utilizes multi-sensor collaborative perception, dynamically triggering cameras to reduce power consumption, optimizing lightweight models to improve real-time performance, enhancing the reliability of target detection in complex environments through confidence-weighted fusion, and prioritizing the three-dimensional decision matrix to ensure obstacle avoidance safety, effectively improving the accuracy and adaptability of a single-machine obstacle avoidance system.

[0044] Reference Figure 1 The embodiments of this application include at least steps S10 to S50.

[0045] S10 transmits frequency-modulated continuous wave signals via millimeter-wave radar, collects raw environmental data, preprocesses it, and outputs a target list containing position, velocity, and signal-to-noise ratio.

[0046] S20: Based on the target list, when the target meets the preset signal-to-noise ratio condition and the speed exceeds the preset speed threshold, the camera is triggered to acquire images, and the images are classified in real time through a lightweight convolutional neural network model, outputting the classification results and bounding boxes.

[0047] S30 performs spatiotemporal alignment between the target list and the classification results, adopts a confidence-weighted fusion algorithm, dynamically allocates the weights of radar data and visual data, and associates the same target through distance calculation, outputting the fused target attributes.

[0048] S40, based on the fused target attributes, constructs a three-dimensional decision matrix according to obstacle type, distance level and collision time, and generates obstacle avoidance strategy according to preset priority rules.

[0049] S50 converts obstacle avoidance strategies into bus control commands, which include obstacle type and distance level, and control the vehicle to perform corresponding alarm or braking response actions.

[0050] Specifically, the system utilizes millimeter-wave radar to collect and preprocess environmental data, outputting a target list including position, velocity, and signal-to-noise ratio. Based on this target list, cameras are dynamically triggered, and images are classified in real-time using a lightweight convolutional neural network, with the results output. Radar and visual data are spatiotemporally aligned, and a confidence-weighted fusion algorithm is used to dynamically allocate weights, associating the same target with fused attributes. A 3D decision matrix is ​​constructed based on these fused attributes to generate an obstacle avoidance strategy. The strategy is then converted into bus commands to control the vehicle's response. This process effectively improves the accuracy and adaptability of a single-machine obstacle avoidance system through multi-sensor collaborative perception, dynamic triggering to reduce power consumption, a lightweight model to improve real-time performance, a fusion algorithm to enhance target detection reliability, and 3D decision-making to ensure obstacle avoidance safety.

[0051] In some embodiments, step S10 specifically includes the following steps: transmitting a frequency-modulated continuous wave signal via millimeter-wave radar, collecting raw environmental data and parsing it into a range-Doppler matrix; setting different noise estimation window sizes for near-field and far-field targets respectively, and calculating the mean background noise of the range-Doppler matrix through a sliding window; applying a dynamic threshold to each range cell, retaining target points with a signal-to-noise ratio exceeding the threshold and valid velocity; filtering neighboring points through Euclidean distance and velocity consistency, merging adjacent point clouds to form a target cluster, and outputting a target list.

[0052] Specifically, environmental data is acquired by transmitting frequency-modulated continuous wave signals via millimeter-wave radar and analyzed into a range-Doppler matrix. Differentiated noise estimation windows are set for near-field and far-field targets, and a sliding window is used to calculate the mean background noise to adapt to the noise characteristics at different distances. Dynamic thresholds are applied to each range cell to retain target points with acceptable signal-to-noise ratios and effective velocities. Neighboring points are selected based on the consistency of Euclidean distance and velocity, and adjacent point clouds are merged to form target clusters and output a target list. This process improves target detection accuracy through regional noise suppression, and the dynamic threshold and clustering algorithm effectively filter out false targets, ensuring that the output target list contains accurate position, velocity, and signal-to-noise ratio information. This provides a reliable radar perception foundation for subsequent camera triggering and multi-sensor data fusion, enhancing the robustness of target detection in harsh environments.

[0053] Furthermore, triggering the camera to acquire images includes: controlling the camera power supply by connecting a switching device to the GPIO pin of the millimeter-wave radar; when the radar detects a valid target, the switching device is turned on to wake up the camera; when the radar does not detect a valid target, the camera power supply is turned off to reduce power consumption.

[0054] In some embodiments, the lightweight convolutional neural network model is a MobileNet series model, which reduces computation by lowering the input image resolution, removes redundant convolutional channels by pruning channels to compress the number of model parameters, uses low-precision quantization to convert the floating-point model into an integer model, and utilizes the SIMD instructions of the embedded processor to accelerate inference operations.

[0055] Specifically, the lightweight convolutional neural network model adopts the MobileNet series architecture. It reduces computation by lowering the input image resolution, removes redundant convolutional channels through channel pruning to compress the number of model parameters, converts the floating-point model to an integer model using low-precision quantization, and accelerates inference operations using embedded processor SIMD instructions. These optimizations significantly improve the real-time processing capabilities of the embedded platform while maintaining classification accuracy, meeting the low-latency requirements for image classification in automotive environments. This provides fast and accurate visual classification results for radar and visual data fusion, thereby supporting the efficient generation of subsequent obstacle avoidance decisions and enhancing the real-time performance and engineering practicality of stand-alone obstacle avoidance systems.

[0056] In some embodiments, the corresponding processing steps after S30 are as follows: the Kalman filter algorithm is used to predict the fused target trajectory, a state vector is constructed based on the target position and velocity, the target position at the current moment is predicted through the state transition matrix, and the prediction result is corrected by combining the observation value to obtain the predicted trajectory; based on the target attributes and the predicted trajectory, a three-dimensional decision matrix is ​​constructed according to the obstacle type, distance level and collision time, and an obstacle avoidance strategy is generated according to the preset priority rules.

[0057] Specifically, the Kalman filter algorithm is used to predict the fused target trajectory. A state vector is constructed based on the target's position and velocity. The current position is predicted using a state transition matrix, and the predicted trajectory is obtained by combining the observation results for correction. Then, based on the target attributes and the predicted trajectory, a three-dimensional decision matrix is ​​constructed according to obstacle type, distance level, and collision time. Obstacle avoidance strategies are generated according to preset priorities. The Kalman filter improves the trajectory estimation accuracy through a prediction-correction mechanism, and the three-dimensional decision matrix integrates multi-dimensional risk parameters to ensure the timeliness and accuracy of the obstacle avoidance strategy, providing a reliable decision basis for vehicle dynamic obstacle avoidance and further enhancing the system's adaptability to complex dynamic environments.

[0058] In some embodiments, S40 specifically includes the following steps: dynamically allocating radar data weights and visual data weights, wherein the radar data weights are positively correlated with the target signal-to-noise ratio and negatively correlated with the ambient light intensity, and the visual data weights are positively correlated with the classification confidence and the ambient light intensity; after normalization processing, the radar data and visual data are fused, and the Euclidean distance between the target in the radar data and the center of the visual detection box is calculated, and targets with a distance less than a preset threshold and the same category are determined to be the same object.

[0059] Furthermore, the decision dimensions of the three-dimensional decision matrix include: obstacle type, distance level, and collision time. The obstacle type includes pedestrians, vehicles, and static obstacles. The distance level quantifies the target distance into multiple levels. The collision time is calculated based on the target distance and relative speed and divided into multiple threshold levels. The priority rule is that pedestrians are preferred over dynamic vehicles, dynamic vehicles are preferred over static obstacles, and the collision time threshold is dynamically adjusted according to environmental conditions.

[0060] Specifically, multi-sensor fusion is achieved by dynamically assigning weights to radar and visual data: radar weights are positively correlated with the target signal-to-noise ratio and negatively correlated with ambient light intensity, while visual weights are positively correlated with classification confidence and ambient light intensity, ensuring data reliability and adaptability across different scenarios. After normalization, the fused data is analyzed by calculating the Euclidean distance between the radar target and the center of the visual detection box, associating targets with a distance less than a threshold and the same category as the same object, achieving spatiotemporal alignment of cross-modal data. The three-dimensional decision matrix is ​​constructed based on obstacle type (pedestrian, vehicle, static obstacle), distance level (quantifying target distance), and collision time (calculated and graded based on distance and relative speed). Priority rules prioritize pedestrians over dynamic vehicles and dynamic vehicles over static obstacles, and the collision time threshold is dynamically adjusted according to environmental conditions, ensuring that the obstacle avoidance strategy balances real-time performance and scene adaptability, improving decision-making accuracy in complex environments.

[0061] The implementation principle of a single-unit obstacle avoidance method based on millimeter-wave radar and camera in this application embodiment is as follows: Time synchronization of radar and visual data is achieved through hardware synchronization signals; spatial alignment is completed by establishing coordinate transformation relationships based on sensor extrinsic parameter calibration; weights of both are dynamically allocated: radar weights are positively correlated with target signal-to-noise ratio and negatively correlated with ambient light intensity, while visual weights are positively correlated with classification confidence and ambient light intensity. After normalization processing, the data are fused, and the same target is associated using Euclidean distance calculation. This fusion method can fully leverage the advantages of radar in distance and velocity detection under adverse weather conditions and the target classification advantages of vision. Dynamic weight allocation can adapt to complex environments such as changes in lighting, improving the accuracy and robustness of target attribute detection, providing reliable fused target attributes for subsequent three-dimensional decision matrices, thereby optimizing obstacle avoidance strategy generation and enhancing the environmental adaptability and decision reliability of the vehicle's single-unit obstacle avoidance system.

[0062] Figure 1 This is a flowchart illustrating a single-machine obstacle avoidance method based on millimeter-wave radar and a camera in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0063] Based on the same technical concept, referring to Figure 2 This application also provides a standalone obstacle avoidance device based on millimeter-wave radar and a camera, employing the following technical solution: The device includes:

[0064] The radar signal module is used to transmit frequency-modulated continuous wave signals through millimeter-wave radar, collect raw environmental data and preprocess it, and output a target list including position, velocity and signal-to-noise ratio.

[0065] The image data module is used to trigger the camera to capture images based on the target list when the target meets the preset signal-to-noise ratio condition and the speed exceeds the preset speed threshold. The image is then classified in real time using a lightweight convolutional neural network model, and the classification results and bounding boxes are output.

[0066] The spatiotemporal alignment module is used to perform spatiotemporal alignment between the target list and the classification results. It adopts a confidence-weighted fusion algorithm to dynamically allocate the weights of radar data and visual data, and associates the same target through distance calculation to output the fused target attributes.

[0067] The obstacle avoidance strategy module is used to construct a three-dimensional decision matrix based on the fused target attributes, obstacle type, distance level and collision time, and generate obstacle avoidance strategies according to preset priority rules.

[0068] The action response module is used to convert obstacle avoidance strategies into bus control commands. The bus control commands include obstacle type and distance level, and control the vehicle to perform corresponding alarm or braking response actions.

[0069] In some embodiments, the radar signal module is specifically used to transmit frequency-modulated continuous wave signals via millimeter-wave radar, collect raw environmental data, and parse it into a range-Doppler matrix.

[0070] Different noise estimation window sizes are set for near-field and far-field targets, and the mean background noise of the range-Doppler matrix is ​​calculated by sliding window.

[0071] Apply a dynamic threshold to each distance cell to retain target points with a signal-to-noise ratio exceeding the threshold and valid velocity.

[0072] Neighboring points are selected based on Euclidean distance and velocity consistency, and adjacent point clouds are merged to form target clusters, outputting a target list.

[0073] In some embodiments, the radar signal module is also used to control the camera power supply by connecting a switching device through the GPIO pin of the millimeter-wave radar. When the radar detects a valid target, it triggers the switching device to turn on to wake up the camera.

[0074] When the radar does not detect a valid target, the camera power is cut off to reduce power consumption.

[0075] In some embodiments, the image data module is specifically used to lightweight the convolutional neural network model as the MobileNet series model. It reduces the amount of computation by reducing the resolution of the input image, removes redundant convolutional channels by pruning channels to compress the number of model parameters, uses low-precision quantization to convert the floating-point model into an integer model, and utilizes the SIMD instructions of the embedded processor to accelerate inference operations.

[0076] In some embodiments, the spatiotemporal alignment module is further configured to use a Kalman filter algorithm to predict the fused target trajectory, construct a state vector based on the target position and velocity, predict the target position at the current moment through a state transition matrix, and correct the prediction result by combining the observation value to obtain the predicted trajectory; based on the target attributes and the predicted trajectory, construct a three-dimensional decision matrix according to the obstacle type, distance level and collision time, and generate an obstacle avoidance strategy according to a preset priority rule.

[0077] In some embodiments, the spatiotemporal alignment module is also used to dynamically allocate radar data weights and visual data weights, wherein the radar data weights are positively correlated with the target signal-to-noise ratio and negatively correlated with the ambient light intensity, and the visual data weights are positively correlated with the classification confidence and positively correlated with the ambient light intensity.

[0078] After normalization, radar data and visual data are fused, and the Euclidean distance between the target in the radar data and the center of the visual detection box is calculated. Targets with a distance less than a preset threshold and the same category are identified as the same object.

[0079] In some embodiments, the decision dimensions of the three-dimensional decision matrix include: obstacle type, distance level, and collision time; obstacle type includes pedestrians, vehicles, and static obstacles; distance level quantifies the target distance into multiple levels; collision time is calculated based on the target distance and relative speed and divided into multiple threshold levels; the priority rule is that pedestrians are preferred over dynamic vehicles, and dynamic vehicles are preferred over static obstacles; the obstacle avoidance strategy module is also used to dynamically adjust the collision time threshold according to environmental conditions.

[0080] This application also discloses a control device.

[0081] Specifically, the control device includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executed to implement the aforementioned stand-alone obstacle avoidance method based on millimeter-wave radar and camera.

[0082] This application also discloses a computer-readable storage medium.

[0083] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the single-machine obstacle avoidance method based on millimeter-wave radar and camera described above. The computer-readable storage medium includes, for example, various media that can store program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A single-machine obstacle avoidance method based on millimeter-wave radar and camera, characterized in that, include: The system transmits frequency-modulated continuous wave signals using millimeter-wave radar, collects raw environmental data, preprocesses it, and outputs a target list containing location, velocity, and signal-to-noise ratio. Based on the target list, when a target meets the preset signal-to-noise ratio condition and its speed exceeds the preset speed threshold, the camera is triggered to capture an image, and the image is classified in real time through a lightweight convolutional neural network model, outputting the classification result and bounding box. The target list and the classification results are spatiotemporally aligned. A confidence-weighted fusion algorithm is used to dynamically allocate the weights of radar data and visual data. The same target is associated through distance calculation, and the fused target attributes are output. Based on the fused target attributes, a three-dimensional decision matrix is ​​constructed according to obstacle type, distance level and collision time, and an obstacle avoidance strategy is generated according to a preset priority rule. The obstacle avoidance strategy is converted into bus control commands, which include the obstacle type and the distance level, and control the vehicle to perform corresponding alarm or braking response actions.

2. The single-machine obstacle avoidance method based on millimeter-wave radar and camera according to claim 1, characterized in that, The process involves transmitting frequency-modulated continuous wave signals via millimeter-wave radar, collecting raw environmental data, preprocessing it, and outputting a target list including position, velocity, and signal-to-noise ratio, including: The system transmits frequency-modulated continuous wave signals using millimeter-wave radar, collects raw environmental data, and analyzes it into a range-Doppler matrix. Different noise estimation window sizes are set for near-field and far-field targets, and the mean background noise of the range-Doppler matrix is ​​calculated by sliding window. Apply a dynamic threshold to each distance cell to retain target points with a signal-to-noise ratio exceeding the threshold and valid velocity. Neighboring points are selected by Euclidean distance and velocity consistency, adjacent point clouds are merged to form target clusters, and the target list is output.

3. The single-machine obstacle avoidance method based on millimeter-wave radar and camera according to claim 1, characterized in that, The trigger for the camera to capture images includes: The camera power supply is controlled by a switching device connected to the GPIO pin of the millimeter-wave radar. When the radar detects a valid target, the switching device is triggered to turn on and wake up the camera. When the radar does not detect a valid target, the camera power is cut off to reduce power consumption.

4. The single-machine obstacle avoidance method based on millimeter-wave radar and camera according to claim 2, characterized in that, The lightweight convolutional neural network model is a MobileNet series model. It reduces the amount of computation by lowering the resolution of the input image, removes redundant convolutional channels by pruning channels to compress the number of model parameters, uses low-precision quantization to convert the floating-point model into an integer model, and utilizes the SIMD instructions of the embedded processor to accelerate inference operations.

5. The single-machine obstacle avoidance method based on millimeter-wave radar and camera according to claim 1, characterized in that, After performing spatiotemporal alignment of the target list and the classification results, employing a confidence-weighted fusion algorithm to dynamically allocate weights between radar data and visual data, associating the same target through distance calculation, and outputting the fused target attributes, the method further includes: The Kalman filter algorithm is used to predict the fused target trajectory. A state vector is constructed based on the target position and velocity. The target position at the current moment is predicted through the state transition matrix. The prediction result is corrected by combining the observation values ​​to obtain the predicted trajectory. Based on the target attributes and the predicted trajectory, a three-dimensional decision matrix is ​​constructed according to the obstacle type, distance level and collision time, and an obstacle avoidance strategy is generated according to a preset priority rule.

6. The single-machine obstacle avoidance method based on millimeter-wave radar and camera according to claim 5, characterized in that, The confidence-weighted fusion algorithm is used to dynamically allocate weights between radar data and visual data, including: The radar data weights and visual data weights are dynamically allocated, wherein the radar data weights are positively correlated with the target signal-to-noise ratio and negatively correlated with the ambient light intensity, and the visual data weights are positively correlated with the classification confidence and the ambient light intensity. After normalization, radar data and visual data are fused, and the Euclidean distance between the target in the radar data and the center of the visual detection box is calculated. Targets with a distance less than a preset threshold and the same category are identified as the same object.

7. A single-machine obstacle avoidance method based on millimeter-wave radar and camera according to claim 6, characterized in that, The decision dimensions of the three-dimensional decision matrix include: the obstacle type, the distance level, and the collision time; The obstacle types include pedestrians, vehicles, and static obstacles. The distance level quantifies the target distance into multiple levels. The collision time is calculated based on the target distance and relative speed and divided into multiple threshold levels. The priority rule is that pedestrians are prioritized over dynamic vehicles, dynamic vehicles are prioritized over static obstacles, and the collision time threshold is dynamically adjusted according to environmental conditions.

8. A stand-alone obstacle avoidance device based on millimeter-wave radar and a camera, characterized in that, The device includes: The radar signal module is used to transmit frequency-modulated continuous wave signals through millimeter-wave radar, collect raw environmental data and preprocess it, and output a target list including position, velocity and signal-to-noise ratio. The image data module is used to trigger the camera to capture images based on the target list when the target meets the preset signal-to-noise ratio condition and the speed exceeds the preset speed threshold, and to perform real-time classification of the images through a lightweight convolutional neural network model, and output the classification results and bounding boxes. The spatiotemporal alignment module is used to perform spatiotemporal alignment between the target list and the classification results. It adopts a confidence-weighted fusion algorithm to dynamically allocate the weights of radar data and visual data, and associates the same target through distance calculation to output the fused target attributes. The obstacle avoidance strategy module is used to construct a three-dimensional decision matrix based on the fused target attributes, obstacle type, distance level and collision time, and generate an obstacle avoidance strategy according to a preset priority rule. The action response module is used to convert the obstacle avoidance strategy into bus control commands, which include the obstacle type and the distance level, and control the vehicle to perform corresponding alarm or braking response actions.

9. A control device, characterized in that, The device includes: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images