Millimeter wave radar ranging interference point elimination method and system based on vision assistance

Abstract

The invention discloses a millimeter wave radar ranging interference point elimination method and system based on vision assistance. The method comprises the steps of obtaining an image containing a target to be subjected to ranging and a receiving echo signal emitting electromagnetic waves at the same position as image shooting; inputting an image containing a target to be subjected to distance measurement into the trained radar denoising model, and outputting a distance measurement prior threshold interval; processing the received echo signal based on the distance measurement prior threshold interval to obtain a distance measurement echo signal after interference point elimination; wherein the training process of the radar denoising model comprises the following steps: acquiring a radar denoising training data set; a radar denoising model is constructed, the model processes a picture containing a to-be-ranged target through a target detection module and outputs detection frame information containing the to-be-ranged target, the detection frame information is processed through a monocular vision ranging module to obtain a ranging priori threshold, and a ranging priori threshold interval is output in combination with a priori maximum absolute error; and training a radar denoising model by using the training data set, and outputting the trained model.

Description

Technical Field

[0001] This invention belongs to the field of surveying and mapping technology, specifically relating to a method and system for eliminating interference points in millimeter-wave radar ranging based on visual assistance. Background Technology

[0002] Millimeter-wave radar ranging technology is widely used in intelligent transportation, high-end security, drones, automotive radar, smart lighting, and autonomous driving, but it still faces a series of challenges in distance detection. A complete millimeter-wave radar system includes transmitting (TX) and receiving (RX) radio frequency (RF) components, analog components such as clocks, and digital components such as analog-to-digital converters (ADCs), microcontrollers (MCUs), and digital signal processors (DSPs). In radar systems, the basic concept of distance measurement refers to the reflection of an electromagnetic signal during its transmission path when it is blocked by an object. The frequency of the signal used in FMCW radar systems increases linearly with time. The millimeter-wave radar board transmits the modulated signal through the transmitting antenna, and the receiving antenna receives the electromagnetic signal emitted back by the target object. There is a phase difference between the previously modulated signal (not transmitted) and the received signal. These two signals are mixed to generate an intermediate frequency (IF) signal. The IF signal is then sampled and subjected to a range-dimensional Fourier transform to obtain the distance measurement value. However, existing ranging algorithms have certain limitations. In complex environments, such as those with strong reflective objects, strong interference points can easily arise. The reflected echo signals from these interference points often exhibit high amplitude in the signal spectrum, making it impossible to determine the target's distance solely based on the amplitude peaks. Therefore, extracting ranging values ​​based solely on amplitude peaks has certain limitations. Summary of the Invention

[0003] To overcome the shortcomings of existing millimeter-wave radar ranging technology, which is prone to strong interference points in complex environments and cannot determine the target distance solely based on the amplitude peaks, this invention provides a vision-assisted millimeter-wave radar ranging interference point removal method and system. By proposing a radar denoising model that combines Faster R-CNN and a monocular visual geometric ranging model for image processing, this method provides an accurate prior threshold for ranging, and thus a prior spatial threshold range. This effectively eliminates or weakens the impact of strong interference points on ranging data in tunnel environments, improving the accuracy of distance detection using millimeter-wave radar.

[0004] According to one aspect of the present invention, a vision-assisted method for eliminating interference points in millimeter-wave radar ranging is provided, comprising:

[0005] Acquire an image containing the target to be ranged, and receive the echo signal of the electromagnetic wave emitted from the same location as the image capture;

[0006] The image containing the target to be ranged is input into the trained radar denoising model, and the range prior threshold interval is output.

[0007] The received echo signal is processed based on the range prior threshold interval to obtain the range received echo signal after interference points are removed.

[0008] The training process of the radar denoising model includes:

[0009] Obtain the training dataset for radar denoising;

[0010] A radar denoising model is constructed. The model processes images containing targets to be measured through a target detection module and outputs detection box information containing targets to be measured. The detection box information is processed through a monocular vision ranging module to obtain a prior ranging threshold. Combined with the prior maximum absolute error, the prior ranging threshold range is output.

[0011] Train a radar denoising model using the training dataset and output the trained model.

[0012] As a further technical solution, the target detection module uses a convolutional neural network to extract features from the input image containing the target to be measured, and obtains a feature map; it uses a region generation network to generate several candidate regions and preliminary bounding box offsets through an anchor point mechanism; it extracts the features of each candidate region through candidate region pooling and adjusts them to a fixed size; and it outputs the detection box information containing the target to be measured through a Fast R-CNN network.

[0013] As a further technical solution, the process of acquiring the detection box information containing the target to be ranged includes:

[0014] The Fast R-CNN network classifies the features of each candidate region extracted by candidate region pooling to determine whether it is the target to be measured. For candidate regions determined to be the target to be measured, bounding box regression is performed based on the preliminary bounding box offset, and finally the detection box information of the target to be measured in the image is output.

[0015] As a further technical solution, the monocular vision ranging module uses the detection box information, as well as the camera's intrinsic parameters and the actual size of the target to be measured, to calculate the prior spatial distance from the shooting position to the target to be measured based on the geometric ranging principle. By combining the prior spatial distance with the prior maximum absolute error, the ranging prior threshold range is obtained.

[0016] As a further technical solution, the process of acquiring training data for radar denoising includes:

[0017] Several sets of shooting distances are obtained based on fixed intervals and preset distance ranges;

[0018] Images containing the target to be measured are captured at different shooting distances, and the absolute distance from the shooting point to the target to be measured is also obtained.

[0019] Unify the resolution of all images containing the target to be measured, and mark the bounding box coordinates of the target in the image after unifying the resolution;

[0020] The images containing the target to be measured, after being processed to a uniform resolution, are integrated with the absolute distance from the corresponding shooting point to the target and the bounding box coordinates of the target. The integrated result of all the captured images containing the target to be measured is used as the training dataset.

[0021] As a further technical solution, the process for obtaining the a priori maximum absolute error is as follows:

[0022] Before obtaining the output radar denoising model, the model processes the range prior threshold output from the validation set for the last time. The range prior threshold is adaptively set to the range prior threshold range. Based on the range prior threshold range, the original range data of the validation set is processed to obtain the range value. The error between the range value and the absolute distance from the shooting point to the target to be ranged is calculated. The maximum error that meets the preset accuracy level is taken as the maximum absolute prior error.

[0023] As a further technical solution, the steps for obtaining the ranging receiver echo signal after interference point removal are as follows:

[0024] Using the prior range threshold interval as the target value segment, the signal falling within the target value segment in the received echo signal is extracted and used as the ranging received echo signal after interference point removal.

[0025] According to one aspect of the present invention, a vision-assisted millimeter-wave radar ranging interference point removal system is provided, comprising:

[0026] The ranging signal acquisition module is used to acquire images containing the target to be ranged, as well as to receive echo signals from electromagnetic waves emitted at the same location as the image capture.

[0027] The ranging prior threshold acquisition module is used to input the image containing the target to be ranging into the trained radar denoising model and output the ranging prior threshold range.

[0028] The ranging signal denoising module is used to process the received echo signal based on the ranging prior threshold range to obtain the ranging received echo signal after interference points are removed.

[0029] The training process of the radar denoising model includes:

[0030] Obtain the training dataset for radar denoising;

[0031] A radar denoising model is constructed. The model processes images containing targets to be measured through a target detection module and outputs detection box information containing targets to be measured. The detection box information is processed through a monocular vision ranging module to obtain a prior ranging threshold. Combined with the prior maximum absolute error, the prior ranging threshold range is output.

[0032] Train a radar denoising model using the training dataset and output the trained model.

[0033] According to another aspect of this specification, an electronic device is provided, including a memory and a processor, the memory storing program instructions executed by the processor, the processor invoking the program instructions to perform a vision-assisted millimeter-wave radar ranging interference point removal method.

[0034] According to another aspect of this specification, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform a vision-assisted millimeter-wave radar ranging interference point removal method.

[0035] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention addresses the problem of an imperfect theoretical framework for strong interference point removal technology in millimeter-wave radar, and proposes a vision-assisted method for extracting ranging values ​​and removing interference points from millimeter-wave radar. By proposing a radar denoising model that combines Faster R-CNN and a monocular visual geometric ranging model, image processing is performed to provide an accurate prior threshold for ranging, thereby providing a prior spatial threshold range. This effectively removes or weakens the impact of strong interference points on ranging data in tunnel environments, improving the accuracy of distance detection using millimeter-wave radar. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating the interference point removal method for vision-assisted millimeter-wave radar ranging provided in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram illustrating the principle of the monocular vision ranging model in an embodiment of the present invention;

[0039] Figure 3 This is a design diagram of the data acquisition experiment in an embodiment of the present invention;

[0040] Figure 4 This is a flowchart illustrating the data preprocessing unit in an embodiment of the present invention;

[0041] Figure 5 This is an example diagram of prior space threshold interval extraction in an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of the results of the vision-assisted millimeter-wave radar ranging interference point elimination system provided in an embodiment of the present invention;

[0043] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] It should be noted that:

[0045] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0048] This invention, based on existing millimeter-wave radar ranging procedures, designs a vision-assisted method for eliminating interference points in millimeter-wave radar ranging, such as... Figure 1 As shown, it includes:

[0049] Step 1: Acquire an image containing the target to be ranged, and receive the echo signal of the electromagnetic wave emitted from the same location as the image capture;

[0050] Step 2: Input the image containing the target to be ranged into the trained radar denoising model and output the range prior threshold interval;

[0051] Step 3: Process the received echo signal based on the ranging prior threshold interval to obtain the ranging received echo signal after interference points are removed.

[0052] Preferably, in step 1, after capturing an image containing the target to be ranged, the image is further preprocessed, including but not limited to distortion correction and resolution adjustment.

[0053] Step 3, the training process of the radar denoising model includes:

[0054] Step 3-1: Obtain the training dataset for radar denoising;

[0055] Step 3-2: Construct a radar denoising model. The model processes images containing targets to be measured through a target detection module, outputs detection box information containing targets to be measured, processes the detection box information through a monocular vision ranging module, obtains the ranging prior threshold, and outputs the ranging prior threshold range by combining the prior maximum absolute error.

[0056] Step 3-3: Train the radar denoising model using the training dataset and output the trained model.

[0057] Furthermore, in step 3-1, the process of obtaining the training dataset for radar denoising includes:

[0058] Several sets of shooting distances are obtained based on fixed intervals and preset distance ranges;

[0059] Images containing the target to be measured are captured at different shooting distances, and the absolute distance from the shooting point to the target to be measured is also obtained.

[0060] Unify the resolution of all images containing the target to be measured, and mark the bounding box coordinates of the target in the image after unifying the resolution;

[0061] The images containing the target to be measured, after being processed to a uniform resolution, are integrated with the absolute distance from the corresponding shooting point to the target and the bounding box coordinates of the target. The integrated result of all the captured images containing the target to be measured is used as the training dataset.

[0062] In step 3-2, the target detection module uses a convolutional neural network to extract features from the input image containing the target to be measured, obtaining a feature map; it uses a region generation network to generate several candidate regions and preliminary bounding box offsets through an anchoring mechanism; it extracts the features of each candidate region through candidate region pooling and adjusts them to a fixed size; and it outputs the detection box information containing the target to be measured through a FastR-CNN network.

[0063] The process of acquiring the detection bounding box information containing the target to be ranged includes:

[0064] The Fast R-CNN network classifies the features of each candidate region extracted by candidate region pooling to determine whether it is the target to be measured. For candidate regions determined to be the target to be measured, bounding box regression is performed based on the preliminary bounding box offset, and finally the detection box information of the target to be measured in the image is output.

[0065] Specifically, the target detection module is essentially a target detector, and the processing procedure is as follows:

[0066] (1) Input: Input an image containing the target to be measured.

[0067] (2) Feature extraction: Use convolutional neural networks (such as VGG, ResNet, etc., the specific one to be selected depends on the actual situation) to extract features from the input image and obtain feature maps.

[0068] (3) RPN generates candidate regions: The RPN slides on the feature map and generates multiple candidate regions through the anchor mechanism. The anchors have different scales and proportions to adapt to targets of different sizes and shapes. For each anchor box (candidate region), the RPN predicts whether it is a target to be measured and obtains the initial bounding box offset.

[0069] (4) Candidate Region Pooling (RoI Pooling): The candidate regions generated by RPN are mapped onto the feature map, the features of each candidate region are extracted, and they are adjusted to a fixed size for subsequent processing.

[0070] (5) Classification and bounding box regression: The Fast R-CNN part classifies the features of the candidate region (to determine whether it is a target) and performs bounding box regression (to correct the bounding box position of the candidate region and obtain a more accurate detection box), and finally outputs the detection box information of the target to be measured in the image (including position coordinates, etc.).

[0071] Specifically, the training process for the object detection module is as follows:

[0072] (1) Data preparation: Collect image data containing the target to be measured, and label the target in each image to determine its bounding box coordinates in the image. Divide the labeled data into training set and test set according to a certain ratio (e.g., 7:3).

[0073] (2) Pre-training: Convolutional neural networks pre-trained on large-scale datasets (such as ImageNet) are usually used as the initial parameters of the feature extraction network, which helps to speed up training and improve model performance.

[0074] (3) Fine-tuning training:

[0075] a. RPN Training: Train the RPN using the training set data to optimize its ability to generate candidate regions. Update the RPN parameters through backpropagation by calculating the difference between the RPN predictions and the ground truth annotations (e.g., cross-entropy loss for classification, smooth L1 loss for bounding box regression).

[0076] b. Fast R-CNN Training: Train the Fast R-CNN part using the candidate regions generated by RPN. Similarly, calculate the classification loss and bounding box regression loss, and update the parameters of Fast R-CNN.

[0077] (4) Model Evaluation and Optimization: Use a test set to evaluate the performance of the trained model, such as calculating the average precision (AP). Based on the evaluation results, model parameters (such as learning rate, anchor point settings, etc.) can be adjusted or data augmentation methods can be added to further optimize the model.

[0078] The monocular vision ranging module uses the detection box information, as well as the camera's intrinsic parameters and the actual size of the target to be measured, to calculate the prior spatial distance from the shooting position to the target to be measured based on the geometric ranging principle. By combining the prior spatial distance with the prior maximum absolute error, the ranging prior threshold range is obtained.

[0079] The process for obtaining the prior maximum absolute error is as follows:

[0080] Before obtaining the output radar denoising model, the model processes the range prior threshold output from the validation set for the last time. The range prior threshold is adaptively set to the range prior threshold range. Based on the range prior threshold range, the original range data of the validation set is processed to obtain the range value. The error between the range value and the absolute distance from the shooting point to the target to be ranged is calculated. The maximum error that meets the preset accuracy level is taken as the maximum absolute prior error.

[0081] Furthermore, a range of horizontal distance measurements that continuously meet the preset accuracy level is obtained as the applicable target range for this method to meet the preset accuracy level.

[0082] Specifically, the principle of the monocular geometric ranging model is as follows: Figure 2 As shown, with the optical center of the camera lens as the intersection point, the actual width W of the photographed object, the horizontal distance D between the optical center and the photographed object, the distance d between the optical center and the imaging plane, and the size w of the photographed object on the imaging plane form a similar triangle relationship. When the camera's imaging plane is located near the camera's focal point, and D is much larger than d, based on the principle of similar triangles and the thin lens imaging formula, d can be approximately equated to the camera's focal length. From similar triangles, we have:

[0083]

[0084] The actual width W of the object being photographed is usually known. Alternatively, image processing software can be used to obtain the distance. To determine the distance of an object from a photo taken with a monocular camera, the camera's focal length also needs to be known. focal length This can be achieved by calibrating the camera, or by setting a distance, i.e., at a distance where the actual size of the object being photographed is known. The pixel value of the photographed object in the photo horizontal distance between the optical center and the subject In this case, calculate the corresponding focal length. As shown below:

[0085]

[0086] Therefore, the formula for calculating monocular visual distance can be expressed as:

[0087]

[0088] Obtain the horizontal distance between the camera and the target. Then, the absolute distance between the camera and the target can be obtained. , can be represented as:

[0089]

[0090] In the formula, Indicates the camera's vertical height. Indicates the vertical height of the target.

[0091] In step 3-3, the steps for obtaining the ranging receiver echo signal after interference point removal are as follows:

[0092] Using the prior range threshold interval as the target value segment, the signal falling within the target value segment in the received echo signal is extracted to obtain the range received echo signal after interference points are removed.

[0093] Specifically, the model structure, input / output, and processing flow of the radar data denoising model are as follows:

[0094] (1) Model structure: It is composed of a monocular vision ranging model and a Faster R-CNN object detection model. The monocular vision ranging model is based on geometric imaging principles (pinhole imaging, similar triangles, etc.), and Faster R-CNN is used for object detection to obtain detection box information.

[0095] (2) Input: Images captured by a monocular camera (after preprocessing) are input into the Faster R-CNN target detection model. Raw ranging data collected by millimeter-wave radar.

[0096] (3) Output: Ranging prior threshold provided for millimeter-wave radar ranging value extraction ( The target distance is obtained based on a monocular vision ranging model. (This represents the maximum absolute error) and is used to accurately extract subsequent millimeter-wave radar ranging values ​​and eliminate interference points.

[0097] (4) Processing flow:

[0098] A. First, the Faster R-CNN object detection model processes the input image, identifies the position of the octagonal reflector in the image, and obtains the detection box information.

[0099] B. Then, the monocular vision ranging model uses detection box information (such as the pixel size of the octagonal reflector in the image), camera intrinsic parameters (focal length, etc.), and the actual size of the octagonal reflector to calculate the distance from the camera to the octagonal reflector based on the principle of geometric ranging. .

[0100] C. Finally, the maximum absolute error determined by the experiment... Generate ranging prior threshold This threshold is used to constrain the extraction range of millimeter-wave radar ranging values, remove interference points that exceed this range, and achieve radar data denoising.

[0101] In the raw data of millimeter-wave radar, strong interference points can cause the "amplitude peaks" of the spectrum to deviate from the true target range cells. By using a priori spatial threshold intervals, the effective range of radar ranging values ​​can be directly limited. Only radar data falling within this interval is considered as valid ranging values, while data outside the interval (mostly spurious signals generated by interference points) is removed, thus achieving data denoising.

[0102] As a specific implementation method, this invention provides a detailed process for a vision-assisted millimeter-wave radar ranging value extraction and interference point removal method. The equipment includes: a monocular camera, an octagonal reflector, a fixed stage, an IWR6843isk radar, and a total station. The monocular camera used in this invention is a Canon EOS 650D camera. The EOS 650D is a high-performance digital single-lens reflex camera with a high-quality CMOS image sensor of approximately 18 million effective pixels, which meets the experimental requirements. The camera is paired with an EF-S 18-135mm f / 3.5-5.6 IS USM lens, with the physical focal length fixed at 135mm in the experiment. The octagonal reflector serves as a target for measurement or monitoring by the millimeter-wave radar; therefore, a monocular camera is used to acquire image information of the octagonal reflector, and a Faster R-CNN model is used to train the octagonal reflector detector. The millimeter-wave radar used is the TI IWR6843isk radar. The purpose of the total station is to provide real observation values ​​for the radar data denoising model and compare them with the distance values ​​obtained by the radar data denoising model proposed in this invention.

[0103] Based on function, the process is mainly divided into five modules: data acquisition unit, monocular camera calibration and processing unit, data preprocessing unit, target detection unit, and radar data denoising model unit, as shown below:

[0104] (a) Data Acquisition Unit. A monocular camera is used to acquire images of the octagonal reflector under different lighting conditions and distances.

[0105] Specifically, the data acquisition experiment design is as follows: Figure 3 As shown, the data acquisition process is as follows:

[0106] (1) Install the camera in a fixed position and set its physical focal length to 135mm. At the same time, place the octagonal reflector 5 meters away from the camera as the initial measurement point;

[0107] (2) Use a camera to photograph the octagonal reflector to ensure that at least 3 high-quality images are obtained at each distance.

[0108] (3) While the camera is taking pictures, use a total station to measure the relative distance D1 between the camera and the octagonal reflector, and use this distance as the actual observed value of this experiment for subsequent accuracy evaluation and analysis;

[0109] After completing the above steps, move the octagonal reflector backward by 0.5m relative to the camera and repeat operations (1), (2), and (3) until the relative distance between the octagonal reflector and the camera is 30m.

[0110] (ii) Monocular camera calibration processing unit. The Zhang Dingyou calibration method is used to calibrate the monocular camera and obtain distortion parameters.

[0111] (III) Data Preprocessing Unit. The data preprocessing process is as follows: Figure 4 As shown, the main processing steps involve distortion correction, data augmentation, and data partitioning of the acquired image data. Distortion correction refers to applying distortion correction to the image data using distortion parameters obtained from calibration. Data augmentation involves adjusting brightness, contrast, hue saturation, gamma correction, and random rotation (-10° to 10°). The Faster R-CNN object detection model has certain requirements for the size of the input data; excessively large image data places excessive demands on hardware and leads to low training efficiency. Therefore, the image data needs to be compressed to 256*256, 512*512, or 1024*1024 according to the original image ratio. To ensure that the image data contains more target features, the compressed image resolution is set to 1024*683. Data partitioning involves dividing the image data processed in the above steps into training and test sets in a 7:3 ratio.

[0112] Specifically, after acquiring the image data according to the above data acquisition experiment, according to... Figure 4 The process for image processing is as follows:

[0113] (1) Image filtering: The collected image data is initially filtered to retain only high-quality images to ensure the accuracy of subsequent processing;

[0114] (2) Distortion correction: The distortion parameters used are used to perform distortion correction on the screened image data. This step can effectively eliminate the influence of lens distortion on the image and make the image features more accurate.

[0115] (3) Image resolution compression: The image resolution is compressed to 1024*683. Therefore, the corrected image data needs to be compressed to this resolution to adapt to the input requirements of the model.

[0116] (4) Pixel value acquisition: Using the image processing software Photoshop, the bounding box and coordinates of the octagonal reflector in the image are manually and accurately marked. Preferably, the length pixel value of the octagonal reflector in the image is also measured. To improve the reliability of the measurement, the length pixel value of the octagonal reflector in each image is measured multiple times and the average value is taken as the true pixel value;

[0117] (5) Focal length calculation: Based on the data processed above, the final calculated focal length of the camera at a resolution of 1024*683 was determined. The pixel value is 5964.2 pixels.

[0118] (iv) Object Detection Unit: An octagonal reflector detector (object detection module) based on the Faster R-CNN model is trained on the preprocessed data. The Faster R-CNN model mainly consists of two parts: a Region Generation Network (RPN) and Fast R-CNN. The RPN is used to generate candidate regions, while Fast R-CNN classifies the candidate regions and performs bounding box regression. The processed data is preprocessed image data containing octagonal reflectors.

[0119] (v) Radar Data Denoising Model Unit. Based on the detection box information, a radar data denoising model is established using a monocular visual geometric ranging model and a Faster R-CNN target detection model to provide a priori threshold range for millimeter-wave radar ranging value extraction.

[0120] The raw ranging data acquired by millimeter-wave radar includes information on octagonal reflectors and interference points in the environment. After identifying the position of the octagonal reflectors in the image using an octagonal reflector detector, further processing is performed based on the detection box information and a monocular visual geometric ranging model. The distance information calculated using the octagonal reflector detection box information is used to establish a ranging prior threshold, and the ranging prior threshold range is adaptively set using this prior threshold.

[0121] Before outputting the radar data denoising model, the range prior threshold output by the validation set on the model at this time is recorded, as well as the range value D2 calculated from the original range data under the range prior threshold range adaptively set with the range prior threshold, as shown in Table 1. Setting sub-meter level as the target detection accuracy, the data in the table shows that within the absolute distance between the octagonal reflector and the camera of 5.0m-20.0m, the target detection accuracy can reach sub-meter level. However, after the distance exceeds 20.0m, its accuracy exceeds sub-meter level. For example, at a distance of 20.5m, the true value is 20.392m, but the measured range value is 18.419m, a difference of 1.973m from the true value. This error is mainly because the octagonal reflector is too far from the camera, and its pixel value in the image is too small, meaning that the distance corresponding to one pixel value in the image exceeds 1m. In summary, the maximum applicable distance for the millimeter-wave radar strong interference point removal method proposed in this embodiment is 20m. After taking the absolute value of the difference between the measured value and the true value for the distance range of 5.0m to 20.0m in the table, the maximum value is 0.78m. This value indicates that within this distance range, the target detection accuracy of the radar data denoising model can reach sub-meter level, demonstrating the high reliability of the model in short to medium ranges. Therefore, in this embodiment, the ranging prior threshold value is... The value range is 5-20m, and the prior maximum absolute error is... The value is set to 0.78m, and the final radar data denoising model is output.

[0122] Table 1. Processing results based on radar data denoising model

[0123] Distance segment / m pixel value Actual observed value D1 / m Distance value D2 / m D1-D2 / m 5.0 255.33 4.777 4.906 -0.129 5.5 251.67 5.309 4.977 0.331 6.0 244.33 5.808 5.127 0.681 6.5 221.67 6.299 5.651 0.648 7.0 191.10 6.820 6.555 0.265 7.5 165.67 7.302 7.561 -0.258 8.0 166.67 7.817 7.515 0.301 8.5 153.33 8.339 8.169 0.170 9.0 147.33 8.828 8.501 0.327 9.5 140.67 9.327 8.904 0.423 10.0 131.67 9.817 9.513 0.304 10.5 125.00 10.313 10.020 0.293 11.0 124.00 10.805 10.101 0.704 11.5 107.40 11.351 11.662 -0.312 12.0 105.60 11.852 11.861 -0.009 12.5 107.33 12.308 11.669 0.639 13.0 102.67 12.815 12.200 0.615 13.5 91.20 13.300 13.734 -0.434 14.0 89.10 13.837 14.057 -0.220 14.5 85.43 14.297 14.661 -0.364 15.0 84.90 14.812 14.753 0.059 15.5 79.00 15.331 15.855 -0.524 16.0 77.67 15.863 16.127 -0.264 16.5 73.67 16.356 17.002 -0.646 17.0 72.67 16.839 17.236 -0.398 17.5 73.67 17.321 17.002 0.318 18.0 67.20 17.858 18.638 -0.780 18.5 69.00 18.345 18.152 0.193 19.0 69.00 18.855 18.152 0.703 19.5 64.33 19.358 19.469 -0.111 20.0 64.67 19.891 19.369 0.522 20.5 68.00 20.392 18.419 1.973 21.0 64.00 20.908 19.570 1.338 21.5 64.00 21.401 19.570 1.830 22.0 58.67 21.889 21.349 0.539 22.5 56.33 22.398 22.234 0.165 23.0 55.00 22.884 22.773 0.111 23.5 54.67 23.364 22.912 0.452 24.0 56.00 23.862 22.366 1.496 24.5 55.33 24.375 22.635 1.739 25.0 55.33 24.850 22.635 2.215 25.5 51.67 25.358 24.242 1.117 26.0 53.67 25.877 23.338 2.539 26.5 54.00 26.372 23.194 3.177 27.0 51.00 26.856 24.559 2.297 27.5 48.00 27.351 26.094 1.257 28.0 50.33 27.824 24.884 2.940 28.5 51.33 28.291 24.399 3.892 29.0 49.33 28.798 25.388 3.409 29.5 48.67 29.331 25.736 3.594 30.0 50.00 29.811 25.050 4.761

[0124] To verify the accuracy and practicality of the method of this invention, some verification data were collected. This invention demonstrates the extraction process of some millimeter-wave radar ranging values, such as... Figure 5 As shown, Figure 5 The target value ranges corresponding to (a), (b), (c), and (d) are 700cm, 1300cm, 1500cm, and 2100cm, respectively.

[0125] from Figure 5Figures (b) and (c) clearly show that the amplitude peaks in the spectrum are not the target's range cell. The proposed vision-assisted millimeter-wave radar ranging value extraction and interference point removal method provides a priori ranging threshold that includes the target range cell, effectively extracting the millimeter-wave radar ranging value. Figure (d) corresponds to a target value segment of 2100cm. According to the radar data denoising model proposed in this embodiment, this target value segment exceeds the applicable range. Therefore, image processing software is used to accurately obtain the pixel information of the octagonal reflector, resulting in a new priori ranging threshold of [20.449, 22.009]. The distance is 21.229m. Combined with Figure (d), it can be seen that the millimeter-wave radar ranging value can be extracted well from the new ranging prior threshold. Experimental results show that the radar data denoising model proposed in this embodiment can effectively extract millimeter-wave radar ranging values ​​within the range of 5-20m, achieving sub-meter accuracy.

[0126] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a vision-assisted millimeter-wave radar ranging interference point removal system. This device is used to execute the vision-assisted millimeter-wave radar ranging interference point removal method in the above method embodiments.

[0127] See Figure 6 The system includes:

[0128] The system includes a ranging signal acquisition module for acquiring images containing the target to be ranged, and a received echo signal from electromagnetic waves emitted at the same location as the image capture. A ranging prior threshold acquisition module is used to input the image containing the target to be ranged into the trained radar denoising model and output a ranging prior threshold interval. A ranging signal denoising module is used to process the received echo signal based on the ranging prior threshold interval to obtain a ranging received echo signal after interference point removal. The training process of the radar denoising model includes: acquiring a radar denoising training dataset; constructing a radar denoising model, whereby the model processes the image containing the target to be ranged through a target detection module, outputs detection box information containing the target to be ranged, processes the detection box information through a monocular vision ranging module to obtain a ranging prior threshold, and outputs a ranging prior threshold interval based on the prior maximum absolute error; training the radar denoising model using the training dataset and outputting the trained model.

[0129] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the system in the above system embodiments by referring to the specific technical solutions in other method embodiments, combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, and ensuring the practicality of the technical solutions, they can obtain corresponding system-class embodiments for implementing the methods in other method-class embodiments.

[0130] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 7 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0131] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0132] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0133] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] Based on the same technical concept as the foregoing embodiments, the present invention provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute a vision-assisted millimeter-wave radar ranging interference point removal method.

[0138] In summary, this invention addresses the problem of an incomplete theoretical framework for strong interference point removal technology in millimeter-wave radar. It proposes a vision-assisted method for extracting ranging values ​​and removing interference points from millimeter-wave radar, thereby improving the accuracy of target range detection by millimeter-wave radar. Experimental results show that the radar data denoising model proposed in this invention can effectively extract millimeter-wave radar ranging values ​​within a range of 5-20m, achieving sub-meter accuracy.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A vision-assisted method for eliminating interference points in millimeter-wave radar ranging, characterized in that, include: Acquire an image containing the target to be ranged, and receive the echo signal of the electromagnetic wave emitted from the same location as the image capture; The image containing the target to be ranged is input into the trained radar denoising model, and the range prior threshold interval is output. The received echo signal is processed based on the range prior threshold interval to obtain the range received echo signal after interference points are removed. The training process of the radar denoising model includes: Obtain the training dataset for radar denoising; A radar denoising model is constructed. The model processes images containing targets to be measured through a target detection module and outputs detection box information containing targets to be measured. The detection box information is processed through a monocular vision ranging module to obtain a prior ranging threshold. Combined with the prior maximum absolute error, the prior ranging threshold range is output. Train a radar denoising model using the training dataset and output the trained model.

2. The method for eliminating interference points in millimeter-wave radar ranging based on vision assistance as described in claim 1, characterized in that, The target detection module uses a convolutional neural network to extract features from the input image containing the target to be measured, obtaining a feature map; it uses a region generation network to generate several candidate regions and preliminary bounding box offsets through an anchor point mechanism; it extracts the features of each candidate region through candidate region pooling and adjusts them to a fixed size; and it outputs the detection box information containing the target to be measured through a Fast R-CNN network.

3. The method for eliminating interference points in millimeter-wave radar ranging based on vision assistance as described in claim 2, characterized in that, The process of acquiring the detection box information containing the target to be ranged includes: The Fast R-CNN network classifies the features of each candidate region extracted by candidate region pooling to determine whether it is the target to be measured. For candidate regions determined to be the target to be measured, bounding box regression is performed based on the preliminary bounding box offset, and finally the detection box information of the target to be measured in the image is output.

4. The method for eliminating interference points in millimeter-wave radar ranging based on vision assistance as described in claim 1, characterized in that, The monocular vision ranging module uses the detection frame information, as well as the camera's intrinsic parameters and the actual size of the target to be measured, to calculate the prior spatial distance from the shooting position to the target to be measured based on the geometric ranging principle. By combining the prior spatial distance with the prior maximum absolute error, the ranging prior threshold range is obtained.

5. The method for eliminating interference points in millimeter-wave radar ranging based on vision assistance as described in claim 1, characterized in that, The process of acquiring the training data for radar denoising includes: Several sets of shooting distances are obtained based on fixed intervals and preset distance ranges; Images containing the target to be measured are captured at different shooting distances, and the absolute distance from the shooting point to the target to be measured is also obtained. Unify the resolution of all images containing the target to be measured, and mark the bounding box coordinates of the target in the image after unifying the resolution; The images containing the target to be measured, after being processed to a uniform resolution, are integrated with the absolute distance from the corresponding shooting point to the target and the bounding box coordinates of the target. The integrated result of all the captured images containing the target to be measured is used as the training dataset.

6. The method for eliminating interference points in millimeter-wave radar ranging based on vision assistance as described in claim 5, characterized in that, The process for obtaining the prior maximum absolute error is as follows: Before obtaining the output radar denoising model, the model processes the range prior threshold output from the validation set for the last time. The range prior threshold is adaptively set to the range prior threshold range. Based on the range prior threshold range, the original range data of the validation set is processed to obtain the range value. The error between the range value and the absolute distance from the shooting point to the target to be ranged is calculated. The maximum error that meets the preset accuracy level is taken as the maximum absolute prior error.

7. The method for eliminating interference points in millimeter-wave radar ranging based on vision assistance as described in claim 1, characterized in that, The steps for obtaining the ranging receiver echo signal after removing interference points are as follows: Using the prior range threshold interval as the target value segment, the signal falling within the target value segment in the received echo signal is extracted and used as the ranging received echo signal after interference point removal.

8. A vision-assisted millimeter-wave radar ranging interference point elimination system, characterized in that, include: The ranging signal acquisition module is used to acquire images containing the target to be ranged, as well as to receive echo signals from electromagnetic waves emitted at the same location as the image capture. The ranging prior threshold acquisition module is used to input the image containing the target to be ranging into the trained radar denoising model and output the ranging prior threshold range. The ranging signal denoising module is used to process the received echo signal based on the ranging prior threshold range to obtain the ranging received echo signal after interference points are removed. The training process of the radar denoising model includes: Obtain the training dataset for radar denoising; A radar denoising model is constructed. The model processes images containing targets to be measured through a target detection module and outputs detection box information containing targets to be measured. The detection box information is processed through a monocular vision ranging module to obtain a prior ranging threshold. Combined with the prior maximum absolute error, the prior ranging threshold range is output. Train a radar denoising model using the training dataset and output the trained model.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 7.

Images