Target bounding box detection system and method based on millimeter wave radar

Abstract

The invention discloses a target bounding box detection system and method based on a millimeter wave radar, and the system comprises a point cloud data generation module which collects a millimeter wave intermediate frequency signal through the millimeter wave radar, and generates the point cloud data of a target; the data optimization module is used for removing noise in the point cloud data by using a reflection saliency coefficient and enhancing the point cloud by using a cumulative matrix; the frame detection module is used for carrying out receiving end beam forming on the millimeter wave intermediate frequency signal to obtain a target reflection energy spatial distribution map, fusing the target reflection energy spatial distribution map and the optimized point cloud data to obtain a fused multichannel feature map, and inputting the fused multichannel feature map into a two-dimensional contour prediction neural network to obtain a two-dimensional contour image of the target; and finally detecting a bounding box of the target in combination with the optimized point cloud data. According to the method, the applicability and reliability of target bounding box detection are improved, and the method can reliably work in various environments with poor light conditions by utilizing millimeter wave radar related algorithm design.

Description

Technical Field

[0001] The present invention belongs to the field of millimeter wave wireless sensing technology, and in particular relates to a target boundary box detection system and method based on millimeter wave radar. Background Art

[0002] Object detection plays a vital role in many applications in production and life, enabling machines to perceive the real world and interact further. In intelligent transportation systems, object detection not only helps vehicle control, but is also increasingly used in new applications such as map generation and vehicle communications. Object detection also plays a key role in intelligent drone systems, which are widely used in rescue missions, delivery services, and wildfire fighting. After detecting the target, it is usually necessary to further determine the boundaries of the target, which is called bounding box estimation. Reliable and accurate bounding box detection is essential for effective interaction between machines and the physical world, for example, ensuring reliable navigation and preventing vehicle collisions in intelligent driving. At present, existing bounding box detection schemes mainly include the following categories:

[0003] 1) Computer vision-based detection solutions, including images captured by ordinary cameras, infrared cameras, and depth cameras. Such technologies usually perform poorly in scenes with adverse visual conditions, such as low light or extreme weather, which are common in driving scenarios.

[0004] 2) LiDAR-based detection solutions use the point cloud obtained by LiDAR scanning for detection. However, the cost of LiDAR systems is usually very high, and the point cloud data volume is large, requiring a lot of computing power and resources to process.

[0005] 3) The detection scheme based on ultrasonic radar obtains the target detection result by analyzing the echo signal of the ultrasonic radar, but the ultrasonic radar has low resolution, poor accuracy and short effective distance. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a target bounding box detection system and method based on millimeter-wave radar, so as to solve the problems that sensors in existing target detection scenarios are susceptible to light limitations and have high deployment and processing costs; the present invention utilizes the signal reflected by the millimeter-wave radar on the target to realize the detection of the target bounding box in the scene.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A target bounding box detection system based on millimeter wave radar of the present invention comprises: a point cloud data generation module, a data optimization module and a frame detection module;

[0009] The point cloud data generation module includes a millimeter wave radar and a point cloud data generation unit;

[0010] Millimeter wave radar is used to transmit millimeter wave frequency modulated continuous wave signals, receive the signals reflected from the target, and obtain millimeter wave intermediate frequency signals through mixing;

[0011] A point cloud data generation unit, used to convert the millimeter wave intermediate frequency signal into the distance, speed, and azimuth of the target to generate point cloud data;

[0012] The data optimization module is used to remove noise points from the point cloud data and enhance the sparse point cloud to obtain optimized point cloud data;

[0013] The border detection module is used to perform a receiving-end beamforming operation on the millimeter-wave intermediate frequency signal to obtain a target reflection energy spatial distribution map, and fuse the target reflection energy spatial distribution map with the optimized point cloud data to obtain a fused multi-channel feature map. The fused multi-channel feature map is input into a two-dimensional contour prediction neural network to obtain a predicted target two-dimensional contour image, which is combined with the optimized point cloud data to finally detect the target's boundary box.

[0014] Furthermore, the point cloud data acquisition module transmits a continuous frequency modulated millimeter wave signal with a frequency of 60 to 64 GHz, and receives the echo signal reflected by the target through multiple receiving antennas. At time t, the transmission signal s of the millimeter wave Tx (t) and the received signal s Rx (t) are respectively expressed by the following formulas:

[0015] s Tx (t) = exp[j(2πf c t+πKt 2 )]

[0016] s Rx (t) = αS Tx [t-2R(t) / c]

[0017] Where j is the imaginary unit, f c is the starting frequency of the frequency modulated continuous wave, K is the rate of change of the frequency, R(t) is the distance between the millimeter wave radar and the target at time t, α is the path loss during reflection, and c is the speed of light in vacuum;

[0018] The intermediate frequency signal s(t) obtained after mixing is expressed as:

[0019] s(t)=s Tx (t)conj[s Rx (t)]≈αexp[j4π(f c +Kt)R(t) / c]

[0020] Among them, conj[s Rx (t)] represents the received signal s Rx (t) conjugation;

[0021] The mixed intermediate frequency signal s(t) contains the aliasing of different target reflection signals. Fast Fourier transform is performed on it to separate the reflection signals from targets at different distances, which is expressed as:

[0022]

[0023] Among them, FFT stands for fast Fourier transform, S i (t) represents the signal component corresponding to a target obtained by separation, which is called the range Fourier signal, R i Indicates the distance between the reflected signal and the millimeter-wave radar.

[0024] Furthermore, the specific steps of generating point cloud data by the point cloud data acquisition module are as follows:

[0025] After the first fast Fourier transform, the phase ω of the range Fourier signal is expressed as:

[0026]

[0027] Among them, ΔR represents the small displacement of the target between adjacent frequency-modulated signals emitted by the millimeter-wave radar, and λ represents the wavelength corresponding to the center frequency of the millimeter-wave frequency-modulated continuous wave signal;

[0028] The distance Fourier signal is then subjected to a fast Fourier transform to obtain the target's velocity v, as shown in the following formula:

[0029]

[0030] Among them, Δω is the change of phase, T c is the duration of linear frequency modulation;

[0031] After the millimeter wave intermediate frequency signal is subjected to the above two fast Fourier transforms, a range-Doppler matrix is ​​obtained. The peak point on the range-Doppler matrix indicates that there is a strong reflection at the position, which is the target point corresponding to the target to be detected. The coordinates corresponding to the peak value represent the distance and speed of the target respectively.

[0032] The azimuth angle θ of the target in space is obtained by performing fast Fourier transform on the multi-path signals obtained by different receiving antennas of the millimeter-wave radar, as follows:

[0033]

[0034] in, is the phase difference of the signals received by adjacent receiving antennas, and l is the distance between adjacent receiving antennas;

[0035] According to the distance, speed and azimuth of the target point obtained above, the position corresponding to the detected target point is mapped to the Cartesian coordinate system in space to obtain point cloud data.

[0036] Furthermore, the data optimization module removes noise points, and the specific steps are as follows:

[0037] Define the reflection significance coefficient RS, the expression is as follows:

[0038] RS=S snr ·S v ·S a

[0039] Among them, S snr is the signal-to-noise ratio significance, S v is the speed significance, S a is the angle significance; the average value of the reflection significance coefficient of all points is calculated as the threshold, and the points with reflection significance coefficient values ​​lower than the threshold are regarded as noise points for removal to complete the noise point removal of point cloud data.

[0040] Furthermore, the data optimization module performs sparse point cloud enhancement, and the specific steps are as follows:

[0041] Define the accumulation matrix and use the continuity of motion in both time and space to enhance the sparse point cloud; divide the plane into M×N grids, and the accumulation matrix BM is defined as follows:

[0042] BM t (m,n)=αBM t-1 (m,n)+count t (m,n) / max(count t )

[0043] Among them, (m,n) represents the coordinates of the grid, BM t (m,n) represents the value of the cumulative matrix at coordinate (m,n) at time t, BM t-1 (m,n) represents the value of the cumulative matrix at the coordinate (m,n) at time t-1, α represents the time decay factor, count t (m,n) represents the number of points in the (m,n) grid at time t, max(count t ) represents the maximum number of points in the grid; PM t Represents the projection of the denoised point cloud in the X-axis and Z-axis planes. The cumulative matrix is ​​used to enhance the sparse point cloud projection. The expression is as follows:

[0044]

[0045] Among them, PM′ t represents the enhanced point cloud, and γ represents the threshold.

[0046] Furthermore, the border detection module realizes the detection of the target boundary box based on the acquired millimeter wave intermediate frequency signal and the optimized point cloud data, and the specific steps are as follows:

[0047] (1) Extracting the spatial distribution of target reflected energy based on beamforming at the receiving end;

[0048] (2) The target reflection energy spatial distribution map and the optimized point cloud data are fused to obtain a fused multi-channel feature map. The fused multi-channel feature map is used as input to design a two-dimensional contour prediction neural network to predict the target bounding box.

[0049] (3) Based on the prediction results, the target’s bounding box is detected in the three-dimensional coordinate system using the target’s depth information and projection rules.

[0050] Furthermore, the step (1) specifically includes:

[0051] The multi-channel intermediate frequency signals obtained by different receiving antennas of the millimeter-wave radar are aligned to the same wavefront to achieve the beamforming effect for a specific β angle direction. The signal x after beamforming β The calculation is as follows:

[0052] x β =[1,e -jπdsinβ ,e -jπ2dsinβ ,…,e -jπ(N-1)dsinβ ]x

[0053] Wherein, β represents the angle between the target and the radial direction of the millimeter-wave radar, N represents the number of millimeter-wave radar receiving antennas, d represents the spacing between adjacent receiving antennas, and x represents the original multi-path signal vector obtained by different receiving antennas;

[0054] Through the above method, the signals from different directions in space are summarized to obtain the spatial distribution map of the target reflected energy.

[0055] Furthermore, the step (2) specifically includes:

[0056] The spatial distribution map of the target's reflected energy is convolved with the projection map of the optimized point cloud in the X-axis and Z-axis planes to obtain a fused point cloud contour image, and a fused multi-channel feature map containing three different dimensions of distance, signal-to-noise ratio, and contour is generated as the input of the two-dimensional contour prediction neural network. The depth image of the target taken by the depth camera is used as the expected output of the two-dimensional contour prediction neural network.

[0057] The two-dimensional contour prediction neural network predicts the two-dimensional contour of the target using a fused multi-channel feature map of three different dimensions. The two-dimensional contour prediction neural network is based on the ResNet structure, starting from the convolution block, passing through the downsampling layer, transitioning through the ResNet block, passing through the upsampling layer, and finally ending with the convolution block; the Tanh layer introduces L2 regularization to improve the generalization performance of the two-dimensional contour prediction neural network; the loss function of the training process is expressed as follows:

[0058]

[0059] in, is the loss function, is the calculated output value, y is the expected output value, θ is the parameter of L2 regularization, L BCE represents the binary cross entropy loss and is expressed as follows:

[0060]

[0061] Among them, m and n represent the coordinates of the plane grid;

[0062] The fused multi-channel feature map is used as the input of the two-dimensional contour prediction neural network, and the corresponding depth image is used as the output to train the two-dimensional contour prediction network; after training, the trained two-dimensional contour prediction neural network is obtained, and the fused multi-channel feature map is converted into a target two-dimensional contour image through the trained two-dimensional contour prediction neural network.

[0063] Furthermore, the step (3) specifically includes:

[0064] The target bounding box is reconstructed in the three-dimensional coordinate system using the distance between the target two-dimensional contour image and the optimized point cloud. The horizontal field of view and vertical field of view of the depth camera are denoted as HFOV and VFOV respectively, and the size of the depth image is I height ×I weight , set the coordinates of a target point to (x, y, z), and its azimuth and elevation angles relative to the millimeter-wave radar are azi and ele respectively. According to the projection principle, the mapping of the point on the depth image is expressed as (i, j), as follows:

[0065]

[0066] The point (i, j) on the target two-dimensional contour image is transformed into a three-dimensional coordinate system using the following formula:

[0067]

[0068] Among them, (x, y, z) is the coordinate of the point in the three-dimensional coordinate system, range is the distance between the target and the millimeter wave radar, and the above transformation is applied to each vertex of the target's two-dimensional contour image to obtain the target's bounding box in the three-dimensional coordinate system.

[0069] A target bounding box detection method based on millimeter wave radar of the present invention, based on the above system, comprises the following steps:

[0070] 1) Transmit millimeter-wave frequency-modulated continuous wave signals;

[0071] 2) Receive the signal reflected by the target and obtain the millimeter wave intermediate frequency signal after mixing;

[0072] 3) The millimeter wave intermediate frequency signal is transformed into a range-Doppler matrix, and the distance and speed of the target are determined according to the coordinates of the peak of the range-Doppler matrix;

[0073] 4) Perform fast Fourier transform on the millimeter wave intermediate frequency signals from different receiving antennas to obtain the azimuth of the target;

[0074] 5) Map the distance, speed and azimuth of the target point into the Cartesian coordinate system of the space to obtain point cloud data;

[0075] 6) Calculate the reflection significance coefficient and remove the noise points in the point cloud data according to the threshold;

[0076] 7) Calculate the cumulative matrix and use the cumulative matrix to enhance the point cloud;

[0077] 8) Perform beamforming at different angles on the millimeter-wave intermediate frequency signals of different receiving antennas to obtain the spatial distribution diagram of the target reflected energy;

[0078] 9) Convolve the spatial distribution map of the target reflected energy with the projection of the optimized point cloud in the XZ axis plane to generate a fused multi-channel feature map;

[0079] 10) using the fused multi-channel feature map as input to train a two-dimensional contour prediction neural network to obtain a trained two-dimensional contour prediction neural network;

[0080] 11) Inputting the fused multi-channel feature map into the trained two-dimensional contour prediction neural network to obtain the target two-dimensional contour image;

[0081] 12) Use the distances corresponding to the two-dimensional contour image and the optimized point cloud to obtain the bounding box of the target in the spatial coordinate system.

[0082] Beneficial effects of the present invention:

[0083] 1. The present invention can achieve high-precision target bounding box detection: the average error of the target bounding box detection is 0.11m.

[0084] 2. The present invention improves the applicability and reliability of target bounding box detection: by utilizing the millimeter wave radar related algorithm design, it can work reliably in various environments with poor lighting conditions.

[0085] 3. The present invention reduces the cost of target bounding box detection: data acquired by low-cost millimeter-wave radar is used to achieve efficient target bounding box detection.

[0086] 4. The present invention is easy to deploy: it has low environmental requirements, is not easily interfered with, has high robustness, and can be deployed in various complex scenarios such as transportation and production to work normally. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 is an architecture diagram of the system of the present invention;

[0088] Figure 2 It is a schematic diagram of the beamforming principle of multiple antennas of the present invention;

[0089] Figure 3a It is a schematic diagram of the spatial distribution of reflected energy of the present invention;

[0090] Figure 3b It is a schematic diagram of a fused multi-channel feature map of the present invention;

[0091] Figure 4 It is a schematic diagram of the structure of the neural network for two-dimensional contour prediction of the present invention;

[0092] Figure 5 It is a schematic diagram of the principle of two-dimensional contour image projection of the present invention. DETAILED DESCRIPTION

[0093] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0094] Reference Figures 1 to 5 As shown, a target bounding box detection system based on millimeter wave radar of the present invention includes: a point cloud data generation module, a data optimization module and a border detection module;

[0095] The point cloud data generation module includes a millimeter wave radar and a point cloud data generation unit;

[0096] Millimeter wave radar is used to transmit millimeter wave frequency modulated continuous wave signals, receive the signals reflected from the target, and obtain millimeter wave intermediate frequency signals through mixing;

[0097] A point cloud data generation unit, used to convert the millimeter wave intermediate frequency signal into the distance, speed, and azimuth of the target to generate point cloud data;

[0098] The point cloud data acquisition module transmits a continuous frequency modulated millimeter wave signal with a frequency of 60 to 64 GHz, and receives the echo signal reflected by the target through multiple receiving antennas. At time t, the transmission signal s of the millimeter wave is Tx (t) and the received signal s Rx (t) are respectively expressed by the following formulas:

[0099] s Tx (t) = exp[j(2πf c t+πKt 2 )]

[0100] s Rx (t) = αS Tx [t-2R(t) / c]

[0101] Where j is the imaginary unit, f c is the starting frequency of the frequency modulated continuous wave, K is the rate of change of the frequency, R(t) is the distance between the millimeter wave radar and the target at time t, α is the path loss during reflection, and c is the speed of light in vacuum;

[0102] The intermediate frequency signal s(t) obtained after mixing is expressed as:

[0103] s(t)=s Tx (t)conj[s Rx (t)]≈αexp[j4π(f c +Kt)R(t) / c]

[0104] Among them, conj[s Rx (t)] represents the received signal s Rx (t) conjugation;

[0105] The mixed intermediate frequency signal s(t) contains the aliasing of different target reflection signals. Fast Fourier transform is performed on it to separate the reflection signals from targets at different distances, which is expressed as:

[0106]

[0107] Among them, FFT stands for fast Fourier transform, S i (t) represents the separated signal component corresponding to a target, called the range Fourier signal, R i Indicates the distance between the reflected signal and the millimeter-wave radar.

[0108] The specific steps of generating point cloud data by the point cloud data acquisition module are as follows:

[0109] After the first fast Fourier transform, the phase ω of the range Fourier signal is expressed as:

[0110]

[0111] Among them, ΔR represents the small displacement of the target between adjacent frequency-modulated signals emitted by the millimeter-wave radar, and λ represents the wavelength corresponding to the center frequency of the millimeter-wave frequency-modulated continuous wave signal;

[0112] The distance Fourier signal is then subjected to a fast Fourier transform to obtain the target's velocity v, as shown in the following formula:

[0113]

[0114] Among them, Δω is the change of phase, T c is the duration of linear frequency modulation;

[0115] After the millimeter wave intermediate frequency signal is subjected to the above two fast Fourier transforms, a range-Doppler matrix is ​​obtained. The peak point on the range-Doppler matrix indicates that there is a strong reflection at the position, which is the target point corresponding to the target to be detected. The coordinates corresponding to the peak value represent the distance and speed of the target respectively.

[0116] The azimuth angle θ of the target in space is obtained by performing fast Fourier transform on the multi-path signals obtained by different receiving antennas of the millimeter-wave radar, as follows:

[0117]

[0118] in, is the phase difference of the signals received by adjacent receiving antennas, and l is the distance between adjacent receiving antennas;

[0119] According to the distance, speed and azimuth of the target point obtained above, the position corresponding to the detected target point is mapped to the Cartesian coordinate system in space to obtain point cloud data.

[0120] The data optimization module is used to remove noise points from the point cloud data and enhance the sparse point cloud to obtain optimized point cloud data;

[0121] Specifically, the data optimization module removes noise points, and the specific steps are as follows:

[0122] Define the reflection significance coefficient RS, the expression is as follows:

[0123] RS=S snr ·S v ·S a

[0124] Among them, S snr is the signal-to-noise ratio significance, S v is the speed significance, S ais the angle significance; the average value of the reflection significance coefficient of all points is calculated as the threshold, and the points with reflection significance coefficient values ​​lower than the threshold are regarded as noise points for removal to complete the noise point removal of point cloud data.

[0125] Specifically, the data optimization module performs sparse point cloud enhancement, and the specific steps are as follows:

[0126] Define the accumulation matrix and use the continuity of motion in both time and space to enhance the sparse point cloud; divide the plane into M×N grids, and the accumulation matrix BM is defined as follows:

[0127] BM t (m,n)=αBM t-1 (m,n)+count t (m,n) / max(count t )

[0128] Among them, (m,n) represents the coordinates of the grid, BM t (m,n) represents the value of the cumulative matrix at coordinate (m,n) at time t, BM t-1 (m,n) represents the value of the cumulative matrix at the coordinate (m,n) at time t-1, α represents the time decay factor, count t (m,n) represents the number of points in the (m,n) grid at time t, maxβ(count t ) represents the maximum number of points in the grid; PM t Represents the projection of the denoised point cloud in the X-axis and Z-axis planes. The cumulative matrix is ​​used to enhance the sparse point cloud projection. The expression is as follows:

[0129]

[0130] Among them, PM′ t represents the enhanced point cloud, and γ represents the threshold.

[0131] The border detection module is used to perform a receiving-end beamforming operation on the millimeter-wave intermediate frequency signal to obtain a target reflection energy spatial distribution map, and fuse the target reflection energy spatial distribution map with the optimized point cloud data to obtain a fused multi-channel feature map, and input the fused multi-channel feature map into a two-dimensional contour prediction neural network to obtain a predicted target two-dimensional contour image, and finally detect the target boundary box by combining the optimized point cloud data;

[0132] The border detection module detects the target boundary box based on the acquired millimeter wave intermediate frequency signal and the optimized point cloud data. The specific steps are as follows:

[0133] (1) Extracting the spatial distribution of target reflected energy based on beamforming at the receiving end;

[0134] (2) The target reflection energy spatial distribution map and the optimized point cloud data are fused to obtain a fused multi-channel feature map. The fused multi-channel feature map is used as input to design a two-dimensional contour prediction neural network to predict the target bounding box.

[0135] (3) Based on the prediction results, the target’s bounding box is detected in the three-dimensional coordinate system using the target’s depth information and projection rules.

[0136] Specifically, the step (1) comprises:

[0137] The multi-channel intermediate frequency signals obtained by different receiving antennas of the millimeter-wave radar are aligned to the same wavefront to achieve the beamforming effect for a specific β angle direction. The signal x after beamforming β The calculation is as follows:

[0138] x β =[1,e -jπdsinβ ,e -jπ2dsinβ ,…,e -jπ(N-1)dsinβ ]x

[0139] Wherein, β represents the angle between the target and the radial direction of the millimeter-wave radar, N represents the number of millimeter-wave radar receiving antennas, d represents the spacing between adjacent receiving antennas, and x represents the original multi-path signal vector obtained by different receiving antennas;

[0140] Through the above method, the signals from different directions in space are summarized to obtain the spatial distribution map of the target reflected energy.

[0141] Specifically, the step (2) comprises:

[0142] The spatial distribution map of target reflected energy is convolved with the projection map of the optimized point cloud in the X-axis and Z-axis planes to obtain a fused point cloud contour image, and a fused multi-channel feature map containing three different dimensions of distance, signal-to-noise ratio, and contour is generated as the input of the two-dimensional contour prediction neural network. The depth image of the target taken by the depth camera is used as the expected output of the two-dimensional contour prediction neural network.

[0143] The two-dimensional contour prediction neural network predicts the two-dimensional contour of the target using a fused multi-channel feature map of three different dimensions. The two-dimensional contour prediction neural network is based on the ResNet structure, starting from the convolution block, passing through the downsampling layer, transitioning through the ResNet block, passing through the upsampling layer, and finally ending with the convolution block; the Tanh layer introduces L2 regularization to improve the generalization performance of the two-dimensional contour prediction neural network; the loss function of the training process is expressed as follows:

[0144]

[0145] in, is the loss function, is the calculated output value, y is the expected output value, θ is the parameter of L2 regularization, L BCE represents the binary cross entropy loss and is expressed as follows:

[0146]

[0147] Among them, m and n represent the coordinates of the plane grid;

[0148] The fused multi-channel feature map is used as the input of the two-dimensional contour prediction neural network, and the corresponding depth image is used as the output to train the two-dimensional contour prediction network; after training, the trained two-dimensional contour prediction neural network is obtained, and the fused multi-channel feature map is converted into a target two-dimensional contour image through the trained two-dimensional contour prediction neural network.

[0149] Specifically, the step (3) comprises:

[0150] The target bounding box is reconstructed in the three-dimensional coordinate system using the distance between the target two-dimensional contour image and the optimized point cloud. The horizontal field of view and vertical field of view of the depth camera are denoted as HFOV and VFOV respectively, and the size of the depth image is I height ×I weight , set the coordinates of a target point to (x, y, z), and its azimuth and elevation angles relative to the millimeter-wave radar are azi and ele respectively. According to the projection principle, the mapping of the point on the depth image is expressed as (i, j), as follows:

[0151]

[0152] The point (i, j) on the target two-dimensional contour image is transformed into a three-dimensional coordinate system using the following formula:

[0153]

[0154] Among them, (x, y, z) is the coordinate of the point in the three-dimensional coordinate system, range is the distance between the target and the millimeter wave radar, and the above transformation is applied to each vertex of the target's two-dimensional contour image to obtain the target's bounding box in the three-dimensional coordinate system.

[0155] A target bounding box detection method based on millimeter wave radar of the present invention, based on the above system, comprises the following steps:

[0156] 1) Transmit millimeter-wave frequency-modulated continuous wave signals;

[0157] 2) Receive the signal reflected by the target and obtain the millimeter wave intermediate frequency signal after mixing;

[0158] 3) The millimeter wave intermediate frequency signal is transformed into a range-Doppler matrix, and the distance and speed of the target are determined according to the coordinates of the peak of the range-Doppler matrix;

[0159] 4) Perform fast Fourier transform on the millimeter wave intermediate frequency signals from different receiving antennas to obtain the azimuth of the target;

[0160] 5) Map the distance, speed and azimuth of the target point into the Cartesian coordinate system of the space to obtain point cloud data;

[0161] 6) Calculate the reflection significance coefficient and remove the noise points in the point cloud data according to the threshold;

[0162] 7) Calculate the cumulative matrix and use the cumulative matrix to enhance the point cloud;

[0163] 8) Perform beamforming at different angles on the millimeter-wave intermediate frequency signals of different receiving antennas to obtain the spatial distribution diagram of the target reflected energy;

[0164] 9) Convolve the spatial distribution map of the target reflected energy with the projection of the optimized point cloud in the XZ axis plane to generate a fused multi-channel feature map;

[0165] 10) using the fused multi-channel feature map as input to train a two-dimensional contour prediction neural network to obtain a trained two-dimensional contour prediction neural network;

[0166] 11) Inputting the fused multi-channel feature map into the trained two-dimensional contour prediction neural network to obtain the target two-dimensional contour image;

[0167] 12) Use the distances corresponding to the two-dimensional contour image and the optimized point cloud to obtain the bounding box of the target in the spatial coordinate system.

[0168] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A target bounding box detection system based on millimeter wave radar, characterized in that: include: Point cloud data generation module, data optimization module and border detection module; The point cloud data generation module includes a millimeter wave radar and a point cloud data generation unit; Millimeter wave radar is used to transmit millimeter wave frequency modulated continuous wave signals, receive the signals reflected from the target, and obtain millimeter wave intermediate frequency signals through mixing; A point cloud data generation unit, used to convert the millimeter wave intermediate frequency signal into the distance, speed, and azimuth of the target to generate point cloud data; The data optimization module is used to remove noise points from the point cloud data and enhance the sparse point cloud to obtain optimized point cloud data; The border detection module is used to perform a receiving-end beamforming operation on the millimeter-wave intermediate frequency signal to obtain a target reflection energy spatial distribution map, and fuse the target reflection energy spatial distribution map with the optimized point cloud data to obtain a fused multi-channel feature map. The fused multi-channel feature map is input into a two-dimensional contour prediction neural network to obtain a predicted target two-dimensional contour image, which is combined with the optimized point cloud data to finally detect the target's boundary box.

2. The target bounding box detection system based on millimeter wave radar according to claim 1, characterized in that: The point cloud data acquisition module transmits a continuous frequency modulated millimeter wave signal with a frequency of 60 to 64 GHz, and receives the echo signal reflected by the target through multiple receiving antennas. At time t, the transmitted signal s of the millimeter wave is Tx (t) and the received signal s Rx (t) are expressed by the following formulas: s Tx (t)=exp[j(2πf c t+πKt 2 )] s Rx (t)=αS Tx [t-2R(t) / c] Where j is the imaginary unit, f c is the starting frequency of the frequency modulated continuous wave, K is the rate of change of the frequency, R(t) is the distance between the millimeter wave radar and the target at time t, α is the path loss during reflection, and c is the speed of light in vacuum; The intermediate frequency signal s(t) obtained after mixing is expressed as: s(t)=s Tx (t)conj[s Rx (t)]≈αexp[j4π(f c +Kt)R(t) / c] Among them, conj[s Rx (t)] represents the received signal s Rx (t) conjugation; The mixed intermediate frequency signal s(t) contains the aliasing of different target reflection signals. Fast Fourier transform is performed on it to separate the reflection signals from targets at different distances, which is expressed as: Among them, FFT stands for fast Fourier transform, S i (t) represents the separated signal component corresponding to a target, called the range Fourier signal, R i Indicates the distance between the reflected signal and the millimeter-wave radar.

3. The target bounding box detection system based on millimeter wave radar according to claim 1, characterized in that: The specific steps of generating point cloud data by the point cloud data acquisition module are as follows: After the first fast Fourier transform, the phase ω of the range Fourier signal is expressed as: Among them, ΔR represents the small displacement of the target between adjacent frequency-modulated signals emitted by the millimeter-wave radar, and λ represents the wavelength corresponding to the center frequency of the millimeter-wave frequency-modulated continuous wave signal; The distance Fourier signal is then subjected to a fast Fourier transform to obtain the target's velocity v, as shown in the following formula: Among them, Δω is the change of phase, T c is the duration of linear frequency modulation; After the millimeter wave intermediate frequency signal is subjected to the above two fast Fourier transforms, a range-Doppler matrix is ​​obtained. The peak point on the range-Doppler matrix indicates that there is a strong reflection at the position, which is the target point corresponding to the target to be detected. The coordinates corresponding to the peak value represent the distance and speed of the target respectively. The azimuth angle θ of the target in space is obtained by performing fast Fourier transform on the multi-path signals obtained by different receiving antennas of the millimeter-wave radar, as follows: in, is the phase difference of the signals received by adjacent receiving antennas, and l is the distance between adjacent receiving antennas; According to the distance, speed and azimuth of the target point obtained above, the position corresponding to the detected target point is mapped to the Cartesian coordinate system in space to obtain point cloud data.

4. The target bounding box detection system based on millimeter wave radar according to claim 1, characterized in that: The data optimization module removes noise points, and the specific steps are as follows: Define the reflection significance coefficient RS, the expression is as follows: RS=S snr ·S v ·S a Among them, S snr is the signal-to-noise ratio significance, S v is the speed significance, S a is the angle significance; the average value of the reflection significance coefficient of all points is calculated as the threshold, and the points with reflection significance coefficient values ​​lower than the threshold are regarded as noise points for removal to complete the noise point removal of point cloud data.

5. The target bounding box detection system based on millimeter wave radar according to claim 1, characterized in that: The data optimization module performs sparse point cloud enhancement, and the specific steps are as follows: Define the accumulation matrix and use the continuity of motion in both time and space to enhance the sparse point cloud; divide the plane into M×N grids, and the accumulation matrix BM is defined as follows: BM t (m,n)=αBM t-1 (m,n)+count t (m,n) / max(count t ) Among them, (m,n) represents the coordinates of the grid, BM t (m,n) represents the value of the cumulative matrix at coordinate (m,n) at time t, BM t-1 (m,n) represents the value of the cumulative matrix at the coordinate (m,n) at time t-1, α represents the time decay factor, count t (m,n) represents the number of points in the (m,n) grid at time t, max(count t ) represents the maximum number of points in the grid; PM t Represents the projection of the denoised point cloud in the X-axis and Z-axis planes. The cumulative matrix is ​​used to enhance the sparse point cloud projection. The expression is as follows: Among them, PM′ t represents the enhanced point cloud, and γ represents the threshold.

6. The target bounding box detection system based on millimeter wave radar according to claim 1, characterized in that: The border detection module detects the target boundary box based on the acquired millimeter wave intermediate frequency signal and the optimized point cloud data. The specific steps are as follows: (1) Extracting the spatial distribution of target reflected energy based on beamforming at the receiving end; (2) The target reflection energy spatial distribution map and the optimized point cloud data are fused to obtain a fused multi-channel feature map. The fused multi-channel feature map is used as input to design a two-dimensional contour prediction neural network to predict the target bounding box. (3) Based on the prediction results, the target’s bounding box is detected in the three-dimensional coordinate system using the target’s depth information and projection rules.

7. The target bounding box detection system based on millimeter wave radar according to claim 6, characterized in that: The step (1) specifically comprises: The multi-channel intermediate frequency signals obtained by different receiving antennas of the millimeter-wave radar are aligned to the same wavefront to achieve the beamforming effect for a specific β angle direction. The signal x after beamforming β The calculation is as follows: x β =[1,e -jπdsinβ ,And -jπ2dsinβ ,…,And -jπ(N-1)dsinβ ]x Wherein, β represents the angle between the target and the radial direction of the millimeter-wave radar, N represents the number of millimeter-wave radar receiving antennas, d represents the spacing between adjacent receiving antennas, and x represents the original multi-path signal vector obtained by different receiving antennas; Through the above method, the signals from different directions in space are summarized to obtain the spatial distribution map of the target reflected energy.

8. The target bounding box detection system based on millimeter wave radar according to claim 7, characterized in that: The step (2) specifically comprises: The spatial distribution map of the target's reflected energy is convolved with the projection map of the optimized point cloud in the X-axis and Z-axis planes to obtain a fused point cloud contour image, and a fused multi-channel feature map containing three different dimensions of distance, signal-to-noise ratio, and contour is generated as the input of the two-dimensional contour prediction neural network. The depth image of the target taken by the depth camera is used as the expected output of the two-dimensional contour prediction neural network. The two-dimensional contour prediction neural network predicts the two-dimensional contour of the target using a fused multi-channel feature map of three different dimensions. The two-dimensional contour prediction neural network is based on the ResNet structure, starting from the convolution block, passing through the downsampling layer, transitioning through the ResNet block, passing through the upsampling layer, and finally ending with the convolution block; the Tanh layer introduces L2 regularization to improve the generalization performance of the two-dimensional contour prediction neural network; the loss function of the training process is expressed as follows: in, is the loss function, is the calculated output value, y is the expected output value, θ is the parameter of L2 regularization, L BCE represents the binary cross entropy loss and is expressed as follows: Among them, m and n represent the coordinates of the plane grid; The fused multi-channel feature map is used as the input of the two-dimensional contour prediction neural network, and the corresponding depth image is used as the output to train the two-dimensional contour prediction network; after training, the trained two-dimensional contour prediction neural network is obtained, and the fused multi-channel feature map is converted into a target two-dimensional contour image through the trained two-dimensional contour prediction neural network.

9. The target bounding box detection system based on millimeter wave radar according to claim 8, characterized in that: The step (3) specifically comprises: The target bounding box is reconstructed in the three-dimensional coordinate system using the distance between the target two-dimensional contour image and the optimized point cloud. The horizontal field of view and vertical field of view of the depth camera are denoted as HFOV and VFOV respectively, and the size of the depth image is I height ×I weight , set the coordinates of a target point to (x, y, z), and its azimuth and elevation angles relative to the millimeter-wave radar are azi and ele respectively. According to the projection principle, the mapping of the point on the depth image is expressed as (i, j), as follows: The point (i, j) on the target two-dimensional contour image is transformed into a three-dimensional coordinate system using the following formula: Among them, (x, y, z) is the coordinate of the point in the three-dimensional coordinate system, range is the distance between the target and the millimeter wave radar, and the above transformation is applied to each vertex of the target's two-dimensional contour image to obtain the target's bounding box in the three-dimensional coordinate system.

10. A method for detecting a target bounding box based on a millimeter wave radar, based on the system according to any one of claims 1 to 9, characterized in that: The method comprises the following steps: 1) Transmit millimeter-wave frequency-modulated continuous wave signals; 2) Receive the signal reflected by the target and obtain the millimeter wave intermediate frequency signal after mixing; 3) The millimeter wave intermediate frequency signal is transformed into a range-Doppler matrix, and the distance and speed of the target are determined according to the coordinates of the peak of the range-Doppler matrix; 4) Perform fast Fourier transform on the millimeter wave intermediate frequency signals from different receiving antennas to obtain the azimuth of the target; 5) Map the distance, speed and azimuth of the target point into the Cartesian coordinate system of the space to obtain point cloud data; 6) Calculate the reflection significance coefficient and remove the noise points in the point cloud data according to the threshold; 7) Calculate the cumulative matrix and use the cumulative matrix to enhance the point cloud; 8) Perform beamforming at different angles on the millimeter-wave intermediate frequency signals of different receiving antennas to obtain the spatial distribution diagram of the target reflected energy; 9) Convolve the spatial distribution map of the target reflected energy with the projection of the optimized point cloud in the XZ axis plane to generate a fused multi-channel feature map; 10) using the fused multi-channel feature map as input to train a two-dimensional contour prediction neural network to obtain a trained two-dimensional contour prediction neural network; 11) Inputting the fused multi-channel feature map into the trained two-dimensional contour prediction neural network to obtain the target two-dimensional contour image; 12) Use the distances corresponding to the two-dimensional contour image and the optimized point cloud to obtain the bounding box of the target in the spatial coordinate system.

Images