Relay reflecting surface reconstruction method and system based on millimeter wave radar

Abstract

The invention provides a relay reflecting surface reconstruction method and system based on a millimeter wave radar, and the method comprises the steps: S1, providing a data collection method based on beam scanning, carrying out the detection of a corner environment through an FMCW radar carried by a mobile trolley, and collecting a radar signal which can be used for the reconstruction of a relay reflecting surface; s2, providing a millimeter wave radar point cloud generation algorithm, and generating a dense and accurate point cloud from a radar intermediate frequency signal; s3, providing a point cloud feature extension method, extending point-level RCS features for the radar point cloud, and enriching semantic information of the point cloud; s4, a multi-scale learnable edge convolutional neural network module (MGL-EdgeConv) is designed, and multi-scale local feature extraction of the point cloud is realized; and S5, designing a point cloud semantic segmentation neural network model MGL-Net taking MGL-EdgeConv as a core module, inputting a radar point cloud, and outputting a semantic category of the point cloud.

Description

Technical Field

[0001] The present invention relates to the field of wireless sensing and ubiquitous computing technology, and in particular, to a method and system for reconstructing a relay reflection surface based on a millimeter-wave radar, and more particularly, to a method and system for reconstructing a relay reflection surface based on a millimeter-wave radar for corner human body perception. Background Art

[0002] In recent years, with the growing demand for automation and intelligent technology in society, mobile robots have been widely used in many industries such as warehousing and logistics, medical services, etc. Traditional perception systems mostly detect the environment within the line of sight of the sensor, while the environment in the obstructed area cannot be detected by traditional perception systems, which greatly reduces the safety of the system and the comprehensiveness of perception. Corner human perception technology can realize the positioning of non-line-of-sight pedestrians, and then prevent collision accidents through better path planning; corner human perception technology can also expand the application scenarios of human-computer interaction. For example, corner gesture recognition technology can enable users to control mobile robots outside the line of sight through different gestures; corner action recognition technology can enable mobile robots to detect dangerous behaviors such as falls and fainting of pedestrians in the non-line-of-sight range, and take corresponding measures in time. Therefore, the research on non-line-of-sight human perception technology is of great significance to improving the safety and functional diversity of mobile perception systems.

[0003] So far, a series of works have used millimeter-wave radar as a sensor for non-line-of-sight perception and successfully verified its effectiveness. The basic principle of millimeter-wave radar-based corner human perception is multiple reflections of the signal. The transmitted millimeter-wave signal will be reflected by the relay reflective surface in the environment to reach the non-line-of-sight target and return to the radar receiving antenna along the original path, which will generate a virtual mirror target point cloud behind the relay reflective surface in the generated radar point cloud. In order to map the point cloud of the non-line-of-sight target back to its real position, reconstructing the geometry of the relay reflective surface is a prerequisite. Existing methods all use additional lidar sensors to reconstruct the geometry of the relay reflective surface. However, not all robots or cars are equipped with lidar. In addition, lidar cannot restore the relay reflective surface of transparent materials such as glass and polycarbonate.

[0004] The present invention aims to design a relay reflector reconstruction system based on millimeter-wave radar without the assistance of laser radar, so as to realize the application of corner human perception with lower cost and wider application range. The present invention proposes a relay reflector reconstruction method based on millimeter-wave radar for corner human perception, and realizes accurate relay reflector reconstruction through millimeter-wave radar. However, the design of such a method faces the following challenges. The first challenge is that the millimeter-wave radar generated by the traditional method is so sparse that it is difficult to reflect the geometric shape of the relay reflector. The proposed method needs to be able to generate dense and accurate relay reflector point cloud; the second challenge is that without the assistance of laser radar, it is difficult to distinguish the relay reflector part and the non-line-of-sight target part in the radar point cloud by relying on the limited characteristics of the millimeter-wave radar itself. The proposed method needs to be able to achieve accurate millimeter-wave radar semantic segmentation. Summary of the invention

[0005] In view of the defects in the prior art, an object of the present invention is to provide a method and system for reconstructing a relay reflection surface based on millimeter-wave radar.

[0006] A method for reconstructing a relay reflection surface based on millimeter wave radar provided by the present invention includes:

[0007] Step S1: Use the millimeter-wave radar carried by the mobile vehicle to detect the corner environment and obtain a radar signal that can be used to reconstruct the relay reflection surface;

[0008] Step S2: Generate a radar point cloud based on the radar signal using a millimeter wave radar point cloud generation algorithm;

[0009] Step S3: Expanding point-level RCS features using a point cloud feature expansion method based on the radar point cloud;

[0010] Step S4: extracting multi-scale local features of the point cloud based on the multi-dimensional features of the radar point cloud;

[0011] Step S5: constructing a point cloud semantic segmentation neural network model MGL-Net; using the point cloud semantic segmentation neural network model MGL-Net, obtaining the semantic category of the point cloud based on the multi-scale local features of the point cloud;

[0012] The point cloud semantic segmentation neural network model MGL-Net is based on MGL-EdgeConv as the core, and is based on the four-dimensional features of the radar point cloud, including: X-axis coordinates, Y-axis coordinates, speed, and point-level RCS, to obtain the semantic category of each point in the point cloud.

[0013] Preferably, the step S1 comprises: using a millimeter-wave radar with a beamforming function to detect the corner environment.

[0014] Preferably, in the step S1, when the millimeter-wave radar carried by the mobile vehicle is used to detect the corner environment, the millimeter-wave radar parameters are set, including: setting the number of chirp signals in each sub-segment to 120, and setting the number of sampling points of each chirp signal to 512.

[0015] Preferably, in the step S1, when the millimeter-wave radar carried by the mobile vehicle is used to detect the corner environment, each frame signal is evenly divided into multiple sub-segments in the time dimension, and in each sub-segment, the phases of multiple transmitting antennas are set to specific values, so as to realize focusing the beam to a specific angle;

[0016] In the kth subsegment, the beamforming angle is θ k When the phase values ​​of the n transmitting antennas TX1, TX2, ..., TXn are set to

[0017]

[0018] Among them, d 2 and d n They represent the distances between TX2 and TX1, and between TXn and TX1 respectively, and λ is the wavelength of the signal.

[0019] Preferably, step S2 comprises:

[0020] Step S2.1: Represent the intermediate frequency signal of each sub-segment as a three-dimensional matrix of K×L×N, where K represents the number of chirp signals contained in a sub-segment, L represents the number of sampling points in a single chirp signal, and N represents the number of receiving antennas;

[0021] Step S2.2: For the intermediate frequency signal of each sub-segment, the distance is estimated by using the distance fast Fourier transform in the fast time dimension, and the speed is estimated by using the Doppler fast Fourier transform in the slow time dimension, and the distance-speed spectrum matrix K′×L′×N under each receiving antenna channel is output, where K′ represents the number of points of the Doppler fast Fourier transform, and L′ represents the number of points of the distance fast Fourier transform;

[0022] Step S2.3: Generate each distance quantization unit l and speed quantization unit k based on the distance-speed spectrum matrix K′×L′×N under each receiving antenna channel; wherein 1≤l≤L′, 1≤k≤K′;

[0023] Step S2.4: using the MUSIC algorithm to estimate the horizontal angle spectrum of the one-dimensional signal x(t)=X[k,l,:] composed of different receiving antennas;

[0024]

[0025] Among them, a(θ) represents the direction vector matrix; P MUSIC (θ) represents the horizontal angle spectrum; E n The eigenvector matrix representing the noise subspace;

[0026] Step S2.5: for the distance-speed-angle three-dimensional matrix, take the average in the speed dimension to generate a distance-angle heat map;

[0027] Step S2.6: Use the 2D CFAR algorithm to extract the peak value of the distance-angle heat map to obtain the candidate distance-angle set {(r 1 ,θ 1 ),...(r n ,θ n )};

[0028] Step S2.7: For each set of distance-angle pairs {(r i ,θ i )} Extract the corresponding Doppler FFT result from the distance-velocity-angle three-dimensional matrix, select the velocity corresponding to the peak value in the Doppler FFT result as the estimated velocity of the target, and calculate the plane coordinates of the target:

[0029] x i =r i cosθ i ,y i =r i sinθ i

[0030] Based on the point cloud generated by the intermediate frequency signal of each sub-segment, the point clouds of the nine sub-segments are merged together to form a complete frame of point cloud data.

[0031] Preferably, step S3 comprises:

[0032] Step S3.1: offline predicting the empirical antenna gain G(φ,θ) corresponding to each beamforming angle φ and horizontal angle θ under different beamforming angles based on the actual experimental method;

[0033] Step S3.2: Calculate the point-level RCS feature based on the empirical antenna gain G(φ,θ) corresponding to each beamforming angle φ and horizontal angle θ under different beamforming angles;

[0034]

[0035] in, represents the empirical value of the antenna gain in the radial direction of the reflection point p, which is equal to G(φ p ,θ p ); Represents the empirical value of the received power of the reflected signal, set to the distance spectrum at distance dp The square of the magnitude at ; Represents the distance between the reflection point p and the radar.

[0036] Preferably, the step S4 comprises:

[0037] Step S4.1: Use the improved k-NN algorithm to find each point p in the feature space i Calculate k nearest neighbors at multiple scales;

[0038] Step S4.2: construct edge features for each point and its neighboring points;

[0039]

[0040] Among them, p i and p j Respectively represent point p i and p j The features of , w represents the learnable weight;

[0041] Step S4.3: Perform maximum pooling on the edge features of each point to obtain updated point features;

[0042]

[0043] Step S4.4: Fuse the updated point features through the attention mechanism.

[0044] Preferably, the step S4.1 comprises:

[0045] Point p i and p j The distance between ij Set to d ij =w T ||p i -p j || 2 , where p i and p j They are point p i and p j The features of , w is the learnable weight;

[0046] After the improved K-NN algorithm, the neighborhood relationships on multiple scales are obtained:

[0047]

[0048] Among them, N m (i) is point p i The set of k nearest neighbors at the mth scale.

[0049] Preferably, step S5 comprises:

[0050] Step S5.1: For the features related to the spatial coordinates in the input point cloud The concatenation matrix of point features and edge features is obtained through the k-nearest neighbor algorithm The features of the point cloud are then extracted through a feedforward neural network φ(·), and the point cloud features are aggregated using a maximum pooling operation to generate a global feature vector Next, the global feature g is passed through a series of fully connected layers f(·) to generate an affine transformation matrix Finally, the spatial left feature of the input point cloud is linearly transformed to obtain the aligned point cloud coordinates X′=XT; the spatial transformation coordinate feature X′ and the other dimensional features of the point cloud Splice to get the transformed point cloud features

[0051] Step S5.2: MGL-Net stacks L MGL-EdgeConv layers, each layer extracts local features of different scales; the input feature of the lth layer is P (l-1) , output feature P (l) for:

[0052] P (l) =MGLEdgeConv(P (l-1) )

[0053]

[0054] Step S5.3: After the multi-layer feature extraction is completed, the local features of all points output by the L-th layer MGL-EdgeConv are aggregated into global features:

[0055]

[0056] Step S5.4: local features of each point It is concatenated with the global feature g and passes through a three-layer fully connected network to finally output the category of each point:

[0057]

[0058] Among them, W fc is the weight of the fully connected network, b fc is the bias of the fully connected network.

[0059] According to the present invention, a millimeter wave radar-based relay reflection surface reconstruction system includes:

[0060] Module M1: Use the millimeter-wave radar carried by the mobile car to detect the corner environment and obtain radar signals that can be used to reconstruct the relay reflection surface;

[0061] Module M2: Generate radar point cloud based on radar signal using millimeter wave radar point cloud generation algorithm;

[0062] Module M3: Extending point-level RCS features based on radar point cloud using point cloud feature extension method;

[0063] Module M4: Extracting multi-scale local features of point clouds based on multi-dimensional features of radar point clouds;

[0064] Module M5: Construct the point cloud semantic segmentation neural network model MGL-Net; use the point cloud semantic segmentation neural network model MGL-Net to obtain the semantic category of the point cloud based on the multi-scale local features of the point cloud;

[0065] The point cloud semantic segmentation neural network model MGL-Net is based on MGL-EdgeConv as the core, and is based on the four-dimensional features of the radar point cloud, including: X-axis coordinates, Y-axis coordinates, speed, and point-level RCS, to obtain the semantic category of each point in the point cloud.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. The present invention provides the first relay reflector reconstruction system based only on millimeter-wave radar without the assistance of laser radar, and has a relay reflector reconstruction accuracy comparable to that of laser radar;

[0068] 2. The present invention adopts a framework of data collection-point cloud generation-feature expansion-point cloud semantic segmentation, which can generate dense and accurate millimeter-wave radar point clouds, and semantically segment radar point clouds into two categories of people and relay reflection surfaces with high precision, thereby realizing the reconstruction of relay reflection surfaces and assisting the application of human corner perception based on millimeter-wave radar;

[0069] 3. The present invention realizes the first relay reflector reconstruction system based only on millimeter-wave radar without the assistance of laser radar. It can be combined with a variety of downstream models to realize a variety of corner human perception applications including target detection, gesture recognition, action recognition, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0071] Figure 1 2 is an example diagram of the system flow in an embodiment of the present invention.

[0072] Figure 2 This is a diagram of a millimeter-wave radar-based relay reflection surface reconstruction system for corner human perception in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0074] Example 1

[0075] According to a method for reconstructing a relay reflection surface based on millimeter-wave radar for corner human perception provided by the present invention, Figures 1 to 2 As shown, including:

[0076] Step S1: Propose a data collection method based on beam scanning, use the FMCW radar carried by the mobile car to detect the corner environment, and collect radar signals that can be used to reconstruct the relay reflection surface;

[0077] Step S2: Propose a millimeter wave radar point cloud generation algorithm to generate dense and accurate point clouds from radar intermediate frequency signals;

[0078] Step S3: Propose a point cloud feature extension method to extend the point-level RCS features for the radar point cloud and enrich the semantic information of the point cloud;

[0079] Step S4: Design a multi-scale learnable edge convolutional neural network module (MGL-EdgeConv) to extract multi-scale local features of point clouds;

[0080] Step S5: design a point cloud semantic segmentation neural network model MGL-Net with MGL-EdgeConv as the core module, input the radar point cloud, and output the semantic category of the point cloud;

[0081] Specifically, the step S1 includes:

[0082] Step S1.1: Due to the mirror reflection effect of the millimeter-wave radar, the relay reflection surface point cloud generated by the traditional radar point cloud generation algorithm is very sparse. In order to enhance the intensity of the diffuse reflection signal of the relay reflection surface, this embodiment proposes to use beamforming technology to improve the transmission signal strength, and achieve omnidirectional gain improvement within the radar field of view through the signal transmission mode of beam scanning. This embodiment selects the TI AWR1843 millimeter-wave radar with beamforming function. This embodiment found through pilot experiments that the 3dB beam width of multiple transmitting antennas after beamforming is about 15°, so the step size of the beam scanning angle change is set to 15°; considering that the field of view of the millimeter-wave radar transmitting antenna is -60° to 60°, the angle range of the beam scanning is set to -60° to 60°.

[0083] Step S1.2: Radar parameter setting. The parameter design of the radar mainly considers the optimization of indicators such as the velocity resolution and the duration of each frame of the millimeter-wave radar. On the one hand, in order to ensure that each frame of the point cloud meets the assumption of quasi-stationarity, the duration of each frame of the signal needs to be shorter. On the other hand, the shorter the sub-segment duration, the lower the velocity resolution of the millimeter-wave radar. After a trade-off between velocity resolution and quasi-stationarity of the point cloud, this embodiment sets the number of chirp signals in each sub-segment (burst) to 120, and the number of sampling points of each chirp signal to 512. Under this parameter setting, the duration of each frame of the signal is 151.7ms, the distance resolution of the millimeter-wave radar is about 3.79cm, the theoretical maximum detection distance is about 22.73m, the velocity resolution is about 0.23m / s, and the theoretical maximum detection speed is about 13.93m / s, which basically meets the system performance requirements of corner human perception.

[0084] Step S1.3: Each frame signal is divided into nine sub-segments in time. In each sub-segment, the phases of multiple transmitting antennas are set to specific values, so as to focus the beam to a specific angle. Specifically, in the kth sub-segment, the beamforming angle is θ k When , the phase values ​​of the three transmitting antennas (TX1, TX2, TX3) are set to

[0085]

[0086] Among them, d 2 and d 3 They represent the distances between TX2 and TX1, and between TX3 and TX1 respectively, and λ is the wavelength of the signal.

[0087] The step S2 comprises:

[0088] Step S2.1: Two-dimensional estimation of distance and speed. The intermediate frequency signal of each sub-segment can be represented as a three-dimensional matrix of size K×L×N, where K=120 represents the number of chirp signals contained in a sub-segment, L=512 represents the number of sampling points in a single chirp signal, and N=4 represents the number of receiving antennas. For the intermediate frequency signal of each sub-segment, the distance is first estimated in the fast time dimension using the distance fast Fourier transform, and then the speed is estimated in the slow time dimension using the Doppler fast Fourier transform. The output result is a matrix of size K′×L′×N, where K′=120 represents the number of Doppler fast Fourier transform points, and L′=512 represents the number of distance fast Fourier transform points. The physical meaning of this matrix is ​​the distance-speed spectrum under each receiving antenna channel.

[0089] Step S2.2: Super-resolution angle estimation. In order to improve the resolution and accuracy of angle estimation, this embodiment adopts a super-resolution angle estimation algorithm, namely the multiple signal classification (MUSIC) algorithm, to perform horizontal angle estimation. Given the three-dimensional matrix X of distance-velocity-receiving antenna of size K′×L′×N outputted in step 1, for each combination of distance quantization unit l (1≤l≤L′) and velocity quantization unit k (1≤k≤K′), this embodiment uses the MUSIC algorithm to estimate the horizontal angle spectrum of the corresponding one-dimensional signal x(t)=X[k,l,:] composed of different receiving antennas. Specifically, firstly, the covariance matrix of the receiving array signal is constructed, and then the eigenvalue decomposition of the covariance matrix is ​​performed to obtain the eigenvector matrix E of the noise subspace. n , E n The matrix spanned by the feature space corresponding to several eigenvalues ​​with the smallest covariance matrix, and finally the MUSIC space spectrum function is constructed according to the orthogonality of the signal subspace and the noise subspace:

[0090]

[0091] Where a(θ) is the direction vector matrix. Traverse the possible angles θ and calculate the spatial spectrum function P MUSIC (θ), the physical meaning of the spatial spectrum function is the energy distribution estimation of the signal received by the array at different angles, so the peak point corresponds to the arrival direction of the signal.

[0092] Step S2.3: Distance-angle peak extraction. First, the distance-speed-angle three-dimensional matrix is ​​averaged in the speed dimension to generate a distance-angle heat map, and then the 2DCFAR algorithm is used to extract the peak of the distance-angle heat map. The algorithm traverses the distance-angle heat map pixel by pixel, and each traversed pixel is called a unit to be detected. A rectangular window containing the surrounding protection units and background units is constructed with the unit to be detected as the center. First, the background noise of each unit to be detected is estimated. The estimation method is: sort all the signal strengths in the background units after excluding the protection units, exclude the largest 30% of the background units, and average the signal strengths of the remaining 70% of the background units to obtain the background noise size of the CUT; then calculate the detection threshold based on the product of the background noise estimate and the threshold factor; then compare the signal strength of the CUT with the detection threshold. If the signal strength of the CUT is greater than the detection threshold, the CUT is determined to be a target, otherwise the CUT is determined to be a background point. Finally, after all pixels have been traversed, all detected target points are further eliminated from non-local peaks, that is, each target point is checked to see if it is the maximum value of its local area. If the target point is a local maximum, it is retained in the final result, otherwise it is eliminated. Finally, the candidate distance-angle set is obtained, which is expressed as {(r 1,θ 1 ),...(r n ,θ n )}.

[0093] Step S2.4: Coordinate transformation and multi-frame merging. For each set of distance-angle pairs {(r i ,θ i )}, we first extract the corresponding Doppler FFT result from the distance-velocity-angle three-dimensional matrix. Then, we select the velocity corresponding to the peak value in the Doppler FFT result as the estimated velocity of the target. Then, we calculate the plane coordinates of the target by the following formula:

[0094] x i =r i cosθ i ,y i =r i sinθ i

[0095] The intermediate frequency signal of each sub-segment is used to generate a point cloud using the above method. The point clouds of the nine sub-segments are merged together to form a complete frame of point cloud data.

[0096] Specifically, step S3 includes:

[0097] Step S3.1: Non-line-of-sight targets such as pedestrians and relay reflectors such as walls also have significant differences in radar detection capabilities, which are mainly due to the differences in their surface materials and geometric shapes. In order to enrich the semantic information of radar point clouds, this embodiment proposes a new millimeter-wave radar point cloud feature, point-level radar cross-section area (PRCS), to measure the radar detectability of each reflection point. The point-level RCS of reflection point p is denoted as σ p , is defined as:

[0098]

[0099] in, represents the received power of the reflected signal from the reflection point p, d p Represents the distance between the reflection point p and the radar, P t Represents the total transmit power of the radar, G p represents the antenna gain in the radial direction of the reflection point p, and λ represents the wavelength of the signal.

[0100] Step S3.2: Offline pre-measurement of antenna radiation pattern. The purpose of the offline pre-measurement step is to measure the antenna radiation pattern at different beamforming angles, that is, the empirical antenna gain corresponding to each beamforming angle φ and horizontal angle θ. This embodiment chooses to calculate G(φ, θ) through actual experiments. The reason for using actual experiments is that when the PRCS, radar transmit power and reflection point distance are fixed, the size of G(φ, θ) is proportional to the received signal strength (RSS), and RSS can be actually measured. Specifically, this step places a corner reflector at a series of azimuth positions, from -90° to 90°, with a step size of 10°. These azimuths cover the entire field of view of the radar and are kept at a fixed distance from the radar. At each azimuth, the radar sequentially beamforms to each angle involved in the beam scanning process. For each set of beamforming angles φ 0 and the horizontal angle θ 0 ,G(φ 0 ,θ 0 ) is calculated as the square amplitude of the corresponding range spectrum peak. Finally, polynomial interpolation is used for the azimuth angle to obtain a more refined antenna radiation pattern G(φ,θ).

[0101] Step S3.3: Online calculation of empirical PRCS. The purpose of the online calculation step is to calculate the empirical PRCS for each reflection point p. The core of this step is to estimate the key variables in the PRCS definition: First, the empirical value of the received power of the reflected signal is set to the distance spectrum at distance d p The square of the amplitude at . The reason why the range spectrum is used instead of the Doppler spectrum in this step is that the number of points in the range FFT is 512, while the number of points in the Doppler FFT is 120. Therefore, the range spectrum has less spectrum leakage than the Doppler FFT. In addition, considering that the range resolution of common millimeter-wave radars is high enough, it can be assumed that different points fall into different distance intervals. Therefore, it is reasonable to use the range spectrum amplitude of the corresponding distance to calculate the RSS of the reflection point. Empirical value of the antenna gain in the radial direction of the reflection point p Equal to G(φ p ,θ p ), where φ p represents the beamforming angle of the subsegment where the reflection point p is located, θ p Represents the horizontal angle of the reflection point p. After estimating the values ​​of the above variables, the formula Calculate the empirical PRCS value of the reflection point p.

[0102] Specifically, the step S4 includes: designing a multi-scale learnable edge convolution module (MGL-EdgeConv).

[0103] Step S4.1: Multi-scale neighborhood construction. This step uses the improved k-NN algorithm to construct a multi-scale neighborhood for each point p in the feature space. ik nearest neighbor points are calculated at various scales. Considering that different point cloud features carry different semantic information, their contributions to semantic segmentation are also different. In order to adaptively set the weights of different features, this embodiment integrates learnable parameters into the distance metric used by the k-NN algorithm to construct the graph. Specifically, we take point p i and p j The distance between ij Set to d ij =w T ||p i -p j || 2 , where p i and p j They are point p i and p j The features of , w are learnable weights.

[0104] Different neighborhood sizes can capture features at different scales. For example, smaller neighborhood sizes tend to extract finer details, while larger neighborhood sizes capture more global features. In order to overcome the difficulty of selecting a single optimal neighborhood size, this embodiment proposes to extract local features at multiple scales through a multi-branch structure. Specifically, we first construct a graph using the k-nearest neighbor algorithm at three empirically predefined neighborhood sizes. After the K-NN algorithm, the neighborhood relationships at three scales are obtained:

[0105]

[0106] Where N m (i) is point p i The set of k nearest neighbors at the mth scale.

[0107] Step S4.2: Construct feature edges. Construct edge features for each point and its neighboring points, expressed as:

[0108]

[0109] Step S4.3: Aggregate neighborhood features. Perform maximum pooling on the edge features of each point to obtain updated point features:

[0110]

[0111] Step S4.4: Multi-scale feature fusion. Finally, the updated point features output by the three branches are fused through the attention mechanism.

[0112] Specifically, step S5 includes: designing a millimeter-wave radar point cloud semantic segmentation neural network MGL-Net, taking the four-dimensional features of the radar point cloud (i.e., x-axis coordinate, y-axis coordinate, speed, point level RCS) as input, and outputting the semantic category of each point in the point cloud (i.e., person, relay reflection surface, other). The core module of MGL-Net is the multi-scale learnable edge convolution module (MGL-EdgeConv) proposed in this embodiment.

[0113] Step S5.1: In order to better align the point cloud data of different spatial perspectives and enhance the robustness of the model, MGL-Net adaptively learns the spatial transformation matrix of the point cloud through the spatial transformation module. First, the concatenation matrix of point features and edge features is obtained through the k-nearest neighbor algorithm Then, a small feed-forward neural network φ(·) is used to extract the features of the point cloud, and a maximum pooling operation is used to aggregate the point cloud features to generate a global feature vector Next, the global feature g is passed through a series of fully connected layers f(·) to generate an affine transformation matrix Finally, the spatial left feature of the input point cloud is linearly transformed to obtain the aligned point cloud coordinates X′=XT. The spatially transformed coordinate feature X′ and the other dimensional features of the point cloud Splice to get the transformed point cloud features

[0114] Step S5.2: Multi-layer feature extraction. MGL-Net stacks L MGL-EdgeConv layers, and each layer extracts local features of different scales. The input feature of the lth layer is P (l-1) , output feature P (l) for

[0115] P (l) =MGLEdgeConv(P (l-1) )

[0116]

[0117] Step S5.3: Global feature aggregation. After the multi-layer feature extraction is completed, the local features of all points output by the L-th layer MGL-EdgeConv are aggregated into global features:

[0118]

[0119] Step S5.4: Segmentation head. The local features of each point It is concatenated with the global feature g and passes through a three-layer fully connected network to finally output the category of each point.

[0120]

[0121] Where W fc is the weight of the fully connected network, b fc is the bias of the fully connected network.

[0122] Example 2

[0123] Embodiment 2 is a preferred embodiment of Embodiment 1

[0124] According to the present invention, a millimeter-wave radar-based relay reflection surface reconstruction system for corner human perception includes:

[0125] Module M1: uses the beam scanning data collection method to detect the corner environment using the FMCW radar carried by the mobile car, and collects radar signals that can be used to reconstruct the relay reflection surface;

[0126] Module M2: using the millimeter wave radar point cloud generation algorithm proposed in this embodiment to generate a dense and accurate point cloud from the radar intermediate frequency signal;

[0127] Module M3: adopt the point cloud feature extension method proposed in this embodiment to extend the point-level RCS features for the radar point cloud and enrich the semantic information of the point cloud;

[0128] Module M4: This embodiment is used to design a point cloud semantic segmentation neural network model MGL-Net with MGL-EdgeConv as the core module, inputs the radar point cloud, and outputs the semantic category of the point cloud;

[0129] Specifically, the module M1 includes:

[0130] Module M1.1: A millimeter-wave radar with beamforming function (such as TIAWR1843) is installed on the mobile car, and the beam scanning angle range is set to -60° to 60°, with a step size of 15°. The number of chirp signals in each subsegment (burst) is set to 120, and the number of sampling points of each chirp signal is set to 512. Under this parameter setting, the duration of each frame signal is 151.7ms, the distance resolution of the millimeter-wave radar is about 3.79cm, the theoretical maximum detection distance is about 22.73m, the speed resolution is about 0.23m / s, and the theoretical maximum detection speed is about 13.93m / s, which basically meets the system performance requirements for corner human perception.

[0131] Module M1.2: Each frame of signal is divided into nine sub-segments in time. In each sub-segment, the phases of multiple reflective antennas are set to specific values ​​to focus the beam to a specific angle. Specifically, in the kth sub-segment, the beamforming angle is θ k When , the phase values ​​of the three transmitting antennas (TX1, TX2, TX3) are set to

[0132]

[0133] where d 2 and d 3 They represent the distances between TX2 and TX1, and between TX3 and TX1 respectively, and λ is the wavelength of the signal.

[0134] Specifically, the module M2 includes:

[0135] Module M2.1: Two-dimensional estimation of distance and velocity. The intermediate frequency signal of each sub-segment can be represented as a three-dimensional matrix of size K×L×N, where K=120 represents the number of chirp signals contained in a sub-segment, L=512 represents the number of sampling points in a single chirp signal, and N=4 represents the number of receiving antennas. For the intermediate frequency signal of each sub-segment, the distance is first estimated using the distance fast Fourier transform in the fast time dimension, and then the speed is estimated using the Doppler fast Fourier transform in the slow time dimension. The output result is a matrix of size K′×L′×N, where K′=120 represents the number of Doppler fast Fourier transform points, and L′=512 represents the number of distance fast Fourier transform points. The physical meaning of this matrix is ​​the distance-velocity spectrum under each receiving antenna channel.

[0136] Module M2.2: Super-resolution angle estimation. In order to improve the resolution and accuracy of angle estimation, this embodiment adopts a super-resolution angle estimation algorithm, namely the multiple signal classification (MUSIC) algorithm, to estimate the horizontal angle. Given the three-dimensional matrix X of distance-velocity-receiving antenna of size K×L×N outputted in step 1, for each combination of distance quantization unit l (1≤l≤L′) and velocity quantization unit k (1≤k≤K′), this embodiment uses the MUSIC algorithm to estimate the horizontal angle spectrum of the corresponding one-dimensional signal x(t)=X[k,l,:] composed of different receiving antennas. Specifically, the covariance matrix of the receiving array signal is first constructed, and then the eigenvalue decomposition of the covariance matrix is ​​performed to obtain the eigenvector matrix E of the noise subspace. n , E n The matrix spanned by the feature space corresponding to several eigenvalues ​​with the smallest covariance matrix, and finally the MUSIC space spectrum function is constructed according to the orthogonality of the signal subspace and the noise subspace:

[0137]

[0138] Where a(θ) is the direction vector matrix. Traverse the possible angles θ and calculate the spatial spectrum function P MUSIC (θ), the physical meaning of the spatial spectrum function is the energy distribution estimation of the signal received by the array at different angles, so the peak point corresponds to the arrival direction of the signal.

[0139] Module M2.3: Distance-angle peak extraction. First, the distance-speed-angle three-dimensional matrix is ​​averaged in the speed dimension to generate a distance-angle heat map, and then the 2DCFAR algorithm is used to extract the peak of the distance-angle heat map. The algorithm traverses the distance-angle heat map pixel by pixel, and each traversed pixel is called a unit to be detected. A rectangular window containing the surrounding protection units and background units is constructed with the unit to be detected as the center. First, the background noise of each unit to be detected is estimated. The estimation method is: sort all the signal strengths in the background units after excluding the protection units, exclude the largest 30% of the background units, and average the signal strengths of the remaining 70% of the background units to obtain the background noise size of the CUT; then calculate the detection threshold based on the product of the background noise estimate and the threshold factor; then compare the signal strength of the CUT with the detection threshold. If the signal strength of the CUT is greater than the detection threshold, the CUT is determined to be a target, otherwise the CUT is determined to be a background point. Finally, after all pixels have been traversed, all detected target points are further eliminated from non-local peaks, that is, each target point is checked to see if it is the maximum value of its local area. If the target point is a local maximum, it is retained in the final result, otherwise it is eliminated. Finally, the candidate distance-angle set is obtained, which is expressed as {(r 1 ,θ 1 ),...(r n ,θ n )}.

[0140] Module M2.4: Coordinate transformation and multi-frame merging. For each set of distance-angle pairs {(r i ,θ i )}, we first extract the corresponding Doppler FFT result from the distance-velocity-angle three-dimensional matrix. Then, we select the velocity corresponding to the peak value in the Doppler FFT result as the estimated velocity of the target. Then, we calculate the plane coordinates of the target by the following formula:

[0141] x i =r i cosθ i ,y i =r i sinθ i

[0142] The intermediate frequency signal of each sub-segment is used to generate a point cloud using the above method. The point clouds of the nine sub-segments are merged together to form a complete frame of point cloud data.

[0143] Specifically, the module M3 includes:

[0144] Module M3.1: This embodiment proposes a new millimeter-wave radar point cloud feature, point-level radar cross-section area (PRCS), to measure the radar detectability of each reflection point. The point-level RCS of the reflection point p is denoted as σ p , is defined as:

[0145]

[0146] in, represents the received power of the reflected signal from the reflection point p, d p Represents the distance between the reflection point p and the radar, P t Represents the total transmit power of the radar, G p represents the antenna gain in the radial direction of the reflection point p, and λ represents the wavelength of the signal.

[0147] Module M3.2: Offline Pre-measurement of Antenna Radiation Pattern. The purpose of the offline pre-measurement step is to measure the antenna radiation pattern at different beamforming angles, that is, the empirical antenna gain G(φ,θ) corresponding to each beamforming angle φ and horizontal angle θ. This embodiment chooses to calculate G(φ,θ) through actual experiments. The reason for using the actual experimental method is that when the PRCS, radar transmit power and reflection point distance are fixed, the size of G(φ,θ) is proportional to the received signal strength (RSS), and RSS can be actually measured. Specifically, this step places a corner reflector at a series of azimuth positions, from -90° to 90°, with a step size of 10°. These azimuths cover the entire field of view of the radar and are kept at a fixed distance from the radar. At each azimuth, the radar sequentially beamforms to each angle involved in the beam scanning process. For each set of beamforming angles φ 0 and the horizontal angle θ 0 ,G(φ 0 ,θ 0 ) is calculated as the square amplitude of the corresponding range spectrum peak. Finally, polynomial interpolation is used for the azimuth angle to obtain a more refined antenna radiation pattern G(φ,θ).

[0148] Module M3.3: Online calculation of empirical PRCS. The purpose of the online calculation step is to calculate the empirical PRCS for each reflection point p. The core of this step is to estimate the key variables in the PRCS definition: First, the empirical value of the received power of the reflected signal is set to the distance spectrum at distance d p The square of the amplitude at the reflection point p; the empirical value of the antenna gain in the radial direction Equal to G(φ p ,θ p ), where φ p represents the beamforming angle of the subsegment where the reflection point p is located, θ pRepresents the horizontal angle of the reflection point p. After estimating the values ​​of the above variables, the formula Calculate the empirical PRCS value of the reflection point p.

[0149] Specifically, the module M4 includes: the input of MGL-Net is the four-dimensional features of the radar point cloud (i.e., x-axis coordinate, y-axis coordinate, speed, point level RCS), and the output is the semantic category of each point in the point cloud (i.e., person, relay reflector, other). The core module of MGL-Net is the multi-scale learnable edge convolution module (MGL-EdgeConv) proposed in this embodiment.

[0150] Module M5.1: Spatial transformation module. For the features related to the spatial coordinates in the input point cloud First, the concatenation matrix of point features and edge features is obtained through the k-nearest neighbor algorithm Then, a small feed-forward neural network φ(·) is used to extract the features of the point cloud, and a maximum pooling operation is used to aggregate the point cloud features to generate a global feature vector Next, the global feature g is passed through a series of fully connected layers f(·) to generate an affine transformation matrix Finally, the spatial left feature of the input point cloud is linearly transformed to obtain the aligned point cloud coordinates X′=XT. The spatially transformed coordinate feature X′ and the other dimensional features of the point cloud Splice to get the transformed point cloud features

[0151] Module M5.2: Multi-layer feature extraction. MGL-Net stacks L MGL-EdgeConv layers, each layer extracts local features of different scales. The input feature of the lth layer is p (l-1) , output feature p (l) for

[0152] p (l) =MGLEdgeConv(P (l-1) )

[0153]

[0154] Module M5.3: MGL-EdgeConv module. First, the k-NN algorithm is used to calculate the nearest neighbor points on three different predefined neighborhoods. The k-NN algorithm uses a learnable distance metric, that is, point p i and p j The distance between ij Set to d ij =w T ||p i -p j || 2, where p i and p j They are point p i and p j The features of , w is the learnable weight. The neighborhood relationships on three scales are obtained: Where N m (i) is point p i The k nearest neighbor set at the mth scale. Then, edge features are constructed for each point and its neighboring points, expressed as Then perform maximum pooling on the edge features of each point to obtain the aggregated point features Finally, the updated point features output by the three branches are fused through the attention mechanism.

[0155] Module M5.4: Global feature aggregation. After the multi-layer feature extraction is completed, the local features of all points output by the L-th layer MGL-EdgeConv are aggregated into global features:

[0156]

[0157] Module M5.5: Segmentation head. The local features of each point It is concatenated with the global feature g and passes through a three-layer fully connected network to finally output the category of each point.

[0158]

[0159] Among them, W fc is the weight of the fully connected network, b fc is the bias of the fully connected network.

[0160] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0161] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for reconstructing a relay reflection surface based on millimeter wave radar, characterized in that: include Step S1: Use the millimeter-wave radar carried by the mobile vehicle to detect the corner environment and obtain a radar signal that can be used to reconstruct the relay reflection surface; Step S2: Generate a radar point cloud based on the radar signal using a millimeter wave radar point cloud generation algorithm; Step S3: Expanding point-level RCS features using a point cloud feature expansion method based on the radar point cloud; Step S4: extracting multi-scale local features of the point cloud based on the multi-dimensional features of the radar point cloud; Step S5: constructing a point cloud semantic segmentation neural network model MGL-Net; using the point cloud semantic segmentation neural network model MGL-Net, obtaining the semantic category of the point cloud based on the multi-scale local features of the point cloud; The point cloud semantic segmentation neural network model MGL-Net is based on MGL-EdgeConv as the core, and is based on the four-dimensional features of the radar point cloud, including: X-axis coordinates, Y-axis coordinates, speed, and point-level RCS, to obtain the semantic category of each point in the point cloud.

2. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 1, characterized in that: The step S1 includes: using a millimeter wave radar with a beamforming function to detect the corner environment.

3. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 1, characterized in that: In the step S1, when the millimeter-wave radar carried by the mobile vehicle is used to detect the corner environment, the millimeter-wave radar parameters are set, including: setting the number of chirp signals in each sub-segment to 120, and setting the number of sampling points of each chirp signal to 512.

4. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 1, characterized in that: In the step S1, when the millimeter-wave radar carried by the mobile vehicle is used to detect the corner environment, each frame signal is evenly divided into multiple sub-segments in the time dimension, and in each sub-segment, the phases of multiple transmitting antennas are set to specific values, so as to realize focusing the beam to a specific angle; In the kth subsegment, the beamforming angle is θ k When the phase values ​​of n transmitting antennas TX1, TX2, ..., TXN are set to Wherein, d2 and d3 represent the distances between TX2 and TX1, and between TXN and TX1, respectively, and λ is the wavelength of the signal.

5. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 4, characterized in that: The step S2 comprises: Step S2.1: Represent the intermediate frequency signal of each sub-segment as a three-dimensional matrix of K×L×N, where K represents the number of chirp signals contained in a sub-segment, L represents the number of sampling points in a single chirp signal, and N represents the number of receiving antennas; Step S2.2: For each sub-segment of the intermediate frequency signal, use the distance fast Fourier transform in the fast time dimension to estimate the distance, and then use the Doppler fast Fourier transform in the slow time dimension to estimate the speed, and output the distance-speed spectrum matrix K under each receiving antenna channel ′ ×L′×N, where K′ represents the number of Doppler fast Fourier transform points and L′ represents the number of range fast Fourier transform points; Step S2.3: Based on the distance-velocity spectrum matrix K under each receiving antenna channel ′ ×L′×N generates each distance quantization unit l and speed quantization unit k; where 1≤l≤L′, 1≤k≤K′; Step S2.4: using the MUSIC algorithm to estimate the horizontal angle spectrum of the one-dimensional signal x(t)=X[k,l,:] composed of different receiving antennas; Where a(θ) represents the direction vector matrix; P MUSIC (θ) represents the horizontal angle spectrum; E n The eigenvector matrix representing the noise subspace; Step S2.5: for the distance-speed-angle three-dimensional matrix, take the average in the speed dimension to generate a distance-angle heat map; Step S2.6: Use the 2D CFAR algorithm to extract the peak value of the distance-angle heat map to obtain the candidate distance-angle set {(r1,θ1),...(r n ,θ n )}; Step S2.7: For each set of distance-angle pairs {(r i ,θ i )} Extract the corresponding Doppler FFT result from the distance-velocity-angle three-dimensional matrix, select the velocity corresponding to the peak value in the Doppler FFT result as the estimated velocity of the target, and calculate the plane coordinates of the target: x i =r i cosθ i ,y i =r i sinθ i Based on the point cloud generated by the intermediate frequency signal of each sub-segment, the point clouds of the nine sub-segments are merged together to form a complete frame of point cloud data.

6. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 1, characterized in that: The step S3 comprises: Step S3.1: offline predicting the empirical antenna gain G(φ,θ) corresponding to each beamforming angle φ and horizontal angle θ under different beamforming angles based on the actual experimental method; Step S3.2: Calculate the point-level RCS feature based on the empirical antenna gain G(φ,θ) corresponding to each beamforming angle φ and horizontal angle θ under different beamforming angles; in, represents the empirical value of the antenna gain in the radial direction of the reflection point p, which is equal to G(φ p ,θ p ); represents the empirical value of the received power of the reflected signal, set to the distance spectrum at distance d p The square of the magnitude at ; Represents the distance between the reflection point p and the radar.

7. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 1, characterized in that: The step S4 comprises: Step S4.1: Use the improved k-NN algorithm to find each point p in the feature space i Calculate k nearest neighbors at multiple scales; Step S4.2: construct edge features for each point and its neighboring points; Among them, p i and p j Respectively represent point p i and p j The features of , w represents the learnable weight; Step S4.3: Perform maximum pooling on the edge features of each point to obtain updated point features; Step S4.4: Fuse the updated point features through the attention mechanism.

8. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 7, characterized in that: The step S4.1 comprises: Point p i and p j The distance between ij Set to d ij =w T ||p i -p j ||2, where p i and p j They are point p i and p j The features of , w is the learnable weight; After the improved K-NN algorithm, the neighborhood relationships on multiple scales are obtained: Among them, N m (i) is point p i The set of k nearest neighbors at the mth scale.

9. The method for reconstructing a relay reflection surface based on millimeter wave radar according to claim 1, characterized in that: The step S5 comprises: Step S5.1: For the features related to the spatial coordinates in the input point cloud The concatenation matrix of point features and edge features is obtained through the k-nearest neighbor algorithm The features of the point cloud are then extracted through a feedforward neural network φ(·), and the point cloud features are aggregated using a maximum pooling operation to generate a global feature vector Next, the global feature g is passed through a series of fully connected layers f(·) to generate an affine transformation matrix Finally, the spatial left feature of the input point cloud is linearly transformed to obtain the aligned point cloud coordinates X′=XT; the spatial transformation coordinate feature X′ and the other dimensional features of the point cloud Splice to get the transformed point cloud features Step S5.2: MGL-Net stacks L MGL-EdgeConv layers, each layer extracts local features of different scales; the input feature of the lth layer is P (l-1) , output feature P (l) for: P (l) =MGLEdgeConv(P (l-1) ) Step S5.3: After the multi-layer feature extraction is completed, the local features of all points output by the L-th layer MGL-EdgeConv are aggregated into global features: Step S5.4: local features of each point It is concatenated with the global feature g and passes through a three-layer fully connected network to finally output the category of each point: Among them, W fc is the weight of the fully connected network, b fc is the bias of the fully connected network.

10. A millimeter wave radar-based relay reflector reconstruction system, characterized in that: include Module M1: Use the millimeter-wave radar carried by the mobile car to detect the corner environment and obtain radar signals that can be used to reconstruct the relay reflection surface; Module M2: Generate radar point cloud based on radar signal using millimeter wave radar point cloud generation algorithm; Module M3: Extending point-level RCS features based on radar point cloud using point cloud feature extension method; Module M4: Extracting multi-scale local features of point clouds based on multi-dimensional features of radar point clouds; Module M5: Construct the point cloud semantic segmentation neural network model MGL-Net; use the point cloud semantic segmentation neural network model MGL-Net to obtain the semantic category of the point cloud based on the multi-scale local features of the point cloud; The point cloud semantic segmentation neural network model MGL-Net is based on MGL-EdgeConv as the core, and is based on the four-dimensional features of the radar point cloud, including: X-axis coordinates, Y-axis coordinates, speed, and point-level RCS, to obtain the semantic category of each point in the point cloud.

Images