RADRadar-based radar scene sensing method, device and equipment

Abstract

The invention relates to a radar scene sensing method, device and equipment based on RADRadar. The method comprises the following steps: constructing an RADRadar model; the RADRadar model comprises a feature similarity learning module and an outer product module; calculating a channel similarity matrix of every two of the RA view, the RD view and the AD view along a feature channel dimension by using a feature similarity learning module, and updating one of the two low-dimensional views subjected to outer product according to the channel similarity matrix to obtain an updated low-dimensional view; and reconstructing the updated low-dimensional view and another original low-dimensional view according to the outer product module to obtain a reconstructed RAD feature cube. By adopting the method, coupling learning among multiple dimensions can be realized.

Description

Technical Field

[0001] The present application relates to the field of radar scene perception technology, and in particular to a radar scene perception method, device and equipment based on RADRadar. Background Art

[0002] Millimeter-wave radar plays a vital role in autonomous driving due to its all-day, all-weather operation and unique speed measurement capabilities. Compared to traditional mechanical scanning radar, multiple-input, multiple-output (MIMO) radar can achieve significantly higher angular resolution. Starting from the radar signal processing flow, radar echoes first exist in the form of large-scale and non-intuitive analog-to-digital converter signals (ADCs). Given the non-intuitive nature of the ADC data format itself, network design for ADC signals is relatively difficult. Therefore, ADC signals are often converted into dense structured radar RAD cubes (range-azimuth-Doppler) through three Fourier transforms. Its regular data format is conducive to the design of learning-based perception networks. However, the complete radar tensor (i.e., radar RAD cube) has a huge data volume, which leads to high computational overhead, thus limiting large-scale applications. Therefore, radar RAD cubes are often filtered into sparse point cloud form through methods such as constant false alarm rate detector (CFAR) to reduce computational overhead. However, the filtering process inevitably causes irreversible information loss, which limits the improvement of the network's perception capabilities.

[0003] Multi-view technology attempts to strike a balance between the two representation paradigms based on the radar cube and radar point cloud. It utilizes multiple low-dimensional views (RA, RD, AD) compressed from the radar cube as input, reducing the network's reliance on computationally expensive layers like 3D convolutions. This reduces computational overhead compared to networks designed directly for the RAD cube. In feature space, multi-view technology concatenates features from multiple views, achieving information complementarity between them. Compared to sparse point cloud representation, multi-view technology preserves both the neat format of radar data and relatively more of the original information, offering advantages. Therefore, multi-view technology is currently an effective means of radar characterization.

[0004] As a classic architecture in the field of radar multi-view technology, TMVA-Net adopts an encoder-decoder architecture. It first compresses the radar RAD cube into RA, RD, and AD views, and then splices the past few frames as the temporal input of the model. The temporal input of the three views is respectively extracted through three independent encoder branches. At the forefront of the encoder, the model uses three-dimensional convolution to extract and compress the temporal information, and then uses two-dimensional convolution to extract features for each view. The outputs of the three independent encoders are directly spliced ​​together along the feature channel dimension in the feature space to form a latent feature space (Latent Feature Space). This latent feature space is then input into two independent decoder branches (RA and RD views) for subsequent decoding, and finally a final prediction is made on the RA and RD views.

[0005] Most existing millimeter-wave radar-based multi-view fusion technologies inherit the learning paradigm of the TMVA-Net network. These techniques directly concatenate multiple views along the feature channel dimension in the feature space between the encoder and decoder. This concatenation method results in the concatenation of non-corresponding physical dimensions (for example, directly concatenating the RA and RD views results in the A-axis and D-axis being incorrectly aligned). This leads to information confusion and a loss of the physical meaning of the dimensions, resulting in erroneous perception. Furthermore, while this direct concatenation method maintains the low dimensionality of the views, it lacks the cube-structured format of the RAD three dimensions, making the model incapable of exploiting the coupling relationships between the three dimensions. Some multi-view fusion techniques utilize implicit attention mechanisms to combine the RA and RD views, using the RD view to supplement the Doppler information missing from the RA view. While this approach avoids direct concatenation of physical dimensions, the implicit learning method cannot construct a cube-structured format with the RAD three dimensions, making the model incapable of exploiting the coupling relationships between the three dimensions. Summary of the Invention

[0006] Based on this, it is necessary to provide a radar scene perception method, device and equipment based on RADRadar that can realize coupled learning among multiple dimensions to address the above technical problems.

[0007] A radar scene perception method based on RADRadar, the method comprising:

[0008] Obtain compressed multi-view radar observation data; the multi-view radar observation data includes RA view, RD view and AD view; construct a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module;

[0009] The feature similarity learning module is used to calculate the channel similarity matrix of each view along the feature channel dimension of the RA view, RD view, and AD view. According to the channel similarity matrix, one of the two low-dimensional views of the outer product is updated to obtain the updated low-dimensional view.

[0010] The updated low-dimensional view and another original low-dimensional view are reconstructed according to the outer product module to obtain a reconstructed RAD feature cube.

[0011] A radar scene perception device based on RADRadar, the device comprising:

[0012] A data acquisition and model building module is used to acquire compressed multi-view radar observation data; the multi-view radar observation data includes RA view, RD view and AD view; and build a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module;

[0013] a feature consistency module, configured to calculate a channel similarity matrix of each of the RA view, the RD view, and the AD view along the feature channel dimension using the feature similarity learning module, and update one of the two low-dimensional views obtained by the outer product according to the channel similarity matrix to obtain an updated low-dimensional view;

[0014] The feature reconstruction module is used to reconstruct the updated low-dimensional view and another original low-dimensional view according to the outer product module to obtain a reconstructed RAD feature cube.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Obtain compressed multi-view radar observation data; the multi-view radar observation data includes RA view, RD view and AD view; construct a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module;

[0017] Calculating a channel similarity matrix of each of the RA view, the RD view, and the AD view along the feature channel dimension using a feature similarity learning module, and updating one of the two low-dimensional views obtained by the outer product according to the channel similarity matrix to obtain an updated low-dimensional view;

[0018] The updated low-dimensional view and another original low-dimensional view are reconstructed according to the outer product module to obtain a reconstructed RAD feature cube.

[0019] The above-mentioned radar scene perception method, device and equipment based on RADRadar, the present application constructs a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module, and the feature similarity learning module is used to calculate the channel similarity matrix of the RA view, RD view and AD view along the feature channel dimension, which can measure the similarity of different views on the feature channel, thereby understanding the association between each view. The feature consistency of the common dimension between different views can be found, providing a basis for subsequent update operations. According to the channel similarity matrix, the low-dimensional view used as the key is updated to obtain an updated key view, whose feature channel has a higher similarity with the low-dimensional view used as the query, effectively ensuring the validity of the subsequent outer product operation, and preparing for the construction of a structured RAD cube. According to the outer product module, the low-dimensional view used as the query and the updated key view are reconstructed to obtain a reconstructed RAD feature cube. The outer product operation can combine the low-dimensional view features into a high-dimensional RAD feature cube. This reconstruction method can explicitly construct a cube structured format with three RAD dimensions in the feature space. In the feature space of the multi-view fusion network, it is the key operation to achieve the reconstruction from the low-dimensional feature layer to the high-dimensional RAD feature cube. It provides strong support for multi-view fusion and the mining of dimensional coupling relationships through the effective combination of low-dimensional features. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 1 is a flow chart of a radar scene perception method based on RADRadar in one embodiment;

[0021] Figure 2 Schematic diagram of the architecture of the RADRadar model in one embodiment;

[0022] Figure 3 is a schematic diagram of a feature similarity learning module in one embodiment;

[0023] Figure 4 is a structural block diagram of a radar scene perception device based on RADRadar in one embodiment;

[0024] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0026] In one embodiment, Figure 1As shown, a radar scene perception method based on RADRadar is provided, comprising the following steps:

[0027] Step 102: Obtain compressed multi-view radar observation data; the multi-view radar observation data includes RA view, RD view and AD view; construct a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module.

[0028] There are many problems with existing multi-view fusion technology. For example, direct splicing will cause multiple views with non-corresponding physical dimensions to be spliced ​​together, causing information confusion and loss of physical meaning of dimensions, and it is impossible to mine the coupling relationship between the three dimensions; and the method using the implicit attention mechanism cannot construct a cube structured format with the three dimensions of RAD, and it also does not have the potential to mine the coupling relationship between the three dimensions. This application aims to solve these problems in order to achieve coupled learning between multiple dimensions. Figure 2 As shown in Figure 1, RADRadar first receives compressed multi-view radar observation data (RA view (range-azimuth view), RD view (range-Doppler view) and AD view (azimuth-Doppler view)), and then performs independent feature extraction on each view in the encoder independently. In the feature space, RADRadar reconstructs the high-dimensional RAD feature cube by introducing an Outer Product Module (OPM). In order to better maintain the physical meaning of the outer product operation, a Feature Similarity Learning Module (FSLM) is introduced (see Figure 3 For the reconstructed RAD feature cube, a 3D convolutional layer is further introduced to learn the spatial coupling between the three physical dimensions. An additional 3D convolutional layer is then used to compress the cube back to the RA and RD views to reduce computational overhead. Finally, decoding is performed on the RA and RD views and the corresponding prediction output is performed.

[0029] Step 104: Calculate the channel similarity matrix of each of the RA view, RD view, and AD view along the feature channel dimension using a feature similarity learning module, and update one of the two low-dimensional views obtained by the outer product according to the channel similarity matrix to obtain an updated low-dimensional view.

[0030] The feature similarity learning module calculates the channel similarity matrix for each of the RA, RD, and AD views along the feature channel dimension. This step measures the similarity between the two low-dimensional views to be outer-producted along the feature channels, thereby understanding the correlation between the views. This method identifies the feature consistency along the common dimension between the two low-dimensional views to be outer-producted, providing a basis for subsequent updates.

[0031] One of the two views being outer-producted is updated based on the channel similarity matrix, resulting in two views with higher feature channel similarity. Because the outer product operation requires good consistency in the common dimensions of the two views, learning and updating the feature similarity learning module ensures that the features of the two low-dimensional views being outer-producted are more closely matched on the common dimensions, ensuring the effectiveness of subsequent outer product operations. This step prepares for the construction of a structured RAD cube, ensuring that the updated features better reflect the relationships between multiple dimensions.

[0032] Step 106 : reconstruct the updated low-dimensional view and another original low-dimensional view according to the outer product module to obtain a reconstructed RAD feature cube.

[0033] In a multi-view fusion network, multiple low-dimensional views (such as RA, RD, and AD views) are compressed from the radar cube. These low-dimensional views have their own feature representations in the feature space. The goal of multi-view fusion is to effectively integrate the information from these low-dimensional views to obtain a more comprehensive and accurate feature representation, thereby improving perception. However, some existing multi-view fusion methods (such as direct splicing) suffer from information confusion and the inability to exploit dimensional coupling relationships.

[0034] In the context of multi-view fusion, since the three low-dimensional views (RA, RD, and AD) all share common dimensions, the outer product operation can combine the information of the low-dimensional feature layers according to certain rules, thereby reconstructing the high-dimensional RAD feature cube from the low-dimensional feature layers. Furthermore, the reconstruction of the high-dimensional RAD feature cube from the low-dimensional feature layers is of great significance to the multi-view fusion network. The high-dimensional RAD feature cube has a regular three-dimensional structure. This structured format can clearly represent the information of radar data in different dimensions (such as azimuth, range, and Doppler). Compared to directly concatenating low-dimensional views, the reconstructed high-dimensional feature cube avoids information confusion and preserves the physical meaning of the dimensions. It also provides a foundation for subsequent operations such as three-dimensional convolution to exploit the coupling relationship between the three dimensions, enabling the model to more fully learn and utilize the information in multi-view data, improving perception capabilities and performance.

[0035] The outer product operation does not exist in isolation. A feature similarity learning module is introduced before the outer product operation. This module is used to learn the consistency of features in the common dimensions of different views to ensure the effectiveness of the outer product operation. Because the effectiveness of the outer product operation depends on the matching degree of the participating features in the common dimensions, the feature similarity learning module adjusts and optimizes the features, making the high-dimensional RAD feature cube generated by the outer product operation more accurate and effective, better achieving the goal of multi-view fusion.

[0036] The above-mentioned radar scene perception method based on RADRadar, the present application constructs a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module, and the feature similarity learning module is used to calculate the channel similarity matrix of the RA view, RD view and AD view along the feature channel dimension. It can measure the similarity of the two low-dimensional views to be outer-producted on the feature channel, thereby understanding the association between the views. The feature consistency of the common dimension between different views can be found to provide a basis for subsequent update operations. According to the channel similarity matrix, one of the two views to be outer-producted is updated to obtain two views with higher feature channel similarity. Prepare for the construction of a structured RAD cube so that the updated features can better reflect the relationship between multiple dimensions. The two updated low-dimensional views are reconstructed according to the outer product module to obtain a reconstructed RAD feature cube. The outer product operation can combine the low-dimensional view features into a high-dimensional RAD feature cube. This reconstruction method can explicitly construct a cube structured format with three RAD dimensions in the feature space. In the feature space of the multi-view fusion network, the outer product module is the key operation to achieve the reconstruction from the low-dimensional feature layer to the high-dimensional RAD feature cube. It provides strong support for multi-view fusion and the mining of dimensional coupling relationships through the effective combination of low-dimensional features.

[0037] In one embodiment, a feature similarity learning module is used to calculate a channel similarity matrix of each of the RA view, the RD view, and the AD view along the feature channel dimension, and one of the two low-dimensional views obtained by the outer product is updated according to the channel similarity matrix to obtain an updated low-dimensional view, including:

[0038] The two input views are regarded as query and key respectively, called query view and key view, and then the channel similarity matrix of the two views along the feature channel dimension is calculated;

[0039] The key view is used as a value to calculate the weighted sum of itself and the channel similarity matrix to obtain the updated key view, and then the original query view and the updated key view are input into the outer product module.

[0040] In one embodiment, the weighted sum of the key view itself and the channel similarity matrix is ​​calculated as the updated key view using the key view as a value, including:

[0041] The key view is used as the value to calculate the weighted sum of itself and the channel similarity matrix, and the updated key view is

[0042] V new (q,k,v)=Softmax(qk T )v

[0043] Among them, V new is the updated key view, where q, k, and v are query, key, and value respectively.

[0044] In a specific embodiment, the correctness of the outer product operation has a high requirement on the consistent feature distribution of the dimensions shared by the two views being outer-producted. Therefore, directly performing the outer product on the views output from two independent encoders may not strictly meet the above consistency requirement, which will lead to a decrease in the confidence of the outer product operation and thus affect the performance of the overall network. In order to achieve correct feature alignment in the shared dimension, we introduce a method before the outer product operation such as Figure 3 The feature similarity learning module shown in Figure 3 is used to ensure the effectiveness of the outer product module.

[0045] A channel similarity matrix (CSM) is designed to learn the channel similarity between two views to be outer-producted. Taking the RA and RD views as input to the FSLM, the FSLM treats the two input views as the query (Q) and key (K), respectively, and then calculates the channel similarity matrix (CSM) of the two views along the feature channel dimension (Channel, C). The FSLM then uses the RD as the value (V), calculates its weighted sum with the CSM, and inputs it into the subsequent outer-product module.

[0046] Specifically, taking the RA and RD views as the input of FSLM, FSLM compresses the azimuth axis and Doppler axis in the RA and RD views respectively to obtain RC RA and RC RD , and learn the similarities between them to obtain the updated RD features.

[0047] In one embodiment, reconstructing the updated low-dimensional view and another original low-dimensional view according to the outer product module to obtain a reconstructed RAD cube includes:

[0048] Taking RA and RD views as an example, the RAD feature cube reconstructed with the R axis as the common dimension The corresponding RA view and RD view Reconstructed by outer product operation. Where B, C, R, A, and D represent batch size, number of feature channels, range, azimuth, and Doppler velocity, respectively.

[0049] In one embodiment, the formula for the outer product operation is:

[0050]

[0051] in, Represents the outer product operation.

[0052] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0053] In one embodiment, Figure 4 As shown, a radar scene perception device based on RADRadar is provided, including: a data acquisition and model building module 402, a feature consistency module 404 and a feature reconstruction module 406, wherein:

[0054] The data acquisition and model building module 402 is used to acquire compressed multi-view radar observation data; the multi-view radar observation data includes RA view, RD view and AD view; and build a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module;

[0055] The feature consistency module 404 calculates a channel similarity matrix of each of the RA view, the RD view, and the AD view along the feature channel dimension using the feature similarity learning module, and updates one of the two low-dimensional views obtained by the outer product according to the channel similarity matrix to obtain an updated low-dimensional view.

[0056] The feature reconstruction module 406 is configured to reconstruct the updated low-dimensional view and another original low-dimensional view according to the outer product module to obtain a reconstructed RAD feature cube.

[0057] The specific limitations of the RADRadar-based radar scene perception device can be found in the limitations of the RADRadar-based radar scene perception method described above and will not be further elaborated here. Each module in the RADRadar-based radar scene perception device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the modules described above can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each of the modules described above.

[0058] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a radar scene perception method based on RADRadar is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0059] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0060] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0061] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0062] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A radar scene perception method based on RADRadar, characterized in that: The method comprises: Acquire compressed multi-view radar observation data; the multi-view radar observation data includes RA view, RD view and AD view; construct a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module; Calculating a channel similarity matrix of each of the RA view, the RD view, and the AD view along the feature channel dimension using a feature similarity learning module, and updating one of the two low-dimensional views obtained by the outer product according to the channel similarity matrix to obtain an updated low-dimensional view; The updated low-dimensional view and another original low-dimensional view are reconstructed according to the outer product module to obtain a reconstructed RAD feature cube.

2. The method according to claim 1, characterized in that Utilizing a feature similarity learning module to calculate a channel similarity matrix of each of the RA view, the RD view, and the AD view along the feature channel dimension, and updating one of the two low-dimensional views obtained by the outer product according to the channel similarity matrix to obtain an updated low-dimensional view, including: The two low-dimensional views of the input are regarded as query and key respectively, called query view and key view, and then the channel similarity matrix of the two views along the feature channel dimension is calculated; The key view is used as a value to calculate the weighted sum of itself and the channel similarity matrix to obtain the updated key view, and then the original query view and the updated key view are input into the outer product module.

3. The method according to claim 2, characterized in that The key view is used as the value to calculate the weighted sum of itself and the channel similarity matrix as the updated key view, including: The key view is used as the value to calculate the weighted sum of itself and the channel similarity matrix, and the updated key view is V new (q,k,v)=Softmax(qk T )v Among them, V new is the updated key view, where q, k, and v are query, key, and value respectively.

4. The method according to any one of claims 1 to 3, characterized in that The updated low-dimensional view and another original low-dimensional view are reconstructed according to the outer product module to obtain a reconstructed RAD feature cube, including: RAD feature cube reconstructed with R axis as common dimension The corresponding RA view and RD view Reconstructed by outer product operation, B, C, R, A, D represent batch size, number of feature channels, distance, azimuth, and Doppler velocity, respectively.

5. The method according to claim 4, characterized in that The formula for the outer product operation is in, Represents the outer product operation.

6. A radar scene perception device based on RADRadar, characterized in that: The device comprises: A data acquisition and model building module is used to acquire compressed multi-view radar observation data; the multi-view radar observation data includes RA view, RD view and AD view; and build a RADRadar model; the RADRadar model includes a feature similarity learning module and an outer product module; a feature consistency module, configured to calculate a channel similarity matrix of each of the RA view, the RD view, and the AD view along the feature channel dimension using the feature similarity learning module, and update one of the two low-dimensional views obtained by the outer product according to the channel similarity matrix to obtain an updated low-dimensional view; The feature reconstruction module is used to reconstruct the updated low-dimensional view and another original low-dimensional view according to the outer product module to obtain a reconstructed RAD feature cube.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images