Explicit doppler encoding and physically constrained radar semantic segmentation method and system

Abstract

The application discloses a radar semantic segmentation method and system based on explicit Doppler coding and physical constraints, and relates to the technical fields of intelligent sensing and radar signal processing. The method comprises the following steps: acquiring original range-angle-Doppler spectrum data of a millimeter wave radar and performing preprocessing; extracting Doppler motion features and generating motion sensing feature maps and dynamic saliency weight maps through baseline estimation and spectrum compensation; fusing the motion sensing feature maps and the dynamic saliency weight maps through a self-attention mechanism and a cross-attention mechanism to generate multi-view fusion feature maps containing motion semantic saliency; wherein, for the multi-view fusion feature maps of continuous time instants, a dynamic persistent constraint loss function is constructed to impose time consistency regularization on the Doppler motion features; and the multi-view fusion feature maps are input into a semantic decoding network to output radar semantic segmentation results. The application effectively improves the accuracy, stability and physical consistency of radar semantic segmentation in complex dynamic scenes.

Description

Technical Field

[0001] This application relates to the fields of intelligent sensing and radar signal processing technology, and in particular to a radar semantic segmentation method and system with explicit Doppler coding and physical constraints. Background Technology

[0002] With the rapid development of autonomous driving and intelligent unmanned systems, environmental perception, as one of its core technologies, places higher demands on the robustness, all-weather capability, and reliability of perception systems. Compared to optical cameras, millimeter-wave radar has the advantages of being insensitive to changes in lighting and being able to operate in all weather conditions. It can still stably acquire target information in harsh environments such as rain, fog, and nighttime, thus gradually becoming an important sensor in autonomous driving perception systems. Radar semantic segmentation, as a key task in radar perception, aims to perform pixel-level or unit-level semantic annotation on radar echo data to distinguish different types of targets such as roads, vehicles, pedestrians, and obstacles. Existing radar semantic segmentation methods are mainly based on multi-view representations of radar spectra, such as range-Doppler view, range-angle view, and angle-Doppler view. They typically use deep neural networks to extract features from different view spectra separately and obtain semantic prediction results through feature concatenation or simple fusion.

[0003] However, the above methods still have many shortcomings in practical applications. First, Doppler information modeling is insufficient. Most current methods treat the Doppler dimension as a regular feature channel, lacking explicit modeling of its physical meaning. This easily leads to the dilution of velocity information during network propagation, making it difficult to effectively distinguish between static backgrounds and dynamic targets. Second, the problem of semantic inconsistency across multiple perspectives is prominent. There are natural spatial mapping differences between different radar perspectives, and simple feature stitching cannot guarantee the semantic consistency of moving targets across multiple perspectives, resulting in contradictory or ambiguous areas in the segmentation results. In addition, the temporal dimension is not fully utilized. Most methods only make predictions based on single-frame or short-time-series radar data, ignoring the temporal continuity of the actual target motion. This causes the semantic results to be prone to jitter or instability in dynamic scenes, making it difficult to meet the high stability and high reliability requirements of autonomous driving perception systems.

[0004] Therefore, how to explicitly model Doppler physical characteristics, effectively fuse multi-view information, and introduce a reasonable time consistency constraint mechanism in radar semantic segmentation tasks to improve the accuracy and stability of radar semantic segmentation in complex dynamic scenarios has become a key problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a radar semantic segmentation method and system based on explicit Doppler coding and physical constraints. By introducing explicit Doppler feature extraction, a multi-view fusion mechanism of Doppler perception, and dynamic continuous constraints into a deep learning framework, the accuracy, stability, and physical consistency of radar semantic segmentation in complex dynamic scenarios are effectively improved.

[0006] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a radar semantic segmentation method based on explicit Doppler coding and physical constraints, comprising: acquiring raw range-angle-Doppler spectrum data of millimeter-wave radar and performing preprocessing to obtain spectral information at a preset resolution; the spectral information includes: range-Doppler spectrum and angle-Doppler spectrum; Based on the distance-Doppler spectrum and the angle-Doppler spectrum, Doppler motion features are extracted through baseline estimation and spectral compensation, and a motion perception feature map and a dynamic saliency weight map are generated; the dynamic saliency weight map is used to characterize the degree of motion saliency at different spatial locations. The radar viewpoint features and the Doppler motion features are input into a multi-view radar feature fusion network, and fused through a self-attention mechanism and a cross-attention mechanism to generate a multi-view fused feature map containing motion semantic saliency; the multi-view radar feature fusion network is obtained after deep interactive modeling based on a semantically saliency-guided feature fusion mechanism. In the training phase of the multi-view radar feature fusion network, a dynamic continuous constraint loss function is constructed for the multi-view fusion feature maps at consecutive time points to apply time consistency regularization; the dynamic continuous constraint loss function is determined based on the dynamic saliency weight map, according to the multi-view fusion feature map or motion perception feature map. The multi-view fused feature map is input into the semantic decoding network, and the radar semantic segmentation result is output.

[0007] In one embodiment, based on the distance-Doppler spectrum and the angle-Doppler spectrum, Doppler motion features are extracted through baseline estimation and spectral compensation, and a motion-aware feature map and a dynamic saliency weight map are generated, including: Based on the distance-Doppler spectrum and the angle-Doppler spectrum, global velocity statistics are extracted by pooling along the Doppler dimension, and the global velocity statistics are input into a multilayer perceptron network to predict the Doppler baseline offset. The Doppler baseline offset is subtracted from the original Doppler spectrum to obtain the baseline-de-baseline Doppler signal; The baseline-de-baseline Doppler signal is subjected to a Fourier transform, multiplied frequency-wise with a learnable spectral filter, and restored to the Doppler axis through an inverse Fourier transform to generate the motion-sensing feature map, and the dynamic saliency weight map is output synchronously.

[0008] In one embodiment, the fusion using self-attention and cross-attention mechanisms includes: modeling the global dependencies within the radar viewpoint features through a self-attention branch to generate self-attention weighted features; injecting Doppler motion semantics into the radar viewpoint features using the radar viewpoint features as queries and the Doppler motion features as keys and values ​​through a cross-attention branch to generate cross-attention weighted features; and concatenating the self-attention weighted features and the cross-attention weighted features along the channel dimension to generate the multi-view fusion feature map.

[0009] In one embodiment, both the self-attention branch and the cross-attention branch employ a residual learning structure, which fuses the attention output with the original radar viewpoint features element by element.

[0010] In one embodiment, the dynamic persistent constraint loss function specifically involves calculating the consistency loss between the multi-view fusion feature map or the motion-aware feature map at adjacent time points only for spatial regions in the dynamic saliency weight map whose weights are greater than a preset threshold.

[0011] In one embodiment, the consistency loss is the sum of squares of the Euclidean distances between feature maps at adjacent time points.

[0012] In one embodiment, the preprocessing includes: amplitude normalization, noise suppression, and size alignment of the raw distance-angle-Doppler spectrum data; wherein the noise suppression includes amplitude threshold filtering, spectral smoothing, or clutter suppression based on statistical characteristics.

[0013] In one embodiment, the radar viewpoint features include range-Doppler viewpoint features or range-angle viewpoint features.

[0014] In one embodiment, the model composed of the multi-view radar feature fusion network and the semantic decoding network is optimized end-to-end using a joint loss function during model training. The joint loss function includes a semantic classification loss term and a dynamic persistence constraint loss term, which is determined by the dynamic persistence constraint loss function.

[0015] Secondly, this application provides a radar semantic segmentation system with explicit Doppler coding and physical constraints, including: a data acquisition and preprocessing module, used to acquire the raw range-angle-Doppler spectrum data of millimeter-wave radar and perform preprocessing to obtain spectral information at a preset resolution; the spectral information includes: range-Doppler spectrum and angle-Doppler spectrum; The Doppler feature extraction module, with its input connected to the output of the data acquisition and preprocessing module, is used to perform baseline estimation and spectral compensation on the distance-Doppler spectrum and the angle-Doppler spectrum to extract Doppler motion features and generate a motion-aware feature map and a dynamic saliency weight map; the dynamic saliency weight map is used to characterize the degree of motion saliency at different spatial locations. The multi-view fusion module has a first input connected to the output of the data acquisition and preprocessing module, and a second input connected to the output of the Doppler feature extraction module. It is used to input radar viewpoint features and Doppler motion features into a multi-view radar feature fusion network, where they are fused using self-attention and cross-attention mechanisms to generate a multi-view fused feature map containing motion semantic saliency. The multi-view radar feature fusion network is obtained after deep interactive modeling based on a semantically saliency-guided feature fusion mechanism. The temporal constraint module, whose input is connected to the output of the multi-view fusion module, is used to construct a dynamic continuous constraint loss function for the multi-view fusion feature maps at consecutive time points during the training phase of the multi-view radar feature fusion network, so as to apply temporal consistency regularization; the dynamic continuous constraint loss function is determined based on the dynamic saliency weight map, according to the multi-view fusion feature map or motion perception feature map. The semantic decoding module, whose input is connected to the output of the multi-view fusion module, is used to perform semantic decoding on the multi-view fusion feature map and output radar semantic segmentation results.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a radar semantic segmentation method and system based on explicit Doppler coding and physical constraints. By setting an explicit Doppler feature extraction step, baseline estimation and spectral compensation are performed on the range-Doppler spectrum and angle-Doppler spectrum, elevating the Doppler dimension from ordinary feature channels to a motion-aware representation with explicit physical meaning. This setting allows the target's velocity structure to be independently modeled during the feature extraction stage, avoiding the problem of Doppler information being diluted or weakened during propagation in deep networks. By setting a feature fusion step guided by semantic saliency, and employing a combination of self-attention and cross-attention mechanisms, the system further enhances the Doppler semantics of radar semantic segmentation. The explicit injection of Doppler motion features into radar viewpoint features overcomes the semantic inconsistency caused by the simple splicing of traditional multi-view features. This enables the multi-view radar feature fusion network to adaptively focus on moving target regions and suppress static background interference at the feature level, thereby improving the spatial consistency and boundary clarity of multi-view semantic segmentation results. By introducing a dynamic persistence constraint loss function and applying temporal consistency regularization, this setting utilizes the prior knowledge of the continuity of real target motion in the temporal dimension to guide the network to learn stable temporal feature representations, effectively suppressing the jitter in semantic segmentation results caused by noise, vehicle motion, or instantaneous interference. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an application environment diagram of a radar semantic segmentation method based on explicit Doppler coding and physical constraints according to an embodiment of this application; Figure 2 A flowchart illustrating a radar semantic segmentation method based on explicit Doppler coding and physical constraints, provided as an embodiment of this application; Figure 3 This is a detailed flowchart illustrating the extraction of Doppler motion features in one embodiment of this application; Figure 4 This is a schematic diagram of the architecture of a multi-view radar feature fusion network in one embodiment of this application; Figure 5 This is a detailed flowchart illustrating the multi-view fusion mechanism in one embodiment of this application; Figure 6 This is a schematic diagram of the attention branch structure in the multi-view fusion module of one embodiment of this application; Figure 7This is a schematic diagram of the functional modules of a radar semantic segmentation system with explicit Doppler coding and physical constraints, provided as an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] This application provides an explicit Doppler coding and physically constrained radar semantic segmentation method, which can be applied to, for example... Figure 1 In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on another server. Terminal 101 can send the raw range-angle-Doppler spectrum data collected from millimeter-wave radar to server 102. After receiving the raw range-angle-Doppler spectrum data, server 102 executes the radar semantic segmentation method based on explicit Doppler coding modeling and physical constraints. This method preprocesses the received radar spectrum data, extracts explicit Doppler features, fuses multi-view features from Doppler perception, and performs semantic decoding to generate a radar semantic segmentation result. Server 102 can then feed back the obtained radar semantic segmentation result to terminal 101 for subsequent applications such as environmental perception, path planning, or target tracking. Furthermore, in some embodiments, the radar semantic segmentation method based on explicit Doppler coding modeling and physical constraints can also be implemented separately by the server 102 or the terminal 101. For example, the terminal 101 can directly perform radar semantic segmentation processing on the raw range-angle-Doppler spectrum data collected by the vehicle millimeter-wave radar and use the segmentation results for local autonomous driving decisions; or the server 102 can read the pre-stored radar spectrum data from the data storage system, perform offline semantic segmentation processing on the batch data, and use it for model training or data annotation.

[0022] Among them, terminal 101 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices and portable wearable devices, and server 102 can be implemented by independent servers or server clusters composed of multiple servers, or it can be a cloud server.

[0023] In one exemplary embodiment, such as Figure 2 As shown, a radar semantic segmentation method based on explicit Doppler coding and physical constraints is provided. This method is executed by a computer device and includes the following steps: S1. Acquire the raw range-angle-Doppler spectrum data of the millimeter-wave radar and preprocess it to obtain spectral information at a preset resolution. The spectral information includes: range-Doppler spectrum, range-angle spectrum, and angle-Doppler spectrum.

[0024] S2, based on distance-Doppler and angle-Doppler spectra, extracts Doppler motion features through baseline estimation and spectral compensation, and generates a motion-aware feature map and a dynamic saliency weight map. The dynamic saliency weight map is used to characterize the degree of motion saliency at different spatial locations.

[0025] S3 inputs radar perspective features and Doppler motion features into a multi-view radar feature fusion network, which fuses them through self-attention and cross-attention mechanisms to generate a multi-view fused feature map containing motion semantic saliency. The multi-view radar feature fusion network is obtained after deep interactive modeling based on a semantically saliency-guided feature fusion mechanism.

[0026] In the training phase of the multi-view radar feature fusion network, a dynamic persistence constraint loss function is constructed for the multi-view fused feature maps at consecutive time points to apply time consistency regularization. The dynamic persistence constraint loss function is determined based on a dynamic saliency weight map, according to the multi-view fused feature map or motion-sensing feature map.

[0027] S4 inputs the multi-view fused feature map into the semantic decoding network and outputs the radar semantic segmentation result.

[0028] By setting explicit Doppler feature extraction steps in steps S1 and S2, baseline estimation and spectral compensation are performed on the range-Doppler spectrum and angle-Doppler spectrum, elevating the Doppler dimension from ordinary feature channels to a motion-aware representation with clear physical meaning. This setting allows the target's velocity structure to be independently modeled during the feature extraction stage, avoiding the problem of Doppler information being diluted or weakened during propagation in the deep network. Through the feature fusion step based on semantic saliency set in step S3, a combination of self-attention and cross-attention mechanisms is used to explicitly inject Doppler motion features into the radar viewpoint features. This setting overcomes the semantic inconsistency defects caused by the simple splicing of traditional multi-view features, enabling the network to adaptively focus on moving target regions and suppress static background interference at the feature level, thereby improving the spatial consistency and boundary clarity of multi-view semantic segmentation results. By introducing a dynamic persistence constraint loss function to apply temporal consistency regularization, this setting utilizes the prior knowledge of the continuity of real target motion in the time dimension to guide the network to learn stable temporal feature representations, effectively suppressing the jitter of semantic segmentation results caused by noise, vehicle motion, or instantaneous interference.

[0029] In specific implementation, step S1 above acquires the raw range-angle-Doppler spectrum data collected by the millimeter-wave radar at continuous intervals. The raw range-angle-Doppler (RAD) spectrum data is represented as a three-dimensional tensor in RAD form, which can be expressed as: .

[0030] in, Indicates the distance dimension index. Indicates the angle dimension index. This indicates the Doppler dimension index.

[0031] To reduce the impact of noise and environmental clutter on subsequent processing, preprocessing operations are performed on the RAD data, including amplitude normalization, noise suppression, and size alignment. Specifically, amplitude normalization is performed on the RAD spectral data to eliminate the influence of amplitude distribution differences under different sampling conditions. During noise suppression, cells with spectral amplitudes below a preset threshold can be suppressed, or clutter regions can be weakened based on statistical distribution. This processing can be expressed as follows: .

[0032] in, This represents a noise suppression function, which includes, but is not limited to, amplitude threshold filtering, spectral smoothing, and clutter suppression operations based on statistical properties. This represents a weakened RAD-form three-dimensional tensor.

[0033] After noise suppression is completed, the radar spectrum data is mapped to range-Doppler and range-angle views. The different view spectrums are then size-aligned by interpolation resampling or cropping and padding to ensure the consistency of multi-view data in spatial dimensions, resulting in range-Doppler, range-angle, and angle-Doppler spectra at preset resolutions.

[0034] In specific implementation, such as Figure 3 As shown, in step S2 above, Doppler motion features are extracted through baseline estimation and spectral compensation, specifically including: S21, global velocity statistics are extracted by pooling along the Doppler dimension, and the global velocity statistics are input into a multilayer perceptron network to predict the Doppler baseline offset.

[0035] Specifically, baseline estimation is performed on the Doppler dimension to model the systematic Doppler shift caused by vehicle motion and static background. In this embodiment, global velocity statistical features are extracted by performing pooling operations on the Doppler dimension. : Furthermore, the statistical features are input into a learnable mapping function to predict the corresponding Doppler baseline offset. : ;in, This represents a learnable function composed of a multilayer sensing mechanism. This represents the pooling operation in the Doppler dimension.

[0036] S22, Subtract the Doppler baseline offset from the original Doppler spectrum to obtain the baseline-de-baseline Doppler signal. .

[0037] Specifically, based on the Doppler baseline, the compensation processing of the original Doppler response is expressed as follows: .

[0038] This step effectively reduces the influence of static background and vehicle motion on the Doppler distribution.

[0039] S23 performs a Fourier transform on the baseline-de-baseline Doppler signal, performs frequency-wise multiplication with a learnable spectral filter, and restores the Doppler axis through an inverse Fourier transform to generate a motion-sensing feature map.

[0040] Specifically, after baseline removal processing, a spectral compensation operation is performed on the Doppler signal to enhance the effective spectral response of the real moving target. Specifically, a frequency domain transformation is performed on the Doppler-dimensional signal to obtain its spectral representation. : .

[0041] in, Indicates frequency index, This represents the frequency domain transform operator along the Doppler dimension. This represents the distance-angle-Doppler response signal after Doppler baseline compensation.

[0042] Subsequently, a learnable spectrum filter was introduced. Frequency-by-frequency modulation is performed on the spectrum to obtain the modulated spectrum. : .

[0043] Furthermore, the Doppler domain is recovered through inverse transform. , means as follows: .

[0044] This spectral compensation process enhances the Doppler response (effective spectral response) of a real moving target while suppressing spectral leakage and random noise, generating a Doppler sensing feature map containing explicit velocity structure information, also known as a motion sensing feature map.

[0045] Furthermore, a dynamic saliency weight map is generated based on the compensated Doppler features. It can be represented as: .

[0046] in, Represents the feature aggregation function, Represents the normalization or activation function. The dynamic significance weighting plot is used to reflect the significance of motion at each spatial location.

[0047] Figure 4This is a schematic diagram of the architecture of the multi-view radar feature fusion network constructed in this application. The multi-view radar feature fusion network constructed in this application includes: three view branch encoders, one Doppler explicit sensing encoder, two cross-view fusion modules, and two view decoders. Specifically, the RA (range-angle) view input is processed by the RA branch encoder to obtain RA features, the RD (range-Doppler) view input is processed by the RD branch encoder to obtain RD features, and the AD (angle-Doppler) view input is processed by the AD branch encoder to obtain AD features. The RD view input and the AD view input are also jointly input to the Doppler explicit sensing encoder to obtain Doppler features. The AD features and the Doppler (Doppler) features are input to the Doppler-AD fusion module, which outputs RA interactive features and passes them to the Doppler-RD fusion module. The RD features, Doppler features, and RA interactive features are jointly input to the Doppler-RD fusion module to obtain the fused features. The fused features are input to the RA decoder and the RD decoder respectively, and the corresponding outputs are the RA view prediction results and the RD view prediction results. The multi-view fusion mechanism of this step will be explained in detail below with reference to the accompanying drawings, such as... Figure 5 As shown, it specifically includes: S31, through a self-attention branch, models the global dependencies within the radar viewpoint features and generates self-attention weighted features.

[0048] Specifically, the range-Doppler viewpoint features or range-angle viewpoint features are input into the Doppler motion features, which are guided by semantic saliency. Let the radar features from the RD or AD viewpoint be represented as... The Doppler semantic features generated by the explicit Doppler coding module are represented as follows: .

[0049] The semantic saliency guidance module includes a self-attention branch and a cross-attention branch. The self-attention branch models the global dependencies within the radar viewpoint features, generating query, key, and value vectors through three 1×1 convolution operations, and then rearranging them to accommodate attention computation. The cross-attention branch uses the radar viewpoint features as queries and Doppler motion features as keys and values, explicitly injecting Doppler motion semantics into the radar viewpoint features. Its attention relationships... It can be represented as: .

[0050] Among them, subscript Indicates cross-attention branches, This represents a query mapping generated from radar viewpoint features. and These represent the key and value mappings generated by Doppler features, respectively.

[0051] S32, through cross-attention branching, uses radar viewpoint features as queries and Doppler motion features as keys and values, injects Doppler motion semantics into radar viewpoint features, and generates cross-attention weighted features.

[0052] Specifically, after obtaining the attention map, the self-attention branch and the cross-attention branch respectively calculate the corresponding weighted feature representations, which are calculated in the following form: .

[0053] .

[0054] in, This represents the self-attention feature calculated based on radar viewpoint characteristics; This represents the value mapping function in the self-attention branch; This represents the attention relation matrix in the self-attention branch; This represents the cross-attention weighted feature of the cross-attention branch output; This represents the attention relationship matrix in the cross-attention branches; Indicates by The generated value mapping.

[0055] Figure 6 This is a schematic diagram of the attention branch structure in the multi-view fusion module of this application. The multi-view fusion module includes a first processing branch and a second processing branch. The first processing branch is used to perform attention enhancement processing on the distance-Doppler features, and the second processing branch is used to perform attention enhancement processing on the Doppler features. The output features of the first processing branch and the second processing branch are concatenated to obtain the final output features.

[0056] Specifically, the first processing branch includes a query unit, a key unit, a value unit, an outer product unit, a Softmax normalization unit, a second outer product unit, and a residual summing unit. The outputs of the query unit and the key unit are input to the Softmax normalization unit after outer product operation. The output of the Softmax normalization unit and the output of the value unit are then input to the residual summing unit after second outer product operation. The residual summing unit adds the result of the second outer product operation to the input distance-Doppler feature to obtain the enhanced distance-Doppler feature. The second processing branch has the same structure as the first processing branch and is used to obtain the enhanced distance-Doppler interaction feature. The enhanced distance-Doppler feature and the enhanced distance-Doppler interaction feature are concatenated by a concatenation unit and then output.

[0057] To maintain the stability of the original radar features while incorporating Doppler semantic information, this embodiment employs a residual learning structure in both attention branches, fusing the attention output with the original radar features element-wise. The format is: .

[0058] in, This represents a linear mapping operator implemented through 1×1 convolution. This indicates the element-wise addition of residuals. For the first Attention-weighted feature functions output by each attention branch; Index for attention branches; This is a self-attention branch; This is a cross-attention branch.

[0059] S33 concatenates the self-attention weighted features and cross-attention weighted features along the channel dimension to generate a multi-view fusion feature map.

[0060] Specifically, the features output by the self-attention branch Features of the cross-attention branch output The features are concatenated along the channel dimension to form a fused saliency-guided feature representation. : .

[0061] in, This represents the feature concatenation operation along the channel dimension. Through the above fusion method, a multi-view fused feature map containing motion semantic saliency is generated.

[0062] In practical implementation, during the training phase of the multi-view radar feature fusion network, a dynamic persistence constraint loss function is introduced for the multi-view fusion feature maps at consecutive time points, and temporal consistency regularization is applied to the target motion state. For consecutive time frames... and Only in spatial regions where the dynamic significance weight is higher than a preset threshold. Internally, the consistency loss between features at adjacent time points is jointly optimized, and the loss function is dynamically and continuously constrained. Defined as: . in, and These represent Doppler sensing features (motion sensing features) or fusion features (multi-view fusion features) at adjacent time points, respectively. Represents a set of dynamically significant regions. Indexed by angle dimension.

[0063] By minimizing the aforementioned dynamic persistence constraint loss function during training, the network is guided to learn the continuous change characteristics of the target motion in the time dimension, thereby effectively suppressing semantic prediction fluctuations caused by instantaneous noise or interference.

[0064] In practice, the final fused feature map (multi-view fused feature map) is input into the semantic decoding network. The semantic decoding network includes multi-layer upsampling modules and convolutional modules, and uses a skip connection structure to fuse shallow and deep features to simultaneously preserve spatial detail information and high-level semantic information. The final output is the radar semantic segmentation results under the range-Doppler view and the range-angle view. .

[0065] in, This represents the semantic segmentation result of radar. This indicates motion perception features or multi-view fusion features. This represents a semantic decoding network.

[0066] The model, composed of a multi-view radar feature fusion network and a semantic decoding network, employs a joint loss function for end-to-end optimization during model training. The loss term includes semantic classification and dynamic persistence constraint, which can be expressed in the following form: .

[0067] in, This represents the semantic classification loss term. This represents the dynamic persistence constraint loss function. These are the weighting coefficients.

[0068] In one exemplary embodiment, such as Figure 7 As shown, a radar semantic segmentation system based on explicit Doppler coding and physical constraints is provided, including: The data acquisition and preprocessing module 201 is used to acquire the raw range-angle-Doppler spectrum data of the millimeter-wave radar and perform preprocessing to obtain spectral information at a preset resolution; the spectral information includes: range-Doppler spectrum and angle-Doppler spectrum.

[0069] The Doppler feature extraction module 202, with its input end connected to the output end of the data acquisition and preprocessing module, is used to perform baseline estimation and spectral compensation on the distance-Doppler spectrum and the angle-Doppler spectrum to extract Doppler motion features and generate a motion-aware feature map and a dynamic saliency weight map; the dynamic saliency weight map is used to characterize the degree of motion saliency at different spatial locations.

[0070] The multi-view fusion module 203 has a first input terminal connected to the output terminal of the data acquisition and preprocessing module, and a second input terminal connected to the output terminal of the Doppler feature extraction module. It is used to input radar view features and Doppler motion features into the multi-view radar feature fusion network, and fuse them through self-attention mechanism and cross-attention mechanism to generate a multi-view fused feature map containing motion semantic saliency. The multi-view radar feature fusion network is obtained after deep interactive modeling based on the semantic saliency-guided feature fusion mechanism.

[0071] The temporal constraint module 204, with its input end connected to the output end of the multi-view fusion module, is used to construct a dynamic continuous constraint loss function for the multi-view fusion feature map at consecutive time points during the training phase of the multi-view radar feature fusion network, so as to apply temporal consistency regularization; the dynamic continuous constraint loss function is determined based on the dynamic saliency weight map and according to the multi-view fusion feature map or motion perception feature map.

[0072] The semantic decoding module 205 has its input end connected to the output end of the multi-view fusion module, and is used to perform semantic decoding on the multi-view fusion feature map and output radar semantic segmentation results.

[0073] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a radar semantic segmentation method based on explicit Doppler coding and physical constraints.

[0074] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0075] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0076] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0077] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Furthermore, any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A radar semantic segmentation method based on explicit Doppler coding and physical constraints, characterized in that, include: The raw range-angle-Doppler spectrum data of the millimeter-wave radar is acquired and preprocessed to obtain spectral information at a preset resolution. The spectral information includes: distance-Doppler spectrum and angle-Doppler spectrum; Based on the distance-Doppler spectrum and the angle-Doppler spectrum, Doppler motion features are extracted through baseline estimation and spectral compensation, and a motion perception feature map and a dynamic saliency weight map are generated; the dynamic saliency weight map is used to characterize the degree of motion saliency at different spatial locations. The radar viewpoint features and the Doppler motion features are input into a multi-view radar feature fusion network, and fused through a self-attention mechanism and a cross-attention mechanism to generate a multi-view fused feature map containing motion semantic saliency; the multi-view radar feature fusion network is obtained after deep interactive modeling based on a semantically saliency-guided feature fusion mechanism. In the training phase of the multi-view radar feature fusion network, a dynamic continuous constraint loss function is constructed for the multi-view fusion feature maps at consecutive time points to apply time consistency regularization; the dynamic continuous constraint loss function is determined based on the dynamic saliency weight map, according to the multi-view fusion feature map or motion perception feature map. The multi-view fused feature map is input into the semantic decoding network, and the radar semantic segmentation result is output. Based on the distance-Doppler spectrum and the angle-Doppler spectrum, Doppler motion features are extracted through baseline estimation and spectral compensation, and motion-sensing feature maps and dynamic saliency weight maps are generated, including: Based on the distance-Doppler spectrum and the angle-Doppler spectrum, global velocity statistics are extracted by pooling along the Doppler dimension, and the global velocity statistics are input into a multilayer perceptron network to predict the Doppler baseline offset. The Doppler baseline offset is subtracted from the original Doppler spectrum to obtain the baseline-de-baseline Doppler signal; The baseline-de-baseline Doppler signal is subjected to a Fourier transform, multiplied frequency-wise with a learnable spectral filter, and restored to the Doppler axis through an inverse Fourier transform to generate the motion-sensing feature map, and the dynamic saliency weight map is output synchronously.

2. The radar semantic segmentation method based on explicit Doppler coding and physical constraints according to claim 1, characterized in that, The fusion through self-attention and cross-attention mechanisms includes: By using a self-attention branch, the global dependencies within the radar viewpoint features are modeled to generate self-attention weighted features. By using cross-attention branches, the radar viewpoint features are used as queries, and the Doppler motion features are used as keys and values. Doppler motion semantics are injected into the radar viewpoint features to generate cross-attention weighted features. The self-attention weighted features and the cross-attention weighted features are concatenated along the channel dimension to generate the multi-view fusion feature map.

3. The radar semantic segmentation method based on explicit Doppler coding and physical constraints according to claim 2, characterized in that, Both the self-attention branch and the cross-attention branch adopt a residual learning structure, which adds and fuses the attention output with the original radar viewpoint features element by element.

4. The radar semantic segmentation method based on explicit Doppler coding and physical constraints according to claim 1, characterized in that, The dynamic persistence constraint loss function is specifically as follows: For spatial regions in the dynamic saliency weight map with weights greater than a preset threshold, the consistency loss between the multi-view fusion feature map or the motion-aware feature map at adjacent time points is calculated.

5. The radar semantic segmentation method based on explicit Doppler coding and physical constraints according to claim 4, characterized in that, The consistency loss is the sum of the squares of the Euclidean distances between feature maps at adjacent time points.

6. The radar semantic segmentation method based on explicit Doppler coding and physical constraints according to claim 1, characterized in that, The preprocessing includes: amplitude normalization, noise suppression, and size alignment of the original distance-angle-Doppler spectrum data; wherein, the noise suppression includes amplitude threshold filtering, spectral smoothing, or clutter suppression based on statistical characteristics.

7. The radar semantic segmentation method based on explicit Doppler coding and physical constraints according to claim 1, characterized in that, The radar view characteristics include range-Doppler view characteristics or range-angle view characteristics.

8. The radar semantic segmentation method based on explicit Doppler coding and physical constraints according to claim 1, characterized in that, The model, composed of the multi-view radar feature fusion network and the semantic decoding network, is optimized end-to-end using a joint loss function during model training. The joint loss function includes a semantic classification loss term and a dynamic persistence constraint loss term, which is determined by the dynamic persistence constraint loss function.

9. A radar semantic segmentation system based on explicit Doppler coding and physical constraints, characterized in that, include The data acquisition and preprocessing module is used to acquire the raw range-angle-Doppler spectrum data of the millimeter-wave radar and perform preprocessing to obtain spectral information at a preset resolution. The spectral information includes: distance-Doppler spectrum and angle-Doppler spectrum; The Doppler feature extraction module, with its input connected to the output of the data acquisition and preprocessing module, is used to perform baseline estimation and spectral compensation on the distance-Doppler spectrum and the angle-Doppler spectrum to extract Doppler motion features and generate a motion-aware feature map and a dynamic saliency weight map; the dynamic saliency weight map is used to characterize the degree of motion saliency at different spatial locations. The multi-view fusion module has a first input connected to the output of the data acquisition and preprocessing module, and a second input connected to the output of the Doppler feature extraction module. It is used to input radar viewpoint features and Doppler motion features into a multi-view radar feature fusion network, where they are fused using self-attention and cross-attention mechanisms to generate a multi-view fused feature map containing motion semantic saliency. The multi-view radar feature fusion network is obtained after deep interactive modeling based on a semantically saliency-guided feature fusion mechanism. The temporal constraint module, whose input is connected to the output of the multi-view fusion module, is used to construct a dynamic continuous constraint loss function for the multi-view fusion feature maps at consecutive time points during the training phase of the multi-view radar feature fusion network, so as to apply temporal consistency regularization; the dynamic continuous constraint loss function is determined based on the dynamic saliency weight map, according to the multi-view fusion feature map or motion perception feature map. The semantic decoding module, whose input end is connected to the output end of the multi-view fusion module, is used to perform semantic decoding on the multi-view fusion feature map and output radar semantic segmentation results. Based on the distance-Doppler spectrum and the angle-Doppler spectrum, Doppler motion features are extracted through baseline estimation and spectral compensation, and motion-sensing feature maps and dynamic saliency weight maps are generated, including: Based on the distance-Doppler spectrum and the angle-Doppler spectrum, global velocity statistics are extracted by pooling along the Doppler dimension, and the global velocity statistics are input into a multilayer perceptron network to predict the Doppler baseline offset. The Doppler baseline offset is subtracted from the original Doppler spectrum to obtain the baseline-de-baseline Doppler signal; The baseline-de-baseline Doppler signal is subjected to a Fourier transform, multiplied frequency-wise with a learnable spectral filter, and restored to the Doppler axis through an inverse Fourier transform to generate the motion-sensing feature map, and the dynamic saliency weight map is output synchronously.

Images