Non-line-of-sight human pose recognition method based on impulse multipath space-time feature learning

Abstract

The application discloses a non-line-of-sight human posture recognition method based on pulse multi-path space-time feature learning, which comprises the following steps: collecting pulse echo signals in a non-line-of-sight scene, performing clutter suppression, time domain truncation alignment and multi-domain feature extraction, splicing a distance-time graph and a micro-Doppler time-frequency spectrum along a channel dimension, and generating a three-dimensional pulse echo tensor; constructing a multi-scale convolution channel attention fusion network, extracting a multi-path space-time feature map from the three-dimensional pulse echo tensor through a parallel multi-scale convolution kernel and a channel attention mechanism; and inputting the time sequence encoder composed of a bidirectional long short-term memory network and a multi-head self-attention module in series to extract a posture time sequence feature vector; inputting the posture time sequence feature vector into a classification decision head to generate the confidence of each posture category, and outputting a human posture recognition result of a fusion decision combined with a preset threshold and a time sequence smoothing arbitration strategy. The application realizes high-precision and high-robustness recognition of human postures in a non-line-of-sight environment.

Description

Technical Field

[0001] This invention relates to the field of human body perception and radar signal processing technology, and in particular to a non-line-of-sight human posture recognition method based on pulse multipath spatiotemporal feature learning. Background Technology

[0002] With the rapid development of security monitoring, smart homes, and emergency rescue, the demand for human body perception under non-line-of-sight conditions is becoming increasingly prominent. Through-wall human posture recognition has significant application value in scenarios such as counter-terrorism, disaster search and rescue, and smart elderly care. Pulse radar, with its nanosecond-level temporal resolution and excellent wall penetration capability, can capture weak motion information and multipath scattering characteristics of human targets under non-line-of-sight conditions, and has become the mainstream technology in the field of non-line-of-sight human body perception. However, the human posture information in existing pulse radar echoes is highly coupled to the complex multipath propagation environment. The echo delay-amplitude patterns corresponding to different postures differ slightly and are significantly affected by factors such as wall material and transmitter-receiver spacing, posing a severe challenge to high-precision posture recognition.

[0003] Currently, the main methods for non-line-of-sight human pose recognition include the following: (1) Methods based on traditional radar micro-Doppler features mainly extract time-spectrum maps from pulse echoes through short-time Fourier transform and use template matching or shallow classifiers to identify limb movement patterns. The calculation process is relatively simple and can effectively distinguish large-amplitude movements such as walking and running in specific scenarios. However, this method usually relies on manually designed feature descriptors, which has limited ability to separate static or semi-static micro-movement postures such as standing, sitting, and squatting. Moreover, when the wall attenuation is large, the signal-to-noise ratio of the spectrum drops sharply, resulting in a serious degradation in recognition accuracy.

[0004] (2) End-to-end methods based on convolutional neural networks treat the distance-time spectrum of pulse echoes as a two-dimensional image, and use 2D-CNN or 3D-CNN to extract spatiotemporal features and map them to pose categories. Such methods can automatically learn the discriminative features in the echoes and achieve high recognition accuracy in line-of-sight scenarios. However, due to the fixed receptive field of the network and the lack of explicit modeling of the multipath propagation structure, in scenarios with severe non-line-of-sight multipath interference such as wall reflections and furniture clutter, the inter-class feature differences of different poses are masked by the multipath superposition components, resulting in increased feature confusion and insufficient generalization ability.

[0005] (3) Based on attention mechanisms and sequence models, some studies in recent years have attempted to introduce Transformer or Long Short-Term Memory networks into the field of radar human perception, using self-attention mechanisms to capture long-range dependencies in echo sequences. However, these methods usually feed the original sensor echoes directly into the attention module, lacking prior embedding of the physical structure of pulse radar echoes. They neither fully utilize the fine temporal structure information of nanosecond-level pulse signals nor incorporate the geometric scattering characteristics of multipath propagation into the network design, making the model lack robustness to changes in non-line-of-sight environments.

[0006] Existing non-line-of-sight human perception technologies have certain limitations in terms of feature resolution, multipath interference suppression, and environmental robustness. Shallow methods based on micro-Doppler lack sufficient feature abstraction levels; CNN-based methods have weak modeling capabilities for multipath structures; and attention-based methods lack prior constraints on radar physics.

[0007] Therefore, it is necessary to propose a solution to improve one or more problems existing in the above-mentioned related technical solutions.

[0008] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] The purpose of this disclosure is to provide a non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0010] A non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning, according to an embodiment of this disclosure, includes the following steps: Acquire pulse echo signals in non-line-of-sight scenarios, perform clutter suppression, time-domain truncation and alignment, and multi-domain feature extraction operations on the pulse echo signals, and stitch the distance-time spectrum and micro-Doppler time spectrum along the channel dimension to generate a three-dimensional pulse echo tensor; A multi-scale convolutional channel attention fusion network is constructed, and multipath spatiotemporal feature maps are extracted from the three-dimensional pulse echo tensor through parallel multi-scale convolutional kernels and channel attention mechanisms. The multipath spatiotemporal feature map is input into a temporal encoder consisting of a bidirectional long short-term memory network and a multi-head self-attention module connected in series to extract the attitude temporal feature vector. The posture temporal feature vector is input into the classification decision head to generate the confidence scores of each human posture category. The human posture recognition result is then output by combining a preset threshold and a temporal smoothing arbitration strategy.

[0011] In an exemplary embodiment of this application, the step of acquiring pulse echo signals in non-line-of-sight scenarios, performing clutter suppression, time-domain truncation and alignment, and multi-domain feature extraction operations on the pulse echo signals, and stitching the distance-time spectrum and the micro-Doppler time spectrum along the channel dimension to generate a three-dimensional pulse echo tensor includes: A single-transmitter dual-receiver pulse radar is used to simultaneously acquire multi-channel echo signals. In a target-free scenario, static background echoes are pre-acquired and mean templates are calculated. Each frame of echo acquired in real time is subtracted from the mean template point by point. Then, a bandpass filter is used to filter out baseline drift and high-frequency noise to obtain a clutter-suppressed pulse echo sequence. The effective echo time window range is determined based on the shortest flight time between the transmitted pulse and the receiving antenna and the maximum detection range of the target. Each frame of the pulse echo sequence is truncated in the time domain. The truncated echo sequence is aligned to a unified time reference according to the trigger time of the transmitted pulse and then digitized and peak normalized to obtain the range-time matrix. A short-time Fourier transform is performed frame by frame along the fast time dimension on the distance-time matrix to obtain the corresponding micro-Doppler time spectrum for each frame. Multiple consecutive distance-time spectra are stacked along the time dimension to form a first three-dimensional tensor. A corresponding number of micro-Doppler time spectrums are stacked along the time dimension to form a second three-dimensional tensor. The first three-dimensional tensor and the second three-dimensional tensor are spliced ​​along the channel dimension to obtain the standardized three-dimensional pulse echo tensor.

[0012] In an exemplary embodiment of this application, the multi-scale convolutional channel attention fusion network includes three parallel convolutional branches and a channel attention module; The kernel sizes of the three parallel convolutional branches are 3×3, 5×5, and 7×7, respectively. The convolutional stride of each branch is 1. The padding size matches the kernel size to ensure that the output feature size remains unchanged. The number of convolutional kernels in a single branch is 64. The channel attention module includes a global average pooling layer and a two-layer fully connected bottleneck structure. The reduction ratio is set to 16. The output of the fully connected layer is activated by Sigmoid to generate a normalized channel weight vector in the range of 0 to 1.

[0013] In an exemplary embodiment of this application, the step of constructing a multi-scale convolutional channel attention fusion network and extracting multipath spatiotemporal feature maps from the three-dimensional pulse echo tensor through parallel multi-scale convolutional kernels and channel attention mechanisms includes: The three-dimensional pulse echo tensor is expanded into a single-channel form along the channel dimension and fed into three parallel convolution branches with different convolution kernel sizes for feature extraction. The outputs of each branch are batch normalized and ReLU activated and then spliced ​​along the channel dimension to obtain a multi-scale fused feature map. The multi-scale fused feature map is input into the channel attention module, and global average pooling, two fully connected mapping layers and sigmoid activation are performed sequentially to generate normalized channel attention weight vectors for each channel. The multi-scale fused feature map is recalibrated by channel-by-channel multiplication using the channel attention weight vectors. The recalibrated feature map and the multi-scale fused feature map are added element-by-element through a skip residual connection to obtain the multipath spatiotemporal feature map.

[0014] In an exemplary embodiment of this application, the step of inputting the multipath spatiotemporal feature map into a temporal encoder composed of a bidirectional long short-term memory network and a multi-head self-attention module connected in series to extract the attitude temporal feature vector includes: The multipath spatiotemporal feature map is split into frame-level feature sequences along the time frame dimension. The frame-level feature sequences are input into a bidirectional long short-term memory network for forward and backward bidirectional encoding. The forward hidden state and backward hidden state of each time step are concatenated along the feature dimension to obtain the context-enhanced feature sequence. The context-enhanced feature sequence is input into the multi-head self-attention module. The global dependency weights between each time step are calculated by scaling dot product attention. The multi-head self-attention output and the context-enhanced feature sequence are summed through a skip residual connection and then processed by layer normalization. The feature vector of the last time step in the layer normalization output is taken as the pose temporal feature vector.

[0015] In an exemplary embodiment of this application, the bidirectional long short-term memory network includes a forward LSTM and a backward LSTM. The hidden layer dimension of both LSTMs is set to 128. The forward hidden state and the backward hidden state at each time step are concatenated to obtain the context enhancement feature sequence with a dimension of 256. The multi-head self-attention module contains four parallel attention heads, each with a model dimension of 64.

[0016] In an exemplary embodiment of this application, the step of inputting the posture temporal feature vector into a classification decision head to generate confidence scores for each human posture category, and outputting a fused decision human posture recognition result by combining a preset threshold and a temporal smoothing arbitration strategy, includes: The posture temporal feature vector is input into a two-layer fully connected classification decision head, and after nonlinear mapping, Dropout regularization and Softmax normalization, confidence score vectors corresponding to each human posture category are generated. Set a high confidence threshold and a confidence difference tolerance threshold. Based on the threshold determination results, choose to directly output the human pose category or start the temporal smoothing arbitration mechanism to output the human pose recognition result of fusion decision.

[0017] In an exemplary embodiment of this application, the step of inputting the posture temporal feature vector into a two-layer fully connected classification decision head, and generating a confidence score vector corresponding to each human posture category through nonlinear mapping, Dropout regularization, and Softmax normalization includes: The pose temporal feature vector is input into the classification decision head, which includes two fully connected hidden layers and a Softmax output layer. The first fully connected hidden layer maps the features to 128 dimensions and applies ReLU activation and Dropout regularization. The second fully connected hidden layer maps the features to the number of pre-human pose categories and inputs them into the Softmax normalization function, outputting the category confidence score vector. The human pose category index corresponding to the largest component in the human pose category confidence score vector is taken as the candidate decision result, and the corresponding highest confidence score is recorded.

[0018] In an exemplary embodiment of this application, the step of setting a high confidence threshold and a confidence difference tolerance threshold, selecting to directly output the human pose category or activate a temporal smoothing arbitration mechanism based on the threshold determination result, and outputting the fused decision human pose recognition result includes: Set a high confidence threshold and a tolerance threshold for confidence differences; If the highest confidence score of the current frame is greater than or equal to the high confidence threshold, the candidate decision result is directly output as the human pose recognition result of the fusion decision. If the highest confidence score of the current frame is lower than the high confidence threshold and the difference between the highest confidence score and the second highest confidence score is greater than the confidence difference tolerance threshold, then the current frame is marked as a frame to be arbitrated, the temporal smoothing arbitration strategy is started, and the arbitration output human pose category of the current frame is determined according to the mode vote of the fusion decision results of the three consecutive frames before the current frame, forming the human pose recognition result of the fusion decision.

[0019] In an exemplary embodiment of this application, if the majority vote of the fusion decision results of the preceding three consecutive frames is tied, the fusion decision result of the fourth frame before the current frame is used as the arbitration output human pose category of the current frame.

[0020] This application proposes a non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning. On the one hand, by suppressing clutter in the pulse echo and concatenating multi-domain features to generate a three-dimensional pulse echo tensor, and employing a multi-scale convolutional channel attention fusion network to extract multipath spatiotemporal features, it achieves high-resolution extraction of spatiotemporal features related to human pose in non-line-of-sight multipath environments, enhancing the ability to distinguish subtle features of static and micro-motion poses. On the other hand, by constructing a temporal encoder through a bidirectional long short-term memory network connected to multi-head self-attention, it mines dynamic temporal information of pose, and outputs recognition results with a threshold and temporal smoothing arbitration strategy. Relying on the temporal correlation of continuous frames to correct misjudgments of low-confidence frames, the continuous recognition process obtains smoother and more robust output results while maintaining response sensitivity, significantly improving the classification accuracy and generalization ability of human pose recognition in non-line-of-sight scenarios. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 This diagram illustrates the steps of a non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning in an exemplary embodiment of this application. Figure 2 This diagram illustrates a non-line-of-sight human pose recognition process based on pulse multipath spatiotemporal feature learning in an exemplary embodiment of this application. Figure 3 This document illustrates a flowchart of pulse echo preprocessing and tensor quantization in an exemplary embodiment of this application. Figure 4 This diagram illustrates a multi-scale convolutional channel attention fusion network in an exemplary embodiment of this application. Figure 5 This diagram illustrates a bidirectional temporal coding and multi-head attention modeling architecture in an exemplary embodiment of this application. Figure 6 This diagram illustrates the arbitration process for classification decision-making and confidence fusion in an exemplary embodiment of this application. Figure 7 This illustrates a confusion matrix diagram on a non-line-of-sight test set in an exemplary experiment of this application; Figure 8 This paper presents a comparison chart of the recognition accuracy of different methods in non-line-of-sight scenarios in exemplary experiments of this application; Figure 9 The t-SNE visualizations of the feature distributions of different methods on the non-line-of-sight test set are shown in the exemplary experiments of this application. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0025] This example implementation first provides a non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning. This method can be applied to a terminal device, such as a mobile terminal like a mobile phone, desktop computer, personal digital assistant, laptop, tablet, or smartwatch. (Reference) Figure 1 , Figure 2 As shown, the method may include the following steps: Step S101: Acquire pulse echo signals in non-line-of-sight scenarios, perform clutter suppression, time-domain truncation and alignment, and multi-domain feature extraction operations on the pulse echo signals, and stitch the distance-time spectrum and micro-Doppler time spectrum along the channel dimension to generate a three-dimensional pulse echo tensor.

[0026] Step S102: Construct a multi-scale convolutional channel attention fusion network, and extract multipath spatiotemporal feature maps from the three-dimensional pulse echo tensor through parallel multi-scale convolutional kernels and channel attention mechanisms.

[0027] Step S103: Input the multipath spatiotemporal feature map into a temporal encoder composed of a bidirectional long short-term memory network and a multi-head self-attention module connected in series, and extract the attitude temporal feature vector.

[0028] Step S104: Input the posture temporal feature vector into the classification decision head to generate the confidence scores of each human posture category, and output the human posture recognition result of fusion decision by combining the preset threshold and the temporal smoothing arbitration strategy.

[0029] This application proposes a non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning. On one hand, it achieves high-resolution extraction of spatiotemporal features related to human pose in non-line-of-sight multipath environments by suppressing clutter in pulse echoes, generating a three-dimensional pulse echo tensor through multi-domain feature concatenation, and extracting multipath spatiotemporal features using a multi-scale convolutional channel attention fusion network, thereby enhancing the ability to distinguish subtle features of static and micro-motion poses. On the other hand, it constructs a temporal encoder by connecting a bidirectional long short-term memory network with multi-head self-attention to mine dynamic temporal information of poses, and outputs recognition results by combining threshold and temporal smoothing arbitration strategies. It also corrects misjudgments of low-confidence frames by relying on the temporal correlation of continuous frames, so that the continuous recognition process can obtain smoother and more robust output results while maintaining response sensitivity, significantly improving the classification accuracy and generalization ability of human pose recognition in non-line-of-sight scenarios.

[0030] Below, as Figures 1-9 As shown, a non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning proposed in this example embodiment will be described in more detail.

[0031] Step S101: Acquire pulse echo signals in non-line-of-sight scenarios, perform clutter suppression, time-domain truncation and alignment, and multi-domain feature extraction operations on the pulse echo signals, and stitch the distance-time spectrum and micro-Doppler time spectrum along the channel dimension to generate a three-dimensional pulse echo tensor.

[0032] It is necessary to understand that, such as Figure 3 As shown, pulse radar echo data is preprocessed and a multi-domain feature tensor is constructed. First, multi-channel echo signals in non-line-of-sight scenarios are continuously acquired using a dual-receiver pulse radar system, and sliding background cancellation and bandpass filtering are performed to remove static wall clutter and low-frequency drift components. Second, short-time Fourier transform and time-domain truncation windowing are performed on single-frame echoes to extract range-slow time maps and micro-Doppler time spectra. The two types of feature maps are then stitched together along the channel dimension to form a three-dimensional pulse echo tensor containing range, time, and Doppler domains.

[0033] Specifically, step S101 includes sub-steps S1001-S1003.

[0034] Step S1001: A single-transmitter dual-receiver pulse radar is used to simultaneously acquire multi-channel echo signals. In a target-free scenario, static background echoes are pre-acquired and a mean template is calculated. Each frame of echo acquired in real time is subtracted from the mean template point by point. Then, a bandpass filter is used to filter out baseline drift and high-frequency noise to obtain a clutter-suppressed pulse echo sequence.

[0035] In this embodiment, a single-transmitter dual-receiver pulse radar system is used. The transmitting antenna is located at the center of the front of the wall, and two receiving antennas are symmetrically arranged on both sides of the transmitting antenna with a distance of 30 cm. The system continuously transmits nanosecond-level Gaussian single-cycle pulses at a pulse repetition frequency of 300 MHz. The receiving antennas synchronously collect the reflected echo signal after being modulated by the human target. In a targetless scene, 100 frames of static background echoes are pre-collected and their mean template is calculated. Then, in real-time acquisition, each frame of echo is subtracted from the mean template point by point to suppress wall reflections and fixed clutter. At the same time, fourth-order Butterworth bandpass filters with cutoff frequencies of 0.1 Hz and 5 kHz are used to filter out ultra-low frequency baseline drift caused by breathing and high frequency circuit noise, resulting in a pulse echo sequence after clutter suppression.

[0036] Step S1002: Determine the effective echo time window range based on the shortest flight time between the transmitted pulse and the receiving antenna and the maximum detection distance of the target. Perform time-domain truncation on each frame of the pulse echo sequence. Align the truncated echo sequence to a unified time reference according to the trigger time of the transmitted pulse and perform digitization and peak normalization to obtain the range-time matrix.

[0037] In this embodiment of the disclosure, the effective echo time window range is determined to be 20 to 80 nanoseconds based on the shortest flight time between the transmitted pulse and the receiving antenna and the maximum detection range of the target. Each frame of the output echo is truncated in the time domain to remove the direct pulse wave and long-distance irrelevant reflections. The truncated echo sequence is aligned to a unified time reference according to the trigger time of the transmitted pulse, and the analog echo signal is digitized with a sampling interval of 0.1 nanoseconds, so that each frame of echo is converted into a discrete sequence containing 600 range units. At the same time, the peak value of the sequence is normalized to eliminate the amplitude difference caused by the fluctuation of the target distance.

[0038] Step S1003: Perform a short-time Fourier transform on the distance-time matrix frame by frame along the fast time dimension to obtain the micro-Doppler time spectrum map corresponding to each frame. Stack multiple consecutive distance-time spectra along the time dimension to form a first three-dimensional tensor. Stack the corresponding number of micro-Doppler time spectrum maps along the time dimension to form a second three-dimensional tensor. Concatenate the first three-dimensional tensor and the second three-dimensional tensor along the channel dimension to obtain the standardized three-dimensional pulse echo tensor.

[0039] In this embodiment, the output range-time matrix is ​​subjected to a Hanning window-truncated short-time Fourier transform frame by frame along the fast time dimension. The Fourier transform window length is set to 64 sampling points and the overlap rate is 50%. The corresponding micro-Doppler time-spectrum map for each frame is calculated, with a frequency resolution of 15.625 MHz. Taking 32 consecutive frames as a processing batch, the 32 range-time maps are stacked along the time dimension to form a three-dimensional tensor. At the same time, the 32 corresponding micro-Doppler time-spectrum maps are stacked along the time dimension to form another three-dimensional tensor. Finally, the above two three-dimensional tensors are spliced ​​along the channel dimension to obtain a normalized three-dimensional pulse-echo tensor with a size of 32×600×2, where 32 is the number of time frames, 600 is the number of range units, and 2 is the number of feature channels.

[0040] Step S102: Construct a multi-scale convolutional channel attention fusion network, and extract multipath spatiotemporal feature maps from the three-dimensional pulse echo tensor through parallel multi-scale convolutional kernels and channel attention mechanisms.

[0041] It is necessary to understand that, such as Figure 4 As shown, a multipath spatiotemporal feature map is extracted from the pulse-echo tensor using a multi-scale convolutional and channel attention fusion network. The three-dimensional pulse-echo tensor is input into the multi-scale convolutional module, where three parallel convolutional branches with kernel sizes of 3×3, 5×5, and 7×7 are used to extract temporal-range features at each scale. The outputs of each branch are batch normalized and ReLU activated, then concatenated along the channel dimension to form multi-scale features. Next, the multi-scale features are input into the channel attention module, where global average pooling, two fully connected layers for dimensionality reduction-up mapping, and Sigmoid activation are sequentially performed to generate normalized weight vectors for each channel. These weight vectors are then used to recalibrate the input features channel by channel, enhancing the feature channels rich in multipath scattering discrimination information and suppressing noise-dominated channels, outputting a multipath spatiotemporal feature map.

[0042] Specifically, step S102 includes sub-steps S2001-S2002.

[0043] Step S2001: Expand the three-dimensional pulse echo tensor into a single-channel form along the channel dimension, and feed it into three parallel convolution branches with convolution kernels of different sizes for feature extraction. After batch normalization and ReLU activation, the outputs of each branch are spliced ​​along the channel dimension to obtain a multi-scale fused feature map. In this embodiment of the disclosure, a multi-domain feature map (three-dimensional pulse-echo tensor) is used as input, and its size is... Expand the tensor along the channel dimension as follows: After the single-channel output, the data is fed into three parallel convolutional branches: the first branch uses 64 two-dimensional convolutional layers with a kernel size of 3×3 and padding set to 1 to ensure the output size remains constant; the second branch uses 64 two-dimensional convolutional layers with a kernel size of 5×5 and padding set to 2; and the third branch uses 64 two-dimensional convolutional layers with a kernel size of 7×7 and padding set to 3. The stride of all three branches is set to 1. After each branch's convolution operation, batch normalization and ReLU activation function mapping are performed sequentially to obtain 64-dimensional multi-scale feature maps. Next, the outputs of the three branches are concatenated along the channel dimension to obtain a three-scale fused feature map containing 192 channels. This multi-scale parallel design enables the network to simultaneously capture the fine temporal structure (small convolutional kernel), the medium-range multipath scattering cluster patterns (medium convolutional kernel), and the broad distribution patterns of wall reflections and target scattering in pulse echoes (large convolutional kernel).

[0044] Step S2002: Input the multi-scale fused feature map into the channel attention module, and sequentially perform global average pooling, two-layer fully connected mapping and Sigmoid activation to generate normalized channel attention weight vectors for each channel. Use the channel attention weight vectors to perform channel-by-channel multiplication recalibration on the multi-scale fused feature map. Add the recalibrated feature map and the multi-scale fused feature map element-by-element through skip residual connections to obtain the multipath spatiotemporal feature map.

[0045] In this embodiment, the three-scale fused feature map is input into a global average pooling layer. Each feature channel is compressed into a single scalar along the spatial dimensions (temporal frame dimension and distance unit dimension) to obtain a channel description vector of length 192. Each element of this vector encodes the global response intensity of the corresponding feature channel. This channel description vector is then fed into a bottleneck structure consisting of two fully connected layers: the first fully connected layer reduces the 192-dimensional feature to 12 dimensions with a reduction ratio of 16 and applies ReLU activation; the second fully connected layer restores the 12-dimensional feature to 192 dimensions and applies Sigmoid activation, generating normalized attention weights for each channel. The vector, where each element takes a value between 0 and 1, represents the channel attention weight of the corresponding feature channel and its importance to the current pose recognition task. The normalized channel attention weight vector is multiplied channel by channel by channel by the three-scale fused feature map output by S2-1, that is, the feature map of each channel is multiplied by its corresponding attention weight value, thereby enhancing the effective feature channels and suppressing the redundant feature channels. A skip residual connection is added between the feature map after channel attention recalibration and the original three-scale fused feature map. The two are added element by element to obtain the multipath spatiotemporal feature map, which ensures the effective propagation of gradients in the deep network.

[0046] Step S103: Input the multipath spatiotemporal feature map into a temporal encoder composed of a bidirectional long short-term memory network and a multi-head self-attention module connected in series, and extract the attitude temporal feature vector.

[0047] It is necessary to understand that, such as Figure 5 As shown, a dynamic model of pose temporal evolution is established through a cascaded bidirectional long short-term memory network and a multi-head self-attention module. First, the multipath spatiotemporal feature map is decomposed into frame-level feature sequences along the time dimension and input into the bidirectional long short-term memory network for forward and backward bidirectional encoding. The concatenation of the forward and backward hidden states output at each time step serves as the context enhancement feature for that frame. Second, the complete output sequence of the bidirectional long short-term memory network is input into the multi-head self-attention module. Global dependency weights between time steps are calculated using scaled dot product attention, and the original multipath spatiotemporal feature map is added to the attention output through skip residual connections to ensure the flow of gradients in forward and backward propagation. Finally, after layer normalization, the pose temporal feature vector is obtained.

[0048] Specifically, step S103 includes sub-steps S3001-S3002.

[0049] Step S3001: The multipath spatiotemporal feature map is split into frame-level feature sequences along the time frame dimension. The frame-level feature sequences are input into a bidirectional long short-term memory network for forward and backward bidirectional encoding. The forward hidden state and backward hidden state of each time step are concatenated along the feature dimension to obtain a context-enhanced feature sequence.

[0050] In this embodiment of the disclosure, firstly, the multipath spatiotemporal feature map is split along the time frame dimension into a frame-level feature sequence of 32 time steps. Each frame is a feature vector of length 192 (corresponding to the number of channels), forming a feature vector of size [missing information]. Temporal feature sequences X is input into a bidirectional Long Short-Term Memory (LSTM) network, which consists of a forward LSTM and a backward LSTM, with both LSTMs having a hidden layer dimension of 128. The forward LSTM operates sequentially from... to Forward traversal At each time step Based on the current frame features Compared to the forward hidden state of the previous time step and unit state Update to obtain forward hidden state ; Backward LSTM from time step to Reverse traversal At each time step Based on the current frame features Backward hidden state at the next time step and unit state Update to obtain backward hidden state The forward hidden state at each time step. Backward hidden state By splicing along the feature dimension, we obtain a size of Context-enhanced feature sequences Through a bidirectional encoding mechanism, the network can simultaneously utilize the temporal context information before the pose change and the subsequent evolutionary trend after the pose change, thereby comprehensively characterizing the dynamic stage of each time step in the complete pose sequence.

[0051] Step S3002: Input the context enhancement feature sequence into the multi-head self-attention module, calculate the global dependency weights between each time step through scaled dot product attention, sum the multi-head self-attention output and the context enhancement feature sequence through skip residual connection and perform layer normalization processing, and take the feature vector of the last time step in the layer normalization output as the pose temporal feature vector.

[0052] In this embodiment of the disclosure, firstly, the context-enhanced feature sequence is... The input is a multi-head self-attention module. This module contains four parallel attention heads, each with a model dimension of 64. Within each attention head, the input feature H is processed by three linear projection matrices. , and Mapped to the query matrix Q, key matrix K, and value matrix V, all three having the same size. Secondly, the scaling dot product attention calculation is performed. First, the dot product between the transposes of Q and K is calculated to obtain a size of... The attention score matrix is ​​obtained by dividing it by a scaling factor of 8 (the square root of 64) to prevent the softmax input gradient from being too small. Then, softmax normalization is applied along the key dimensions to obtain the attention weight matrix. Finally, the attention weight matrix is ​​multiplied by V to obtain a matrix of size V. The weighted output. The outputs of the four attention heads are concatenated along the feature dimension. The matrix is ​​then restored to its original state through a linear fusion projection. The multi-head self-attention output and the context-enhanced feature sequence H output by S3-1 are subjected to skipped residual connections and summation. The result is then processed by layer normalization to stabilize the training process. The last time step in the layer normalization output ( The feature vectors of the first time step are extracted and used as the attitude temporal feature vectors that compress all time step information, with a length of 256. This vector retains the encoding results of the bidirectional long short-term memory network for the local dynamics of each frame, and also incorporates the reweighting of global inter-frame dependencies by multi-head self-attention, thus having the ability to compactly express the complete attitude sequence.

[0053] Step S104: Input the posture temporal feature vector into the classification decision head to generate the confidence scores of each human posture category, and output the human posture recognition result of fusion decision by combining the preset threshold and the temporal smoothing arbitration strategy.

[0054] Specifically, such as Figure 6 As shown, pose classification is performed based on the pose temporal feature vector, and a fusion decision result is generated. The pose temporal feature vector is fed into a classification decision head containing two hidden layers. The first hidden layer has a dimension of 128, is activated by ReLU, and is randomly deactivated with a Dropout ratio of 0.3. The second hidden layer maps the features to a preset number of pose categories and performs Softmax normalization to obtain the confidence score vector of each candidate category. A confidence threshold is set. If the highest confidence in the current frame exceeds the threshold, the corresponding category label is directly output. If the highest confidence is lower than the threshold but the difference between the top two confidences is lower than the difference tolerance, the temporal smoothing arbitration mechanism is activated. The mode of the decision results of the previous three consecutive frames is used as the output category of the current frame, and finally the fusion decision result is formed.

[0055] In this embodiment, to achieve a reliable mapping from pose temporal features to the final pose category, and to ensure the consistency and stability of the output results for single frames and consecutive frames, a 256-dimensional pose temporal feature vector is input into the classification decision head. This classification decision head consists of two fully connected hidden layers and a Softmax output layer. The first hidden layer maps the 256-dimensional features to 128 dimensions and applies a ReLU activation function, while performing Dropout regularization with a dropout probability of 0.3 to prevent overfitting. The second hidden layer maps the 128-dimensional features to a preset total number of pose categories. The output is then fed into the Softmax normalization function, where N is set to 6, corresponding to the posture categories of standing, sitting, squatting, lying down, bending over, and walking slowly. The Softmax output layer will then be the second hidden layer. Transform the dimensional real numbers into a class probability distribution that sums to 1, where the dimensional real number is denoted by _i_. The component indicates that the current frame belongs to the first [element]. The confidence score for a pose is formed by all N components, which constitute the class confidence score vector. During single-frame inference, the class index corresponding to the largest component in the class confidence score vector is taken as the candidate decision result for the current frame, and the corresponding highest confidence score is denoted as . By combining multi-layer nonlinear mapping with probability normalization, the network can compress high-dimensional pose temporal features into a semantically clear category space and quantify the credibility of each candidate category.

[0056] Set a high confidence threshold The tolerance for difference at confidence level is 0.8. It is 0.15. For the current frame, if its highest confidence level is... Greater than or equal to If so, the candidate decision for that frame is directly output as the fusion decision result, without triggering the arbitration mechanism. Secondly, if the highest confidence level of the current frame... Below And the difference between the highest confidence level and the second highest confidence level is greater than If the model's confidence level for that frame is insufficient, the current frame is marked as a frame to be arbitrated. For frames to be arbitrated, a temporal smoothing arbitration strategy is initiated: the three consecutive frames preceding the current frame (i.e., the first, second, and third frames) are statistically analyzed. , , The fusion decision result of the frame is used. If there is a result that has been covered by arbitration in the previous frame, the arbitration result shall prevail. The frequency of each category in the above three frames shall be calculated, and the category with the highest frequency shall be used as the arbitration output category of the current frame. Furthermore, if the mode of the previous three frames is tied, the process is further backtracked to the first frame. The frame is then used as the arbitration output for the current frame, with the arbitration result of that frame serving as the arbitration output. Through the combined strategy of confidence threshold filtering and temporal smooth arbitration, single-frame misjudgments can be effectively suppressed at low-confidence frames and attitude switching transition frames, enabling the entire continuous recognition process to achieve smoother and more robust output results while maintaining response sensitivity.

[0057] To verify the effectiveness of the non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning proposed in this application, the following experimental verification analysis was conducted.

[0058] This experiment used an IR-UWB pulse radar system as the data acquisition platform, with a transmitted pulse center frequency of 4.3 GHz, a bandwidth of 1.5 GHz, and a sampling rate of 10 GHz. The experimental environment was a real indoor through-wall scenario with a 24 cm thick solid brick wall. The experimental environment was based on the Ubuntu 20.04 operating system, using the PyTorch 2.0 framework and CUDA 11.8 acceleration, and was trained and tested on an NVIDIA GeForce RTX 3090 GPU. The dataset contained 6 pose classes (standing, sitting, squatting, lying down, bending over, and walking), with 800 sample sequences (32 frames per sequence) collected for each pose class, divided into training, validation, and test sets in a 7:2:1 ratio. Training used the Adam optimizer, with an initial learning rate set to... The batch size was set to 32, and the training ran for 120 epochs. The learning rate was reduced to one-tenth of its current value in the 60th and 90th epochs, respectively. The loss function was a weighted combination of cross-entropy loss and center loss, with weight coefficients of 1.0 and 0.1, respectively.

[0059] like Figure 7As shown, the method in this application achieved high classification accuracy in six non-line-of-sight human posture recognition tasks. Specifically, the recognition accuracy for standing and sitting postures reached 96.7% and 95.3%, respectively, indicating that the bidirectional temporal coding network can effectively capture the subtle differences in multipath scattering patterns between upright and sitting postures. A small number of misclassifications occurred between squatting and bending postures, mainly because the similar torso tilt angles of the two postures led to partial overlap of the multipath time delay structures. The recognition accuracy for lying posture reached 98.1%, the highest among the six categories. This is attributed to the orthogonal geometric relationship between the human body and the radar line of sight when lying down, which produces a multipath feature pattern significantly different from other postures. The recognition accuracy for the walking category was 93.5%, lower than that for static postures, indicating that the micro-Doppler broadening caused by movement under non-line-of-sight conditions introduces some interference to posture feature extraction.

[0060] like Figure 8 As shown, the method in this application achieves an overall recognition accuracy of 95.2% in non-line-of-sight scenarios, significantly outperforming the comparative methods. Among them, the method based on artificial features and support vector machines achieves an average accuracy of 72.1%, but due to its limited feature representation capabilities, the blurred boundaries of micro-Doppler features in different poses under wall attenuation conditions lead to numerous misclassifications. The end-to-end method based on 2D-CNN achieves an accuracy of 81.4%, a significant improvement over traditional methods, but its fixed receptive field struggles to adapt to scale variations in multipath scattering. The Transformer-based method achieves an accuracy of 87.6%, benefiting from the global modeling capabilities of the self-attention mechanism, but its feature resolution in non-line-of-sight scenarios remains insufficient due to the lack of prior embedding with temporal structure of radar echoes. The method in this invention achieves optimal recognition performance by using a multi-scale convolutional channel attention fusion network for fine extraction of multipath structures and bidirectional temporal coding for complete modeling of pose dynamics, demonstrating significant advantages in non-line-of-sight human pose recognition tasks.

[0061] like Figure 9As shown, the deep features extracted by the method in this application exhibit clear intra-class clustering and inter-class separation characteristics in the t-SNE embedding space. In contrast, the feature distribution of the 2D-CNN method shows more overlapping regions between different categories, especially for the crouching and bending classes, indicating that its fixed-scale convolution is also unable to extract fine-grained class discrimination clues in multipath scattering. The feature distribution of the Transformer method is better than that of 2D-CNN, but the boundary between the standing and sitting classes is still somewhat ambiguous. This is because the pure attention mechanism lacks the ability to explicitly capture the local structure of near-range multipath clusters. The method of this invention combines multi-scale convolution for fine extraction of local multipath structures with channel attention for adaptive enhancement of discriminative channels. At the same time, a cascaded architecture of bidirectional LSTM and multi-head attention is introduced in the temporal modeling stage, so that the final deep features not only retain the fine-grained information of the multipath spatial structure, but also incorporate the dynamic context of the temporal dimension, thereby obtaining a more compact and separable feature distribution structure in the embedding space.

[0062] Table 1 Comparison of attitude recognition accuracy under different through-wall materials As shown in Table 1, the method of this application maintained a high and stable recognition accuracy under three typical wall material conditions. As the wall attenuation decreased, the overall accuracy increased from 95.2% under a 24cm solid brick wall to 98.4% under a 2cm wooden partition wall, a trend consistent with the physical law of signal penetration attenuation. Under the 24cm solid brick wall condition, the recognition accuracy for each posture category remained above 92%, with the crouching posture showing the lowest accuracy at 93.8%, mainly because the thick wall significantly absorbed the high-frequency components of the pulse signal, leading to blurring of the micro-Doppler features. The lying posture showed the highest accuracy under all wall conditions, indicating that this posture has the strongest discriminative power in non-line-of-sight multipath modes. The data in Table 1 show that the method of this invention has good robustness to changes in wall material and can still maintain high recognition performance in scenarios with severely attenuated solid walls, basically meeting the practical application requirements for non-line-of-sight human perception in typical security and emergency scenarios.

[0063] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0064] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0065] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

Claims

1. A non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning, characterized in that, Includes the following steps: Acquire pulse echo signals in non-line-of-sight scenarios, perform clutter suppression, time-domain truncation and alignment, and multi-domain feature extraction operations on the pulse echo signals, and stitch the distance-time spectrum and micro-Doppler time spectrum along the channel dimension to generate a three-dimensional pulse echo tensor; A multi-scale convolutional channel attention fusion network is constructed, and multipath spatiotemporal feature maps are extracted from the three-dimensional pulse echo tensor through parallel multi-scale convolutional kernels and channel attention mechanisms. The multipath spatiotemporal feature map is input into a temporal encoder consisting of a bidirectional long short-term memory network and a multi-head self-attention module connected in series to extract the attitude temporal feature vector. The posture temporal feature vector is input into the classification decision head to generate the confidence scores of each human posture category. The human posture recognition result is then output by combining a preset threshold and a temporal smoothing arbitration strategy.

2. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 1, characterized in that, The steps of acquiring pulse echo signals in non-line-of-sight scenarios, performing clutter suppression, time-domain truncation and alignment, and multi-domain feature extraction on the pulse echo signals, and stitching the distance-time spectrum and micro-Doppler time spectrum along the channel dimension to generate a three-dimensional pulse echo tensor include: A single-transmitter dual-receiver pulse radar is used to simultaneously acquire multi-channel echo signals. In a target-free scenario, static background echoes are pre-acquired and mean templates are calculated. Each frame of echo acquired in real time is subtracted from the mean template point by point. Then, a bandpass filter is used to filter out baseline drift and high-frequency noise to obtain a clutter-suppressed pulse echo sequence. The effective echo time window range is determined based on the shortest flight time between the transmitted pulse and the receiving antenna and the maximum detection range of the target. Each frame of the pulse echo sequence is truncated in the time domain. The truncated echo sequence is aligned to a unified time reference according to the trigger time of the transmitted pulse and then digitized and peak normalized to obtain the range-time matrix. A short-time Fourier transform is performed frame by frame along the fast time dimension on the distance-time matrix to obtain the corresponding micro-Doppler time spectrum for each frame. Multiple consecutive distance-time spectra are stacked along the time dimension to form a first three-dimensional tensor. A corresponding number of micro-Doppler time spectrums are stacked along the time dimension to form a second three-dimensional tensor. The first three-dimensional tensor and the second three-dimensional tensor are spliced ​​along the channel dimension to obtain the standardized three-dimensional pulse echo tensor.

3. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 1, characterized in that, The multi-scale convolutional channel attention fusion network includes three parallel convolutional branches and a channel attention module; The kernel sizes of the three parallel convolutional branches are 3×3, 5×5, and 7×7, respectively. The convolutional stride of each branch is 1. The padding size matches the kernel size to ensure that the output feature size remains unchanged. The number of convolutional kernels in a single branch is 64. The channel attention module includes a global average pooling layer and a two-layer fully connected bottleneck structure. The reduction ratio is set to 16. The output of the fully connected layer is activated by Sigmoid to generate a normalized channel weight vector in the range of 0 to 1.

4. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 3, characterized in that, The step of constructing a multi-scale convolutional channel attention fusion network to extract multipath spatiotemporal feature maps from the three-dimensional pulse echo tensor through parallel multi-scale convolutional kernels and channel attention mechanisms includes: The three-dimensional pulse echo tensor is expanded into a single-channel form along the channel dimension and fed into three parallel convolution branches with different convolution kernel sizes for feature extraction. The outputs of each branch are batch normalized and ReLU activated and then spliced ​​along the channel dimension to obtain a multi-scale fused feature map. The multi-scale fused feature map is input into the channel attention module, and global average pooling, two fully connected mapping layers and sigmoid activation are performed sequentially to generate normalized channel attention weight vectors for each channel. The multi-scale fused feature map is recalibrated by channel-by-channel multiplication using the channel attention weight vectors. The recalibrated feature map and the multi-scale fused feature map are added element-by-element through a skip residual connection to obtain the multipath spatiotemporal feature map.

5. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 1, characterized in that, The step of inputting the multipath spatiotemporal feature map into a temporal encoder consisting of a bidirectional long short-term memory network and a multi-head self-attention module connected in series to extract the attitude temporal feature vector includes: The multipath spatiotemporal feature map is split into frame-level feature sequences along the time frame dimension. The frame-level feature sequences are input into a bidirectional long short-term memory network for forward and backward bidirectional encoding. The forward hidden state and backward hidden state of each time step are concatenated along the feature dimension to obtain the context-enhanced feature sequence. The context-enhanced feature sequence is input into the multi-head self-attention module. The global dependency weights between each time step are calculated by scaling dot product attention. The multi-head self-attention output and the context-enhanced feature sequence are summed through a skip residual connection and then processed by layer normalization. The feature vector of the last time step in the layer normalization output is taken as the pose temporal feature vector.

6. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 5, characterized in that, The bidirectional long short-term memory network includes a forward LSTM and a backward LSTM. The hidden layer dimension of both LSTMs is set to 128. The forward hidden state and the backward hidden state at each time step are concatenated to obtain the context enhancement feature sequence with a dimension of 256. The multi-head self-attention module contains four parallel attention heads, each with a model dimension of 64.

7. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 1, characterized in that, The steps of inputting the posture temporal feature vector into the classification decision head to generate confidence scores for each human posture category, and combining a preset threshold with a temporal smoothing arbitration strategy to output the fused decision human posture recognition result include: The posture temporal feature vector is input into a two-layer fully connected classification decision head, and after nonlinear mapping, Dropout regularization and Softmax normalization, confidence score vectors corresponding to each human posture category are generated. Set a high confidence threshold and a confidence difference tolerance threshold. Based on the threshold determination results, choose to directly output the human pose category or start the temporal smoothing arbitration mechanism to output the human pose recognition result of fusion decision.

8. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 7, characterized in that, The step of inputting the pose temporal feature vector into a two-layer fully connected classification decision head, and generating the confidence score vector corresponding to each human pose category through nonlinear mapping, Dropout regularization, and Softmax normalization includes: The pose temporal feature vector is input into the classification decision head, which includes two fully connected hidden layers and a Softmax output layer. The first fully connected hidden layer maps the features to 128 dimensions and applies ReLU activation and Dropout regularization. The second fully connected hidden layer maps the features to the number of pre-human pose categories and inputs them into the Softmax normalization function, outputting the category confidence score vector. The human pose category index corresponding to the largest component in the human pose category confidence score vector is taken as the candidate decision result, and the corresponding highest confidence score is recorded.

9. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 8, characterized in that, The steps of setting a high confidence threshold and a confidence difference tolerance threshold, and selecting to directly output the human pose category or activate the temporal smoothing arbitration mechanism based on the threshold determination result, and outputting the human pose recognition result of the fusion decision, include: Set a high confidence threshold and a tolerance threshold for confidence differences; If the highest confidence score of the current frame is greater than or equal to the high confidence threshold, the candidate decision result is directly output as the human pose recognition result of the fusion decision. If the highest confidence score of the current frame is lower than the high confidence threshold and the difference between the highest confidence score and the second highest confidence score is greater than the confidence difference tolerance threshold, then the current frame is marked as a frame to be arbitrated, the temporal smoothing arbitration strategy is started, and the arbitration output human pose category of the current frame is determined according to the mode vote of the fusion decision results of the three consecutive frames before the current frame, forming the human pose recognition result of the fusion decision.

10. The non-line-of-sight human pose recognition method based on pulse multipath spatiotemporal feature learning according to claim 9, characterized in that, In the majority voting, if the majority voting results of the fusion decision results of the previous three consecutive frames are tied, then the fusion decision result of the fourth frame before the current frame is used as the arbitration output human pose category for the current frame.

Images