Human pose estimation method based on multimodal attention network

Abstract

This invention relates to the fields of wireless sensing and deep learning, specifically to a human pose estimation method based on a multimodal attention network. In video modality learning, a multi-resolution network (FCN) is designed to extract pose features. By applying a local self-attention network to the generated heatmap, the weights and attention levels of different regions are adaptively adjusted to capture more refined keypoint information. In CSI modality learning, a spatiotemporal attention network and a multimodal-guided linear spatial position variation layer are designed to facilitate self-learning of its spatiotemporal features. A teacher-student architecture is adopted in the overall multimodal learning network to allow the Wi-Fi signal-based pose estimation model to learn more from the deep learning capabilities of the video modality. By transferring the correct human pose estimation information from the video learning network to the Wi-Fi signal learning network, accurate human pose information can be estimated using only the CSI modality as input.

Description

Technical Field

[0001] This invention relates to the fields of wireless sensing and deep learning, and particularly to key point location estimation in the field of deep learning, specifically to a human pose estimation method based on a multimodal attention network. Background Technology

[0002] With the rapid development of IoT, big data, and AI technologies, human-machine-thing integration has become an important requirement and a new development model for future intelligent computing. Perception is the foundation of human-machine-thing integration applications. The various behavioral perception and recognition technologies that have emerged in recent years have not only permeated the concept of AIoT into all aspects of daily life, but have also created application demands for the long-term development of this technology.

[0003] Currently, most human body recognition activities are still single-modal perception tasks. However, single-modal perception models are often limited by the sensing capabilities and physical characteristics of the devices themselves. For example, vision-based perception cannot protect privacy and is powerless in conditions of occlusion and darkness; perception based on commercial Wi-Fi is relatively coarse in accuracy and subject to many environmental and user constraints. Due to the obvious shortcomings of various sensing devices, joint learning and perception between multiple modalities is a good option. When existing human body recognition methods cannot meet the increasingly demanding application requirements, using Wi-Fi and video to jointly complete human body recognition tasks is a current research trend.

[0004] This invention focuses on the research of human pose estimation algorithms for Wi-Fi signals based on multimodal attention networks. Through analysis of the mechanisms of existing Wi-Fi signals and the characteristics of human pose estimation tasks, the network structure is redesigned from a multimodal learning perspective, thereby achieving higher pose estimation performance. Furthermore, considering that Wi-Fi wireless transmission is a complex process, with electromagnetic waves undergoing diffraction, scattering, and non-mirror reflection during propagation, which can easily obscure effective human pose features, this invention proposes a human pose estimation network structure based on a teacher-student architecture using CSI and video. This network extracts modally consistent pose features, further validating the effectiveness of the proposed design concept in multiple scenarios and providing varying degrees of reference value for related multimodal perception tasks.

[0005] This invention presents a human pose estimation algorithm based on a multimodal attention network, primarily addressing the problems of joint learning among multiple modalities and the localization and combination of key points in human pose. It possesses significant development potential and room for improvement in both data preprocessing and algorithm implementation.

[0006] Specifically, the present invention mainly solves the following problems:

[0007] Multimodal sensing information is highly heterogeneous, possessing not only significantly different dimensions and patterns but also potential temporal asynchrony, making multimodal fusion extremely challenging. With the rise of deep learning, the field of wireless sensing has made numerous attempts to implement deep learning methods. However, past multimodal sensing efforts often relied on established deep learning methods, neglecting the inherent features and characteristics of each modality, severely impacting the accuracy of the recognition results.

[0008] To leverage deep learning models, most existing multimodal perception work directly fits keypoint coordinates from the pose skeleton sequence of video signals to human pose features. This approach ignores the inherent characteristics of wireless sensing signals, lacks detailed spatiotemporal modeling based on the wireless signal space, and relies solely on extensive data training and repeated parameter adjustments to maintain recognition performance. Therefore, it fails to correctly explain the mechanism by which CSI performs pose perception tasks.

[0009] In summary, early work on multimodal human perception neglected the specific characteristics of each modality, relying solely on deep learning models and lacking sufficient understanding and utilization of Wi-Fi signal features, leading to the loss of many unique modal features. This problem will become increasingly prominent as perception granularity becomes more refined and perception tasks become more complex. Furthermore, due to the time-consuming acquisition of CSI data and the difficulty in extracting effective features, existing Wi-Fi signal-based perception algorithms still suffer from poor universality, high learning costs, and weak system generalization ability; this field remains to be explored. Summary of the Invention

[0010] This invention designs both a video-based pose feature extraction network and a CSI-based pose feature extraction network. In video modality learning, an FCN-based multi-resolution network is designed for pose feature extraction. By applying a local self-attention network to the generated heatmap, the weights and attention levels of different regions are adaptively adjusted to capture more refined keypoint information. In CSI modality learning, a spatiotemporal attention network and a multimodal-guided linear spatial position variation layer are designed to meet the self-learning requirements of its spatiotemporal features. In both the heatmap fusion stage and the final human pose estimation stage, the learning objective is to minimize the error between the predictions of the CSI network and the predictions from the video source. The overall multimodal learning network employs a teacher-student architecture to allow the Wi-Fi signal-based pose estimation model to learn more from the deep learning capabilities of the video modality. By transferring the correct human pose estimation information from the video learning network to the Wi-Fi signal learning network, accurate human pose information can ultimately be estimated using only the CSI modality as input.

[0011] (1) Video-based key point detection network

[0012] In this method, the video-based keypoint detection network primarily uses a Fully Convolutional Network (FCN) for image segmentation and encoding, and then employs a Local Attention Network to extract spatial feature information from the image to identify key points of the human skeleton in motion. The FCN network is a deep learning-based semantic segmentation method that can segment pixel-level input images into different object categories. Compared to traditional methods, the FCN network has achieved great success in image segmentation tasks because it can be trained end-to-end, does not require manual feature extraction, and has strong generalization capabilities.

[0013] Specifically, by using the FCN network to extract and encode features from the input video frames, the FCN network can be trained end-to-end throughout the entire video sequence, thus leveraging the temporal information of the video to better understand the evolution of actions.

[0014] The backbone of the video keypoint detection network uses the VGG network, which includes a multi-scale supervised structure. The network training is divided into three stages: FCN_32, FCN_16, and FCN_8, representing the training progress from... and The final resolution representation is generated. The network structure outputting the resolution can be represented at high, medium, and low resolutions by the pooling layers of the VGG network. FCN_32 only uses the low-resolution representation and then performs a 32x transposed convolutional upsampling operation, with only L... D1 A monitoring signal. The monitoring signal for FCN_16 is L. D1 This algorithm fuses FCN_32 and FCN_16 as another output and uses multi-scale supervision, i.e., L... D1 +L D2 +L F1 The supervisory signal for FCN_8 is L. D3 The outputs of FCN_32, FCN_16, and FCN_8 are fused to obtain the final output, and multi-scale supervision, i.e., L, is used. D1 +L D2 +L D3 +L F1 +L F2By supervising the fused output, the network learns more fully the complementary information required by features at different resolutions, leading to more accurate keypoint detection results. Maintaining high-resolution representation throughout the entire computation process, it begins with a set of high-resolution convolutions, then progressively adds low-resolution convolutional branches, connecting them in parallel. The final network consists of several stages, where the nth stage contains n convolutional branches with n different resolutions. Throughout the process, information is continuously exchanged between parallel computational combinations through multi-resolution fusion.

[0015] Because FCN networks can be trained and tested at different resolutions, they can adapt to video data of different sizes and resolutions, generating pixel-level heatmaps to describe the location and intensity of each keypoint. Local self-attention networks are used to weight local regions of each heatmap to extract keypoint features that are more representative.

[0016] (2) Video-side attention convolution module

[0017] Local Self-Attention Networks (LSNs) are neural network models based on attention mechanisms that can automatically learn spatial relationships and interactions between features in an image. By fusing heatmaps of different scales obtained from an FCN network, and then processing these heatmaps using an LSN, the accuracy of keypoint detection is further improved to obtain the final keypoint detection result. To enable the video signal to learn better spatiotemporal feature information, the feature extraction stage at the video end follows the design of HRNet, starting with a high-resolution convolutional stream as the first stage and progressively adding high- to low-resolution streams as new stages. The multi-resolution streams are connected in parallel. The main body consists of a series of stages. In each stage, the feature representation of each resolution stream is independently updated by multiple encoders and repeatedly exchanges cross-resolution information with the convolutional multi-scale fusion module. Figure 2 This explains the entire network architecture.

[0018] In local self-attention networks, for each input position, only the self-attention weights between it and its neighboring positions are calculated. This reduces computational cost while preserving local relationships between inputs. Assume the input sequence... Where T represents the sequence length and d represents the vector dimension at each position. To compute local self-attention, a window size w is introduced, indicating that each position only needs to consider its w neighboring positions. Specifically, for each position i, the attention weight a between it and its neighboring positions is calculated. i，j Then, its representation vector y is obtained by weighted summation. i ,Right now:

[0019]

[0020]

[0021] Where e i，j The correlation between position i and position j can be represented by dot product, addition, concatenation, etc. The main computational complexity of local self-attention networks comes from calculating the attention weights; therefore, introducing a window size can reduce the computational cost. Simultaneously, this method can preserve the local relationships of the input sequence, better handling the modeling of long sequences.

[0022] (3) Key point detection network based on CSI

[0023] The goal of this method is to establish a CSI keypoint estimation network that can explicitly capture global dependencies between different parts of the human body. It describes a method for capturing global interactions using a self-attention module and establishes a connection between video network keypoint estimation networks and the activation maximization principle of CSI skeleton extraction.

[0024] The CSI keypoint estimation network consists of three parts: a spatiotemporal attention network to extract time-frequency feature information from the CSI; a pose-based feedforward network to capture long-range spatial interactions between feature vectors; and a pose learning network for predicting keypoint heatmaps. The backbone network utilizes a CNN network with a Transformer-based encoder architecture. This is because transforming CSI sequence information into a pure heatmap prediction task is simply an encoding task; it compresses the received spectrogram and the phase of the CSI into a compact keypoint location representation.

[0025] In pose feature models, given an input spectrogram Suppose the CNN backbone outputs a two-dimensional spatial structure image feature. Its feature dimension is transformed into d through a 1×1 convolution. This is then combined with a given input phase information. The spectral feature map is flat as a sequence Where L = H × W passes through N attention layers and feedforward networks (FFNs). Then, assume that the CNN backbone network outputs a two-dimensional spatial structure image feature. Its feature dimension is transformed to d through a 1×1 convolution. The network's input is connected to the decoder, and the output... To predict K types of key point heatmaps Where H * W * =H I / 4, W I / 4. Then reshape E to The primary method uses a 1×1 convolution of order 1E to reduce the number of channels to order 2k. If H, W is not equal to H.* W * Then, additional bilinear interpolation or a 4×4 transposed convolution is used for upsampling before the 1×1 convolution. A 1×1 convolution is completely equivalent to a position-level linear transformation layer.

[0026] In extracting the spatiotemporal features of CSI, a spatiotemporal attention network based on the Transformer structure was mainly used. The core mechanism of the encoder is multi-head self-attention, which first takes an input sequence... Project to query key Sum Through three matrices Then, calculate the attention score matrix. for:

[0027]

[0028] Each queue (i.e., the feature vector at position i) calculates its similarity with all keys to obtain This determines the degree to which each queue depends on the preceding sequence. Then, the linear sum of all elements in the value matrix V is summed with w. i The corresponding weights are added to x. i In this way, an increment is obtained. In this way, the attention map can be regarded as a dynamic weight determined by specific image content, reweighting the information flow in the forward propagation.

[0029] The spatiotemporal attention module captures the total contribution of the prediction to each micro-Doppler map location. Each contribution from a different location can be reflected by the gradient and the attention weights given by the video attention network. Therefore, a specific analysis of x... j How does position j in the graph sequence affect the activation prediction keypoint heatmap at position i? Calculate... (Key point type of K) The derivative of wrt, further assuming As a function, WRT a given attention score, A i，j .get:

[0030]

[0031] in These are static weights (fixed during inference) and shared across all image locations. It can then be seen that the function G is related to A. i，j The relationship is approximately linear, meaning that the prediction h... i Its contribution level depends directly on its attention score at the image location.

[0032] In particular, the last attention layer acts as an aggregator, with WiViPose collecting contributions from all image locations based on attention points and forming the maximum activation in the predicted keypoint heatmap. The convolutional layers in the pose-based feedforward neural network (FFN) and pose learning network cannot be ignored; they represent locational information and transform the contributions from all locations approximately linearly through the same transformation without altering their relative proportions.

[0033] The goal of pose learning networks is to find the location of maximum activation, which represents the position of the keypoint. The interpretability of activation maximization lies in the fact that the input region that maximizes the activation of a particular neuron can explain what that activated neuron is searching for. In this task, WiViPose's learning objective is to find the location of the maximum activation in the heatmap. * Location Neuron activation is maximized, where i * The basic true location representing a key point:

[0034]

[0035] Assuming the model has already optimized the parameters θ * To predict the location of a specific keypoint (maximum location), the model's prediction can be interpreted as follows: for these locations J, element j has a higher attention score (≥δ), and for i, there are dependencies that significantly contribute to the prediction. These dependencies can be found in the following way:

[0036] J = {j|A i，j (θ * ,I)≥δ} (6)

[0037] in, This is the attention graph of the last attention layer, and it also contains a function wrtθ. * and I, that is, A=A(θ * Given an image I and a query location i, A i，： It can reveal the dependencies that a predicted location I is highly dependent on, and define its dependency region. A ：，jThis can reveal the most affected region at a location j, defining its affected region. Traditional CNN-based methods also use heatmap activations as key locations, but due to the depth and high non-linearity of deep CNNs, interpretable patterns in the predictions cannot be directly found. AM-based methods may provide insights, but they require additional optimization costs to learn the interpretable patterns that the convolutional kernels prefer to find. In contrast, AM is extended to heatmap-based localization via transformers, and no additional optimization costs are required because the optimization is implicitly done during training, i.e., A = A(θ). * The defined dependency region is the pattern being searched, which can reveal dependencies specific to images and keypoints.

[0038] (4) Loss Function

[0039] Since the CSI attention network implements regression learning, the L2 norm loss function commonly used in regression learning is applicable to the network. The loss function requires calculating the output S of the student network. t (·) and the ground truth value G t The loss between (·). Furthermore, G t (·) confidence information c i This demonstrates its correlation with the true pose, therefore it is necessary to take it into account when defining the loss function. Finally, the loss function is defined as follows:

[0040]

[0041] In the formula, This indicates the calculation of L2 loss; and Let represent the ground truth value and model prediction value for the i-th pose, respectively. This represents the confidence level of the i-th skeleton point.

[0042] After satisfying the confidence information, the output S of the student network in this invention t (·) represents the location of the key points of the human pose, which are composed of the skeleton node sequence generated by the network.

[0043] Beneficial effects

[0044] This invention proposes a human pose estimation algorithm based on a multimodal attention network, namely the WiViPose model. It aims to fuse and integrate information from different modalities to adaptively adjust the weights and attention levels of different regions in the skeletal heatmap, thereby achieving more accurate and interpretable pose estimation. It exhibits good scalability, robustness, and energy utilization optimization, and its beneficial effects are mainly reflected in the following three aspects:

[0045] (1) This invention can achieve centimeter-level human pose estimation using conventional Wi-Fi devices, and can automatically detect and generate key skeletal features during human movement to achieve good joint positioning. To solve the shortcomings of low recognition accuracy in previous pose estimation work, this invention extracts relevant action features of different modalities and performs attention learning, which can reduce the differences between modalities and reduce the influence of environmental irrelevant factors, greatly improving the learning performance of the model.

[0046] (2) This invention models the relationships between different modalities to adaptively adjust the weights and attention levels of different regions. Finally, it uses skeletal heatmaps to regress key points of the human body, achieving more accurate and interpretable pose estimation. Network parameters are controlled according to different practical needs, thereby improving the algorithm's adaptability and fit.

[0047] (3) Compared with wearable devices, the present invention, which is based on Wi-Fi for human posture estimation, does not require the subject to wear any device; compared with camera-based posture recognition, the present invention is not constrained by light or darkness and does not infringe on human privacy. Attached Figure Description

[0048] Figure 1 This is a diagram illustrating the overall architecture of the present invention;

[0049] Figure 2 This is a flowchart of the CSI pose feature extraction network of the present invention;

[0050] Figure 3 To generate a skeleton instance diagram;

[0051] Figure 4 The system performance graphs are shown in the LoS and NLoS scenarios.

[0052] Figure 5 A system performance diagram for pose estimation of different volunteers;

[0053] Figure 6 System performance diagrams for transceiver equipment under different environments;

[0054] Figure 7 This is a system performance diagram for the transceiver equipment at different distances. Detailed Implementation

[0055] The present invention will be further described below with reference to the accompanying drawings.

[0056] Example 1

[0057] This example uses data packet transmission based on a wireless network as an example to illustrate the implementation of the invention. The effects and functions of the invention are shown below with reference to the accompanying drawings. Figure 1 This is a diagram of the overall architecture of the present invention. Figure 2 This is a flowchart of the CSI pose feature extraction network of the present invention.

[0058] The specific steps are as follows:

[0059] The embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0060] This experiment aims to collect CSI data of Wi-Fi signals using the PicoScenes platform, while a camera simultaneously captures images of human motion, in order to study the human perception state in different environments caused by Wi-Fi signals and images. It is crucial to ensure that both projects start and end simultaneously, with strict time alignment. This step involves data alignment and model validity; therefore, maintaining spatiotemporal consistency is a fundamental prerequisite for conducting the experiment.

[0061] 1. Experimental Platform and Scenarios

[0062] The system in this paper uses two Lenovo E73S desktop computers equipped with AX210 network cards. One computer uses one external antenna as the transmitter, and the other computer uses two antennas as the receiver to collect CSI data. Both computers run Ubuntu 20.04 with kernel version 5.14. To collect CSI data, the PicoScense CSI Tool development tool is installed on both computers. Additionally, a Kinect V2 camera is used to collect camera data. The specific experimental steps are as follows:

[0063] First, a test environment needs to be prepared, such as an indoor room or an open outdoor area. Ensure that there are no interfering signals in the test environment and that the signal strength remains relatively stable. In the PicoScenes platform, configure the platform's acquisition parameters and use the development tools to configure the Wi-Fi network card to acquire CSI data. The specific steps are as follows:

[0064] 1. Log in to the PicoScenes platform and create a new data collection task.

[0065] 2. In the data acquisition task, set the following parameters: Wi-Fi network card model AX210, channel 165, bandwidth 20MHz, frequency 5GHz, sampling rate 900.

[0066] 3. On both sides of the transceiver device, find the PhyPath ID of the Wi-Fi NIC and switch to the same channel and channel mode.

[0067] 4. Configure the CSI data acquisition mode in Monitor mode in the data acquisition task. The transmitter injects data packets in 802.11n format, and the CSI data is continuously collected by different receivers at the specified locations at the same time.

[0068] Place the camera equipment, then position the Kinect V2 camera in the middle of the transmitting device's antenna, at a height of 1.6 meters. The tester stands 2.2 meters away from the Kinect V2. Position the Kinect V2 at a 60-degree angle vertically (officially recommended). Set the camera to collect 30 frames per second at a resolution of 640×480.

[0069] 2 Data Acquisition and Processing

[0070] During the experimental data collection process, users were allowed to perform various daily activities without specific guidance to evaluate the system's performance. These activities included muscle strengthening exercises, balance and stretching exercises, aerobic exercise, and dance. Daily activities included raising arms, waving, walking, jogging, and using a smartphone. Users were allowed free movement within the sensing area and were not always facing the Wi-Fi transceiver. During the experiment, both the Wi-Fi transceiver and the Kinect V2 camera captured the user's posture and motion information.

[0071] 3 Data Preprocessing

[0072] The acquisition frequency of CSI data is related to the packet reception frequency of the receiving end. Due to path attenuation of Wi-Fi signals during propagation and device instability, repeated packet capture and packet loss may occur during the experiment, which will also affect the acquisition frequency of CSI data. Therefore, CSI data preprocessing is required.

[0073] 3.1 Data Interpolation Processing

[0074] It is important to note that in previous experiments, although the transmitter was programmed to send 900 data packets per second, it could not be guaranteed that the receiver would receive a data packet at the same frequency. Sampling jitter was found to be common, sometimes exceeding 300 milliseconds. This jitter was caused by data packet loss, possibly due to interference from surrounding electromagnetic waves or accidental deviations during antenna collection. For cases of dropped measurement results, interpolation is required for more accurate signal evaluation. Using the obtained overall sequence CFR samples, a 50-millisecond interval was set between consecutive values ​​at uniformly spaced time points. Discontinuous timestamps were considered as lost measurement results. A first-order interpolation method was used to overlap and extend the CFR samples, calculating the missing value in the middle from the values ​​at both ends of the lost measurement result. Assuming the measured value of the x0th measurement result is y0, the xth... n The measured value of each measurement result is yn Then it is necessary to perform (x) n Interpolation processing of (x0-1) measurement results. Given coordinates (x0, y0) and (x... n y n To obtain [x0, x n The value of a certain position on the straight line within the interval:

[0075]

[0076] Since the value is known, the value can be obtained from the formula:

[0077]

[0078] By performing first-order linear interpolation estimation, the continuity of signal timestamps and the integrity of sampling are guaranteed, thus providing an effective basis for the alignment of different signals.

[0079] 4 Feature Extraction

[0080] The collected CSI data reflects two main characteristics of motion changes: time-domain features and frequency-domain features. Time-domain features include phase, while frequency-domain features refer to Doppler shift. Human motion affects both the time and frequency domains of Wi-Fi signals. Evaluating random human motion requires extracting subtle changes from complex movements, a major challenge in feature extraction. Time-domain features contain more usable temporal information, ensuring the continuity of human motion, while frequency-domain features include the intensity and drasticness of motion changes. Considering the advantages and disadvantages of both, and in the context of motion quality assessment experiments, a combination of time-domain and frequency-domain features can be used to extract these features for human posture perception tasks.

[0081] 4.1 Phase Characteristics - Phase

[0082] The causes of CSI phase errors are generally divided into two categories: carrier frequency offset (CFO) and sampling frequency offset (SFO). CFO is caused by the missynchronization of the channel center frequencies at the transmitter and receiver, and can be automatically compensated and calibrated by the receiver's CFO calibrator. However, in previous work, due to incomplete network card hardware, there was a clock asynchronization between the sampling frequencies of the transmitter and receiver. In the PicoScenes platform used, due to its superior network card equipment, the incomplete network card hardware resulted in more sufficient CFO compensation; SFO can also be automatically compensated by the receiver's SFO corrector, but the excess SFO leading to phase rotation error is significantly reduced. Therefore, the time delay caused by SFO and the phase offset caused by CFO are less, so no additional phase correction is needed here. The subcarrier set can be directly adjusted.

[0083] H = [H1, H2, H3, ..., H N ] T (10)

[0084] Where i ∈ [1, N], and N is the number of subcarriers. Each H i This represents the amplitude and phase of a subcarrier. H appears in the complex form a+bi, and its modulus can be obtained. and argument That is, the corresponding amplitude and phase.

[0085] 4.2 Frequency Domain Characteristics - Micro-Doppler

[0086] In this method, the input Doppler signal is first converted into a time-frequency representation using short-time Fourier transform. Then, a micro-Doppler descriptor is established to extract features from local time-frequency patches centered on the torso frequency. After feature extraction, matrix-based principal component analysis is used to reduce the size of the feature map.

[0087] 4.2.1 Time-Frequency Segment

[0088] When the Doppler signal is reflected back from a moving person, the returned Wi-Fi signal includes multiple time-varying frequencies caused by different moving parts of the body. To capture these time-varying Doppler frequencies, the Wi-Fi signal is converted into a time-frequency representation. The Short-Time Fourier Transform (STFT) is commonly used to convert a discrete-time Doppler signal x(n) into its time-frequency representation. Mathematically, the STFT is derived from...

[0089]

[0090] In the formula, n is the discrete time, k is the discrete frequency, N is the number of frequency points, and w(m) is the window function. The size of the window function is determined empirically to achieve better classification. The spectrum is obtained by taking the size of the STFT, i.e., S(n,k)=|X(n,k)|. Figure 3 The spectrum of a person walking with swinging arms is shown. The time-varying Doppler frequency around 200 Hz is caused by the oscillating torso and is considered the spine of the spectrum. The main flanks represent the micro-Doppler of leg swinging, and one flank represents the micro-Doppler of arm swinging. The spectrum is shown on a logarithmic scale with a threshold of -20 dB.

[0091] Because the torso has the largest surface area of ​​the human body, it is the area with the most torso position information sensed by CSI during WiFi signal propagation. The region with the highest spectral energy is determined by finding the maximum value in each column of the spectrum, which typically represents the torso. The frequency distribution of reflections from other parts of the body can be estimated based on the phase difference between the main and lateral wings, which usually represents the signal reflection and diffraction phenomena caused by leg movement or arm movement during walking. Most micro-Doppler signal features exhibited by moving body parts are distributed around the spine in the spectrum; the brightest part is called the main wing region, and the other areas of reflected and diffracted signal energy are called the lateral wing regions. To avoid extracting other irrelevant human posture information and reduce the influence of environmental noise, it is not necessary to process the entire spectrum. Therefore, to extract information only related to the human body...

[0092] Two-dimensional time-frequency segments centered on the spine of the spectrogram were extracted and used as input for the feature extraction stage. It is important to note that before estimating the flank height, each column of the spectrogram is convolved with a one-dimensional median filter, and the spectrogram is segmented using a threshold calculated from the spectrogram of the background scene. The specific calculated frequency distribution is as follows:

[0093]

[0094] Where M is the total number of time points. The frequency profile is F. p (k), normalized to [0,1]. The maximum frequency of the main flanking peak satisfies:

[0095]

[0096] Where θ is a given threshold, k t Here are the frequency coordinates of the spine. Subsequently, the height of the main lateral flank is calculated as h = Kk. t The frequency range representing the patch height is calculated as [y]. t -h,y t +h]. On the other hand, the width of the patch is chosen to capture at least one complete motion cycle; the optimal size will be determined experimentally. To compensate for variations in frequency and time, the extracted patch is adjusted to a predefined height N before feature extraction. y and width N x Since the torso pixels have higher intensity values ​​than the micro-Doppler components, the contrast is enhanced. This technique improves the contrast of the micro-Doppler patch. Let S(i,j), i∈[1,N]. y ], j∈[1, N x [] represents a microDoppler segment. For contrast enhancement, the Naka-Rushton function is used, by...

[0097]

[0098] In the formula, μ is the average intensity value of the patch:

[0099]

[0100] A time-frequency patch is displayed before and after the application of contrast enhancement technology.

[0101] 4.2.2 Feature Extraction

[0102] To extract features from time-frequency patches, a micro-Doppler descriptor based on the key descriptor principle was designed. The proposed micro-Doppler descriptor employs a log-Gabor filter to capture missing global-level information. Here, the Gabor filter proposed by Gield is used, which, compared to a regular Gabor filter, has no DC component and bandwidth limitations. Therefore, a small segment of the log-Gabor filter is sufficient to cover the desired spectrum. On a linear frequency scale, the Gabor filter has a Gaussian function shape, while the log-Gabor filter has a similar shape on a logarithmic frequency scale, with an extended tail in the high-frequency region. Due to the singularity of the origin-logarithmic Gabor function, a log-Gabor filter was designed in the frequency domain. In polar coordinates, the log-Gabor filter can be expressed as:

[0103]

[0104] Among them, f i θ is the center frequency of the filter. j It is the j-th direction angle of the filter, N f It is the number of scales, N θ It is the number of directions, R=N f N θ It is the total number of log-Gabor filters, σ f and σ θ These are the radial and angular bandwidths, respectively. The micro-Doppler descriptor uses a log-Gabor filter in the frequency domain. The first step in calculating the micro-Doppler characteristics is to pass the input CFR signal through a set of log-Gabor filters σ. f and σ θ {G1, ..., G k , ..., G R Perform convolution. Let I denote the Fourier transform of the contrast enhancement patch. The real part Z(k, 1) and the imaginary part Z(k, 0) of the output of the k-th log-Gabor filter are:

[0105]

[0106]

[0107] Among them, IFFT2 is the two-dimensional inverse fast Fourier transform. and These are real and imaginary operators, respectively. A half-wave rectification operation is performed, decomposing each filtered image into mapped and unmapped parts:

[0108]

[0109] This thresholding operation is also applied to the imaginary part of the filtered image, i.e., Z. k，0 , k∈[1,R]. Then the Naka-Rashton transform is applied to the map and outside the map respectively.

[0110]

[0111] in,

[0112] The final step of feature extraction is to arrange the extracted features into a matrix. Each contrast enhancement region is divided into d non-overlapping sub-regions in each direction (horizontal and vertical) along its outer edge, and the average value of each sub-region is used as a feature. The average value calculated from the real part of the k-th filtered image is stored in the vector μ. k , l = [μ1, ... μ L ] T L = d 2 In, its l-th element is

[0113]

[0114] After calculating the features of all filtered CFRs, the features are arranged into a matrix.

[0115] A = [μ 1，1，0 μ 1，0，0 μ 1，1，1 , ..., μ R，1，0 μ R，0，0 μ R，1，1 ] (twenty two)

[0116] After the feature extraction stage, it is fed into the WiViPose network for keypoint estimation.

[0117] 5. Experimental Demonstration

[0118] To better visualize the model's performance, 3D human skeletons built for different activities are displayed as the user performs various activities. Figure 3 Four examples of these constructed 3D skeletons are shown.

[0119] For each subgraph (e.g., Figure 3 (a) The first row displays time-series video frames recorded by an RGB camera for visual reference, and the second row displays the ground skeleton recorded by Kinect 2.0. The results generated by the CSI self-attention network are displayed in the third row. Figure 3 (a) and (b) show the different activities the user is performing while walking. Figure 3 It shows the user walking around randomly.

[0120] The results for LoS and NLoS scenarios are as follows: Figure 4 As shown, the average joint positioning error is 4.7 cm in the Loss (LoS) scenario, while it is slightly higher at 5.3 cm in the Non-LoS (NLoS) scenario. Wi-Fi signals can construct highly accurate moving human postures; however, in the NLoS scenario, mispredicted and distorted body parts are more pronounced. For example, Figure 3 In frame (d), the anatomy of the subject's left leg is inaccurate. Figure 3 In (d), some slight deformations in the skeletal posture are easily observed. However, these results also indicate that the system can accurately construct moving human postures using Wi-Fi signals.

[0121] Example 2

[0122] To investigate the impact of different environments and distances on the system performance of Wi-Fi devices, distance is defined as the distance between two adjacent receivers, which users can adjust according to room size. Both the transmitter and receiver use AX210 network cards when acquiring CSI data. Each data packet collected from the AX210 wireless network card yields an n x m complex CSI matrix, where n represents the number of received data packets and m represents the number of subcarriers. In the experiment, n was set to 900 and m to 57. Therefore, the CSI parsed from the data packets received by the receiver is a 900 × 57 complex matrix. If the transmitter uses one antenna and the receiver uses two antennas, the matrix dimension received per second is 1 × 2 × 900 × 57. Here, each collection session lasts 360 seconds.

[0123] The acquired CSI data was stored in the receiver and processed using MATLAB. One-eighth of the pose estimation samples were used for testing, and the remainder for training. Eight-fold cross-validation was employed to estimate the model to ensure accuracy. Experimental results are as follows: Figure 5 As shown, the average positioning error of the system for different volunteers is basically around 5cm, indicating that the system has stability.

[0124] To verify the system's environment independence, performance remained constant when furniture positions were changed, as discussed in the overall performance section. This section further investigates system performance in more challenging scenarios. Specifically, the system model is trained using data collected in one environment (e.g., a living room or dining room), and its performance is evaluated when running in different environments (e.g., a bedroom), with static environment deletion also performed during this evaluation. Figure 6 The displayed bedrooms, living rooms, and dining rooms were trained by the system in different environments. For example... Figure 6 As shown, the average joint localization errors for the bedroom, living room, and dining room were 5.2 cm, 5.7 cm, and 4.7 cm, respectively. Although no test environment was observed during the training phase, the joint localization error remained highly accurate. This is because the learning model relies on the micro-Doppler characteristics of signals reflected solely by the human body, which are independent of the background environment. Therefore, the system can be trained in one environment and then run in a new one.

[0125] The model was trained at the default distance to evaluate its performance at different distances, with the distance between the transceiver devices set to 2.5m, 3m, and 3.5m. CSI packets were collected within this distance, and after data preprocessing and feature extraction, they were input into the WiViPose network for performance testing. Figure 7 The corresponding average errors were 4.7 cm, 4.9 cm, and 5.1 cm, respectively. Observations revealed that the system performance improved as the distance decreased. This is because shorter propagation distances result in higher received signal strength. The results indicate that the system operates well across different distances between the transceiver and receiver.

Claims

1. A human pose estimation method based on a multi-modal attention network, characterized in that, A video-based pose feature extraction network and a CSI-based pose feature extraction network are designed. In the learning of the video modality, a multi-resolution network based on FCN is designed to extract pose features, and a local self-attention network is applied to the generated heat map to adaptively adjust the weights and attention of different regions to capture finer key point information. In the learning of the CSI modality, a spatio-temporal attention network and a spatial position linear variation layer guided by multiple modalities are designed to meet the self-learning of its spatio-temporal features. In the heat map fusion stage and the final human pose estimation stage, the learning goal is to minimize the error between the prediction of the CSI network and the prediction result of the video end. In the overall multi-modal learning network, a teacher-student architecture is adopted to transfer the correct information of human pose estimation from the learning network of the video to the learning network of the Wi-Fi signal, and finally only the CSI modality is input to accurately estimate the human pose information. The CSI key point estimation network consists of three parts: a spatio-temporal attention network to extract CSI time-frequency feature information; a pose-based feedforward network to capture long-range spatial interactions between feature vectors; and a pose learning network to predict key point heat maps; the backbone network uses a CNN network, and the network structure follows the encoder of the Transformer architecture, because converting CSI sequence information into a pure heat map prediction task is only an encoding task, which compresses the frequency spectrum and the phase of CSI into a compact key point position representation. 2.The human pose estimation method based on the multi-modal attention network of claim 1, wherein, Specifically, the steps include: (1) Video-based key point detection network The training of the network is divided into three stages FCN_32, FCN_16 and FCN_8, representing the generation of the final and resolution output from the representation of the resolution, the fusion of FCN_32 and FCN_16 as another output, and the use of multi-scale supervision, i.e. D1 +L D2 +L F1 ; the supervision signal of FCN_8 is L D3 , the fusion of the outputs of FCN_32, FCN_16 and FCN_8 as the final output, and the use of multi-scale supervision, i.e. D1 +L D2 +L D3 +L F1 +L F2 ; Maintain high-resolution representation throughout the entire process, which starts with a set of high-resolution convolutions, then progressively adds low-resolution convolution branches, and connects them in parallel; the final network consists of several stages, where the nth stage contains n convolution branches and has n different resolutions; throughout the process, information is constantly exchanged between parallel operation combinations through multi-resolution fusion; (2) Video attention convolution module In a local self-attention network, for each input position, only the self-attention weights between it and its neighboring positions are calculated; assuming the input sequence... Where T represents the sequence length and d represents the vector dimension at each position; to compute local self-attention, a window size w is introduced, indicating that each position only needs to consider w adjacent positions; specifically, for each position i, the attention weight a between it and its adjacent positions is calculated. i,j Then, its representation vector y is obtained by weighted summation. i ,Right now: where e i，j represents the correlation between position i and position j, and its calculation method includes dot product, addition, and splicing; (3) CSI-based key point detection network In the pose feature model, given an input spectral graph Suppose the CNN backbone outputs a two-dimensional spatial structure image feature The feature dimension is converted to d by 1x1 convolution; plus a given input phase information Spectral feature map is flattened as a sequence Where L=HxW goes through N attention layers and feed-forward networks FFNs; then, assuming the CNN backbone network outputs a two-dimensional spatial structure image feature The feature dimension is converted to d by 1x1 convolution; the input of the network is connected to the decoder, and the output To predict K types of keypoint heatmaps Where H * ,W * =H I / 4,W I / 4; then E is reshaped to 1st order E using 1x1 convolution to reduce channels to 2nd order k; if H,W * ,W * , then use additional bilinear interpolation or 4x4 transpose convolution for upsampling before 1x1 convolution; a 1x1 convolution is completely equivalent to a position-wise linear transformation layer; In the process of extracting the spatio-temporal features of CSI, a spatio-temporal attention network based on the Transformer structure is used; the core mechanism of the encoder is multi-head self-attention, which first projects an input sequence Project to query Key And value Through three matrices Then, the attention score matrix Is: Each queue That is, the feature vector of position i calculates the similarity with all keys to obtain This determines the degree of each queue dependence on the previous sequence; then, the linear sum of all elements in the value matrix V is added to x i with the corresponding weight in x i , to obtain an increment; in this way, the attention map can be regarded as a dynamic weight determined by the specific image content, reweighting the information flow in the forward propagation; The spatio-temporal attention module captures the total contribution of each micro-Doppler map position prediction, and each contribution from different positions can be reflected through gradients, as well as the attention weights given by the video attention network; Thus, by analyzing x j How the position j in the sequence of images influences the activation prediction heat map for position i, is computed The derivative of the keypoint type w.r.t K, further assuming As a function w.r.t. a given attention score, A i,j Obtained: wherein is a static weight that is fixed at inference time and shared across all image locations; it can then be seen that the function G is a function of A i，j approximately linear, i.e. the degree of contribution of a prediction h i to the final prediction depends directly on its attention score at the image location; In particular, the last attention layer acts as an aggregator, WiViPose collects the contributions of all image positions according to the attention points, and forms the maximum activation in the predicted key point heat map; the convolution layers in the pose-based feedforward network and the pose learning network cannot be ignored, they are position-based information representations, which approximately linearly transform the contributions of all positions through the same transformation without changing their relative proportions; The content in the pose learning network is to find the maximum position of activation, which represents the position of the key point; the interpretability of activation maximization is that the input region that can maximize the activation of a certain neuron can explain what the activated neuron is looking for; in this task, the learning goal of WiViPose is to expect that the i * position of the heat map neuron activation is maximally activated, where i * represents the basic true position of a key point: Assuming the model has optimized parameters θ * and predicting the position of a particular keypoint, and the maximum position, the model's prediction can be interpreted as these positions J, element j, have a higher attention score (≥ δ) for i, are dependent relationships that contribute significantly to the prediction; these dependencies can be found by: J = {j | A i,j (θ * ,I) ≥ δ} (6) where, is the attention map of the last attention layer, and there is a function w.r.t θ * and I, i.e., A = A(θ * ,I); given an image I and a query location i, A i，: can reveal a dependency relationship that a predicted location I highly depends on, defining its dependency region; A ：,j can reveal a region that location j is most affected by, defining its affected region; display the dependency relationship of images and key points; (4) Loss function Since the CSI attention network realizes regression learning, the L2 norm loss function commonly used in regression learning is suitable for the network; The loss function needs to compute the loss between the output S of the student network t (·) and the ground truth value G t (·). The loss function is defined as follows: wherein, represents the L2 loss calculation; and respectively represent the ground truth value and the model predicted value of the i-th pose; represents the confidence of the i-th skeleton point. After satisfying the confidence information, the output S of the student network t (·) is the position of the human body posture key point composed of the skeleton node sequence generated by the network.

Images