Human body posture estimation method based on multi-modal contrast learning

Abstract

The invention discloses a human body posture estimation method based on multi-modal contrast learning. The method comprises the following steps: firstly, carrying out noise reduction and dimension reduction preprocessing on input Wi-Fi modal data, and obtaining a spectrogram by adopting short-time Fourier transform; the method comprises the following steps of: preprocessing input LiDAR modal data, acquiring a pair of enhanced point cloud data by adopting data enhancement, extracting a human skeleton in the point cloud data by adopting a Point Net + + network, and enhancing the quality of the human skeleton by using intra-modal contrast learning; then carrying out joint training on the two pre-processed modalities, and for a spectrogram of the Wi-Fi modalities, extracting human motion posture features of the Wi-Fi modalities by adopting CNN and ResNet34; for a human skeleton sequence of a LiDAR mode, ST-GCN is adopted to extract human motion posture features of point clouds. Finally, cross-modal contrast learning is adopted, feature alignment between heterogeneous modals is achieved, and the aligned features are used for human body posture estimation.

Description

Technical Field

[0001] The present invention relates to the fields of wireless perception and deep learning, and in particular to a human posture estimation method based on multimodal comparative learning of Wi-Fi signals and LiDAR point cloud data. Background Art

[0002] In today's wave of digitalization and intelligent technology, the Artificial Intelligence of Things (AIoT), a cross-industry convergence technology, is deeply integrated into and driving the transformation and development of various industries with its superior intelligent analysis and real-time perception capabilities, becoming a key force driving global digital transformation. With continuous technological advancements, AIoT, combining the extensive connectivity of the Internet of Things with the intelligent analysis capabilities of AI, has achieved full coverage of the entire process from data collection to intelligent decision-making, demonstrating tremendous potential in the field of intelligent perception technology. For example, robot dogs equipped with a variety of perception sensors are already widely used in areas such as intelligent power inspection. Highly intelligent humanoid robots will soon be able to replace many types of human labor and bring significant convenience to people's lives. Human pose estimation, a key research area in human behavior recognition, focuses on accurately determining the spatial position and connections of various human body parts. This technology identifies and locates key parts such as the head, torso, and limbs, collects their coordinate data, and then constructs a skeletal model that reflects the dynamic changes of the human body, enabling accurate assessment of individual poses. This technology has a wide range of applications, encompassing virtual reality, autonomous driving, metaverse simulation, 3D gaming, and other fields.

[0003] Thanks to the rapid development of artificial intelligence and sensor technology, a large amount of research has emerged in the field of human pose estimation. Currently, there are three main approaches to human pose estimation: vision-based, wearable device-based, and wireless signal-based. Vision-based approaches primarily rely on video data captured by cameras. Image processing and computer vision techniques are used to extract human joints and contour information, which are then assembled into a human skeleton. While this approach can provide rich spatial information, it also has limitations, such as sensitivity to lighting conditions, susceptibility to occlusion, and potential privacy violations. Furthermore, the high computational resource requirements of vision-based approaches limit their application in specific scenarios. Wearable device-based approaches acquire data directly from the human body through sensors such as accelerometers, gyroscopes, and IMUs. This technology provides continuous and real-time pose information and offers the advantage of being robust to environmental conditions and unaffected by lighting and occlusion. However, wearable devices require users to wear them continuously, which can cause discomfort and pose significant limitations in smart home scenarios. For example, requiring users to wear the devices continuously during sleep is often impractical. Perception methods based on Wi-Fi wireless signals avoid occlusion and privacy issues and do not require wearable devices. However, they have two major limitations: the Wi-Fi modality is subject to interference from the external environment, resulting in poor robustness; and the low resolution of Wi-Fi itself leads to low recognition accuracy. Summary of the Invention

[0004] In the field of wireless perception, in order to solve the two major limitations of Wi-Fi signals, namely, poor robustness due to external environmental interference and low recognition accuracy due to low resolution of Wi-Fi itself, the present invention provides a human posture estimation method based on multimodal contrastive learning, which adopts the method of using LiDAR modality to assist Wi-Fi modality for training. In the training phase, the model is trained with Wi-Fi modality as the main modality and laser radar as the auxiliary modality for joint training. In the testing phase, only Wi-Fi single modality is used for testing and verification, and the two multimodalities are integrated to give full play to their respective advantages. On the one hand, the accuracy of Wi-Fi perception is improved by providing high-precision three-dimensional spatial information through the LiDAR modality. On the other hand, the robustness of Wi-Fi perception is better improved by providing information in multi-view scenarios through the LiDAR modality. At the same time, the present invention adopts contrastive learning technology to solve the problem of alignment between different modal data. By learning the similarities and differences between the data, the features of different modalities are mapped into a unified feature space, thereby achieving feature alignment. This invention adopts a multimodal fusion approach to provide richer information from multiple perspectives for model training, thereby more accurately perceiving human behavior. It has important theoretical and practical significance for the advancement of AIoT in the field of human posture estimation.

[0005] In order to solve the above technical problems, the present invention proposes a method for estimating human posture based on multimodal contrastive learning, comprising the following steps:

[0006] Step 1: Deploy Wi-Fi devices and Ouster OS1 32-channel LiDAR devices in a real-world scenario. Collect and synchronize data using ROS, and use NTP to time-align Wi-Fi and LiDAR data.

[0007] Step 2: Data preprocessing is performed on the synchronously input Wi-Fi data and LiDAR data. For the synchronously input Wi-Fi data (CSI), conjugate multiplication and a Butterworth filter are first used to eliminate random phase errors and static environmental noise in the CSI, respectively. The denoised data is then reduced in dimension using PCA. Finally, a short-time Fourier transform is used to perform time-frequency analysis on the reduced-dimensional data to obtain a spectrum of the Wi-Fi modality. For the synchronously input LiDAR data, data augmentation techniques are first used to obtain a pair of enhanced point cloud data frames for each frame. The point cloud feature extraction network PointNet++ is then used to extract human skeleton features from each pair of point clouds. Intra-modal contrastive learning is then used to enhance the quality of the point cloud feature representation.

[0008] Step 3: Using Wi-Fi data as the primary modality and LiDAR data as the auxiliary modality, multimodal joint training is performed. This includes: using the Wi-Fi modality spectrogram obtained in Step 2, a convolutional neural network (CNN) and a residual network (ResNet34) are used to extract human motion posture features from the Wi-Fi modality. Using the human skeleton represented by the point cloud features obtained in Step 2, a spatiotemporal graph convolutional network (ST-GCN) is used to extract human motion posture features from the LiDAR modality. Finally, cross-modal comparative learning is used to complete self-supervised training of the model, achieving feature alignment between the two heterogeneous modalities to obtain coordinates for human pose estimation.

[0009] Step 4: In the testing phase, only Wi-Fi single modality is used as input data and input into the trained model as a test to obtain the result of human posture estimation.

[0010] Furthermore, in step 2, conjugate multiplication and a Butterworth filter are used on the synchronously input Wi-Fi data CSI to eliminate random phase errors and static environmental noise, respectively. The process is as follows: For the CSI from different antennas on the same network card at the receiving end, a conjugate multiplication algorithm is used, and the phase offset in the CSI is eliminated by adjusting the power values of different antennas. A fifth-order Butterworth filter is then used to remove environmental noise contained in the CSI, and the denoised CSI is normalized. The preprocessing of the synchronously input LiDAR data is as follows: For the input synchronous LiDAR data frame, a pair of enhanced point cloud data is obtained through random combination of translation, random dropout, and random dithering. Each pair of point cloud data is input into the point cloud feature extraction network PointNet++ to extract a pair of point cloud features. Finally, each pair of enhanced point cloud features is used as a positive sample pair for intra-modal contrastive learning to enhance the quality of the human skeleton represented by each frame of point cloud data.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] (1) In the field of wireless perception, in order to solve the two major limitations of Wi-Fi signals, namely, poor robustness due to external environmental interference and low recognition accuracy due to low resolution of Wi-Fi itself, the present invention proposes a method of using LiDAR modality to assist Wi-Fi modality in training. In the training phase, the model is trained with Wi-Fi modality as the main modality and LiDAR as the auxiliary modality for joint training. In the testing phase, only Wi-Fi modality is used for testing and verification, and the two modalities are integrated to give full play to their respective advantages. On the one hand, the accuracy of Wi-Fi perception is improved by providing high-precision three-dimensional spatial information through the LiDAR modality. On the other hand, the robustness of Wi-Fi perception is better improved by providing information in multi-view scenarios through the LiDAR modality.

[0013] (2) In the scenario of multimodal fusion, similar to the differences between image and text modalities, there are large differences between 2D Wi-Fi images and 3D LiDAR point cloud data. How to solve the heterogeneous problem between features of different dimensions is also another challenge. The present invention adopts contrastive learning technology to solve the problem of alignment between different modal data. By learning the similarities and differences between data, the features of different modalities are mapped into a unified feature space, thereby achieving feature alignment. At the same time, in the multimodal contrast training process, the global information of Wi-Fi modality training can provide the overall motion contour of the body, and the local information extracted by LiDAR modality can provide the movement details of the posture. Contrastive learning is used to achieve mutual complementation between global information and local information, thereby improving the quality of feature representation.

[0014] (3) This paper adopts PointNet++ to extract the joint features of human body posture and obtains the spatiotemporal relationship of human body movement through ST-GCN. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flowchart of the human body posture estimation method based on multimodal contrast learning proposed by the present invention;

[0016] Figure 2 This is the synchronous data collection scenario in the present invention;

[0017] Figure 3 It is a system framework diagram for implementing the method of the present invention;

[0018] Figure 4 This is the spectrum diagram generated by the Wi-Fi mode of the present invention;

[0019] Figure 5 This is a schematic diagram of generating a human skeleton using point cloud modality according to the present invention;

[0020] Figure 6 Schematic diagram of cross-modal contrastive learning in the present invention;

[0021] Figure 7 Schematic diagram of the network structure of the residual network ResNet34 in the present invention;

[0022] Figure 8 It is a schematic diagram of the network structure of the spatiotemporal graph convolutional network ST-GCN in the present invention. DETAILED DESCRIPTION

[0023] The design concept of the human posture estimation method based on multimodal contrastive learning proposed in the present invention is as follows: for the input Wi-Fi modal data, the CSI data is first preprocessed. This includes using the conjugate multiplication method to eliminate the random phase error of the CSI, using the Butterworth filter to eliminate the environmental noise, and using PCA to reduce the dimension of the denoised data. Then, the short-time Fourier transform is used to obtain the spectrum of the Wi-Fi data. For the input LiDAR modal data, this method first extracts the point cloud human skeleton from the point cloud data of the lidar. Specifically, for each point cloud data frame, data enhancement technology is first used to obtain a pair of enhanced point cloud data. Then, the PointNet++ network is used to extract the human skeleton of this pair of point cloud data respectively, and the point cloud intra-modal contrast learning method is used to enhance the quality of the point cloud feature representation of the human skeleton. This method will then jointly train the two modalities after preprocessing. For the spectrum diagram of the Wi-Fi modality, the convolutional network CNN and the residual network ResNet34 will be used to extract the human motion posture features of the Wi-Fi modality; for the human skeleton sequence of the LiDAR modality, the spatiotemporal graph convolutional network ST-GCN will be used to extract the human motion posture features of the point cloud data. Finally, cross-modal contrast learning is used to achieve feature alignment between heterogeneous modalities, and the aligned features are used for human posture estimation. In the present invention, richer features can be obtained from multiple perspectives, and the trained model is tested using a single Wi-Fi modality. This method is used to solve the problem of poor robustness and poor accuracy of estimated posture in Wi-Fi perception due to the influence of the environment on the signal. The specific steps of the human posture estimation method based on multi-modal contrast learning of the present invention are as follows:

[0024] Step 1: Deploy Wi-Fi devices and an Ouster OS1 32-channel LiDAR device in a real-world scenario for data collection. Data collection and synchronization are achieved through the Robot Operating System (ROS). Wi-Fi and LiDAR data are time-aligned using NTP (Network Time Protocol) to ensure multimodal data synchronization. Figure 1 It is a flow chart of the scheme of the present invention; Figure 2 It is the dataset collection scenario; Figure 3 It is a system framework diagram of the present invention.

[0025] Step 2: Data preprocessing, including:

[0026] (1) For the synchronously input Wi-Fi data CSI (Channel State Information).

[0027] First, conjugate multiplication and a Butterworth filter are used to remove random phase errors and static environmental noise from the CSI, respectively. Principal Component Analysis (PCA) is then used to reduce the dimensionality of the denoised data. Finally, a short-time Fourier transform (SFT) is used to perform time-frequency analysis on the reduced-dimensional data, generating a spectrum diagram of the Wi-Fi mode. For the CSI from different antennas on the same network card at the receiver, a conjugate multiplication algorithm is used, and the power values of the different antennas are adjusted to eliminate phase offsets in the CSI. A fifth-order Butterworth filter is then used to remove environmental noise contained in the CSI, and the denoised CSI is normalized. The details are as follows:

[0028] Step 2A-1) For CSI1 and CSI2 on different antennas in the same network card, the result of conjugate multiplication can be expressed as:

[0029]

[0030] In the above formula, A noise (f, t) is the environmental noise, H s1 (f,t) and H d1 (f, t) are the static and dynamic components of CSI1, and The dynamic and static components of CSI2 are shown in Table 1. Item (1) is the product of the static path component of CSI1 and the static path component of CSI2. By definition, this item can be considered a constant. Item (2) is the product of the static path component of CSI1 and the dynamic path component of CSI2. Item (3) is the product of the dynamic path component of CSI1 and the static path component of CSI2. Item (4) is the product of the dynamic path components of the two link CSIs. Since the energy caused by the dynamic path is much smaller than the energy of the static environment, the value of this item will be very small and can be ignored.

[0031] Step 2A-2) can obtain the Doppler velocity from the product term of item (3). To reduce the interference caused by item (4), the power of the static path component on antenna 1 is reduced by subtracting a value α from CSI1 of antenna 1, and the power of the static path component of CSI2 on antenna 2 is increased by adding a value β. Through the above-mentioned power amplification and reduction steps, item (3) plays a dominant role in the final calculation result, thereby eliminating the phase offset of CSI and can be used in subsequent spectrum generation.

[0032] In the final formula of step 2A-3, we can see that the conjugate multiplication method can eliminate the CSI phase offset, but it further amplifies the CSI amplitude noise. Based on this, the present invention uses a fifth-order Butterworth filter with a cutoff frequency range of 10 Hz to 80 Hz, effectively removing high-frequency noise while retaining low-frequency human motion information. Finally, to ensure a reasonable interval distribution of the data, a normalization step is performed to uniformly scale the CSI values to a range of -1 to 1 for subsequent data processing.

[0033] Step 2A-4) Perform PCA dimensionality reduction on the CSI data processed in step 2A-3). This includes calculating the covariance matrix of the normalized data and performing eigendecomposition. The eigenvectors corresponding to the first k largest eigenvalues are selected to form a dimensionality reduction matrix. This matrix is used to project the original CSI data into a low-dimensional space, thereby achieving data dimensionality reduction.

[0034] Step 2A-5) For the data after dimensionality reduction in step 2A-4), a short-time Fourier transform is used to obtain a spectrum of the CSI data. This includes:

[0035] In human motion monitoring in the field of wireless sensing, Doppler shift is one of the important characteristics reflecting the motion state. Its calculation can be expressed as follows:

[0036]

[0037] Where λ represents the signal wavelength and d(t) represents the length of the CSI propagation path.

[0038] In order to convert the CSI data after PCA dimensionality reduction into a Doppler spectrum, the present invention uses short-time Fourier transform to convert it into the time-frequency domain. The formula is as follows:

[0039]

[0040] Among them, X(t,f) is the representation of the signal converted to the time-frequency domain, x(n) is the original signal, w(nt) is the sliding window function, t is time, and f is frequency.

[0041] The conversion steps are as follows: First, the CSI measurement waveform is divided into multiple segments using a sliding window technique. To better track changes in signal frequency, this method uses a window size of 1024 and a sliding step size of 1. Then, a fast Fourier transform is performed on each segment to generate the corresponding spectrum. Figure 4 This is a spectrum diagram of the Wi-Fi mode obtained by the present invention.

[0042] (2) For synchronous input LiDAR data

[0043] First, data augmentation technology is used to obtain a pair of enhanced point cloud data frames for each frame to enhance the data quality of the point cloud. Then, the point cloud feature extraction network PointNet++ is used to extract the features of the human skeleton of this pair of point clouds. Finally, intra-modal contrast learning is used to enhance the quality of the point cloud feature representation and obtain the human skeleton represented by the enhanced point cloud feature. Figure 5 This is the point cloud modality extraction of the human skeleton in this invention. The preprocessing of synchronous input LiDAR data involves: obtaining a pair of enhanced point cloud data from each frame of synchronous LiDAR data through random combination of translation, random discarding, and random dithering; inputting each pair of point cloud data into the point cloud feature extraction network PointNet++ to extract a pair of point cloud features; and finally, using each pair of enhanced point cloud features as a positive sample pair for intra-modal comparative learning to enhance the quality of the human skeleton represented by each frame of point cloud data. The details are as follows:

[0044] Step 2B-1) For each input 3D point cloud data frame p i , the present invention constructs its enhanced version p i t1 and p i t2 , by randomly combining transformations from the transformation set T in a sequential manner.

[0045] Step 2B-2) The transformation set includes random translation, random dithering, random dropout, and other transformations, and the transformation matrix parameters are randomly initialized. Through data enhancement, two different enhanced versions of the point cloud human body data frames can be obtained.

[0046] Step 2B-3) The present invention uses PointNet++ as a point cloud feature extractor, and simultaneously inputs two different enhanced point cloud data frames into the point cloud feature extraction network PointNet++, thereby converting p i t1 and p i t2 Mapped to feature space.

[0047] Step 2B-4) PointNet++ is based on a layered sampling strategy. First, keypoints are selected from the input point cloud. These keypoints serve as the center points of local regions. Then, local regions are defined around each keypoint, and local features are extracted from the points within this region. This process then uses a weight-sharing MLP to learn feature representations within the local region. Finally, an aggregation operation is used to integrate the local features onto the keypoints to form a higher-level feature representation. This process is repeated at multiple levels, gradually extracting features from the local to the global level, allowing the model to capture structural information at different scales.

[0048] Step 2B-5) uses an intra-modality contrastive learning algorithm to enhance the quality of the human skeleton represented by each frame of point cloud data, including:

[0049] Step 2B-5-1) The present invention uses each pair of enhanced point cloud features as a positive sample pair and randomly selects other point cloud features from the same dataset as negative samples. By maximizing the similarity between the positive sample pairs and minimizing the similarity with the negative sample pairs, a higher-quality human skeleton sequence is obtained.

[0050] Step 2B-5-2) The present invention uses NT-Xent loss to distinguish instances at this stage and calculate the positive example pair z i t1 and z i t2 The contrast loss function l(i,t1,t2) is:

[0051]

[0052] In formula (4), τ is the temperature coefficient, s(·) represents the cosine similarity function; when the mini-batch size is N, the intra-modal instance discrimination loss function ι of the mini-batch sample is imid It is expressed as follows:

[0053]

[0054] The intra-modal instance discrimination loss function ι for this mini-batch of samples imid Represents the invariance imposed by various transformations within the point cloud modality.

[0055] Step 3: Multimodal joint training. During model training, Wi-Fi signals are used as the primary modality and LiDAR point cloud data as the auxiliary modality for joint training. Multimodal joint training is performed using Wi-Fi data as the primary modality and LiDAR data as the auxiliary modality; this includes:

[0056] (1) Spectrum diagram for Wi-Fi mode.

[0057] Convolutional neural network (CNN) and residual network (ResNet34) are used to extract human motion posture features in Wi-Fi mode. The specific contents are as follows:

[0058] Step 3A-1) Using a convolutional neural network and a residual network ResNet34 as the feature extraction network for the Wi-Fi modality;

[0059] The CSI feature extraction spectrogram is treated as a two-dimensional image. Using 2D convolution, we can capture spatial and temporal features with smaller parameters. The convolutional neural network includes a convolutional layer for image extraction, a 3×3×64 convolution kernel, BatchNorm and ReLU activation functions, and maximum pooling to produce a compressed feature map.

[0060] Step 3A-2) For the local feature map output by the convolutional neural network, input the ResNet34 deep network to extract global features, and the residual structure is expressed as:

[0061] y=F(x)+x (6)

[0062] Where x represents the input of the residual block, y represents the output of the residual block, and F(x) represents the output of the feedforward neural network.

[0063] The residual network ResNet34 includes 5 stages, namely Stage0, Stage1, Stage2, Stage3, and Stage4. Stage0 uses a 7×7 convolution to perform preliminary feature extraction on the input image to further reduce the size of the feature map; Stage1 to Stage4 are composed of 3, 4, 6, and 3 BasicBlock residual blocks respectively. Their structures are as follows: Figure 7 shown.

[0064] Step 3A-3) After completing the feature encoding of ResNet34, connect the fully connected FC layer to facilitate subsequent comparative learning with the features of the LiDAR modality.

[0065] (2) Human skeleton sequence for LiDAR modality.

[0066] The present invention uses a spatiotemporal graph convolutional network (ST-GCN) to obtain spatial motion features and temporal features in a spatiotemporal skeleton graph. This includes:

[0067] Step 3B-1) ST-GCN is a network architecture for human activity recognition based on skeletons. It extracts features of predicted joints in the form of a graph structure. Its input is skeleton key point data with spatiotemporal attributes. Each node represents a joint, and the edge represents the natural connection and temporal continuity between joints. Its structure is as follows: Figure 8 shown.

[0068] Step 3B-2) Design LiDAR-assisted modality extraction features. For the human skeleton sequence extracted from point cloud, each point cloud feature vector is regarded as an independent node, and the edges between nodes are defined based on the inherent topological structure of point cloud data. The present invention connects the frame-by-frame global features with each joint to generate joint features. As the construction of the graph node. The input joint features represent the skeleton sequence of the human body. At the same time, the input skeleton data first enters the batch normalization layer. This preprocessing step is essential. It eliminates the impact of different joint data due to the difference in measurement scale and ensures that all joint data are processed at the same scale.

[0069] Step 3B-3) The spatiotemporal graph convolutional network (ST-GCN) operates within a local neighborhood of the graph network structure data to extract features. Based on the sampling function and weight function, the graph convolution process is expressed as:

[0070]

[0071] In formula (7), f in (v tj ) is the node v tj Input features, f out (v ti ) is the node v tj The output features, Z ti (v tj ) is a normalization term, indicating the difference between tj The number of nodes with the same label is used to balance the contribution of different subsets to the output.

[0072] Step 3B-4) The spatiotemporal graph convolution block includes two parts: spatial graph convolution (GCN) and temporal graph convolution (TCN). ST-GCN contains a total of 10 layers of blocks, but the first layer does not have a residual structure. Its structure is as follows Figure 8 After the above network layer-by-layer feature extraction, at the output end of ST-GCN, the feature vector is input into the fully connected layer, and after nonlinear transformation, the human motion posture characteristics of the point cloud modality are obtained.

[0073] (3) In addition to contrastive learning within the point cloud modality, the present invention also introduces cross-modal contrastive learning.

[0074] In this paper, comparative learning between the LiDAR and Wi-Fi modalities is used to achieve feature alignment between the two heterogeneous modalities, thereby achieving complementarity between global and local features and enhancing feature representation capabilities. The aligned features are then used to obtain coordinates for human pose estimation, thereby enabling human pose estimation. Figure 6 It is cross-modal contrastive learning.

[0075] Step 3C-1) On the Wi-Fi side, use the Wi-Fi backbone network to embed the spectrogram image into the feature space and define the projected Wi-Fi feature as h i At the same time, on the LiDAR side, the feature vector z of the LiDAR modality is obtained through the spatiotemporal graph convolutional network ST-GCNi ;

[0076] Step 3C-2) In the feature space, the next step is to make z i With h i The similarity of z is maximized because they all correspond to the same posture results. To this end, the present invention adopts a cross-modal contrast loss function, which improves the feature representation ability of the model by introducing more different positive and negative sample pairs between modalities; calculate the positive sample z i and h i The contrast loss function l(i,z,h) is:

[0077]

[0078] When the mini-batch size is N, the cross-modal contrast loss function ι under mini-batch training is cmid Expressed as:

[0079]

[0080] The cross-modal contrast loss function ι under this mini-batch training cmid Represents the feature fusion relationship between the 3D point cloud and 2D spectrogram added in the feature space;

[0081] Step 3C-3) The total loss function obtained during the training process is ι imid and ι cmid Combination of . imid represents the invariance imposed by various transformations within the point cloud modality, and ι cmid Represents the feature fusion relationship between the 3D point cloud and 2D spectrogram added in the feature space, and the total loss function is:

[0082] ι=ι imid +ι cmid (10)

[0083] Step 3C-4) After completing cross-modal contrastive learning, the acquired feature vector Using the mapping of multiple layers of fully connected layers, the final output of the human body key points is expressed as:

[0084] Y=W2σ(W1f+b)1+b (11)

[0085] In formula (11), Y is the coordinate of the key points of the human body, W1 and W2 are weight matrices, b1 and b2 are biases, and σ is the ReLU activation function.

[0086] Step 4: Contrastive learning optimizes the overall semantic relationship between modalities and achieves the complementarity of global and local information. During the model testing phase, only the Wi-Fi modality is used for testing to complete human posture estimation under a single Wi-Fi network.

[0087] The CNN and ResNet34 in step 3 are used as feature extractors for Wi-Fi data in the test phase. Only the Wi-Fi mode is used as test data to generate a spectrogram, which is then input into the trained CNN and ResNet34 to obtain the human pose estimation results.

[0088] Example:

[0089] The human body posture estimation method based on multimodal contrastive learning proposed in the present invention specifically comprises the following steps in the posture estimation task in a real scene:

[0090] Step 1: Use the TP-Link N750 and the Ouster O 32-channel lidar to collect synchronized data in a real-world scenario.

[0091] 1-1) Wi-Fi data is collected from the TP-Link N750 using the Atheros CSI tool at a sampling rate of 1000 Hz, ultimately providing a 100 Hz CSI data stream. This data stream includes one transmit antenna and three receive antennas. Each antenna pair transmits 114 subcarriers with a 40 MHz bandwidth, forming a 3 × 114 × 10 matrix over an average time period of 100 ms. LiDAR data is collected by an Ouster OS1 32-channel lidar at a frequency of 10 Hz, capturing a dense point cloud of up to 1,310,720 points per second.

[0092] 1-2) Data is synchronized between Wi-Fi and LiDAR data using the Network Time Protocol (NTP). All devices are connected to the same mini-PC and use ROS for data collection and synchronization. ROS is installed on the mini-PC for data collection and synchronization. During data collection, the platform is placed 3.0 meters away from the sensing target, such as Figure 2 In this way, good synchronization of subsequent input multimodal data can be guaranteed.

[0093] 1-3) Taking into account the differences brought about by different populations, data collection included 10 experimental personnel, including 11 women and 29 men, with an average age of 25.3±2.8, a weight range of 66.1±12.0kg, a height range of 172.3±7.9cm, and a body mass index (BMI) of 22.2±3.2. The final raw data collected included 27 actions repeatedly performed by all participants within 30 seconds.

[0094] Step 2: Data preprocessing and modeling. Process the collected Wi-Fi data and LiDAR data, convert the CSI data into a spectrogram, and convert the LiDAR data into a human skeleton sequence. The specific steps are as follows:

[0095] 2-1) For the CSI data received from different antennas on the same network card, the conjugate multiplication algorithm of formula (1) is used to adjust the power values of different antennas to eliminate the phase offset and static components in the CSI.

[0096] 2-2) Based on this, a fifth-order Butterworth filter is used with a cutoff frequency range of 10 Hz to 80 Hz. This effectively removes high-frequency noise while retaining low-frequency human motion information. Finally, to ensure a reasonable data distribution, the CSI values are normalized to a range of -1 to 1.

[0097] 2-3) PCA dimensionality reduction is performed on the initially processed CSI data. The covariance matrix of the standardized data is calculated and eigendecomposition is performed. The eigenvectors corresponding to the first k largest eigenvalues are selected to form a dimensionality reduction matrix. This matrix is used to project the original CSI data into a low-dimensional space, thus achieving data dimensionality reduction.

[0098] 2-4) For the data after dimensionality reduction, use formula (3) and short-time Fourier transform to obtain the spectrum of CSI data. The spectrum of different actions is as follows: Figure 4 shown.

[0099] 2-5) For the input synchronized LiDAR point cloud data frame, data enhancement techniques such as translation, random discarding, and random dithering are used to obtain a pair of enhanced versions of 3D point cloud data.

[0100] 2-6) Input this pair of point cloud data into the PointNet++ point cloud feature extraction network to obtain the human skeleton sequence of the point cloud.

[0101] 2-7) Then, the intra-modal contrast learning algorithm is used to enhance the quality of the human skeleton of this frame of point cloud data. The process of extracting the human skeleton sequence is as follows: Figure 5 shown

[0102] Step 3: Model training. Divide the data set in step 1 into 80% as the training set and 20% as the test set. Both the training set and the test set contain different users, different environments, and different actions. For Wi-Fi data and LiDAR data, the network model is used for training. The specific steps are as follows:

[0103] 3-1) Convolutional neural network (CNN) and residual network (ResNet34) are used as feature extraction networks for Wi-Fi mode.

[0104] 3-2) The convolutional network design includes a convolutional layer for image extraction, a 3×3×64 convolution kernel, BatchNorm and ReLU activation functions, and max pooling to produce a compressed feature map. The ResNet34 network consists of five stages, of which Stage 0 uses a 7×7 convolution to perform preliminary feature extraction on the input image, further reducing the size of the feature map. Stages 1 through 4 are composed of 3, 4, 6, and 3 residual blocks, respectively, as shown in the BasicBlocks. After completing the ResNet34 feature encoding, a fully connected FC layer is connected to facilitate subsequent comparative learning with LiDAR modality features.

[0105] 3-3) The spatiotemporal graph convolutional network ST-GCN is used to obtain the spatial motion features and temporal features in the spatiotemporal skeleton graph. The frame-by-frame global features are connected with each joint to generate the joint features. As the construction of the graph node. The input joint features represent the skeleton sequence of the human body. The input skeleton data first enters the batch normalization layer. This preprocessing step is essential. It can eliminate the influence of the measurement scale difference between different joint data and ensure that all joint data are processed at the same scale.

[0106] 3-4) The ST-GCN network extracts features layer by layer, ultimately outputting a 256-dimensional feature vector. This feature vector not only condenses spatiotemporal information but also provides an informative feature representation for the final regression task. Subsequently, at the output of the ST-GCN, the resulting feature vector is fed into a fully connected layer. After undergoing nonlinear transformations, the FC layer further transforms and integrates the features to better suit the subsequent contrastive learning task.

[0107] 3-5) For the features of two different modalities, use the cross-modal contrast learning in formula (8) to obtain the final human posture features. After completing the cross-modal contrast learning, for the obtained feature vector Using multi-layer fully connected layer mapping, the output is represented as the coordinates of 17 key points of the human body.

[0108] 3-6) The model was trained for 80 epochs, with the model with the lowest loss being the final model. Each batch processed 128 samples, with an initial learning rate of 0.001, which was halved every ten epochs using a multi-step decay strategy. The training process used the stochastic gradient descent (SGD) optimizer, with random shuffling applied when loading each batch of data. This parameter configuration allows for the efficient construction of a model with stable performance and high accuracy.

[0109] Step 4: Single-model testing: Use the CNN and ResNet34 trained in Step 3 as the feature extraction network for the Wi-Fi modality in the test phase. Use the CNN and ResNet34 from Step 3 as the feature extractor for Wi-Fi data in the test phase. Use only the Wi-Fi modality as test data. Generate a spectrogram and input it into the trained CNN and ResNet34 to obtain human pose estimation results.

[0110] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many improvements and changes without departing from the purpose of the present invention, which are all protected by the present invention.

Claims

1. A human posture estimation method based on multimodal contrast learning, characterized in that: The following steps are involved: Step 1: Deploy Wi-Fi devices and Ouster OS1 32-channel LiDAR devices in a real-world scenario. Collect and synchronize data using ROS, and use NTP to time-align Wi-Fi and LiDAR data. Step 2: Preprocess the Wi-Fi data and LiDAR data input simultaneously, including: For the synchronous input Wi-Fi CSI data, conjugate multiplication and a Butterworth filter are first used to remove random phase errors and static environmental noise in the CSI, respectively. PCA is then used to reduce the dimensionality of the denoised data. Finally, a short-time Fourier transform is used to perform time-frequency analysis on the reduced-dimensional data to obtain a spectrum of the Wi-Fi modality. For the synchronous input LiDAR data, data augmentation technology is first used to obtain a pair of enhanced point cloud data frames for each frame. Then, the point cloud feature extraction network PointNet++ is used to extract the human skeleton features of each pair of point clouds. Intra-modal contrast learning is used to enhance the quality of the point cloud feature representation. Step 3: Use Wi-Fi data as the primary modality and LiDAR data as the auxiliary modality to perform multimodal joint training, including: For the Wi-Fi modality spectrum obtained in step 2, the convolutional neural network (CNN) and the residual network (ResNet34) are used to extract the human motion posture features of the Wi-Fi modality. For the human skeleton represented by the point cloud features obtained in step 2, the spatiotemporal graph convolutional network ST-GCN is first used to extract the human motion posture features of the LiDAR modality; Using cross-modal contrastive learning, the model is trained through self-supervision, achieving feature alignment between two heterogeneous modalities to obtain coordinates for human pose estimation. Step 4: In the testing phase, only Wi-Fi single modality is used as input data and input into the trained model as a test to obtain the result of human posture estimation.

2. The human body posture estimation method according to claim 1, wherein In step 2, conjugate multiplication and a Butterworth filter are used on the synchronously input Wi-Fi data CSI to eliminate random phase errors and static environmental noise in the CSI. The process is as follows: For the CSI of different antennas on the same network card at the receiving end, a conjugate multiplication algorithm is used, and the power values of different antennas are adjusted to eliminate the phase offset in the CSI. Then, a fifth-order Butterworth filter is used to remove environmental noise contained in the CSI, and the denoised CSI is normalized.

3. The method for estimating human body posture according to claim 1, wherein: In step 2, the specific contents of pre-processing the synchronously input Wi-Fi data CSI include: Step 2A-1) For CSI1 and CSI2 on different antennas in the same network card, the result of conjugate multiplication can be expressed as: In the above formula, A noise (f, t) is the environmental noise, H s1 (f,t) and H d1 (f, t) are the static and dynamic components of CSI1, and The dynamic component and static component of CSI2; Among them, item (1) is the product of the static path component of CSI1 and the static path component of CSI2. According to the definition, this item can be regarded as a constant; item (2) is the product of the static path component of CSI1 and the dynamic path component of CSI2; item (3) is the product of the dynamic path component of CSI1 and the static path component of CSI2; item (4) is the product of the dynamic path components of the two link CSIs, and the value of this item is negligible; Step 2A-2) Obtain the Doppler velocity from the product term of item (3) of equation (1), reduce the power of the static path component on antenna 1 by subtracting a value α from CSI1 of antenna 1, and increase the power of the static path component on antenna 2 CSI2 by adding a value β, thereby eliminating the phase offset of CSI; Step 2A-3) Use a fifth-order Butterworth filter with a cutoff frequency range of 10 Hz to 80 Hz, and normalize the CSI value to a range of -1 to 1. Step 2A-4) Performing PCA dimensionality reduction on the CSI data processed in step 2A-3), including: calculating the covariance matrix of the normalized data and performing eigendecomposition, selecting the eigenvectors corresponding to the first k largest eigenvalues to form a dimensionality reduction matrix, and projecting the original CSI data into a low-dimensional space using this matrix; Step 2A-5) For the data after dimensionality reduction in step 2A-4), a short-time Fourier transform is used to obtain a spectrum of the CSI data, including: Step 2A-5-1) In the field of wireless sensing for human motion monitoring, the Doppler shift calculation is expressed as follows: Where λ represents the signal wavelength and d(t) represents the length of the CSI propagation path. The short-time Fourier transform is used to convert it into the time-frequency domain, and the formula is as follows: Where X(t,f) is the representation of the signal converted to the time-frequency domain, x(n) is the original signal, w(nt) is the sliding window function, t is time, and f is frequency; In step 2A-5-2), the CSI measurement waveform is divided into multiple segments using a sliding window technique, with the window size set to 1024 and the sliding step size set to 1. Then, each segment is subjected to a fast Fourier transform to generate a corresponding spectrum diagram.

4. The method for estimating human body posture according to claim 1, wherein: In step 2, the preprocessing of the synchronous input LiDAR data is as follows: for the input synchronous LiDAR data frame, a pair of enhanced point cloud data is obtained by randomly combining translation, random discarding, and random dithering; each pair of point cloud data is input into the point cloud feature extraction network PointNet++ to extract a pair of point cloud features; finally, each pair of enhanced point cloud features is used as a positive sample pair for intra-modal contrast learning to enhance the quality of the human skeleton represented by each frame of point cloud data.

5. The method for estimating human body posture according to claim 1, wherein The specific contents of preprocessing the synchronous input LiDAR data include: Step 2B-1) For each input point cloud data frame p i , construct its enhanced version p i t1 and p i t2 , by randomly combining transformations from the transformation set T in a sequential manner; Step 2B-2) The transformation set includes random translation, random dithering, and random dropping, wherein the transformation matrix parameters are all randomly initialized; two different enhanced point cloud data frames are obtained through data enhancement; Step 2B-3) The two different enhanced point cloud data frames are simultaneously input into the point cloud feature extraction network PointNet++, thereby p i t1 and p i t2 Mapping to feature space; Step 2B-4) PointNet++ is based on layered sampling. First, keypoints are selected from the input point cloud and used as the center points of local regions. Then, a local region is defined around each keypoint. Local features are extracted from the points within the region, and a shared-weight MLP is used to learn the feature representation within the local region. Finally, the local features are integrated into the keypoints through aggregation to form a higher-level feature representation. This process is repeated at multiple levels, gradually extracting features from local to global, allowing the model to capture structural information at different scales. Step 2B-5) uses an intra-modality contrastive learning algorithm to enhance the quality of the human skeleton represented by each frame of point cloud data, including: Step 2B-5-1) Use each pair of enhanced point cloud features as a positive sample pair, and randomly select other point cloud features from the same dataset as negative samples; obtain a higher quality human skeleton sequence by maximizing the similarity between the positive sample pairs and minimizing the similarity with the negative sample pairs; Step 2B-5-2) Use NT-Xent loss to distinguish instances and calculate the positive example pair z i t1 and z i t2 The contrast loss function l(i,t1,t2) is: In formula (4), τ is the temperature coefficient, s(·) represents the cosine similarity function; when the mini-batch size is N, the intra-modal instance discrimination loss function ι of the mini-batch sample is imid It is expressed as follows: The intra-modal instance discrimination loss function ι for this mini-batch of samples imid Represents the invariance imposed by various transformations within the point cloud modality.

6. The method for estimating human body posture according to claim 1, wherein: In step 3, for the Wi-Fi modality spectrum obtained in step 2, a convolutional neural network (CNN) and a residual network (ResNet34) are used to extract the human motion posture features of the Wi-Fi modality. include: Step 3A-1) Using a convolutional neural network and a residual network ResNet34 as the feature extraction network for the Wi-Fi modality; The spectrum of the CSI feature extraction is treated as a two-dimensional image, and 2D convolution is used to capture spatial and temporal features. The convolutional neural network includes a convolution layer for extracting the image, a 3×3×64 convolution kernel, BatchNorm and ReLU activation functions, and maximum pooling to obtain a compressed feature map. Step 3A-2) The local feature map output by the convolutional neural network is input to the residual network ResNet34 to extract global features. The residual structure is represented as: y=F(x)+x (6) Where x represents the input of the residual block, y represents the output of the residual block, and F(x) represents the output of the feedforward neural network; The residual network ResNet34 includes 5 stages, namely Stage0, Stage1, Stage2, Stage3, and Stage4. Stage0 uses a 7×7 convolution to perform preliminary feature extraction on the input image to further reduce the size of the feature map; Stage1 to Stage4 are composed of 3, 4, 6, and 3 BasicBlock residual blocks respectively; Step 3A-3) After completing the feature encoding of ResNet34, connect the fully connected FC layer to facilitate subsequent comparative learning with the features of the LiDAR modality.

7. The method for estimating human body posture according to claim 5, wherein: In step 3, for the human skeleton represented by the point cloud features obtained in step 2, the spatiotemporal graph convolutional network ST-GCN is used to extract the human motion posture features of the point cloud data, including: Step 3B-1) The spatiotemporal graph convolutional network (ST-GCN) is a network architecture for skeleton-based human activity recognition. It extracts features for predicting joints in the form of a graph structure. Its input is skeleton keypoint data with spatiotemporal properties. Each node represents a joint, and the edges represent the natural connection and temporal continuity between joints. Step 3B-2) Design LiDAR-assisted modality extraction features. For the human skeleton sequence extracted from point cloud, each point cloud feature vector is regarded as an independent node, and the edges between nodes are defined according to the inherent topological structure of the point cloud data; the frame-by-frame global features are connected with each joint to generate joint features. As the construction of graph nodes; the input joint features represent the skeleton sequence of the human body. At the same time, the input skeleton data first enters the batch normalization layer to eliminate the impact of different joint data due to measurement scale differences, ensuring that all joint data are processed at the same scale; Step 3B-3) The spatiotemporal graph convolutional network ST-GCN extracts features by performing graph convolution operations. Based on the sampling function and weight function, the graph convolution process is expressed as: In formula (7), f in (v tj ) is the node v tj Input features, f out (v ti ) is the node v tj The output features, Z ti (v tj ) is a normalization term, indicating the difference between tj The number of nodes with the same label is used to balance the contribution of different subsets to the output; Step 3B-4) The block of the spatiotemporal graph convolutional network ST-GCN contains two parts: spatial graph convolution (GCN) and temporal graph convolution (TCN). ST-GCN contains a total of 10 layers of blocks, of which the first layer of blocks has no residual structure. After the above network layer-by-layer feature extraction, the feature vector obtained at the output end of the spatiotemporal graph convolutional network ST-GCN is input into the fully connected layer. After nonlinear transformation, the human motion posture characteristics of the point cloud modality are obtained.

8. The method for estimating human body posture according to claim 5, wherein: In step 3, cross-modal contrastive learning is used to complete self-supervised training of the model, achieve feature alignment between the two heterogeneous modalities, and use the aligned features for human pose estimation, including: Step 3C-1) On the Wi-Fi side, use the Wi-Fi backbone network to embed the spectrogram image into the feature space and define the projected Wi-Fi feature as h i At the same time, on the LiDAR side, the feature vector z of the LiDAR modality is obtained through the spatiotemporal graph convolutional network ST-GCN i ; Step 3C-2) In the feature space, the next step is to make z i With h i The similarity of the positive sample z is maximized, and the cross-modal contrast loss function is used to improve the feature representation ability of the model by introducing more different positive and negative sample pairs between the modalities; i and h i The contrast loss function l(i,z,h) is: When the mini-batch size is N, the cross-modal contrast loss function ι under mini-batch training is cmid Expressed as: The cross-modal contrast loss function ι under this mini-batch training cmid Represents the feature fusion relationship between the 3D point cloud and 2D spectrogram added in the feature space; Step 3C-3) The total loss function obtained during the training process is the intra-modality contrast loss ι imid and cross-modal contrast loss ι cmid combination of ι＝ι imid +ι cmid (10) Step 3C-4) After completing cross-modal contrastive learning, the acquired feature vector Using the mapping of multiple layers of fully connected layers, the final output of the human body key points is expressed as: Y=W2σ(W1f+b)1+b (11) In formula (1 1), Y is the coordinate of the key points of the human body, W1 and W2 are weight matrices, b1 and b2 are biases, and σ is the ReLU activation function.

Images