Human body posture estimation method based on Wi-Fi channel state information

Abstract

The invention provides a human body posture recognition method based on Wi-Fi channel state information, and the method comprises the steps: firstly constructing a wireless human body posture estimation data set, collecting CSI data and a human body posture video at the same time under different postures of a human body during the construction, and directly changing the last 32 bits of a hardware NIC timestamp into the last 32 bits of a system timestamp in a CSI data packet, and performing software time alignment. According to the invention, a human body posture key point prediction deep learning model based on pyramid cavity convolution and a residual network is further constructed; and when prediction is needed, to-be-predicted CSI data under any action of a person is obtained, and the model is input to obtain a human body posture action representation. According to the method, the human body posture recognition based on the Wi-Fi channel state information can be realized, and meanwhile, the problem of data synchronization is solved.

Description

Technical Field

[0001] The invention belongs to the field of wireless human body posture estimation, and in particular relates to a human body posture estimation method based on Wi-Fi channel state information. Background Art

[0002] With the rapid development of science and technology and the improvement of people's living standards, smart life has become an inevitable trend in modern society, in which human perception technology plays an important role, providing strong technical support for applications such as smart home and virtual reality. Traditional human behavior perception technology mostly relies on physical devices or wearable devices, which not only limits the user's range of activities, but also has many privacy concerns. Therefore, it is of great significance to develop a non-contact, high-precision human behavior perception technology.

[0003] In recent years, the technology of fine-grained human posture estimation based on Wi-Fi channel state information has attracted widespread attention. As a widely deployed wireless communication technology, Wi-Fi has the advantages of wide coverage, low cost, and easy access. By extracting features from Wi-Fi-based CSI signals (channel state information), rich environmental and human posture information can be obtained, thereby realizing human posture estimation. This technology overcomes some limitations of traditional human posture estimation technology, provides a new perspective for human behavior perception methods, and has great application development potential.

[0004] However, there are significant differences in signal characteristics between Wi-Fi signals and traditional dedicated wireless sensing devices. Traditional dedicated wireless sensing device signals are known for their large bandwidth and high resolution, while Wi-Fi signals are the opposite. They are limited by their small bandwidth and low spatial resolution, making it difficult to achieve high-accuracy and real-time human posture estimation. Summary of the invention

[0005] In view of this, the present invention proposes a human posture recognition method based on Wi-Fi channel state information. On the basis of a customized wireless human posture estimation dataset, a suitable model framework is established to obtain a deep learning model for predicting key points of human posture, thereby realizing human posture recognition based on Wi-Fi channel state information and solving the problem of data synchronization, thereby solving the problems of large model, slow speed and low accuracy of existing deep learning methods.

[0006] In order to solve the above technical problems, the present invention is implemented as follows.

[0007] A human body posture recognition method based on Wi-Fi channel state information, comprising:

[0008] Step 1: Establish a wireless human posture estimation dataset: Under different human postures, channel state information (CSI) data and human posture videos are collected simultaneously and time-aligned; human posture images are obtained from human posture videos, and the coordinates of human posture key points in the human posture images and their confidence are annotated to obtain label supervision data; CSI data and label supervision data constitute training samples, thereby establishing a wireless human posture estimation dataset;

[0009] The time alignment includes: during the data collection process, the video recording time is the system timestamp T1, the CSI data is collected by the open source measurement software and carried in the CSI data packet to be provided to the sender, which contains the hardware NIC timestamp information T2 when sending, and there is a difference between the system timestamp T1 and the hardware NIC timestamp T2; the underlying code of the open source measurement software for CSI data is modified, and the last 32 bits of the original hardware NIC timestamp in the CSI data packet sent by it are directly changed to the last 32 bits of the system timestamp, that is, the system timestamp offset information; the device that receives the CSI data packet extracts the timestamp from the CSI data, and the device calculates the time when the last 32 bits of the system timestamp closest to the current time is 0, and superimposes the last 32 bits of the system timestamp offset information in the CSI data packet to obtain the system time when the CSI data packet is sent;

[0010] Step 2: Establish and train a deep learning model for predicting key points of human posture: design a network model that maps CSI data to coordinates of key points of human posture, the network model including a pyramid dilated convolution and a residual network; input the wireless human posture estimation dataset into the network model for training;

[0011] Step 3: Human posture prediction: Obtain the CSI data to be predicted under any human action, input it into the human posture key point prediction deep learning model, and obtain the human posture key point sequence corresponding to the current CSI data; combine the human posture key point sequence according to the time dimension to obtain the human posture action representation.

[0012] Preferably, in step 1, different postures of the human body include postures of different body types when standing, walking, sitting, lying, bending, stretching and rotating.

[0013] Preferably, in step 1, after collecting the CSI data, CSI data packet loss processing is further performed, specifically:

[0014] Import the .dat file generated by the Linux 802.11n CSI tool measurement into Matlab software;

[0015] Determine the timestamp difference between two CSI data packets according to the sampling frequency;

[0016] According to the tolerable error range, based on the timestamp difference, a threshold m of the reception time difference of two CSI data packets under normal circumstances is determined;

[0017] Calculate the timestamp difference between the two CSI data packets actually received, and compare it with the threshold m. If it is greater than the threshold m, it is determined that there is packet loss in the data packet transmission.

[0018] The nearest neighbor interpolation method is used for data completion to fill the missing CSI data due to packet loss.

[0019] Preferably, the labeling of the coordinates of the key points of human posture in the human posture image and their confidences is: labeling the coordinates of the key points of human posture in the human posture image and their confidences using Alphapose.

[0020] Preferably, the deep learning model for predicting key points of human posture is set as follows: after CSI data is input into a dilated convolution with a dilation rate of {1, 2, 4, 8}, the obtained data is subjected to feature fusion in an additive manner, and then feature extraction, regularization and activation operations are performed, and the data is input into a 50-layer residual network, the number of residual blocks is {3, 4, 6, 3} in sequence, and the residual blocks mainly include convolution and activation operations; finally, a fully connected layer is added to realize the output of 34 key points;

[0021] The loss function indicator Loss in the deep learning model is the cumulative sum of the Euclidean distance confidence weighted between the predicted value and the true value of each human posture key point:

[0022]

[0023] Among them, C i is the confidence of the i-th human body pose key point in the training sample, pd i is the predicted value of the key point of the i-th human body posture, gt i is the true value of the key point of the i-th human body posture; The symbol in the figure indicates the Euclidean distance calculation.

[0024] 6. The method as claimed in claim 1 is characterized in that the human body posture key points include 17 key points, representing nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle respectively.

[0025] Beneficial effects:

[0026] (1) This method achieves the synchronization of CSI reception and human posture video recording by timestamp calculation, which solves the problem of being unable to control antenna transmission and camera simultaneously due to hardware limitations, ensures the correspondence between CSI data and human posture supervision information, and provides a reliable data basis for human posture estimation.

[0027] (2) The present invention is applied to the field of wireless human posture recognition, and uses a residual network model with a pyramid-shaped atrous convolution structure to perform supervised training on Wi-Fi signals. The pyramid-shaped atrous convolution structure can achieve multi-scale information fusion, increase the receptive field of the convolution layer without losing spatial resolution, and retain the rich spatial information and change information contained in the Wi-Fi signal. The use of the residual network can avoid the model degradation problem caused by the increase in network depth, has a good fitting effect, and achieves high accuracy and real-time human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of the algorithm processing of the embodiment of the present invention.

[0029] Figure 2 Schematic diagram of the change in data size during CSI data interpolation.

[0030] Figure 3 Schematic diagram of CSI data packet loss interpolation.

[0031] Figure 4 It is a flowchart of the data alignment process.

[0032] Figure 5 is a schematic diagram of the network structure in step 3 of the present invention; wherein Figure 5 (a) is a schematic diagram of a pyramid-shaped hole convolutional network. Figure 5 (b) is a schematic diagram of ResNet50. Figure 5 (c) is a schematic diagram of the overall flow of the network structure.

[0033] Figure 6 This is a diagram of the actual experimental scene.

[0034] Figure 7 This is a schematic diagram of the model training results. Figure 7 (a) and Figure 7 (b) shows two examples, where the left side shows the true value of human posture and the right side shows the predicted value of human posture.

[0035] Figure 8 Schematic diagram of 17 key points of human body posture. DETAILED DESCRIPTION

[0036] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0037] The purpose of the present invention is to solve the problems of large network scale and low accuracy in existing networks, and proposes a deep learning method for human posture recognition. Figure 1 : is an algorithm processing flow chart of an embodiment of the present invention. Figure 1 As shown, the present invention is implemented by the following steps:

[0038] Step 1: Modeling the impact of human motion on CSI signals

[0039] Channel state information (CSI) describes the attenuation factor of the signal on each transmission path, that is, the value of each element in the channel gain matrix H, such as signal scattering, environmental attenuation, distance attenuation and other information, which can reflect the characteristics of the wireless channel in a fine-grained manner.

[0040] In the OFDM system, the signal received by the receiver can be expressed as:

[0041] Y=HX+R

[0042] Where X and Y represent the transmitted and received signals respectively, H is the channel state information (CSI) matrix, and R is Gaussian white noise. Therefore, the CSI matrix can be approximately expressed as:

[0043]

[0044] In order to further accurately describe the characteristics of multiple signal propagation paths, H can be expressed as:

[0045]

[0046] where a n (t) is the amplitude attenuation factor, τ n (t) is the propagation delay, f is the carrier frequency, and t is the current time. The channel frequency response at a specific carrier frequency is the CSI signal.

[0047] Let N be the number of subcarriers, then at time i, the channel state information H(i, j) of the jth receiving antenna can be expressed as:

[0048] H(i,j)=[H(i,j,1),H(i,j,2),…,H(i,j,N)]

[0049] The channel state information of the kth subcarrier can be expressed as:

[0050]

[0051] in, It is the phase information, which reflects the phase offset during the transmission of wireless signals.

[0052] In a typical indoor space, wireless signals have multiple propagation paths. In addition to the line-of-sight propagation path (the main propagation path), there are also multiple reflection paths generated by other environmental factors such as walls and furniture. When someone enters the room, these movement factors will introduce additional propagation paths related to the human body.

[0053] In wireless communication systems, line-of-sight propagation signals are mainly affected by free space path loss. According to the free space model and Friss free space equation, when the signal propagates from the transmitting antenna to the receiving antenna at a distance d, the received power can be expressed as:

[0054]

[0055] Among them, P t is the transmission power, P r (d) is the received power, which is a function of the line-of-sight propagation distance d, G t is the transmitting antenna gain, G r is the receiving antenna gain, and λ is the wavelength of the electromagnetic wave. In addition to considering line-of-sight propagation, it is also necessary to consider the impact of other key reflection paths on the received power. After introducing the environmental factor reflection path, the received power can be expressed as:

[0056]

[0057] Where h is the distance from other reflection paths to the line-of-sight propagation path (LoS path).

[0058] When the human body is in an indoor environment, multiple scattered paths will be generated. These scattered powers will also be added to the final received power. The human body scattered power can be expressed as:

[0059]

[0060] where Δ is the additional path length caused by the human body.

[0061] If a person is stationary in the environment, P r The scattering path of the human body will change at a faster speed during a fall, which directly leads to a change in the received power.

[0062] The total received power of the receiver is the sum of the received powers from each propagation path. Assume that the strengths of the line-of-sight path, the indoor environment (floor, wall, etc.) reflection path, and the human body scattering path are ELoS 、E ref and E Sca , the total received power can be expressed as:

[0063] P∝|E LoS +E ref +E Sca | 2

[0064] In the above formula, when in a specific scenario, due to the invariance of the environment, E LoS and E ref It is generally a constant. When the human body is at rest, the component E Sca It is also basically in a constant state; when the human body is in motion, E Sca will change over time and can be used to represent human motion.

[0065] Wi-Fi signals can reach various parts of the human body, which will cause reflection and attenuation during the propagation process, thus affecting the CSI value. Furthermore, the changes in indoor communication links caused by the scattering paths introduced by the human body can be modeled: Under indoor multipath conditions, CSI can be expressed as:

[0066]

[0067] Among them, H s represents the channel characteristics of the static signal propagation path introduced by the indoor environment, the subscript l represents the static signal propagation path index, and L is the number of static signal propagation paths; H m represents the channel characteristics of the dynamic signal propagation path introduced by the moving human body, the subscript d represents the dynamic signal propagation path index, and D is the number of dynamic signal propagation paths. From the above definition and analysis, we can know that H s is the time-invariant information, and H m Contains human motion information and changes over time. H contains human motion information, which provides theoretical support for the model to extract human posture information from CSI signals.

[0068] Step 2: Data collection and preprocessing

[0069] In order to establish a wireless human posture estimation dataset, the present invention takes the following steps:

[0070] In the data collection phase, this solution uses a camera with a self-encoding function to capture the corresponding human posture video images while collecting CSI data. After obtaining the human posture estimation data, in order to ensure the synchronization relationship between the CSI data and the human posture image, it is necessary to process the CSI data and the human posture video data separately, so as to construct an accurate Wi-Fi wireless human posture estimation dataset.

[0071] For CSI data, since CSI signals are easily affected by multiple factors such as the environment and equipment, it is difficult for the receiving antenna to capture all CSI data values, resulting in data loss, abnormal values, etc. Therefore, it is necessary to preprocess the collected raw CSI data, including amplitude information extraction, nearest neighbor interpolation, Kalman filtering, and bilinear interpolation, to obtain accurate and reliable data.

[0072] Specifically, during the measurement process, each set of data contains 3000 data packets, and the actual received data is approximately between 2500 and 3000. In order to solve these problems, this solution first increases the CSI data volume to 3000 based on the nearest neighbor interpolation method of the time series. The Kalman filter is used to filter out the signal outliers, and the size of the CSI signal data obtained at this time is 3000×3×3×30. Due to the data volume requirements of deep learning, each large data is split into small data blocks of 10×3×3×30 based on Python processing, and the data size is changed to 300×3×3 after resizing transformation. In order to improve the smoothness of the CSI data and increase the data volume, the bilinear interpolation method is finally used to interpolate the data to a size of 300×136×136. The size of the CSI data changes during the processing as shown below: Figure 2 shown.

[0073] For human posture motion videos, this solution first uses the OpenCv-Python library to decompose the video by frame, obtains and uniformly samples the decomposed image set to meet the matching relationship with the CSI data. Then, based on Alphapose, the present invention annotates the key point coordinates and confidence of the human posture in the picture to replace the manually annotated data set, greatly improving the annotation rate and accuracy. The json file obtained after annotation is processed to extract the key points and confidence data corresponding to each picture, thereby obtaining complete label supervision data.

[0074] In the actual data processing process, packet loss and data alignment are problems that need to be solved:

[0075] (1) CSI data packet loss

[0076] In the process of data collection, due to the complexity of wireless network communication, factors such as channel interference, transmission errors, and hardware failures are often difficult to avoid. These factors may cause some CSI data packets to fail to successfully reach the receiving end or be completely recorded during the transmission process. For deep learning models, especially when it comes to building human posture estimation models, the fixed amount of data and the accuracy of data quality are particularly critical. Therefore, the integrity and accuracy of CSI data have a decisive impact on the effect of model training. Solving the problem of CSI data packet loss is crucial. The processing process is as follows: Figure 3 shown.

[0077] Since the .dat data structure obtained when collecting CSI does not have the function of recording the packet loss location, we need to determine the packet loss location by judging its sending time. In the data processing stage at the receiving end, this solution first imports the .dat file generated by the Linux 802.11nCSI Tool measurement into the Matlab software to further extract the amplitude information of CSI. The timestamp_low indicator in each CSI data packet is the timestamp record of the NIC card at the transmitting end, which is the last 32 bits of the network card time. Its standard data sampling rate is 10 6 That is to say, assuming that the sampling frequency is set to 100 Hz in this experimental scheme, the standard hardware timestamp difference between the two CSI data packets should be 10000.

[0078] Therefore, the time of the waveform horizontal axis can be determined by the physical quantity of the timestamp. Combined with the above standards, if the error is allowed, the difference between the timestamps of the two previous and subsequent data packets is calculated, and the variable threshold m is set to effectively identify whether there is packet loss in the data packet transmission. If packet loss is found, in order to ensure the continuity of the data sequence and the training quality of the model, this solution uses the nearest neighbor interpolation method to complete the data, so as to fill the CSI data missing due to packet loss. This method not only improves the integrity of the data, but also helps to improve the accuracy and stability of the subsequent human posture estimation model. Through this series of data processing measures, we provide more reliable and effective data support for the training of deep learning models.

[0079] (2) Alignment of CSI data and RGB images

[0080] In order to correctly map the human posture in the image to the human motion contained in the CSI data, CSI data collection and video recording should be started at the same time during data collection. To ensure the accuracy of this experiment, it can be solved based on hardware facilities and software processing respectively. The processing idea diagram is shown in the figure below. Figure 4 shown.

[0081] (1) Solution based on hardware equipment: During the data collection process, two computers are used to control the antenna transmission and reception respectively, and the hardware NIC time of the two computers is synchronized. A script is used on one of the computers to simultaneously control the antenna reception and camera operation to achieve data synchronization during the measurement phase;

[0082] (2) Software-based solution: Use timestamp calculation to synchronize CSI data and video frames. This method requires recording a reference point timestamp at the beginning of data acquisition, which is generally the default sending timestamp. Then, at the beginning of CSI data acquisition and video recording, each timestamp is recorded. These timestamps can be obtained from the system time or other accurate time sources.

[0083] After data acquisition and recording are completed, the CSI data and video images are stored in corresponding files, and timestamp information is included in the file naming or metadata to facilitate data alignment in the subsequent data processing stage. In the data processing stage, the CSI data and video image files are read and aligned according to the timestamp. The time offset between the CSI data and the video image can be determined by comparing the difference in timestamps, and then one of the data sources (such as CSI data or video frames) is corrected in time so that the two data sources can be aligned in time.

[0084] However, due to the limitation of hardware equipment, we were unable to control the antenna reception and the camera on one computer at the same time in this experiment. Therefore, the present invention uses software processing and timestamp calculation to synchronize CSI data and video frames.

[0085] When using software processing to solve the problem, you need to pay attention to the difference between hardware timestamp and software timestamp. Timestamp is the number of seconds (excluding leap seconds) that have passed since January 1, 1970 (midnight UTC / GMT), used to represent a time point. Initially, the hardware timestamp and software timestamp are not necessarily the same. The timestamp in the CSI data structure collected by the Linux 802.11n CSI Tool is the last 32 bits of the NIC network card time. The NIC hardware time is increased by 1 every 1 microsecond interval, and it is repeated approximately every 4300 seconds (72 minutes) (starting from 2 0 ~2 32 About 4300s). Therefore, the actual time of sending a data packet can be calculated from two aspects:

[0086] ①Use PTP precise time protocol

[0087] Precision Time Protocol (PTP) is a protocol used for clock synchronization in the network. When used in conjunction with hardware, PTP can achieve sub-microsecond accuracy. During experimental measurements, the PTP protocol can be used to achieve complete synchronization of software time (computer time) and NIC network card time, thereby directly reading the time information in the CSI data. However, it should be noted that the PTP protocol has high requirements for devices, and some classic network cards do not support this function, such as the Intel 5300 series used in this solution.

[0088] ② Improve software

[0089] Since Intel 5300 does not support the PTP synchronization protocol, this paper modifies the underlying code of Linux 802.11n CSI Tool and writes the corresponding software timestamp in a specific txt file while sending data packets, thereby skipping the software and hardware time synchronization and obtaining the CSI time information. After obtaining the CSI timestamp information, this solution uses a self-written program to calculate the last 32 bits of the software timestamp closest to the current time as 0 time, and superimposes the last 32 bits of the software timestamp in the CSI data to obtain the complete time information when the data packet is sent.

[0090] (3) Completeness of data set

[0091] The completeness of the Wi-Fi wireless human posture estimation dataset is crucial for solution verification and model design. A complete dataset needs to provide diverse, real-world samples that cover a variety of posture types, posture changes, and angles. Only in this way can the dataset help train a more robust and generalized model, enabling it to more accurately estimate the key points of human posture in practical applications. During the initial experimental phase of testing, the model output results were consistent and approximately the average value of the key points of each sample. After excluding the influence of related factors such as the algorithm, it is believed that the completeness of the dataset has an important impact on human posture estimation based on Wi-Fi signals.

[0092] The completeness of the data set is not only related to the training effect, but also to the test effect. For an incomplete data set, its test results are often not representative. This is because in the actual experimental process, the use of deep learning for wireless human posture estimation will inevitably learn the inherent characteristics of the data set itself. Taking key point estimation as an example, if the human behavior data contains too many hand movements and rarely contains running, jumping and other movements involving leg joints, the estimation accuracy of the hand joints will be much lower than that of the leg joints, and the output human posture image tends to keep one leg joint unchanged. Using such a data set for training and even testing is undoubtedly not objective, and its result indicators are not representative. Therefore, in the process of data set collection, special attention should be paid to the completeness of the data set.

[0093] A complete dataset should take into account both the scale of the dataset and the diversity of human postures, comprehensively cover the changes in different human postures, and posture samples of different individuals (tall, short, fat, thin). This will help improve the diversity and representativeness of the dataset and ensure that the dataset has a wider applicability in key point posture recognition.

[0094] Therefore, when constructing a data set, the present invention collects data in different postures of the human body, and the different postures at least include the postures of standing, walking, sitting, lying, bending, stretching and rotating for people of different heights, weights and body shapes.

[0095] Step 3: Establish a human posture prediction network model and train it;

[0096] Pyramid dilated convolution is a multi-scale convolution operation that can extract features within different receptive fields. By using pyramid dilated convolution to capture posture features at different scales and applying different dilation rates to the convolution layer, feature representations at different levels can be obtained, achieving effective capture of posture information from local to global.

[0097] Residual network is a method that uses skip connections to solve the gradient vanishing and gradient exploding problems in deep network training. The present invention introduces residual connections to construct a deep network structure. Residual connections can directly transfer the features of the previous layer to the subsequent layers, so that the network can better learn the fitting information, avoid the attenuation of information during the network transmission process, and help improve the network's convergence speed and model performance.

[0098] By comprehensively applying pyramid atrous convolution and residual network, the model can model posture features at different scales, and retains detail information and contextual relationships through residual connections, thereby improving the prediction accuracy of posture key points. In addition, according to the size and form of the input CSI data, the present invention improves the channel size and connection layer of the original network to adapt to different data features and model complexity, thereby improving the performance and applicability of the model. The network structure diagram is shown in Figure 5 shown.

[0099] Specifically, the CSI data is input into the dilation rate of {1, 2, 4, 8} after the hole convolution, the obtained data is added for feature fusion, and then simple feature extraction, regularization and activation operations are performed before input into the 50-layer residual network, where the number of residual blocks is {3, 4, 6, 3}, and the residual blocks mainly include convolution and activation operations. Finally, a fully connected layer is added to achieve the output of 34 key points.

[0100] When using AlphaPose to extract 17 key nodes in human pose estimation, not only the positions of these key points are obtained, but also the confidence scores of the corresponding key points can be obtained, which reflects the credibility of the AlphaPose algorithm's predicted value for a specific key point position.

[0101] Therefore, in order to accurately evaluate the difference between the human pose joints predicted by the quantization network and the actual pose and guide the model optimization direction, for single-person human pose estimation, this scheme defines a loss function indicator as shown in the following formula. The loss function indicator is the weighted cumulative sum of the Euclidean distance confidence between the predicted value and the actual value of each human pose key point. The loss contribution of each key point depends not only on the Euclidean distance between its predicted position and the actual position, but also on the confidence of the point.

[0102]

[0103] Among them, C i is the confidence of the key point of the i-th human body pose in the sample, pd i is the predicted value of the key point of the i-th human body. Similarly, gt i is the true value. The symbol is used to represent the calculation of Euclidean distance.

[0104] Through the above design principles and improvement measures, the human posture prediction model can effectively extract features from CSI data and accurately predict the positions of key points of human posture.

[0105] The training samples obtained in step 2 are input into the above-mentioned human posture prediction model and trained to obtain a trained prediction model.

[0106] Step 4: Human body posture prediction;

[0107] In the human posture prediction stage, the CSI signal to be predicted is obtained by collecting data of any human movement. The collected CSI signal is preprocessed and segmented so that it can be input into the human posture key point prediction model for analysis and inference.

[0108] The preprocessed signal is input into the human posture key point prediction model, and the human posture key point sequence corresponding to the CSI signal can be obtained through key point value regression analysis. Finally, these key point sequences are combined according to the time dimension to obtain a complete human posture action representation.

[0109] Through the above steps, the present invention constructs a wireless human posture estimation solution, uses a deep learning model to process and predict CSI data, and realizes accurate prediction of key points of human posture. The model combines pyramid-type dilated convolution and residual network, and optimizes the network channel size according to the characteristics of CSI data to improve the performance and accuracy of posture estimation.

[0110] In order to verify the human posture estimation method based on Wi-Fi channel state information proposed in this invention, data measurement is performed on the posture of a single person in an indoor scene. The scene layout is as follows: Figure 6 shown.

[0111] This embodiment is based on the Intel 5300 wireless network card equipped with three 5G antennas, Linux 802.11n CSI Tool and Ubuntu 14.04 LTS operating system for CSI data acquisition, and the human posture video acquisition based on the Hikvision 0s-E14a camera. The experiment uses a CSI data acquisition frequency of 100Hz, and the corresponding video frame rate is 10fps. Every 10 frames of CSI data correspond to a human posture key point supervision information. In the data processing process, it mainly includes filtering, interpolation and other related operations to improve the quality and accuracy of the data.

[0112] The wireless human posture estimation dataset contains 3.6w sets of valid data, of which 90% of the data is used as a training set for the model to learn and optimize the model parameters. The remaining 10% of the data is used as an independent test set to evaluate the performance and generalization ability of the model. The processed data set is input into the algorithm model for iteration, using the Adam optimizer with a learning rate of 0.001. Good results are obtained when the number of training times is 501. Two simple test prediction graphs are shown in the figure below. Figure 7 As shown in Figure (a) and Figure (b).

[0113] Taking into account the AlphaPose keypoint annotations, the inventors measured the algorithm performance using the percentage of correct keypoints (PCK) as an indicator based on a slight modification of the Coco dataset:

[0114]

[0115] Where P is the number of people in the test frame, i represents the index of the body joint, i∈{1, 2, .., 17), is the predicted value of the key point of the i-th human body in the P-th sample. Similarly, is the true value. p is the width of the human pose bounding box, h p is the height of the human pose bounding box. is the Euclidean pixel distance between the predicted value and the true value, and passes through the diagonal length of the person’s bounding box Normalize. Ⅱ represents the logic judgment operation. If the normalized error value is less than a, the judgment logic value is 1, otherwise it is 0. In order to obtain the bounding box of the person, the body joint coordinates of AlphaPose are aligned with the bounding box of the key points of the human body posture after drawing during the processing.

[0116] The human posture estimation implemented in this scheme has 17 key points, which are averagely segmented based on PCK, and the MPCK index is established as shown in the following formula:

[0117]

[0118] Using the existing test set for indicator evaluation, the PCK results are shown in the following table:

[0119]

[0120] Among them, the 17 key points represent the nose (Nose), left eye (Left Eye), right eye (Right Eye), left ear (Left Ear), right ear (Right Ear), left shoulder (Left Shoulder), right shoulder (Right Shoulder), left elbow (Left Elbow), right elbow (Right Elbow), left wrist (Left Wrist), right wrist (Right Wrist), left hip (Left Hip), right hip (Right Hip), left knee (Left Knee), right knee (Right Knee), left ankle (Left Ankle) and right ankle (Right Ankle), such as Figure 7 shown.

[0121] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A human body posture recognition method based on Wi-Fi channel state information, characterized in that: include: Step 1: Establish a wireless human posture estimation dataset: Under different human postures, simultaneously collect channel state information (CSI) data and human posture videos, and perform time alignment; Obtaining a human posture image from a human posture video, and annotating the coordinates of key human posture points in the human posture image and their confidences to obtain label supervision data; CSI data and label supervision data constitute training samples to establish a wireless human posture estimation dataset; The time alignment includes: during the data collection process, the video recording time is the system timestamp, the CSI data is collected by the open source measurement software and carried in the CSI data packet to be provided to the sender, which contains the hardware NIC timestamp information when sending, and there is a difference between the system timestamp and the hardware NIC timestamp; the open source measurement software underlying code of the CSI data is modified, and the last 32 bits of the original hardware NIC timestamp in the CSI data packet sent by it are directly changed to the last 32 bits of the system timestamp, that is, the system timestamp offset information; the device that receives the CSI data packet extracts the timestamp from the CSI data, and the device calculates the time when the last 32 bits of the system timestamp closest to the current time is 0, and superimposes the last 32 bits of the system timestamp offset information in the CSI data packet to obtain the system time when the CSI data packet is sent; Step 2: Establish and train a deep learning model for predicting key points of human posture: design a network model that maps CSI data to coordinates of key points of human posture, the network model including a pyramid dilated convolution and a residual network; input the wireless human posture estimation dataset into the network model for training; Step 3: Human posture prediction: Obtain the CSI data to be predicted under any human action, input it into the human posture key point prediction deep learning model, and obtain the human posture key point sequence corresponding to the current CSI data; combine the human posture key point sequence according to the time dimension to obtain the human posture action representation.

2. The method according to claim 1, characterized in that In step 1, different postures of the human body include postures of different body types when standing, walking, sitting, lying, bending, stretching and rotating.

3. The method according to claim 1, characterized in that In step 1, after collecting CSI data, CSI data packet loss processing is further performed, specifically: Import the .dat file generated by the Linux 802.11n CSI tool measurement into Matlab software; Determine the timestamp difference between two CSI data packets according to the sampling frequency; According to the tolerable error range, based on the timestamp difference, a threshold m of the reception time difference of two CSI data packets under normal circumstances is determined; Calculate the timestamp difference between the two CSI data packets actually received, and compare it with the threshold m. If it is greater than the threshold m, it is determined that there is packet loss in the data packet transmission. The nearest neighbor interpolation method is used for data completion to fill the missing CSI data due to packet loss.

4. The method according to claim 1, characterized in that The labeling of the coordinates of the key points of human posture in the human posture image and their confidences is as follows: labeling the coordinates of the key points of human posture in the human posture image and their confidences by using Alphapose.

5. The method according to claim 1, characterized in that The setting of the human posture key point prediction deep learning model is as follows: after the CSI data is input into the dilation rate {1, 2, 4, 8} of the hollow convolution, the obtained data is added to perform feature fusion, and then feature extraction, regularization and activation operations are performed, and input into a 50-layer residual network, the number of residual blocks is {3, 4, 6, 3}, and the residual block mainly includes convolution and activation operations; finally, a fully connected layer is added to realize the output of 34 key points; The loss function indicator Loss in the deep learning model is the cumulative sum of the Euclidean distance confidence weighted between the predicted value and the true value of each human posture key point: Among them, C i is the confidence of the i-th human pose key point in the training sample, pd i is the predicted value of the key point of the i-th human body posture, gt i is the true value of the key point of the i-th human body posture; The symbol in the figure indicates the Euclidean distance calculation.

6. The method according to claim 1, characterized in that The human body posture key points include 17 key points, representing nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle respectively.

Images