WiFi human body posture estimation method fusing channel attention and multi-branch learning
By using a multi-branch codec network and channel attention mechanism, the cross-modal variability problem in human pose estimation of WiFi signals is solved, achieving efficient and low-cost accurate human pose prediction, which is suitable for security management and privacy protection in various locations.
Patent Information
- Application Number
- CN202511044508.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-28
- Publication Date
- 2025-11-11
AI Technical Summary
Existing human pose estimation algorithms based on images and sensors perform poorly in low light and occlusion conditions, and sensors are expensive and infringe on privacy. Human pose estimation algorithms based on WiFi signals suffer from cross-modal differences, resulting in large prediction errors.
A multi-branch encoder-decoder network structure is adopted, combined with a channel attention mechanism, to extract and fuse features using channel state information data, and to perform joint loss calculation using the outputs of multiple decoder branches, thereby achieving accurate prediction of human skeleton information.
It improves the accuracy and efficiency of WiFi signal human posture estimation, reduces costs, and is suitable for security management and privacy protection in various locations.
Smart Images
Figure CN120929916A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human pose estimation, and more particularly to a WiFi signal human pose estimation algorithm based on channel attention mechanism and multi-branch encoding / decoding learning. Background Technology
[0002] Human pose estimation algorithms are widely used in daily life, playing a vital role in smart cities, smart homes, fall detection, and other scenarios. These algorithms come in various forms. Based on deep learning algorithms, they can predict the coordinates of the human skeleton's joints using images collected from surveillance cameras or information from sensors worn close to the body, further analyzing people's real-time movement performance.
[0003] In practice, image-based human pose estimation methods face significant challenges. They fail to function effectively in low light or when obstructed by objects. Furthermore, in certain locations, video surveillance poses a risk of infringing on the privacy of others. While sensor-based human pose estimation methods can better avoid the problems of light, obstructions, and personal privacy, the sensors required to be worn by humans and are expensive, resulting in limited adoption.
[0004] A human pose estimation algorithm based on WiFi signals utilizes channel state information data generated by changes in human movement between WiFi signal transmitters. With the support of a deep learning model algorithm, it can predict the skeletal joints of the human body between WiFi signal transmitters through training. In this WiFi-based method, WiFi signal transmission is not only unaffected by light or obstructions, effectively avoiding privacy issues, but it can also achieve specific functions using only a home WiFi router, featuring low cost and strong scalability. This technology has enormous application prospects, not only for security management in public places such as hospitals, office buildings, and schools, but also for private locations such as residences, and even for security monitoring in industrial production. Summary of the Invention
[0005] The WiFi human pose estimation method of this invention, which integrates channel attention and multi-branch learning, addresses the cross-modal differences between WiFi signal data and human pose information data. Specifically, directly predicting two-dimensional human skeleton information using one-dimensional WiFi signal data will result in significant errors. This invention employs a multi-branch encoder-decoder backbone network structure, utilizing a single-branch encoder to extract features of human pose information from WiFi signals, and integrating a channel attention mechanism to enhance the expressive power of key features. Feature reconstruction is performed through a multi-branch decoder, with mutual feature fusion and learning between decoder branches. During model training, the outputs of multiple decoder branches are used for joint loss calculation. During prediction, the decoder branch with the most complete feature fusion is used as the final output, thereby significantly improving algorithm performance.
[0006] The technical solution of this invention is:
[0007] This invention provides a WiFi human pose estimation method that integrates channel attention and multi-branch learning, the method comprising the following steps:
[0008] Step 1: Process the channel state information data data0, which changes due to human movement between WiFi transceivers, into a tensor format of N×S×T1×T2, where N is the number of sequences, S is the number of WiFi signal carriers, T1 is the number of antennas of the WiFi signal transmitter, and T2 is the number of antennas of the WiFi signal receiver.
[0009] data0 = {N × S × T1 × T2}
[0010] Step 2: Input the original channel state information data data0 into the convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The input channels of this convolutional layer are N×S and the output channels are C, which performs dimensionality reduction on the channel state information data; then, the generalization ability and nonlinearity of the network are enhanced by the batch normalization layer BacthNorm and the Mish activation function layer.
[0011] data0 = Mish(BatchNorm(Conv) 1×1 (data0)))Mish=x*tanh(ln(1+e x ))
[0012]
[0013] data0 = {C × T1 × T2}
[0014] Step 3: Use the deconvolutional network DeConv to upsample the channel state information data to expand the first and second dimensions of the data. T1×T2 is converted into H and W, where H and W represent the height and width of the tensor and are of equal size. Then, through the batch normalization layer BacthNorm and the Mish activation function layer, the preprocessed data data1 is obtained.
[0015] data1=Mish(BatchNorm(DeConv(data0)))
[0016] data1 = {C × H × W}
[0017] Step 4: Input the upsampled channel state information data1 into the backbone network, which includes one encoder branch (Encoder) and two decoder branches (Decoder). ass And the final decoder branch, Decoder, and the encoder and decoder have the same number of layers and similar structures;
[0018] Backbone = {Encoder, Decoder} ass Decoder}
[0019] Step 5: Input the channel state information data data1 into the encoder branch Encoder. The encoder branch Encoder consists of multiple layers, each with a similar basic block structure, and the output data of each layer of the encoder branch is {En1, En2, ..., En...} n}, where En i In this context, 'i' represents the depth of the hierarchy.
[0020] Step 6: Output the last layer of the encoder branch Encoder. n Input to the auxiliary decoder branch Decoder ass In the middle, and retain the output of each layer of the encoder {En1, En2, ..., En...} n} is used to assist the feature fusion and output of each layer of the decoder, and the output data of each layer of the auxiliary decoder branch is {De_ass}. n De_ass n-1 ,…,De_ass1}, where De_ass i In this context, 'i' represents the hierarchy depth, which corresponds to the decoder branch hierarchy depth and sequence number.
[0021] Step 7: Output the last layer of the encoder branch Encoder. n The input is fed into the final decoder branch, Decoder, while retaining the output of each layer of the auxiliary decoder, {De_ass}.n De_ass n-1 ,…,De_ass1}, used for feature fusion and output of each layer of the final decoder;
[0022] Step 8: For the outputs of the auxiliary decoder De_ass1 and the decoder De1, perform global averaging layer AdaptiveAvg and 1×1 convolutional kernel convolutional network layer operation Conv 1×1 The resolution of the input tensor is converted into the resolution of the target output tensor, which is 2×18×18, where 18 is the number of human skeleton joints and 2 represents the x and y coordinates of the joints, thus obtaining the final predictions Pre1 and Pre2 of the human skeleton joints.
[0023] Pre1 = Mish(BatchNorm(Conv) 1×1 (AdaptiveAvg(De_ass1))))
[0024] Pre2 = Mish(BatchNorm(Conv) 1×1 (AdaptiveAvg(De1))))
[0025] Pre1 = {2 × 18 × 18}
[0026] Step 9: During the training phase, the network's predictions Pre1 and Pre2 are used to calculate the loss. During the prediction phase, only the output Pre2 of the final decoder branch is used as the prediction result.
[0027] Loss=(||Pd x -Gt x ||+||Pd y -Gt y ||)
[0028] Where Pd x and Gt x It predicts the x-coordinate values corresponding to the key points and label key points, Pd y and Gt y is the predicted y-coordinate value of the key point and the label key point, and ‖‖ is the squared difference function.
[0029] Furthermore, the specific implementation method of the Channel_attn module is as follows:
[0030] The channel attention module employs a squeeze-and-excitation mechanism. First, it performs a global adaptive average pooling (AdaptiveAvg2dPool) on the input feature map to compress spatial dimension information into channel descriptors. Then, it learns the interdependencies between channels through a combination of two linear layers and activation functions ReLU and Sigmoid to generate channel attention weights. Finally, it performs element-wise multiplication with the original feature map to weight the importance of features from different channels.
[0031] Channel_attn = x * Sigmoid(Linear(ReLu(Linear(AdaptiveAvg2dPool(x))))) The first linear layer reduces the number of channels to 1 / 16 of the original number, and the second linear layer restores the original number of channels. This design enhances the feature representation capability of the model while maintaining computational efficiency.
[0032] Furthermore, in step 5, the encoder branch performs specific feature extraction, and the specific method for data processing at each layer is as follows:
[0033] Step 5.1: The encoder base block includes a multi-scale convolution module and residual connections;
[0034] Step 5.2, transfer the data En i The input is fed into the encoder base block with a resolution of C×H×W. It first passes through 3 parallel convolutional networks with kernel sizes of 1×1, 3×3, and S×5, stride of 1, and padding of 1, 2, and 3, resulting in 3 tensors with the same resolution of C×H×W. Then, through the linking operation Cat, the tensors are linked sequentially according to the channel dimension to obtain tensor data with a resolution of 3C×H×W.
[0035] En i_1×1 =Mish(BatchNorm(Conv) 1×1 (En i )))
[0036] En i_3×3 =Mish(BatchNorm(Conv) 3×3 (En i )))
[0037] En i_5×5 =Mish(BatchNorm(Conv) 5×5 (En i )))
[0038] En i_mid =Cat(En i_1×1En i_3×3 En i_5×5 )
[0039] En i_mid ={3C×H×W}
[0040] Step 5.3: Then pass through a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The input channel is 3C, the output channel is C, the step size is 1, and the padding is 0, resulting in a tensor with a resolution of C×H×W. This tensor is then compared with the encoder's input En. i Perform pixel addition to obtain the final output En. i+1 ;
[0041] En i+1 =Mish(BatchNorm(Conv) 1×1 (En i_m id)))
[0042] En i+1 ={C×H×W}
[0043] En i+1 =En i +En i+1
[0044] Step 5.4: For the links between each layer of the encoder, convert the output En of the decoder base block of the previous layer... i With a resolution of C×H×W, a convolutional network layer Conv with a 2x2 kernel is first constructed. 2×2 The step size is 2, and the padding is 0, resulting in data with the first and second dimensions halved, with a resolution of [resolution value missing]. In the Conv convolutional network layer with a 1x1 kernel 1×1 With a step size of 1 and padding of 0, the data with doubled channel dimensions is obtained, which becomes the input data for the basic block of the next layer decoder, with a resolution of [resolution missing].
[0045] En i+1 =Mish(BatchNorm(Conv) 2×2 (En i+1 )))
[0046]
[0047] En i+1 =Mish(BatchNorm(Conv) 2×2 (En i+1 )))
[0048]
[0049] Furthermore, in step 6, the auxiliary decoder branch performs specific feature restoration, and the specific method for data processing at each layer is as follows:
[0050] Step 6.1: In the auxiliary decoder basic block, the structure is similar to that of the encoder basic block, with multi-scale convolutional modules and residual connections, plus a 1×1 convolutional kernel convolutional layer network Conv 1×1 A combination of BatchNorm and the activation function Mish is used to correct the resolution, ensuring that the data resolution is the same as that output by the corresponding encoder level.
[0051] Step 6.2, Output of the auxiliary decoder base block De_ass i , and the corresponding encoder output En for each layer i Same resolution, En i First, channel weights are assigned using the Channel_attn module to obtain the corresponding layer feature data for the decoder that extracts channel features. Then, based on the channel dimension, De_ass i With En i Perform the connection operation Cat, and attach a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The channel dimensionality is reduced, and then batch normalization (BatchNorm) and activation function (Mish) are applied to obtain the final output De_ass of each layer of the auxiliary decoder. i ;
[0052] Channel_attn
[0053] =x*Sigmoid(Linear(ReLu(Linear(AdaptiveAvg2dPool(x)))))
[0054] ReLU = max(0, x)
[0055]
[0056] En i =Channel_attn(En i )
[0057] De_ass i =Cat(De_ass i En i )
[0058] De_ass i =Mish(BatchNorm(Conv) 1×1 (De_ass i )))
[0059] Where Linear() represents a linear layer;
[0060] Step 6.3: The interconnection of each layer in the auxiliary decoder is a bottom-up operation, using a deconvolutional network layer DeConv with 2×2 convolutional kernels. 2×2 With a step size of 2 and padding of 0, we obtain feature data with doubled dimensions H and W, ensuring that it has the same resolution as the output data of the corresponding encoder layer. That is, for the output data De_ass of the i-th layer of the auxiliary decoder branch... i The resolution is {C×H×W}, and the input data De_ass of the (i-1)th layer is... i-1 The resolution is {C×2H×2W}, and the corresponding encoder branch has a data resolution of {C×2H×2W} for the (i-1)th layer.
[0061] De_ass i-1 =DeConv 2×2 (De_ass i )
[0062] De_ass i =(C×H×W}
[0063] De_ass i-1 ={C×2H×2W}
[0064] En i-1 ={C×2H×2W}
[0065] Furthermore, in step 7, the final decoder branch performs specific feature restoration, and the specific method for data processing at each layer is as follows:
[0066] Step 7.1: In the final decoder base block, the structure is similar to that of the encoder and auxiliary decoder base blocks, featuring multi-scale convolutional modules and residual connections, along with a 1×1 convolutional kernel convolutional layer network Conv. 1×1 The combination of BatchNorm and the activation function Mish is used to correct the resolution, ensuring that the data resolution is the same as that output by the corresponding auxiliary decoder layer.
[0067] Step 7.2, the final decoder base block output De i The corresponding auxiliary decoder outputs De_ass at each layer. i Same resolution, De_ass i First, channel weights are assigned using the Channel_attn module to obtain the corresponding layer feature data for the decoder that extracts channel features. Then, based on the channel dimension, De... i With De_ass iPerform the connection operation Cat, and attach a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The channel dimensionality is reduced, and then batch normalization (BatchNorm) and activation function (Mish) are applied to obtain the final output De of each layer of the auxiliary decoder. i ;
[0068] Step 7.3: The interconnection of each layer in the final decoder is a bottom-up operation, using a deconvolutional network layer DeConv with 2×2 convolutional kernels. 2×2 With a step size of 2 and padding of 0, we obtain feature data with doubled dimensions H and W, ensuring that the resolution of the output data is the same as that of the corresponding auxiliary decoder layer.
[0069] De i-1 =DeConv z×2 (De i )
[0070] De i ={C×H×W}
[0071] De i-1 ={C×2H×2W}
[0072] De_ass i-1 ={C×2H×2W}
[0073] The beneficial effects of this invention are:
[0074] The advantages of this invention compared to the prior art are as follows: This invention uses supervised enhancement based on channel attention and multi-branch learning for data prediction, which can more effectively utilize the potential information in WiFi signals, thereby more efficiently predicting human posture information data.
[0075] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description
[0076] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.
[0077] Figure 1 This is a flowchart of a WiFi human pose estimation method that integrates channel attention and multi-branch learning.
[0078] Figure 2 It is a backbone network diagram of a multi-branch learning encoder-decoder.
[0079] Figure 3It is a diagram of the basic blocks for multi-scale residual processing and its channel attention linking operation network. Detailed Implementation
[0080] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0081] like Figure 1 As shown, this invention provides a WiFi human pose estimation method that integrates channel attention and multi-branch learning. The method includes the following steps:
[0082] Step 1: Process the channel state information data data0, which changes due to human movement between WiFi transceivers, into a tensor format of N×S×T1×T2, where N is the number of sequences, S is the number of WiFi signal carriers, T1 is the number of antennas of the WiFi signal transmitter, and T2 is the number of antennas of the WiFi signal receiver.
[0083] data0 = {N × S × T1 × T2}
[0084] Step 2: Input the original channel state information data data0 into the convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The input channels of this convolutional layer are N×S and the output channels are C, which performs dimensionality reduction on the channel state information data; then, the generalization ability and nonlinearity of the network are enhanced by the batch normalization layer BacthNorm and the Mish activation function layer.
[0085] data0 = Mish(BatchNorm(Conv) 1×1 (data0)))
[0086] Mish=x*tanh(ln(1+e x ))
[0087]
[0088] data0 = {C × T1 × T2}
[0089] Step 3: Use the deconvolutional network DeConv to upsample the channel state information data to expand the first and second dimensions of the data. T1×T2 is converted into H and W, where H and W represent the height and width of the tensor and are of equal size. Then, through the batch normalization layer BacthNorm and the Mish activation function layer, the preprocessed data data1 is obtained.
[0090] data1=Mish(BatchNorm(DeConv(data0)))
[0091] data1 = {C × H × W}
[0092] Step 4: Input the upsampled channel state information data1 into the backbone network, such as... Figure 2 As shown, the backbone network includes one encoder branch (Encoder) and two decoder branches, namely, auxiliary decoder branches (Decoder). ass And the final decoder branch, Decoder, and the encoder and decoder have the same number of layers and similar structures;
[0093] Backbone = {Encoder, Decoder} ass, Decoder}
[0094] Step 5: Input the channel state information data data1 into the encoder branch Encoder. The encoder branch Encoder consists of multiple layers, each with a similar basic block structure, such as... Figure 3 As shown, the output data of each layer of the encoder branch is {En1, En2, ..., En...} n}, where En i In this context, 'i' represents the depth of the hierarchy.
[0095] Step 6: Output the last layer of the encoder branch Encoder. n Input to the auxiliary decoder branch Decoder ass In the middle, and retain the output of each layer of the encoder {En1, En2, ..., En... n} is used to assist the feature fusion and output of each layer of the decoder, and the output data of each layer of the auxiliary decoder branch is {De_ass}. n De_ass n-1 ,…,De_ass1}, where De_ass i In this context, 'i' represents the hierarchy depth, which corresponds to the decoder branch hierarchy depth and sequence number.
[0096] Step 7: Output the last layer of the encoder branch Encoder. n The input is fed into the final decoder branch, Decoder, while retaining the output of each layer of the auxiliary decoder, {De_ass}. n De_ass n-1 ,...,De_ass1}, used for feature fusion and output of each layer of the final decoder, such as Figure 3 As shown;
[0097] Step 8: For the outputs of the auxiliary decoder De_ass1 and the decoder De1, perform global averaging layer AdaptiveAvg and 1×1 convolutional kernel convolutional network layer operation Conv 1×1 The resolution of the input tensor is converted into the resolution of the target output tensor, which is 2×18×18, where 18 is the number of human skeleton joints and 2 represents the x and y coordinates of the joints, thus obtaining the final predictions Pre1 and Pre2 of the human skeleton joints.
[0098] Pre1 = Mish(BatchNorm(Conv) 1×1 (AdaptiveAvg(De_ass1))))
[0099] Pre2 = Mish(BatchNorm(Conv) 1×1 (AdaptiveAvg(De1))))
[0100] Pre1 = {2 × 18 × 18}
[0101] Step 9: During the training phase, the network's predictions Pre1 and Pre2 are used to calculate the loss. During the prediction phase, only the output Pre2 of the final decoder branch is used as the prediction result.
[0102] Loss=(||Pd x -Gt x ||+||Pd y -Gt y ||)
[0103] Where Pd x and Gt x It predicts the x-coordinate values corresponding to the key points and label key points, Pd y and Gt y is the predicted y-coordinate value of the key point and the label key point, and ‖‖ is the squared difference function.
[0104] Furthermore, the specific implementation method of the Channel_attn module is as follows:
[0105] The channel attention module employs a squeeze-and-excitation mechanism. First, it performs a global adaptive average pooling (AdaptiveAvg2dPool) on the input feature map to compress spatial dimension information into channel descriptors. Then, it learns the interdependencies between channels through a combination of two linear layers and activation functions ReLU and Sigmoid to generate channel attention weights. Finally, it performs element-wise multiplication with the original feature map to weight the importance of features from different channels.
[0106] Channel_attn=x*Sigmoid(Linear(ReLu(Linear(AdaptiveAvg2dPool(x)))))
[0107] The first linear layer reduces the number of channels to 1 / 16 of the original number, while the second linear layer restores the original number of channels. This design enhances the model's feature representation ability while maintaining computational efficiency.
[0108] Furthermore, in step 5, the encoder branch performs specific feature extraction, and the specific method for data processing at each layer is as follows:
[0109] Step 5.1: The encoder base block includes a multi-scale convolution module and residual connections;
[0110] Step 5.2, transfer the data En i The input is fed into the encoder base block with a resolution of C×H×W. It first passes through three parallel convolutional networks with kernel sizes of 1×1, 3×3, and 5×5, strides of 1, and padding of 0, 1, and 2, resulting in three tensors with the same resolution of C×H×W. Then, through the linking operation Cat, the tensors are linked sequentially according to the channel dimension to obtain tensor data with a resolution of 3C×H×W.
[0111] En i1×1 =Mish(BatchNorm(Conv) 1×1 (En i )))
[0112] En i_3×3 =Mish(BatchNorm(Conv) 3×3 (En i )))
[0113] En i_5×5 =Mish(BatchNorm(Conv) 5×5 (En i )))
[0114] En i_mid =Cat(En i_1×1 En i_3×3 En i_5×5 )
[0115] En i_mid ={3C×H×W}
[0116] Step 5.3: Then pass through a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1The input channel is 3C, the output channel is C, the step size is 1, and the padding is 0, resulting in a tensor with a resolution of C×H×W. This tensor is then compared with the encoder's input En. i Perform pixel addition to obtain the final output En. i+1 ;
[0117] En i+1 =Mish(BatchNorm(Conv) 1×1 (En i_mid )))
[0118] En i+1 ={C×H×W}
[0119] En i+1 =En i +En i+1
[0120] Step 5.4: For the links between each layer of the encoder, convert the output En of the decoder base block of the previous layer... i With a resolution of C×H×W, a convolutional network layer Conv with a 2x2 kernel is first constructed. 2×2 The step size is 2, and the padding is 0, resulting in data with the first and second dimensions halved, at a resolution of [resolution value missing]. In the Conv convolutional network layer with a 1x1 kernel 1×1 With a step size of 1 and padding of 0, the data with doubled channel dimensions is obtained, which becomes the input data for the basic block of the next layer decoder, with a resolution of [resolution missing].
[0121] En i+1 =Mish(BatchNorm(Conv) 2×2 (En i+1 )))
[0122]
[0123] En i+1 =Mish(BatchNorm(Conv) 2×2 (En i+1 )))
[0124]
[0125] Furthermore, in step 6, the auxiliary decoder branch performs specific feature restoration, and the specific method for data processing at each layer is as follows:
[0126] Step 6.1: In the auxiliary decoder basic block, the structure is similar to that of the encoder basic block, with multi-scale convolutional modules and residual connections, plus a 1×1 convolutional kernel convolutional layer network Conv 1×1 A combination of BatchNorm and the activation function Mish is used to correct the resolution, ensuring that the data resolution is the same as that output by the corresponding encoder level.
[0127] Step 6.2, Output of the auxiliary decoder base block De_ass i , and the corresponding encoder output En for each layer i Same resolution, En i First, channel weights are assigned using the Channel_attn module to obtain the corresponding layer feature data for the decoder that extracts channel features. Then, based on the channel dimension, De_ass i With En i Perform the connection operation Cat, and attach a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The channel dimensionality is reduced, and then batch normalization (BatchNorm) and activation function (Mish) are applied to obtain the final output De_ass of each layer of the auxiliary decoder. i ;
[0128] Channel_attn
[0129] =x*Sigmoid(Linear(ReLu(Linear(AdaptiiveAvg2dPool(x)))))
[0130] ReLU = max(0, x)
[0131]
[0132] En i =Channel_attn(En i )
[0133] De_ass i =Cat(De_ass i En i )
[0134] De_ass i =Mish(BatchNorm(Conv) 1×1 (De_ass i )))
[0135] Where LinearO is a linear layer;
[0136] Step 6.3: The interconnection of each layer in the auxiliary decoder is a bottom-up operation, using a deconvolutional network layer DeConv with 2×2 convolutional kernels. 2×2 With a step size of 2 and padding of 0, we obtain feature data with doubled dimensions H and W, ensuring that it has the same resolution as the output data of the corresponding encoder layer. That is, for the output data De_ass of the i-th layer of the auxiliary decoder branch... i The resolution is {C×H×W}, and the input data De_ass of the (i-1)th layer is... i-1 The resolution is {C×2H×2W}, and the corresponding encoder branch has a data resolution of {C×2H×2W} for the (i-1)th layer.
[0137] De_ass i-1 =DeConv 2×2 (De_ass i )
[0138] De_ass i ={C×H×W}
[0139] De_ass i-1 ={C×2H×2W}
[0140] En i-1 ={C×2H×2W}
[0141] Furthermore, in step 7, the final decoder branch performs specific feature restoration, and the specific method for data processing at each layer is as follows:
[0142] Step 7.1: In the final decoder base block, the structure is similar to that of the encoder and auxiliary decoder base blocks, featuring multi-scale convolutional modules and residual connections, along with a 1×1 convolutional kernel convolutional layer network Conv. 1×1 The combination of BatchNorm and the activation function Mish is used to correct the resolution, ensuring that the data resolution is the same as that output by the corresponding auxiliary decoder layer.
[0143] Step 7.2, the final decoder base block output De i The corresponding auxiliary decoder outputs De_ass at each layer. i Same resolution, De_ass i First, channel weights are assigned using the Channel_attn module to obtain the corresponding layer feature data for the decoder that extracts channel features. Then, based on the channel dimension, De... i With De_ass i Perform the connection operation Cat, and attach a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1The channel dimensionality is reduced, and then batch normalization (BatchNorm) and activation function (Mish) are applied to obtain the final output De of each layer of the auxiliary decoder. i ;
[0144] Step 7.3: The interconnection of each layer in the final decoder is a bottom-up operation, using a deconvolutional network layer DeConv with 2×2 convolutional kernels. 2×2 With a step size of 2 and padding of 0, we obtain feature data with doubled dimensions H and W, ensuring that the resolution of the output data is the same as that of the corresponding auxiliary decoder layer.
[0145] De i-1 =DeConv 2×2 (De i )
[0146] De i ={C×H×W}
[0147] De i-1 ={C×2H×2W}
[0148] De_ass i-1 ={C×2H×2W}.
Claims
1. A WiFi human pose estimation method integrating channel attention and multi-branch learning, characterized in that, The method includes the following steps: Step 1: Process the channel state information data data0, which changes due to human movement between WiFi transceivers, into a tensor format of N×S×T1×T2, where N is the number of sequences, S is the number of WiFi signal carriers, T1 is the number of antennas of the WiFi signal transmitter, and T2 is the number of antennas of the WiFi signal receiver. data0 = {N × S × T1 × T2} Step 2: Input the original channel state information data data0 into the convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The input channels of this convolutional layer are N×S and the output channels are C, which performs dimensionality reduction on the channel state information data; then, the generalization ability and nonlinearity of the network are enhanced by the batch normalization layer BacthNorm and the Mish activation function layer. data0=Mish(BatchNorm(Conv 1×1 (data0))) Mish=x tanh(ln(1+e x )) data0 = {C × T1 × T2} Step 3: Use the deconvolutional network DeConv to perform upsampling operations on the channel state information data to expand the first and second dimension data. T1×T2 is converted into H and W, where H and W represent the height and width of the tensor and are of equal size. Then, through the batch normalization layer BacthNorm and the Mish activation function layer, the preprocessed data data1 is obtained. data1=Mish(BatchNorm(DeConv(data0))) data1 = {C × H × W} Step 4: Input the upsampled channel state information data1 into the backbone network, which includes one encoder branch (Encoder) and two decoder branches (Decoder). ass And the final decoder branch, Decoder, and the encoder and decoder have the same number of layers and similar structures; Backbone={Encoder,Decoder ass ,Decoder} Step 5: Input the channel state information data data1 into the encoder branch Encoder. The encoder branch Encoder consists of multiple layers, each with a similar basic block structure, and the output data of each layer of the encoder branch is {En1, En2, ..., En...} n }, where En i In this context, 'i' represents the depth of the hierarchy. Step 6: Output the last layer of the encoder branch Encoder. n Input to the auxiliary decoder branch Decoder ass In the middle, and retain the output of each layer of the encoder {En1, En2, ..., En... n } is used to assist the feature fusion and output of each layer of the decoder. During feature fusion, the output En_(i) of the corresponding layer of the encoder is weighted through the channel attention module Channel_attn to enhance the expressive power of key features. The output data of each layer of the auxiliary decoder branch is {De_ass n De_ass n-1 ,...,De_ass1}, where De_ass i In this context, 'i' represents the hierarchy depth, which corresponds to the decoder branch hierarchy depth and sequence number. Step 7: Output the last layer of the encoder branch Encoder. n The input is fed into the final decoder branch, Decoder, while retaining the output of each layer of the auxiliary decoder, {De_ass}. n De_ass n-1 , ...,De_ass1}, are used for feature fusion and output of each layer of the final decoder, and during feature fusion, the output De_ass of the corresponding layer of the auxiliary decoder is fused. i Channel weights are assigned through the Channel_attn module to achieve effective feature fusion between multiple branches; Step 8: For the outputs of the auxiliary decoder De_ass1 and the decoder De1, perform global averaging layer AdaptiveAvg and 1×1 convolutional kernel convolutional network layer operation Conv 1×1 The resolution of the input tensor is converted into the resolution of the target output tensor, which is 2×18×18, where 18 is the number of human skeleton joints and 2 represents the x and y coordinates of the joints, thus obtaining the final predictions Pre1 and Pre2 of the human skeleton joints. Pre1=Mish(BatchNorm(Conv 1×1 (AdaptiveAvg(De_ass1)))) Pre2=Mish(BatchNorm(Conv 1×1 (AdaptiveAvg(De1)))) Pre1 = {2 × 18 × 18} Step 9: During the training phase, the network's predictions Pre1 and Pre2 are used to calculate the loss. During the prediction phase, only the output Pre2 of the final decoder branch is used as the prediction result. Loss=(Pd x -Gt x +Pd y -Gt y ) Where Pd x and Gt x It predicts the x-coordinate values corresponding to the key points and label key points, Pd y and Gt y It is the predicted y-coordinate value of the key point and the label key point, and it is the squared difference function.
2. The WiFi human pose estimation method integrating channel attention and multi-branch learning according to claim 1, characterized in that... The feature extraction process for each layer of the single-branch encoder in step 5 is as follows: Step 5.1: The encoder base block includes a multi-scale convolution module and residual connections; Step 5.2, transfer the data En i The input is fed into the encoder base block with a resolution of C×H×W. It first passes through three parallel convolutional networks with kernel sizes of 1×1, 3×3, and 5×5, stride of 1, and padding of 0, 1, and 2, resulting in three tensors with the same resolution of C×H×W. Then, through the linking operation Cat, the tensors are linked sequentially according to the channel dimension to obtain tensor data with a resolution of 3C×H×W. One i_1×1 =Mish(BatchNorm(Conv 1×1 (One i ))) One i_3×3 =Mish(BatchNorm(Conv 3×3 (One i ))) UN i_5×5 =Mish(BatchNorm(Conv 5×5 (One i ))) And i_mid =Cat(En i_1×1 ,And i_3×3 ,And i_5×5 ) In i_mid ={3C×H×W} Step 5.3: Then pass through a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The input channel is 3C, the output channel is C, the step size is 1, and the padding is 0, resulting in a tensor with a resolution of C×H×W. This tensor is then compared with the encoder's input En. i Perform pixel addition to obtain the final output En. i+1 ; One i+1 =Mish(BatchNorm(Conv 1×1 (One i_mid ))) In i+1 ={C×H×W} And i+1 =And i +And i+1 Step 5.4: For the links between each layer of the encoder, convert the output En of the decoder base block of the previous layer... i With a resolution of C×H×W, a convolutional network layer Conv with a 2x2 kernel is first constructed. 2×2 The step size is 2, and the padding is 0, resulting in data with the first and second dimensions halved, with a resolution of [resolution value missing]. In the Conv convolutional network layer with a 1x1 kernel 1×1 With a step size of 1 and padding of 0, the data with doubled channel dimensions is obtained, which becomes the input data for the basic block of the next layer decoder, with a resolution of [resolution missing]. One i+1 =Mish(BatchNorm(Conv 2×2 (One i+1 ))) One i+1 =Mish(BatchNorm(Conv 2×2 (One i+1 ))) 3. The WiFi human pose estimation method integrating channel attention and multi-branch learning according to claim 1, characterized in that... The network flow for restoring the features of each layer of the auxiliary decoder and fusing them with the corresponding layer features of the encoder in step 6 is as follows: Step 6.1: In the auxiliary decoder basic block, the structure is similar to that of the encoder basic block, with multi-scale convolutional modules and residual connections, plus a 1×1 convolutional kernel convolutional layer network Conv 1×1 A combination of BatchNorm and the activation function Mish is used to correct the resolution, ensuring that the data resolution is the same as that output by the corresponding encoder level. Step 6.2, Output of the auxiliary decoder base block De_ass i , and the corresponding encoder output En for each layer i Same resolution, En i First, channel weights are assigned using the Channel_attn module to obtain the corresponding layer feature data for the decoder that extracts channel features. Then, based on the channel dimension, De_ass i With En i Perform the connection operation Cat, and attach a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The channel dimensionality is reduced, and then batch normalization (BatchNorm) and activation function (Mish) are applied to obtain the final output De_ass of each layer of the auxiliary decoder. i ; Channel_attn =x Sigmoid(Linear(ReLu(Linear(AdaptiveAvg2dPool(x))))) ReLU = max(0,x) And i =Channel_attn(En i ) That's it i =Cat(The_ass i ,And i ) That's it i =Mish(BatchNorm(Conv 1×1 (That is i ))) Where Linear() represents a linear layer; Step 6.3: The interconnection of each layer in the auxiliary decoder is a bottom-up operation, using a deconvolutional network layer DeConv with 2×2 convolutional kernels. 2×2 With a step size of 2 and padding of 0, we obtain feature data with doubled dimensions H and W, ensuring that it has the same resolution as the output data of the corresponding encoder layer. That is, for the output data De_ass of the i-th layer of the auxiliary decoder branch... i The resolution is {C×H×W}, and the resolution of the input data De_assi-1 of the (i-1)th layer is {C×2H×2W}, and the resolution of the corresponding encoder branch data of the (i-1)th layer is {C×2H×2W}. That's it i-1 =DeConv 2×2 (That is i ) That's it i ={C×H×W} That's it i-1 ={C×2H×2W} In i-1 ={C×2H×2W} 4. The WiFi human pose estimation method integrating channel attention and multi-branch learning according to claim 1, characterized in that... The network process for restoring the features of each layer of the final decoder and fusing them with the corresponding layer features of the auxiliary decoder in step 7 is as follows: Step 7.1: In the final decoder base block, the structure is similar to that of the encoder and auxiliary decoder base blocks, featuring multi-scale convolutional modules and residual connections, along with a 1×1 convolutional kernel convolutional layer network Conv. 1×1 The combination of BatchNorm and the activation function Mish is used to correct the resolution, ensuring that the data resolution is the same as that output by the corresponding auxiliary decoder layer. Step 7.2, the final decoder base block output De i , and the corresponding auxiliary decoder outputs De_ass at each layer i Same resolution, De_ass i First, channel weights are assigned using the Channel_attn module to obtain the corresponding layer feature data for the decoder that extracts channel features. Then, based on the channel dimension, De... i With De_ass i Perform the connection operation Cat, and attach a convolutional network layer Conv with a 1×1 convolutional kernel. 1×1 The channel dimensionality is reduced, and then batch normalization (BatchNorm) and activation function (Mish) are applied to obtain the final output De of each layer of the auxiliary decoder. i ; Step 7.3: The interconnection of each layer in the final decoder is a bottom-up operation, using a deconvolutional network layer DeConv with 2×2 convolutional kernels. 2×2 With a step size of 2 and padding of 0, we obtain feature data with doubled dimensions H and W, ensuring that the resolution of the output data is the same as that of the corresponding auxiliary decoder layer. Of i-1 =DeConv 2×2 (Of i ) Of i ={C×H×W} Of i-1 ={C×2H×2W} That's it i-1 ={C×2H×2W} 5. The WiFi human pose estimation method integrating channel attention and multi-branch learning according to claim 1, characterized in that... The specific implementation of the channel attention module Channel_attn is as follows: Channel_attn = xSigmoid(Linear(ReLu(Linear(AdaptiveAvg2dPool(x))))) Where x is the input feature map, AdaptiveAvg2dPool is the global adaptive average pooling, Linear is the linear layer, ReLU is the ReLU activation function, and Sigmoid is the Sigmoid activation function. This module assigns different weights to different channels of the input features, highlighting important feature channels and enhancing the expressive power of key features.