Human body posture estimation method based on semantic enhancement and multi-scale feature fusion

Abstract

The invention discloses a human body posture estimation method based on semantic enhancement and multi-scale feature fusion. The method comprises the following steps: constructing a feature pyramid on the basis of a MobileNetV2 network, unifying resolutions of different scale features, and splicing the resolutions; the spliced features are input into a scale perception semantic module based on the OfficientVit, and global semantic representation of scale perception is generated; constructing a deep semantic injection module, fusing scale-perceived global semantics with corresponding features, and selectively strengthening related features by adopting a context-guided attention mechanism; adaptive multi-scale feature fusion is carried out, and the network can adaptively adjust the size of a receptive field so as to better capture human body key point features of different scales and improve the utilization rate of deep and shallow layer features; and constructing a key point detection head by using a 1 * 1 convolutional layer to obtain final key point heat map output. The model provided by the invention can better sense the semantic features and multi-scale visual clues of the human body postures, so that the generalization ability of the model is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human body posture estimation, and in particular to a human body posture estimation method based on semantic enhancement and multi-scale feature fusion. Background Art

[0002] With the gradual development of the field of computer vision, human pose estimation, as an indispensable component of human-computer interaction, has gradually entered people's field of vision. It involves identifying people from input images or videos and using a certain number of interconnected nodes to represent their movements. In practice, researchers usually convert this task into key point detection, in which a specified number of key points are extracted from the human body and then connected according to a certain logic to form a human skeleton. Human pose estimation technology is not only the core support for human tracking, action recognition, three-dimensional posture analysis, and human-computer interaction applications, but also a key link in identifying and understanding human behavior. Therefore, through the continuous improvement and innovation of this basic technology, it will help to further enhance the performance of computer vision in application scenarios such as human-computer interaction.

[0003] However, since keypoints are usually small and extremely sensitive to position, spatial information and semantic information are crucial for the accuracy of overall pose estimation. In order to obtain these two types of information, researchers have proposed some high-resolution convolutional networks, aiming to improve the prediction accuracy of human keypoint detection. SimpleBaseline restores low-resolution features by connecting a three-layer deconvolution module after the last convolutional layer, thereby enhancing the ability to locate keypoints. Although this method improves the prediction accuracy to a certain extent, the last high-resolution feature is sampled from the previous low-resolution feature, which may lead to the loss of image information. To solve this problem, HRNet adopts a high-resolution representation strategy, forming a multi-stage network structure by gradually adding high-resolution to low-resolution sub-networks. Although HRNet has achieved high accuracy indicators in human pose estimation, it still has some limitations. Its network model is too complex, and running the model requires a lot of load, which is heavily dependent on efficient computing devices and large-capacity storage devices. In the CPN network, the feature pyramid structure is used to effectively fuse features at different levels in a bottom-up manner, aiming to obtain richer feature representations and combine features through the head network. Although these methods differ in the way they fuse features, they face a huge gap in receptive fields when aggregating features at different levels, thus learning relatively coarse representations that hinder the precise localization of human poses. In addition, these methods use fixed convolution kernels when fusing features, which limits their ability to adapt to objects of different scales.

[0004] In order to achieve a good balance between the accuracy and complexity of the network, the present invention uses the lightweight network MobileNetV2 as the backbone network to provide a human posture estimation model with rich spatial and semantic information. The feature representation capability of the lightweight network MobileNet is improved by combining the adaptive multi-scale feature fusion module, the attention mechanism and the semantic injection module. The present invention aims to achieve a higher accuracy standard while ensuring acceptable memory and computational overhead. Summary of the invention

[0005] In view of this, the purpose of the embodiments of the present application is to provide a human posture estimation method based on semantic enhancement and multi-scale feature fusion to solve the problems of information loss and excessive model complexity pointed out in the background technology.

[0006] In order to achieve the above technical objectives, the technical solutions adopted in this application are as follows:

[0007] The present application provides a method for human posture estimation based on semantic enhancement and multi-scale feature fusion, the method comprising:

[0008] Step (1): Build a feature pyramid based on the MobileNetV2 network, unify the resolution of features of different scales and splice them;

[0009] Step (2): Input the new features obtained by concatenating the multi-scale features in the feature pyramid into the EfficientViT module to extract scale-aware semantics;

[0010] Step (3): Construct a deep semantic injection module to fuse the scale-aware global semantics with the corresponding features, and use a context-guided attention mechanism to alleviate the semantic gap and selectively enhance relevant features;

[0011] Step (4): Design a multi-scale feature fusion module with adaptive receptive field to extract and fuse effective information from features of different scales;

[0012] Step (5): Concatenate the multi-scale features output from step 4, use a 1×1 convolutional layer to generate the final key point heat map, and use unbiased data processing to correct the heat map, and then predict the key points of the human body.

[0013] Furthermore, the feature extraction backbone network MobileNetV2 is used to construct a feature pyramid, including:

[0014] Based on the MobileNetV2 network, the features of the second, third, fifth and seventh layers of the MobileNetV2 network are output to construct a feature pyramid to obtain features of different scales.

[0015] {F1,F2,…,F N}, where N = 4 represents the number of scales; the features {F1, F2, ..., F N}Average pooling to the resolution size of the seventh layer output of the feature pyramid.

[0016] Furthermore, the scale-aware semantic extraction module based on EfficientVit:

[0017] This module consists of a memory-efficient sandwich layout, a cascaded group attention module, and a parameter redistribution strategy, which improves model efficiency from three aspects: memory, computation, and parameters. The implementation process is as follows:

[0018] 1) Sandwich layout: Specifically, it applies a single attention layer Φ A For spatial mixing, this layer is sandwiched between the FFN layer Φ F The calculation formula is:

[0019] X i+1 =Φ F (Φ A (Φ F (X i ))) (1)

[0020] Among them, X i and X i+1 denote the input and output features of the i-th building block, Φ F It consists of a feedforward neural network layer and a deep convolution, where FFN realizes the information interaction between different feature channels, and the deep convolution replaces the position embedding encoding by introducing the inductive bias of local structural information. A It is a cascaded group attention mechanism;

[0021] 2) Cascaded group attention: Provide different splits of the complete features for each head, thereby explicitly decomposing the attention calculation across heads. The mechanism of attention is expressed as:

[0022] X ij =Attention(X ij W ij Q ,X ij W ij K ,X ij W ij V ) (2)

[0023] X i+1 =Concat[X ij ] j＝1:h W i P (3)

[0024] The jth attention head calculates self-attention, which acts on X ij , that is, input feature X i The jth segmentation of i =[X i1 ,X i2 ,…,X ih ], 1≤j≤h, where h is the total number of attention heads, W ij Q , W ij K and W ij V is a projection layer that maps the input feature segmentation to different subspaces, W i P is a linear layer that projects the concatenated output features back to the same dimension as the input. EfficientVit's self-attention mechanism computes the attention map of each head in a cascaded manner, which adds the output of each head to the subsequent head to gradually improve the feature representation:

[0025] X' ij =X ij +X i(j-1) ,1<j≤h (4)

[0026] X' ij is the jth input segmentation X ij and the output X of the (j-1)th head i(j-1) The result of the calculation and addition is substituted for X ij becomes the new input feature of the j-th head when computing self-attention, and another token interaction layer is applied after the Q-projection.

[0027] 3) Parameter redistribution: Reduce the channel dimension of each head Q and K projection, set the V projection to be consistent with the dimension of the input embedding, reduce the FFN expansion ratio from 4 to 2, and weaken parameter redundancy.

[0028] Furthermore, the deep semantic injection module:

[0029] After obtaining scale-aware semantics, the global semantic features are divided into four parts according to the channel and combined with the features {F1, F2, …, F N} Fusion, injecting deep semantics into the local features of the feature pyramid;

[0030] The semantic injection module takes the local features of the feature pyramid module and the global semantics of the visual transformer as input;

[0031] Local Mark X l Through a 1×1 convolution layer, followed by batch normalization to produce the feature f to be injectedlocal ; Global semantics X g It is fed to a 1×1 convolution layer, followed by a batch normalization layer and a sigmoid layer to produce a semantic weight global_weight. At the same time, the global semantics also passes through a 1×1 convolution layer and then batch normalization to obtain the feature f global , these three outputs have the same number of channels;

[0032] Inject the global semantics into the corresponding local mark to obtain the preliminary injection feature f injected :

[0033] f injected =f local *global_weight (5)

[0034] In addition, the context-guided attention CGA module is introduced to transform the global semantic features f global Add to the previously obtained injection feature f injected ,To further enhance the feature representation, the CGA module assigns a unique spatial importance weight to each feature channel, ,and fuses channel attention and spatial attention, so that local and global features can fully interact and ,fuse in both channel and spatial dimensions;

[0035] f injected and f global Input into the CGA module, calculate the weight of each channel feature modulation, and CGA simultaneously processes the low-level features f injected by preliminary semantics injected and the corresponding high-level features f global , to capture the relationship between them and reflect the importance of each spatial location;

[0036] W=CGA(f injected +f global ) (6)

[0037] By weighted summation, f injected and f global The fusion formula is as follows:

[0038] X fused =Conv 1x1 (f injected *W+f global *(1-W)+f injected +f global ) (7)

[0039] Among them, f injected The weight is W, f global The weight of is (1-W).

[0040] Furthermore, the adaptive multi-scale feature fusion module:

[0041] Taking FPN as the baseline, the adaptive multi-scale feature fusion (AMFF) module is used to improve the fusion mode in the top-down path, so that the model focuses on the feature adjustment between the current layer and the adjacent layer. The transformation completed by AMFF is expressed as F(·), then AMFF can be expressed as follows:

[0042] P i =F(C i ,C i+1 ),i=2,3,4 (8)

[0043] in is the output of AMFF. i and C i+1 are the inputs of AMFF, which represent the feature maps of the i-th layer and the i+1-th layer respectively;

[0044] The AMFF module includes: selectable convolution kernel (LSK) branch, cross-layer interactive CEAM branch and branch integration;

[0045] The LSK branch generates multi-scale features with rich contextual information through a series of depth-separable convolutions with different receptive fields, and uses a spatial kernel selection mechanism to adaptively select the most relevant features. i After a series of transformation processes of the LSK branch, the output features are recorded as

[0046] The CEAM branch aims to seek the intrinsic connection between adjacent layers. Through the cross-layer attention mechanism, the low-level fine-grained features can be fused with the top-level coarse-grained features to improve the feature fusion effect. It can be calculated as:

[0047]

[0048] in is the output of the CEAM branch, sig(·) is the sigmoid activation function, It is a pixel-by-pixel multiplication. Under the guidance of the features of the i-th layer, this branch provides more valuable contextual interaction information for fusion;

[0049] The outputs of LSK and CEAM are fused with the original features through residual integration to achieve adaptive feature adjustment within and across layers. The outputs of LSK and CEAM are integrated with the original i-th layer feature map in a residual manner as follows:

[0050]

[0051] in It is the pixel-by-pixel summation;

[0052] AMFF is extended to the bottom-up path applicable to PAFPN, denoted as AMFF*, where AMFF* is represented by P i and P i-1 As input, the transformation can be expressed as:

[0053] L i =F'(P i ,P i-1 ),i=3,4,5 (11)

[0054] in is the output of AMFF*. In AMFF*, LSK is based on P i is the input, and the output is obtained after transformation, denoted as The output of CEAM is:

[0055]

[0056] Then the output L of AMFF* i It can be calculated as:

[0057]

[0058] Extending AMFF to AMFF* and applying it to the bottom-up feature fusion path of PAFPN can effectively fuse high-level semantic information with low-level positioning information.

[0059] Further, the key point heat map is generated:

[0060] Finally, the features of the AMFF / AMFF* module are upsampled and concatenated, and the key point heat map is obtained through a 1×1 convolution layer and corrected using unbiased data processing. Then, the key points of each part of the human body are predicted.

[0061] The invention adopting the above technical solution has the following advantages:

[0062] 1. This paper designs an attention-guided deep semantic injection module to effectively fuse scale-aware global semantics with corresponding local features to enhance the semantic representation of posture keypoints.

[0063] 2. An adaptive multi-scale feature fusion (AMFF) module is proposed, which can fully utilize the complementarity of multi-scale features and provide richer and more effective feature representation for pose estimation tasks.

[0064] 3. In particular, since the AMFF module integrates a selectable convolution kernel module and a cross-layer interaction mechanism, the former can adaptively adjust the receptive field size and capture multi-scale visual clues; the latter enhances the correlation between deep and shallow layer features, effectively reducing information loss in the feature fusion process.

[0065] 4. Compared with traditional methods, the model of the present invention can better perceive the semantic features and multi-scale visual cues of human posture, thereby significantly improving the estimation performance and is suitable for mobile applications in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The present application may be further described by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate certain embodiments of the present application and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings may be obtained based on these drawings without creative effort.

[0067] Figure 1 This is the overall flow chart of this application;

[0068] Figure 2 This is a schematic diagram of the overall network model architecture in this application;

[0069] Figure 3 This is a schematic diagram of the deep semantic injection module in this application;

[0070] Figure 4 A schematic diagram of the context-guided attention module in this application;

[0071] Figure 5 Schematic diagram of the adaptive multi-scale feature fusion module and details of the AMFF module in this application;

[0072] Figure 6 This is a schematic diagram of the LSK module in this application. DETAILED DESCRIPTION

[0073] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that in the drawings or descriptions, similar or identical parts use the same figure numbers, and the implementation methods not shown or described in the drawings are forms known to ordinary technicians in the relevant technical field. In the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0074] Please refer to Figure 1-Figure 6 This embodiment provides a method for estimating human posture based on semantic enhancement and multi-scale feature fusion. Figure 1 For a specific implementation flow chart, the method includes:

[0075] Step (1), design a lightweight human posture estimation network based on the MobileNetV2 network and construct a feature pyramid;

[0076] like Figure 2As shown in the figure, a lightweight human posture estimation network is constructed based on the MobileNetV2 network. Specifically, in the backbone network of the present invention, the second, third, fifth and seventh layers of the MobileNetV2 network are used as outputs. The output features of these four different levels are used to construct a feature pyramid. The number of stacks of the inverted residual network structure of the output layer of the MobileNetV2 network and the dimension of the output features are described in detail below.

[0077] The input image size is 3×256×192. The training image passes through a stem module, namely the initial convolution layer, which consists of a standard 3×3 convolution layer and a 1×1 convolution layer, which is used to perform preliminary extraction of input features and adjust the number of channels. The output feature dimension is 32×128×96. The main part of the MobileNetV2 network is composed of multiple inverted residual modules stacked together. Each inverted residual module consists of 3 sublayers connected in series: 1) a 1×1 convolution layer for expanding the number of input channels; 2) a 3×3 depth-separable convolution layer for feature extraction; 3) another 1×1 convolution layer for shrinking the number of output channels. This bottleneck structure can greatly reduce model parameters and computational complexity. At different depths, the number of stacked inverted residual modules varies. Among them, the second layer, stacking 2 inverted residual modules, the output feature dimension is 24×64×48; the third layer, stacking 3 inverted residual modules, the output feature dimension is 32×32×24; the fifth layer, stacking 3 inverted residual modules, the output feature dimension is 96×16×12; the seventh layer, stacking 3 inverted residual modules, the output feature dimension is 320×8×6. The output feature of each scale is denoted as F i , then the set of MobileNetV2 output features of different scales is {F1,F2,…,F N}, where N represents the number of scales, N=4.

[0078] Furthermore, the output features of the second, third, and fifth layers are aggregated to the feature size of the seventh layer, i.e., 8×6, by downsampling. Finally, the markers of different scales are concatenated along the channel dimension to generate new markers. The new markers will be fed into the efficient visual Transformer to generate scale-aware semantics.

[0079] Step (2), using efficient visual Transformer (EfficientVit) to generate deep scale-aware semantics;

[0080] This module consists of a memory-efficient sandwich layout, a cascaded group attention module, and a parameter redistribution strategy, which improves model efficiency from three aspects: memory, computation, and parameters. The specific implementation process is as follows:

[0081] Sandwich layout, which uses less memory-constrained self-attention layers and more memory-efficient FFN layers for channel interactions. Specifically, it applies a single self-attention layer Φ A For spatial mixing, this layer is sandwiched between the FFN layer Φ F The calculation formula is:

[0082] X i+1 =Φ F (Φ A (Φ F (X i ))) (14)

[0083] Among them, X i and X i+1 denote the input and output features of the i-th building block respectively. Φ F It consists of a feedforward neural network layer (FFN) and a deep convolution (DWConv), where FFN realizes information interaction between different feature channels, and DWConv replaces the position embedding encoding by introducing the inductive bias of local structural information. A It is a cascaded group attention mechanism.

[0084] Cascaded group attention. In the Transformer model, attention head redundancy is a serious problem in the multi-head self-attention mechanism, which leads to computational inefficiency. The grouping concept of cascaded group attention is similar to the grouped convolution in efficient CNN. It provides different splits of the complete features to each head, thereby explicitly decomposing the attention calculation across heads. This can effectively improve the computational efficiency. Formally speaking, this attention can be expressed as:

[0085] X ij =Attention(X ij W ij Q ,X ij W ij K ,X ij W ij V ) (15)

[0086] X i+1 =Concat[X ij ] j＝1:h W i P (16)

[0087] The jth attention head calculates self-attention, which acts on X ij , that is, input feature X i The jth segmentation of i =[X i1 ,Xi2 ,…,X ih ], where 1≤j≤h. Where h is the total number of attention heads, W ij Q , W ij K and W ij V is a projection layer that maps the input feature segmentation to different subspaces, W i P is a linear layer that projects the concatenated output features back to the same dimension as the input. The multi-head self-attention mechanism of the EfficientVit module computes the attention map of each head in a cascaded manner, which adds the output of each head to the subsequent head to gradually improve the feature representation:

[0088] X' ij =X ij +X i(j-1) ,1<j≤h (17)

[0089] X' ij is the jth input segmentation X ij and the output X of the (j-1)th head i(j-1) The result of the addition. It replaces X ij becomes the new input feature of the jth head when computing self-attention. In addition, another token interaction layer is applied after the Q-projection, which enables the self-attention to jointly capture local and global relations and further enhance the feature representation.

[0090] This cascade design has two advantages. First, providing each head with a different feature split can improve the diversity of attention maps. Similar to grouped convolution, cascaded grouped attention can save h times the number of floating-point operations and parameters because the input and output channels of the QKV layer are reduced by h times. Second, cascading attention heads can increase the network depth, thereby further improving the model capacity without introducing any additional parameters. It only incurs a small latency overhead because the attention map calculation of each head uses a smaller QK channel dimension.

[0091] Parameter redistribution. The core idea is to redistribute parameters in the network by expanding the channel width of key modules and reducing the channel width of unimportant modules. Specifically, the Q and K projections of all stages of each head are set with smaller channel dimensions, and the V projection is set to have the same dimension as the input embedding. Due to the parameter redundancy of FFN, its expansion ratio is also reduced from 4 to 2. With this strategy, important modules have more channels in high-dimensional space to learn representations, avoiding the loss of feature information. At the same time, redundant parameters in unimportant modules are removed to speed up reasoning and improve model efficiency.

[0092] Step (3), such as Figure 3 As shown, a scale-aware deep semantic injection module is built;

[0093] After obtaining scale-aware semantics, the global semantic features are divided into four parts according to the channel, and they are combined with other markers F generated in step (1) i However, in the case of {F1, F2, …, F N There is a significant semantic gap between the} and scale-aware semantics. To this end, a semantic injection module is introduced to alleviate the semantic gap before fusing these markers. Figure 3 As shown in Figure 2, the semantic injection module (SIM) takes the local features of the feature pyramid module and the global semantics of the visual transformer as input. l Through a 1×1 convolution layer, followed by batch normalization to produce the feature f to be injected local . Global semantics X g It is fed to a 1×1 convolution layer, followed by a batch normalization layer and a sigmoid layer to produce a semantic weight global_weight. At the same time, the global semantics also passes through a 1×1 convolution layer and then batch normalization to obtain the feature f global These three outputs have the same size. Then, the global semantics is injected into the corresponding local tags to obtain the preliminary injection feature f injected The local features injected into the global semantics, the global features and the outputs of several SIMs share the same number of channels, denoted as M. The preliminary injected features are expressed by the formula:

[0094] f injected =f local *global_weight (18)

[0095] Afterwards, global semantics are also added to the preliminary injection feature f injected If we use traditional methods such as simple weighted summation or cascading to convert the global semantics f local and f injected Fusion may lead to the loss of useful information or gradient loss. Therefore, the present invention introduces a context-guided attention CGA module, which can emphasize which spatial regions and information in channels the model should pay attention to by assigning a unique spatial importance weight to each feature channel. This content-based attention mechanism can ensure that the feature fusion process focuses more specifically on valuable information, thereby improving the fusion effect. In addition, the calculation method of CGA combines channel attention and spatial attention, which enables local and global features to fully interact and fuse in both channel and space dimensions. This cross-dimensional feature combination helps to extract more discriminative semantic feature representations.

[0096] Furthermore, the detailed process of CGA is as follows Figure 4 shown.

[0097] X∈R C×H×W Representing the ongoing input features, the goal of CGA is to generate a channel-specific spatial importance map with the same dimension as X, i.e., W∈R C×H×W . First calculate the corresponding W c and W s .

[0098] W c =Conv 1×1 (ReLU(Conv 1×1 (X c GAP ))) (19)

[0099] W s =Conv 7×7 ([X s GAP ,X s GMP ]) (20)

[0100] Among them, ReLU represents the activation function, Conv k×k (·) denotes a convolutional layer with a kernel size of k×k, and [·] denotes a channel-wise concatenation operation. c GAP , X s GAP and X s GMP denote the features processed by global average pooling operations across spatial dimensions, global mean pooling operations across channel dimensions, and global maximum pooling operations across channel dimensions, respectively. In order to reduce the number of parameters and limit the complexity of the model, the first 1×1 convolution reduces the channel dimension from C to ( r represents the reduction rate), and the second 1×1 convolution expands it back to C. In the specific implementation, we choose to reduce the channel dimension to a fixed value, i.e., 16. Then, W is added by a simple addition operation following the broadcast rule. c and W s Fusion together to obtain a rough spatial importance map W coa ∈R C×H×W .

[0101] W coa =W c +W s (twenty one)

[0102] In order to obtain the final refined spatial importance map W, W coaEach channel of is adjusted according to the corresponding input feature. We use the content of the input features as guidance to generate the final channel-specific spatial importance map W. In particular, W coa Each channel of and X is rearranged in an alternating manner by a channel shuffle operation.

[0103] W = σ(GConv 7×7 (shuffle([X,W coa ]))) (twenty two)

[0104] in, σ Represents the sigmoid operation, shuffle represents the channel shuffle operation, GConv k×k represents a grouped convolutional layer with a kernel size of k×k. In the specific implementation, the number of groups is set to C.

[0105] Next, the present invention will f injected and f global Input into the CGA module to calculate the weight of each channel feature modulation. CGA will also process the low-level features f injected by the preliminary semantics injected and the corresponding high-level features f global , to capture the relationship between them and reflect the importance of each spatial location.

[0106] W=CGA(f injected +f global ) (twenty three)

[0107] Then, we use weighted summation to convert f injected and f global To be integrated. Among them, f injected The weight is W, f global The weight of is (1-W).

[0108] X fused =Conv 1×1 (f injected *W+f global *(1-W)+f injcted +f global )(twenty four)

[0109] This fusion method ensures that the local features f injected and the global feature f global Fully interact and combine in spatial and channel dimensions. At the same time, the jump connection of the original features is retained to alleviate the problem of information loss and gradient disappearance. Finally, a 1×1 convolution layer is used to fusion feature X fused Projection is performed to obtain the final output features.

[0110] Step (4), building an adaptive multi-scale feature fusion module AMFF;

[0111] Based on the traditional feature pyramid network (FPN), an adaptive multi-scale feature fusion module (AMFF) is designed. AMFF not only inherits the core idea of ​​FPN, but also introduces an adaptive receptive field adjustment mechanism on its basis, and strengthens the interaction between the current layer and the adjacent layer, so as to more effectively capture the diverse characteristics of the target. The specific details of the AMFF module are as follows: Figure 5 As shown. We denote the transformation completed by AMFF as F(·), then AMFF can be expressed as follows:

[0112] P i =F(C i ,C i+1 ),i=2,3,4 (25)

[0113] in is the output of AMFF. i and C i+1 are the input of AMFF, which represent the feature maps of the i-th layer and the i+1-th layer respectively.

[0114] Furthermore, the main structure of the AMFF module is the large selectable convolution kernel (LSK) branch, the cross attention branch (CEAM) and the branch integration. The specific details of these three parts will be elaborated in the following.

[0115] LSK branch: The specific details of the LSK module are as follows Figure 6 As shown in Figure 2. The LSK branch allows the model to adaptively use different large convolution kernels on the i-th layer feature map and adjust the receptive field of each target in the space as needed. Its specific composition structure is as follows:

[0116] 1) Large kernel convolution

[0117] In order to model a series of multiple long-range contexts for adaptive selection, LSK builds larger kernel convolutions through a sequence of depthwise convolutions with large growing kernels and increasing dilation. Specifically, the kernel size k, dilation rate d, and dilation of the receptive field RF of the i-th depthwise convolution in the series are defined as follows:

[0118] k i-1 ≤k i ; d1=1,d i-1 <d i ≤RF i-1 (26)

[0119] RF1=k1,RF i =d i (k i -1)+RF i-1(27)

[0120] The increase in kernel size and dilation rate ensures that the receptive field expands fast enough. We set an upper limit on the dilation rate to ensure that the dilated convolution does not introduce gaps between feature maps. For example, a large kernel can be decomposed into 2 or 3 depth convolutions. In a specific implementation of the present invention, a large kernel is decomposed into 2 depth convolutions.

[0121] In order to obtain features with rich contextual information from different ranges of the input X, a series of factorized depthwise convolutions with different receptive fields are applied:

[0122] U0=X,U i =Conv i dw (U i-1 ) (28)

[0123] Among them, Conv i dw (·) is a kernel with k i and expansion d i Suppose there are N decomposition kernels, each kernel consists of a 1×1 convolution layer Conv 1×1 (·) Further processing:

[0124]

[0125] This allows channel mixing for each spatial feature vector. Then, kernels for various objects are dynamically selected based on the obtained multi-scale features through a selection mechanism, which will be introduced below.

[0126] 2) Spatial core selection

[0127] In order to improve the network's ability to focus on the most relevant spatial context regions to detect objects, a spatial selection mechanism is used to spatially select feature maps from large convolutional kernels of different scales. First, the features obtained from different kernels are concatenated with features of different receptive field ranges:

[0128]

[0129] Then, channel-based average pooling and maximum pooling are performed, denoted as Pavg(·) and Pmax(·), respectively.

[0130]

[0131] Where SAavg and SAmax are the average and maximum pooled spatial feature descriptors. To allow information interaction between different spatial descriptors, the spatially pooled features are concatenated and convolutional layer F is used. 2→N(·) Convert the pooled features (with 2 channels) into N spatial attention maps:

[0132] SA=F 2→N ([SAavg,SAmax]) (32)

[0133] For each spatial attention map SA i , apply the sigmoid activation function to obtain a separate spatial selection mask for each decomposed large kernel:

[0134] SA i =σ(SA i ) (33)

[0135] in σ Denotes the sigmoid function. Then, the features from the decomposed large kernel sequence are weighted by their corresponding spatial selection masks and fused through the convolutional layer F(·) to obtain the attention feature S:

[0136]

[0137] The final output of the LSK module is the element-wise product between the input features X and S:

[0138] Y=X·S (35)

[0139] The LSK module has two advantages. First, it explicitly produces multiple features with various large receptive fields, which makes later kernel selection easier. Second, the sequential decomposition is more efficient than simply applying a single larger kernel. Under the same theoretical receptive field, the decomposed convolution kernel greatly reduces the number of parameters compared to the standard large convolution kernel.

[0140] CEAM branch: The CEAM branch aims to seek the intrinsic connection between adjacent layers and adaptively fuse valuable features through the cross-layer attention mechanism. It can be calculated as

[0141]

[0142] in is the output of the CEAM branch, sig(·) is the sigmoid activation function, is a pixel-by-pixel multiplication. Under the guidance of the i-th layer features, this branch provides more valuable contextual interaction information for fusion.

[0143] Branch integration: After the above effective operations, the output of LSK and CEAM is added to the original i-th layer feature map in a residual manner as follows:

[0144]

[0145] in is the pixel-by-pixel sum. In summary, the i-th layer features are adaptively adjusted through intra-layer and cross-layer attention mechanisms.

[0146] In order to further improve the detection performance of the network, we extend AMFF to the bottom-up path in PAFPN, denoted as AMFF*. i and P i-1 As input, the transformation can be expressed as

[0147] L i =F'(P i ,P i-1 ),i=3,4,5 (38)

[0148] in is the output of AMFF*. In AMFF*, the execution of LSK branch is similar to AMFF. LSK is based on P i is the input and the output is The output of CEAM is:

[0149]

[0150] Then the output L of AMFF* i It can be calculated as:

[0151]

[0152] The structure of AMFF* is similar to Figure 5 AMFF in. Figure 5 In the example, the current feature map C i Replace with P i , map the adjacent feature maps C i+1 Replace with P i-1 AMFF* utilizes cross-attention in a bottom-up manner to integrate high-level semantic information with low-level localization information. With the help of AMFF*, the performance of PAFPN will be further improved.

[0153] Step (5), finally, the features output by the AMFF* module are upsampled and concatenated together. At this time, the feature resolution is 64×48. Finally, a 1×1 convolution is performed to obtain the key point heat map, and the heat map is corrected using unbiased data processing. The loss function is defined as the mean square error to evaluate the performance of the model.

[0154] Through the above technical solutions, the semantic information and spatial information required for human posture estimation can be effectively obtained. Through the attention-guided deep semantic injection module, the scale-aware global semantics are effectively fused with the corresponding local features to enhance the model's ability to represent key points. Secondly, the adaptive multi-scale feature fusion (AMFF) module dynamically adjusts the size of the receptive field by integrating the selectable convolution kernel module (LSK) and the cross-layer interaction mechanism, captures richer multi-scale visual clues, and enhances the correlation between deep and shallow features, effectively reducing information loss during feature fusion. Through the coordinated operation of these technical means, the model's ability to understand, analyze and accurately estimate complex human postures is greatly enhanced.

[0155] In the embodiments provided in the present application, it should be understood that the disclosed devices, equipment and methods can also be implemented in other ways. The above-described device, equipment and method embodiments are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0156] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A human posture estimation method based on semantic enhancement and multi-scale feature fusion, characterized in that: The method comprises: Step (1): Build a feature pyramid based on the MobileNetV2 network, unify the resolution of features of different scales and splice them; Step (2): Input the new features obtained by concatenating the multi-scale features in the feature pyramid into the EfficientViT module to extract scale-aware semantics; Step (3): Construct a deep semantic injection module to fuse the scale-aware global semantics with the corresponding features, and use a context-guided attention mechanism to alleviate the semantic gap and selectively enhance relevant features; Step (4): Design a multi-scale feature fusion module with adaptive receptive field to extract and fuse effective information from features of different scales; Step (5): Concatenate the multi-scale features output from step 4, use a 1×1 convolutional layer to generate the final key point heat map, and use unbiased data processing to correct the heat map, and then predict the key points of the human body.

2. The method for human posture estimation based on semantic enhancement and multi-scale feature fusion according to claim 1, characterized in that: The feature extraction backbone network MobileNetV2 is used to construct a feature pyramid, including: Based on the MobileNetV2 network, the features of the second, third, fifth and seventh layers of the MobileNetV2 network are output to construct a feature pyramid to obtain features of different scales {F1, F2, …, F N }, where N = 4 represents the number of scales; the features {F1, F2, ..., F N }Average pooling to the resolution size of the seventh layer output of the feature pyramid.

3. The method for human posture estimation based on semantic enhancement and multi-scale feature fusion according to claim 2, characterized in that: The scale-aware semantic extraction module based on EfficientVit: This module consists of a memory-efficient sandwich layout, a cascaded group attention module, and a parameter redistribution strategy, which improves model efficiency from three aspects: memory, computation, and parameters. The implementation process is as follows: 1) Sandwich layout: Specifically, it applies a single attention layer Φ A For spatial mixing, this layer is sandwiched between the FFN layer Φ F The calculation formula is: X i+1 =Φ F (F A (F F (X i ))) (1) Among them, X i and X i+1 denote the input and output features of the i-th building block, Φ F It consists of a feedforward neural network layer and a deep convolution, where FFN realizes the information interaction between different feature channels, and the deep convolution replaces the position embedding encoding by introducing the inductive bias of local structural information. A It is a cascaded group attention mechanism; 2) Cascaded group attention: Provide different splits of the complete features for each head, thereby explicitly decomposing the attention calculation across heads. The mechanism of attention is expressed as: X ij =Attention(X ij W ij Q ,X ij W ij K ,X ij W ij V ) (2) X i+1 =Concat[X ij ] j＝1:h W i P (3) The jth attention head calculates self-attention, which acts on X ij , that is, input feature X i The jth segmentation of i =[X i1 ,X i2 ,…,X ih ], 1≤j≤h, where h is the total number of attention heads, W ij Q , W ij K and W ij V is a projection layer that maps the input feature segmentation to different subspaces, W i P is a linear layer that projects the concatenated output features back to the same dimension as the input. EfficientVit’s self-attention mechanism computes the attention map of each head in a cascaded manner, which adds the output of each head to the subsequent head to gradually improve the feature representation: X' ij =X ij +X i(j-1) ,1<j≤h (4) X' ij is the jth input segmentation X ij and the output X of the (j-1)th head i(j-1) The result of the calculation and addition is substituted for X ij becomes the new input feature of the jth head when computing self-attention, and another token interaction layer is applied after the Q-projection. 3) Parameter redistribution: Reduce the channel dimension of each head Q and K projection, set the V projection to be consistent with the dimension of the input embedding, reduce the FFN expansion ratio from 4 to 2, and weaken parameter redundancy.

4. The method for human posture estimation based on semantic enhancement and multi-scale feature fusion according to claim 3, characterized in that: The deep semantic injection module: After obtaining scale-aware semantics, the global semantic features are divided into four parts according to the channel and combined with the features {F1, F2, …, F N } Fusion, injecting deep semantics into the local features of the feature pyramid; The semantic injection module takes the local features of the feature pyramid module and the global semantics of the visual Transformer as input; the local mark X l Through a 1×1 convolution layer, followed by batch normalization to produce the feature f to be injected local ; Global semantics X g It is fed to a 1×1 convolution layer, followed by a batch normalization layer and a sigmoid layer to produce a semantic weight global_weight. At the same time, the global semantics also passes through a 1×1 convolution layer, followed by batch normalization to obtain the feature f global , these three outputs have the same number of channels; inject the global semantics into the corresponding local tags to obtain the preliminary injection feature f injected : f injected =f local *global_weight (5) In addition, the context-guided attention CGA module is introduced to transform the global semantic features f global Add to the previously obtained injection feature f injected ,To further enhance the feature representation, the CGA module assigns a unique spatial importance weight to each feature channel., ,CGA fuses channel attention and spatial attention, so that local and global features can fully interact and fuse in both channel and spatial dimensions; f injected and f global Input into the CGA module, calculate the weight of each channel feature modulation, and CGA simultaneously processes the low-level features f injected by preliminary semantics injected and the corresponding high-level features f global , to capture the relationship between them and reflect the importance of each spatial location; W=CGA(f injected + f global ) (6) By weighted summation, f injected and f global The fusion formula is as follows: X fused =Conv 1x1 (f injected * W+f global *(1-W)+f injected +f global ) (7) Among them, f injected The weight is W, f global The weight of is (1-W).

5. The method for human posture estimation based on semantic enhancement and multi-scale feature fusion according to claim 4, characterized in that: The adaptive multi-scale feature fusion module: Taking FPN as the baseline, the adaptive multi-scale feature fusion (AMFF) module is used to improve the fusion mode in the top-down path, so that the model focuses on the feature adjustment between the current layer and the adjacent layer. The transformation completed by AMFF is expressed as F(·), then AMFF can be expressed as follows: P i =F(C i ,C i+1 ),i=2,3,4 (8) in is the output of AMFF. i and C i+1 are the inputs of AMFF, which represent the feature maps of the i-th layer and the i+1-th layer respectively; The AMFF module includes: selectable convolution kernel (LSK) branch, cross-layer interactive CEAM branch and branch integration; The LSK branch generates multi-scale features with rich contextual information through a series of depth-separable convolutions with different receptive fields, and uses a spatial kernel selection mechanism to adaptively select the most relevant features. i After a series of transformation processes of the LSK branch, the output features are recorded as The CEAM branch aims to seek the intrinsic connection between adjacent layers. Through the cross-layer attention mechanism, the low-level fine-grained features can be fused with the top-level coarse-grained features to improve the feature fusion effect. It can be calculated as: in is the output of the CEAM branch, sig(·) is the sigmoid activation function, It is a pixel-by-pixel multiplication. Under the guidance of the features of the i-th layer, this branch provides more valuable contextual interaction information for fusion; The outputs of LSK and CEAM are fused with the original features through residual integration to achieve adaptive feature adjustment within and across layers. The outputs of LSK and CEAM are integrated with the original i-th layer feature map in a residual manner as follows: in It is the pixel-by-pixel summation; AMFF is extended to the bottom-up path applicable to PAFPN, denoted as AMFF*, where AMFF* is represented by P i and P i-1 As input, the transformation can be expressed as: L i =F'(P i ,P i-1 ),i=3,4,5 (11) in is the output of AMFF*. In AMFF*, LSK is based on P i is the input, and the output is obtained after transformation, denoted as The output of CEAM is: Then the output L of AMFF* i It can be calculated as: Extending AMFF to AMFF* and applying it to the bottom-up feature fusion path of PAFPN can effectively fuse high-level semantic information with low-level positioning information.

6. The method for human posture estimation based on semantic enhancement and multi-scale feature fusion according to claim 5, characterized in that: The key point heat map is generated: Finally, the features of the AMFF / AMFF* module are upsampled and concatenated, and the key point heat map is obtained through a 1×1 convolution layer and corrected using unbiased data processing. Then, the key points of each part of the human body are predicted.

Images