A human pose estimation method based on semantic enhancement and multi-scale feature fusion

Abstract

The application discloses a human pose estimation method based on semantic enhancement and multi-scale feature fusion. The method comprises the following steps: constructing a feature pyramid on the basis of a MobileNetV2 network, unifying the resolutions of different scale features and splicing; inputting the features after splicing into a scale perception semantic module based on EfficientVit to generate a scale perception global semantic representation; constructing a deep semantic injection module to fuse the scale perception global semantic and the corresponding features, and selectively strengthening the related features by using a context-guided attention mechanism; adaptively fusing the multi-scale features, so that the network can adaptively adjust the receptive field size to better capture human key point features of different scales and improve the utilization rate of deep and shallow layer features; and using a 1*1 convolution layer to construct a key point detection head to obtain the final key point heat map output. The model can better perceive the semantic features and multi-scale visual clues of the human pose, thereby significantly improving the generalization ability of the model.

Description

Technical Field

[0001] This invention relates to the field of human pose estimation technology, and more specifically, to a human pose estimation method based on semantic enhancement and multi-scale feature fusion. Background Technology

[0002] With the gradual development of computer vision, human pose estimation, as an indispensable component of human-computer interaction, has gradually come into focus. It involves identifying people from input images or videos and representing their motion using a certain number of interconnected nodes. In practice, researchers often transform this task into keypoint detection, where a specified number of keypoints are extracted from the human body and then connected according to certain logic to form the human skeleton. Human pose estimation technology is not only a core support for human tracking, action recognition, 3D pose analysis, and human-computer interaction applications, but also a key link in recognizing and understanding human behavior. Therefore, continuous improvement and innovation of this fundamental technology will help further enhance the performance of computer vision in human-computer interaction and other application scenarios.

[0003] However, since keypoints are typically small and extremely sensitive to location, spatial and semantic information is crucial for the accuracy of overall pose estimation. To obtain this information, researchers have proposed several high-resolution convolutional networks (HRNets) aimed at improving the prediction accuracy of human keypoint detection. SimpleBaseline enhances keypoint localization by recovering low-resolution features after the last convolutional layer. While this method improves prediction accuracy to some extent, the last high-resolution feature is sampled from the previous low-resolution feature, which may lead to the loss of image information. To address this issue, HRNet employs a high-resolution representation strategy, forming a multi-stage network structure by progressively adding high-resolution to low-resolution subnetworks. Although HRNet achieves high accuracy in human pose estimation, it still has some limitations. Its network model is too complex, requiring a large workload to run, and heavily relying on efficient computing devices and large-capacity storage devices. In the CPN network, a feature pyramid structure is used to effectively fuse features from different levels in a bottom-up manner, aiming to obtain richer feature representations, and then combining features through the head network. Although these methods differ in their feature fusion approaches, they face significant receptive field gaps when aggregating features at different levels, resulting in relatively coarse representations that hinder accurate human pose localization. Furthermore, the use of fixed convolutional kernels during feature fusion limits their ability to adapt to targets of varying scales.

[0004] To address the aforementioned issues and achieve a good balance between network accuracy and complexity, this invention uses the lightweight MobileNetV2 network as its backbone, providing a human pose estimation model rich in spatial and semantic information. By combining an adaptive multi-scale feature fusion module, an attention mechanism, and a semantic injection module, the feature representation capabilities of the lightweight MobileNet network are enhanced. This invention aims to achieve high accuracy while maintaining acceptable memory and computational overhead. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a human pose estimation method based on semantic enhancement and multi-scale feature fusion, so as to solve the problems of information loss and excessive model complexity mentioned in the background art.

[0006] To achieve the above technical objectives, the technical solution adopted in this application is as follows:

[0007] This application provides a human pose estimation method based on semantic enhancement and multi-scale feature fusion, the method comprising:

[0008] Step (1): Construct a feature pyramid based on the MobileNetV2 network, unify the resolution of features at different scales, and stitch them together;

[0009] Step (2): Input the new features after splicing the multi-scale features in the feature pyramid into the EfficientViT module to extract scale-aware semantics;

[0010] Step (3): Construct a deep semantic injection module to fuse scale-aware global semantics with corresponding features, and use a context-guided attention mechanism to alleviate semantic gaps and selectively enhance relevant features;

[0011] Step (4): Design an adaptive receptive field multi-scale feature fusion module to extract and fuse effective information from features at different scales;

[0012] Step (5): The multi-scale features output in step 4 are spliced ​​together, a 1×1 convolutional layer is used to generate the final key point heatmap, and unbiased data processing is used to correct the heatmap. Then, human key points are predicted.

[0013] Furthermore, a feature pyramid is constructed using the feature extraction backbone network MobileNetV2, including:

[0014] Based on the MobileNetV2 network, feature pyramids are constructed using the feature outputs of layers 2, 3, 5, and 7 of the MobileNetV2 network to obtain features at different scales.

[0015] {F1,F2,…,F N}, where N=4 represents the number of scales; the features {F1,F2,…,F} are... N The average resolution size of the output from the seventh layer of the feature pyramid.

[0016] Furthermore, the scale-aware semantic extraction module based on EfficientVit:

[0017] This module consists of a memory-efficient sandwich layout, a cascaded attention module, and a parameter reallocation strategy, which improve model efficiency from three aspects: memory, computation, and parameters. The implementation process is as follows:

[0018] 1) Sandwich layout: Specifically, it applies a single attention layer Φ A Spatial mixing is performed, and this layer is sandwiched between FFN layers Φ. F The calculation formula is as follows:

[0019] X i+1 =Φ F (Φ A (Φ F (X i ))) (1)

[0020] Among them, X i and X i+1 Let Φ represent the input and output features of the i-th building block, respectively. F It consists of a feedforward neural network layer and a depthwise convolutional layer. The FFN layer enables information exchange between different feature channels, while the depthwise convolutional layer replaces positional embedding encoding by introducing inductive biases that incorporate local structural information. Φ A It is a cascaded grouping attention mechanism;

[0021] 2) Cascaded Group Attention: Provides distinct segmentations of complete features for each head, thereby explicitly decomposing attention computation across heads. The mechanism of this attention is described as follows:

[0022] X ij =Attention(X) ij W ij Q ,X ij W ij K ,X ij W ij V (2)

[0023] X i+1 =Concat[X ij ] j＝1:h W i P (3)

[0024] The j-th attention head calculates self-attention, which acts on X. ij That is, input feature X i The j-th segment, i.e., X i =[X i1 ,X i2 ,…,X ih ], 1≤j≤h, where h is the total number of attention heads, W ij Q W ij K and W ij V It is a projection layer that segments and maps input features to different subspaces, W i P It is a linear layer that projects the concatenated output features back to the same dimension as the input. EfficientVit's self-attention mechanism computes the attention map for each head in a cascaded manner, adding the output of each head to subsequent heads to progressively improve the feature representation.

[0025] X' ij =X ij +X i(j-1) ,1<j≤h (4)

[0026] X' ij It is the j-th input segment X ij The output X of the (j-1)th head i(j-1) The result of the calculation and addition is used to replace X. ij It becomes a new input feature of the j-th head when computing self-attention, and another token interaction layer is applied after Q-projection.

[0027] 3) Parameter reallocation: Reduce the channel dimension of each head Q and K projection, set the V projection to be consistent with the dimension of the input embedding, reduce the FFN expansion ratio from 4 to 2, and reduce parameter redundancy.

[0028] Furthermore, the deep semantic injection module:

[0029] After obtaining scale-aware semantics, the global semantic features are divided into four parts according to channels, and then compared with the feature {F1, F2, ..., F...} N Fusion injects deep semantics into the local features of the feature pyramid;

[0030] The semantic injection module takes the local features of the feature pyramid module and the global semantics of the visual Transformer as input;

[0031] Local marker X l The feature f to be injected is generated by using 1×1 convolutional layers followed by batch normalization.local Global semantics X g The data is fed into a 1×1 convolutional layer, followed by a batch normalization layer and a sigmoid layer to produce the semantic weights global_weight. Simultaneously, the global semantics are also passed through a 1×1 convolutional layer, followed by batch normalization to obtain the features f. global These three outputs have the same number of channels;

[0032] Global semantics are injected into the corresponding local tags to obtain preliminary injected features f. injected :

[0033] f injected =f local *global_weight (5)

[0034] Furthermore, the context-guided attention CGA module introduces global semantic features f global Add to the previously obtained injection feature f injected To further enhance feature representation, the CGA module assigns a unique spatial importance weight to each feature channel, and integrates channel attention and spatial attention, enabling local and global features to fully interact and merge in both channel and spatial dimensions.

[0035] f injected and f global The input is fed into the CGA module, which calculates the weights of the modulated features for each channel. The CGA also processes the low-level features f from the initial semantic injection. injected and corresponding high-level features f global To capture the relationships between them and reflect the importance of each spatial location;

[0036] W = CGA(f injected +f global (6)

[0037] f is calculated using a weighted summation method. injected and f global The fusion process is as follows:

[0038] X fused =Conv 1x1 (f injected *W+f global *(1-W)+f injected +f global (7)

[0039] Among them, f injected The weights are W and f global The weight is (1-W).

[0040] Furthermore, the adaptive multi-scale feature fusion module:

[0041] Using FPN as a baseline, the fusion mode in the top-down path is improved by the Adaptive Multi-Scale Feature Fusion (AMFF) module, which makes the model focus on feature adjustment between the current layer and adjacent layers. The transformation performed by AMFF is represented as F(·), and AMFF can be expressed as follows:

[0042] P i =F(C i C i+1 ), i = 2, 3, 4 (8)

[0043] in This is the output of AMFF. C i and C i+1 These are the inputs to AMFF, representing the feature maps of the i-th and (i+1)-th layers, respectively;

[0044] The AMFF module includes: Selectable kernel (LSK) branch, cross-layer interactive CEAM branch, and branch integration;

[0045] The Selectable Kernel (LSK) branch generates multi-scale features with rich contextual information through a series of depthwise separable convolutions with different receptive fields, and adaptively selects the most relevant features using a spatial kernel selection mechanism. The input feature C... i After the LSK branching process, the resulting output feature is denoted as...

[0046] The CEAM branch aims to seek the intrinsic connections between adjacent layers. Through a cross-layer attention mechanism, it can fuse fine-grained features from lower layers with coarse-grained features from top layers to improve feature fusion performance. It can be calculated as follows:

[0047]

[0048] in This is the output of the CEAM branch, and sig(·) is the sigmoid activation function. It is a pixel-wise multiplication, and guided by the features of the i-th layer, this branch provides more valuable contextual interaction information for fusion;

[0049] The outputs of LSK and CEAM are fused with the original features through residual ensemble, achieving adaptive feature adjustment within and across layers. The outputs of LSK and CEAM are integrated with the original feature map of the i-th layer in a residual manner as follows:

[0050]

[0051] in It sums the values ​​pixel by pixel;

[0052] AMFF is extended to a bottom-up path applicable to PAFPN, denoted as AMFF*, where AMFF* is P i and P i-1 As input, the transformation can be represented as:

[0053] L i =F'(P i ,P i-1 ), i = 3, 4, 5 (11)

[0054] in This is the output of AMFF*. In AMFF*, LSK is P i The input is transformed to obtain the output, denoted as . The output of CEAM is:

[0055]

[0056] Then the output L of AMFF* i It can be calculated as:

[0057]

[0058] Extending AMFF to AMFF* and applying it to the bottom-up feature fusion path of PAFPN can effectively fuse high-level semantic information with low-level localization information.

[0059] Furthermore, the key point heatmap is generated:

[0060] Finally, the features processed by the AMFF / AMFF* module are upsampled and stitched together, and then passed through a 1×1 convolutional layer to obtain a key point heatmap. The heatmap is then corrected using unbiased data processing, and the key points of each part of the human body are predicted.

[0061] The invention employing the above technical solution has the following advantages:

[0062] 1. This invention designs an attention-guided deep semantic injection module that effectively integrates scale-aware global semantics with corresponding local features to enhance the semantic representation of pose key points.

[0063] 2. An adaptive multi-scale feature fusion (AMFF) module is proposed, which can make full use of the complementarity of multi-scale features to provide richer and more effective feature representations for pose estimation tasks.

[0064] 3. In particular, the AMFF module integrates a selectable convolutional kernel module and a cross-layer interaction mechanism. The former can adaptively adjust the receptive field size and capture multi-scale visual cues; the latter enhances the correlation between features in deep and shallow layers, effectively reducing information loss during feature fusion.

[0065] 4. Compared with traditional methods, the model of this invention can better perceive the semantic features of human posture and multi-scale visual cues, thereby significantly improving estimation performance and making it suitable for mobile applications in complex scenarios. Attached Figure Description

[0066] This application can be further illustrated by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate some embodiments of this application and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any inventive effort.

[0067] Figure 1 This is the overall flowchart of this application;

[0068] Figure 2 This is a schematic diagram of the overall network model architecture in this application;

[0069] Figure 3 This is a schematic diagram of the deep semantic injection module in this application;

[0070] Figure 4 This is a schematic diagram of the context-guided attention module in this application;

[0071] Figure 5 This application includes a schematic diagram of the adaptive multi-scale feature fusion module and details of the AMFF module.

[0072] Figure 6 This is a schematic diagram of the LSK module in this application. Detailed Implementation

[0073] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts are referred to by the same reference numerals in the drawings or description. Implementations not shown or described in the drawings are forms known to those skilled in the art. In the description of this application, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0074] Please refer to Figures 1-6 This embodiment provides a human pose estimation method based on semantic enhancement and multi-scale feature fusion. Figure 1 As shown in the specific implementation flowchart, the method includes:

[0075] Step (1): Design a lightweight human pose estimation network based on the MobileNetV2 network and construct a feature pyramid;

[0076] like Figure 2As shown, a lightweight human pose estimation network is constructed based on the MobileNetV2 network. Specifically, in the backbone network of this invention, layers 2, 3, 5, and 7 of the MobileNetV2 network are used as outputs. A feature pyramid is constructed using the output features from these four different layers. The following section will detail the number of stacks of the inverted residual network structure of the MobileNetV2 network output layers and the dimensions of the output features.

[0077] The input image size is 3×256×192, which is the training image. It passes through a stem module, i.e., the initial convolutional layer, which consists of a standard 3×3 convolutional layer and a 1×1 convolutional layer, used for initial feature extraction and channel number adjustment. The output feature dimension is 32×128×96. The main body of the MobileNetV2 network consists of multiple stacked inverse residual modules. Each inverse residual module consists of three cascaded sublayers: 1) a 1×1 convolutional layer for expanding the input channel number; 2) a 3×3 depthwise separable convolutional layer for feature extraction; and 3) another 1×1 convolutional layer for shrinking the output channel number. This bottleneck structure can significantly reduce model parameters and computational cost. The number of stacked inverse residual modules varies at different depths. The second layer stacks two inverse residual modules, with an output feature dimension of 24×64×48; the third layer stacks three inverse residual modules, with an output feature dimension of 32×32×24; the fifth layer stacks three inverse residual modules, with an output feature dimension of 96×16×12; and the seventh layer stacks three inverse residual modules, with an output feature dimension of 320×8×6. The output feature at each scale is denoted as F. i Then the set of features at different scales output by MobileNetV2 is {F1, F2, ..., F...} N}, where N represents the number of scales, N = 4.

[0078] Furthermore, the output features from the second, third, and fifth layers are downsampled and aggregated into a feature size of 8×6 for the seventh layer. Finally, labels of different scales are concatenated along the channel dimension to generate new labels. These new labels are then fed into an efficient visual Transformer to produce scale-aware semantics.

[0079] Step (2): Utilize an efficient visual Transformer (EfficientVit) to generate deep scale-aware semantics;

[0080] This module consists of a memory-efficient sandwich layout, a cascaded attention module, and a parameter reallocation strategy, improving model efficiency in terms of memory, computation, and parameters, respectively. The specific implementation process is shown below:

[0081] The sandwich layout uses a self-attention layer with fewer memory constraints and more memory-efficient FFN layers for channel interactions. Specifically, it applies a single self-attention layer Φ. A Spatial mixing is performed, and this layer is sandwiched between FFN layers Φ. F Between. The calculation formula is:

[0082] X i+1 =Φ F (Φ A (Φ F (X i (14)

[0083] Among them, X i and X i+1 Let Φ represent the input and output features of the i-th building block, respectively. F It consists of a feedforward neural network layer (FFN) and a deep convolutional layer (DWConv). The FFN enables information exchange between different feature channels, while the DWConv replaces positional embedding encoding by introducing inductive biases based on local structural information. A It is a cascaded grouping attention mechanism.

[0084] Cascaded group attention. In Transformer models, attention head redundancy is a serious problem in multi-head self-attention mechanisms, leading to computational inefficiency. The grouping concept of cascaded group attention is similar to grouped convolution in efficient CNNs. It provides different segments of complete features for each head, thus explicitly decomposing the attention computation across heads. This significantly improves computational efficiency. Formally, this attention can be expressed as:

[0085] X ij =Attention(X) ij W ij Q ,X ij W ij K ,X ij W ij V (15)

[0086] X i+1 =Concat[X ij ] j＝1:h W i P (16)

[0087] The j-th attention head calculates self-attention, which acts on X. ij That is, input feature X i The j-th segment, i.e., X i =[X i1 ,Xi2 ,…,X ih ], where 1≤j≤h. Here, h is the total number of attention heads, and W ij Q W ij K and W ij V It is a projection layer that segments and maps input features to different subspaces, W i P It is a linear layer that projects the concatenated output features back to the same dimension as the input. The multi-head self-attention mechanism of the EfficientVit module computes the attention map of each head in a cascaded manner, adding the output of each head to subsequent heads to progressively improve the feature representation:

[0088] X' ij =X ij +X i(j-1) ,1<j≤h (17)

[0089] X' ij It is the j-th input segment X ij The output X of the (j-1)th head i(j-1) The result of the addition. It replaces X. ij This becomes the new input feature for the j-th head when computing self-attention. Furthermore, another token interaction layer is applied after Q-projection, enabling self-attention to jointly capture local and global relationships and further enhance the feature representation.

[0090] This cascaded design has two advantages. First, providing different feature segmentations to each head increases the diversity of the attention map. Similar to grouped convolutions, cascaded grouped attention can save h times the number of floating-point operations and parameters because the input and output channels of the QKV layer are reduced by h times. Second, cascading attention heads can increase network depth, thereby further increasing model capacity without introducing any additional parameters. It only incurs a small latency overhead because the attention map computation for each head uses a smaller QK channel dimension.

[0091] Parameter reallocation. The core idea is to reallocate parameters in the network by increasing the channel width of critical modules while decreasing the channel width of less important modules. Specifically, the Q and K projections for all stages of each head are set to a smaller channel dimension, while the V projection is set to have the same dimension as the input embedding. Due to the parameter redundancy of FFN, its expansion ratio is also reduced from 4 to 2. Using this strategy, important modules have more channels in the high-dimensional space to learn representations, avoiding the loss of feature information. Simultaneously, redundant parameters in less important modules are removed, accelerating inference and improving model efficiency.

[0092] Step (3), as Figure 3 As shown, a scale-aware deep semantic injection module is constructed;

[0093] After obtaining scale-aware semantics, the global semantic features are divided into four parts according to channels, and these parts are combined with other labels F generated in step (1). i They can be added directly. However, in the notation {F1, F2, ..., F...} N There is a significant semantic gap between} and scale-aware semantics. To address this, a semantic injection module is introduced to mitigate this gap before fusing these tags. For example... Figure 3 As shown, the Semantic Injection Module (SIM) takes the local features of the Feature Pyramid module and the global semantics of the Visual Transformer as input. Local label X l The feature f to be injected is generated by using 1×1 convolutional layers followed by batch normalization. local Global semantics X g The data is fed into a 1×1 convolutional layer, followed by a batch normalization layer and a sigmoid layer to produce the semantic weights global_weight. Simultaneously, the global semantics are also passed through a 1×1 convolutional layer, followed by batch normalization to obtain the features f. global These three outputs have the same size. Then, the global semantics are injected into the corresponding local labels to obtain the initial injected features f. injected The local features, global features, and outputs of several SIMs injected with global semantics share the same number of channels, denoted as M. The initial injected features are expressed by the formula:

[0094] f injected =f local *global_weight (18)

[0095] Subsequently, global semantics are added to the initially injected feature f. injected In the middle. If traditional methods, such as simple weighted summation or concatenation, are used, the global semantics f will be... local and f injected Feature fusion can lead to the loss of useful information or gradient loss. Therefore, this invention introduces a context-guided attention CGA module. By assigning a unique spatial importance weight to each feature channel, it emphasizes which spatial regions and channels the model should focus on. This content-based attention mechanism ensures that the feature fusion process focuses more specifically on valuable information, thereby improving fusion performance. Furthermore, CGA's computation method integrates channel attention and spatial attention, allowing local and global features to fully interact and fuse in both channel and spatial dimensions. This cross-dimensional feature combination helps extract more discriminative semantic feature representations.

[0096] Furthermore, the detailed process of CGA is as follows: Figure 4 As shown.

[0097] X∈R C×H×W Representing the input features in progress, the goal of CGA is to generate a channel-specific spatial importance map with the same dimension as X, i.e., W∈R. C×H×W First, calculate the corresponding W. c and W s .

[0098] W c =Conv 1×1 (ReLU(Conv 1×1 (X c GAP ))) (19)

[0099] W s =Conv 7×7 ([X s GAP ,X s GMP (20)

[0100] Where ReLU represents the activation function, Conv k×k (·) represents a convolutional layer with a kernel size of k×k, and [·] represents a channel-wise concatenation operation. X c GAP X s GAP and X s GMP These represent features processed by global average pooling across spatial dimensions, global mean pooling across channel dimensions, and global max pooling across channel dimensions, respectively. To reduce the number of parameters and limit model complexity, the first 1×1 convolution reduces the channel dimension from C to [value missing]. ( r (representing the reduction rate), the second 1×1 convolution expands it back to C. In the specific implementation, the channel dimension is chosen to be reduced to a fixed value, i.e., 16. Then, W is expanded by simple addition operations following the broadcast rules. c and W s By combining these elements, a coarse spatial importance map W is obtained. coa ∈R C×H×W .

[0101] W coa =W c +W s (twenty one)

[0102] To obtain the final refined spatial importance map W, W coaEach channel is adjusted based on the corresponding input features. We use the content of the input features as guidance to generate the final channel-specific spatial importance map W. In particular, W coa Each channel of X is rearranged in an alternating manner through a channel shuffling operation.

[0103] W=σ(GConv 7×7 (shuffle([X,W coa ]))) (twenty two)

[0104] in, σ GConv represents the sigmoid operation, shuffle represents the channel shuffling operation. k×k This represents a grouped convolutional layer with a kernel size of k×k. In a specific implementation, the number of groups is set to C.

[0105] Next, the present invention will f injected and f global The input is fed into the CGA module, which calculates the weights for feature modulation of each channel. CGA also processes the low-level features f from the initial semantic injection. injected and corresponding high-level features f global This is to capture the relationships between them and reflect the importance of each spatial location.

[0106] W = CGA(f injected +f global ) (twenty three)

[0107] Then, we use a weighted summation method to calculate f. injected and f global To merge. Where, f injected The weights are W and f global The weight is (1-W).

[0108] X fused =Conv 1×1 (f injected *W+f global *(1-W)+f injcted +f global )(twenty four)

[0109] This fusion method ensures that the local features f injected through deep semantics are preserved. injected and global features f global It achieves full interaction and combination in both spatial and channel dimensions. Simultaneously, it preserves skip connections of the original features to mitigate information loss and gradient vanishing issues. Finally, it uses 1×1 convolutional layers to fuse the features X. fused Projection is performed to obtain the final output features.

[0110] Step (4): Construct the adaptive multi-scale feature fusion module AMFF;

[0111] Based on the traditional Feature Pyramid Network (FPN), an Adaptive Multi-Scale Feature Fusion Module (AMFF) was designed. AMFF not only inherits the core ideas of FPN but also introduces an adaptive receptive field adjustment mechanism and enhances the interaction between the current layer and adjacent layers, thereby more effectively capturing the diverse features of the target. Specific details of the AMFF module are as follows... Figure 5 As shown. If we denote the transformation performed by AMFF as F(·), then AMFF can be represented as follows:

[0112] P i =F(C i C i+1 ), i = 2, 3, 4 (25)

[0113] in This is the output of AMFF. C i and C i+1 These are the inputs to AMFF, representing the feature maps of the i-th and (i+1)-th layers, respectively.

[0114] Furthermore, the main structure of the AMFF module consists of a large selectable kernel (LSK) branch, a cross attention branch (CEAM), and branch ensemble. The specific details of these three parts will be elaborated below.

[0115] LSK branch: Details of the LSK module are as follows Figure 6 As shown. The LSK branch allows the model to adaptively use different large convolutional kernels on the feature map of layer i and adjust the receptive field of each target in the space as needed. Its specific structure is as follows:

[0116] 1) Large kernel convolution

[0117] To model a series of multiple long-range contexts for adaptive selection, LSK constructs larger kernel convolutions by using depthwise convolution sequences with large growing kernels and increased dilation. Specifically, the kernel size k, dilation rate d, and dilation of the receptive field RF for the i-th depthwise convolution in this series are defined as follows:

[0118] k i-1 ≤k i ;d1=1,d i-1 <d i ≤RF i-1 (26)

[0119] RF1 = k1, RF i =d i (k i -1)+RF i-1(27)

[0120] Increasing the kernel size and dilation rate ensures that the receptive field dilates quickly enough. We set an upper limit on the dilation rate to ensure that dilated convolutions do not introduce gaps between feature maps. For example, a large kernel can be decomposed into 2 or 3 depthwise convolutions. In a specific implementation of this invention, a large kernel is decomposed into 2 depthwise convolutions.

[0121] To obtain features with rich contextual information from different ranges of input X, a series of decomposition depthwise convolutions with different receptive fields are applied:

[0122] U0=X,U i =Conv i dw (U i-1 (28)

[0123] Where Conv i dw (·) is a nucleus k i and expansion d i Depthwise convolution. Assume there are N decomposition kernels, each consisting of a 1×1 convolutional layer Conv. 1×1 (·) Further processing:

[0124]

[0125] This allows for channel mixing of each spatial feature vector. Then, a selection mechanism is used to dynamically select the kernel for various objects based on the obtained multi-scale features, which will be described below.

[0126] 2) Spatial kernel selection

[0127] To improve the network's ability to focus on the most relevant spatial context regions for target detection, a spatial selection mechanism is used to spatially select feature maps from large convolutional kernels of different scales. First, features obtained from different kernels are concatenated with features from different receptive field ranges:

[0128]

[0129] Then, channel-based average pooling and max pooling are used, represented as Pavg(·) and Pmax(·) respectively.

[0130]

[0131] SAavg and SAmax are the average and maximum pooled spatial feature descriptors, respectively. To allow information exchange between different spatial descriptors, the spatially pooled features are concatenated and a convolutional layer F is used. 2→N(·) Convert the aggregated features (with 2 channels) into N spatial attention maps:

[0132] SA = F 2→N ([SAavg,SAmax]) (32)

[0133] For each spatial attention diagram SA i The sigmoid activation function is applied to obtain a separate spatial selection mask for each decomposition kernel:

[0134] SA i =σ(SA) i (33)

[0135] in σ Let S represent the sigmoid function. Then, the features from the decomposed large kernel sequence are weighted by their corresponding spatial selection masks and fused through a convolutional layer F(·) to obtain the attention features S:

[0136]

[0137] The final output of the LSK module is the element-wise product between the input features X and S:

[0138] Y = X·S (35)

[0139] The LSK module has two advantages. First, it explicitly generates multiple features with various large receptive fields, making subsequent kernel selection easier. Second, sequential decomposition is more efficient than simply applying a single, larger kernel. Under the same theoretical receptive field, decomposing the convolution kernel significantly reduces the number of parameters compared to standard large convolution kernels.

[0140] CEAM Branch: The CEAM branch aims to seek the intrinsic connections between adjacent layers, adaptively fusing valuable features through a cross-layer attention mechanism. It can be computed as...

[0141]

[0142] in This is the output of the CEAM branch, and sig(·) is the sigmoid activation function. It is a pixel-by-pixel multiplication. Guided by the features of the i-th layer, this branch provides more valuable contextual interaction information for fusion.

[0143] Branch ensemble: After the above effective calculations, the outputs of LSK and CEAM are added to the original i-th layer feature map as residuals.

[0144]

[0145] in It involves summing pixel by pixel. In summary, the features of the i-th layer are adaptively adjusted through intra-layer and cross-layer attention mechanisms.

[0146] To further improve the network's detection performance, we extend AMFF to a bottom-up path suitable for PAFPN, denoted as AMFF*. AMFF* uses P i and P i-1 As input, the transformation can be represented as

[0147] L i =F'(P i ,P i-1 ), i = 3, 4, 5 (38)

[0148] in This is the output of AMFF*. In AMFF*, the execution of the LSK branch is similar to that in AMFF. LSK starts with P... i Let the input be denoted as and the output be denoted as . The output of CEAM is:

[0149]

[0150] Then the output L of AMFF* i It can be calculated as:

[0151]

[0152] The structure of AMFF* is similar to Figure 5 AMFF in. In Figure 5 In the middle, the current feature map C i Replace with P i Map adjacent features to C i+1 Replace with P i-1 AMFF* utilizes cross-attention in a bottom-up manner to integrate high-level semantic information with low-level localization information. With the help of AMFF*, the performance of PAFPN will be further improved.

[0153] Step (5): Finally, the features output by the AMFF* module are upsampled and concatenated. At this point, the feature resolution is 64×48. A 1×1 convolution is then performed to obtain a keypoint heatmap, which is then corrected using unbiased data processing. The loss function is defined as mean squared error to evaluate the model's performance.

[0154] The aforementioned technical solutions effectively acquire the semantic and spatial information required for human pose estimation. The attention-guided deep semantic injection module effectively fuses scale-aware global semantics with corresponding local features, enhancing the model's ability to represent key points semantically. Secondly, the adaptive multi-scale feature fusion (AMFF) module, by integrating a selectable convolutional kernel module (LSK) and a cross-layer interaction mechanism, dynamically adjusts the receptive field size, captures richer multi-scale visual cues, and enhances the correlation between shallow and deep features, effectively reducing information loss during feature fusion. Through the synergistic operation of these techniques, the model's ability to understand, analyze, and accurately estimate complex human poses is greatly enhanced.

[0155] In the embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can also be implemented in other ways. The apparatus, devices, and methods embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0156] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A human pose estimation method based on semantic enhancement and multi-scale feature fusion, characterized in that, The method comprises: Step (1): constructing a feature pyramid on the basis of a MobileNetV2 network, unifying the resolutions of different scale features and splicing; Step (2): inputting new features spliced after multi-scale features in the feature pyramid into an EfficientViT module, and extracting scale perception semantics; Step (3): constructing a deep semantic injection module, fusing scale perception semantics and corresponding features, and adopting a context-guided attention mechanism to relieve semantic gaps and selectively strengthen features related to human body posture key points; Step (4): designing a multi-scale feature fusion module with adaptive receptive fields, and extracting and fusing effective information in different scale features; Taking FPN as a baseline, the fusion mode in the top-down path is improved through an adaptive multi-scale feature fusion (AMFF) module, so that the model focuses on feature adjustment between the current layer and the adjacent layer, and the transformation completed by the AMFF is represented as F(·), and the AMFF can be represented as follows: P i = F(C i ,C i+1 ), i = 2, 3, 4 wherein is the output of the AMFF, C i and C i+1 are the inputs of the AMFF, which represent the feature maps of the i-th layer and the i+1-th layer, respectively; The AMFF module comprises: a selectable convolution kernel (LSK) branch, a cross-layer interaction CEAM branch and a branch integration; The optional convolution kernel (LSK) branch generates multi-scale features with rich context information through a series of deep separable convolutions with different receptive fields, and adopts a spatial kernel selection mechanism to adaptively select the most relevant features, input features C i After the series of transformation processes of the LSK branch, the output features obtained are denoted as The CEAM branch aims to seek the internal relationship between adjacent layers, and can fuse low-layer fine-grained features with top-layer coarse-grained features through a cross-layer attention mechanism to improve the feature fusion effect, which can be calculated as: wherein is the output of the CEAM branch, sig(·) is the sigmoid activation function, is the pixel-wise multiplication, under the guidance of the i-th layer feature, this branch provides more valuable context interaction information for fusion; The outputs of the LSK and the CEAM are fused with the original features through a residual integration mode to realize adaptive feature adjustment within the layer and across the layers, and the outputs of the LSK and the CEAM are integrated with the original i-th layer feature map in a residual mode as follows: wherein is a pixel-wise summation; AMFF extends to the bottom-up paths applicable in PAFPN, denoted as AMFF*, AMFF* with P i and P i-1 as inputs, the transformation can be represented as: L i = F'(P i , P i-1 ), i = 3, 4, 5 where is the output of AMFF*, in which LSK is transformed with P i as input to get the output, denoted as the output of CEAM is The output L of the AMFF* is then i may be calculated as: The AMFF is extended to AMFF*, and applied to the bottom-up feature fusion path of the PAFPN, which can effectively fuse high-layer semantic information with low-layer positioning information; Step (5): splicing the multi-scale features output in step 4, generating the final key point heat map using a 1x1 convolution layer, and correcting the heat map using unbiased data processing, and then predicting the human body key points. 2.The human pose estimation method based on semantic enhancement and multi-scale feature fusion according to claim 1, characterized in that, The feature pyramid is constructed by using a feature extraction backbone network MobileNetV2, comprising: On the basis of the MobileNetV2 network, the feature outputs of the second layer, the third layer, the fifth layer and the seventh layer of the MobileNetV2 network are constructed into a feature pyramid to obtain different scale features {F1, F2, …, F N}, wherein N=4 represents the number of scales; and the features {F1, F2, …, F N} are averaged and collected to the resolution size of the output of the seventh layer of the feature pyramid. 3.The human pose estimation method based on semantic enhancement and multi-scale feature fusion according to claim 2, characterized in that, The scale perception semantic extraction module based on EfficientVit comprises: The module is composed of a memory-efficient sandwich layout, a cascaded group attention module and a parameter redistribution strategy, which respectively improves the model efficiency from the aspects of memory, calculation and parameters, and the implementation process is as follows: 1) Sandwich layout: Specifically, it applies a single attention layer Φ A between FFN layers Φ F with the formula: X i+1 = Φ F (Φ A (Φ F (X i ))) where X i and X i+1 denote the input and output features of the ith building block, respectively, Φ F is composed of a feed-forward neural network (FFN) layer and a deep convolution, where the FFN realizes the information interaction among different feature channels, and the deep convolution replaces the positional embedding encoding by introducing an inductive bias of local structure information, Φ A is a cascaded group attention mechanism; 2) Cascaded group attention: providing different partitions of complete features for each head, thereby explicitly decomposing cross-head attention calculation, and the mechanism of the attention is expressed as: The jth attention head computes self-attention, which acts on X ij , i.e., the jth partition of input features X i , i.e., X i = [X i1 , X i2 , …, X ih ], 1≤j≤h, where h is the total number of attention heads, W ij Q , W ij K , and W ij V are projection layers that map the input feature partitions to different subspaces, W i P is a linear layer that projects the concatenated output features back to the dimension consistent with the input, the self-attention mechanism of EfficientVit computes the attention map for each head in a cascading manner, which adds the output of each head to the subsequent head to gradually improve the feature representation: X ' ij is the j-th input split X ij and the output of the (j-1)-th head the result of the post-addition computation, replacing X ij becomes the new input feature for the j-th head when computing self-attention, after Q projection another token interaction layer is applied; 3) Parameter redistribution: reducing the channel dimension of Q and K projection of each head, setting the dimension of V projection to be consistent with the dimension of input embedding, expanding the FFN ratio from 4 to 2, and weakening parameter redundancy.

4. The human pose estimation method based on semantic enhancement and multi-scale feature fusion according to claim 3, characterized in that, The deep semantic injection module comprises: After obtaining the scale-aware semantics, the global semantic features are divided into 4 parts by channel, and fused with the features {F1, F2, …, F N} to inject deep semantics into the local features of the feature pyramid. The semantic injection module takes the local features of the feature pyramid module and the global semantics of the visual Transformer as input; the local label X l is fed into a 1x1 convolutional layer, followed by batch normalization to produce the features f local to be injected; the global semantic X g is fed into a 1x1 convolutional layer, followed by a batch normalization layer and a sigmoid layer to produce the semantic weight global_weight, while the global semantic is also passed through a 1x1 convolutional layer, followed by batch normalization to obtain the feature f global The three outputs have the same number of channels; the global semantic is injected into the corresponding local label to obtain the preliminary injected feature f injected : f injected = f local *global_weight In addition, the context-guided attention (CGA) module introduces global semantic features f global add to the injection features f injected obtained in the foregoing, further enhances the feature representation, and the CGA module assigns a unique spatial importance weight to each feature channel. The CGA fuses channel attention and spatial attention, enabling local and global features to fully interact and fuse in both channel and spatial dimensions. f injected and f global are input into the CGA module, which computes the weights of the feature modulations for each channel, and simultaneously processes the low-level features f injected and the corresponding high-level features f global to capture the relationships between them and reflect the importance of each spatial location. W = CGA(f injected +f global ) The f injected and f global are fused by means of a weighted sum, the formula being as follows: X fused = Conv 1x1 (f injected * W + f global * (1 - W) + f injected + f global ) wherein f injected has a weight of W, and f global has a weight of (1-W).

5. The human pose estimation method based on semantic enhancement and multi-scale feature fusion according to claim 4, characterized in that, The key point heat map generation comprises: Finally, the features passing through the AMFF / AMFF* module are upsampled and spliced, a 1x1 convolution layer is used to obtain a key point heat map, and the heat map is corrected using unbiased data processing, and then the key points of each part of the human body are predicted.

Images