A lightweight human pose estimation system and method fusing channel and spatial activation

Abstract

The present application relates to the technical field of human posture recognition, solves the technical problems that the traditional method faces the multi-scale fusion and increases the calculation burden, and has difficulties in realizing efficient network operation, especially relates to a light-weight human posture estimation system and method fusing channels and space activation, input image is input into EDCNet network, and then the key point position of human posture in the input image is obtained through output. The EDCNet network fuses the extraction process of local and global features, realizes the fusion of different resolution information under double scales, integrates a spatial segmentation module and a double-scale cross attention module. The present application can improve the accuracy of human key point estimation without stacking the traditional Transformer, and realizes a better balance between efficiency and accuracy compared with the traditional stacked ViT architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human pose recognition, and in particular to a lightweight human pose estimation system and method fusing channel and spatial activation. BACKGROUND

[0002] Human pose estimation is a key task in the field of computer vision, and its main goal is to accurately identify the key points of the human body in an image. The accuracy and efficiency of pose estimation are closely related to human measurement, and the key lies in providing reliable, real-time and accurate human measurement results in various environments, which is particularly important for practical applications such as robot motion, human-computer interaction, human tracking and action recognition.

[0003] Convolutional Neural Networks (CNN) are robust to rotation, scaling and translation, providing a new solution for human key point positioning. Deep Pose Network (Deeppose) is the first to use CNN for human pose estimation, overcoming the complexity of manually constructing a diagram structure model. Subsequently, the Pyramid Network (CPN) adopts a cascaded pyramid structure and achieves significant results on the COCO dataset. Then, the High-Resolution Network (HRNet) improves the pose estimation performance through multi-scale fusion technology, which can integrate multi-resolution information to enhance the feature expression ability.

[0004] Due to the effectiveness of multi-scale fusion, various research fields have also begun to introduce multi-feature local information representation methods into various applications. However, we found that directly applying multi-resolution fusion in human joint positioning increases the network burden and still has room for optimization. Unlike these strategies, we focus on how to balance the accuracy and efficiency of the network at different resolutions.

[0005] In CNN, multi-scale fusion is used to extract local features at different resolutions, while Transformer has a significant advantage in obtaining global information and has been successfully applied to tasks such as object detection. Similarly, Transformer has achieved comparable accuracy to convolutional methods in human pose estimation. Directly combining the basic Vision Transformer (ViT) with convolution helps to balance efficiency and accuracy in key point recognition, and by enhancing the capacity of Transformer, the accuracy of key point recognition can be improved, which is similar to the case in object detection and instance segmentation. However, at the same time, with the deepening of the self-attention module, the network efficiency will also decrease. SUMMARY

[0006] In view of the deficiencies of the prior art, the application provides a lightweight human posture estimation system and method fusing channels and spatial activation, which solves the technical problems that the traditional method increases the computational burden when facing multi-scale fusion and has difficulty in realizing efficient network operation.

[0007] To solve the above technical problems, the application provides the following technical solutions: a lightweight human posture estimation system fusing channels and spatial activation, which obtains the key point positions of human posture in an input image based on an EDCNet network;

[0008] The EDCNet network comprises:

[0009] The spatial division module takes a feature map as input, is used for dividing channels to obtain two groups of high-resolution features and low-resolution features, and is used for dividing channels to obtain two groups of high-resolution features and low-resolution features.

[0010] In addition, the double-scale CSViT is applied on two different scales to construct a cross attention mechanism, so as to extract and fuse the local and global feature information across spaces, and realize a double-scale cross attention module for human posture estimation.

[0011] Further, the double-scale CSViT has two resolution CSViT structures, namely a high-resolution CSViT and a low-resolution CSViT, and is connected through cross attention between two different resolutions, which is used for obtaining feature information of different scales and promoting feature perception effect between different scales.

[0012] Further, the CSViT structure comprises:

[0013] The channel normalization unit is used for keeping the dimension constant without expanding the D-dimensional length, and the expression is: After the flattening processing, a two-dimensional patch sequence is obtained

[0014]

[0015] wherein, represents the two-dimensional patch sequence output after normalization processing; LN represents layer normalization; Lin represents a linear layer;

[0016] In addition, the spatial activation unit based on local and global information captures short-range and long-range dependencies in an image, which is used for identifying key human joint points in the image, and the expression is:

[0017]

[0018] The local feature information output by the spatial activation unit ​By multiplying the value V with the result of the influence of the query Q and the key K and adjusting the dimension through convolution and dimension conversion, the expression is:

[0019]

[0020] wherein, represents a two-dimensional patch sequence processed by a spatial activation unit; is a function; represents a 1x1 convolution; represents a matrix multiplication; RE represents a dimension conversion operation; and HP is a hyperparameter;

[0021] and a channel activation unit further activates the channels from the spatial information extracted from the spatial activation unit, for fusing the information of different channels at the same spatial position to enhance the channel perception ability, and the expression is:

[0022]

[0023] wherein, represents a 3x3 convolution; is an activation function.

[0024] The technical scheme also provides a human pose estimation method based on the above-mentioned lightweight human pose estimation system, and the method comprises the following steps:

[0025] S1, extracting a heat map feature of an input image through a CNN backbone network to obtain a feature map , wherein , H and W are the height and width of the input image , respectively; , H, W and C are the height, width and channel of the feature map , respectively; S2, dividing the channels through a spatial division module to obtain two groups of double-scale features, i.e., high-resolution features and low-resolution features;

[0026] S3, adopting a double-scale CSViT to construct a cross-attention mechanism for realizing information transmission between different spatial scales;

[0027] S4, combining the cross-attention mechanism through a double-scale cross-attention module to extract and fuse the local and global feature information across the space to obtain the key point positions of the human pose in the input image

[0028] .

[0029] Further, the CNN backbone network adopts a ResNet50 network. ​

[0030] Further, in step S2, the specific process includes the following steps:

[0031] Firstly, the feature map is down-sampled to obtain two resolution features, i.e., a high-resolution feature and a low-resolution feature .

[0032] Subsequently, the high-resolution feature and the low-resolution feature are spatially divided along the channel dimension to generate four spatial features, expressed as:

[0033]

[0034]

[0035] wherein Split[ ] represents a channel separation operation; both represent the high-resolution feature; both represent the low-resolution feature.

[0036] Further, the spatial dimensions of the high-resolution feature and are both , while the spatial dimensions of the low-resolution feature and are both .

[0037] Further, in the spatial splitting module, network efficiency is improved by compressing the spatial dimension, and the specific settings are as follows:

[0038]

[0039] wherein are the height and width of the input image , respectively; are the height and width of the feature map , respectively.

[0040] Further, in step S3, the specific process includes the following steps:

[0041] Firstly, the high-resolution feature and are obtained from the spatial splitting module, and the low-resolution feature and are obtained by down-sampling;

[0042] Then, any high-resolution feature and any low-resolution feature are input into a dual-scale CSVit; wherein high-resolution features 、 are input into a high-resolution CSVit and low-resolution features are input into a low-resolution CSVit respectively and processed in parallel;

[0043] When processing low-resolution features , low-resolution sequences , , are obtained; then, spatial learning is performed on the low-resolution sequences and through dimension transformation and 1x1 convolution, and spatially expanded low-resolution sequences and are obtained through padding upsampling;

[0044] Meanwhile, high-resolution features are processed to obtain high-resolution sequences , wherein ; then, the low-resolution sequences and are input into a high-resolution CSVit together with the high-resolution sequences to obtain sequences through cross-attention;

[0045] Meanwhile, the low-resolution sequences , , are processed through a low-resolution CSVit to obtain sequences ;

[0046] Finally, the sequences are upsampled and added to the sequences to obtain the final cross-attention output, expressed as:

[0047]

[0048] wherein, represents element-wise addition; represents an upsampling operation.

[0049] By means of the above technical solution, the present application provides a lightweight human pose estimation system and method fusing channel and spatial activation, which at least has the following beneficial effects:

[0050] 1. The human key point estimation precision can be improved without stacking the traditional transformer, the latest performance level is achieved compared with the prior art method, and the speed is fast, more rapid and light.

[0051] 2. The double-scale cross-attention module uses a double cross-attention mechanism to combine CSViT to model local and global dependency relationships. The spatial division module efficiently extracts multiple spatial features and seamlessly connects with the double-scale cross-attention module to form the EDCNet network, achieving a balance between accuracy and efficiency.

[0052] 3. The application proposes a new module CSViT, which includes a spatial and channel activation unit, and enhances the extraction capability of local and global features. The CSViT replaces the traditional ViT, and the posture estimation precision can be improved without stacking, and a channel normalization unit is introduced to improve its adaptability to image sequences.

[0053] 4. The application constructs a double-scale CSViT through different scale CSViTs, and performs cross-attention inside the double-scale CSViT. Based on the local and global feature extraction capability of a single CSViT, the double-scale cross-attention further enhances the local and global feature processing capability under different scales. This novel way has better human joint information capture effect.

[0054] 5. The method proposed in the application is superior to the prior art method on COCO and MPII data sets. The EDCNet network obtains higher accuracy with fewer parameters and faster speed on the COCO validation set, the average accuracy reaches 74.6, and the image processing speed on the RTX2080Ti graphics card is more than 160 frames per second, which is higher than the advanced method at the same period. BRIEF DESCRIPTION OF DRAWINGS

[0055] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0056] Figure 1 The flowchart of the human pose estimation method in the application;

[0057] Figure 2 The network structure diagram of the EDCNet network in the application;

[0058] Figure 3 The network structure diagram of the CSViT in the application;

[0059] Figure 4 The flowchart of the human pose estimation based on the EDCNet network in the application;

[0060] Figure 5 Network structure diagram for the cross-attention mechanism composed of double-scale CSVit in the present application;

[0061] Figure 6 Example diagram of the visualization result of the present application on the COCO dataset;

[0062] Figure 7 Example diagram of the display result comparison between the present application and the existing method SHaRPose-S. DETAILED DESCRIPTION

[0063] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments. The implementation process of how to apply technical means to solve technical problems and achieve technical effects of the present application can be fully understood and implemented.

[0064] Although the prior art has made significant progress in accuracy, it still faces challenges in the following two aspects:

[0065] 1) Multi-scale fusion increases the computational burden: Single-scale methods are difficult to capture information at different resolutions, so high-resolution networks significantly improve the accuracy of pose recognition through multi-scale fusion. These methods always maintain high resolution in the network and combine multiple resolutions (usually more than three) to obtain detailed information. However, this fusion strategy needs to process multiple high-resolution inputs, significantly increasing the computational load of the network. Therefore, it is still a challenge to improve the efficiency of multi-scale fusion while ensuring accuracy.

[0066] 2) Difficulty in implementing efficient operation of stacked Transformer architecture: Given the limitations of CNN in capturing global information, existing technologies use the self-attention mechanism of Transformer to capture the overall dependency in the image, but it is difficult to achieve ideal results, indicating that relying on a small number of Transformer structures cannot achieve better performance. Subsequently, researchers combined multi-scale fusion with Transformer, not only increasing the number of Transformer layers, but also strengthening the fusion strategy between different scales. Although this method improves the accuracy of pose estimation, the network speed decreases significantly, and the efficiency problem remains unsolved.

[0067] To address the above challenges, the present application rethinks the extraction strategy of image information, namely a lightweight human state estimation system and method fusing channel and spatial activation. The method is realized based on an efficient dual-scale cross-attention network (EDCNet) to solve the technical problems faced by the prior art, namely:

[0068] Change the original stack Transformer method to make the network achieve faster speed and better performance with less overhead.

[0069] Change the original multi-scale fusion feature transmission method to ensure effective fusion of different scales in a low network overhead form, so that the present application can effectively solve the balance problem between human pose estimation accuracy and efficiency compared with the prior art.

[0070] Embodiment one

[0071] The present embodiment proposes a lightweight human pose estimation system fusing channel and spatial activation. The core idea of the present embodiment is to input the input image into the EDCNet (Efficient Dual-scale Cross-attention Network) network, and then obtain the key point position of the human pose in the input image through output. The EDCNet network fuses the extraction process of local and global features to realize the fusion of different resolution information under dual scale.

[0072] Although the prior art can improve the accuracy of human pose estimation by increasing the width and depth of the original Transformer, it will sacrifice network speed. To solve this problem, the present application proposes an efficient dual-scale cross-attention network (EDCNet) for human key point positioning, and introduces a space split module (SSM) and a dual-scale cross-attention module (DCM). As shown in Figure 2 The EDCNet network integrates a space split module (SSM) and a dual-scale cross-attention module (DCM), which achieves a better balance between efficiency and accuracy compared with the traditional stacked ViT architecture.

[0073] The space splitting module takes a feature map as input, and is used to divide the channels to obtain two groups of high-resolution features (HF) and low-resolution features (LF). The double-scale cross-attention module uses a double-scale CSViT to construct a cross-attention mechanism at two different scales, so as to extract and fuse the local and global feature information across the space, and realize more accurate human pose estimation.

[0074] Firstly, a parallel space is obtained by the space splitting module. The core component of the double-scale cross-attention module is the CSViT, which can capture short-range and long-range dependency relationships. Then, the double-scale CSViT is used to obtain cross-attention. Finally, two cross-attention mechanisms are applied in the double-scale cross-attention module to extract and fuse the local and global feature information across the space, so as to realize accurate human pose estimation. The double-scale cross-attention module captures local and global feature dependency relationships between different resolutions through innovative strategies. In this module, the low-resolution Key and Value are fused with the Query from the high-resolution through the double-scale CSViT. The present embodiment applies the DCM in multiple spaces to mine deeper cross-space correlations.

[0075] The double-scale CSViT has two resolution CSViT structures, i.e. a high-resolution CSViT and a low-resolution CSViT, and is connected through cross-attention between two different resolutions, so as to obtain feature information of different scales and promote feature perception effects between different scales.

[0076] Specifically, the core of the double-scale cross-attention module is a new visual self-attention mechanism, the Channel-Spatial Vision Transformer (CSVit), which includes two groups of double-scale CSViT in the double-scale cross-attention module. Each group of CSViT has two resolution CSViT, i.e. a high-resolution CSViT and a low-resolution CSViT in Figure 2 The two are connected through cross-attention between two different resolutions, so as to obtain feature information of different scales, and the cross-attention can promote feature perception effects between different scales. Therefore, the cross-attention mechanism composed of the double-scale CSVit enables the double-scale cross-attention module to capture local and global information across the space.

[0077] The CSVit fuses independent spatial information from Query, Key and Value through a spatial activation unit, effectively integrating short-range and long-range dependency relationships. At the same time, the present embodiment introduces a channel activation unit in the CSVit, which enhances the channel perception ability through continuous convolution.

[0078] The implementation basis of the dual-scale cross-attention module is a CSViT structure, which is the core of the present application, as shown in Figure 3 The dual-scale CSViT is composed of two CSViTs of different scales, and the last two groups of dual-scale CSViTs are added together to implement the dual-scale cross-attention module. Inside the different resolution CSViTs, their structures are completely the same. Among them, the CSViT structure includes a channel activation unit (CAU), a spatial activation unit (SAU), and a channel normalization unit (CNU). The present embodiment constructs a dual-scale CSViT through a CSViT structure of different scales, and performs cross-attention inside the dual-scale CSViT. On the basis of the local and global feature extraction capability of a single CSViT, the dual-scale cross-attention further enhances the local and global feature processing capability under different scales. This novel approach has better effect on capturing human joint information.

[0079] The advantages of the two modules in CSVit are as follows: one is that in the process of global interaction, spatial interaction is realized for K and Q to complete local feature fusion and better realize short and long distance feature acquisition. The other is to change the channel (first increase and then decrease) after the previous spatial expansion, and the spatial size remains unchanged, realizing the continuous fusion of features in the same spatial position in different channels, improving the spatial perception in the channel, and not increasing the additional calculation amount. The proposal of CSVit is conducive to realizing fast recognition of human joints in real scenes and better balancing the efficiency and effect of human pose estimation.

[0080] First, the present embodiment normalizes the input sequence information by using a channel normalization unit. Then, the spatial activation unit captures short-range and long-range dependency relationships. Finally, the channel activation unit enhances the channel perception capability and improves the overall performance of the CSVit.

[0081] For simplicity, it is assumed that the initial size of the input to the CSViT is , and a two-dimensional patch sequence is obtained after flattening processing, where = , represents the number of patches into which the image is divided, , represents the resolution of each feature block patch. CSVit uses a fixed latent vector dimension D in all layers, and projects the patch with a size of to a D-dimensional space through a trainable linear transformation.

[0082] Unlike the traditional Transformer, the embodiment replaces the original self-attention mechanism and MLP with a spatial activation unit (SAU) and a channel activation unit (CAU), respectively. The spatial activation unit (SAU) is used to reduce the length of the input sequence, while the channel activation unit (CAU) iteratively adjusts the number of channels after reshaping the input sequence. To further manage the length of the input sequence, the embodiment also introduces a channel normalization unit (CNU) in the CSVit.

[0083] The channel normalization unit normalizes the feature map , that is, after flattening, a two-dimensional patch sequence is obtained to maintain a constant dimension without expanding the D-dimensional length . Specifically, the embodiment introduces a channel normalization unit to solve the irregular sequence length problem in CSVit before inputting the two-dimensional patch sequence into the model. After inputting K, Q, and V, the input channels are smoothed to improve channel perception. This improves the recognition effect of the EDCNet network on human pose positions. When the input sequence is NxD, the channel normalization unit normalizes the length of dimension D. The channel normalization unit maintains a constant dimension without expanding the D-dimensional length, and its calculation method is as follows:

[0084]

[0085] where LN represents layer normalization (Layer Normalization), and Lin represents a linear layer. The embodiment makes the mean of each patch (patch) in the sequence zero and the standard deviation one through two layer normalizations, thereby completing the normalization of the sequence; the linear layer enhances the expression ability of dimension D. In order to reduce the additional computational overhead as much as possible, the embodiment keeps the dimension D unchanged, so that the normalized sequence is more suitable for human pose recognition tasks.

[0086] The spatial activation unit captures short-range and long-range dependencies in images based on local and global information, and is used to identify key human joint points in images. Specifically, the spatial activation unit generates three matrices of the same size: Key (K), Query (Q), and Value (V), all of which have a dimension size of . First, the D-dimensional channels in K and V are flattened to , and then a 1x1 convolution is used to reduce the dimension of the one-dimensional vector from N=256 to M=128. In the experiment on the COCO dataset, P=4, D=256, h=4, ​= 64. Then, Q and K are multiplied to capture the global spatial dependency. After that, 1x1 convolution and Softmax function are used to determine the relationship between local features and the influence of Q and K on global features, and the expression is:

[0087]

[0088] wherein, represents 1x1 convolution; represents matrix multiplication; RE represents a reshape operation. The formula expresses the interaction of Q and K, and generally, Q and K are global matrices for realizing global interaction, while the convolution Conv of the present application realizes local interaction, so that the formula simultaneously incorporates convolution operation in Q and K for global interaction, and the dimension change realizes the switching between the three-dimensional matrix of convolution and the two-dimensional matrix of QK, so that the spatial activation unit can realize sufficient interaction of local and global information.

[0089] Finally, the local feature information output by the spatial activation unit is The result of the influence of V on Q and K is multiplied and the dimension is adjusted by convolution and dimension conversion, and the expression is:

[0090]

[0091] wherein, the hyperparameter HP = 256. The intermediate matrix obtains the spatial size through dimension conversion RE, and then uses convolution to extract local feature information. On the basis of sufficient interaction of global and local information, the value V is used to calculate the value of each key point in the image. By calculating the interaction between each pixel point, the relationship between different key points and the probability that the image feature belongs to which key point are obtained, and then V is calculated, that is, the position of the key point on the feature map belongs to which specific joint is better obtained. In fact, Q, K and V are the same at the beginning, only after the two formulas, Q and K first calculate the relationship between key points and the probability of position, and then use this result to assist V to better obtain the position of the key point.

[0092] Through the above calculation, the spatial activation unit can capture short-range and long-range dependencies in the image based on local and global information, which helps to identify key human joint points in the image, whether it is an isolated joint or a related joint.

[0093] The channel activation unit performs dimension conversion to further activate the channel from the spatial information extracted from the spatial activation unit, which is used to fuse the information of different channels at the same spatial position to enhance the channel perception ability. Specifically, after obtaining the local feature information After that, the channel activation unit performs dimension transformation to further activate the channel from the spatial information extracted by the spatial activation unit. Unlike the linear layer, the channel activation unit uses convolution to realize channel activation, with the expression as follows:

[0094]

[0095] First, the spatial size is expanded to twice the original size using a 3x3 convolution; then, the local spatial information is extracted using a 1x1 convolution, activated by the activation function GeLU, and finally restored to the spatial size by another 1x1 convolution. This process enables the channel activation unit to capture spatial information while activating the channel. After completing the dimension transformation, the spatial feature information output by the channel activation unit is obtained At this time, the spatial size of the local feature information can be regarded as the sequence length of the spatial feature information , and the dimension of the local feature information is the same as that of the spatial feature information .

[0096] In the channel activation unit, local information is extracted by convolution, rather than directly inputting the local feature information into the MLP to obtain it. During the calculation of the input local feature information , the number of channels changes (first increases and then decreases), while the spatial size remains unchanged. This design enables the fusion of information from different channels at the same spatial position, thereby enhancing the channel perception ability.

[0097] It is worth noting that the proposed CSViT has the following differences from the basic ViT (i.e., the Transformer mentioned earlier):

[0098] CSViT contains a spatial activation unit (Spatial Activation Unit) that fuses the independent spatial information of Query, Key, and Value, enabling the model to efficiently combine short-range and long-range dependencies. While reducing computational overhead, CSViT enhances the ability to obtain local information, thereby improving the overall learning ability of the model during global interaction.

[0099] This embodiment replaces the traditional MLP with a channel activation unit (Channel Activation Unit), enabling CSViT to further learn local features after obtaining global interaction information. By introducing a channel perception mechanism, the model can more accurately perceive the feature information of key points at specific positions.

[0100] To avoid the irregularity of sequence length in CSViT, a channel normalization unit is introduced to ensure the adaptability of CSViT in image sequence operation.

[0101] In the EDCNet network, the spatial segmentation module delivers two groups of image features with different resolutions to the dual-scale cross-attention module, which extracts and fuses dual-scale information using two cross-attention mechanisms. This embodiment uses deconvolution to restore the key point positions, achieving a good balance between efficiency and effectiveness.

[0102] Embodiment Two

[0103] This embodiment proposes a human pose estimation method that fuses channel and spatial activation. The EDCNet network processes input images through a spatial segmentation module (SSM) and a dual-scale cross-attention module (DCM) to identify human pose key points. The spatial segmentation module (SSM) can obtain multiple spatial features through a low-overhead spatial division method and output the required features for different scales for the dual-scale cross-attention module (DCM). The core of the dual-scale cross-attention module (DCM) is CSViT, which aims to extract local and global information from heatmaps. Subsequently, we introduce a cross-attention mechanism in the dual-scale cross-attention module (DCM) to achieve accurate positioning of human joints through dual-scale CSViT, ensuring that the EDCNet network can capture cross-scale spatial feature correlations with low computational overhead. As shown in Figure 1 The method comprises the following steps:

[0104] S1, extracting heat map features of an input image through a CNN backbone network to obtain a feature map , wherein are the height and width of the image , respectively, are the height, width, and channel of the feature map . As shown in Figure 4 , the CNN backbone network in this embodiment uses a ResNet50 network.

[0105] S2, dividing the channels through a spatial segmentation module (SSM) to obtain two groups of dual-scale features, namely high-resolution features (HF) and low-resolution features (LF). The specific process includes:

[0106] First, downsample the feature map to obtain two resolution features, namely high-resolution feature and low-resolution feature .

[0107] Subsequently, high-resolution features and low-resolution features are spatially divided along the channel dimension to generate four spatial features, expressed as:

[0108]

[0109]

[0110] where Split[ ] represents a channel separation operation; all represent high-resolution features; all represent low-resolution features. The spatial dimensions of the high-resolution features and are both , and the spatial dimensions of the low-resolution features and are both .

[0111] Through the spatial splitting module (SSM), the embodiment realizes spatial division at a lower computational cost, obtains multiple spatial features, and provides direct dual-scale input for the subsequent dual-scale cross-attention module (DCM). In the spatial splitting module (SSM), the embodiment improves network efficiency by compressing the spatial dimension, and is specifically configured as:

[0112]

[0113] S3, a dual-scale CSViT is adopted to construct a cross-attention mechanism (CA) for realizing information transmission between different spatial scales. As shown in Figure 5 , the cross-attention structure composed of the dual-scale CSVit includes two cross-attention mechanisms (CA) and can provide cross-space local and global information on the dual scale. First, the CSVit can effectively capture short-range and long-range dependencies. In addition, the cross-attention fuses global and local information on different scales, thereby guiding the dual-scale cross-attention module (DCM) to discover the correlation between key points between different scales.

[0114] In Figure 5 , by using two CSVit at different resolutions, the K and V of the low-resolution features are upsampled to the high-resolution, realizing the cross-attention mechanism. Finally, the high-resolution output fuses the low-resolution features, effectively capturing the detailed information of the high-resolution and the global features of the low-resolution. A cross-attention structure is obtained.

[0115] Since the spatial segmentation module outputs two high-resolution features and two low-resolution features, two sets of features can be obtained, namely one high-resolution and one low-resolution. These two sets of features are respectively input into the cross-attention structure, and after obtaining feature heatmaps, they are added together to obtain the final output, which is the dual-scale cross-module structure of this application.

[0116] For convenience, this embodiment... Figure 5 The other set of cross-attention structures with the same structure has been omitted. The following is a general introduction to the dual-scale cross module in conjunction with the feature acquisition and processing process:

[0117] First, two high-resolution features are obtained from the spatial segmentation module. and Meanwhile, low-resolution features are obtained through downsampling. and .

[0118] Then, any high-resolution feature and any low-resolution feature The inputs are combined into a dual-scale CSVit. High-resolution features are included. 、 With low-resolution features Input to high-resolution CSVit and low-resolution CSVit respectively Parallel processing.

[0119] Processing low-resolution features At that time, low resolution Obtain low-resolution sequence , , Subsequently, the low-resolution sequence was processed through dimensionality transformation and 1×1 convolution. and Spatial learning is performed, and spatially expanded low-resolution sequences are obtained by padding upsampling. and Meanwhile, from high resolution Processing high-resolution features Obtain high-resolution sequences ,in Next, the low-resolution sequence and With high-resolution sequences Input together to high resolution Cross attention is performed to obtain the sequence Meanwhile, low-resolution sequences , , Through low resolution The processing sequence .

[0120] Finally, the embodiment will sequence After upsampling and adding sequence , the final cross-attention output is obtained, the expression is:

[0121]

[0122] wherein, represents element-wise addition (Element-wise Add); represents the upsampling operation.

[0123] In the embodiment, two double-scale CSViTs implement parallel processing of multi-space features at different scales, and after four spatial outputs, two high resolutions and two low resolutions are divided. As shown in Figure 3 , the two high resolutions and the two low resolutions come from two different split channels, that is, the high and low resolutions are realized when splitting, and then two high and two low resolutions are obtained by splitting the channels, and then one high and one low are input into the double-scale CSViT. The purpose of this process is to interact with different spatial scales.

[0124] S4, the double-scale cross-attention module (DCM) combines two cross-attention mechanisms (CA) to extract and fuse local and global feature information across spaces, and obtain the key point position of the human body in the input image .

[0125] The double-scale cross-attention module (DCM) proposed in the embodiment brings significant advantages to the method, that is:

[0126] The ingenious CSVit and cross-attention strategy are designed, which combines the advantages of high and low resolutions, and effectively captures the cross-space short-range and long-range dependencies of the key nodes in the image;

[0127] The synergistic effect of the double-scale cross-attention module (DCM) and the spatial segmentation module (SSM) enables the EDCNet network to have multi-scale global information, thereby improving the human key point recognition capability.

[0128] Experimental example

[0129] The present application is tested on the COCO (Common Objects in Context) human pose dataset.

[0130] This dataset is one of the widely used benchmark datasets in the field of computer vision, mainly used for human key point detection (Human Pose Estimation) task, and the key point positions of human body are labeled to support 2D pose estimation research of multiple people.

[0131] Validation set (val2017): about 5000 images. For each person, 17 key points are labeled, including: nose, left / right eye, left / right ear, left / right shoulder, left / right elbow, left / right wrist, left / right hip, left / right knee, left / right ankle.

[0132] Evaluation index (OKS): COCO uses Object Keypoint Similarity (OKS) to evaluate the closeness of the prediction result to the labeled data, which is similar to the concept of IoU.

[0133] The method of the present application is compared with two current advanced methods in the COCO val verification set, and the comparison results are shown in Table 1. The accuracy of the present application reaches 74.6, while the overall accuracy of the existing methods MSPENet, HRNet-W32 and SHaRPose-S is 72.7, 73.4 and 74.2 respectively. The results show that the method of the present application is obviously better, and the speed reaches 168 frames per second, which is faster, can more real-time recognize human posture, in addition, the parameter is smaller, only 8.7M, therefore, compared with the current similar method, the method of the present application can achieve faster, lighter, more accurate recognition effect.

[0134] Table 1 Comparison results on COCO val verification set

[0135]

[0136] In Table 1, #Params represents the parameter size, GFLOPs represents the calculation size of the network model, and AP represents the overall accuracy. The speed (FPS, frames per second), average accuracy (AP) and average recall rate (AR) are used as the final evaluation index in this embodiment.

[0137] The comparison methods in the table are all existing known methods, and the sources are as follows:

[0138] MSPENet: J. Xu, W. Liu, W. Xing, and X. J. T. V. C. Wei, "MSPENet: multi-scale adaptive fusion and position enhancement network for human pose estimation," vol. 39, no. 5, pp. 2005-2019, 2023.

[0139] SHaRPose-S: X. An, L. Zhao, C. Gong, N. Wang, D. Wang, and J. Yang, "SHaRPose: Sparse High-Resolution Representation for Human Pose Estimation," in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, vol.38, no. 2, pp. 691-699.

[0140] The quantitative effects of the parts of the present application are shown as follows:

[0141] Table 2 Comparison results of setting a spatial segmentation module

[0142]

[0143] Obviously, without the spatial segmentation module, the average accuracy of the present application is 0.6 percentage points lower.

[0144] Table 3 Comparison results of setting cross attention

[0145]

[0146] Without cross attention, the average accuracy of the present application is also 0.6 percentage points lower. This further illustrates the importance of the two modules of the present application.

[0147] Table 4 Comparison results of setting CSViT units

[0148]

[0149] This experiment confirms the importance and effect of each part of the CSViT of the present application.

[0150] Visualization results (displayed on COCO dataset): The visualization of the present application on COCO dataset: The present application performs well in some complex human body structures, occlusions, environmental disturbances and the like, as shown in Figure 6

[0151] Meanwhile, compared with the advanced method, the method of the present application achieves more accurate human body posture in some specific key red circle positions. As shown in Figure 7

[0152] The method proposed in the present application is superior to the existing method on COCO and MPII datasets. The EDCNet network obtains higher accuracy with fewer parameters and faster speed on the COCO validation set, with an average accuracy of 74.6, and has an image processing speed of more than 160 frames per second on an RTX2080Ti graphics card, exceeding the advanced method of the same period.

[0153] In summary, the present application proposes a lightweight real-time human body posture estimation system and method fusing channel and spatial activation, which is mainly realized by relying on an efficient dual-scale cross-attention network (EDCNet), and improves the accuracy of human key point estimation without stacking traditional Transformers through a spatial segmentation module (SSM) and a dual-scale cross-attention module (DCM). First, the SSM obtains multiple spatial features through low-cost spatial division and provides necessary dual-scale input for the DCM. Second, the DCM is a cross-attention structure realized by a dual-scale CSVit. Unlike the traditional Transformer, the CSVit adopts a spatial activation unit and a channel activation unit to replace the self-attention mechanism and the MLP layer, thereby realizing effective fusion of short-range and long-range dependencies.

[0154] The present application combines the spatial segmentation module (SSM) and the dual-scale cross-attention module (DCM) to capture the correlation of the image at different scales. The multi-space strategy of the SSM helps the DCM to understand the spatial relationship between the key points more deeply, so that the DCM can capture rich local and global information through multi-resolution cross-attention. The experimental results based on the COCO dataset show that the method proposed in the present application reaches the latest performance level compared with the existing method, and is fast, more rapid and lightweight. At the same time, the present application also analyzes each component module of the EDCNet in detail, and verifies its actual contribution in posture estimation.

[0155] ​​Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by instructing relevant hardware through programs, therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment in the form of a combination of software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.

[0156] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0157] The above embodiments have been described in detail, and the principles and implementation manners of the present application are described by applying specific examples; the above embodiment descriptions are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A lightweight human pose estimation system that integrates channel and spatial activation, characterized in that, Key point locations of human pose in input images are obtained based on the EDCNet network; The EDCNet network includes: Using feature maps as input, a spatial segmentation module is used to divide channels and obtain two sets of high-resolution features and low-resolution features; Furthermore, a cross-attention mechanism is constructed by applying dual-scale CSViT at two different scales to extract and fuse local and global feature information across space, thereby realizing a dual-scale cross-attention module for human pose estimation. The dual-scale CSViT has two resolution CSViT structures, namely high-resolution CSViT and low-resolution CSViT, and is connected between the two different resolutions through cross-attention to obtain feature information at different scales and promote feature perception between different scales. The CSViT structure includes: For feature maps After flattening, a two-dimensional patch sequence is obtained. The channel normalization unit, used to maintain dimensionality constant without expanding the D-dimensional length, is expressed as: ； in, This represents the two-dimensional patch sequence output after normalization; LN represents layer normalization; Lin represents a linear layer. Furthermore, spatial activation units that capture short-range and long-range dependencies in images based on local and global information are used to identify key human joints in the image, expressed as: ； Local feature information output by spatial activation unit The expression is obtained by multiplying the value V by the effects of query Q and key K, and then adjusting the dimensions through convolution and dimensionality transformation: ； in, Represents a two-dimensional patch sequence Processed by the spatial activation unit; for function; Represents a 1×1 convolution; Represents matrix multiplication; RE represents dimension transformation operation; HP is a hyperparameter; In addition, a dimensional transformation is performed to further activate channel activation units from the spatial information extracted by the spatial activation units. This is used to fuse information from different channels at the same spatial location to enhance channel perception capabilities. The expression is as follows: ； in, This represents a 3×3 convolution; for Activation function.

2. A lightweight human pose estimation method that integrates channel and spatial activation, employing the lightweight human pose estimation system described in claim 1, characterized in that, The method includes the following steps: S1. Extract the input image using the CNN backbone network. Heatmap features are used to obtain feature maps ,in The input images are respectively Height and width, Feature maps Height, width, and passageway; S2. The channel is divided by the spatial segmentation module to obtain two sets of dual-scale features, namely high-resolution features and low-resolution features. S3. A cross-attention mechanism for information transfer between different spatial scales is constructed using dual-scale CSViT. S4, the dual-scale cross-attention module, combines a cross-attention mechanism to extract and fuse local and global feature information across spatial dimensions to obtain the input image. Key points in human posture.

3. The human posture estimation method according to claim 2, characterized in that, In step S2, the specific process includes the following steps: First, for the feature map Downsampling is performed to obtain features at two resolutions: high-resolution features. and low-resolution features ; Subsequently, high-resolution features were analyzed along the channel dimension. and low-resolution features The space is partitioned, generating four spatial features, expressed as follows: ； ； Where Split[ ] represents the channel splitting operation; All represent high-resolution features; All of these represent low-resolution features.

4. The human posture estimation method according to claim 2, characterized in that, In step S3, the specific process includes the following steps: First, high-resolution features are obtained from the spatial segmentation module. and Meanwhile, low-resolution features are obtained through downsampling. and ; Then, any high-resolution feature and any low-resolution feature The common input is fed into a dual-scale CSVit; where high-resolution features... 、 With low-resolution features Input to high-resolution CSVit and low-resolution CSVit respectively Parallel processing; Processing low-resolution features At that time, low resolution Obtain low-resolution sequence , , Subsequently, the low-resolution sequence was processed through dimensionality transformation and 1×1 convolution. and Spatial learning is performed, and spatially expanded low-resolution sequences are obtained by padding upsampling. and ; At the same time, from high resolution Processing high-resolution features Obtain high-resolution sequences ,in Next, the low-resolution sequence and With high-resolution sequences Input together to high resolution Cross attention is performed to obtain the sequence ; Meanwhile, low-resolution sequences , , Through low resolution Processing yields the sequence ; Finally, the sequence Upsampling and sequence Adding them together yields the final cross-attention output, expressed as: ； in, This indicates element-wise addition; This indicates an upsampling operation.

5. The human posture estimation method according to claim 3, characterized in that, The high-resolution features and All space dimensions are low-resolution features and All space dimensions are .

6. The human pose estimation method according to claim 5, characterized in that, In the spatial partitioning module, network efficiency is improved by compressing the spatial size, specifically configured as follows: ； ； in, The input images are respectively Height and width; Feature maps The height and width.

7. The human posture estimation method according to claim 2, characterized in that, The CNN backbone network uses the ResNet50 network.

Images