Video human body posture estimation method and device based on Mama architecture and multi-scale feature fusion

Abstract

The invention discloses a video human body posture estimation method and device based on Mama architecture and multi-scale feature fusion, and the method comprises the steps: obtaining multi-scale fusion features of different stages, extracting posture clues through the time sequence information contained in a front frame and a rear frame, completing the cross-frame multi-scale time sequence feature fusion, intra-frame multi-scale feature fusion, and residual decoding operation, and carrying out the video human body posture estimation. Information among different features is fully mined; an advanced feature extraction and enhancement strategy is adopted, a fusion method is innovated, richer feature information is provided, a multi-scale feature enhancement Mama module efficiently captures a data dependency relationship, focuses human body feature content and fades background interference, and the feature expression ability is further improved; according to the method, the time sequence information can be fully combined, the posture relation is deeply mined, the human body posture estimation precision in the video scene is remarkably improved, and the method has certain practical significance and application value and has important significance in the industries such as safety protection and automatic driving.

Description

Technical Field

[0001] The present invention belongs to the technical field of human posture estimation in computer vision, and in particular relates to a video human posture estimation method and device based on Mamba architecture and multi-scale feature fusion. Background Art

[0002] The core task of human pose estimation is to use network models to detect the joints of people in images or videos, which is a prerequisite for many downstream applications in the field of computer vision technology. In recent years, human pose estimation has received extensive attention and research in industry and academia, and has achieved breakthrough developments in the fields of behavior recognition, surveillance and security, and human-computer collaboration. It has broad application prospects and huge practical value; however, most of the current human pose estimation methods are aimed at image tasks, and there are still many challenging difficulties for more complex video tasks.

[0003] In image tasks, feature fusion is often used to increase feature representation capabilities so that the network model can learn richer feature representations. Representative networks include High-Resolution Network (HRNet). While maintaining high-resolution feature maps throughout the process, the network uses feature maps of different resolutions in parallel to perform feature fusion at different stages, thereby providing the network with rich feature representations. Therefore, the network has achieved relatively good results in image-level human pose estimation tasks. However, people in video scenes cannot correctly display complete human poses due to rapid movement, self-occlusion, object occlusion, etc. If traditional estimation methods for image tasks are used to estimate human pose frame by frame, not only will the network efficiency be low due to massive information processing, but the support of the temporal information contained in the video data for pose estimation will not be considered, resulting in unsatisfactory detection results. The existing human pose estimation methods based on videos are optimized and improved on the basis of image models. They also ignore the spatiotemporal similarity and continuity between video frames, fail to fully consider the gradual characteristics of human motion in videos, and cannot accurately estimate people in complex scenes or with too many joint occlusions. Summary of the invention

[0004] In view of the above, the present invention provides a method and device for video human posture estimation based on Mamba architecture and multi-scale feature fusion. The method can obtain the human posture information in the target frame at a deeper level, and fully combine the timing information provided by the previous and next supporting frames to optimize the target posture, thereby improving the accuracy of human posture estimation in video scenes.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A video human posture estimation method based on Mamba architecture and multi-scale feature fusion, the method comprising the following steps:

[0007] (1) Obtain a video frame dataset for human posture estimation and perform feature extraction at different scales;

[0008] (2) The acquired multi-scale features are subjected to feature residual construction, mask generation, SSD layer mapping, Skip connection and normalization operations, and the features are enhanced and optimized at different feature levels to obtain the first output features;

[0009] (3) The first output features obtained from the target frame and the previous and next support frames are fused with temporal dimension information through concatenated convolution and Mamba to obtain the second output features;

[0010] (4) The first output feature obtained from the target frame is convolved layer by layer and concatenated upward, and finally feature fusion is completed through the Mamba layer to obtain the third output feature;

[0011] (5) The second output feature and the third output feature are decoded and fused in corresponding dimensions to obtain the final human posture estimation result.

[0012] Furthermore, the specific implementation of step (1) is as follows: annotating 15 joint points of the characters in the acquired data set, and setting additional labels for joint visibility; dividing the data set into three parts: a training set, a validation set, and a test set; performing dense posture annotation on the middle 30 frames of the training set, and providing posture annotation for every four frames in the validation set; Figure 2 As shown, a high-resolution network feature extraction module is constructed for feature extraction at different scales. The high-resolution network feature extraction module is based on HRNet. In the first stage, the module uses a high-resolution branch to process the high-resolution features of the input frame. At the same time, the second, third and fourth stages are composed of multiple parallel multi-resolution branches. Each stage adds a low-resolution branch by downsampling, and multi-scale feature fusion is performed between the branches. The model inputs the first, middle and last three frames as a batch each time, and then selects the lowest scale features after the multi-scale fusion operation in the second, third and fourth stages as the input feature sequence of the subsequent steps of the present invention.

[0013] Furthermore, the step (2) constructs a multi-scale feature enhancement Mamba module for enhancing and optimizing the features, which is constructed by feature residual, mask generation, and Mamba layer, wherein the Mamba layer includes an SSD layer, a Skip connection, and a normalization operation; the front, middle, and back three frames obtained in step (1) are divided into a large-scale feature set, a medium-scale feature set, and a small-scale feature set according to the feature scale, and then respectively input into the multi-scale feature enhancement Mamba module, wherein in the large-scale feature set, two residual features are obtained by subtracting the target frame features from the front and back support frame features, and two different masks are obtained by using channel compression and variant Sigmoid function transformation, and then The target frame features and the front and back support frame features are spliced ​​in the channel dimension to obtain large-scale spliced ​​features. Subsequently, the large-scale spliced ​​features are point-multiplied with the two masks and added, and spliced ​​with the large-scale temporal fusion features in the channel dimension to obtain large-scale preliminary fusion features. The large-scale preliminary fusion features are then input into the four-layer stacked Mamba for information fusion. The designed state space model enables the model to focus on the human content features and weaken the background interference information. Finally, the large-scale enhanced features are obtained through upsampling operations; the medium-scale feature set and the small-scale feature set also refer to the above large-scale feature operations to obtain medium-scale enhanced features and small-scale enhanced features, respectively. The above specific operation methods are expressed as follows:

[0014] res_a=target_lar-post_sup_lar

[0015] res_b=target_lar-pre_sup_lar

[0016] mask1,mask2=varsigmoid(comp(res_a,res_b))

[0017]

[0018] varsigmoid(x)=1 / (2+e -|x| )

[0019] z=large_fus_out+Mamba(LayerNorm(large_fus_out))

[0020] r=UP(z,z',z')

[0021] Among them: target_lar, post_sup_lar, pre_sup_lar represent the target frame features, post-support frame features, and pre-support frame features, respectively; res_a and res_b represent the post-feature residuals between the target frame features and the post-support frame features, and the pre-feature residuals between the target frame features and the pre-support frame features, respectively; large_fus represents the large-scale splicing features; large_fus_out represents the large-scale preliminary fusion features; mask1, mask2 represent the corresponding masks; comp represents channel compression; varsigmoid() represents the variant Sigmoid function; represents concatenation, ⊙ represents dot product, SUM represents addition, y represents the intermediate value, z, z', z" represent the three scale features after the Mamba layer respectively, LayerNorm() represents layer normalization, UP() represents upsampling operation, and r represents the enhanced feature set of three scales.

[0022] Furthermore, the step (3) constructs a cross-frame multi-scale temporal fusion module for cross-fusion of the three scale enhancement features obtained from the front, middle and back three frames in temporal order. The specific contents are:

[0023] The large-scale enhanced features obtained by the previous and next support frames and the target frame in step (2) are concatenated, and then the large-scale temporal fusion enhanced features are obtained through convolution operation. The medium-scale enhanced features and the small-scale enhanced features are concatenated and Mamba layered to obtain the medium-scale and small-scale temporal fusion enhanced features. The above specific operation is expressed as follows:

[0024]

[0025] Among them: lg_fus, md_fus and sm_fus represent large-scale enhancement features, medium-scale enhancement features and small-scale enhancement features respectively, pre_sup represents the previous support frame, post_sup represents the post-support frame, and target represents the target frame; lg_t_fus, md_t_fus and sm_t_fus represent large-scale temporal fusion enhancement features, medium-scale temporal fusion enhancement features and small-scale temporal fusion enhancement features respectively; Conv represents convolution operation, and Mamba represents Mamba layer.

[0026] Furthermore, the step (4) constructs an intra-frame multi-scale feature fusion module for taking the multi-scale enhanced features of the target frame obtained in step (2) as input, and the specific contents are as follows:

[0027] First, the medium-scale enhancement feature is convolved and upsampled by 2 times, and then spliced ​​and convolved with the large-scale enhancement feature after convolution to obtain a first-level large-scale inter-frame fusion enhancement feature. Secondly, the small-scale enhancement feature is convolved and upsampled by 2 times and spliced ​​and convolved with the medium-scale enhancement feature to obtain a first-level medium-scale inter-frame fusion enhancement feature. Thirdly, the first-level medium-scale inter-frame fusion enhancement feature is upsampled by 2 times and spliced ​​and convolved with the first-level large-scale inter-frame fusion enhancement feature to obtain a second-level large-scale inter-frame fusion enhancement feature. Finally, the small-scale enhancement feature is upsampled by 4 times and spliced ​​with the second-level large-scale inter-frame fusion enhancement feature and obtained through the Mamba layer. The above specific operation methods are expressed as follows:

[0028]

[0029] Among them: UP 2* Indicates 2x upsampling, UP 4* It represents 4 times upsampling, target_lg_fus represents the large-scale enhancement feature of the target frame, target_md_fus represents the medium-scale enhancement feature of the target frame, target_sm_fus represents the small-scale enhancement feature of the target frame, one_lg_zj_fus represents the first-level large-scale inter-frame fusion enhancement feature, one_md_zj_fus represents the first-level medium-scale inter-frame fusion enhancement feature, two_lg_zj_fus and three_lg_zj_fus represent the second-level and third-level large-scale inter-frame fusion enhancement features respectively, and Conv represents the convolution operation.

[0030] Furthermore, the step (5) constructs a posture feature decoding module for realizing human posture estimation, and uses the large-scale temporal fusion enhancement feature, the medium-scale temporal fusion enhancement feature, the small-scale temporal fusion enhancement feature obtained in step (3) and the three-level large-scale inter-frame fusion enhancement feature obtained in step (5) as module input, and inputs them into decoders with residual structures to further learn information of features of different scales, and obtain output features, wherein the medium-scale temporal fusion enhancement feature and the small-scale temporal fusion enhancement feature are upsampled by 2 times and 4 times respectively after passing through the decoder, so as to obtain the same scale size as the large-scale temporal fusion enhancement feature and the four-level large-scale inter-frame fusion enhancement feature, and then performs feature addition and 1x1 convolution operation on all outputs, and finally obtains the position coordinates of each key point of the character in the target frame. The above specific operation method is expressed as follows:

[0031] A,B,C,D=Decoder(lg_t_fus,md_t_fus,sm_t_fus,three_lg_zj_fus)

[0032]

[0033] Among them: A, B, C, D represent the output of large-scale temporal fusion enhancement features, medium-scale temporal fusion enhancement features, small-scale temporal fusion enhancement features, and three-level large-scale inter-frame fusion enhancement features after passing through the decoder Decoder, and Out is the position coordinate output of each key point of the character in the target frame, which is the result of human posture estimation.

[0034] The present invention makes full use of the feature extraction capability of high-resolution networks, the ability of structured state space sequence models to efficiently capture complex dependencies in sequence data, and the rich multi-scale feature interaction information and temporal continuous information contained in video sequences to deal with the problem of difficult posture estimation in complex video scenes, enhances the reasoning ability of the model, can better estimate the key parts of the human body, and improves the accuracy of posture estimation. It is of great significance in industries such as safety protection and autonomous driving that require human posture extraction and prediction. Its beneficial technical effects are mainly reflected in the following four aspects:

[0035] 1. The present invention effectively captures the timing information in the video by adopting a multi-scale feature extraction, enhancement, and fusion method and an end-to-end encoder-decoder architecture, and realizes cross-frame multi-scale timing feature fusion and intra-frame multi-scale feature fusion, fully mining the effective information between different features, and solving the pain point problem of the difficulty in utilizing timing information in traditional single-frame posture estimation methods, thereby improving the accuracy and robustness of posture estimation.

[0036] 2. The present invention effectively processes video sequence data and makes full use of the adjacent frames to extract useful posture information clues to optimize the target posture. It not only enhances the model's ability to adjust to dynamic human body changes in the video, but also significantly improves the accuracy of human body posture estimation in complex scenes due to motion blur, occlusion, etc.

[0037] 3. The present invention adopts advanced feature extraction and enhancement strategies, and effectively extracts richer feature information through multi-scale branch extraction, feature fusion and structured state space sequence model. The proposed multi-scale feature enhancement Mamba module efficiently captures complex dependencies, focuses on human feature content, reduces background interference, and further enhances the expressiveness of features.

[0038] 4. The cross-frame multi-scale temporal fusion module and the intra-frame multi-scale feature fusion module of the present invention adopt an innovative fusion strategy, which realizes the cross-frame and intra-frame feature fusion through splicing, fusion, convolution and SSM operations. This fusion strategy not only enhances the temporal coherence of the features, but also improves the ability to capture the dynamic posture changes of the human body in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1Schematic diagram of the overall process of the human body posture estimation method of the present invention.

[0040] Figure 2 Schematic diagram of the structure of the high-resolution network feature extraction module.

[0041] Figure 3 Schematic diagram of the structure of the multi-scale feature enhancement Mamba module.

[0042] Figure 4 Schematic diagram of the structure of the cross-frame multi-scale temporal fusion module.

[0043] Figure 5 Schematic diagram of the structure of the intra-frame multi-scale feature fusion module.

[0044] Figure 6 This is a structural diagram of the posture feature decoding module.

[0045] Figure 7 It is a structural diagram of a video human posture estimation device based on Mamba architecture and multi-scale feature fusion of the present invention. DETAILED DESCRIPTION

[0046] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0047] like Figure 1 As shown, a video human posture estimation method based on Mamba architecture and multi-scale feature fusion of the present invention comprises the following steps:

[0048] (1) Use a video dataset for human pose estimation and set it to three frames as an input sequence.

[0049] In this implementation, three datasets, PoseTrack17, PoseTrack18, and PoseTrack21, are selected as benchmarks for training data, and a dataset suitable for human pose estimation is constructed: the three datasets are divided into three parts, namely, a training set, a validation set, and a test set. 15 joint points are annotated for the characters in the dataset, and additional labels for joint visibility are set; dense pose annotations are performed on the middle 30 frames of the training set, and pose annotations are provided for every four frames in the validation set; the number of video frame inputs in the dataset is set to three frames as an input sequence, and in the case where the front supporting frame of the first frame and the back supporting frame of the last frame do not exist, this method uses the method of copying the target frame to supplement the missing supporting frames to ensure the compatibility and robustness of the network model.

[0050] (2) Construct a high-resolution network feature extraction module to extract features from the input frames.

[0051] like Figure 2As shown in the figure, it is the structure diagram of the HRNet backbone network. After the backbone network receives the input, it uses a high-resolution branch to process the features in the first stage. In the second, third and fourth stages, low-resolution branches are added step by step by downsampling, and multi-scale feature fusion is performed between the branches. Specifically, the resolution of the original input is 256*192. After the multi-scale feature fusion of four stages, a richer feature representation is obtained. Then the model extracts the lowest scale features of 128*96, 64*48, and 32*24 as output features in the second, third and fourth stages, respectively, recorded as

[0052] (3) Construct a multi-scale feature enhancement Mamba module to enhance multi-scale features.

[0053] like Figure 3 As shown in the figure, the front, middle and back three frames are divided into a large-scale feature set, a medium-scale feature set and a small-scale feature set according to the feature scale, and then input into the enhancement module respectively. In the large-scale feature set, two residual features are obtained by subtracting the target frame features from the front and back support frame features respectively, and two different masks are obtained by using channel compression and variant Sigmoid function transformation. Then, the three features are spliced ​​in the channel dimension to obtain a large-scale spliced ​​feature. Subsequently, the large-scale spliced ​​feature is point-multiplied with the two masks respectively and added, and then spliced ​​with the large-scale temporal fusion feature in the channel dimension, and then information fusion is performed by stacking four layers of Mamba. The designed structured state space sequence mechanism enables the model to focus on the content features of the human body and weaken the background interference information. Finally, the large-scale enhanced feature is obtained by upsampling. The mesoscale feature set and the small-scale feature set also refer to the above large-scale feature operation to obtain the mesoscale enhanced features. and small-scale enhanced features

[0054] (4) Construct a cross-frame multi-scale temporal fusion module to cross-fuse the multi-scale enhanced features obtained from the target frame and the support frame.

[0055] like Figure 4 As shown, the output of the support frame and the target frame after step (3) is the enhanced feature Then the large-scale enhanced features Perform concatenation operation, and then convolve to obtain large-scale temporal fusion enhanced feature T a ;Mesoscale enhancement features Perform splicing and Mamba operations to obtain the mid-scale temporal fusion enhanced feature T b ; Small scale enhancement features After concatenation and Mamba operation, the small-scale temporal fusion enhanced feature T is obtained. c; By combining the features of the support frame and the target frame, the model captures the temporal continuity and dynamic changes of the video posture, helping the model to understand complex motion transformations, and adopts different extraction methods for features of different scales to more carefully capture local information and enhance feature connections.

[0056] (5) Construct an intra-frame multi-scale feature fusion module to scale-fuse the multi-scale enhanced features of the target frame.

[0057] like Figure 5 As shown, the three scale features of the target frame obtained after step (3) Perform upsampling fusion step by step, specifically: first, the mid-scale enhanced features Upsampling by 2 times and large-scale enhancement features Perform concatenation and convolution operations to obtain the first-level large-scale inter-frame fusion enhancement features Secondly, small-scale enhancement features Upsampling by 2 times and mid-scale enhancement features Perform concatenation and convolution operations to obtain the first-level mid-scale inter-frame fusion enhancement features Again, the first-level mid-scale inter-frame fusion enhancement feature Upsampling by 2 times and first-level large-scale inter-frame fusion to enhance features Perform concatenation and convolution operations to obtain the second-level large-scale inter-frame fusion enhancement features Finally, the small-scale enhanced features Upsample by 4 times and fuse with the secondary large-scale frame to enhance the features Perform concatenation and convolution operations and input them into the Mamba layer to obtain three-level large-scale inter-frame fusion enhancement features. The final output is 128*96*64 features

[0058] (6) Construct a posture feature decoding module to decode the outputs of the cross-frame multi-scale temporal fusion module and the intra-frame multi-scale feature fusion module to obtain the key point heat map output.

[0059] like Figure 6 As shown, the large-scale temporal fusion enhancement feature T obtained in step (4) is a , medium-scale time series fusion enhanced features T b , Small-scale temporal fusion enhanced feature T c And the three-level large-scale inter-frame fusion enhancement feature obtained in step (5) The decoders with residual structures are respectively input to further learn feature information of different scales. The decoder with residual structure is composed of transposed convolution, batch normalization and ReLU activation function, which ensures that more feature information is retained, which helps the model understand and integrate multi-scale information. The scale-time series fusion enhancement features and scale-time series fusion enhancement features will be upsampled by 2 times and 4 times after passing through the decoder to reach the scale of 128*96*64, and then further added and fused with the large-scale time series fusion enhancement features and the three-level large-scale inter-frame fusion enhancement features after the decoder, and a 1*1 convolution layer is used to obtain a 128*96*15 human posture heat map. Finally, the coordinates are decoded by Argmax to obtain the corresponding joint point results.

[0060] (7) Use the video frames in the training set to train the model.

[0061] 7.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate and optimizer;

[0062] 7.2 Input the training data sets of PoseTrack17, PoseTrack18, and PoseTrack21 into the model in batches for training. The training cycle is set to 210 epochs and the learning rate is set to 0.0005. The model forward propagation output obtains the corresponding pose estimation results, i.e., the position coordinates of the key points. The MSE function is used to calculate the loss between the pose estimation results and the annotation information, and the network parameters are iteratively updated until the loss function converges. Finally, the optimal parameter weight file is saved.

[0063] 7.3 Import the saved optimal parameter weight file into the model, perform mean average precision (mAP) evaluation on the model of the present invention on the validation set of the PoseTrack17, PoseTrack18, and PoseTrack21 datasets, and perform quantitative analysis on the estimation results obtained by the model and the estimation results obtained by other existing models, as shown in Table 1:

[0064] Table 1

[0065]

[0066] The average precision of the model of the present invention on the validation sets of the three datasets PoseTrack17, PoseTrack18, and PoseTrack21 is higher than that of the existing models; the PoseTrack17 dataset is a classic video human posture estimation dataset, and the model of the present invention achieves the best performance on this dataset, with an average precision higher than other existing methods, 0.9% higher than PoseWarper and DCPose, and good precision is also achieved on the wrist and ankle joints that are prone to motion blur, which is higher than the existing methods; the PoseTrack18 dataset has more occluded and blurred joints, and the model of the present invention still achieves performance superior to other methods, improving the precision of the wrist and ankle joints by 4.4% and 4.5% compared with DCPose, and the average precision is also improved by 2.4%. The PoseTrack21 dataset contains more complex scenes, and the average accuracy of the model of the present invention is still 2.7% higher than the previous optimal method DCPose. This proves that the model of the present invention has excellent accuracy and robustness in human posture estimation, and has achieved significant improvements in accuracy for difficult joint estimation where occlusion and blur are more common in videos. Compared with existing methods, it still maintains the best performance in more challenging scenarios.

[0067] Corresponding to the aforementioned embodiment of a method for estimating human body posture through video based on the fusion of Mamba architecture and multi-scale features, the present invention also provides an embodiment of a device for estimating human body posture through video based on the fusion of Mamba architecture and multi-scale features.

[0068] See also Figure 7 A video human posture estimation device based on Mamba architecture and multi-scale feature fusion provided by an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a video human posture estimation method based on Mamba architecture and multi-scale feature fusion in the above embodiment.

[0069] An embodiment of a video human posture estimation device based on the Mamba architecture and multi-scale feature fusion provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 7As shown in FIG. 1 , a hardware structure diagram of a video human posture estimation device based on Mamba architecture and multi-scale feature fusion provided by the present invention is provided in any device with data processing capability, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0070] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0071] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0072] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a video human posture estimation method based on Mamba architecture and multi-scale feature fusion in the above embodiment is implemented.

[0073] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0074] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for estimating human body posture in a video based on the Mamba architecture and multi-scale feature fusion is implemented.

[0075] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A video human posture estimation method based on Mamba architecture and multi-scale feature fusion, the method comprising the following steps: (1) Obtain a video frame dataset for human posture estimation and perform feature extraction at different scales; (2) The acquired multi-scale features are subjected to feature residual construction, mask generation, SSD layer mapping, Skip connection and normalization operations, and the features are enhanced and optimized at different feature levels to obtain the first output features; (3) The first output features obtained from the target frame and the previous and next support frames are fused with temporal dimension information through concatenated convolution and Mamba to obtain the second output features; (4) The first output feature obtained from the target frame is convolved layer by layer and concatenated upward, and finally feature fusion is completed through the Mamba layer to obtain the third output feature; (5) The second output feature and the third output feature are decoded and fused in corresponding dimensions to obtain the final human posture estimation result.

2. The method for estimating human body posture from video according to claim 1, characterized in that: In the step (1), 15 joint points of the person in the acquired data set are annotated, and additional labels for joint visibility are set; the data set is divided into three parts: a training set, a validation set, and a test set, dense posture annotations are performed on the middle 30 frames of the training set, and posture annotations are provided for every four frames in the validation set.

3. The video human posture estimation method according to claim 1, characterized in that: The step (1) constructs a high-resolution network feature extraction module for feature extraction at different scales. The high-resolution network feature extraction module is based on HRNet. In the first stage, the module uses a high-resolution branch to process the high-resolution features of the input frame. At the same time, the second, third and fourth stages are composed of multiple parallel multi-resolution branches. Each stage adds a low-resolution branch by downsampling, and multi-scale feature fusion is performed between the branches. The model inputs the first, middle and last three frames as a batch each time, and then selects the lowest scale features after the multi-scale fusion operation in the second, third and fourth stages as the input feature sequence of the subsequent steps of the present invention.

4. The method for estimating human body posture from video according to claim 3, characterized in that: The step (2) constructs a multi-scale feature enhancement Mamba module for enhancing and optimizing features, which is composed of feature residual construction, mask generation, SSD layer mapping, Skip connection and normalization operation; the front, middle and back three frames obtained in step (1) are divided into a large-scale feature set, a medium-scale feature set and a small-scale feature set according to the feature scale, and then respectively input into the multi-scale feature enhancement Mamba module, wherein in the large-scale feature set, two residual features are obtained by subtracting the target frame features from the front and back support frame features, and two different masks are obtained by using channel compression and variant Sigmoid function transformation, and then the target frame features and the front and back support frame features are spliced ​​in the channel dimension to obtain a large-scale spliced ​​feature, and then the large-scale spliced ​​feature is respectively multiplied with the two masks and added, and spliced ​​with the large-scale spliced ​​feature in the channel dimension to obtain a large-scale preliminary fusion feature, and then the large-scale preliminary fusion feature is input into the four-layer stacked Mamba for information fusion, and finally the large-scale enhanced feature is obtained by upsampling operation; The mesoscale feature set and the small-scale feature set also refer to the above large-scale feature operation to obtain the mesoscale enhanced features and the small-scale enhanced features respectively. The above specific operation methods are expressed as follows: res_a=target_lar-post_sup_lar res_b=target_lar-pre_sup_lar mask1,mask2=varsigmoid(comp(res_a,res_b)) varsigmoid(x)=1 / (2+e -|x| ) z=large_fus_out+Mamba(LayerNorm(large_fus_out)) r=UP(z,z',z') Among them: target_lar, post_sup_lar, pre_sup_lar represent the target frame features, post-support frame features, and pre-support frame features, respectively; res_a and res_b represent the post-feature residuals between the target frame features and the post-support frame features, and the pre-feature residuals between the target frame features and the pre-support frame features, respectively; large_fus represents the large-scale splicing features; large_fus_out represents the large-scale preliminary fusion features; mask1, mask2 represent the corresponding masks; comp represents channel compression; varsigmoid() represents the variant Sigmoid function; represents concatenation, ⊙ represents dot product, SUM represents addition, y represents the intermediate value, z, z', z" represent the three scale features after the Mamba layer respectively, LayerNorm() represents layer normalization, UP() represents upsampling operation, and r represents the enhanced feature set of three scales.

5. The video human posture estimation method according to claim 1, characterized in that: In step (4), the number of video frame inputs in the data set is set to three frames per input sequence. In the case where the previous supporting frame of the first frame and the subsequent supporting frame of the last frame do not exist, the missing supporting frames are supplemented by copying the target frames.

6. The method for estimating human body posture from video according to claim 1, characterized in that: The step (3) constructs a cross-frame multi-scale temporal fusion module for performing temporal cross-fusion of the three scale enhancement features obtained from the front, middle and back three frames; the specific content is: the large-scale enhancement features obtained by the front and back support frames and the target frame in step (2) are spliced, and then the large-scale temporal fusion enhancement features are obtained through convolution operation, and the medium-scale enhancement features and the small-scale enhancement features are respectively spliced ​​and Mamba layered to obtain the medium-scale and small-scale temporal fusion enhancement features. The above specific operation method is expressed as follows: Among them: lg_fus, md_fus and sm_fus represent large-scale enhancement features, medium-scale enhancement features and small-scale enhancement features respectively, pre_sup represents the previous support frame, post_sup represents the post-support frame, and target represents the target frame; lg_t_fus, md_t_fus and sm_t_fus represent large-scale temporal fusion enhancement features, medium-scale temporal fusion enhancement features and small-scale temporal fusion enhancement features respectively; Conv represents convolution operation, and Mamba represents Mamba layer.

7. The method for estimating human body posture from video according to claim 4, characterized in that: The step (4) constructs an intra-frame multi-scale feature fusion module for taking the multi-scale enhanced features of the target frame obtained in step (2) as input, and specifically comprises: first, convolving the medium-scale enhanced features and upsampling them by 2 times, and then splicing and convolving the convolved large-scale enhanced features to obtain a first-level large-scale inter-frame fusion enhanced feature; second, convolving the small-scale enhanced features and upsampling them by 2 times and splicing and convolving the medium-scale enhanced features to obtain a first-level medium-scale inter-frame fusion enhanced feature; third, upsampling the first-level medium-scale inter-frame fusion enhanced feature by 2 times and splicing and convolving the first-level large-scale inter-frame fusion enhanced feature; finally, upsampling the small-scale enhanced features by 4 times and splicing them with the second-level large-scale inter-frame fusion enhanced features, and then obtaining a third-level large-scale inter-frame fusion enhanced feature through the Mamba layer. The above specific operation methods are expressed as follows: Among them: UP 2* Indicates 2x upsampling, UP 4* It represents 4 times upsampling, target_lg_fus represents the large-scale enhancement feature of the target frame, target_md_fus represents the medium-scale enhancement feature of the target frame, target_sm_fus represents the small-scale enhancement feature of the target frame, one_lg_zj_fus represents the first-level large-scale inter-frame fusion enhancement feature, one_md_zj_fus represents the first-level medium-scale inter-frame fusion enhancement feature, two_lg_zj_fus and three_lg_zj_fus represent the second-level and third-level large-scale inter-frame fusion enhancement features respectively, and Conv represents the convolution operation.

8. The method for estimating human body posture from video according to claim 7, characterized in that: The step (5) constructs a posture feature decoding module for realizing human posture estimation, and uses the large-scale temporal fusion enhancement feature, the medium-scale temporal fusion enhancement feature, the small-scale temporal fusion enhancement feature obtained in step (3) and the three-level large-scale inter-frame fusion enhancement feature obtained in step (5) as module input, and inputs them into decoders with residual structures to further learn information of features of different scales, and obtain output features, wherein the medium-scale temporal fusion enhancement feature and the small-scale temporal fusion enhancement feature are upsampled by 2 times and 4 times respectively after passing through the decoder, so as to obtain the same scale size as the large-scale temporal fusion enhancement feature and the four-level large-scale inter-frame fusion enhancement feature, and then performs feature addition and 1x1 convolution operation on all outputs, and finally obtains the position coordinates of each key point of the character in the target frame. The above specific operation method is expressed as follows: A,B,C,D=Decoder(lg_t_fus,md_t_fus,sm_t_fus,three_lg_zj_fus) Among them: A, B, C, D represent the output of large-scale temporal fusion enhancement features, medium-scale temporal fusion enhancement features, small-scale temporal fusion enhancement features, and three-level large-scale inter-frame fusion enhancement features after passing through the decoder Decoder, and Out is the position coordinate output of each key point of the character in the target frame, which is the result of human posture estimation.

9. A video human posture estimation device based on Mamba architecture and multi-scale feature fusion, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a video human posture estimation method based on Mamba architecture and multi-scale feature fusion is implemented as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a video human posture estimation method based on Mamba architecture and multi-scale feature fusion as described in any one of claims 1 to 8 is implemented.

Images