A parallel space internal and external interaction system and method for human posture estimation

Abstract

The present invention relates to the technical field of human posture estimation, solves the technical problem that the traditional method does not perform spatial feature extraction with the same resolution in multiple identical spaces, and particularly relates to a parallel space internal and external interaction system and method for human posture estimation, the system comprising a downsampling module for acquiring an input image and quickly downsampling the input image to obtain a downsampled image feature, a channel adjustment module for expanding the channel size of the downsampled image feature to obtain a channel expansion feature, and separating the channels of the channel expansion feature to obtain four parallel space features with the same spatial size and adjacent channels. The present invention explores the influence of parallel space interaction on human posture estimation by performing spatial interaction within and between parallel spaces, and completes global feature interaction both internally and externally, which is of great benefit to capturing the relationship between key points of human posture estimation and ensures the high efficiency of human posture estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of human body posture estimation, and in particular to a parallel space internal and external interaction system and method for human body posture estimation. Background Art

[0002] Human pose estimation is a fundamental task in the field of vision, which involves locating the position of all people in an image. Human pose estimation is commonly used in a wide range of applications such as motion recognition, human-computer interaction, and augmented reality. Although computer vision has made great progress with the development of deep learning, human pose estimation is still a problem worth studying.

[0003] At present, human posture estimation is very sensitive to the scale of spatial features, and multi-scale spatial features can be used to improve positioning accuracy. For example, the High Resolution Network (HRNet) uses multiple parallel spaces to locate key points, but HRNet mainly relies on a single branch with high resolution to improve positioning accuracy, and multiple parallel subspaces have different scales.

[0004] As mentioned above, most current methods tend to set parallel branch spaces to different resolutions, and in the acquisition of parallel branch spaces, they are also like HRNet, habitually acquiring different spatial scales in sequence through downsampling, and have not yet performed spatial feature extraction of the same resolution in multiple identical spaces. Therefore, it is unclear whether the parallel space learning strategy is effective for the network. On the other hand, the interaction of parallel spaces is not friendly to the parameters and computation of the network, and feature extraction within parallel spaces is bound to put higher requirements on the research methods of human posture estimation. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a parallel space internal and external interaction system and method for human body posture estimation. Through the design of parallel space, spatial interaction is performed within and between parallel spaces, and the impact of parallel space interaction on human body posture estimation is explored, which can effectively solve the problems in the background technology.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a parallel space internal and external interactive system for human posture estimation, comprising a downsampling module for acquiring an input image and quickly downsampling the input image to obtain downsampled image features, and further comprising:

[0007] A channel adjustment module, the channel adjustment module is used to expand the channel size of the downsampled image feature to obtain a channel expansion feature, and separate the channels of the channel expansion feature to obtain four parallel space features with the same spatial size and adjacent channels;

[0008] A parallel space interaction module, the parallel space interaction module is used to perform parallel space internal interaction and parallel space external interaction on the four parallel space features to obtain parallel space interaction features;

[0009] The interaction inside the parallel space takes precedence over the interaction outside the parallel space;

[0010] A channel regularization unit, the channel regularization unit is used to regularize the channel of the parallel space interaction feature to obtain a channel regularization feature;

[0011] The deconvolution layer is used to restore the spatial size of the channel regularization feature, obtain the human joint position for network training, and calculate the loss value with the true value.

[0012] Furthermore, the channel adjustment module includes a module for downsampling image features Expand to The channel expansion feature is used to enhance the human posture perception ability in the extended channel, and the channel of the channel expansion feature is separated to obtain four parallel space features with the same space size and adjacent channels. Channel separation unit.

[0013] Furthermore, the parallel space interaction module includes a split head Transformer sub-block for performing parallel space internal interaction on the four parallel space features, and a convolution sub-block for gradually and interleavingly performing parallel space external interaction after performing the parallel space internal interaction.

[0014] Furthermore, there are four split head Transformer sub-blocks in total, and each of the split head Transformer sub-blocks performs parallel space internal interaction on each parallel space feature separately to obtain a first parallel space internal interaction feature, a second parallel space internal interaction feature, a third parallel space internal interaction feature, and a fourth parallel space internal interaction feature;

[0015] The convolution sub-block includes a first convolution for outputting a first parallel space external interaction feature according to the first parallel space internal interaction feature and the second parallel space internal interaction feature;

[0016] and outputting a second convolution of a second parallel space external interaction feature according to the first parallel space external interaction feature and the third parallel space internal interaction feature;

[0017] And a third convolution is performed to output the parallel space interaction feature according to the second parallel space external interaction feature and the fourth parallel space internal interaction feature.

[0018] Furthermore, the split head Transformer sub-block includes a global position relationship of human joints of different scales within each parallel space feature, and finally outputs a head split attention feature Head split attention, and head split attention features Perform linear transformation and convolution calculation to obtain linear convolution transformation features Linear convolutional transformation layer.

[0019] Furthermore, the head split attention includes a method for calculating the self-attention value by using three calculation matrices: value V, keyword K, and query Q. and The total number of attention heads of the head split attention is N, and the numbers of attention heads of the high spatial self-attention and the low spatial self-attention are (1-b)×N and b×N respectively;

[0020] And the self-attention value and Fusion is performed to obtain the global position relationship of human joints of different scales within each of the parallel space features, and finally the head split attention feature of the head split attention output is obtained. Attention fusion unit.

[0021] Furthermore, the linear convolutional transformation layer includes a head split attention feature The first linear layer with the number of channels expanded to twice that of the first linear layer, and the deep convolution that processes spatial information in the channels after the expansion is doubled , and restore the number of channels to the same as the head split attention feature after processing the spatial information The second linear layer with the same number of original channels.

[0022] Furthermore, a value for the high spatial self-attention is set between the low spatial self-attention and the high spatial self-attention. , Keywords A space reduction unit for compressing the space size.

[0023] The technical solution also provides a parallel space internal and external interaction method applied to the above parallel space internal and external interaction system, comprising the following steps:

[0024] Get the input image and perform fast downsampling to obtain downsampled image features;

[0025] After expanding the channel size of the downsampled image features, four parallel spatial features with the same spatial size and adjacent channels are obtained by separation;

[0026] Performing parallel space internal interaction and parallel space external interaction on the four parallel space features in turn to obtain parallel space interaction features;

[0027] Regularizing the channels of the parallel space interaction features to obtain channel regularization features;

[0028] The spatial size of the channel regularization features is restored to obtain the human joint positions used for network training.

[0029] Furthermore, the specific process of obtaining the parallel space interaction characteristics is as follows:

[0030] The self-attention value is calculated based on the three calculation matrices of value V, keyword K, and query Q, where:

[0031] Self-attention value for high spatial self-attention The calculation is as follows:

[0032] ;

[0033] Self-attention value for low spatial self-attention The calculation is as follows:

[0034] ;

[0035] In the above formula, , are the value and keyword of high spatial self-attention respectively; T represents transposition; , Represent the value and keyword of low spatial self-attention respectively; and are all constants;

[0036] Self-attention value and Fusion is performed to obtain the global positional relationship of human joints of different scales within each parallel spatial feature, and finally the head split attention feature is obtained. ;

[0037] Split attention features on the head Perform linear transformation and convolution calculation to obtain linear convolution transformation features , the expression is:

[0038] ;

[0039] in, Indicates that the convolution kernel is a 3×3 depth convolution; represents the first linear layer, represents the second linear layer; Indicates dimension conversion; Represents the activation function GeLU;

[0040] Transform features of linear convolution by transforming feature dimensions Perform spatial dimension conversion and four linear convolution conversion features Performing parallel space internal interaction on each parallel space feature respectively, to obtain a first parallel space interaction feature, a second parallel space interaction feature, a third parallel space interaction feature, and a fourth parallel space interaction feature;

[0041] Multiply the interactive features in the first parallel space by the interactive features in the second parallel space, perform spatial interaction between the interactive features in the first parallel space and the interactive features in the second parallel space, and then obtain the interactive features outside the first parallel space through the first convolution;

[0042] Multiply the first parallel space external interaction feature and the third parallel space internal interaction feature, perform spatial interaction between the first parallel space external interaction feature and the third parallel space internal interaction feature, and then obtain the second parallel space external interaction feature through a second convolution;

[0043] Multiply the interaction features outside the second parallel space and the interaction features inside the fourth parallel space, perform spatial interaction between the interaction features outside the second parallel space and the interaction features inside the fourth parallel space, and then obtain the interaction features outside the third parallel space through the third convolution;

[0044] Finally, the parallel space external interaction of the four parallel space characteristics is completed to obtain the parallel space interaction characteristics.

[0045] By means of the above technical solution, the present invention provides a parallel space internal and external interaction system and method for human body posture estimation, which has at least the following beneficial effects:

[0046] 1. The present invention explores the impact of parallel space interaction on human posture estimation by performing spatial interaction within and between parallel spaces. Global feature interaction is completed both internally and externally, which is of great benefit for capturing the relationship between key points of human posture estimation and ensures the high efficiency of human posture estimation.

[0047] 2. The present invention constructs four parallel space features of the same size and connected by channels, and uses a parallel space interaction module to perform parallel space internal interaction and parallel space external interaction on the four parallel space features. This process realizes the internal and external interaction of the four parallel space features, improves the degree of interaction of the parallel space features, and thus enhances the ability of the present invention to construct relationships between different key points.

[0048] 3. The present invention effectively improves the self-attention capture capability of different spatial sizes by performing internal interaction of parallel spatial features, and the design of low spatial self-attention effectively improves the efficiency of the overall method, ensuring that the present invention is capable of completing accurate key point estimation with lower parameters and computational complexity. In addition, the present invention enhances the feature learning capability between different spaces by performing external interaction of parallel spaces.

[0049] 4. The linear convolution transformation layer constructed by the present invention can further improve the perception ability of the channel, and subsequently use deep convolution to process spatial information in the extended channel to help reduce the information loss of the network when learning features in parallel space.

[0050] 5. The present invention utilizes a channel adjustment module and a channel regularization unit to expand, transform and split the channel. Since the four parallel space features are split along the channel, the setting of the channel adjustment module and the channel regularization unit effectively avoids the loss of channel details caused by channel splitting, improves the learning ability of the channel, and ultimately improves the overall effect of human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0052] Figure 1 A network structure diagram of the parallel space internal and external interactive system in the present invention;

[0053] Figure 2 This is a network structure diagram of the split head Transformer sub-block in the present invention;

[0054] Figure 3 This is a diagram showing the human body posture estimation results of the parallel space internal and external interaction method in the present invention on the COCO dataset;

[0055] Figure 4 This is a diagram of the human body posture estimation results of the parallel space internal and external interaction method in the present invention on the MPII dataset. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0057] To perform spatial interaction within and between parallel spaces and explore the impact of parallel space interaction on human pose estimation, please refer to Figure 1-Figure 2 This embodiment proposes a parallel space internal and external interactive system for human posture estimation, including a downsampling module for obtaining an input image and quickly downsampling the input image to obtain downsampled image features. Batch normalization Batch Norm and activation function ReLU are also provided between the downsampling module and the channel adjustment module to improve the feature learning ability of the downsampled image features. In this application, for the task of human posture estimation, the downsampled image can better retain the global information of the image. More importantly, the downsampled image is convenient for network learning, otherwise it will seriously affect the running speed of the network.

[0058] Specifically, the channel adjustment module is used to expand the channel size of the downsampled image features to obtain channel expansion features, and separate the channels of the channel expansion features to obtain four parallel space features with the same spatial size and adjacent channels; the parallel space interaction module performs parallel space internal interaction and parallel space external interaction on the four parallel space features to obtain parallel space interaction features; in this embodiment, parallel space internal interaction takes precedence over parallel space external interaction.

[0059] The channels of the parallel space interaction features are regularized by the channel regularization unit to obtain the channel regularization features. The channel regularization unit is a depth convolution with a convolution kernel of 3×3. Finally, the deconvolution layer restores the spatial size of the channel regularization features to obtain the human joint positions used for network training, and calculates the loss value with the true value. In this application, the position of the true value is provided by the data set annotation. Generally, the L2 loss value is used to calculate the true value and the human joint position of the network training. The human joint position of the network training is presented in the form of a heatmap in this application, which can further improve the accuracy of key point estimation. The testing and training process of this application is consistent and will not be repeated.

[0060] The channel adjustment module includes a function for downsampling image features Expand to The channel expansion feature is used to enhance the human posture perception ability in the extended channel, and the channel of the channel expansion feature is separated to obtain four parallel space features with the same space size and adjacent channels. The channel separation unit has four parallel space features: the first parallel space feature , the second parallel space feature , the third parallel space features And the fourth parallel space characteristics In this embodiment, channel separation is implemented by using the Python Split() function, which is a commonly used instruction in Python and is therefore not described in detail. The formula is as follows:

[0061] ;

[0062] Specifically, the channel expansion unit is used to expand the channel size of the downsampled image features. Expand to The channel expansion feature enhances the human posture perception ability in the expanded channel. In fact, after the channel is expanded, although the overall parameters and computational complexity of the network increase slightly, the enlarged channel has a stronger feature learning ability. When it comes to the key point recognition of human posture estimation, the enlarged channel is conducive to capturing more detailed spatial information, thereby improving the final effect of human posture estimation. Indicates size The set of downsampled image features, Indicates size The set of channel expansion features, Respectively represent the height and width of the feature and the channel size. In this application, is H / 16, is W / 16, where H and W are the height and width of the input image respectively.

[0063] The parallel space interaction module includes a split head Transformer sub-block for performing parallel space internal interaction on four parallel space features, and a convolution sub-block for gradually interleaving parallel space external interaction after performing parallel space internal interaction.

[0064] There are four split head Transformer sub-blocks in total. Each split head Transformer sub-block performs parallel space internal interaction on each parallel space feature separately to obtain the first parallel space interaction feature, the second parallel space interaction feature, the third parallel space interaction feature and the fourth parallel space interaction feature.

[0065] The convolution sub-block includes a first convolution for outputting an interactive feature outside the first parallel space according to the interactive feature inside the first parallel space and the interactive feature inside the second parallel space; a second convolution for outputting an interactive feature outside the second parallel space according to the interactive feature outside the first parallel space and the interactive feature inside the third parallel space; and a third convolution for outputting a parallel space interactive feature according to the interactive feature outside the second parallel space and the interactive feature inside the fourth parallel space. Both the first convolution and the second convolution are convolutions with a convolution kernel of 3×3, and the third convolution is a convolution with a convolution kernel of 5×5.

[0066] The split head Transformer sub-block includes the global position relationship of human joints of different scales within each parallel spatial feature, and finally outputs the head split attention feature. Head split attention, and head split attention features Perform linear transformation and convolution calculation to obtain linear convolution transformation features Linear convolutional transformation layer.

[0067] This embodiment performs parallel space internal interaction on parallel space features through the split head Transformer sub-block. The head split attention in the split head Transformer sub-block effectively improves the self-attention capture ability of different spatial sizes, and the design of low spatial self-attention effectively improves the efficiency of the overall method, ensuring that the present invention has the ability to complete accurate key point estimation with lower parameters and computational complexity.

[0068] In addition, the constructed linear convolution conversion layer can further improve the perception of the channel, and then use deep convolution to process spatial information in the extended channel to help reduce the information loss of the network in learning parallel space features. In addition, the present invention improves the feature learning ability of this method between different spaces by designing convolution sub-blocks to perform parallel space external interaction on the four parallel space features.

[0069] The head split attention includes the self-attention value calculated by the three calculation matrices of value V, keyword K, and query Q. and High spatial self-attention and low spatial self-attention, as well as the self-attention value and Fusion is performed to obtain the global positional relationship of human joints of different scales within each parallel spatial feature, and finally the head split attention feature of the head split attention output is obtained. In this embodiment, the total number of attention heads of the head split attention is N, and the number of attention heads of the high spatial self-attention and the low spatial self-attention are (1-b)×N and b×N respectively. Specifically, in this application, N is generally taken as 8, and b is 0.5.

[0070] A value for high spatial self-attention is set between low spatial self-attention and high spatial self-attention , Keywords The spatial reduction unit is used to compress the size of the space, and the compression scale is M = 4. Specifically, the spatial reduction unit is set so that when the number of attention heads is the same, the loss of low spatial self-attention is lower than that of high spatial self-attention, which ensures that the training process of the present invention is carried out under a lower parameter and calculation amount.

[0071] In one embodiment of the present application, the attention fusion unit is implemented using the Concat() function. and Fusion is performed. Of course, this application does not limit the fusion method, that is, matrix multiplication or addition can be used to achieve fusion. The Concat() function is used here to ensure that the final number of output heads is equal to the total number of attention heads of the head split attention.

[0072] The linear convolutional transformation layer includes the attention features for head splitting The first linear layer with the number of channels expanded to twice that of the first linear layer, and the deep convolution that processes spatial information in the channels after the expansion is doubled , and restore the number of channels to the same as the head split attention feature after processing the spatial information The second linear layer with the same number of original channels. The number of channels of the head split attention feature can be increased before convolution. The increase in the number of channels effectively improves the detail feature capture effect in the convolution process, which is beneficial to the final human posture estimation performance. Therefore, the linear convolution conversion layer constructed by the present invention can further improve the perception of the channel, and subsequently use deep convolution to process spatial information in the extended channel to assist in reducing the information loss of the network in learning features in parallel space.

[0073] The present invention is a neural network model, a new idea for human posture estimation, which uses multiple modules to extract the position of human joints in an image. The specific steps are introduced below. First, the input image is obtained and the input image is quickly downsampled to obtain the downsampled image feature; then the channel size of the downsampled image feature is expanded using the channel adjustment module to obtain the channel expansion feature, and the channel of the channel expansion feature is separated to obtain four parallel space features with the same space size and adjacent channels.

[0074] Next, the parallel space interaction module is used to perform parallel space internal interaction and parallel space external interaction on the four parallel space features. Among them, the parallel space features are interacted internally in parallel space through the split head Transformer sub-block. The head split attention in the split head Transformer sub-block effectively improves the self-attention capture ability of different spatial sizes, and the constructed linear convolution conversion layer can further improve the perception ability of the channel; in addition, the four parallel space features are interacted externally in parallel space through the designed convolution sub-block to obtain the parallel space interaction features.

[0075] Then, the channel regularization unit is used to regularize the channel of the parallel space interaction feature to obtain the channel regularization feature; finally, the deconvolution layer is used to restore the spatial size of the channel regularization feature to obtain the human joint position used for network training, and the loss value is calculated with the true value to complete the network training of this application. The present invention uses a channel adjustment module and a channel regularization unit to expand, transform and split the channel. Since the four parallel space features are split along the channel, the setting of the channel adjustment module and the channel regularization unit effectively avoids the loss of channel details caused by channel splitting, improves the learning ability of the channel, and ultimately improves the overall effect of human posture estimation.

[0076] This embodiment explores the impact of parallel space interaction on human posture estimation by performing spatial interaction within and between parallel spaces. Global feature interaction is completed both internally and externally, which is of great benefit for capturing the relationship between key points of human posture estimation and ensures the high efficiency of human posture estimation.

[0077] For the above parallel space internal and external interactive system, please refer to Figure 3-Figure 4 This embodiment also proposes a parallel space internal and external interaction method for human body posture estimation, which includes the following steps:

[0078] S1. Obtain an input image and perform fast downsampling to obtain downsampled image features. In the present application, the input image can be downsampled by adopting a conventional downsampling mechanism. For the task of human posture estimation, the downsampled image can better retain the global information of the image. More importantly, the downsampled image is convenient for network learning, otherwise it will seriously affect the running speed of the network.

[0079] S2, after expanding the channel size of the downsampled image feature, separation is performed to obtain four parallel spatial features with the same spatial size and adjacent channels; as a further preferred implementation of this embodiment, the specific process includes:

[0080] First, the downsampled image features are convolved using a 5×5 convolution kernel. Expand to ;

[0081] Then use a linear layer to improve the channel to , and then improve the learning ability of key positions through linear norm and activation function GeLU;

[0082] Then use another linear layer to restore the channel to , and obtain channel expansion features through dimension transformation Reshape , the expression is:

[0083] ;

[0084] in, Indicates dimension conversion; Indicates a convolution with a convolution kernel of 5×5; and Respectively represent Linear Layer linear layer and Linear Norm linear norm; Represents the activation function GeLU.

[0085] Finally, the Python Split() function is used to expand the channel features. The channels are separated to obtain four parallel spatial features with the same spatial size and adjacent channels. , respectively, the first parallel space characteristics , the second parallel space feature , the third parallel space features And the fourth parallel space characteristics .

[0086] S3, performing parallel space internal interaction and parallel space external interaction on the four parallel space features in sequence to obtain parallel space interaction features; as a further preferred implementation of this embodiment, the specific process includes:

[0087] The self-attention value is calculated based on the three calculation matrices of value V, keyword K, and query Q, where:

[0088] Self-attention value for high spatial self-attention The calculation is as follows:

[0089] ;

[0090] Self-attention value for low spatial self-attention The calculation is as follows:

[0091] ;

[0092] Among them, the dimensions of each calculation matrix are as follows:

[0093] , and M are both constants, with values ​​of 256 and 4 respectively. Here, M refers to the compression scale in the spatial reduction unit; N refers to the length of the one-dimensional vector converted by the spatial scale in the head split attention, that is, , which belongs to the common sense technology in Transformer. Among them, is H / 16, is W / 16, H and W are the height and width of the input image; , They respectively represent the feature dimension representation before space reduction.

[0094] In the above formula, , are the value and keyword of high spatial self-attention respectively; T represents transposition; , Represent the value and keyword of low spatial self-attention respectively; and are all constants.

[0095] In the above embodiment, the spatial vector of low spatial self-attention is smaller than that of high spatial self-attention. In this way, high spatial self-attention and low spatial self-attention can capture global features of different scales in each parallel spatial feature. This approach also avoids the defect of overly similar global feature extraction caused by the consistency of parallel spatial features, and improves the network's learning ability for human body postures of different scales. In addition, a spatial reduction unit is provided between low spatial self-attention and high spatial self-attention. Therefore, when the number of heads is the same, the loss of low spatial self-attention is lower than that of high spatial self-attention, ensuring that the training process of the present invention is carried out at a lower parameter and computational complexity.

[0096] Self-attention value and Fusion is performed to obtain the global positional relationship of human joints of different scales within each parallel spatial feature, and finally the head split attention feature is obtained. ;

[0097] Split attention features on the head Perform linear transformation and convolution calculation to obtain linear convolution transformation features , the expression is:

[0098] ;

[0099] in, Indicates that the convolution kernel is a 3×3 depth convolution; represents the first linear layer, represents the second linear layer; Indicates dimension conversion; Represents the activation function GeLU.

[0100] Among them, the first linear layer increases the number of channels of the head split attention feature to two times, and then uses deep convolution to process spatial information in the extended channel to reduce the loss of parallel spatial information. The second linear layer restores the number of channels to be consistent with the number of channels of the head split attention feature. This approach mainly increases the number of channels of the head split attention feature before convolution. The increase in the number of channels effectively improves the effect of capturing detailed features during the convolution process, which is beneficial to the final human posture estimation performance.

[0101] Transform features of linear convolution by transforming feature dimensions Perform spatial dimension conversion and four linear convolution conversion features The internal interaction in the parallel space is performed on each parallel space feature respectively, and the first parallel space interaction feature, the second parallel space interaction feature, the third parallel space interaction feature and the fourth parallel space interaction feature are obtained.

[0102] In this application, executing the internal interaction of parallel spaces can complete the global feature extraction of different parallel spaces. According to previous methods, it can be known that spatial interaction is beneficial to human posture estimation. For example, in the prior art application number CN202310201240.7, the invention name is a human posture estimation method based on multidimensional space interaction (hereinafter referred to as the multidimensional interaction method), which records the impact of multidimensional space interaction on human posture estimation. However, the multidimensional interaction method does not explore the interactive operation performed inside the parallel space, but uses Transformer to complete the overall global interaction, and the interaction inside the parallel space has not been studied in depth. Compared with the multi-interaction method, this application not only interacts inside the parallel space, but also connects the external interaction of the parallel space to the internal interaction, and completes the interaction process inside and outside multiple parallel spaces. It not only overcomes the previous situation that the internal interaction performance of the parallel space is unclear, but also completes the global interaction of different scales in the parallel space through the designed split head Transformer sub-block, effectively avoiding the defects of excessive parameters and calculations, which is very important for the efficiency of human posture estimation.

[0103] Multiply the interactive features in the first parallel space by the interactive features in the second parallel space, perform spatial interaction between the interactive features in the first parallel space and the interactive features in the second parallel space, and then obtain the interactive features outside the first parallel space through the first convolution;

[0104] Multiply the first parallel space external interaction feature and the third parallel space internal interaction feature, perform spatial interaction between the first parallel space external interaction feature and the third parallel space internal interaction feature, and then obtain the second parallel space external interaction feature through a second convolution;

[0105] Multiply the interaction features outside the second parallel space and the interaction features inside the fourth parallel space, perform spatial interaction between the interaction features outside the second parallel space and the interaction features inside the fourth parallel space, and then obtain the interaction features outside the third parallel space through the third convolution;

[0106] Finally, the parallel space external interaction of the four parallel space characteristics is completed to obtain the parallel space interaction characteristics.

[0107] In the present application, the acquisition of four parallel space features is only an example. In fact, those skilled in the art can perform different interactions inside and outside parallel spaces by changing the number of parallel space features. This change is foreseeable and therefore also falls within the scope of this application.

[0108] S4. Regularize the channels of the parallel space interaction features to obtain channel regularization features. Specifically, in this embodiment, a deep convolution with a convolution kernel of 3×3 can be used to implement this step, or other channel regularization operations can be used to implement this step, such as implementing the channel expansion unit in the present invention, and then using convolution to reduce the number of channels.

[0109] S5. Restore the spatial size of the channel regularization feature to obtain the human joint position for network training, and calculate the loss value with the true value. In this application, the position of the true value is provided by the data set annotation, and the L2 loss value is generally used to calculate the true value and the human joint position of the network training. The human joint position of the network training is presented in the form of a heatmap in this application, which can further improve the accuracy of key point estimation. The test and training process of this application are consistent, so it will not be repeated.

[0110] According to the records of the above method, the present invention gives the results on the COCO dataset, the calculation amount is in GFLOPs, and the running speed is in FPS, as shown in the following table.

[0111] Experimental comparison results of the proposed method with other advanced methods on COCO val

[0112]

[0113] The method of this application can achieve an average accuracy of 74.4% on the COCO val dataset, and the speed reaches 180 frames per second when the input image size is 256×192. It can be seen that this application has achieved a better balance in both accuracy and running speed.

[0114] Among them, TransPoseA4 is the method in the paper "TransPose: Keypoint localization via Transformer"; Poseur is the method in the paper "Poseur: Direct human pose regression with transformers"; HRNet-W32 is the method in the paper "Deep high-resolution representation learning for human pose estimation"; PPT-S is the method in the paper "Ppt: token-pruned pose transformer for monocular and multi-view human pose estimation"; DistilPose is the method in the paper "DistilPose: Tokenized pose regression with heatmap distillation".

[0115] The present invention explores the impact of parallel space interaction on human posture estimation by performing spatial interaction within and between parallel spaces. Global feature interaction is completed both internally and externally, which is of great benefit for capturing the relationship between key points of human posture estimation and ensures the high efficiency of human posture estimation.

[0116] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0117] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A parallel space internal and external interactive system for human posture estimation, comprising a downsampling module for acquiring an input image and quickly downsampling the input image to obtain downsampled image features, characterized in that: Also includes: A channel adjustment module, the channel adjustment module is used to expand the channel size of the downsampled image feature to obtain a channel expansion feature, and separate the channels of the channel expansion feature to obtain four parallel space features with the same spatial size and adjacent channels; A parallel space interaction module, the parallel space interaction module is used to perform parallel space internal interaction and parallel space external interaction on the four parallel space features to obtain parallel space interaction features; The interaction inside the parallel space takes precedence over the interaction outside the parallel space; A channel regularization unit, the channel regularization unit is used to regularize the channel of the parallel space interaction feature to obtain a channel regularization feature; The deconvolution layer is used to restore the spatial size of the channel regularization feature, obtain the human joint position for network training, and calculate the loss value with the true value.

2. The parallel space internal and external interactive system according to claim 1, characterized in that: The channel adjustment module includes a module for downsampling image features Expand to The channel expansion feature is used to enhance the human posture perception ability in the extended channel, and the channel of the channel expansion feature is separated to obtain four parallel space features with the same space size and adjacent channels. Channel separation unit.

3. The parallel space internal and external interactive system according to claim 1, characterized in that: The parallel space interaction module includes a split head Transformer sub-block for performing parallel space internal interaction on the four parallel space features, and a convolution sub-block for gradually and interleavingly performing parallel space external interaction after performing the parallel space internal interaction.

4. The parallel space internal and external interactive system according to claim 3, characterized in that: There are four split head Transformer sub-blocks in total, and each split head Transformer sub-block performs parallel space internal interaction on each parallel space feature separately to obtain a first parallel space internal interaction feature, a second parallel space internal interaction feature, a third parallel space internal interaction feature, and a fourth parallel space internal interaction feature; The convolution sub-block includes a first convolution for outputting a first parallel space external interaction feature according to the first parallel space internal interaction feature and the second parallel space internal interaction feature; and outputting a second convolution of a second parallel space external interaction feature according to the first parallel space external interaction feature and the third parallel space internal interaction feature; And a third convolution is performed to output the parallel space interaction feature according to the second parallel space external interaction feature and the fourth parallel space internal interaction feature.

5. The parallel space internal and external interactive system according to claim 3 or 4, characterized in that: The split head Transformer sub-block includes a global position relationship of human joints of different scales within each parallel space feature, and finally outputs the head split attention feature Head split attention, and head split attention features Perform linear transformation and convolution calculation to obtain linear convolution transformation features Linear convolutional transformation layer.

6. The parallel space internal and external interactive system according to claim 5, characterized in that: The head split attention includes a method for calculating the self-attention value through three calculation matrices: value V, keyword K, and query Q. and The total number of attention heads of the head split attention is N, and the numbers of attention heads of the high spatial self-attention and the low spatial self-attention are (1-b)×N and b×N respectively; And the self-attention value and Fusion is performed to obtain the global position relationship of human joints of different scales within each of the parallel space features, and finally the head split attention feature of the head split attention output is obtained. Attention fusion unit.

7. The parallel space internal and external interactive system according to claim 5, characterized in that: The linear convolutional transformation layer includes the head split attention feature The first linear layer with the number of channels expanded to twice that of the first linear layer, and the deep convolution that processes spatial information in the channels after the expansion is doubled , and restore the number of channels to the same as the head split attention feature after processing the spatial information The second linear layer with the same number of original channels.

8. The parallel space internal and external interactive system according to claim 6, characterized in that: A value for the high spatial self-attention is set between the low spatial self-attention and the high spatial self-attention. , Keywords A space reduction unit for compressing the space size.

9. A parallel space internal and external interaction method applied to the parallel space internal and external interaction system according to any one of claims 1 to 8, characterized in that: The following steps are involved: Get the input image and perform fast downsampling to obtain downsampled image features; After expanding the channel size of the downsampled image features, four parallel spatial features with the same spatial size and adjacent channels are obtained by separation; Performing parallel space internal interaction and parallel space external interaction on the four parallel space features in turn to obtain parallel space interaction features; Regularizing the channels of the parallel space interaction features to obtain channel regularization features; The spatial size of the channel regularization features is restored to obtain the human joint positions used for network training.

10. The parallel space internal and external interaction method according to claim 9, characterized in that: The specific process of obtaining the parallel space interaction characteristics is as follows: The self-attention value is calculated based on the three calculation matrices of value V, keyword K, and query Q, where: Self-attention value for high spatial self-attention The calculation is as follows: ； Self-attention value for low spatial self-attention The calculation is as follows: ； In the above formula, , are the value and keyword of high spatial self-attention respectively; T represents transposition; , Represent the value and keyword of low spatial self-attention respectively; and are all constants; Self-attention value and Fusion is performed to obtain the global positional relationship of human joints of different scales within each parallel spatial feature, and finally the head split attention feature is obtained. ; Split attention features on the head Perform linear transformation and convolution calculation to obtain linear convolution transformation features , the expression is: ； in, Indicates that the convolution kernel is a 3×3 depth convolution; represents the first linear layer, represents the second linear layer; Indicates dimension conversion; Represents the activation function GeLU; Transform features of linear convolution by transforming feature dimensions Perform spatial dimension conversion and four linear convolution conversion features Performing parallel space internal interaction on each parallel space feature respectively, to obtain a first parallel space interaction feature, a second parallel space interaction feature, a third parallel space interaction feature, and a fourth parallel space interaction feature; Multiply the interactive features in the first parallel space by the interactive features in the second parallel space, perform spatial interaction between the interactive features in the first parallel space and the interactive features in the second parallel space, and then obtain the interactive features outside the first parallel space through the first convolution; Multiply the first parallel space external interaction feature and the third parallel space internal interaction feature, perform spatial interaction between the first parallel space external interaction feature and the third parallel space internal interaction feature, and then obtain the second parallel space external interaction feature through a second convolution; Multiply the interaction features outside the second parallel space and the interaction features inside the fourth parallel space, perform spatial interaction between the interaction features outside the second parallel space and the interaction features inside the fourth parallel space, and then obtain the interaction features outside the third parallel space through the third convolution; Finally, the parallel space external interaction of the four parallel space characteristics is completed to obtain the parallel space interaction characteristics.

Images