Human pose estimation method, system and device based on binary self-attention

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a human pose estimation method, system and device based on two-part self-attention, aiming to solve the problem that the attention mechanism module is difficult to meet the high real-time requirement of algorithm landing in the process of use, and affects the system inference speed. The present application comprises: obtaining a human image to be pose estimated as an input image; inputting the input image into a two-part self-attention-based human pose estimation network trained to obtain a pose estimation heat map; and the two-part self-attention-based human pose estimation network is constructed based on a convolutional neural network. The two-part self-attention-based human pose estimation network given in the present application can better extract semantic features related to human pose key points through feature selection and strengthening by an attention module on the basis of down-sampling and up-sampling structure.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a human pose estimation method, system and device based on two-part self-attention. BACKGROUND

[0002] Human pose estimation is applied more and more widely. With the advent of the big data era, deep learning has been successfully applied in the field of computer vision. In recent years, more and more researches use deep learning to solve the problem of human pose estimation, and various methods emerge in an endless stream. Some scholars propose that adding an attention module can further improve the algorithm accuracy. The basic idea of the attention mechanism is to let the system ignore irrelevant information and focus on important information, so as to select important features from the input information and help detection by obtaining important semantic information. However, due to the need to capture context information, the attention mechanism module has a large computational complexity and memory capacity.

[0003] With the development of deep learning technology, new algorithm models are emerging, and the current Top-down method generally achieves high algorithm accuracy. Taking SimpleBaseline as an example, a simple down-sampling plus up-sampling structure can achieve good detection effect. On this basis, Alphapose adds an SE module as an attention module in the down-sampling part, and uses PixelShuffle for up-sampling, which improves the accuracy while taking into account the speed. The latest research is to use a complex network design to obtain more rich multi-scale features, and even rely on Transformer to obtain a larger receptive field and stronger representation ability. For example, HRNet maintains high resolution throughout the network, and uses parallel branches to connect different resolution feature maps and repeatedly multi-scale fusion to reduce information loss, so as to obtain higher resolution representation. This structure brings higher performance. ViTPose uses pure ViT as an encoder to learn features, and uses a simple decoder to decode the learned features to obtain more accurate prediction results.

[0004] However, complex network design means huge computational load, which seriously affects the inference speed, so it is difficult to meet the high real-time demand of algorithm landing. Considering the requirement of inference speed, the structure based on down-sampling plus up-sampling is still widely used in the industry. Although the attention module can bring a certain degree of accuracy improvement, its computational complexity, especially the impact on inference speed, should not be ignored. Based on this, the application proposes a human pose estimation method, system and device based on two-part self-attention. SUMMARY

[0005] To solve the problems in the prior art that the attention mechanism module is difficult to meet the high real-time requirement of algorithm landing in use, and affects the inference speed of the system, the present application provides a human pose estimation method based on a two-part self-attention, which comprises:

[0006] An image of a human body to be pose estimated is acquired as an input image.

[0007] The input image is input into a human pose estimation network based on two-part self-attention which has been trained to obtain a pose estimation heat map.

[0008] The human pose estimation network based on two-part self-attention is constructed based on a convolutional neural network.

[0009] In some preferred embodiments, the input image is input into the human pose estimation network based on two-part self-attention which has been trained to obtain a pose estimation heat map, and the method is as follows:

[0010] The feature extraction module of the human pose estimation network based on two-part self-attention extracts feature maps of multiple scales of the input image.

[0011] The feature enhancement module of the human pose estimation network based on two-part self-attention enhances and fuses the feature maps of multiple scales to obtain a fused feature map.

[0012] The fused feature map is input into the heat map generation module of the human pose estimation network based on two-part self-attention to generate a heat map of human pose key points in the input image.

[0013] The method for generating the heat map of human pose key points is as follows: the fused feature map is converted to a heat map of human pose key points through convolution after the number of channels is equal to the number of human pose key points.

[0014] In some preferred embodiments, the feature enhancement module of the human pose estimation network based on two-part self-attention enhances and fuses the feature maps of multiple scales to obtain a fused feature map, and the method is as follows:

[0015] Step A100, initializing a first feature map; wherein when the first feature map is initialized, the first feature map is the feature map with the smallest scale among the multiple scale feature maps.

[0016] Step A200, upsampling the first feature map to obtain a second feature map.

[0017] Step A300, performing information fusion processing on the second feature map through convolution with a set size to obtain a third feature map.

[0018] Step A400, performing channel feature enhancement processing on the third feature map through a pre-constructed two-part self-attention module to obtain a fourth feature map;

[0019] Step A500, taking the fourth feature map as a first feature map, jumping to step A200, and inputting the fourth feature map as a fusion feature map until the size of the fourth feature map is equal to the size of a feature map with the largest size among the plurality of size feature maps.

[0020] In some preferred embodiments, the third feature map is processed by a pre-constructed two-part self-attention module for channel feature enhancement, and the method is as follows:

[0021] Step A411, inputting a feature map X, i.e., the third feature map, the channel number of the feature map X being C1, and performing 1x1 convolution on the feature map X to obtain a feature map Y, the channel number of the feature map Y being C2;

[0022] Step A412, dividing the Y by channel number to obtain A and B, multiplying the A and the B, and performing global average pooling to obtain a feature map after global average pooling, i.e., a C2 / 2-dimensional vector F sq ;

[0023] Step A413, performing channel feature learning on the feature map after global average pooling, converting the F sq back to a C1-dimensional vector F ex through an FC layer, normalizing the F ex through a LayerNorm, and then obtaining a weight vector w through a sigmoid function.

[0024] Step A414, multiplying the C1-dimensional weight vector w and the C1-dimensional X in the channel to obtain a feature map with channel attention, i.e., w*X, and then obtaining a feature map after channel feature enhancement.

[0025] The up-sampling ratio when performing up-sampling is the same as the ratio of two feature maps with adjacent sizes among the plurality of size feature maps, wherein the ratio of the two feature maps with adjacent sizes is the ratio of a feature map with a larger size to a feature map with a smaller size.

[0026] In some preferred embodiments, the method for performing information fusion and feature enhancement on the plurality of size feature maps U is as follows:

[0027] U i-1 = attention(conva*b(PixelShuffle(U i )));

[0028] Wherein, the i is: the number of the plurality of scale feature maps U, the a*b is the size of the convolution kernel, the a and the b are any number between 0-10.

[0029] In some preferred embodiments, the calculation method of the weight vector w is:

[0030] w=sigmoid(FC(pool(WX|split1⊙WX|split2))).

[0031] The second aspect of the application provides a human pose estimation method based on binary self-attention, which comprises an image acquisition module and a heat map generation module.

[0032] The image acquisition module is configured to acquire a human image to be pose estimated as an input image.

[0033] The heat map generation module is configured to input the input image into a human pose estimation network based on binary self-attention which has been trained to obtain a pose estimation heat map.

[0034] The human pose estimation network based on binary self-attention is constructed based on a convolutional neural network.

[0035] The third aspect of the application provides a storage device, wherein a plurality of programs are stored, the programs being suitable for being loaded and executed by a processor to realize the human pose estimation method based on binary self-attention.

[0036] The fourth aspect of the application provides a processing device, comprising a processor and a storage device; the processor is suitable for executing each program; the storage device is suitable for storing a plurality of programs; characterized in that the programs are suitable for being loaded and executed by the processor to realize the human pose estimation method based on binary self-attention.

[0037] The beneficial effects of the application are:

[0038] (1) The human pose estimation network based on binary self-attention provided by the application can better extract semantic features related to human pose key points through the attention module for feature selection and strengthening on the basis of the down-sampling and up-sampling structure.

[0039] (2) In the up-sampling stage, PixelShuffle is combined, and spatial attention is also considered, which is helpful for accurate positioning of key point positions; at the same time, due to the simple structure of the attention module and high calculation efficiency, although the calculation complexity increases slightly, the calculation time consumption does not increase significantly. BRIEF DESCRIPTION OF DRAWINGS

[0040] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in conjunction with the accompanying drawings:

[0041] Figure 1 is a flowchart of a human pose estimation method based on a two-part self-attention according to an embodiment of the application;

[0042] Figure 2 is a flowchart of a Resnet50-based feature extraction process of a human pose estimation method based on a two-part self-attention according to an embodiment of the application;

[0043] Figure 3 is a schematic diagram of the working principle of a two-part self-attention module of a human pose estimation method based on a two-part self-attention according to an embodiment of the application;

[0044] Figure 4 is a flowchart of a two-part self-attention module of a human pose estimation method based on a two-part self-attention according to an embodiment of the application;

[0045] Figure 5 is a schematic diagram of the framework of a human pose estimation system based on a two-part self-attention according to an embodiment of the application;

[0046] Figure 6 is a schematic diagram of the structure of a computer system of a server for implementing the method, system, and device embodiments of the application. DETAILED DESCRIPTION

[0047] The application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the related application, and not to limit the application. In addition, it should be noted that only parts related to the application are shown in the drawings for ease of description.

[0048] It should be noted that the embodiments and features in the embodiments of the application can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0049] Referring to Figure 1 , the human pose estimation method based on a two-part self-attention according to the first embodiment of the application, the method comprises:

[0050] obtaining a human image to be pose estimated as an input image;

[0051] inputting the input image into the trained human pose estimation network based on a two-part self-attention to obtain a pose estimation heat map;

[0052] The human pose estimation network based on the two-part self-attention is constructed based on a convolutional neural network.

[0053] The input image is preprocessed after being acquired, and the image enhancement operation includes brightness transformation, contrast enhancement, histogram equalization, color space conversion, etc.

[0054] In the embodiment, the input image is input into the human pose estimation network based on the two-part self-attention which is trained, to obtain a pose estimation heat map, and the method is as follows:

[0055] The feature extraction module 100 of the human pose estimation network based on the two-part self-attention extracts feature maps of multiple scales of the input image.

[0056] The feature enhancement module 200 of the human pose estimation network based on the two-part self-attention enhances and fuses the feature maps of multiple scales to obtain a fused feature map.

[0057] The fused feature map is input into the heat map generation module 300 of the human pose estimation network based on the two-part self-attention to generate a heat map of human pose key points in the input image.

[0058] The method for generating the heat map of human pose key points is as follows: the fused feature map is converted to a channel number equal to the number of human pose key points through convolution to obtain a heat map of human pose key points.

[0059] For example, the feature maps of multiple scales include stage1, stage2, stage3 and stage4, that is, feature map D1, feature map D2, feature map D3 and feature map D4. Preferably, the scale of the feature map D1 is 1 / 4 of the resolution of the input image; the scale of the feature map D2 is 1 / 8 of the resolution of the input image; the scale of the feature map D3 is 1 / 16 of the resolution of the input image; and the scale of the feature map D4 is 1 / 32 of the resolution of the input image.

[0060] Preferably, the feature enhancement module 200 of the human pose estimation network based on the two-part self-attention enhances and fuses the feature maps of multiple scales to obtain a fused feature map, and the method is as follows:

[0061] Step A100, initializing a first feature map; when the first feature map is initialized, the first feature map is the feature map with the smallest scale in the multiple scale feature maps.

[0062] Step A200, upsampling the first feature map to obtain a second feature map.

[0063] Step A300, information fusion processing is performed on the second feature map through a convolution of a set size to obtain a third feature map;

[0064] Step A400, channel feature enhancement processing is performed on the third feature map through a pre-constructed binary self-attention module 400 to obtain a fourth feature map;

[0065] Step A500, the fourth feature map is taken as a first feature map, and step A200 is jumped to until the scale of the fourth feature map is equal to the scale of a feature map with the largest scale among the plurality of scale feature maps, and the fourth feature map is input as a fusion feature map.

[0066] For example, as shown in FIG. 1, a Resnet50 is taken as a backbone network for feature extraction. Figure 2 It should be noted that the method is not limited to the backbone network, and any network with multi-scale feature extraction capability can be used. The binary self-attention module 400 is inserted at the end of each stage of the Resnet50. Since the binary self-attention module 400 does not change the size of the feature map, the original structure of the backbone network will not be damaged.

[0067] Preferably, the way of information fusion and channel feature enhancement of the plurality of scale feature maps U is as follows:

[0068] U i-1 = attention(conv a*b(PixelShuffle(U i ))) (1)

[0069] Wherein, the i is: the number of the plurality of scale feature maps, and the a*b is the size of the convolution kernel.

[0070] Wherein, the value range of the a and the b is 0-10, and preferably 3.

[0071] For example, let the D4=U4, that is, the first feature map, the first feature map is up-sampled through the PixelShuffle method, and the up-sampled U4 is denoted as a two-scale feature map U4_up, that is, a second feature map, the two-scale feature map is information fused, and preferably, the information fusion can be performed through a 3*3 convolution to obtain U3, that is, a third feature map, the third feature map is channel feature enhanced, and the scale of the U3 is the same as that of the D3, both of which are 1 / 16 of the resolution of the input image. In this way, the U3 is information fused and channel feature enhanced to obtain U2; the U2 is information fused and channel feature enhanced in the same way, so as to obtain U1 which is 1 / 4 of the resolution of the input image, that is, a fourth feature map.

[0072] Preferably, as shown in FIG. 2,Figure 3 、 Figure 4 , performing channel feature enhancement processing on the third feature map through the pre-constructed two-part self-attention module 400, the method being:

[0073] Step A411, input the feature map X, that is, the third feature map, the channel number of the feature map X is C1, and the feature map Y is obtained by performing 1x1 convolution on the feature map X, the channel number of the feature map Y is C2;

[0074] Step A412, divide the Y by channel number to obtain A and B, multiply the A and the B, and then perform global average pooling to obtain a feature map after global average pooling, that is, a C2 / 2-dimensional vector F sq ;

[0075] Step A413, performing channel feature learning on the feature map after global average pooling, and converting the F sq back to a C1-dimensional vector F ex through an FC layer, and then performing LayerNorm normalization and a sigmoid function to obtain a weight vector w; ex

[0076] Step A414, multiplying the C1-dimensional weight vector w and the C1-dimensional X in the channel to obtain a feature map with channel attention, that is, w*X, and then obtaining a feature map after channel feature enhancement.

[0077] Wherein, the 1x1 convolution is represented as: conv1x1.

[0078] Wherein, Figure 3 H is the height of the feature maps of multiple scales, and W is the width of the feature maps of multiple scales.

[0079] Preferably, the up-sampling ratio when up-sampling is the same as the ratio of two feature maps of adjacent scales in the feature maps of multiple scales, wherein the ratio of two feature maps of adjacent scales is the ratio of a feature map of a larger scale to a feature map of a smaller scale.

[0080] For example, 1 / 4 divided by 1 / 8 equals 2, and the up-sampling ratio is 2.

[0081] Wherein, the calculation method of the weight vector w is:

[0082] w=sigmoid(FC(pool(WX|split1⊙WX|split2))) (2)

[0083] ​Wherein, the pool is a pooling calculation, and the PixelShuffle is an upsampling method, which can effectively enlarge the reduced feature map. By setting the upsampling ratio, a high-resolution image of a specified multiple can be obtained from a low-resolution image. Here, the upsampling ratio is 2, so the low-resolution image of the first four channels can be spliced into the high-resolution first channel space.

[0084] Wherein, the two-part self-attention module 400, although it enhances the channel features, can be converted into attention to the spatial position through the subsequent PixelShuffle operation. Therefore, the two-part self-attention of upsampling has the effect of spatial attention.

[0085] Wherein, compared with the SE, CBAM and other attention modules, the two-part self-attention module 400 does not perform pooling at the beginning, so more information can be retained; at the same time, the matrix point multiplication is faster and can provide better nonlinearity. Therefore, although the computational complexity increases slightly, the speed and effect are better than SE and CBAM.

[0086] Wherein, after obtaining the pose estimation heat map, the pose estimation heat map is subjected to L2 loss calculation with the GroundTruth heat map and is corrected.

[0087] As a verification, first, the speed of the two-part self-attention module is tested in terms of model inference time, rather than in terms of parameter amount and calculation amount to evaluate the speed of the method. The two-part self-attention module is compared with SE and CBAM in an experimental comparison, and the average time consumption of 1000 inferences is recorded for feature maps of different scales. The speed experiment is based on an entry-level RTX2060 graphics card, and the experimental results are shown in Table 1:

[0088] Inferred speed (ms) SE CBAM Ours (1,512,32,32) 0.67 0.78 0.57 (1,128,128,128) 1.58 1.96 0.75

[0089] It can be seen that for feature maps of different resolutions, the attention module achieves faster inference speed than SE and CBAM.

[0090] Method AP AR SimpleBaseline_res50 70.4 76.3 Alphapose_res50 71.2 77.1 Alphapose_res50+CBAM 71.3 77.1 Ours_res50 71.6 77.4

[0091] As shown in Table 2, compared with the SimpleBaseline, the increase of the attention module indeed improves the algorithm accuracy. The difference between the human pose estimation network given by the application and Alphapose lies in the different attention modules. Since the two-part self-attention module utilizes more original feature information, it is helpful for the extraction of semantic features; the two-part self-attention module in the upsampling stage can play the role of spatial attention, which is also helpful for the positioning of key points. Although the number of attention modules is increased, the overall time consumption does not increase due to the higher calculation efficiency.

[0092] Although the above-mentioned embodiments are described in the above-mentioned order, it is understood by those skilled in the art that, in order to achieve the effect of the embodiments, the different steps do not have to be executed in such an order, and can be executed simultaneously (in parallel) or in a reversed order, and these simple changes are within the protection scope of the present application.

[0093] The human pose estimation system based on the two-part self-attention of the second embodiment of the present application, see Figure 5 The system comprises a heat map generation module 300 and an image acquisition module 500.

[0094] The image acquisition module 500 is configured to acquire a human image to be pose estimated as an input image.

[0095] The heat map generation module 300 is configured to input the input image into the human pose estimation network based on the two-part self-attention which is trained to obtain a pose estimation heat map.

[0096] The human pose estimation network based on the two-part self-attention is constructed based on a convolutional neural network.

[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related description of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0098] It should be noted that the human pose estimation system based on the two-part self-attention provided in the above-mentioned embodiments is only exemplified by the division of the above-mentioned functional modules, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiments of the present application are further decomposed or combined, for example, the modules of the above-mentioned embodiments can be combined into one module, or can be further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules and steps, and should not be considered as an improper limitation of the present application.

[0099] The third embodiment of the present application is a storage device, wherein a plurality of programs are stored, and the programs are suitable for being loaded and executed by a processor to realize the human pose estimation method based on the two-part self-attention described above.

[0100] The fourth embodiment of the present application is a processing device comprising a processor and a storage device; the processor is suitable for executing each program; the storage device is suitable for storing a plurality of programs; characterized in that the programs are suitable for being loaded and executed by the processor to realize the human pose estimation method based on the two-part self-attention described above.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the storage device and the processing device described above and the related descriptions can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0102] Those skilled in the art can clearly understand that the modules and method steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described in the foregoing description. Whether the functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0103] Reference is made below to Figure 6 which shows a structural schematic diagram of a computer system of a server for implementing the method, system and device embodiments of the present application. Figure 6 The server shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0104] As shown in Figure 6 , the computer system includes a central processing unit (CPU) 601 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage portion 608 into a random access memory (RAM) 603. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] The following components are connected to the I / O interface 605: an input section 606 including input devices such as a keyboard and mouse; an output section 607 including output devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage section 608 as necessary.

[0106] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0107] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0108] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0109] The terms "first", "second", etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

[0110] The terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0111] The technical scheme of the present application has been described in combination with the preferred embodiments shown in the drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical schemes after the changes or replacements will all fall within the protection scope of the present application.

Claims

1. A human pose estimation method based on bipartite self-attention, characterized in that, The method comprises: obtaining a human body image to be estimated as an input image; inputting the input image into a trained human body pose estimation network based on a two-division self-attention to obtain a pose estimation heat map: extracting feature maps of multiple scales of the input image through a feature extraction module of the human body pose estimation network based on two-division self-attention; enhancing and fusing the feature maps of multiple scales through a feature enhancement module of the human body pose estimation network based on two-division self-attention to obtain a fused feature map: inputting the fused feature map into a heat map generation module of the human body pose estimation network based on two-division self-attention to generate a heat map of human body pose key points in the input image; wherein the method for generating the heat map of human body pose key points is: after the fused feature map is converted to a channel number equal to the number of human body pose key points through convolution, a heat map of human body pose key points is obtained; enhancing and fusing the feature maps of multiple scales through a feature enhancement module of the human body pose estimation network based on two-division self-attention to obtain a fused feature map, and the method is: Step A100, initializing a first feature map; wherein when the first feature map is initialized, the first feature map is the smallest scale feature map in the multiple scale feature maps; Step A200, upsampling the first feature map to obtain a second feature map; Step A300, performing information fusion processing on the second feature map through convolution of a set size to obtain a third feature map; Step A400, performing channel feature enhancement processing on the third feature map through a pre-constructed two-division self-attention module to obtain a fourth feature map; Step A500, taking the fourth feature map as the first feature map and jumping to Step A200 until the scale of the fourth feature map is equal to the scale of the largest scale feature map in the multiple scale feature maps, and inputting the fourth feature map as the fused feature map; The human body pose estimation network based on two-division self-attention is constructed based on a convolutional neural network.

2. The human pose estimation method based on bipartite self-attention according to claim 1, characterized in that, The method for performing channel feature enhancement processing on the third feature map through a pre-constructed two-division self-attention module is: Step A411, inputting a feature map X, i.e., the third feature map, wherein the channel number of the feature map X is C1, and performing 1x1 convolution on the feature map X to obtain a feature map Y, wherein the channel number of the feature map Y is C2; Step A412, divide the Y by the number of channels to obtain A and B, and multiply the A and the B, and then perform global average pooling to obtain a feature map after global average pooling, that is, a C2 / 2-dimensional vector F sq ; Step A413, channel feature learning is performed on the feature map after global average pooling, and the F sq The vector F is transformed back to the C1 dimension through the FC layer ex , F ex After normalization by LayerNorm, the weight vector w can be obtained through the sigmoid function. Step A414, multiplying the weight vector w of the C1 dimension with the X of the C1 dimension on the channel, That is, the feature map with channel attention, and then the feature map after channel feature enhancement is obtained.

3. The human pose estimation method based on bipartite self-attention according to claim 2, characterized in that, The upsampling ratio when upsampling is the same as the ratio of two feature maps of adjacent scales in the multiple scale feature maps, wherein the ratio of the two feature maps of adjacent scales is the ratio of the larger scale feature map to the smaller scale feature map.

4. The human pose estimation method based on bipartite self-attention according to claim 3, characterized in that, The method for performing information fusion and channel feature enhancement on the multiple scale feature maps U is: ； wherein the i is: the number of the feature map U of the plurality of scale feature maps, the is the size of the convolution kernel, and the a and the b are any number between 0-10.

5. The human pose estimation method based on bipartite self-attention according to claim 4, characterized in that, The calculation method of the weight vector w is: 。 6. The system for human pose estimation based on bipartite self-attention according to any one of claims 1 to 5, characterized in that, The system comprises an image acquisition module and a heat map generation module; The image acquisition module is configured to obtain a human body image to be estimated as an input image; The heat map generation module is configured to input the input image into a trained human body pose estimation network based on two-division self-attention to obtain a pose estimation heat map; The human pose estimation network based on the two-part self-attention is constructed based on a convolutional neural network.

7. A storage device in which a plurality of programs are stored, characterized by The program is applicable to being loaded and executed by a processor to implement the human pose estimation method based on the two-part self-attention according to any one of claims 1-5.

8. A processing device comprising a processor and a storage device; the processor is adapted to execute each program; the storage device is adapted to store a plurality of programs; characterized in that, The program is applicable to being loaded and executed by a processor to implement the human pose estimation method based on the two-part self-attention according to any one of claims 1-5.

Images