Joint point detection method, posture recognition method and device

Abstract

The present invention provides a joint point detection method, a posture recognition method and a device, and relates to the technical field of image processing. The method comprises: obtaining a target image to be detected; performing joint point detection on the target image through a joint point detection model to obtain a corresponding joint point heat map in the target image; wherein the joint point detection model comprises a feature extraction network and a feature fusion network; the feature extraction network comprises a dense connection block and an orthogonal attention module between the dense connection blocks; the orthogonal attention module is used to extract multi-scale features; the feature fusion network comprises a second-order fusion module of multiple stages, and the second-order fusion module of each stage is used to weightedly fuse the multi-scale features and the aggregated features output by the second-order fusion module of the previous stage; and determining the joint points in the target image based on the joint point heat map. The present invention can efficiently detect and obtain joint point detection results with high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a joint point detection method, a posture recognition method and a device. Background Art

[0002] Human posture recognition aims to locate key bone points (also called joints) such as eyes, ears and shoulders in images. With the development of neural network research and hardware facilities, the detection accuracy of joints has been greatly improved. Since multi-scale information plays an important role in joint detection, the current node detection model usually uses a cascade pyramid framework to extract multi-scale information. Although such joint models have high detection accuracy, the cascade pyramid framework is too large, resulting in low joint detection efficiency. At present, the joint detection model of a single pyramid framework is a lightweight model. The single pyramid framework is light enough compared to the cascade pyramid framework. However, the joint detection model of a single pyramid framework cannot extract enough multi-scale information, which will lead to low joint detection accuracy. In summary, the existing joint detection methods cannot efficiently detect and obtain high-precision joint detection results. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a joint point detection method, a posture recognition method and a device, which can efficiently detect and obtain joint point detection results with high accuracy.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a joint point detection method, comprising: acquiring a target image to be detected; performing joint point detection on the target image through a joint point detection model to obtain a joint point heat map corresponding to the target image; wherein the joint point detection model comprises a feature extraction network and a feature fusion network; the feature extraction network is used to extract multi-scale features; the feature fusion network comprises a plurality of stages of second-order fusion modules, and the second-order fusion module of each stage is used to perform weighted fusion on the multi-scale features and the aggregated features output by the second-order fusion module of the previous stage; and determining the joint points in the target image based on the joint point heat map.

[0006] In one embodiment, the joint point detection model also includes a heat map generation network; the feature extraction network includes a densely connected block and an orthogonal attention module between the densely connected blocks; the step of performing joint point detection on the target image through the joint point detection model to obtain a joint point heat map corresponding to the target image includes: performing feature extraction on the target image through each of the orthogonal attention modules in the feature extraction network to obtain multiple multi-scale features corresponding to the target image; performing weighted fusion of the aggregated features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregated features through the second-order fusion module of each stage in the feature fusion network to obtain the aggregated features output by the second-stage fusion subnetwork of this stage; the heat map generation network processes the aggregated features output by the feature fusion network to generate a joint point heat map corresponding to the target image.

[0007] In one embodiment, the orthogonal attention module includes multiple orthogonal attention units connected in sequence; the step of extracting features of the target image through each of the orthogonal attention modules in the feature extraction network to obtain multiple multi-scale features corresponding to the target image includes: for the first orthogonal attention unit in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage are used as input features of the first orthogonal attention unit, and the multi-scale features are output for the input features through the first orthogonal attention unit; for the remaining orthogonal attention units in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage and the multi-scale features output by each orthogonal attention unit located before the orthogonal attention unit are used as input features of the orthogonal attention unit, and the multi-scale features are output for the input features through the orthogonal attention unit.

[0008] In one embodiment, the step of outputting a multi-scale feature for the input feature through the orthogonal attention unit includes: performing a feature inversion operation on the input feature through the mask attention subunit in the orthogonal attention unit to obtain a mask feature map; performing weighted processing on the input feature through the channel attention subunit in the orthogonal attention unit to obtain a channel feature map; and performing feature fusion on the mask feature map and the channel feature map to obtain the multi-scale feature output by the orthogonal attention unit.

[0009] In one embodiment, the step of performing weighted fusion on the aggregate features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregate features through the second-order fusion module of each stage in the feature fusion network to obtain the aggregate features output by the second-order fusion module of this stage includes: for the second-order fusion module of each stage in the feature fusion network, the aggregate features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregate features are spliced ​​to obtain a relationship weight graph; based on the relationship weight graph, the aggregate features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregate features are weightedly fused to obtain the aggregate features output by the second-order fusion module of this stage.

[0010] In one embodiment, the step of determining the joint points in the target image based on the joint point heat map includes: determining the probability that each pixel point in the joint point heat map is a joint point according to the joint point heat map; determining the target pixel point in the joint point heat map according to the probability; and mapping the target pixel point to the target image to obtain the joint point in the target image.

[0011] In one embodiment, the step of obtaining the target image to be detected includes: obtaining the original image to be detected; inputting the original image into the target detection network to obtain a bounding box where the target object is located; and intercepting the area where the bounding box is located to obtain the target image to be detected.

[0012] In one embodiment, the training step of the joint point detection model includes: inputting the training sample into the initial detection model to obtain the joint point heat map corresponding to the training sample; calculating the loss value based on the preset loss function and the joint point heat map corresponding to the training sample, and updating the parameters of the initial detection model according to the loss value to obtain the joint point detection model.

[0013] In a second aspect, an embodiment of the present invention further provides a posture recognition method, comprising: performing joint point detection on a target image to be detected using any of the methods provided in the first aspect to obtain multiple joint points of a target object contained in the target image; and determining the posture of the target object based on the positions of the multiple joint points of the target object.

[0014] In the third aspect, an embodiment of the present invention provides a joint point detection device, comprising: a target image acquisition module, used to acquire a target image to be detected; a heat map acquisition module, used to perform joint point detection on the target image through a joint point detection model, and obtain a joint point heat map corresponding to the target image; wherein the joint point detection model includes a feature extraction network and a feature fusion network; the feature extraction network is used to extract multi-scale features; the feature fusion network includes a plurality of stages of second-order fusion modules, and the second-order fusion module of each stage is used to perform weighted fusion on the multi-scale features and the aggregated features output by the second-order fusion module of the previous stage; a joint point determination module, used to determine the joint points in the target image based on the joint point heat map.

[0015] In a fourth aspect, an embodiment of the present invention provides a posture recognition device, comprising: a joint detection module, used to perform joint point detection on a target image to be detected using any of the methods provided in the first aspect, to obtain multiple joint points of a target object contained in the target image; and a posture recognition module, used to determine the posture of the target object based on the positions of the multiple joint points of the target object.

[0016] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory; a computer program is stored on the memory, and when the computer program is run by the processor, it executes any one of the methods provided in the first aspect, or executes the method provided in the second aspect.

[0017] In a sixth aspect, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used for any one of the methods provided in the first aspect, or for executing and storing computer software instructions used for the method provided in the second aspect.

[0018] The embodiment of the present invention provides a joint point detection method and device, which first obtains a target image to be detected, then performs joint point detection on the target image through a joint point detection model, obtains a joint point heat map corresponding to the target image, and determines the joint points in the target image based on the joint point heat map, wherein the joint point detection model includes a feature extraction network and a feature fusion network, the feature extraction network is used to extract multi-scale features, and the feature fusion network includes a second-order fusion module of multiple stages, and the second-order fusion module of each stage is used to weightedly fuse the multi-scale features and the aggregated features output by the second-order fusion module of the previous stage. The above method proposes a new joint point detection model, which can efficiently detect and obtain joint point detection results with higher accuracy using the new joint point detection model. Specifically, by utilizing a feature extraction network to extract multi-scale features of different spatial sizes, the joint point detection model can extract more multi-scale information, and by utilizing a feature fusion network to perform weighted fusion of multi-scale features and corresponding aggregate features, the multi-scale information can be more fully utilized. Since the key point detection model has a more concise structure compared to the cascade pyramid framework, the embodiment of the present invention can more efficiently detect and obtain joint point detection results with higher accuracy compared to the neural network model of the cascade pyramid framework. Compared to the lightweight neural network model of the single pyramid framework, the embodiment of the present invention can effectively improve the joint point detection accuracy by extracting more multi-scale information, that is, the embodiment of the present invention comprehensively improves the efficiency and accuracy of joint point detection.

[0019] The embodiment of the present invention provides a posture recognition method and device, which uses any one of the methods provided by the above-mentioned joint point detection method to perform joint point detection on a target image to be detected, obtains multiple joint points of a target object contained in the target image, and recognizes the posture of the target object based on the multiple joint points of the target object. The above-mentioned method uses a joint point detection method with high detection efficiency and high detection accuracy to detect all joint points contained in the target image, thereby recognizing the posture of the target object based on the detected joint points, and further can more efficiently recognize and obtain a more accurate human body posture of the target object.

[0020] Other features and advantages of the embodiments of the present invention will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned techniques of the embodiments of the present invention.

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown;

[0024] Figure 2 A schematic diagram of a flow chart of a joint point detection method provided by an embodiment of the present invention is shown;

[0025] Figure 3 A schematic diagram of the structure of a joint point detection model provided by an embodiment of the present invention is shown;

[0026] Figure 4 A schematic diagram of the structure of an orthogonal attention module provided by an embodiment of the present invention is shown;

[0027] Figure 5 A schematic diagram of the structure of an orthogonal attention unit provided by an embodiment of the present invention is shown;

[0028] Figure 6 A schematic diagram of the structure of a second-order fusion module provided by an embodiment of the present invention is shown;

[0029] Figure 7 A schematic diagram of a flow chart of a gesture recognition method provided by an embodiment of the present invention is shown;

[0030] Figure 8 A schematic structural diagram of a joint point detection device provided by an embodiment of the present invention is shown;

[0031] Fig. 9 A schematic structural diagram of a gesture recognition device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be described below in conjunction with the drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0033] At present, the efficiency of joint point detection by the neural network model of the cascade pyramid framework is low, while the accuracy of joint point detection by the neural network model of the single pyramid framework is low. To improve this problem, the embodiments of the present invention provide a joint point detection method, a posture recognition method and a device. The technology can be applied to scenes requiring joint point detection or scenes requiring posture recognition. The embodiments of the present invention are described in detail below.

[0034] Embodiment 1:

[0035] First, refer to Figure 1 An example electronic device 100 for implementing a joint point detection method, a posture recognition method and an apparatus according to an embodiment of the present invention is described.

[0036] like Figure 1 The electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108, and an image acquisition device 110. These components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are only exemplary and not restrictive. The electronic device may have Figure 1 Some of the components shown may also have Figure 1 Other components and structures are not shown.

[0037] The processor 102 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 102 can be a central processing unit (CPU), a graphics processing unit (GPU), or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0038] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0039] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0040] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0041] The image acquisition device 110 can capture images desired by the user (eg, photos, videos, etc.), and store the captured images in the storage device 104 for use by other components.

[0042] Exemplarily, an example electronic device for implementing the joint point detection method, gesture recognition method and apparatus according to the embodiments of the present invention may be implemented as an intelligent terminal such as a smart phone, a tablet computer, or a computer.

[0043] Embodiment 2:

[0044] See also Figure 2 The flowchart of a joint point detection method shown in FIG. 1 mainly includes the following steps S202 to S206:

[0045] Step S202, obtaining a target image to be detected.

[0046] Among them, the target image can be an image containing a target object, and the target image can be obtained based on the original image to be detected, the original image contains at least one target image, and the target image can be obtained by intercepting the target image contained in the original image. In some embodiments, the original image can be obtained by photographing a device with a shooting function (for example, a smart phone or a camera), and a manual upload channel can also be provided for the user to obtain the original image uploaded by the user. In order to better detect the joint points of each target object in the original image, the embodiment of the present invention can intercept the area where each target object is located from the original image, so as to obtain a target image containing only one target object, and then detect all the relevant nodes of the target object in the target image.

[0047] Step S204, performing joint point detection on the target image through a joint point detection model to obtain a joint point heat map corresponding to the target image.

[0048] Among them, the joint point detection model includes a feature extraction network and a feature fusion network, the feature extraction network is used to extract multi-scale features in the target image from top to bottom, and the feature fusion network is used to fuse the multi-scale features extracted by the feature extraction network from bottom to top, and then generate a joint point heat map based on the fused features. In one embodiment, the feature extraction network includes a dense connection block and an orthogonal attention module (OAB, Orthogonal Attention Block) between the dense connection blocks, and the orthogonal attention module is used to extract multi-scale features. For example, OAB module 1 is used to extract multi-scale features of feature maps with a spatial size of l1*w1, and OAB module 2 is used to extract multi-scale features of feature maps with a spatial size of l2*w2; in addition, the feature fusion network also includes a plurality of stages of second-order fusion modules, and the second-order fusion modules of each stage are used to weightedly fuse the multi-scale features and the aggregate features output by the second-order fusion module of the previous stage, for example, the second-order fusion module of the second stage is used to weightedly fuse the multi-scale features and the aggregate features output by the second-order fusion module of the first stage, thereby obtaining the aggregate features output by the second-order fusion module of the second stage.

[0049] Step S206, determining the joint points in the target image based on the joint point heat map.

[0050] Among them, the joint point heat map corresponds to all the joint points contained in the target image one by one. In one embodiment, the probability values ​​represented by each pixel in the joint point heat map can be statistically analyzed, and the pixel with the largest probability value can be determined, and the coordinate value corresponding to the pixel can be obtained. By mapping the coordinate value to the target image, the joint point corresponding to the joint point heat map can be obtained. In the specific implementation, the above-mentioned processing is performed on the joint point heat map corresponding to each joint point contained in the target image, and all the joint points contained in the target image can be determined.

[0051] The above-mentioned joint point detection method provided by the embodiment of the present invention proposes a new joint point detection model, and the new joint point detection model can efficiently detect and obtain joint point detection results with higher accuracy. Specifically, by using each orthogonal attention module to extract multi-scale features of different spatial sizes respectively, the joint point detection model can extract more multi-scale information, and the multi-scale features and corresponding aggregate features are weighted and fused using a feature fusion network to make more full use of multi-scale information. Since the key point detection model is more streamlined than the cascade pyramid framework, the embodiment of the present invention can more efficiently detect and obtain joint point detection results with higher accuracy compared to the neural network model of the cascade pyramid framework. Compared with the lightweight neural network model of the single pyramid framework, the embodiment of the present invention can effectively improve the joint point detection accuracy by extracting more multi-scale information, that is, the embodiment of the present invention comprehensively improves the efficiency and accuracy of joint point detection.

[0052] In practical applications, considering that the original image may contain many target objects, it is not conducive to detecting the joint points of each target object in the original image. Therefore, the embodiment of the present invention cuts out the area where each target object is located from the original image to obtain the target image, and then detects the joint points contained in each target image respectively, so as to improve the accuracy of joint point detection to a certain extent. The embodiment of the present invention provides a specific implementation method for obtaining the target image to be detected, which can be specifically referred to the following steps (1) to (3):

[0053] (1) Obtaining the original image to be detected. The original image may include an image captured by an electronic device with a shooting function such as a smart phone or a camera, or an image uploaded by a user or downloaded from the Internet by the electronic device. In one embodiment, an image upload channel may be provided for the user, and the user may select and upload an image for joint point detection from a large number of images as the original image; in another embodiment, the image currently captured by the electronic device with a camera function may be used as the original image to be detected.

[0054] (2) Inputting the original image into the target detection network to obtain the original image marked with a person bounding box. The target detection network is used to detect the target object contained in the original image, and can use a network such as Megdet network, R-CNN (Region-based Convolutional Network) network, YOLO (You Only Look Once) network or SSD (Single Shot Multi-Box Detector) network, etc. The person bounding box is used to mark the area where the target object is located in the original image. In practical applications, a rectangular box can be used as the person bounding box. In one embodiment, in order to obtain a person bounding box with higher detection quality, the embodiment of the present invention can use the Megdet network as the target detection network to obtain the original image marked with a person bounding box.

[0055] (3) The area where the character's bounding box is located is captured to obtain the target image to be detected. In order to further improve the accuracy of joint point detection, the embodiment of the present invention can also scale the captured target image to a specified size. Assuming the specified size is 384×288, the area where the captured character's bounding box is located is scaled to 384×288, and the scaled image is used as the target image. In the subsequent detection process, joint point detection is performed on the target image of the specified size.

[0056] For ease of understanding, the present invention further provides a joint point detection model, see Figure 3 An exemplary structural diagram of a joint point detection model is provided, wherein the joint point detection model also includes a heat map generation network, the heat map generation network is used to generate a joint point heat map, and the feature fusion network includes multiple second-order fusion modules (SFU, Second-order Fusion Unit) for weighted fusion of multi-scale features and aggregated features output by the second-order fusion module in the previous stage. Figure 3 The area covered by the left slash in the middle represents the feature map, and the area covered by the right slash represents the heat map of the joint point. The input of the feature extraction network is the target image, and the output is multi-scale features of different spatial sizes; the input of the feature fusion network is multi-scale features of different spatial sizes, and the output is aggregated features; the input of the heat map generation network is aggregated features, and the output is the heat map of the joint point.

[0057] exist Figure 3 Based on the above, an embodiment of the present invention provides a specific implementation method of detecting joint points of a target image by using a joint point detection model to obtain a joint point heat map of all joint points contained in the target image, see steps 1 to 3 below:

[0058] Step 1: extract features of the target image through each orthogonal attention module in the feature extraction network to obtain multiple multi-scale features corresponding to the target image. The orthogonal attention module includes multiple orthogonal attention units connected in sequence.

[0059] To facilitate understanding of step 1, the embodiment of the present invention further explains the feature extraction network. The feature extraction network provided by the embodiment of the present invention also includes multiple first convolutional layers (conv) and pooling layers (Pooling), wherein: Figure 3 The arrows marked with "conv+Pooling" in the figure represent the first convolutional layer and the pooling layer. Figure 3 It can be seen that the number of the first convolutional layers is one less than the number of orthogonal attention modules, and the first convolutional layer is connected one-to-one with the remaining orthogonal attention modules except the orthogonal attention module at the end of the feature fusion network. It can also be understood that the orthogonal attention modules are connected through the first convolutional layer and the pooling layer. Among them, the first convolutional layer can be, for example, a 1*1 convolution kernel, and the pooling layer can be a 2*2 pooling layer with a step size of 2. For example, if the orthogonal attention modules provided in the embodiment of the present invention are 4, then the number of the first convolutional layer and the pooling layer are both 3.

[0060] Based on the above feature extraction network, the embodiment of the present invention provides a specific implementation method of extracting features of a target image through each orthogonal attention module in the feature extraction network to obtain multiple multi-scale features, wherein the spatial sizes of the multi-scale features output by different orthogonal attention modules are different, as shown in the following (1) and (2):

[0061] (1) For the orthogonal attention module of the first stage in the feature extraction network, the orthogonal attention module is used to extract features of the target image to obtain multiple multi-scale features of the same spatial size, and the first convolution layer and pooling layer connected to the orthogonal attention module of this stage are used to downsample each multi-scale feature of the spatial size to obtain a multi-scale feature output by the orthogonal attention module. The same spatial size can be, for example, 1 / 4 of the original size of the target image, which is not limited in the present embodiment of the application. Figure 4 As shown, the multi-scale features are input into the orthogonal attention module of the first stage in the feature extraction network. Assuming the original size is S, the orthogonal attention module of the first stage will extract features of the target image with a spatial size of 1 / 4S, and obtain multiple multi-scale features with a spatial size of 1 / 4S. Each multi-scale feature is input into the first convolution layer and the pooling layer connected to the orthogonal attention module. The first convolution layer and the pooling layer are used to downsample the input multiple multi-scale features to obtain a multi-scale feature with a spatial size of 1 / 8S.

[0062] (2) For the orthogonal attention modules of the remaining stages, the orthogonal attention modules of the current stage perform feature extraction on the multi-scale features output by the orthogonal attention modules of the previous stage to obtain multiple multi-scale features of the same spatial size, and downsample the multi-scale features of the spatial size through the first convolutional layer and pooling layer connected to the orthogonal attention module of the current stage to obtain a multi-scale feature output by the orthogonal attention module. The same spatial size can be 1 / 8, 1 / 16 or 1 / 32 of the original size of the target image. Figure 3 For example, the orthogonal attention module of the second stage in the feature extraction network is used to extract the multi-scale features with a spatial size of 1 / 8S, and multiple multi-scale features with a spatial size of 1 / 8S are obtained. Each multi-scale feature is input into the first convolution layer and pooling layer connected to the orthogonal attention module of this stage, and the first convolution layer and the pooling layer are used to downsample the input multi-scale features to obtain a multi-scale feature with a spatial size of 1 / 16S. Similarly, the orthogonal attention module of the third stage in the feature graph network is used to extract the multi-scale features with a spatial size of 1 / 16S, and multiple multi-scale features with a spatial size of 1 / 16 are obtained, thereby obtaining a multi-scale feature with a spatial size of 1 / 32. Then, the orthogonal attention module of the fourth stage in the feature extraction network is used to extract the multi-scale features with a spatial size of 1 / 32, and multiple multi-scale features with a spatial size of 1 / 32 are obtained.

[0063] In order to facilitate understanding of the multiple multi-scale features of the same spatial size obtained by extracting features through the orthogonal attention module, the embodiment of the present invention further provides an orthogonal attention module. Figure 4 The schematic diagram of the structure of an orthogonal attention module shown in FIG. 1 , the orthogonal attention module includes a plurality of orthogonal attention units connected in sequence, an exemplary Figure 4 The orthogonal attention module shown includes three orthogonal attention units connected in sequence, each of which includes a mask attention unit (MAU) subunit and a channel attention unit (CAU) subunit. Figure 4 Based on this, an embodiment of the present invention provides an implementation method in which each orthogonal attention module extracts features of a target image to obtain multiple multi-scale features corresponding to the target image. Specifically, the following steps 1.1 to 1.2 can be used:

[0064] Step 1.1, for the first orthogonal attention unit in each orthogonal attention module in the feature extraction network, the multi-scale feature output by the orthogonal attention module in the previous stage is used as the input feature of the first orthogonal attention unit, and the multi-scale feature is output by the first orthogonal attention unit for the input feature. For example, the input of the first OAB unit in the orthogonal attention module is the multi-scale feature A output by the orthogonal attention module in the previous stage, and the multi-scale feature B is output for the multi-scale feature A.

[0065] Step 1.2, for the remaining orthogonal attention units in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage and the multi-scale features output by each orthogonal attention unit located before the orthogonal attention unit are used as the input features of the orthogonal attention unit, and the orthogonal attention unit outputs the multi-scale features for the input features. For example, the input of the second OAB unit in the orthogonal attention module is the input feature obtained by fusing the multi-scale feature A and the multi-scale feature B, and the multi-scale feature C is output for the input feature; the input of the third OAB unit in the orthogonal attention module is the input feature obtained by fusing the multi-scale feature A, the multi-scale feature B and the multi-scale feature C, and the multi-scale feature D is output for the input feature, and so on, until the last OAB unit outputs the multi-scale feature.

[0066] The embodiment of the present invention provides an OAB unit, see Figure 5 A structural schematic diagram of an orthogonal attention unit is shown, wherein the orthogonal attention unit includes a mask attention subunit and a channel attention subunit, the MAU subunit includes a 1*1 convolution kernel, a DW9*9 convolution kernel (that is, a depth-separable convolution kernel) and a Sigmoid activation function, the CAU subunit includes a GPA (global-average-pooling, global pooling layer), two 1*1 convolution kernels and a Sigmoid activation function, and the OAB unit also includes a 1*1 convolution kernel and a 3*3 convolution kernel. Based on this, an embodiment of the present invention provides an implementation method for outputting multi-scale features for the input features through the orthogonal attention unit, see (i) to (iii) below:

[0067] (i) The masked attention subunit in the orthogonal attention unit performs a feature inversion operation on the input features to obtain a masked feature map. The MAU subunit masks the features represented by the input features contained in the output masked feature map, thereby forcing the information of different layers to be as rich and diverse as possible. Figure 5As shown in FIG. 1 , the input features are subjected to 1*1 convolution kernel, DW9*9 convolution kernel and Sigmoid activation function to obtain an attention map M (also referred to as mask M), wherein the features represented by the attention map M are consistent with the features represented by the input features, and then the attention map is inverted using F(x)=1-x to obtain the mask feature map (1-M), at which point the features represented by the mask feature map (1-M) are completely opposite to the features represented by the input features. It should be noted that the MAU subunit includes not only the above-mentioned convolution kernel and activation function, but also inversion and multiplication operations.

[0068] (ii) The input features are weighted by the channel attention sub-unit in the orthogonal attention unit to obtain a channel feature map. Figure 5 As shown in the figure, the input features are processed by GPA, two 1*1 convolution kernels and Sigmoid activation function to establish the dependency relationship between the feature channels in the spliced ​​graph, and the channel weight α is obtained. The channel weight α is bitwise multiplied with the input features to obtain the channel feature map.

[0069] (iii) Fusing the mask feature map and the channel feature map to obtain the multi-scale features output by the orthogonal attention unit. The CAU subunit is used to explore the interdependence between different feature channels to improve the nonlinearity of a single network layer, so that the output channel feature map represents features different from the input features. Before fusing the mask feature map with the channel feature map, the channel feature map is first passed through a 1*1 convolution kernel and a 3*3 convolution kernel, and then the features output by the 3.3 convolution kernel are bitwise multiplied with the mask feature map to obtain the multi-scale features output by the orthogonal attention unit. In order to make the output multi-scale features more different from the input features, the embodiment of the present invention outputs a mask feature map that is completely opposite to the features represented by the input features through the MAU subunit, and the mask feature map is bitwise multiplied with the channel feature map to obtain a richer and more diverse multi-scale feature.

[0070] The embodiment of the present invention continuously generates new multi-scale features through the above-mentioned OAB module, and splices the generated multi-scale features with the existing multi-scale features to form a tightly connected structure, thereby achieving the purpose of extracting more multi-scale features. Under the joint action of the MAU subunit and the CAU subunit, the OAB module can also have rich multi-scale features in a single spatial size.

[0071] Step 2: The aggregate features output by the second-order fusion module of each stage in the feature fusion network are weightedly fused with the multi-scale features of the same spatial size as the aggregate features, to obtain the aggregate features output by the second-order fusion module of this stage. For the second-order fusion module of the first stage in the feature fusion network, the multi-scale features output by the orthogonal attention module of the last stage in the feature extraction network can be deconvolved, and then the aggregate features obtained by the deconvolution operation and the multi-scale features of the same spatial size as the aggregate features are weightedly fused through the second-order fusion module of the first stage to obtain the aggregate features output by the second-order fusion module of this stage; for the second-order fusion modules of the remaining stages in the feature fusion network, the aggregate features output by the second-order fusion module of the previous stage can be deconvolved, and then the aggregate features obtained by the deconvolution operation and the multi-scale features of the same spatial size as the aggregate features are weightedly fused through the second-order fusion module of this stage to obtain the aggregate features output by the second-order fusion module of this stage. In addition, the feature fusion network also includes Transposed Conv, which is used to perform deconvolution operations on the aggregated features or the multi-scale features output by the orthogonal attention module in the last stage of the feature extraction network.

[0072] Since the OAB module outputs multiple multi-scale features, when performing weighted fusion, it is necessary to compress the multi-scale features of the same spatial size, and then perform weighted fusion on the compressed feature map and the aggregated features. In practical applications, in order to compress the multi-scale features of the same spatial size, the feature extraction network provided by the embodiment of the present invention further includes multiple second convolutional layers (conv) and normalization layers (bn) connected in sequence, wherein: Figure 3 The arrows marked with "conv+bn" in the figure represent the second convolutional layer and the normalization layer. The number of second convolutional layers is the same as the number of orthogonal attention modules. The second convolutional layer is connected to the orthogonal attention modules one-to-one. By inputting multiple multi-scale features of the same spatial size into the second convolutional layer and the normalization layer, multiple multi-scale features of the spatial size can be obtained for compression.

[0073] In addition, the embodiment of the present invention also exemplarily provides a second-order fusion module, see Figure 6 The schematic diagram of the structure of a second-order fusion module is shown in FIG. 1 , wherein the second-order fusion module includes a Concat function for concatenating the compressed feature map Fbu and the aggregated feature Ftd to obtain a relationship weight map, and also includes a 1*1 convolution kernel, a DW9*9 convolution kernel, and a Sigmoid activation function. Figure 6The second-order fusion module shown, the embodiment of the present invention provides an implementation method of obtaining the aggregated features output by the second-order fusion module of the previous stage by weighted fusion of the aggregated features output by the second-order fusion module of each stage in the feature fusion network and the multi-scale features with the same spatial size as the aggregated features, see the following steps 2.1 to 2.2:

[0074] Step 2.1: For each second-order fusion module in the feature fusion network, the aggregated features output by the second-order fusion module in the previous stage are concatenated with the multi-scale features with the same spatial size as the aggregated features to obtain a relationship weight map. In one embodiment, the multi-scale features with the same spatial size can be compressed by the second convolution layer and the normalization layer. Figure 3 As shown, the second convolution layer and normalization layer connected to the first OAB module compress multiple multi-scale features with a spatial size of 1 / 4S into a feature map with a spatial size of 1 / 4S; similarly, the second convolution layer and normalization layer connected to the second OAB module compress multiple multi-scale features with a spatial size of 1 / 8S into a feature map with a spatial size of 1 / 8S; the second convolution layer and normalization layer connected to the third OAB module compress multiple multi-scale features with a spatial size of 1 / 16S into a feature map with a spatial size of 1 / 16S; the second convolution layer and normalization layer connected to the fourth OAB module compress multiple multi-scale features with a spatial size of 1 / 32S into a feature map with a spatial size of 1 / 32S. Then, the compressed feature map and the aggregated features are spliced ​​to obtain a relationship weight map. In one embodiment, the compressed feature map and the aggregated features can be spliced ​​by the Concat function to obtain a relationship weight map.

[0075] Step 2.2, based on the relationship weight graph, the aggregated features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregated features are weightedly fused to obtain the aggregated features output by the second-order fusion module of this stage. In one embodiment, the feature fusion weight can be calculated based on the relationship weight graph, and the difference between the preset weight and the feature fusion weight is calculated. Specifically, the relationship weight graph is subjected to the above-mentioned 1*1 convolution kernel, DW9*9 convolution kernel and Sigmoid activation function to obtain the weight fusion weight λ. Assuming that the preset weight is 1, the difference 1-x between 1 and λ is calculated, that is, the weight fusion weight λ is negated using F(x)=1-x to obtain the difference 1-λ. After determining the feature fusion weight and the above-mentioned difference, the product of the compressed feature map and 1-λ can be calculated to obtain the first feature map, and the product of the aggregated feature and λ can be calculated to obtain the second feature map, and the first feature map and the second feature map are bitwise added to obtain the aggregated features output by the second-order fusion module of this stage.

[0076] To facilitate further understanding of the above weighted fusion process, the embodiment of the present invention combines Figure 3 The feature fusion process is explained. First, the second convolutional layer and normalization layer connected to the fourth OAB module are used to compress multiple multi-scale features with a spatial size of 1 / 32S to obtain a feature map with a spatial size of 1 / 32S. The first TransposedConv is used to perform a deconvolution operation on the feature map with a spatial size of 1 / 32S to obtain an aggregate feature with a spatial size of 1 / 16S. In addition, the second convolutional layer and normalization layer connected to the third OAB module are used to compress multiple multi-scale features with a spatial size of 1 / 16S. At this time, the feature map with a spatial size of 1 / 16S and the aggregate feature are used as the input of the first SFU sub-network. The first SFU sub-network will calculate the feature fusion weight, and perform weighted fusion on the input feature map and aggregate feature based on the feature fusion weight to obtain an aggregate feature with a spatial size of 1 / 16S. Similarly, the second convolutional layer and normalization layer connected to the second OAB module are used to compress multiple multi-scale features with a spatial size of 1 / 8S to obtain a feature map with a spatial size of 1 / 8S, and the second Transposed Conv performs a deconvolution operation on the aggregated features with a spatial size of 1 / 16S to obtain aggregated features with a spatial size of 1 / 8S. At this time, the feature map and aggregated features with a spatial size of 1 / 8S are used as the input of the second SFU sub-network, and the aggregated features with a spatial size of 1 / 8S are output, and so on.

[0077] The second-order fusion module provided in the embodiment of the present invention utilizes the completely opposite modes of λ and 1-λ, so that similar features between weighted feature maps are suppressed and different features are strengthened. Finally, the weighted feature maps are added to obtain the fused aggregated features, so that there is less redundant information and more effective information in the fused feature maps. In the embodiment of the present invention, features at different spatial sizes are continuously fused through SFU to form the final deep aggregated features, so that the aggregated features output by the terminal second-order fusion network contain sufficient multi-scale information and have powerful characterization capabilities.

[0078] Step 3, the heat map generation network processes the aggregated features output by the feature fusion network to generate a joint point heat map corresponding to the target image. In one embodiment, after the feature fusion network generates the aggregated features (that is, the aggregated features output by the second-order fusion module in the last stage), it will be sent to a heat map generation network to generate a joint point heat map, wherein the heat map generation network can be a regressor, and the convolutional layer can be selected as a regressor. The heat map contains K channels, where K represents the number of bone points (that is, the above-mentioned joint points) to be detected. Each channel can be a two-dimensional image with a size of 1 / 4 of the original image. That is, the regressor will output K joint point heat maps according to the feature fusion network, and each joint point heat map corresponds to a joint point one by one.

[0079] In one implementation, the embodiment of the present invention may perform the above-mentioned step of determining the joint points in the target image based on the joint point heat map according to the following steps a to c:

[0080] Step a: According to the joint point heat map, determine the probability that each pixel in the joint point heat map is a joint point. The joint point heat map is composed of pixels, each pixel may be / not a joint point, and determine the probability that each pixel is a joint point.

[0081] Step b, according to the probability, determine the target pixel in the joint point heat map. In one embodiment, the pixel with the highest probability can be used as the target pixel in the joint point heat map, and the coordinate value of the target pixel is obtained. Among them, the higher the probability value corresponding to each pixel in the joint point heat map, the higher the possibility that the pixel is a skeleton point, and vice versa, the lower the possibility that the pixel is a skeleton point. Therefore, the embodiment of the present invention uses the pixel with the highest probability value as the target pixel in the joint point heat map. The embodiment of the present invention provides an implementation method for obtaining the coordinate value of the target pixel, firstly, the joint point heat map is low-pass filtered to remove the noise in the joint point heat map, and then the pixel with the highest probability value in the entire joint point heat map is found, and it is determined as the target pixel. However, due to the limited precision of the pixel, the coordinate value obtained is only an integer. In order to improve the precision of the coordinate value, the embodiment of the present invention performs an offset processing on the integer coordinate value. Specifically, by determining the sub-extreme point corresponding to the coordinate value, and offsetting the integer coordinate value toward the sub-extreme point, the coordinate value of the target pixel is obtained.

[0082] Step c, mapping the target pixel points to the target image to obtain the joint points in the target image. In the specific implementation, the target pixel points determined from the heat map of each joint point are mapped to the target image, and all the joint points in the target image can be determined.

[0083] In addition, an embodiment of the present invention provides a method for training a joint point detection model, first obtaining a training sample, then inputting the training sample into an initial detection model, obtaining a heat map of the joint points corresponding to the training sample, and calculating a loss value based on a preset loss function and the heat map of the joint points corresponding to the training sample, and updating the parameters of the initial detection model according to the loss value. For example, the initial detection model at the completion of the training is determined as the joint point detection model, for example, when the number of iterative updates of the parameters of the initial detection model reaches a specific number, or the loss value is less than or equal to a specific value, or the training time reaches a specific time, the updated initial detection model at this time is determined as the joint point detection model. In one embodiment, the training sample can be a target image with various joint points annotated, and the loss value is calculated based on the preset loss function, the heat map of the joint points corresponding to the training sample, and the joint points annotated by the training sample, so as to train the initial detection model using the loss value.

[0084] In summary, the joint point detection method provided by the embodiment of the present invention improves the neural network of the single pyramid framework: on the one hand, the feature extraction network is improved by using the OAB module, and the multi-scale information between different layers under the same spatial size is extracted by using the densely connected OAB module, and the multi-scale information extracted by different layers is made more abundant and diverse through the attention mechanism, so that the features of a single spatial size can also contain more multi-scale information; on the other hand, the feature fusion network is improved by using the SFU sub-network, and the SFU sub-network can better suppress the redundant information in the features, so that the proportion of effective information in the final aggregated features is higher, wherein the effective information refers to the information that can bring gain to the final result, and the embodiment of the present invention can better improve the utilization rate of the features under a single spatial size, so that the fused features have more representation capabilities. Therefore, the embodiment of the present invention can extract more multi-scale information, and at the same time make the multi-scale information have sufficient representation capabilities, so that the joint point detection model of the single-layer pyramid framework reaches the accuracy comparable to the cascade pyramid model, and at the same time, the speed and efficiency of the joint point detection model in detecting joint points are better improved.

[0085] Embodiment three:

[0086] Based on the joint point detection method of embodiment 2, the embodiment of the present invention provides a posture recognition method, see Figure 7 The flowchart of a gesture recognition method shown in FIG. 1 mainly includes the following steps S702 to S704:

[0087] Step S702, using a joint point detection method to perform joint point detection on the target image to be detected, and obtain multiple joint points of the target object contained in the target image. In one embodiment, the target image can be input into the joint point detection model provided in Example 2, and the multi-scale features of multiple spatial dimensions are extracted by the feature extraction network in the joint point detection model, and the extracted multi-scale features are fused by the feature fusion network in the joint point detection model, so as to generate a joint point heat map using the fused feature map, and then determine each joint of the target object based on the joint point heat map, wherein the structure of the joint point detection model can refer to the aforementioned Example 2, and the embodiment of the present invention will not be repeated here.

[0088] Step S704, determining the posture of the target object according to the positions of the multiple joints of the target object. In one embodiment, the posture of the target object can be obtained by connecting the joints according to the human skeleton structure, such as connecting the shoulder, elbow and wrist to obtain the posture of the target object's arm.

[0089] The above-mentioned posture recognition method provided by the embodiment of the present invention utilizes a joint point detection method with faster detection efficiency and higher detection accuracy to detect all joint points contained in the target image, thereby identifying the posture of the target object based on the detected joint points, and further can more efficiently identify a more accurate human body posture of the target object.

[0090] Embodiment 4:

[0091] For the joint point detection method provided in the second embodiment, the present invention provides a joint point detection device, see Figure 8 The structure diagram of a joint point detection device shown in FIG. 1 includes the following modules:

[0092] The target image acquisition module 802 is used to acquire the target image to be detected.

[0093] The heat map acquisition module 804 is used to perform joint point detection on the target image through the joint point detection model to obtain the joint point heat map corresponding to the target image; wherein the joint point detection model includes a feature extraction network and a feature fusion network; the feature extraction network is used to extract multi-scale features; the feature fusion network includes multiple stages of second-order fusion modules, and the second-order fusion module of each stage is used to perform weighted fusion on the multi-scale features and the aggregated features output by the second-order fusion module of the previous stage.

[0094] The joint point determination module 806 is used to determine the joint points in the target image based on the joint point heat map.

[0095] The joint point detection device provided by the embodiment of the present invention proposes a new joint point detection model, which can efficiently detect and obtain joint point detection results with higher accuracy. Specifically, by using each orthogonal attention module to extract multi-scale features of different spatial sizes, the joint point detection model can extract more multi-scale information, and the multi-scale features and corresponding aggregate features are weighted and fused using a feature fusion network to make more full use of the multi-scale information. Since the key point detection model is more streamlined than the cascade pyramid framework, the embodiment of the present invention can more efficiently detect and obtain joint point detection results with higher accuracy compared to the neural network model of the cascade pyramid framework. Compared with the lightweight neural network model of the single pyramid framework, the embodiment of the present invention can effectively improve the joint point detection accuracy by extracting more multi-scale information, that is, the embodiment of the present invention comprehensively improves the efficiency and accuracy of joint point detection.

[0096] In one embodiment, the joint point detection model also includes a heat map generation network; the feature extraction network includes a densely connected block and an orthogonal attention module between the densely connected blocks; the above-mentioned heat map acquisition module 804 is also used to: extract features of the target image through each orthogonal attention module in the feature extraction network to obtain multiple multi-scale features corresponding to the target image; through the second-order fusion module of each stage of the feature fusion network, the aggregated features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregated features are weightedly fused to obtain the aggregated features output by the second-order fusion module of this stage; the heat map generation network processes the aggregated features output by the feature fusion network to generate a joint point heat map corresponding to the target image.

[0097] In one embodiment, the orthogonal attention module includes multiple orthogonal attention units connected in sequence; the above-mentioned heat map acquisition module 804 of the target image is also used for: for the first orthogonal attention unit in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage are used as the input features of the first orthogonal attention unit, and the multi-scale features are output for the input features through the first orthogonal attention unit; wherein the orthogonal attention module includes multiple orthogonal attention units connected in sequence; for the remaining orthogonal attention units in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage and the multi-scale features output by each orthogonal attention unit located before the orthogonal attention unit are used as the input features of the orthogonal attention unit, and the multi-scale features are output for the input features through the orthogonal attention unit.

[0098] In one embodiment, the above-mentioned heat map acquisition module 804 is also used to: perform feature inversion operation on the input features through the mask attention sub-unit in the orthogonal attention unit to obtain a mask feature map; perform weighted processing on the input features through the channel attention sub-unit in the orthogonal attention unit to obtain a channel feature map; perform feature fusion on the mask feature map and the channel feature map to obtain the multi-scale features output by the orthogonal attention unit.

[0099] In one embodiment, the heat map acquisition module 804 is also used for: for the second-order fusion module of each stage in the feature fusion network, the aggregated features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregated features are spliced ​​to obtain a relationship weight map; based on the relationship weight map, the aggregated features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregated features are weightedly fused to obtain the aggregated features output by the second-order fusion module of this stage.

[0100] In one embodiment, the joint point determination module 806 is also used to: determine the probability that each pixel point in the joint point heat map is a joint point based on the joint point heat map; determine the target pixel point in the joint point heat map based on the probability; map the target pixel point to the target image to obtain the joint point in the target image.

[0101] In one embodiment, the target image acquisition module 802 is further used to: acquire the original image to be detected; input the original image into the target detection network to obtain the bounding box where the target object is located; and capture the area where the bounding box is located to obtain the target image to be detected.

[0102] In one embodiment, the above-mentioned device also includes a training module, which is used to: input the training sample into the initial detection model to obtain the joint point heat map corresponding to the training sample; calculate the loss value based on the preset loss function and the joint point heat map corresponding to the training sample, and update the parameters of the initial detection model according to the loss value to obtain the joint point detection model.

[0103] For the gesture recognition method provided in the third embodiment, the present invention provides a gesture recognition device, see Fig. 9 The structure diagram of a gesture recognition device shown in FIG. 1 includes the following modules:

[0104] The joint detection module 902 is used to perform joint point detection on the target image to be detected by using any method provided in the second embodiment to obtain multiple joint points of the target object contained in the target image.

[0105] The posture recognition module 904 is used to determine the posture of the target object according to the positions of multiple joint points of the target object.

[0106] The above-mentioned posture recognition device provided by the embodiment of the present invention uses a joint point detection method with faster detection efficiency and higher detection accuracy to detect all joint points contained in the target image, thereby identifying the posture of the target object based on the detected joint points, and further can more efficiently identify and obtain more accurate human body posture of the target object.

[0107] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0108] Embodiment five:

[0109] The computer program product of the joint point detection method, posture recognition method and device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0110] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0111] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0112] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0113] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A joint point detection method, characterized in that: include: Acquire the target image to be detected; The target image is subjected to joint point detection by a joint point detection model to obtain a joint point heat map corresponding to the target image; wherein the joint point detection model includes a feature extraction network and a feature fusion network; the feature extraction network is used to extract multi-scale features; the feature fusion network includes a second-order fusion module of multiple stages, and the second-order fusion module of each stage is used to perform weighted fusion on the multi-scale features and the aggregated features output by the second-order fusion module of the previous stage; Determine the joint points in the target image based on the joint point heat map; The joint point detection model also includes a heat map generation network; the feature extraction network includes a dense connection block and an orthogonal attention module between the dense connection blocks; the step of performing joint point detection on the target image through the joint point detection model to obtain a joint point heat map corresponding to the target image includes: Extracting features of the target image through each of the orthogonal attention modules in the feature extraction network to obtain a plurality of multi-scale features corresponding to the target image; The second-order fusion module at each stage in the feature fusion network performs weighted fusion on the aggregated features output by the second-order fusion module at the previous stage and the multi-scale features with the same spatial size as the aggregated features, so as to obtain the aggregated features output by the second-order fusion module at this stage; The heat map generation network processes the aggregated features output by the feature fusion network to generate a heat map of the joint points corresponding to the target image; The orthogonal attention module includes a plurality of orthogonal attention units connected in sequence; the step of extracting features of the target image through each of the orthogonal attention modules in the feature extraction network to obtain a plurality of multi-scale features corresponding to the target image includes: For the first orthogonal attention unit in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage are used as the input features of the first orthogonal attention unit, and the multi-scale features are output for the input features through the first orthogonal attention unit; for the remaining orthogonal attention units in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage and the multi-scale features output by each orthogonal attention unit located before the orthogonal attention unit are used as the input features of the orthogonal attention unit, and the multi-scale features are output for the input features through the orthogonal attention unit.

2. The method according to claim 1, characterized in that The step of outputting a multi-scale feature for the input feature through the orthogonal attention unit comprises: Performing a feature inversion operation on the input feature through the masked attention subunit in the orthogonal attention unit to obtain a masked feature map; The input features are weighted by the channel attention sub-unit in the orthogonal attention unit to obtain a channel feature map; The mask feature map and the channel feature map are feature fused to obtain the multi-scale features output by the orthogonal attention unit.

3. The method according to claim 2, characterized in that The step of performing weighted fusion on the aggregated features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregated features through the second-order fusion module of each stage in the feature fusion network to obtain the aggregated features output by the second-order fusion module of the current stage includes: For the second-order fusion module at each stage in the feature fusion network, the aggregated features output by the second-order fusion module at the previous stage are concatenated with multi-scale features having the same spatial size as the aggregated features to obtain a relationship weight graph; Based on the relationship weight graph, the aggregated features output by the second-order fusion module of the previous stage and the multi-scale features with the same spatial size as the aggregated features are weightedly fused to obtain the aggregated features output by the second-order fusion module of this stage.

4. The method according to claim 1, characterized in that: The step of determining the joint points in the target image based on the joint point heat map comprises: According to the joint point heat map, determining the probability that each pixel point in the joint point heat map is a joint point; According to the probability, determine the target pixel point in the joint point heat map; The target pixel points are mapped to the target image to obtain joint points in the target image.

5. The method according to claim 1, characterized in that The step of acquiring the target image to be detected comprises: Obtain the original image to be detected; Input the original image into the target detection network to obtain the bounding box where the target object is located; The area where the bounding box is located is intercepted to obtain the target image to be detected.

6. The method according to claim 1, characterized in that The training steps of the joint point detection model include: Input the training sample into the initial detection model to obtain the joint point heat map corresponding to the training sample; The loss value is calculated based on a preset loss function and the joint point heat map corresponding to the training sample, and the parameters of the initial detection model are updated according to the loss value to obtain the joint point detection model.

7. A gesture recognition method, characterized in that: include: Perform joint point detection on the target image to be detected by using the method according to any one of claims 1 to 6 to obtain multiple joint points of the target object contained in the target image; The posture of the target object is determined according to the positions of the multiple joint points of the target object.

8. A joint point detection device, characterized in that: include: A target image acquisition module is used to acquire a target image to be detected; A heat map acquisition module, used to perform joint point detection on the target image through a joint point detection model to obtain a heat map of joint points corresponding to the target image; wherein the joint point detection model includes a feature extraction network and a feature fusion network; The feature extraction network is used to extract multi-scale features; the feature fusion network includes a plurality of stages of second-order fusion modules, and the second-order fusion modules in each stage are used to perform weighted fusion on the multi-scale features and the aggregated features output by the second-order fusion module in the previous stage; A joint point determination module, used to determine the joint points in the target image based on the joint point heat map; The joint point detection model also includes a heat map generation network; the feature extraction network includes a dense connection block and an orthogonal attention module between the dense connection blocks; the heat map acquisition module is specifically used for: Extracting features of the target image through each of the orthogonal attention modules in the feature extraction network to obtain a plurality of multi-scale features corresponding to the target image; The second-order fusion module at each stage in the feature fusion network performs weighted fusion on the aggregated features output by the second-order fusion module at the previous stage and the multi-scale features with the same spatial size as the aggregated features, so as to obtain the aggregated features output by the second-order fusion module at this stage; The heat map generation network processes the aggregated features output by the feature fusion network to generate a heat map of the joint points corresponding to the target image; The orthogonal attention module includes a plurality of orthogonal attention units connected in sequence; the heat map acquisition module is specifically used for: For the first orthogonal attention unit in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage are used as the input features of the first orthogonal attention unit, and the multi-scale features are output for the input features through the first orthogonal attention unit; for the remaining orthogonal attention units in each orthogonal attention module in the feature extraction network, the multi-scale features output by the orthogonal attention module in the previous stage and the multi-scale features output by each orthogonal attention unit located before the orthogonal attention unit are used as the input features of the orthogonal attention unit, and the multi-scale features are output for the input features through the orthogonal attention unit.

9. A gesture recognition device, characterized in that: include: A joint detection module, configured to perform joint point detection on a target image to be detected by using the method described in any one of claims 1 to 6, to obtain a plurality of joint points of a target object contained in the target image; The posture recognition module is used to determine the posture of the target object according to the positions of multiple joint points of the target object.

10. An electronic device, characterized in that: including a processor and a memory; The memory stores a computer program, and when the computer program is executed by the processor, the computer program executes the method according to any one of claims 1 to 6, or the method according to claim 7.

11. A computer storage medium, characterized in that: Used to store computer software instructions used for the method according to any one of claims 1 to 6, or to execute computer software instructions stored for the method according to claim 7.

Images