Human body key point processing method based on grouping convolution and electronic equipment

Abstract

The invention discloses a human body key point processing method based on grouped convolution and electronic equipment, the method is applied to a key point detection model, and the method comprises the following steps: carrying out shallow feature extraction operation on an input image based on a first convolutional neural network, and obtaining an initial feature of the input image; grouping feature information corresponding to each human body key point in the initial features to form a plurality of first feature groups, and performing convolution operation on each first feature group by using a second convolutional neural network to obtain deep features; based on the deep features, determining a first key point feature map corresponding to the human body key points; adjusting the first key point feature map into a second key point feature map by using a fusion network based on a dependency relationship among the human body key points; and on the basis of a deconvolution network, grouping feature information corresponding to each human body key point in the second key point feature map to form a plurality of second feature groups, and performing deconvolution operation on the second feature groups to generate a heat map.

Description

Technical Field

[0001] This application relates to the field of neural networks, and in particular to a method and electronic device for processing human key points based on grouped convolution. Background Technology

[0002] Human keypoints are crucial locations on the human body in an image, such as movable joints. Human keypoint detection is a computer vision task that aims to determine the coordinates of keypoints within an input image of the human body. This process has wide applications in video surveillance, motion analysis, and human-computer interaction. However, current keypoint detection models for determining the coordinates of human keypoints do not adequately utilize the independence of different keypoints. While shallow information needs to be extracted in the initial stage of feature extraction, deeper stages require different features to estimate different keypoints. Using ordinary convolutional layers for feature extraction applies all features to estimate each keypoint, ignoring the degree of independence between different keypoints. This leads to inaccuracies in determining the coordinates of keypoints using keypoint detection models. Summary of the Invention

[0003] The purpose of this application is to provide a method and electronic device for processing human key points based on grouped convolution. This method effectively improves the accuracy of detecting human key points in input images.

[0004] To achieve the above objectives, this application provides a method for processing human key points based on grouped convolution, applied in a key point detection model, including:

[0005] The input image is subjected to shallow feature extraction based on the first convolutional neural network to obtain the initial features of the input image, wherein the initial features include the feature information of the input image and shallow information associated with each of the human body key points in the input image;

[0006] The feature information corresponding to each of the human body key points in the initial features is grouped to form multiple first feature groups. A second convolutional neural network is used to perform a convolution operation on each of the first feature groups to obtain deep features containing the independent features of each of the human body key points.

[0007] Based on the deep features, a first key point feature map corresponding to the key points of the human body is determined;

[0008] Based on the dependencies between the various human body key points, the first key point feature map is adjusted into the second key point feature map using a fusion network.

[0009] Based on a deconvolutional network, the feature information corresponding to each human key point in the second key point feature map is grouped into multiple second feature groups. The second feature groups are then deconvolved to generate corresponding heatmaps, and the coordinates of the human key points in the input image are determined based on the heatmaps.

[0010] Optionally, the shallow feature extraction operation on the input image based on the first convolutional neural network includes:

[0011] Determine the morphological parameters and number of color channels of the input image;

[0012] Based on the morphological parameters and number of color channels of the input image, the first convolutional neural network is used to extract features from the input image.

[0013] Optionally, the step of grouping the feature information corresponding to each of the human body key points in the initial features to form multiple first feature groups, and using a second convolutional neural network to perform a convolution operation on each of the first feature groups to obtain deep features containing the independent features of each of the human body key points, includes:

[0014] The convolutional kernel with the fewest channels in the second convolutional neural network is used to perform a convolution operation on each of the first feature groups to obtain the feature information of each of the first feature groups.

[0015] Accordingly, determining the first key point feature map corresponding to the human body key points based on the deep features includes:

[0016] By concatenating the feature information of all the first feature groups, the first key point feature map is obtained.

[0017] Optionally, determining the first keypoint feature map corresponding to the human body keypoints based on the deep features includes:

[0018] Based on pre-constructed human body structural information and prior knowledge, the first constraint condition is determined;

[0019] Under the constraints of the first constraint, the first key point feature map is determined based on the deep features.

[0020] Optionally, the step of adjusting the first keypoint feature map into a second keypoint feature map using a fusion network based on the dependencies between the various human body keypoints includes:

[0021] Using the convolution kernel of the fusion network, the convolutional data of all channels corresponding to the human body key points in the first key point feature map are weighted and summed to obtain the second key point feature map.

[0022] Optionally, the step of adjusting the first keypoint feature map into a second keypoint feature map using a fusion network based on the dependencies between the various human body keypoints includes:

[0023] Based on pre-constructed human body structural information and prior knowledge, the first constraint condition is determined;

[0024] Based on the first constraint and the dependency relationship between the key points of the human body, the first key point feature map is adjusted to the second key point feature map.

[0025] Optionally, the determination of the first constraint based on pre-constructed human structural information and prior knowledge includes:

[0026] Based on the loss function, the similarity relationship between the human body structure in the input image and the human body structure in the flipped image of the input image is constrained to form the first constraint condition, wherein the loss function is:

[0027]

[0028] Where z i The feature vector of the human body key points in the input image; The feature vector of the symmetrical key points in the flipped image of the input image; τ is the feature vector of symmetrical key points in the flipped image of the reference image during the training process of the key point detection model; τ is the temperature coefficient.

[0029] Optionally, the step of grouping the feature information corresponding to each human key point in the second key point feature map into multiple second feature groups based on the deconvolution network, and performing a deconvolution operation on the second feature groups, includes:

[0030] During the deconvolution operation on each of the second feature groups, the deep features are used as part of the input to the deconvolution network.

[0031] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0032] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0033] The human keypoint processing method based on grouped convolution in this application obtains relatively independent deep features for each human keypoint by grouping the extracted initial features and then performing convolution operations. This allows the second keypoint feature map obtained based on the deep features to reflect both the overall feature information of the input image and the independent feature information of each human keypoint. While achieving regularization of the entire keypoint detection model, it also more accurately determines the positional information of human keypoints in the input image. Attached Figure Description

[0034] Figure 1 This is a schematic diagram illustrating human key point detection on an input image according to an embodiment of this application;

[0035] Figure 2 This is a flowchart of a method for processing human key points based on grouped convolution, according to an embodiment of this application.

[0036] Figure 3 This is a flowchart illustrating a specific embodiment of the processing method described in this application.

[0037] Figure 4 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S100;

[0038] Figure 5 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S300;

[0039] Figure 6 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S400;

[0040] Figure 7 This is a schematic diagram illustrating the constraint of the similarity relationship between human structures in the input image and its flipped image according to an embodiment of this application;

[0041] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0042] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0043] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0044] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0045] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0046] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0047] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0048] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0049] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0050] This application provides a method for processing human key points based on grouped convolution, which is applied to a key point detection model, such as... Figure 1 As shown, this keypoint detection model can detect the location of key points on the human body in an image, and has wide applications in video surveillance, motion analysis, and human-computer interaction. Human keypoints are crucial locations on the human body in an image, such as the head, hands, elbows, feet, and joints. This processing method can determine the corresponding heatmap based on the RGB image of the human body, and then determine the two-dimensional coordinates of the human keypoints in the image based on the heatmap.

[0051] The method includes: performing shallow feature extraction on the input image based on a first convolutional neural network to obtain the initial features of the input image. The input image is a target image containing a human figure. The first convolutional neural network can perform shallow feature extraction on the input image using ordinary convolution. Shallow feature extraction extracts relatively basic and intuitive shallow features from the input image. For example, the extracted image features are closer to the original image, containing more pixel information, such as color, texture, edges, and sharp corners. Compared to deep features, these features focus more on fine-grained information of the image.

[0052] After obtaining the initial features through shallow feature extraction, the feature information corresponding to each human key point in the initial features is grouped to form multiple first feature groups. The specific grouping method can be based on the number of human key points; for example, the feature information corresponding to one human key point in the initial features can be grouped into a group to form the corresponding first feature group.

[0053] Then, a second convolutional neural network is used to perform deep convolution operations on each of the first feature groups to obtain deep features containing the independent features of each human body keypoint. These deep features have undergone processing by multiple layers of the second convolutional neural network, resulting in less noise, a larger receptive field, and stronger semantic information. Because the first feature groups are obtained through group convolution, the deep features obtained by performing deep convolution operations on each of the first feature groups using the second convolutional neural network possess the independent features of their respective human body keypoints.

[0054] Then, based on deep features, a first keypoint feature map corresponding to the human body keypoints is determined. This first keypoint feature map contains the features corresponding to the human body keypoints. Next, based on the dependencies between various human body keypoints, such as the coordination relationships between various human body keypoints during movement, a fusion network is used to weight and sum the channel data in the first keypoint feature map to obtain a second keypoint feature map. This second keypoint feature map more accurately reflects the features of each human body keypoint.

[0055] Based on a deconvolutional network with a grouped structure, the feature information corresponding to each human keypoint in the second keypoint feature map is grouped into multiple second feature groups. Each second feature group is then deconvolved to obtain the feature information of each corresponding human keypoint. A heatmap is then generated based on the feature information of all human keypoints. This heatmap can be a graphic containing numbers or characters representing probabilities, which can represent the position of the human keypoints. Based on this heatmap, the position of the human keypoints in the input image can be determined and represented by coordinates.

[0056] The method for processing human key points based on grouped convolution of this application will be described in detail below with reference to the accompanying drawings. Figure 2 This is a flowchart of a human keypoint processing method based on grouped convolution, as described in an embodiment of this application. Figure 2 As shown and combined Figure 3 The method includes the following steps:

[0057] S100, shallow feature extraction is performed on the input image based on the first convolutional neural network to obtain the initial features of the input image, wherein the initial features include the feature information of the input image and shallow information associated with each of the human body key points in the input image.

[0058] For example, the first convolutional neural network can be a neural network with a common structure (such as a CNN with a common structure) capable of performing feature extraction on the input image. This extraction operation can be a shallow feature extraction operation on the input image, which extracts relatively basic and intuitive features from the input image. For example, the extracted image features are relatively close to the original image, containing more pixel information, such as details like color, texture, edges, and corners. The initial features of the obtained input image include shallow features, as well as edge information of the input image associated with each human body keypoint.

[0059] The initial features can be used as input to the second convolutional neural network, which will then further extract features from them.

[0060] S200, the feature information corresponding to each of the human body key points in the initial features is grouped into multiple first feature groups, and a second convolutional neural network is used to perform a convolution operation on each of the first feature groups to obtain deep features containing the independent features of each of the human body key points.

[0061] For example, the initial features include feature information corresponding to each human body key point, that is, each human body key point has its own feature information. In this embodiment, the feature information corresponding to the human body key points can be grouped based on the number of human body key points. For example, if there are 17 human body key points, the feature information corresponding to each of the human body key points can be grouped to form 17 first feature groups.

[0062] By using a second convolutional neural network to perform convolution operations on each first feature group, the independent deep features of each human keypoint are fully acquired. Specifically, this second convolutional neural network can perform group convolutions on each first feature group to extract independent deep features. For example, processing through multiple layers in the second convolutional neural network can reduce noise in the obtained deep features, resulting in a larger receptive field and stronger semantic information. This group convolution operation using the second convolutional neural network can effectively reduce the number of parameters in the entire keypoint detection model.

[0063] S300, Based on the deep features, determine the first key point feature map corresponding to the key points of the human body.

[0064] For example, since each deep feature can represent the deep feature content of the corresponding human body key point, the second convolutional neural network can determine the corresponding first key point feature map and output it based on the deep features and the structural relationship of the human body key points.

[0065] S400, based on the dependency relationship between the various human body key points, the first key point feature map is adjusted into a second key point feature map using a fusion network.

[0066] For example, the dependency between key points in the human body can be a physical relationship of the human body structure. For instance, when the human body moves, the first key point corresponding to the upper arm drives the second key point corresponding to the hand, which shows that the second key point is dependent on the first key point.

[0067] The fusion network can be pre-built based on the dependencies between various human keypoints. When used, the fusion network can be used to adjust the feature information corresponding to each human keypoint in the first keypoint feature map. For example, if the feature information in the first keypoint feature map shows that the second human keypoint is too far away from the first human keypoint, the fusion network can adjust the corresponding feature information according to the known dependencies, so that the feature information can show that the second human keypoint is closer to the first human keypoint.

[0068] In one embodiment, after adjusting all similar feature information, the obtained first key point feature map is then segmented along the channels corresponding to the human body key points (each channel contains feature information) to obtain the second key point feature map.

[0069] S500: Based on a deconvolutional network, the feature information corresponding to each human key point in the second key point feature map is grouped into multiple second feature groups. The second feature groups are deconvolved to generate corresponding heatmaps, so as to determine the coordinates of the human key points in the input image based on the heatmaps.

[0070] For example, since the feature information of each human body key point has already established interdependencies through the fusion network, the process of estimating the heatmap based on the feature information corresponding to the human body key points can still be carried out by processing the feature information corresponding to each human body key point independently. This can be achieved by using a deconvolutional network with a grouped structure to independently estimate the feature information corresponding to each human body key point in the second key point feature map. Specifically, the feature information corresponding to each human body key point in the second key point feature map can be grouped into multiple second feature groups, with each human body key point corresponding to a second feature group. Then, each second feature group is deconvolved to generate the corresponding heatmap.

[0071] Since heatmaps can represent the locations of key points on the human body, their coordinates in the input image can be determined based on the heatmap. In one embodiment, a point can be selected in the input image as a reference point, and the coordinates of each key point in the input image can be determined based on this reference point.

[0072] The human keypoint processing method based on grouped convolution in this application obtains relatively independent deep features for each human keypoint by grouping the extracted initial features and then performing convolution operations. This allows the second keypoint feature map obtained based on the deep features to reflect both the overall feature information of the input image and the independent feature information of each human keypoint. While achieving regularization of the entire keypoint detection model, it also more accurately determines the positional information of human keypoints in the input image.

[0073] In one embodiment of this application, the shallow feature extraction operation on the input image based on the first convolutional neural network is as follows: Figure 4 As shown, it includes the following steps:

[0074] S110, determine the morphological parameters and number of color channels of the input image;

[0075] S120, based on the morphological parameters and number of color channels of the input image, feature extraction is performed on the input image using the first convolutional neural network.

[0076] For example, the shape parameters of the input image are parameters used to represent the shape of the input image, such as the length, width, and height of the input image.

[0077] Each image has one or more color channels. The default number of color channels in an image depends on its color mode; that is, the color mode of an image determines the number of its color channels. The primary color channels of an input image can be the three primary color channels, namely RGB channels, where the number of color channels is 3, and "R" represents "red", "G" represents "green", and "B" represents "blue". In RGB color mode, the input image is a composite of the R, G, and B channels.

[0078] In one specific embodiment, the input image of the keypoint detection model is an RGB image of the human body. Where H and W are the height and width of the input image, respectively, and 3 corresponds to the three RGB channels of the image. First, a first convolutional neural network E with several layers is used... l This method extracts low-level information from the input image, such as edge information. Since this low-level information is crucial for key points on the human body, the low-level feature extraction network uses a standard convolutional structure, and the output is...

[0079] In one embodiment of this application, the step of grouping the feature information corresponding to each of the human body key points in the initial features to form multiple first feature groups, and using a second convolutional neural network to perform a convolution operation on each of the first feature groups to obtain deep features containing the independent features of each of the human body key points includes the following steps:

[0080] The convolutional kernel with the fewest channels in the second convolutional neural network is used to perform a convolution operation on each of the first feature groups to obtain the feature information of each of the first feature groups.

[0081] Accordingly, determining the first key point feature map corresponding to the human body key points based on the deep features includes:

[0082] By concatenating the feature information of all the first feature groups, the first key point feature map is obtained.

[0083] For example, after extracting shallow information, since the estimation of different human keypoints requires attention to different types of high-level features, a second convolutional neural network (deep feature extraction network E) with a grouped convolutional structure is used. h This allows for independent deep feature extraction of the feature information corresponding to different key points on the human body.

[0084] The first convolutional neural network can be a regular convolutional network, where the convolutional kernel performs convolution on all channels of the input image. The parameter count is h1×w1×C1×C2, where h1 and w1 are the kernel sizes, and C1 and C2 are the number of input and output channels, respectively. In contrast, the second convolutional neural network uses grouped convolution to divide the initial features into several first feature groups. Each group is then independently convolved using a kernel with the fewest or fewest channels in the second convolutional neural network. The parameter count is h1×w1×(C1 / g)×(C2 / g)×g, which is only 1 / g of the original count, where g is the number of groups. Therefore, the deep feature extraction network based on grouped convolution (the second convolutional neural network) is more lightweight than that based on a regular convolutional network (such as the first convolutional neural network).

[0085] Furthermore, set d l If the initial features (which can be represented based on feature maps) are divisible by J (where J is the number of key points in the human body), then the first feature group can be divided into J groups along the channel dimension. Each first feature group is convolved independently, and then the feature information of the first feature group corresponding to each human body key point is concatenated along the feature channel dimension. The final output of the second convolutional neural network is as follows: The first key point feature map corresponding to the stitched human body key points is as follows

[0086] In one embodiment of this application, the step of determining a first keypoint feature map corresponding to the human body keypoints based on the deep features is as follows: Figure 5 As shown, it includes:

[0087] S310, based on pre-constructed human body structural information and prior knowledge, determines the first constraint condition.

[0088] For example, human body structural information refers to information related to the inherent appearance, shape, and movement of the human body. For instance, human body structural information might define the human body as including the head, body, two upper limbs, and two lower limbs. This human body structural information defines the structure of the human body.

[0089] Prior information refers to the mutual constraints between key points of the human body during static or dynamic processes. For example, during upper limb movement, the upper arm drives the forearm and hand; the distance between the foot and leg joints will not exceed a preset value.

[0090] Based on the aforementioned human body structural information and prior knowledge, a first constraint condition is determined, which limits the structural relationship between key points of the human body.

[0091] S320, Under the constraint of the first constraint condition, the first key point feature map is determined based on the deep features.

[0092] For example, in the process of determining the first keypoint feature map based on deep features, the second convolutional neural network is constrained by the first constraint condition, so that the second convolutional neural network can limit the first keypoint feature map based on the basic structure of the human body when outputting the first keypoint feature map. This results in a more accurate first keypoint feature map.

[0093] In one embodiment of this application, adjusting the first keypoint feature map into a second keypoint feature map using a fusion network based on the dependencies between the various human keypoints includes:

[0094] Using the convolution kernel of the fusion network, the convolutional data of all channels corresponding to the human body key points in the first key point feature map are weighted and summed to obtain the second key point feature map.

[0095] For example, to compensate for the problem of human structure dependencies caused by grouped convolution, this method, after extracting the feature information corresponding to each human key point using a second convolutional neural network, also uses a fusion network R to model the dependencies between different human key points, thus obtaining a second key point feature map. This feature fusion network adopts a common convolutional structure. The convolutional kernels weight and sum the channel data corresponding to all human key points, thus enabling the modeling of the relationship between different human key points and obtaining an accurate second key point feature map.

[0096] In one embodiment of this application, based on the dependencies between the various human body key points, the first key point feature map is adjusted to a second key point feature map using a fusion network, such as... Figure 6 As shown, it includes:

[0097] S410, based on pre-constructed human body structural information and prior knowledge, determine the first constraint condition;

[0098] S420, based on the first constraint and the dependency relationship between the key points of the human body, adjust the first key point feature map to the second key point feature map.

[0099] For example, similar to the embodiments described above, human body structural information refers to information related to the inherent appearance, shape, and movements of the human body. Prior information refers to the mutual constraints between various key points of the human body during static or dynamic processes.

[0100] Based on the aforementioned human body structural information and prior knowledge, a first constraint condition is determined, which limits the structural relationship between key points of the human body.

[0101] The process of adjusting the first keypoint feature map to the second keypoint feature map also needs to be constrained based on the first constraint condition and the dependency relationship between human body keypoints, thereby optimizing the first keypoint feature map. This, in turn, achieves regularization of the keypoint detection model for human body keypoints.

[0102] In one embodiment of this application, determining the first constraint based on pre-constructed human structural information and prior knowledge includes:

[0103] Based on the loss function, the similarity relationship between the human body structure in the input image and the human body structure in the flipped image of the input image is constrained to form the first constraint condition, wherein the loss function is:

[0104]

[0105] Where z i The feature vector of the human body key points in the input image; The feature vector of the symmetrical key points in the flipped image of the input image; τ is the feature vector of symmetrical key points in the flipped image of the reference image during the training process of the key point detection model; τ is the temperature coefficient.

[0106] For example, since the keypoint detection model of this application may not extract features of the entire image during processing, but may extract features only corresponding to each keypoint of the human body, it can apply additional constraints at the feature level based on prior knowledge of human structure to obtain the first feature constraint condition. The symmetry of human structure is a significant characteristic that distinguishes it from other data. This feature constraint content in the first feature constraint condition simultaneously affects the initial first keypoint feature map F. ori and the optimized second key point feature map F ref Apply constraints.

[0107] Combination Figure 7 Consider an input image (RGB image) whose horizontally flipped image is a mirror image of the original image. The human body keypoint features on the left side of the input image and on the right side of the flipped image have similar physical meanings. We can use the NT-Xent Loss function to add similarity constraints. The smaller the NT-Xent Loss, the better the constraint effect. The formula for calculating the NT-Xent Loss is:

[0108]

[0109] Where z i The feature vector of human key points in the input image; The feature vectors of symmetrical key points in the flipped image of the input image; τ is the feature vector of symmetrical key points in the flipped image of the reference image during the key point detection model training process; τ is the temperature coefficient.

[0110] In one embodiment of this application, the step of grouping the feature information corresponding to each of the human body key points in the second key point feature map into multiple second feature groups based on a deconvolution network, and performing a deconvolution operation on the second feature groups, includes:

[0111] During the deconvolution operation on each of the second feature groups, the deep features are used as part of the input to the deconvolution network.

[0112] For example, in the process of converting the second keypoint feature map into a heatmap, since the feature information of each human keypoint has already been modeled with interdependencies through a fusion network, the heatmap estimation stage based on the second keypoint feature map can still adopt the approach of processing the feature information corresponding to each human keypoint independently. That is, a deconvolutional network with a grouped structure can be used to independently estimate the heatmap of each human keypoint. Specifically, for the optimized second keypoint feature map F... ref The data is divided into J groups along the channel dimension, and each group is independently deconvolved to obtain the corresponding heatmap. Where h and w represent the height and width of the output heatmap, respectively. To avoid the vanishing gradient phenomenon and improve the overall performance of the keypoint detection model, the keypoint detection model can also adopt the skip connection approach, using the deep features output from the feature extraction stage as part of the input to the deconvolutional network.

[0113] This application also provides an electronic device, such as... Figure 8 As shown, it includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method described above.

[0114] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, electronic devices, computer-readable storage media, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Furthermore, this application can take the form of a computer program product implemented on one or more computer-readable storage media containing computer-readable program code. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0116] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0117] The aforementioned memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0118] The aforementioned readable storage medium may be a magnetic disk, optical disk, DVD, USB, read-only memory (ROM) or random access memory (RAM), etc. This application does not limit the specific form of storage medium.

[0119] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A method for processing human key points based on grouped convolution, characterized in that, Applied to keypoint detection models, including: The input image is subjected to shallow feature extraction based on the first convolutional neural network to obtain the initial features of the input image, wherein the initial features include the feature information of the input image and shallow information associated with each of the human body key points in the input image; The feature information corresponding to each of the human body key points in the initial features is grouped to form multiple first feature groups. A second convolutional neural network is used to perform a convolution operation on each of the first feature groups to obtain deep features containing the independent features of each of the human body key points. Based on the deep features, a first key point feature map corresponding to the key points of the human body is determined; Based on the dependencies between the various human body key points, the first key point feature map is adjusted into the second key point feature map using a fusion network. Based on a deconvolutional network, the feature information corresponding to each human key point in the second key point feature map is grouped into multiple second feature groups. The second feature groups are then deconvolved to generate corresponding heatmaps, and the coordinates of the human key points in the input image are determined based on the heatmaps.

2. The method for processing human key points based on grouped convolution according to claim 1, characterized in that, The shallow feature extraction operation based on the first convolutional neural network on the input image includes: Determine the morphological parameters and number of color channels of the input image; Based on the morphological parameters and number of color channels of the input image, the first convolutional neural network is used to extract features from the input image.

3. The method for processing human key points based on grouped convolution according to claim 1, characterized in that, The step of grouping the feature information corresponding to each of the human body key points in the initial features to form multiple first feature groups, and using a second convolutional neural network to perform a convolution operation on each of the first feature groups to obtain deep features containing the independent features of each of the human body key points includes: The convolutional kernel with the fewest channels in the second convolutional neural network is used to perform a convolution operation on each of the first feature groups to obtain the feature information of each of the first feature groups. Accordingly, determining the first key point feature map corresponding to the human body key points based on the deep features includes: By concatenating the feature information of all the first feature groups, the first key point feature map is obtained.

4. The method for processing human key points based on grouped convolution according to claim 1, characterized in that, The step of determining the first key point feature map corresponding to the human body key points based on the deep features includes: Based on pre-constructed human body structural information and prior knowledge, the first constraint condition is determined; Under the constraints of the first constraint, the first key point feature map is determined based on the deep features.

5. The method for processing human key points based on grouped convolution according to claim 1, characterized in that, The step of adjusting the first key point feature map into a second key point feature map using a fusion network based on the dependencies between the various human body key points includes: Using the convolution kernel of the fusion network, the convolutional data of all channels corresponding to the human body key points in the first key point feature map are weighted and summed to obtain the second key point feature map.

6. The method for processing human key points based on grouped convolution according to claim 1, characterized in that, The step of adjusting the first key point feature map into a second key point feature map using a fusion network based on the dependencies between the various human body key points includes: Based on pre-constructed human body structural information and prior knowledge, the first constraint condition is determined; Based on the first constraint and the dependency relationship between the key points of the human body, the first key point feature map is adjusted to the second key point feature map.

7. The method for processing human key points based on grouped convolution according to claim 4 or 6, characterized in that, The determination of the first constraint based on pre-constructed human structural information and prior knowledge includes: Based on the loss function, the similarity relationship between the human body structure in the input image and the human body structure in the flipped image of the input image is constrained to form the first constraint condition, wherein the loss function is: Where z i The feature vector of the human body key points in the input image; The feature vector of the symmetrical key points in the flipped image of the input image; τ is the feature vector of symmetrical key points in the flipped image of the reference image during the training process of the key point detection model; τ is the temperature coefficient.

8. The method for processing human key points based on grouped convolution according to claim 1, characterized in that, The method based on a deconvolutional network involves grouping the feature information corresponding to each human key point in the second key point feature map into multiple second feature groups, and then performing a deconvolution operation on the second feature groups, including: During the deconvolution operation on each of the second feature groups, the deep features are used as part of the input to the deconvolution network.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method as claimed in claim 1.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in claim 1.

Images