Joint processing method, equipment and product for key point detection and part analysis of human body

Abstract

The embodiment of the present invention provides a joint processing method, device and product for human body key point detection and part analysis, the method comprising: acquiring a target image; using a target network model trained to convergence to perform joint processing of human body key point detection and part analysis on the target image to obtain human body key point detection results and human body part analysis results, the target network model is obtained after performing an optimal network structure search on the initial network model using a neural network automatic search technology during the process of training the initial network model using training samples; outputting the human body key point detection results and human body part analysis results through the target network model can effectively improve the accuracy of human body key point detection and part analysis, and does not rely on the design experience and parameter adjustment experience of the network model, effectively improving the efficiency of human body key point detection and part analysis.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a joint processing method, device and product for key point detection and part analysis of a human body. Background Art

[0002] With the development of artificial intelligence, various deep learning models have been gradually applied to the field of human visual understanding. Human key point detection and human body part analysis are two important tasks in human visual understanding. They play a vital role in analyzing human posture. These two tasks are highly correlated and can promote each other.

[0003] Therefore, in the prior art, a deep learning model is used to jointly process the key point detection and body part analysis of the human body. However, in the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: the deep learning model used in the prior art is manually designed by the user based on experience, which requires a lot of network model design experience and parameter adjustment experience, which is time-consuming and labor-intensive, and the designed network model is not the optimal network model, which leads to a low accuracy rate of key point detection and body part analysis. Summary of the invention

[0004] The embodiments of the present invention provide a joint processing method, device and product for human body key point detection and part analysis, which are used to solve the technical problem that the deep learning model used in the prior art is manually designed by the user based on experience, requires a lot of network model design experience and parameter adjustment experience, is time-consuming and labor-intensive, and the designed network model is not the optimal network model, which leads to low accuracy of human body key point detection and part analysis.

[0005] In a first aspect, an embodiment of the present invention provides a joint processing method for human body key point detection and part analysis, comprising:

[0006] Get the target image;

[0007] Using a target network model trained to convergence to perform joint processing of human key point detection and body part analysis on the target image to obtain human key point detection results and human body part analysis results, the target network model is obtained by searching the initial network model for the optimal network structure using a neural network automatic search technology during the process of training the initial network model using training samples;

[0008] The target network model outputs the human body key point detection results and human body part analysis results.

[0009] In a second aspect, an embodiment of the present invention provides a joint processing method for human body key point detection and part analysis, including:

[0010] Acquire a training sample, wherein the training sample includes an image to be trained, a real human body key point heat map corresponding to the image to be trained, and a real human body part analysis map;

[0011] In the process of training the initial network model using the training samples, the neural network automatic search technology is used to search for the optimal network structure of the initial network model;

[0012] Determine whether the trained network model meets the preset convergence conditions;

[0013] If it is determined that the trained network model meets the preset convergence condition, the network model meeting the preset convergence condition is determined as the target network model trained to convergence.

[0014] In a third aspect, an embodiment of the present invention provides a joint processing device for human body key point detection and part analysis, including:

[0015] An acquisition module, used for acquiring a target image;

[0016] A processing module, used for performing joint processing of human key point detection and part analysis on the target image using a target network model trained to convergence, so as to obtain human key point detection results and human part analysis results, wherein the target network model is obtained by searching for the optimal network structure of the initial network model using a neural network automatic search technology during the process of training the initial network model using training samples;

[0017] The output module is used to output the human body key point detection results and human body part analysis results through the target network model.

[0018] In a fourth aspect, an embodiment of the present invention provides a joint processing device for human body key point detection and part analysis, including:

[0019] An acquisition module is used to acquire training samples, wherein the training samples include an image to be trained, a real human body key point heat map and a real human body part analysis map corresponding to the image to be trained;

[0020] A training search module, used for searching the optimal network structure of the initial network model by using the neural network automatic search technology in the process of training the initial network model by using the training samples;

[0021] A judgment module is used to judge whether the trained network model meets the preset convergence conditions;

[0022] The determination module is used to determine the network model that meets the preset convergence condition as the target network model trained to convergence if it is determined that the trained network model meets the preset convergence condition.

[0023] In a fifth aspect, an embodiment of the present invention provides an electronic device, including: at least one processor and a memory;

[0024] The memory stores computer-executable instructions;

[0025] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to the first aspect or the second aspect.

[0026] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect or the second aspect.

[0027] In a seventh aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described in the first aspect or the second aspect when executed by a processor.

[0028] The joint processing method, device and product for human body key point detection and part analysis provided by the embodiments of the present invention obtain a target image; use a target network model trained to convergence to perform joint processing of human body key point detection and part analysis on the target image to obtain human body key point detection results and human body part analysis results, wherein the target network model is obtained by using a neural network automatic search technology to search for the optimal network structure of the initial network model during the training of the initial network model using training samples; the human body key point detection results and human body part analysis results are output through the target network model. Since the target network model trained to convergence is a network model that automatically searches for the optimal network structure in a candidate search space using a neural network automatic search technology, when performing the joint processing of the two tasks of human body key point detection and human body part analysis, the accuracy of human body key point detection and part analysis can be effectively improved, and it does not rely on the design experience and parameter adjustment experience of the network model, effectively improving the efficiency of human body key point detection and part analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0030] Figure 1 This is an application scenario diagram that can implement the joint processing method of human body key point detection and part analysis according to the embodiment of the present invention;

[0031] Figure 2 It is another application scenario diagram that can implement the joint processing method of human body key point detection and part analysis according to the embodiment of the present invention;

[0032] Figure 3 It is a flowchart of a joint processing method for key point detection and part analysis of a human body provided by an embodiment of the present invention;

[0033] Figure 4 is a flow chart of a joint processing method for key point detection and part analysis of a human body provided by another embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of the architecture of a target network model in another embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of the architecture of a target interaction sub-network model in another embodiment of the present invention;

[0036] Figure 7 1 is a flow chart of a joint processing method for human body key point detection and part analysis provided by another embodiment of the present invention;

[0037] Figure 8 It is a structural schematic diagram of a joint processing device for human body key point detection and part analysis provided by an embodiment of the present invention;

[0038] Fig. 9 is a structural schematic diagram of a joint processing device for human body key point detection and part analysis provided by another embodiment of the present invention;

[0039] Fig.10 is a first block diagram of an electronic device for implementing the joint processing method for key point detection and part analysis of a human body according to an embodiment of the present invention;

[0040] Fig.11 This is a second block diagram of an electronic device for implementing the joint processing method for human body key point detection and part analysis according to an embodiment of the present invention.

[0041] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] First, the terms involved in the embodiments of the present invention are explained:

[0044] Human key point detection: Human key point detection is to obtain the position of human key points in the image through computer vision technology. The position of human key points can be the core key point position, including: the top of the head, facial features, neck, limbs and other major joint positions.

[0045] Human body part parsing: Human body part parsing aims to parse human body images into different semantic regions, such as parsing human body images into hair region, skin region, top region, pants region, shoe region, etc.

[0046] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail.

[0047] Human key point detection and human part parsing are two important tasks in human visual understanding. They play a vital role in analyzing human posture. These two tasks are highly correlated and can promote each other. On the one hand, human key points are contained in different parts and regions, and human part parsing can provide semantic information for human key point detection. On the other hand, human key points include structural information of the human body, which can guide the semantic segmentation of different parts.

[0048] In the prior art, a deep learning model is used to perform human key point detection and human body part analysis separately, but the two tasks cannot be processed jointly. When a deep learning model is used to jointly process human key point detection and human body part analysis, the deep learning model is manually designed by the user based on experience, which requires a lot of network model design experience and parameter adjustment experience, which is time-consuming and labor-intensive, and the designed network model is not the optimal network model, which leads to low accuracy of human key point detection and part analysis.

[0049] Therefore, in the face of technical problems in the prior art, the inventors have concluded through creative research that in order to get rid of the problem that manually designing deep learning models based on experience requires a lot of network model design experience and parameter adjustment experience, and the problem that the designed network model is not the optimal network model, it is necessary to use the neural network automatic search technology in the process of training the initial network model using training samples to automatically search for the optimal network structure in the candidate search space, and use the network model corresponding to the optimal network structure as the target network model to perform joint processing of the two tasks of human key point detection and human body part analysis. To achieve the purpose of improving the accuracy of human key point detection and part analysis.

[0050] Therefore, based on the above creative findings, the inventors have proposed a technical solution of the embodiment of the present invention. The application scenario of the joint processing method for human key point detection and part analysis provided by the embodiment of the present invention is introduced below.

[0051] like Figure 1 As shown, the application scenario of the joint processing method of human key point detection and part analysis provided by this embodiment can be a batch processing scenario of images. Then, in this application scenario, it can include: a storage device 1 and an electronic device 2. Among them, the storage device 1 is used to store a target image including a human body, and the electronic device 2 pre-stores a target network model trained to convergence. When the conditions for the joint processing of human key point detection and part analysis are met, the target image is obtained from the storage device 1, and the target image is input into the target network model trained to convergence, and the target image is subjected to the joint processing of human key point detection and part analysis using the target network model to obtain the human key point detection results and the human body part analysis results, and the human key point detection results and the human body part analysis results are output, and the target image is associated with the human key point detection results and the human body part analysis results and sent to the storage device 1, so that the storage device 1 associates and stores the target image with the human key point detection results and the human body part analysis results. Since the target network model is obtained after searching the optimal network structure of the initial network model using the neural network automatic search technology during the training of the initial network model using training samples, the target network model is a model that is more suitable for the joint processing of human body key point detection and part analysis. Therefore, using the target network model for the joint processing of human body key point detection and part analysis can effectively improve the accuracy of human body key point detection and part analysis.

[0052] It is understandable that the application scenario of the joint processing method of human key point detection and part analysis provided in this embodiment can also be a scenario in which a certain application is used to perform joint processing of human key point detection and part analysis on an image. For example, in an application scenario using augmented reality technology (AR) or virtual reality technology (VR), Figure 2 As shown, it includes a terminal device 3 and an electronic device 2. The user selects or shoots a target image including a human body from a gallery through the operation interface of a certain application program through the terminal device 3, and the electronic device 2 obtains the target image from the terminal device 3, and inputs the target image into the target network model trained to convergence, and uses the target network model to perform joint processing of human key point detection and body part analysis on the target image to obtain human key point detection results and human body part analysis results, and outputs the human key point detection results and human body part analysis results, and performs subsequent processing according to the human key point detection results and human body part analysis results. For example, after obtaining the human key point detection results and human body part analysis results, the human body in the image is virtually dressed up, and a special effect picture is added to a certain position of the target image, and the target image that is subsequently processed according to the human key point detection results and human body part analysis results is sent to the terminal device 3 so that the terminal device can display it.

[0053] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0054] Embodiment 1

[0055] Figure 3 FIG. 1 is a flow chart of a joint processing method for human key point detection and part analysis provided by an embodiment of the present invention. Figure 3 As shown, the execution subject of the joint processing method for human key point detection and part analysis provided in this embodiment is a joint processing device for human key point detection and part analysis, and the joint processing device for human key point detection and part analysis can be integrated in an electronic device. Then the joint processing method for human key point detection and part analysis provided in this embodiment includes the following steps:

[0056] Step 101, obtaining a target image.

[0057] In this embodiment, the target image is an image that is jointly processed for human body key point detection and part analysis, and the target image may include a human body region.

[0058] In this embodiment, when acquiring the target image, the acquisition method may be different according to different application scenarios. For example, in the scenario of batch processing of target images, the target image is stored in a storage device, and the storage device is accessed to acquire the target image. For example, when the method of the embodiment of the present invention is used in a certain application, a component for acquiring the target image may be provided to the user through the operation interface of the application. The user triggers the component to acquire the target image from the gallery of the terminal device or by shooting, and after the user clicks the "Confirm" component on the operation interface, the terminal device sends the target image to the electronic device.

[0059] The target image may be a grayscale image or a color image, which is not limited in this embodiment.

[0060] Step 102, using a target network model trained to convergence to perform joint processing of human key point detection and part analysis on the target image to obtain human key point detection results and human part analysis results. The target network model is obtained by searching the initial network model for the optimal network structure using a neural network automatic search technology during the process of training the initial network model using training samples.

[0061] Step 103, outputting the human body key point detection results and human body part analysis results through the target network model.

[0062] In this embodiment, the training samples include the image to be trained, the real human body key point heat map and the real human body part analysis map corresponding to the image to be trained, wherein the image to be trained includes the human body area.

[0063] In this embodiment, the initial network model includes multiple initial network sub-models, such as: an initial backbone sub-network model, an initial key point detection sub-network model, an initial part analysis sub-network model, and an initial interaction sub-network model.

[0064] Among them, at least one initial network sub-model includes a searchable conventional operation unit and / or a searchable dimensionality reduction operation unit, the searchable conventional operation unit includes multiple candidate operation spaces, and the searchable dimensionality reduction operation unit includes multiple candidate operation spaces. For example, the initial backbone sub-network model includes multiple searchable conventional operation units and searchable dimensionality reduction operation units. The key point detection sub-network model includes multiple searchable conventional operation units. The initial part analysis sub-network model includes multiple searchable conventional operation units, and the initial interaction sub-network model includes multiple searchable conventional operation units.

[0065] The searchable regular operation unit is composed of multiple nodes connected, each node corresponds to the feature map input in the initial sub-network model or the feature map output by the previous searchable regular operation unit, and there are multiple candidate operation spaces between the nodes, and the multiple candidate operation spaces may include convolution, pooling, cross-connection and other operation spaces. After the searchable regular operation unit performs the operation of the candidate operation space with the corresponding feature map, the dimension remains unchanged.

[0066] Similarly, the searchable dimensionality reduction operation unit is composed of multiple nodes connected, each node corresponds to the feature map input in the initial sub-network model or the feature map output by the previous searchable conventional operation unit or the feature map output by the previous searchable dimensionality reduction operation unit, and there are multiple candidate operation spaces between the nodes, and the multiple candidate operation spaces may include convolution, pooling and cross-connection operation spaces. After the searchable dimensionality reduction operation unit performs the operation of the candidate operation space with the corresponding feature map, the dimension is reduced.

[0067] Specifically, in this embodiment, a target network model trained to convergence is used to perform joint processing of human key point detection and part analysis on a target image. Before obtaining the human key point detection results and human part analysis results, a neural network automatic search technology is used to perform an optimal network structure search on the initial network model in a candidate search space during the training of the initial network model using training samples to obtain an optimal network structure network model. The network model of the optimal network structure is the target network model trained to convergence. The target network model trained to convergence includes multiple target sub-network models of the optimal network structure. At least one target sub-network model includes a target conventional operation unit and / or a target dimensionality reduction operation unit of the optimal network structure. Among them, the target conventional operation unit and / or the target dimensionality reduction operation unit of the optimal network structure include a target operation space. Among them, the target operation space is one of convolution, pooling and cross connection.

[0068] Specifically, in this embodiment, after obtaining the target network model trained to convergence, the target image is input into the target network model trained to convergence, and the target network model trained to convergence performs joint processing of human key point detection and body part analysis on the target image, and outputs human key point detection results and human body part analysis results. Among them, the human key point detection result can be a human key point heat map in the target image. The human body part analysis result can be a human body part analysis map in the target image.

[0069] Among them, when the target network model trained to convergence performs joint processing of human key point detection and part analysis on the target image, the target backbone sub-network model can first extract common features of the target image, and then the target key point detection sub-network model extracts human key point features, and the target part analysis sub-network model extracts human part features. Finally, the target interaction sub-network model performs feature interaction between human key point features and human part features according to the determined optimal interaction method.

[0070] It is understandable that the target network model trained to convergence may also perform other joint processing of human body key point detection and part analysis on the target image, which is not limited in this embodiment.

[0071] The joint processing method for human body key point detection and part analysis provided in the present embodiment comprises the following steps: acquiring a target image; performing joint processing of human body key point detection and part analysis on the target image using a target network model trained to convergence, so as to obtain human body key point detection results and human body part analysis results, wherein the target network model is obtained by performing an optimal network structure search on the initial network model using a neural network automatic search technology during the process of training the initial network model using training samples; outputting the human body key point detection results and human body part analysis results through the target network model, and since the target network model trained to convergence is a network model that automatically searches for the optimal network structure in a candidate search space using a neural network automatic search technology, when performing the joint processing of the two tasks of human body key point detection and human body part analysis, the accuracy of human body key point detection and part analysis can be effectively improved, and the efficiency of human body key point detection and part analysis can be effectively improved without relying on the design experience and parameter adjustment experience of the network model.

[0072] Embodiment 2

[0073] Figure 4 FIG. 1 is a flow chart of a joint processing method for human key point detection and part analysis provided by another embodiment of the present invention. Figure 4 As shown, the joint processing method for human key point detection and part analysis provided in this embodiment is based on the joint processing method for human key point detection and part analysis provided in Example 1, and further refines step 102. The joint processing method for human key point detection and part analysis provided in this embodiment includes the following steps:

[0074] Step 201, acquiring a target image.

[0075] In this embodiment, the implementation of step 201 is similar to the implementation of step 101 in the first embodiment of the present invention, and will not be described in detail here.

[0076] Step 202, using the target network model to perform human body key point detection and body part analysis processing on the target image, so as to obtain a first human body key point detection feature map and a first human body part analysis feature map respectively.

[0077] As an optional implementation, Figure 5 As shown, the target network model includes: a target backbone sub-network model, a target key point detection sub-network model, and a target part analysis sub-network model. Then step 202 includes the following steps:

[0078] Step 2021, using the target backbone sub-network model to extract common features of the target image to obtain multiple common feature maps, each common feature map has a different resolution.

[0079] Alternatively, if Figure 5 As shown, the target backbone subnetwork model includes: a feature preservation module and multiple feature dimensionality reduction modules, the feature preservation module includes multiple third target conventional operation units, and the feature dimensionality reduction module includes: a first target dimensionality reduction operation unit and multiple fourth target conventional operation units.

[0080] Then step 2021 includes the following steps:

[0081] Step 2021a, arrange the feature preservation module and multiple feature dimension reduction modules in sequence.

[0082] like Figure 5 As shown in FIG. 1 , the feature preservation module is arranged at the front end of multiple feature dimension reduction modules. The feature dimension reduction modules are arranged in sequence at the back end. The target image is first input into the feature preservation module, processed by the feature preservation module, input into the feature dimension reduction module, processed by the previous feature dimension reduction module, and then input into the next feature dimension reduction module for further processing until the last feature dimension reduction module outputs the processing result.

[0083] Step 2021b, operates the target image and the third target conventional operation unit in the feature preservation module according to the corresponding target operation space to obtain a first common feature map.

[0084] Specifically, in this embodiment, the multiple target conventional operation units in the feature preservation module included in the target trunk sub-network model are used as the third target conventional operation unit, and the multiple target conventional operation units in the feature dimension reduction module included in the target trunk sub-network model are used as the fourth target conventional operation unit. The target operation space in the third target conventional operation unit and the fourth target conventional operation unit is determined after the initial network model is trained to convergence through training samples, so it may be any one of convolution, pooling and cross connection.

[0085] In this embodiment, the target image is first operated with the first third target conventional operation unit in the feature retention module according to the corresponding target operation space, and then the feature map after operation is operated with the second third target conventional operation unit according to the corresponding target operation space, and the feature map after operation is operated with the third third target conventional operation unit according to the corresponding target operation space, and so on, until the feature map after operation is operated with the last third target conventional operation unit according to the corresponding target operation space, the feature map obtained is the first common feature map, and the size of the first common feature map is consistent with the size of the target image. It can be understood that the corresponding target operation space in each third target conventional operation unit can be different.

[0086] Among them, the first common feature map represents the more stable features of the target image.

[0087] Step 2021c: determine the first common feature map as the previous common feature map corresponding to the current feature dimension reduction module.

[0088] Step 2021d, for each current feature dimensionality reduction module, the previous common feature map is operated with the corresponding target operation space in the first target dimensionality reduction operation unit and the fourth target regular operation unit in the current feature dimensionality reduction module to obtain the previous common feature map corresponding to the next feature dimensionality reduction module.

[0089] Specifically, in this embodiment, a plurality of feature dimension reduction modules are included in the target backbone subnetwork model. Each feature dimension reduction module is arranged in sequence. First, the first common feature map and the first target dimension reduction operation unit in the first feature dimension reduction module are operated according to the corresponding target operation space, and the feature map after operation and the fourth target conventional operation unit in the first feature dimension reduction module are operated according to the corresponding target operation space, and the previous common feature map of the second feature dimension reduction module is output. Then, the previous common feature map and the first target dimension reduction operation unit in the second feature dimension reduction module are operated according to the corresponding target operation space, and the feature map after operation and the fourth target conventional operation unit in the second feature dimension reduction module are operated according to the corresponding target operation space, and the previous common feature map of the third feature dimension reduction module is output. And so on, until the last common feature map is obtained.

[0090] Among them, the feature dimension reduction module has the function of extracting features and reducing the dimension of the target image. Therefore, after each processing of a feature dimension reduction module, the dimension of the output common feature map is half of the dimension of the input common feature map. Therefore, each common feature map has a different resolution, that is, the size of each common feature map is different. Figure 5 As shown in the figure, if there are three feature dimension reduction modules, the size of the final output common feature map is 1 / 8 of the target image.

[0091] Step 2022: Use the target key point detection sub-network model to extract human key point features from multiple shared feature maps to obtain a first human key point detection feature map.

[0092] Alternatively, if Figure 5 As shown, the target key point detection sub-network model includes: a third fusion module and a posture refinement module, and the posture refinement module includes multiple fifth target conventional operation units.

[0093] Then step 2022 includes the following steps:

[0094] Step 2022a, using a bilinear interpolation algorithm to upsample the low-resolution common feature map in two adjacent common feature maps, so that the upsampled common feature map has the same resolution as the adjacent common feature maps.

[0095] Step 2022b: Use a third fusion module to gradually fuse the up-sampled common feature map with adjacent common feature maps of the same resolution to obtain a fused common feature map.

[0096] In this embodiment, for multiple common feature maps, they are sorted from low to high according to resolution. The common feature map with the lowest resolution is obtained, and the common feature map with the lowest resolution is upsampled using a bilinear interpolation algorithm. The upsampled common feature map has the same resolution as the adjacent common feature map. The upsampled common feature map is fused with the adjacent common feature map with the same resolution through the third fusion module. The common feature map after this fusion has a different resolution from the adjacent common feature map. The bilinear interpolation algorithm is continued to be used for upsampling. The common feature map after the upsampling has the same resolution as the adjacent common feature map. The upsampled common feature map is fused with the adjacent common feature map with the same resolution through the third fusion module. And so on, until the third fusion module fuses the last two common feature maps to obtain the final fused common feature map.

[0097] Step 2022c, the fused common feature map is operated with the fifth target conventional operation unit in the posture refinement module according to the corresponding target operation space to obtain the first human key point detection feature map.

[0098] In this embodiment, the multiple target conventional operation units in the posture refinement module included in the target key point detection sub-network model are used as the fifth target conventional operation unit. The target operation space in the fifth target conventional operation unit is determined after the initial network model is trained to convergence by training samples, so it may be any one of convolution, pooling and cross connection.

[0099] In this embodiment, the fused common feature map is first operated with the first fifth target conventional operation unit in the posture refinement module according to the corresponding target operation space, and then the operated feature map is operated with the second fifth target conventional operation unit according to the corresponding target operation space, and the operated feature map is operated with the third fifth target conventional operation unit according to the corresponding target operation space, and so on, until the operated feature map is operated with the last fifth target conventional operation unit according to the corresponding target operation space, and the obtained feature map is the first human key point detection feature map. It can be understood that the corresponding target operation space in each fifth target conventional operation unit can be different.

[0100] Among them, the first human key point detection feature map is a feature map that highlights the key point features of the human body.

[0101] Step 2023: Use the target part parsing sub-network model to extract human body part features from multiple shared feature maps to obtain a first human body part parsing feature map.

[0102] Alternatively, if Figure 5 As shown, the target part parsing sub-network model includes: a searchable dilated convolution pooling pyramid unit and a part parsing refinement module; the searchable dilated convolution pooling pyramid unit includes: a sixth target conventional operation unit and a second target dimensionality reduction operation unit, and the part parsing refinement module includes: multiple seventh target conventional operation units.

[0103] Then step 2023 includes the following steps:

[0104] Step 2023a, uses a searchable atrous convolution pooling pyramid unit to perform operations on multiple common feature maps according to the corresponding target operation space in the sixth target conventional operation unit and the second target dimensionality reduction operation unit to obtain a human body part analytical feature map after operation.

[0105] In this embodiment, a plurality of common feature maps are input into a searchable atrous convolutional pooling pyramid unit, and the structure of the searchable atrous convolutional pooling pyramid unit is similar to the original atrous convolutional pooling pyramid (English: atrousspatial pyramid pooling, referred to as: ASPP), except that the original conventional convolutional layer is replaced by the sixth target conventional operation unit, and the original dimension reduction convolutional layer is replaced by the second target dimension reduction operation unit, so as to achieve the searchability of the network structure of the atrous convolutional pooling pyramid unit. The atrous convolutional pooling pyramid unit with the optimal network structure is determined, and the optimal atrous convolutional pooling pyramid unit is used to operate the plurality of common feature maps according to the corresponding target operation space in the sixth target conventional operation unit and the second target dimension reduction operation unit, so as to obtain the human body part analysis feature map after the operation.

[0106] Step 2023b, operates the calculated human body part analysis feature map and the seventh target conventional operation unit in the part analysis refinement module according to the corresponding target operation space to obtain a first human body part analysis feature map.

[0107] In this embodiment, the multiple target conventional operation units in the part parsing refinement module included in the target part parsing sub-network model are used as the seventh target conventional operation unit. The target operation space in the seventh target conventional operation unit is determined after the initial network model is trained to convergence through training samples, so it may be any one of convolution, pooling and cross connection.

[0108] In this embodiment, the calculated human body part analysis feature map is firstly calculated with the first seventh target conventional operation unit in the part analysis refinement module according to the corresponding target operation space, and then the calculated feature map is calculated with the second seventh target conventional operation unit according to the corresponding target operation space, and the calculated feature map is calculated with the third seventh target conventional operation unit according to the corresponding target operation space, and so on, until the calculated feature map is calculated with the last seventh target conventional operation unit according to the corresponding target operation space, and the obtained feature map is the first human body part analysis feature map. It can be understood that the corresponding target operation space in each seventh target conventional operation unit can be different.

[0109] Among them, the first human body part analysis feature map is a feature map that highlights the characteristics of the human body part.

[0110] Step 203, using the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human body part analysis feature map according to the determined optimal interaction method, so as to obtain the second human key point detection feature map and the second human body part analysis feature map after interaction.

[0111] As an optional implementation, Figure 5 As shown, in this embodiment, the target network model also includes: a target interaction sub-network model. Figure 6 As shown, the target interaction sub-network model includes: two target searchable modules and two fusion modules, the target searchable module includes multiple first target regular operation units, and the fusion module includes multiple second target regular operation units.

[0112] Then step 203 includes the following steps:

[0113] Step 2031, the first human key point detection feature map and the first target conventional operation unit in the first target searchable module are operated according to the corresponding target operation space to obtain the human key point detection feature map after operation, and the human key point detection feature map after operation and the first human body part analysis feature map are operated using the target operation space corresponding to the second target conventional operation unit in the first fusion module to obtain the second human key point detection feature map.

[0114] In this embodiment, the two target searchable modules included in the target interaction sub-network model are respectively the first target searchable module and the second target searchable module. The two fusion modules included in the target interaction sub-network model are respectively the first fusion module and the second fusion module.

[0115] In this embodiment, the target conventional operation unit existing in the target searchable module included in the target interactive sub-network model is used as the first target conventional operation unit. The target conventional operation unit existing in the fusion module included in the target interactive sub-network model is used as the second target conventional operation unit. The target operation space in the first target conventional operation unit and the first target conventional operation unit is determined after the initial network model is trained to convergence through training samples, so it may be any one of convolution, pooling and cross connection.

[0116] In this embodiment, the first human key point detection feature map is first operated with the first first target regular operation unit in the first target searchable module according to the corresponding target operation space, and then the feature map after operation is operated with the second first target regular operation unit according to the corresponding target operation space, and so on, until the feature map after operation is operated with the last first target regular operation unit according to the corresponding target operation space, and the feature map obtained is the human key point detection feature map after operation. It can be understood that the corresponding target operation space in each first target regular operation unit can be different.

[0117] Similarly, if Figure 6 As shown, the calculated human key point detection feature map and the first human body part analysis feature map are calculated using the target operation space corresponding to the first second target conventional operation unit in the first fusion module to obtain the calculated feature map, and then the calculated feature map and the second second target conventional operation unit are calculated according to the corresponding target operation space, and the calculated feature map and the third second target conventional operation unit are calculated according to the corresponding target operation space, and so on, until the calculated feature map and the last second target conventional operation unit are calculated according to the corresponding target operation space, and the obtained feature map is the second human key point detection feature map. It can be understood that the corresponding target operation space in each second target conventional operation unit can be different.

[0118] Step 2032, the first human body part parsing feature map and the first target conventional operation unit in the second target searchable module are operated according to the corresponding target operation space to obtain the human body part parsing feature map after operation, and the human body part parsing feature map after operation and the first human body key point detection feature map are operated using the target operation space corresponding to the second target conventional operation unit in the second fusion module to obtain the second human body part parsing feature map.

[0119] This step is similar to the step 2031. First, the first human body part analysis feature map and the first first target regular operation unit in the second target searchable module are operated according to the corresponding target operation space, and then the feature map after operation is operated with the second first target regular operation unit according to the corresponding target operation space, and so on, until the feature map after operation is operated with the last first target regular operation unit according to the corresponding target operation space, and the feature map obtained is the human body part analysis feature map after operation. It can be understood that the corresponding target operation space in each first target regular operation unit can be different.

[0120] Similarly, if Figure 6 As shown, the calculated human body part analysis feature map and the first human body key point detection feature map are calculated using the target operation space corresponding to the first second target conventional operation unit in the second fusion module to obtain the calculated feature map, and then the calculated feature map and the second second target conventional operation unit are calculated according to the corresponding target operation space, and the calculated feature map and the third second target conventional operation unit are calculated according to the corresponding target operation space, and so on, until the calculated feature map and the last second target conventional operation unit are calculated according to the corresponding target operation space, and the obtained feature map is the second human body part analysis feature map. It can be understood that the corresponding target operation space in each second target conventional operation unit can be different.

[0121] Step 204, determining a human body key point detection result based on the first human body key point detection feature map and the second human body key point detection feature map, and determining a human body part analysis result based on the first human body part analysis feature map and the second human body part analysis feature map.

[0122] Optionally, in this embodiment, if Figure 5 As shown, the target network model also includes: a convolution layer. Then in step 204, determining the human key point detection result according to the first human key point detection feature map and the second human key point detection feature map includes the following steps:

[0123] Step 2041, connect the first human key point detection feature map and the second human key point detection feature map in the channel dimension to obtain a key point detection feature map after the channel dimension connection.

[0124] In this embodiment, the first human key point detection feature map and the second human key point detection feature map have three dimensions, respectively, which are length and width, which are the number of channels. Therefore, the first human key point detection feature map and the second human key point detection feature map are spliced ​​together according to the dimension of the channel to form a key point detection feature map after the channel dimension is connected.

[0125] Step 2042, using the convolution kernel in the convolution layer, performs a convolution operation on the key point detection feature map after the channel dimension connection to obtain a target human key point heat map.

[0126] Step 2043, determining the target human body key point heat map as the human body key point detection result.

[0127] In this embodiment, the convolution layer includes a convolution kernel, and the key point detection feature map after the channel dimension is connected is convolved with the convolution kernel, and the map obtained after the convolution calculation is determined as the human body key point heat map. In the human body key point heat map, the heat value of the key point is significantly different from the heat value of the non-key point, so as to obtain the human body key point detection result.

[0128] Optionally, in this embodiment, if Figure 5 As shown, the target network model also includes: a deconvolution layer. Then, in step 204, determining the human body part parsing result according to the first human body part parsing feature map and the second human body part parsing feature map includes the following steps:

[0129] Step 2044, the first human body part parsing feature map and the second human body part parsing feature map are connected in the channel dimension to obtain a part parsing feature map after the connection in the channel dimension.

[0130] This step is similar to the implementation of step 2041. The first human body part analysis feature map and the second human body part analysis feature map have three dimensions, which are length and width, which are the number of channels. Therefore, the first human body part analysis feature map and the second human body part analysis feature map are spliced ​​together according to the dimension of the channel to form a key point detection feature map after the channel dimension is connected.

[0131] Step 2045, using a deconvolution layer to upsample the part analysis feature map after the channel dimension connection to obtain the target human body part analysis map.

[0132] Step 2046, determining the target human body part analysis diagram as the human body part analysis result.

[0133] In this embodiment, the key point detection feature map after the channel dimension connection is input into the deconvolution layer, and the deconvolution layer performs an upsampling operation on the part analysis feature map after the channel dimension connection, and the map after the upsampling operation is the target human body part analysis map. The size of the target human body part analysis map is the same as the size of the target image. Pixel classification has been performed in the target human body part analysis map to obtain the analysis result of the human body part.

[0134] Step 205: outputting the human body key point detection results and human body part analysis results through the target network model.

[0135] In this embodiment, the implementation of step 205 is similar to the implementation of step 103 in the first embodiment, and will not be described in detail here.

[0136] The joint processing method for human key point detection and part analysis provided in this embodiment, when using the target network model trained to convergence to perform joint processing of human key point detection and part analysis on the target image to obtain the human key point detection result and the human part analysis result, the target network model is used to perform human key point detection and part analysis processing on the target image respectively to obtain the first human key point detection feature map and the first human part analysis feature map respectively, and the target interaction sub-network model is used to perform feature interaction on the first human key point detection feature map and the first human part analysis feature map according to the determined optimal interaction mode to obtain the second human key point detection feature map and the second human part analysis feature map after interaction respectively, and the human key point detection result is determined according to the first human key point detection feature map and the second human key point detection feature map, and the human part analysis result is determined according to the first human part analysis feature map and the second human part analysis feature map. Since the human key point detection task and the human part analysis task can perform feature interaction according to the optimal interaction mode, the accuracy of human key point detection and part analysis can be further improved.

[0137] Embodiment 3

[0138] Figure 7 FIG. 1 is a flow chart of a combined processing method for human key point detection and part analysis provided by another embodiment of the present invention. Figure 7 As shown, the joint processing method for human key point detection and part analysis provided in this embodiment is based on the joint processing method for human key point detection and part analysis provided in Example 1 or Example 2, and further includes the following steps:

[0139] Step 301, obtaining training samples, the training samples including the image to be trained, the real human body key point heat map corresponding to the image to be trained and the real human body part analysis map.

[0140] In this embodiment, there are multiple training samples, and the training set can be composed of multiple training samples. When obtaining the training samples, the training samples can be obtained by accessing the storage device. Each training sample includes an image to be trained, and also includes a real human key point heat map and a real human part analysis map corresponding to the image to be trained.

[0141] Step 302, in the process of training the initial network model using the training samples, the neural network automatic search technology is used to search for the optimal network structure of the initial network model.

[0142] Optionally, in this embodiment, the initial network model includes multiple initial network sub-models, at least one initial network sub-model includes a searchable conventional operation unit and / or a searchable dimensionality reduction operation unit, the searchable conventional operation unit includes multiple candidate operation spaces, and the searchable dimensionality reduction operation unit includes multiple candidate operation spaces.

[0143] Among them, if at least one initial network sub-model includes: an initial backbone sub-network model, an initial key point detection sub-network model, an initial part parsing sub-network model and an initial interaction sub-network model. Then the initial backbone sub-network model includes multiple searchable conventional operation units and searchable dimensionality reduction operation units. The key point detection sub-network model includes multiple searchable conventional operation units. The initial part parsing sub-network model includes multiple searchable conventional operation units, and the initial interaction sub-network model includes multiple searchable conventional operation units. The multiple candidate operation spaces included in the searchable conventional operation unit and the searchable dimensionality reduction operation unit can be operation spaces such as convolution, pooling and cross-connection.

[0144] Accordingly, step 302 specifically includes:

[0145] In the process of training the initial network model using training samples, a neural network automatic search technology is used to optimally search the candidate operation space of the corresponding searchable conventional operation unit and / or searchable dimensionality reduction operation unit in at least one initial network sub-model.

[0146] Specifically, in this embodiment, if at least one initial network sub-model includes: an initial backbone sub-network model, an initial key point detection sub-network model, an initial part parsing sub-network model and an initial interaction sub-network model, the training sample is first input into the initial backbone sub-network model, and the initial backbone sub-network model extracts common features from the training samples to obtain multiple initial common feature maps, each of which has a different resolution. Then, the initial key point detection sub-network model is used to extract the key point features of the human body from the multiple initial common feature maps to obtain the first initial key point detection feature map, the initial part analysis sub-network model is used to extract the part features of the human body from the multiple initial common feature maps to obtain the first initial part analysis feature map, the initial interaction sub-network model is used to perform feature interaction on the first initial key point detection feature map and the first initial part analysis feature map according to the candidate interaction mode to obtain the second initial key point detection feature map and the second initial part analysis feature map after interaction respectively, the initial predicted key point heat map is determined according to the first initial key point detection feature map and the second initial key point detection feature map, and the initial predicted part analysis map is determined according to the first initial part analysis feature map and the second initial part analysis feature map. This process is a training process for the initial network model, and any candidate operation space is used in each initial sub-network model in this training process. Then, according to the deviation between the initial predicted human body key point heat map and the real human body key point heat map, as well as the deviation between the initial predicted human body part parsing map and the real human body part parsing map, the neural network automatic search technology is used to search the optimal candidate operation space for the candidate operation space. In each training process, the deviation between the initial predicted human body part parsing map and the real human body part parsing map, as well as the deviation between the initial predicted human body part parsing map and the real human body part parsing map are continuously reduced, or the search times reach the preset search times, and finally the target network model with the optimal network structure is obtained.

[0147] Step 303, determining whether the trained network model meets the preset convergence condition.

[0148] Optionally, in this embodiment, step 303 includes the following steps:

[0149] Step 3031, determine whether the value of the total loss function corresponding to the trained network model is less than a preset threshold or whether the number of searches reaches a preset number of searches.

[0150] Optionally, if determining whether the trained network model satisfies the preset convergence condition is determining whether the value of the total loss function corresponding to the trained network model is less than a preset threshold, then before determining whether the value of the total loss function corresponding to the trained network model is less than the preset threshold, the following steps are also included:

[0151] Step 302-31, determining a first loss function based on the predicted human body key point heat map and the real human body key point heat map determined during the training of the initial network model.

[0152] Among them, the first loss function is the deviation between the predicted human body key point heat map and the real human body key point heat map determined during the training of the initial network model.

[0153] Step 302-32, determining a second loss function according to the predicted human body part analysis graph and the real human body part analysis graph determined during the training of the initial network model.

[0154] Among them, the second loss function is the deviation between the predicted human body part analysis map and the actual human body part analysis map determined during the training of the initial network model.

[0155] Step 302-33, determining a total loss function according to the first loss function and the second loss function.

[0156] In this embodiment, the total loss function is a weighted sum of the first loss function and the second loss function, wherein the weights corresponding to the first loss function and the second loss function can be pre-configured.

[0157] Step 3032: If it is determined that the total loss function is less than a preset threshold or the number of searches reaches a preset number of searches, it is determined that a preset convergence condition is met.

[0158] In this embodiment, after determining the total loss function, the value of the total loss function can be calculated after each training. The smaller the value of the total loss function, the more accurate the human key point detection results and human body part analysis results. If the value of the total loss function is less than the preset threshold, it means that the trained network model has met the convergence condition. Among them, the preset threshold can be configured in advance.

[0159] Alternatively, in this embodiment, a preset number of searches is preconfigured, and the preset number of searches may be equal to the number of preset model iterations, to determine whether the number of searches has reached the preset number of searches. If the preset number of searches has been reached, it indicates that the trained network model has met the convergence conditions.

[0160] Step 304: If it is determined that the trained network model meets the preset convergence condition, the network model meeting the preset convergence condition is determined as the target network model trained to convergence.

[0161] In this embodiment, the trained network model that meets the preset convergence conditions is the network model with the optimal network structure. The network model with the optimal network structure is determined as the target network model, and the target network model is used to perform joint processing of human body key point detection and part analysis on the target image.

[0162] The joint processing method for human body key point detection and part analysis provided in the present embodiment obtains training samples, and the training samples include images to be trained, real human body key point heat maps and real human body part analysis maps corresponding to the images to be trained; in the process of training the initial network model using the training samples, the neural network automatic search technology is used to search the initial network model for the optimal network structure; it is judged whether the trained network model meets the preset convergence conditions; if it is determined that the trained network model meets the preset convergence conditions, the network model that meets the preset convergence conditions is determined as the target network model trained to convergence, because the target network model is automatically searched in the candidate search space using the neural network automatic search technology, the target network model obtained after training is a network model with the optimal network structure, which is more suitable for the joint processing of the two tasks of human body key point detection and human body part analysis, thereby effectively improving the accuracy of human body key point detection and part analysis.

[0163] Embodiment 4

[0164] Figure 8 FIG. 1 is a schematic diagram of a joint processing device for human body key point detection and part analysis provided by an embodiment of the present invention. Figure 8 As shown, the joint processing device 40 for human body key point detection and part analysis provided in this embodiment includes: an acquisition module 41, a processing module 42, and an output module 43.

[0165] Among them, the acquisition module 41 is used to acquire the target image. The processing module 42 is used to use the target network model trained to convergence to perform joint processing of human key point detection and body part analysis on the target image to obtain the human key point detection results and the human body part analysis results. The target network model is obtained by using the neural network automatic search technology to search for the optimal network structure of the initial network model during the training of the initial network model using the training samples. The output module 43 is used to output the human key point detection results and the human body part analysis results through the target network model.

[0166] The combined processing device for human body key point detection and part analysis provided in this embodiment can perform Figure 2 The technical solution of the method embodiment shown in the figure has the same implementation principle and technical effect as Figure 2 The method embodiments shown are similar and will not be described in detail here.

[0167] Optionally, in this embodiment, the target network model includes: a target interaction sub-network model;

[0168] Correspondingly, the processing module 42 is specifically used to: use the target network model to perform human key point detection and part analysis processing on the target image, so as to obtain a first human key point detection feature map and a first human part analysis feature map, respectively; use the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human part analysis feature map according to a determined optimal interaction method, so as to obtain a second human key point detection feature map and a second human part analysis feature map after interaction, respectively; determine the human key point detection result according to the first human key point detection feature map and the second human key point detection feature map, and determine the human part analysis result according to the first human part analysis feature map and the second human part analysis feature map.

[0169] Optionally, the target interaction sub-network model includes: two target searchable modules and two fusion modules, the target searchable module includes a plurality of first target conventional operation units, and the fusion module includes a plurality of second target conventional operation units.

[0170] Accordingly, the processing module, when using the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human body part analysis feature map according to the determined optimal interaction mode to respectively obtain the second human key point detection feature map and the second human body part analysis feature map after interaction, is specifically used for:

[0171] The first human body key point detection feature map and the first target conventional operation unit in the first target searchable module are operated according to the corresponding target operation space to obtain the human body key point detection feature map after operation, and the human body key point detection feature map after operation and the first human body part analysis feature map are operated using the target operation space corresponding to the second target conventional operation unit in the first fusion module to obtain the second human body key point detection feature map; the first human body part analysis feature map and the first target conventional operation unit in the second target searchable module are operated according to the corresponding target operation space to obtain the human body part analysis feature map after operation, and the human body part analysis feature map after operation and the first human body key point detection feature map are operated using the target operation space corresponding to the second target conventional operation unit in the second fusion module to obtain the second human body part analysis feature map.

[0172] Optionally, the target network model also includes: a target backbone sub-network model, a target key point detection sub-network model, and a target part analysis sub-network model.

[0173] Accordingly, the processing module 42, when using the target network model to perform human key point detection and part analysis processing on the target image to obtain the first human key point detection feature map and the first human part analysis feature map, is specifically used to:

[0174] The target backbone sub-network model is used to extract common features of the target image to obtain multiple common feature maps, each of which has a different resolution; the target key point detection sub-network model is used to extract human body key point features from multiple common feature maps to obtain a first human body key point detection feature map; the target part analysis sub-network model is used to extract human body part features from multiple common feature maps to obtain a first human body part analysis feature map.

[0175] Optionally, the target backbone subnetwork model includes: a feature preservation module and multiple feature dimensionality reduction modules, the feature preservation module includes multiple third target conventional operation units, and the feature dimensionality reduction module includes: a first target dimensionality reduction operation unit and multiple fourth target conventional operation units.

[0176] Accordingly, when the processing module 42 uses the target backbone sub-network model to extract common features from the target image to obtain multiple common feature maps, it is specifically used to:

[0177] The feature preservation module and multiple feature dimension reduction modules are arranged in sequence; the target image and the third target conventional operation unit in the feature preservation module are operated according to the corresponding target operation space to obtain a first common feature map; the first common feature map is determined as the previous common feature map corresponding to the current feature dimension reduction module; for each current feature dimension reduction module, the previous common feature map is operated with the first target dimension reduction operation unit and the corresponding target operation space in the fourth target conventional operation unit in the current feature dimension reduction module to obtain the previous common feature map corresponding to the next feature dimension reduction module.

[0178] Optionally, the target key point detection sub-network model includes: a third fusion module and a posture refinement module, and the posture refinement module includes multiple fifth target conventional operation units.

[0179] Accordingly, the processing module 42, when using the target key point detection sub-network model to extract human key point features from a plurality of shared feature maps to obtain a first human key point detection feature map, is specifically used to:

[0180] A bilinear interpolation algorithm is used to upsample the low-resolution common feature maps in two adjacent common feature maps, and the common feature map after upsampling has the same resolution as the adjacent common feature map; a third fusion module is used to gradually fuse the common feature map after upsampling with the adjacent common feature maps with the same resolution to obtain a fused common feature map; the fused common feature map and the fifth target conventional operation unit in the posture refinement module are operated according to the corresponding target operation space to obtain a first human key point detection feature map.

[0181] Optionally, the target part parsing sub-network model includes: a searchable dilated convolution pooling pyramid unit and a part parsing refinement module; the searchable dilated convolution pooling pyramid unit includes: a sixth target conventional operation unit and a second target dimensionality reduction operation unit, and the part parsing refinement module includes: multiple seventh target conventional operation units.

[0182] Accordingly, the processing module 42, when using the target part parsing sub-network model to extract human body part features from a plurality of common feature maps to obtain a first human body part parsing feature map, is specifically used to:

[0183] A searchable atrous convolution pooling pyramid unit is used to perform operations on multiple common feature maps according to the corresponding target operation space in the sixth target conventional operation unit and the second target dimensionality reduction operation unit to obtain a human body part parsing feature map after operation; the human body part parsing feature map after operation is operated with the seventh target conventional operation unit in the part parsing refinement module according to the corresponding target operation space to obtain a first human body part parsing feature map.

[0184] Optionally, the target network model also includes: a convolutional layer.

[0185] Accordingly, when determining the human body key point detection result according to the first human body key point detection feature map and the second human body key point detection feature map, the processing module 42 is specifically used to:

[0186] The first human key point detection feature map and the second human key point detection feature map are connected in the channel dimension to obtain the key point detection feature map after the channel dimension connection; the convolution kernel in the convolution layer is used to perform a convolution operation on the key point detection feature map after the channel dimension connection to obtain a target human key point heat map; the target human key point heat map is determined as the human key point detection result.

[0187] Optionally, the target network model also includes: a deconvolution layer.

[0188] Accordingly, when determining the human body part analysis result according to the first human body part analysis feature map and the second human body part analysis feature map, the processing module 42 is specifically used to:

[0189] The first human body part parsing feature map and the second human body part parsing feature map are connected in channel dimension to obtain a part parsing feature map after the connection in channel dimension; a deconvolution layer is used to upsample the part parsing feature map after the connection in channel dimension to obtain a target human body part parsing map; and the target human body part parsing map is determined as the human body part parsing result.

[0190] The combined processing device for human body key point detection and part analysis provided in this embodiment can perform Figure 4The technical solution of the method embodiment shown in the figure has the same implementation principle and technical effect as Figure 4 The method embodiments shown are similar and will not be described in detail here.

[0191] Embodiment 5

[0192] Fig. 9 FIG. 1 is a schematic diagram of a joint processing device for detecting key points of a human body and analyzing the parts of a human body provided by another embodiment of the present invention. Fig. 9 As shown, the joint processing device 50 for human body key point detection and part analysis provided in this embodiment includes: an acquisition module 51, a training search module 52, a judgment module 53, and a determination module 54.

[0193] Among them, the acquisition module 51 is used to obtain training samples, and the training samples include the image to be trained, the real human body key point heat map corresponding to the image to be trained, and the real human body part analysis map. The training search module 52 is used to use the neural network automatic search technology to search for the optimal network structure of the initial network model in the process of training the initial network model using the training samples. The judgment module 53 is used to judge whether the trained network model meets the preset convergence conditions. The determination module 54 is used to determine the network model that meets the preset convergence conditions as the target network model trained to convergence if it is determined that the trained network model meets the preset convergence conditions.

[0194] The combined processing device for human body key point detection and part analysis provided in this embodiment can perform Figure 7 The technical solution of the method embodiment shown in the figure has the same implementation principle and technical effect as Figure 7 The method embodiments shown are similar and will not be described in detail here.

[0195] Optionally, the initial network model includes multiple initial network sub-models, at least one initial network sub-model includes a searchable conventional operation unit and / or a searchable dimensionality reduction operation unit, the searchable conventional operation unit includes multiple candidate operation spaces, and the searchable dimensionality reduction operation unit includes multiple candidate operation spaces.

[0196] Accordingly, the training search module 52 is specifically used for:

[0197] In the process of training the initial network model using training samples, a neural network automatic search technology is used to optimally search the candidate operation space of the corresponding searchable conventional operation unit and / or searchable dimensionality reduction operation unit in at least one initial network sub-model.

[0198] Optionally, the judgment module 53 is specifically used to: judge whether the value of the total loss function corresponding to the trained network model is less than a preset threshold or whether the search number reaches a preset search number; if it is determined that the value of the total loss function is less than a preset threshold or the search number reaches a preset search number, it is determined that the preset convergence condition is met.

[0199] Optionally, the determination module 54 is also used to: determine a first loss function based on a predicted human body key point heat map and a real human body key point heat map determined during the training of the initial network model; determine a second loss function based on a predicted human body part analysis map and a real human body part analysis map determined during the training of the initial network model; determine a total loss function based on the first loss function and the second loss function.

[0200] Embodiment 6

[0201] Fig.10 The first block diagram of an electronic device for implementing the joint processing method of human body key point detection and part analysis according to an embodiment of the present invention is shown in FIG. Fig.10 As shown, the electronic device 60 includes: at least one processor 62 and a memory 61.

[0202] The memory 61 stores computer-executable instructions. At least one processor 62 executes the computer-executable instructions stored in the memory, so that at least one processor executes the method of any one of the above-mentioned embodiments 1 to 3.

[0203] Embodiment 7

[0204] Fig.11 is a second block diagram of an electronic device for implementing the joint processing method for human body key point detection and part analysis according to an embodiment of the present invention, such as Fig.11 As shown, the electronic device can be a computer, a digital broadcast terminal, a message sending and receiving device, a tablet device, a personal digital assistant, a server, a server cluster, etc.

[0205] The electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0206] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0207] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0208] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0209] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0210] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0211] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0212] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0213] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0214] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0215] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of an electronic device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0216] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables a terminal device to execute the joint processing method of human body key point detection and part analysis of the above electronic device.

[0217] In an exemplary embodiment, a computer program product is further provided, including a computer program, where a processor executes the method in any one of the above-mentioned embodiments 1 to 3.

[0218] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0219] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A joint processing method for human body key point detection and part analysis, characterized in that: include: Get the target image; Using a target network model trained to convergence to perform joint processing of human key point detection and body part analysis on the target image to obtain human key point detection results and human body part analysis results, the target network model is obtained by searching the initial network model for the optimal network structure using a neural network automatic search technology during the process of training the initial network model using training samples; Outputting the human body key point detection results and human body part analysis results through the target network model; The target network model includes: a target interaction sub-network model; The method of using the target network model trained to convergence to perform joint processing of human key point detection and body part analysis on the target image to obtain human key point detection results and human body part analysis results includes: Using the target network model to perform human key point detection and body part analysis processing on the target image, so as to obtain a first human key point detection feature map and a first human body part analysis feature map respectively; Using the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human body part analysis feature map according to the determined optimal interaction mode, so as to obtain a second human key point detection feature map and a second human body part analysis feature map after interaction respectively; A human body key point detection result is determined according to the first human body key point detection feature map and the second human body key point detection feature map, and a human body part analysis result is determined according to the first human body part analysis feature map and the second human body part analysis feature map.

2. The method according to claim 1, characterized in that The target interaction sub-network model includes: two target searchable modules and two fusion modules, the target searchable module includes a plurality of first target conventional operation units, and the fusion module includes a plurality of second target conventional operation units; The method of using the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human body part analysis feature map according to the determined optimal interaction mode to obtain a second human key point detection feature map and a second human body part analysis feature map after interaction, respectively, includes: The first human key point detection feature map is operated with the first target conventional operation unit in the first target searchable module according to the corresponding target operation space to obtain a human key point detection feature map after operation, and the human key point detection feature map after operation is operated with the first human body part analysis feature map using the target operation space corresponding to the second target conventional operation unit in the first fusion module to obtain a second human key point detection feature map; The first human body part parsing feature map and the first target conventional operation unit in the second target searchable module are operated according to the corresponding target operation space to obtain the human body part parsing feature map after operation, and the human body part parsing feature map after operation and the first human body key point detection feature map are operated using the target operation space corresponding to the second target conventional operation unit in the second fusion module to obtain the second human body part parsing feature map.

3. The method according to claim 1, characterized in that The target network model also includes: a target backbone sub-network model, a target key point detection sub-network model, and a target part analysis sub-network model; The method of using the target network model to perform human key point detection and part analysis processing on the target image to obtain a first human key point detection feature map and a first human part analysis feature map respectively includes: Using the target backbone sub-network model to extract common features from the target image to obtain a plurality of common feature maps, each of which has a different resolution; Using the target key point detection sub-network model to extract human key point features from a plurality of shared feature maps to obtain a first human key point detection feature map; The target part parsing sub-network model is used to extract human body part features from a plurality of common feature maps to obtain a first human body part parsing feature map.

4. The method according to claim 3, characterized in that The target backbone sub-network model includes: a feature preservation module and multiple feature dimension reduction modules, the feature preservation module includes multiple third target conventional operation units, and the feature dimension reduction module includes: a first target dimension reduction operation unit and multiple fourth target conventional operation units; The using the target backbone sub-network model to extract common features from the target image to obtain a plurality of common feature maps includes: Arranging the feature preserving module and multiple feature dimension reduction modules in sequence; The target image and the third target conventional operation unit in the feature preservation module are operated according to the corresponding target operation space to obtain a first common feature map; Determine the first common feature map as the previous common feature map corresponding to the current feature dimension reduction module; For each current feature dimensionality reduction module, the previous common feature map is operated with the corresponding target operation space in the first target dimensionality reduction operation unit and the fourth target regular operation unit in the current feature dimensionality reduction module to obtain the previous common feature map corresponding to the next feature dimensionality reduction module.

5. The method according to claim 3, characterized in that: The target key point detection sub-network model includes: a third fusion module and a posture refinement module, and the posture refinement module includes a plurality of fifth target conventional operation units; The method of extracting human key point features from a plurality of shared feature maps using the target key point detection sub-network model to obtain a first human key point detection feature map includes: A bilinear interpolation algorithm is used to upsample the low-resolution common feature maps of two adjacent common feature maps, so that the upsampled common feature maps have the same resolution as the adjacent common feature maps; The third fusion module is used to gradually fuse the up-sampled common feature map with the adjacent common feature maps of the same resolution to obtain a fused common feature map; The fused common feature map and the fifth target conventional operation unit in the posture refinement module are operated according to the corresponding target operation space to obtain a first human key point detection feature map.

6. The method according to claim 3, characterized in that The target part parsing sub-network model includes: a searchable dilated convolution pooling pyramid unit and a part parsing refinement module; the searchable dilated convolution pooling pyramid unit includes: a sixth target conventional operation unit and a second target dimensionality reduction operation unit, and the part parsing refinement module includes: a plurality of seventh target conventional operation units; The step of extracting human body part features from a plurality of shared feature maps using the target part parsing sub-network model to obtain a first human body part parsing feature map includes: A searchable atrous convolution pooling pyramid unit is used to perform operations on multiple common feature maps according to the corresponding target operation space in the sixth target conventional operation unit and the second target dimension reduction operation unit to obtain a human body part analytical feature map after operation; The calculated human body part analysis feature map and the seventh target conventional operation unit in the part analysis refinement module are operated according to the corresponding target operation space to obtain a first human body part analysis feature map.

7. The method according to claim 1, characterized in that The target network model also includes: a convolutional layer; The determining of the human body key point detection result according to the first human body key point detection feature map and the second human body key point detection feature map comprises: Connecting the first human key point detection feature map and the second human key point detection feature map in a channel dimension to obtain a key point detection feature map after the connection in the channel dimension; Using the convolution kernel in the convolution layer, a convolution operation is performed on the key point detection feature map after the channel dimension connection to obtain a target human key point heat map; The target human body key point heat map is determined as the human body key point detection result.

8. The method according to claim 1, characterized in that The target network model also includes: a deconvolution layer; The determining of the human body part analysis result according to the first human body part analysis feature map and the second human body part analysis feature map comprises: Connecting the first human body part analysis feature map and the second human body part analysis feature map in channel dimension to obtain a part analysis feature map after connection in channel dimension; A deconvolution layer is used to upsample the part analysis feature map after the channel dimension connection to obtain a target human body part analysis map; The target human body part analysis diagram is determined as a human body part analysis result.

9. A joint processing method for human body key point detection and part analysis, characterized in that: include: Acquire a training sample, wherein the training sample includes an image to be trained, a real human body key point heat map corresponding to the image to be trained, and a real human body part analysis map; In the process of training the initial network model using the training samples, the neural network automatic search technology is used to search for the optimal network structure of the initial network model; Determine whether the trained network model meets the preset convergence conditions; If it is determined that the trained network model meets the preset convergence condition, the network model that meets the preset convergence condition is determined as the target network model trained to convergence, so as to use the target network model to perform human key point detection and part analysis processing on the target image respectively, so as to obtain a first human key point detection feature map and a first human part analysis feature map respectively, and use the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human part analysis feature map according to the determined optimal interaction method, so as to obtain a second human key point detection feature map and a second human part analysis feature map after interaction respectively, determine the human key point detection result according to the first human key point detection feature map and the second human key point detection feature map, and determine the human part analysis result according to the first human part analysis feature map and the second human part analysis feature map, the target network model is obtained by searching the optimal network structure of the initial network model using a neural network automatic search technology in the process of training the initial network model using a training sample, and the human key point detection result and the human part analysis result are output through the target network model.

10. The method according to claim 9, characterized in that The initial network model includes a plurality of initial network sub-models, at least one of the initial network sub-models includes a searchable conventional operation unit and / or a searchable dimensionality reduction operation unit, the searchable conventional operation unit includes a plurality of candidate operation spaces, and the searchable dimensionality reduction operation unit includes a plurality of candidate operation spaces; In the process of training the initial network model using the training samples, the neural network automatic search technology is used to search for the optimal network structure of the initial network model, including: In the process of training the initial network model using the training samples, a neural network automatic search technology is used to optimally search the candidate operation space of the corresponding searchable conventional operation units and / or searchable dimensionality reduction operation units in the at least one initial network sub-model.

11. The method according to claim 9, characterized in that The step of determining whether the trained network model meets a preset convergence condition includes: Determine whether the value of the total loss function corresponding to the trained network model is less than a preset threshold or whether the number of searches reaches a preset number of searches; If it is determined that the value of the total loss function is less than a preset threshold or the number of searches reaches a preset number of searches, it is determined that the preset convergence condition is met.

12. The method according to claim 11, characterized in that Before determining whether the value of the total loss function corresponding to the trained network model is less than a preset threshold, the method further includes: Determine a first loss function based on a predicted human body key point heat map and a real human body key point heat map determined during the training of the initial network model; Determine a second loss function based on the predicted human body part parsing graph and the real human body part parsing graph determined during the training of the initial network model; The total loss function is determined according to the first loss function and the second loss function.

13. A joint processing device for human body key point detection and part analysis, characterized in that: include: An acquisition module, used for acquiring a target image; A processing module, used for performing joint processing of human key point detection and part analysis on the target image using a target network model trained to convergence, so as to obtain human key point detection results and human part analysis results, wherein the target network model is obtained by searching for the optimal network structure of the initial network model using a neural network automatic search technology during the process of training the initial network model using training samples; An output module, used to output human key point detection results and human body part analysis results through the target network model; The target network model includes: a target interaction sub-network model; The processing module is specifically used for: Using the target network model to perform human key point detection and body part analysis processing on the target image, so as to obtain a first human key point detection feature map and a first human body part analysis feature map respectively; Using the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human body part analysis feature map according to the determined optimal interaction mode, so as to obtain a second human key point detection feature map and a second human body part analysis feature map after interaction respectively; A human body key point detection result is determined according to the first human body key point detection feature map and the second human body key point detection feature map, and a human body part analysis result is determined according to the first human body part analysis feature map and the second human body part analysis feature map.

14. A joint processing device for key point detection and part analysis of human body, characterized in that: include: An acquisition module is used to acquire training samples, wherein the training samples include an image to be trained, a real human body key point heat map and a real human body part analysis map corresponding to the image to be trained; A training search module, used for searching the optimal network structure of the initial network model by using the neural network automatic search technology in the process of training the initial network model by using the training samples; A judgment module is used to judge whether the trained network model meets the preset convergence conditions; A determination module, for determining the network model that satisfies the preset convergence condition as a target network model trained to convergence if it is determined that the trained network model satisfies the preset convergence condition, so as to use the target network model to perform human key point detection and part analysis processing on the target image respectively, so as to obtain a first human key point detection feature map and a first human part analysis feature map respectively, and use the target interaction sub-network model to perform feature interaction on the first human key point detection feature map and the first human part analysis feature map according to the determined optimal interaction mode, so as to obtain a second human key point detection feature map and a second human part analysis feature map after interaction respectively, determine the human key point detection result according to the first human key point detection feature map and the second human key point detection feature map, and determine the human part analysis result according to the first human part analysis feature map and the second human part analysis feature map, the target network model is obtained by searching the optimal network structure of the initial network model using a neural network automatic search technology in the process of training the initial network model using a training sample, and outputs the human key point detection result and the human part analysis result through the target network model.

15. An electronic device, characterized in that: include: at least one processor, memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 8 or any one of claims 9 to 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 or any one of claims 9 to 12 when executed by a processor.

Images