Human body image segmentation method, human body image segmentation model training method and device

Abstract

The present invention provides a human body image segmentation method, a training method and a device for a human body image segmentation model; wherein the method comprises: obtaining an image to be processed containing a target human body; inputting the image to be processed into a pre-trained human body image segmentation model, and outputting an initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information comprises detection results of preset key points of the target human body and / or edge detection results of various parts; adjusting the initial segmentation result according to the auxiliary information, and obtaining a final segmentation result of the target human body in the image to be processed. In this method, the human body image segmentation model not only outputs the initial segmentation result of the target human body, but also outputs the auxiliary information, and the initial segmentation result can be adjusted based on the auxiliary information, thereby improving the accuracy of the segmentation result and improving the overall segmentation effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a human body image segmentation method, a human body image segmentation model training method and a device. Background Art

[0002] Human image segmentation refers to the automatic identification and location of key parts of the human body from an image, such as hair, face, arms, legs, feet and other key parts. Human image segmentation can be applied to a variety of fields such as virtual reality, video surveillance, and behavior analysis. In related technologies, human image segmentation algorithms based on deep learning are mostly implemented using ordinary image semantic segmentation networks; however, compared with other image segmentation tasks, there are more unfavorable factors that may affect the effect of human image segmentation, such as the human body being covered by various types of clothing, and the human body having many postures and large deformations, which lead to poor overall effect of human image segmentation. Summary of the invention

[0003] In view of this, the object of the present invention is to provide a human body image segmentation method, a training method and a device for a human body image segmentation model, so as to improve the accuracy of the segmentation result and improve the overall effect of the segmentation.

[0004] In a first aspect, an embodiment of the present invention provides a human body image segmentation method, the method comprising: obtaining an image to be processed containing a target human body; inputting the image to be processed into a pre-trained human body image segmentation model, and outputting an initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes detection results of preset key points of the target human body and / or edge detection results of various parts; adjusting the initial segmentation result according to the auxiliary information to obtain a final segmentation result of the target human body in the image to be processed.

[0005] The above-mentioned human image segmentation model includes: a feature extraction network, a human image segmentation network, a key point detection network and an edge detection network; wherein the feature extraction network is used to: extract feature information of the image to be processed based on the image to be processed; the human image segmentation network is used to: output the initial segmentation result of the target human body based on the feature information; the key point detection network is used to: output the detection result of the preset key point based on the feature information of the image to be processed; the edge detection network is used to: output the edge detection result of each part based on the feature information.

[0006] The above-mentioned initial segmentation result includes: the part category of each part of the segmented target human body; the auxiliary information includes the detection results of preset key points; the above-mentioned step of adjusting the initial segmentation result according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed includes: according to the detection results of the preset key points, detecting whether the part category of each part of the target human body in the initial segmentation result is correct; if there is a target part with an incorrect part category, determining the part category of the target part according to the detection results of the preset key points.

[0007] The above-mentioned step of detecting whether the part category of each part of the target human body in the initial segmentation result is correct based on the detection results of the preset key points includes: determining the relative position relationship of each part of the target human body based on the detection results of the preset key points; based on the relative position relationship, detecting whether the part category of each part of the target human body in the initial segmentation result is correct.

[0008] The above-mentioned initial segmentation result includes: the edge position of each part of the segmented target human body; the auxiliary information includes the edge detection results of each part of the target human body; the above-mentioned step of adjusting the initial segmentation result according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed includes: adjusting the edge position of each part according to the edge detection result, and determining the adjusted edge position of each part as the final segmentation result of the target human body in the image to be processed.

[0009] In second aspect, an embodiment of the present invention provides a training method for a human image segmentation model, the method comprising: obtaining a training sample set; wherein the sample images in the training sample set carry part segmentation labels and auxiliary information labels; the auxiliary information labels include position labels of preset key points of the human body and / or position labels of part edges; and training a preset initial model based on the training sample set to obtain a human image segmentation model.

[0010] The above-mentioned auxiliary information labels include position labels of preset key points of the human body and position labels of part edges; the above-mentioned step of training the preset initial model based on the training sample set to obtain a human image segmentation model includes: determining a sample image from the training sample set; inputting the sample image into the initial model, and outputting the initial segmentation result of the human body in the sample image, the detection result of preset key points and the detection result of part edges; determining a first loss value based on the initial segmentation result and the human body part segmentation label; determining a second loss value based on the detection result of the preset key points and the position labels of the preset key points; determining a third loss value based on the part edge detection result and the position labels of the part edges; updating the weight parameters of the initial model based on the first loss value, the second loss value and the third loss value; continuing to execute the step of determining the sample image from the training sample set until the initial model converges to obtain a human image segmentation model.

[0011] In a third aspect, an embodiment of the present invention provides a human body image segmentation device, the device comprising: an image acquisition module, used to acquire an image to be processed containing a target human body; an output module, used to input the image to be processed into a pre-trained human body image segmentation model, and output an initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes detection results of preset key points of the target human body and / or edge detection results of various parts; an adjustment module, used to adjust the initial segmentation result according to the auxiliary information, and obtain the final segmentation result of the target human body in the image to be processed.

[0012] In a fourth aspect, an embodiment of the present invention provides a training device for a human image segmentation model, the device comprising: a sample set acquisition module, used to acquire a training sample set; wherein the sample images in the training sample set carry part segmentation labels and auxiliary information labels; the auxiliary information labels include position labels of preset key points of the human body and / or position labels of part edges; a training module, used to train a preset initial model based on the training sample set to obtain a human image segmentation model.

[0013] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned human image segmentation method, or the above-mentioned human image segmentation model training method.

[0014] In a sixth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned human image segmentation method, or the above-mentioned human image segmentation model training method.

[0015] The embodiments of the present invention bring the following beneficial effects:

[0016] In the above-mentioned human image segmentation method, human image segmentation model training method, device and electronic device, an image to be processed containing a target human body is input into a pre-trained human image segmentation model, and an initial segmentation result and auxiliary information of the target human body are output; wherein the auxiliary information includes the detection results of preset key points of the target human body and / or the edge detection results of each part; the initial segmentation result is adjusted according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed. In this method, the human image segmentation model not only outputs the initial segmentation result of the target human body, but also outputs the auxiliary information, and the initial segmentation result can be adjusted based on the auxiliary information, thereby improving the accuracy of the segmentation result and improving the overall segmentation effect.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of a human body image segmentation method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the model structure of a human body image segmentation model provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the model structure of another human image segmentation model provided by an embodiment of the present invention;

[0023] Figure 4 A flowchart of a method for training a human image segmentation model provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the structure of a human body image segmentation device provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the structure of a training device for a human image segmentation model provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0028] In the related technology, there are two main categories of human image segmentation algorithms. One category is the traditional human image segmentation algorithm. The traditional human image segmentation algorithm usually uses artificially designed rules to extract human features. The extracted human features can be, for example, bag-of-words features, color features, etc., and then trains the human image segmentation model, inputs the human features into the trained human image segmentation model, and obtains the initial results of human image segmentation; finally, the initial results are post-processed using probability graph models such as conditional random fields, and finally a smoother and more accurate result is obtained. The traditional human image segmentation algorithm lacks high-level semantic information of human images and also includes complex post-processing processes. The entire algorithm process is relatively cumbersome and cannot be optimized end-to-end. Therefore, it is generally less accurate and slower.

[0029] The other type is the human image segmentation algorithm based on deep learning. Deep learning can automatically learn features suitable for human image segmentation directly from the original human image without manual intervention. Since human image segmentation can be regarded as a special case of image semantic segmentation, the usual practice is to use a mature semantic segmentation network, input the human image into the network, and directly output the final segmentation result. However, this method of using an ordinary image semantic segmentation network does not take into account the particularity of the human image segmentation task. To a certain extent, human image segmentation can be regarded as a more fine-grained semantic segmentation task. The boundaries of different parts of the human body are more difficult to identify under complex clothing conditions. At the same time, compared with other objects, the deformation of the human body is larger, which greatly increases the difficulty of human body part recognition, resulting in poor overall effect of human image segmentation.

[0030] Based on the above, the embodiments of the present invention provide a human image segmentation method, a human image segmentation model training method, a device and an electronic device, which can be applied to various image segmentation scenarios such as human image segmentation and animal segmentation. To facilitate understanding of this embodiment, a human image segmentation method disclosed in an embodiment of the present invention is first introduced in detail.

[0031] like Figure 1 As shown, the method comprises the following steps:

[0032] Step S102, obtaining an image to be processed containing a target human body;

[0033] The image to be processed may contain one or more target human bodies.

[0034] Step S104, inputting the image to be processed into a pre-trained human image segmentation model, and outputting the initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes the detection results of preset key points of the target human body and / or the edge detection results of various parts;

[0035] After the image to be processed is input into the human image segmentation model, the human image segmentation model outputs two contents, one of which is the initial segmentation result of the target human body; the initial segmentation result can specifically be the location area of ​​each part of the target human body after segmentation, and the name of each part; the initial segmentation model can be marked on the image to be processed. The other content is auxiliary information, which can simultaneously include the detection results of the preset key points of the target human body and the edge detection results of each part, or only include one of the two. Among them, the preset key points can be pre-set, and the preset key points can specifically be the position points such as the wrist, elbow, knee, etc.; detecting the positions of these key points of the target human body is conducive to knowing the posture of the target human body and the position of each part relative to the key points, so that the parts with incorrect names in the initial segmentation results can be corrected. For example, the part near the wrist key point is generally a hand or an arm. If a part near the wrist key point is identified as a foot, it means that the part is identified incorrectly and needs to be corrected.

[0036] In addition, the initial segmentation results include the location areas of various parts of the target human body, which may not accurately identify the location areas of various parts, for example, the location areas may be too large, too small or offset. The edge detection results of the above-mentioned various parts can more accurately indicate the edge positions of the various parts, so the edge detection results can make the location areas of the various parts in the initial segmentation results more accurate.

[0037] Step S106, adjusting the initial segmentation result according to the auxiliary information to obtain a final segmentation result of the target human body in the image to be processed.

[0038] The auxiliary information can be used to determine whether there are errors in the initial segmentation results; for example, when the auxiliary information includes the detection results of preset key points of the target human body, the auxiliary information can be used to determine whether the identification names of various parts in the initial segmentation results are correct. If there are parts that are misidentified, the misidentified parts can be corrected based on the auxiliary information. When the auxiliary information includes the edge detection results of various parts, the auxiliary information can be used to determine whether the location areas of various parts in the initial segmentation results are accurate. If they are inaccurate, the location areas of various parts can be adjusted based on the auxiliary information.

[0039] In the above-mentioned human image segmentation method, the image to be processed containing the target human body is input into a pre-trained human image segmentation model, and the initial segmentation result and auxiliary information of the target human body are output; wherein the auxiliary information includes the detection results of the preset key points of the target human body and / or the edge detection results of each part; the initial segmentation result is adjusted according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed. In this method, the human image segmentation model not only outputs the initial segmentation result of the target human body, but also outputs the auxiliary information, and the initial segmentation result can be adjusted based on the auxiliary information, thereby improving the accuracy of the segmentation result and improving the overall segmentation effect.

[0040] The following embodiments continue to describe the model structure of the human body image segmentation model. Figure 2 As shown, the human image segmentation model includes a feature extraction network, a human image segmentation network, a key point detection network and an edge detection network; the feature extraction network is connected to the human image segmentation network, the key point detection network and the edge detection network respectively; wherein the feature extraction network is used to: extract feature information of the image to be processed based on the image to be processed; the input of the feature extraction network is the image to be processed, and the output is the feature information of the image to be processed; the feature information is respectively input to the subsequent human image segmentation network, the key point detection network and the edge detection network; the feature information can also be called shared information or public features; the feature extraction network can be implemented by a combination of convolutional layers and activation functions.

[0041] The above-mentioned human body image segmentation network is used to: output the initial segmentation result of the target human body based on the feature information; the input of the human body image segmentation network is the feature information of the image to be processed; the human body image segmentation network can be implemented through a convolutional layer or a fully connected layer; the initial segmentation result output by the human body image segmentation network usually includes a vector of each pixel position in the image to be processed; the feature points in the vector are used to indicate the probability that the pixel position belongs to a certain part; based on the initial segmentation result, the target human body can be segmented.

[0042] In addition, the key point detection network is used to: output the detection results of preset key points based on the feature information of the image to be processed; the input of the key point detection network is the feature information of the image to be processed; the key point detection network can be implemented by a convolutional layer or a fully connected layer; the detection results of preset key points output by the key point detection network usually include the coordinates of each preset key point in the image to be processed. The edge detection network is used to: output the edge detection results of each part based on the feature information; the input of the edge detection network is the feature information of the image to be processed; the edge detection network can be implemented by a convolutional layer or a fully connected layer; the edge detection results of each part output by the edge detection network usually include a vector of each pixel position in the image to be processed, and the feature points in the vector are used to indicate whether the pixel position belongs to the edge of the part.

[0043] Figure 3 The model structure of another human image segmentation model is shown in the figure. Figure 2 The human image segmentation model shown in FIG. 1 is implemented on the basis of FIG. 1 . In this model, the feature extraction network includes four modules, namely Block1, Block2, Block3, and Block4, which are connected in sequence; each module can be composed of a convolutional layer and an activation function connected in sequence. In other implementations, the feature extraction network may also include more or fewer modules; the number of modules included in the feature extraction network can be determined according to the speed and accuracy requirements of the human image segmentation model. It can be understood that the more modules the feature extraction network includes, the deeper the network is, the better the performance is, but the slower the speed is.

[0044] The image to be processed can be represented as X∈R H×W×3 ; Where H represents the height of the image to be processed, W represents the width of the image to be processed, and 3 represents the RGB channels of the image to be processed; that is, the size of the image to be processed can be expressed as H×W×3. After the image to be processed is processed by the feature extraction network, the feature information of the image to be processed can be output, and the feature information can be specifically a feature map; the size of the feature map can be h×w×c; where h represents the height of the feature map, w represents the width of the feature map, and c represents the number of channels of the feature map, also known as the channel number.

[0045] The human image segmentation network includes a Block4-1 module, which can be implemented by a fully connected layer or a convolutional layer. The output of the human image segmentation network is the initial segmentation result, which can be expressed as It can be seen that the height and width of the initial segmentation result are the same as those of the image to be processed, and then each pixel position corresponds to a feature vector of length C. The value of C is related to the preset part type; for example, there are 10 types of human body parts, and the value of C is also 10; for a certain pixel position, in the feature vector of length C of the pixel position, each feature point corresponds to a part type, and the feature value of the feature point represents the probability that the pixel position belongs to the part type.

[0046] The above key point detection network includes a Block4-2 module, which can be implemented by a fully connected layer or a convolutional layer. The output of the key point detection network is the detection result of the preset key point, which can be expressed as It can be seen that the size of the detection result of the preset key point is N*2, which can be specifically expressed as a matrix of N rows and 2 columns, where each row includes the x-coordinate value and the y-coordinate value of a key point.

[0047] The above edge detection network includes the Block4-3 module, which can be implemented by a fully connected layer or a convolutional layer. The output of the edge detection network is the edge detection result of each part, which can be expressed as It can be seen that the height and width of the edge detection result are the same as the image to be processed, and then each pixel position corresponds to a feature vector with a length of 2. In this feature vector, the feature value of one feature point represents the probability that the current pixel position belongs to the edge of the part, and the feature value of another feature point represents the probability that the current pixel position does not belong to the edge of the part.

[0048] In the above-mentioned human body image segmentation method, the auxiliary information output by the human body image segmentation model includes the detection results of preset key points of the target human body and the edge detection results of each part, which can improve the accuracy of part type recognition in the segmentation results, and can also improve the accuracy of the position area of ​​part segmentation, thereby further improving the accuracy of the segmentation results and improving the overall effect of segmentation.

[0049] The following embodiments further describe the specific implementation method of adjusting the initial segmentation result based on auxiliary information. First, when the auxiliary information includes the detection results of preset key points, the auxiliary information can be used to adjust the part categories of the various parts of the target human body segmented in the initial segmentation result. It can be understood that the initial segmentation result includes the part categories of the various parts of the target human body segmented. When adjusting the initial segmentation result according to the auxiliary information, the part categories of the various parts of the target human body in the initial segmentation result can be detected based on the detection results of the preset key points. If there is a target part with an incorrect part category, the part category of the target part is determined based on the detection results of the preset key points.

[0050] The detection results of preset key points include the position coordinates of multiple key points. For a certain part, the part is usually near a certain key point, or between multiple key points. For example, the hand part is usually near the key point of the elbow, and if the hand part is near the key point of the knee in the initial segmentation result, it can usually be determined that the part category of the hand part is recognized incorrectly.

[0051] In specific implementation, the relative position relationship of each part of the target human body can be determined according to the detection results of the preset key points; then based on the relative position relationship, whether the part category of each part of the target human body in the initial segmentation result is correct can be detected. Among them, according to the position coordinates of each key point, the posture of the target human body can be determined, and then the relative position relationship of each part of the target human body can be determined based on the posture of the target human body, for example, the position relationship between the hand and the arm, the position relationship between the head and the shoulder, the position relationship between the foot and the leg, etc.; the position relationship can be that part A is located below, above, to the left of part B, etc., and the position relationship can also be expressed by an angle.

[0052] After the relative position relationship of each part is determined based on the key points, check whether the relative position relationship of the part categories of each part of the target human body in the initial segmentation result is the same as the relative position relationship of the parts determined based on the key points; if the relative position relationship between a part and its surrounding parts is different from the relative position relationship between the part and its surrounding parts determined based on the key points, it means that the part category of the part is identified incorrectly.

[0053] When determining the part category of the target part based on the detection results of preset key points, the part category of the target part can be inferred based on the key points around the target part; continuing with the above example, the part near the key point knee usually includes the calf, thigh and foot; then, based on factors such as the distance between the target part and the key point and the shape of the target part itself, the part category of the target part is corrected to the foot.

[0054] Through the key point detection mentioned above, the human body image segmentation model can extract the overall information of each part of the human body, thereby improving the accuracy of classification of each part and reducing the probability of misclassification (such as predicting a leg as an arm).

[0055] In addition, when the auxiliary information includes edge detection results of various parts of the target human body, the auxiliary information can be used to adjust the edge positions of the various parts of the target human body segmented in the initial segmentation result; that is, the initial segmentation result includes the edge positions of the various parts of the target human body segmented. When the initial segmentation result is adjusted according to the auxiliary information, the edge positions of the various parts can be adjusted according to the edge detection results, and the adjusted edge positions of the various parts are determined as the final segmentation result of the target human body in the image to be processed.

[0056] In actual implementation, for each part, the edge position of the part in the edge detection result can be set as the final edge position of the part, and the final edge position of each part and the part category of each part can be determined as the final segmentation result of the target human body. In another implementation, the final edge position of the part can be determined based on the edge position of the part in the edge detection result and the edge position of the part in the initial detection result; for example, the edge position of the part in the edge detection result and the edge position of the part in the initial detection result can be weighted averaged to obtain the final edge position of the part.

[0057] Through the above-mentioned edge detection, the human body image segmentation model can more accurately identify the boundaries of different parts of the human body.

[0058] In the above method, the part type of each part in the initial segmentation result is adjusted by the detection results of the preset key points in the auxiliary information, thereby improving the accuracy of part type recognition in the segmentation result; the position area of ​​each part in the initial segmentation result is adjusted by the edge detection results of each part in the auxiliary information, thereby improving the accuracy of the position area of ​​the part segmentation, which can further improve the accuracy of the segmentation result and improve the overall effect of segmentation.

[0059] The following embodiments further describe the training method of the human body image segmentation model involved in the above embodiments. Figure 4 As shown, the method comprises the following steps:

[0060] Step S402, obtaining a training sample set; wherein the sample images in the training sample set carry part segmentation labels and auxiliary information labels; the auxiliary information labels include position labels of preset key points of the human body and / or position labels of part edges;

[0061] Step S404: training the preset initial model based on the training sample set to obtain a human body image segmentation model.

[0062] The human image segmentation model trained in the above manner can output the initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes the detection results of the preset key points of the target human body and / or the edge detection results of each part; and then the initial segmentation result can be adjusted according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed. In this method, the human image segmentation model not only outputs the initial segmentation result of the target human body, but also outputs the auxiliary information, based on which the initial segmentation result can be adjusted, thereby improving the accuracy of the segmentation result and improving the overall segmentation effect.

[0063] In order to further improve the accuracy of the segmentation results, the auxiliary information labels carried by the sample images also include the position labels of the preset key points of the human body and the position labels of the edges of the parts. When training the preset initial model based on the training sample set, it can be achieved through the following steps:

[0064] Step 502, determining a sample image from a training sample set;

[0065] Step 504, inputting the sample image into the initial model, and outputting the initial segmentation result of the human body in the sample image, the detection result of the preset key points and the part edge detection result;

[0066] The model structure of the initial model can refer to the model structure of the human body image segmentation model provided in the above embodiment; the initial model includes a feature extraction network, a human body image segmentation network, a key point detection network and an edge detection network; wherein the human body image segmentation network outputs the initial segmentation result of the human body in the sample image; the key point detection network outputs the detection result of the preset key points of the human body in the sample image; the edge detection network outputs the edge detection result of the human body parts in the sample image.

[0067] Step 506, determining a first loss value according to the initial segmentation result and the human body part segmentation label; determining a second loss value according to the detection result of the preset key point and the position label of the preset key point; determining a third loss value according to the part edge detection result and the position label of the part edge;

[0068] The above initial segmentation result can be expressed as The height of the initial segmentation result is H, the width is W, and the number of channels is C; the height of the initial segmentation result is the same as the height of the sample image, and the width of the initial segmentation result is the same as the width of the sample image; for each pixel position (i, j) in the sample image, the vector of the pixel position in the initial segmentation result is This vector is input into the softmax function to obtain the probability vector The formula for each eigenvalue in the probability vector is as follows:

[0069]

[0070] in, Representation vector The kth element of .

[0071] According to the human body part segmentation label Y∈R H×W , and the above probability vector The loss value of the segmentation can be calculated, that is, the first loss value L mentioned above segmentation :

[0072]

[0073] in, Representation vector Y i,j elements.

[0074] The detection results of the above preset key points can be expressed as Where N represents the number of key points, 2 means that each key point corresponds to two data, namely the x-coordinate value and y-coordinate value of the key point; the position label of the preset key point can be expressed as P∈R N×2 ; Based on this, the loss function of key point detection can be calculated, that is, the second loss value mentioned above:

[0075]

[0076] Among them, ||M|| 1 Represents the sum of the absolute values ​​of the elements of the matrix M.

[0077] The position labels of the above key points can be used for training the auxiliary task of key point detection. Through key point detection, the initial model can learn the overall information of each part, thereby improving the accuracy of classification of each part and reducing the probability of misclassification (such as predicting a leg as an arm).

[0078] The above part edge detection results can be expressed as Among them, the height of the part edge detection result is H, the width is W, and the number of channels is 2; the height of the part edge detection result is the same as the height of the sample image, and the width of the part edge detection result is the same as the width of the sample image; for each pixel position (i, j) in the sample image, the vector of the pixel position in the part edge detection result is The vector includes two data, one representing the probability that the pixel position belongs to the edge of the part, and the other representing the probability that the pixel position does not belong to the edge of the part. Input to the softmax function to get the probability vector The formula for each eigenvalue in the probability vector is as follows:

[0079]

[0080] in, Representation vector The kth element of .

[0081] According to the above probability vector As well as the position label of the part edge, the loss value of edge detection can be determined, that is, the third loss value L edge :

[0082]

[0083] in, Representation vector E i,j elements.

[0084] The position labels of the edges of the above parts are used for training the auxiliary task of edge detection. Through edge detection, the initial model can more accurately identify the boundaries of different parts.

[0085] Step 508, based on the first loss value, the second loss value and the third loss value, update the weight parameters of the initial model; continue to execute the step of determining the sample image from the training sample set until the initial model converges to obtain the human image segmentation model.

[0086] Based on the first loss value, the second loss value and the third loss value, the overall loss value of the initial model can be obtained. Specifically, the overall loss value is equal to the sum of the first loss value, the second loss value and the third loss value, that is, the overall loss value L = L edge +L segmentation +L point Then, the back propagation algorithm is used to find the derivative of the overall loss function L with respect to all parameters W in the initial model. Next, update the parameters of the initial model through the stochastic gradient descent algorithm: The parameters of the initial model are updated iteratively until convergence. α is the learning rate, which is a manually pre-set hyperparameter with common values ​​of 0.01, 0.001, etc.

[0087] After the initial model training is completed, the trained initial model can also be tested based on the test set. During the test, only the human image segmentation results output by the initial model can be obtained. For each pixel position i∈[1,…,H],j∈[1,…,W], the pixel position includes the probability corresponding to the part category in C, and the part category corresponding to the maximum value is determined as the part category of the pixel position; then, based on the determined part category and category label, the test result of the initial model is obtained. If the test result indicates that the segmentation accuracy of the initial model is low, the initial model can be trained by continuing to use the training sample set in the above embodiment until the segmentation accuracy of the initial model indicated by the test result reaches a preset accuracy threshold.

[0088] The above human image segmentation model, also known as a multi-task convolutional neural network, can handle three tasks at the same time: human image segmentation, edge detection, and key point positioning. Through the auxiliary task of edge detection, the model can pay more attention to the boundaries of different parts of the human body and improve the positioning accuracy of the boundaries of different parts; through key point detection, the model can learn the overall information of each part, thereby improving the accuracy of classification of each part and reducing the probability of misclassification.

[0089] When the above human image segmentation model is applied to human image segmentation, compared with the existing traditional human image segmentation algorithm, the above human image segmentation model is a human image segmentation algorithm based on deep learning, which can automatically learn multi-level semantic features related to human body parts, with high accuracy, and does not require complex result post-processing, and has a fast speed. Compared with the existing human image segmentation algorithm based on deep learning, the above human image segmentation model introduces the idea of ​​multi-task learning to further improve the accuracy of human image segmentation. Through the auxiliary task of edge detection, the model can pay more attention to the boundaries of different parts of the human body and improve the positioning accuracy of the boundaries of different parts of the human body; through the detection of key points of the human body, the network can learn the overall information of various parts of the human body, thereby improving the accuracy of classification of various parts of the human body and reducing the probability of misclassification.

[0090] Corresponding to the above method embodiment, see Figure 5 A schematic structural diagram of a human body image segmentation device is shown, the device comprising:

[0091] An image acquisition module 50 is used to acquire an image to be processed containing a target human body;

[0092] The output module 52 is used to input the image to be processed into the pre-trained human image segmentation model, and output the initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes the detection results of the preset key points of the target human body and / or the edge detection results of each part;

[0093] The adjustment module 54 is used to adjust the initial segmentation result according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed.

[0094] In the above-mentioned human image segmentation device, the image to be processed containing the target human body is input into the pre-trained human image segmentation model, and the initial segmentation result and auxiliary information of the target human body are output; wherein the auxiliary information includes the detection results of the preset key points of the target human body and / or the edge detection results of each part; the initial segmentation result is adjusted according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed. In this method, the human image segmentation model not only outputs the initial segmentation result of the target human body, but also outputs the auxiliary information, and the initial segmentation result can be adjusted based on the auxiliary information, thereby improving the accuracy of the segmentation result and improving the overall segmentation effect.

[0095] Furthermore, the above-mentioned human image segmentation model includes: a feature extraction network, a human image segmentation network, a key point detection network and an edge detection network; wherein the feature extraction network is used to: extract feature information of the image to be processed based on the image to be processed; the human image segmentation network is used to: output the initial segmentation result of the target human body based on the feature information; the key point detection network is used to: output the detection result of the preset key point based on the feature information of the image to be processed; the edge detection network is used to: output the edge detection result of each part based on the feature information.

[0096] The above-mentioned initial segmentation results include: the part categories of each part of the segmented target human body; the auxiliary information includes the detection results of preset key points; the above-mentioned adjustment module is also used to: detect whether the part categories of each part of the target human body in the initial segmentation results are correct based on the detection results of preset key points; if there are target parts with incorrect part categories, determine the part category of the target part based on the detection results of preset key points.

[0097] The adjustment module is also used to: determine the relative position relationship of each part of the target human body according to the detection results of the preset key points; based on the relative position relationship, detect whether the part category of each part of the target human body in the initial segmentation result is correct.

[0098] The above-mentioned initial segmentation result includes: the edge position of each part of the segmented target human body; the auxiliary information includes the edge detection result of each part of the target human body; the above-mentioned adjustment module is also used to: adjust the edge position of each part according to the edge detection result, and determine the adjusted edge position of each part as the final segmentation result of the target human body in the image to be processed.

[0099] See also Figure 6 A structural schematic diagram of a training device for a human image segmentation model is shown, the device comprising:

[0100] The sample set acquisition module 60 is used to acquire a training sample set; wherein the sample images in the training sample set carry part segmentation labels and auxiliary information labels; the auxiliary information labels include position labels of preset key points of the human body and / or position labels of part edges;

[0101] The training module 61 is used to train a preset initial model based on a training sample set to obtain a human image segmentation model.

[0102] The human image segmentation model trained by the above device can output the initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes the detection results of the preset key points of the target human body and / or the edge detection results of each part; and then the initial segmentation result can be adjusted according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed. In this way, the human image segmentation model not only outputs the initial segmentation result of the target human body, but also outputs the auxiliary information. The initial segmentation result can be adjusted based on the auxiliary information, thereby improving the accuracy of the segmentation result and improving the overall segmentation effect.

[0103] The above-mentioned auxiliary information labels include position labels of preset key points of the human body and position labels of part edges; the above-mentioned training module is also used to: determine a sample image from a training sample set; input the sample image into the initial model, and output the initial segmentation result of the human body in the sample image, the detection result of preset key points and the detection result of part edges; determine a first loss value based on the initial segmentation result and the human body part segmentation label; determine a second loss value based on the detection result of the preset key points and the position labels of the preset key points; determine a third loss value based on the part edge detection result and the position labels of the part edges; update the weight parameters of the initial model based on the first loss value, the second loss value and the third loss value; continue to execute the step of determining the sample image from the training sample set until the initial model converges to obtain a human image segmentation model.

[0104] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned human image segmentation method, or the above-mentioned human image segmentation model training method.

[0105] See also Figure 7 As shown, the electronic device includes a processor 100 and a memory 101, wherein the memory 101 stores machine executable instructions that can be executed by the processor 100, and the processor 100 executes the machine executable instructions to implement the above-mentioned human image segmentation method, or the above-mentioned human image segmentation model training method.

[0106] Further, Figure 7The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0107] Among them, the memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 102 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0108] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 100. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method of the above embodiment in combination with its hardware.

[0109] This embodiment also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned human image segmentation method, or the above-mentioned human image segmentation model training method.

[0110] The computer program products of the human image segmentation method, human image segmentation model training method, device and electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0112] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0113] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0114] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A human body image segmentation method, characterized in that: The method comprises: Acquire an image to be processed containing a target human body; Input the image to be processed into a pre-trained human body image segmentation model, and output the initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes the detection results of preset key points of the target human body and / or the edge detection results of various parts; Adjusting the initial segmentation result according to the auxiliary information to obtain a final segmentation result of the target human body in the image to be processed; The step of adjusting the initial segmentation result according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed includes: judging whether there is an error in the initial segmentation result according to the auxiliary information; if there is an error, adjusting the initial segmentation result according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed.

2. The method according to claim 1, characterized in that The human image segmentation model includes: a feature extraction network, a human image segmentation network, a key point detection network and an edge detection network; Wherein, the feature extraction network is used to: extract feature information of the image to be processed based on the image to be processed; the human body image segmentation network is used to: output the initial segmentation result of the target human body based on the feature information; The key point detection network is used to output the detection results of the preset key points based on the feature information of the image to be processed; the edge detection network is used to output the edge detection results of each part based on the feature information.

3. The method according to claim 1, characterized in that The initial segmentation result includes: the part category of each part of the target human body segmented; the auxiliary information includes the detection result of the preset key point; The step of adjusting the initial segmentation result according to the auxiliary information to obtain a final segmentation result of the target human body in the image to be processed includes: According to the detection result of the preset key points, detecting whether the part category of each part of the target human body in the initial segmentation result is correct; If there is a target part with an incorrect part category, the part category of the target part is determined according to the detection result of the preset key point.

4. The method according to claim 3, characterized in that The step of detecting whether the part categories of the parts of the target human body in the initial segmentation result are correct according to the detection results of the preset key points comprises: Determining the relative position relationship of various parts of the target human body according to the detection results of the preset key points; Based on the relative position relationship, it is detected whether the part category of each part of the target human body in the initial segmentation result is correct.

5. The method according to claim 1, characterized in that The initial segmentation result includes: the edge positions of the segmented parts of the target human body; the auxiliary information includes the edge detection results of the parts of the target human body; The step of adjusting the initial segmentation result according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed includes: adjusting the edge position of each part according to the edge detection result, and determining the adjusted edge position of each part as the final segmentation result of the target human body in the image to be processed.

6. A training method for a human image segmentation model, characterized in that: The method comprises: Acquire a training sample set; wherein the sample images in the training sample set carry part segmentation labels and auxiliary information labels; the auxiliary information labels include position labels of preset key points of the human body and / or position labels of part edges; the auxiliary information is used to determine whether there is an error in the initial segmentation result of the image to be processed; if there is an error, adjust the initial segmentation result according to the auxiliary information to obtain a final segmentation result of the target human body in the image to be processed; The preset initial model is trained based on the training sample set to obtain a human body image segmentation model.

7. The method according to claim 6, characterized in that The auxiliary information tags include position tags of preset key points of the human body and position tags of the edges of the parts; The step of training a preset initial model based on the training sample set to obtain a human image segmentation model comprises: Determine a sample image from the training sample set; Inputting the sample image into the initial model, and outputting the initial segmentation result of the human body in the sample image, the detection result of the preset key points and the part edge detection result; Determine a first loss value according to the initial segmentation result and the human body part segmentation label; determine a second loss value according to the detection result of the preset key point and the position label of the preset key point; determine a third loss value according to the part edge detection result and the position label of the part edge; Based on the first loss value, the second loss value and the third loss value, the weight parameters of the initial model are updated; and the step of determining the sample image from the training sample set is continued until the initial model converges to obtain the human image segmentation model.

8. A human body image segmentation device, characterized in that: The device comprises: An image acquisition module is used to acquire an image to be processed containing a target human body; An output module, used to input the image to be processed into a pre-trained human image segmentation model, and output the initial segmentation result and auxiliary information of the target human body; wherein the auxiliary information includes the detection results of preset key points of the target human body and / or the edge detection results of various parts; An adjustment module, used for adjusting the initial segmentation result according to the auxiliary information to obtain a final segmentation result of the target human body in the image to be processed; The adjustment module is used to determine whether there is an error in the initial segmentation result according to the auxiliary information; if there is an error, adjust the initial segmentation result according to the auxiliary information to obtain a final segmentation result of the target human body in the image to be processed.

9. A training device for a human image segmentation model, characterized in that: The device comprises: A sample set acquisition module is used to acquire a training sample set; wherein the sample images in the training sample set carry part segmentation labels and auxiliary information labels; the auxiliary information labels include position labels of preset key points of the human body and / or position labels of part edges; the auxiliary information is used to determine whether there is an error in the initial segmentation result of the image to be processed; if there is an error, the initial segmentation result is adjusted according to the auxiliary information to obtain the final segmentation result of the target human body in the image to be processed; The training module is used to train the preset initial model based on the training sample set to obtain a human image segmentation model.

10. An electronic device, characterized in that: It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the human image segmentation method described in any one of claims 1 to 5, or the training method of the human image segmentation model described in claim 6 or 7.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to implement the human image segmentation method described in any one of claims 1 to 5, or the training method of the human image segmentation model described in claim 6 or 7.

Images