Data processing method and device, electronic equipment and storage medium
By adding human key point information as a loss function constraint during the training of the human portrait segmentation network model, the problem of inaccurate matting of detailed parts in the human portrait segmentation algorithm is solved, and higher matting accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2026-04-14
AI Technical Summary
Existing portrait segmentation algorithms are prone to missing or inaccurately extracting details such as hands and feet when cutting out images, resulting in insufficient accuracy.
When training a human image segmentation network model, human key point information is added as a constraint to the loss function. By learning the semantic association between key points, the network's attention weights to human key points are improved.
It achieves precise and detailed human portrait segmentation, improving the accuracy of image cutout.
Smart Images
Figure CN116664618B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology
[0002] Human face segmentation can identify the complete outline of the human body in an image, segment it from the background, and return a grayscale image and a foreground image of the human face. It can be applied to scenarios such as photo compositing. However, the accuracy of current human face segmentation algorithms still needs improvement. Summary of the Invention
[0003] In view of the above problems, this application proposes a data processing method, apparatus, electronic device, and storage medium to improve the above problems.
[0004] In a first aspect, embodiments of this application provide a data processing method, the method comprising: acquiring a human keypoint dataset, the human keypoint dataset including a human image, a mask image corresponding to the human image, and human keypoint images corresponding to the human image; inputting the human image and the mask image into a human image segmentation network model to be trained, thereby training the human image segmentation network model; acquiring a first loss parameter output by the human image segmentation network model to be trained, and using the model obtained when the first loss parameter converges as a first human image segmentation network model; inputting the human image, the mask image, and the human keypoint images into the first human image segmentation network model to train the first human image segmentation network model; acquiring a second loss parameter output by the first human image segmentation network model, and using the model obtained when the second loss parameter converges as a second human image segmentation network model, wherein the attention weight of the second human image segmentation network model for human keypoints is higher than the attention weight of the first human image segmentation network model for human keypoints.
[0005] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising: a dataset acquisition module, configured to acquire a human keypoint dataset, the human keypoint dataset including a human image, a mask image corresponding to the human image, and human keypoint images corresponding to the human image; a first training module, configured to input the human image and the mask image into a human image segmentation network model to be trained, for training the human image segmentation network model; a first processing module, configured to acquire a first loss parameter output by the human image segmentation network model to be trained, and use the model obtained when the first loss parameter converges as a first human image segmentation network model; a second training module, configured to input the human image, the mask image, and the human keypoint images into the first human image segmentation network model, for training the first human image segmentation network model; and a second processing module, configured to acquire a second loss parameter output by the first human image segmentation network model, and use the model obtained when the second loss parameter converges as a second human image segmentation network model, wherein the attention weight of the second human image segmentation network model for human keypoints is higher than the attention weight of the first human image segmentation network model for human keypoints.
[0006] Thirdly, this application provides an electronic device including one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method described in the first aspect above.
[0007] Fourthly, this application provides a computer-readable storage medium storing program code, wherein the method described in the first aspect is executed when the program code is run.
[0008] This application provides a data processing method, system, electronic device, and storage medium. The method involves acquiring a human keypoint dataset, which includes a human image, a mask image corresponding to the human image, and human keypoint images corresponding to the human image. The human image and the mask image are input into a human image segmentation network model to be trained. A first loss parameter is obtained from the output of the human image segmentation network model, and the model obtained when the first loss parameter converges is used as a first human image segmentation network model. The human image, the mask image, and the human keypoint images are then input into the first human image segmentation network model to train it. A second loss parameter is obtained from the output of the first human image segmentation network model, and the model obtained when the second loss parameter converges is used as a second human image segmentation network model. The attention weights of the human keypoints in the second human image segmentation network model are higher than those in the first human image segmentation network model. Thus, by using the above method, we can add human key point information as a loss function constraint when training the human image segmentation network model, thereby enabling the network to focus its attention on the key point information of the human body, learn the semantic relationships between key points, and achieve accurate and detailed segmentation of the human image, that is, improve the image matting accuracy of the human image segmentation algorithm. Attached Figure Description
[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 A flowchart of a data processing method according to an embodiment of this application is shown.
[0011] Figure 2 A flowchart of a data processing method according to another embodiment of this application is shown.
[0012] Figure 3 A flowchart of a data processing method according to another embodiment of this application is shown.
[0013] Figure 4 An example diagram of a human body image provided in an embodiment of this application is shown.
[0014] Figure 5 The embodiments of this application are shown. Figure 4 The image shown is an example of a mask image corresponding to a human body image.
[0015] Figure 6 The embodiments of this application are shown. Figure 4 The image shown is an example of a human body key point image corresponding to the human body image shown.
[0016] Figure 7 A flowchart of a data processing method according to another embodiment of this application is shown.
[0017] Figure 8 The embodiments of this application are shown. Figure 4 The human body image shown is a visualization of the key points of the human body.
[0018] Figure 9 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown.
[0019] Figure 10 A structural block diagram of an electronic device for performing a data processing method according to an embodiment of the present application is shown.
[0020] Figure 11 It is a storage unit in this application embodiment for storing or carrying program code that implements a data processing method according to this application embodiment. Detailed Implementation
[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0022] Semantic segmentation is a deep learning algorithm that associates labels or categories with each pixel of an image. It is used to identify sets of pixels that constitute distinguishable categories. For example, autonomous vehicles need to identify vehicles, pedestrians, traffic signals, crosswalks, and other road features. Compared to image classification or object detection, semantic segmentation gives us a more detailed understanding of images. This understanding is crucial in many fields such as autonomous driving, robotics, and image search engines. Portrait segmentation is a type of semantic segmentation. It can identify the complete outline of a human body in an image, segment it from the background, and return a grayscale image and a foreground portrait image, which can be applied to scenarios such as photo compositing. However, current portrait segmentation algorithms are prone to missing or inaccurately extracting details such as hands and feet, and the accuracy of image extraction needs further improvement.
[0023] To address the aforementioned issues, the inventors, through extensive research, proposed the data processing method, apparatus, electronic device, and storage medium provided in the embodiments of this application. This method can incorporate human key point information as a loss function constraint when training a human image segmentation network model, thereby directing network attention to the key point information of the human body, learning the semantic relationships between key points, and ultimately achieving accurate and detailed segmentation of the human image, thus improving the image matting accuracy of the human image segmentation algorithm.
[0024] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0025] Please see Figure 1 One embodiment of this application provides a data processing method that can be applied to electronic devices.
[0026] The method includes:
[0027] Step S110: Obtain a human body key point dataset, which includes a human body image, a mask image corresponding to the human body image, and a human body key point image corresponding to the human body image.
[0028] In this embodiment, the human body keypoint dataset consists of human body images, mask images corresponding to the human body images, and human body keypoint images corresponding to the human body images. The source of the human body images is not limited; they can be images downloaded from the internet, locally stored images, or images captured by electronic devices. There can be multiple human body images, each with different mask images and corresponding human body keypoint images. The pose of the human body in the images is not limited; for example, the human body can be standing, sitting, or lying down.
[0029] One approach is to pre-acquire and store a dataset of human key points when training a human image segmentation network model, so that the model can be trained on the dataset later.
[0030] Step S120: Input the human body image and the mask image into the human image segmentation network model to be trained, so as to train the human image segmentation network model.
[0031] In this embodiment, training the human image segmentation network model based on the human keypoint dataset can be divided into two stages. As one implementation, the first stage involves inputting the human image from the human keypoint dataset and the corresponding mask image into the human image segmentation network model to be trained.
[0032] The specific type of the human image segmentation network model to be trained is not limited.
[0033] Step S130: Obtain the first loss parameter output by the human image segmentation network model to be trained, and use the model obtained when the first loss parameter converges as the first human image segmentation network model.
[0034] In this embodiment of the application, the first loss parameter represents the loss parameter of the mask image when the human face segmentation network model to be trained is trained in the first stage. The first loss parameter can be L1 loss.
[0035] As one implementation method, the first loss parameters output by the portrait segmentation network model to be trained can be obtained, and the model obtained when the first loss parameters converge can be used as the first portrait segmentation network model.
[0036] Step S140: Input the human body image, the mask image, and the human body key point image into the first human image segmentation network model to train the first human image segmentation network model.
[0037] As one implementation method, in the second stage, the human body image, the mask image corresponding to the human body image, and the human body key point image corresponding to the human body image can be input into the first human body segmentation network model to further train the first human body segmentation network model.
[0038] Compared to the first stage, the second stage adds human (skeleton) key points as loss constraints when training the human portrait segmentation network model, which can help the first human portrait segmentation network model improve its ability to accurately segment detailed parts of the portrait.
[0039] Step S150: Obtain the second loss parameter output by the first portrait segmentation network model, and use the model obtained when the second loss parameter converges as the second portrait segmentation network model. The attention weight of the second portrait segmentation network model to the human body key points is higher than that of the first portrait segmentation network model to the human body key points.
[0040] In this embodiment, the second loss parameter represents the loss parameter of the mask image when training the first portrait segmentation network model in the second stage. The second loss parameter can be expressed as L1loss + loss_keypoint, where loss_keypoint represents the keypoint weighted loss parameter. The role of loss_keypoint is to assign higher attention weights to human body keypoints, allowing the network to learn the correlation of whole-body information. For example, seeing "shoulder" will lead to finding "elbow". Therefore, in this embodiment, the attention weight of the second portrait segmentation network model to human body keypoints is higher than that of the first portrait segmentation network model to human body keypoints.
[0041] As one implementation method, the second loss parameters output by the first portrait segmentation network model can be obtained, and the model obtained when the second loss parameters converge can be used as the second portrait segmentation network model.
[0042] This application provides a data processing method that involves acquiring a human keypoint dataset, which includes a human image, a mask image corresponding to the human image, and human keypoint images corresponding to the human image; inputting the human image and the mask image into a human image segmentation network model to train the human image segmentation network model; acquiring a first loss parameter output by the human image segmentation network model to train the first human image segmentation network model; acquiring a second loss parameter output by the first human image segmentation network model to train the first human image segmentation network model; acquiring a second loss parameter output by the first human image segmentation network model to train the second ... Thus, by using the above method, we can add human key point information as a loss function constraint when training the human image segmentation network model, thereby enabling the network to focus its attention on the key point information of the human body, learn the semantic relationships between key points, and achieve accurate and detailed segmentation of the human image, that is, improve the image matting accuracy of the human image segmentation algorithm.
[0043] Please see Figure 2 Another embodiment of this application provides a data processing method that can be applied to electronic devices, the method comprising:
[0044] Step S210: Obtain a human body key point dataset, which includes a human body image, a mask image corresponding to the human body image, and a human body key point image corresponding to the human body image.
[0045] The specific implementation of step S210 can be referred to the relevant description in step S110 of the foregoing embodiments, and will not be repeated here.
[0046] Step S220: Input the human body image and the mask image into the human image segmentation network model to be trained, so as to train the human image segmentation network model.
[0047] The specific implementation of step S220 can be referred to the relevant description in step S120 in the foregoing embodiments, and will not be repeated here.
[0048] Step S230: Obtain the first predicted human image output by the human image segmentation network model to be trained based on the human image and the mask image.
[0049] As one implementation method, during the first stage of training the human image segmentation network model, the human image segmentation network model can perform human image segmentation prediction based on the human image and the mask image corresponding to the human image, and then the predicted human image segmentation image can be used as the first predicted human image.
[0050] Step S240: Obtain the first loss parameter based on the first predicted human image and the mask image, and use the model obtained when the first loss parameter converges as the first human image segmentation network model.
[0051] In this embodiment, the mask image corresponding to the human image can be understood as a ground truth mask or a ground truth (labels in the sample set, which can be manually labeled) and is used to supervise the training of the human image segmentation network model.
[0052] To improve the matting accuracy of the human image segmentation network model under training, one approach is to obtain a first loss parameter based on a first predicted human image and a mask image. This first loss parameter is then used to supervise the accuracy of the prediction results of the human image segmentation network model under training, and the model obtained when the first loss parameter converges is taken as the first human image segmentation network model. In at least one embodiment, the matting accuracy of the first human image segmentation network model is higher than that of the human image segmentation network model under training.
[0053] As a specific implementation method, the absolute value of the difference between the first predicted human image and the mask image can be used as the first loss parameter. The calculation method of the first loss parameter is as follows:
[0054] L1loss = |B1 - C|
[0055] Where L1loss represents the first loss parameter, B1 represents the first predicted human image, and C represents the mask image.
[0056] Step S250: Input the human body image, the mask image, and the human body key point image into the first human image segmentation network model to train the first human image segmentation network model.
[0057] The specific implementation of step S250 can be referred to the relevant description in step S140 of the foregoing embodiments, and will not be repeated here.
[0058] Step S260: Obtain the second loss parameter output by the first portrait segmentation network model, and use the model obtained when the second loss parameter converges as the second portrait segmentation network model. The attention weight of the second portrait segmentation network model to the human body key points is higher than that of the first portrait segmentation network model to the human body key points.
[0059] The specific implementation of step S260 can be referred to the relevant description in step S150 of the foregoing embodiments, and will not be repeated here.
[0060] As one implementation method, after the model obtained when the second loss parameter converges is used as the second portrait segmentation network model, image segmentation can be performed on the image data to be processed based on the second portrait segmentation network model. Here, the specific type of image data to be processed is not limited; for example, the image data to be processed can be images of the elderly, images of children, single photos, group photos, etc. The process of performing image segmentation on the image data to be processed based on the second portrait segmentation network model is similar to the process of training the second portrait segmentation network model described above, and will not be repeated here.
[0061] In this embodiment, the training process of a human image segmentation network model for matting human images is described as an example. In actual implementation, other types of images can also be trained using image segmentation network models for matting other types of images. The training process is similar to the training process described in this embodiment. In this way, other types of images, in addition to human images, can include object images, animal images, landscape images, or video images, etc., and are not specifically limited.
[0062] It should be noted that, based on this implementation method, those skilled in the art can further subdivide the training stages of the portrait segmentation network model. The parameters used to train the portrait segmentation network model in each stage can be different, and the number of parameters used to train the portrait segmentation network model in a later stage can be more than the number of parameters used to train the portrait segmentation network model in a previous stage, thereby obtaining a portrait segmentation network model with better image matting accuracy.
[0063] This application provides a data processing method that involves acquiring a human keypoint dataset, which includes a human image, a mask image corresponding to the human image, and human keypoint images corresponding to the human image; inputting the human image and the mask image into a human image segmentation network model to train the human image segmentation network model; obtaining a first loss parameter based on a first predicted human image and the mask image, and using the model obtained when the first loss parameter converges as a first human image segmentation network model; inputting the human image, the mask image, and the human keypoint images into the first human image segmentation network model to train the first human image segmentation network model; inputting the human image, the mask image, and the human keypoint images into the first human image segmentation network model to train the first human image segmentation network model; obtaining a second loss parameter output by the first human image segmentation network model, and using the model obtained when the second loss parameter converges as a second human image segmentation network model, wherein the attention weight of the human keypoints in the second human image segmentation network model is higher than the attention weight of the human keypoints in the first human image segmentation network model. Thus, by using the above method, we can add human key point information as a loss function constraint when training the human image segmentation network model, thereby enabling the network to focus its attention on the key point information of the human body, learn the semantic relationships between key points, and achieve accurate and detailed segmentation of the human image, that is, improve the image matting accuracy of the human image segmentation algorithm.
[0064] Please see Figure 3 Another embodiment of this application provides a data processing method that can be applied to electronic devices, the method comprising:
[0065] Step S310: Obtain a human body key point dataset, which includes a human body image, a mask image corresponding to the human body image, and a human body key point image corresponding to the human body image.
[0066] The specific implementation of step S310 can be referred to the relevant description in step S110 in the foregoing embodiments, and will not be repeated here.
[0067] Step S320: Input the human body image and the mask image into the human image segmentation network model to be trained, so as to train the human image segmentation network model.
[0068] The specific implementation of step S320 can be referred to the relevant description in step S120 in the foregoing embodiments, and will not be repeated here.
[0069] Step S330: Obtain the first predicted human image output by the human image segmentation network model to be trained based on the human image and the mask image.
[0070] The specific implementation of step S330 can be referred to the relevant description in step S230 in the foregoing embodiments, and will not be repeated here.
[0071] Step S340: Obtain the first loss parameter based on the first predicted human image and the mask image, and use the model obtained when the first loss parameter converges as the first human image segmentation network model.
[0072] The specific implementation of step S340 can be referred to the relevant description in step S240 in the foregoing embodiments, and will not be repeated here.
[0073] Step S350: Input the human body image, the mask image, and the human body key point image into the first human image segmentation network model to train the first human image segmentation network model.
[0074] The specific implementation of step S350 can be referred to the relevant description in step S140 of the foregoing embodiments, and will not be repeated here.
[0075] Step S360: Obtain the second predicted human image output by the first human image segmentation network model based on the human image and the mask image.
[0076] Based on the aforementioned implementation method, during the training of the first human image segmentation network model in the second stage, the learning rate of the first human image segmentation network model can be reduced. Then, the first human image segmentation network model performs human image segmentation prediction based on the human image and the mask image corresponding to the human image. The predicted human image segmentation image can then be used as the second predicted human image.
[0077] Step S370: Obtain keypoint weighted loss parameters based on the second predicted human body image, the mask image, and the human body keypoint image.
[0078] To further improve the matting accuracy of the first portrait segmentation network model, one approach is to obtain keypoint weighted loss parameters based on the second predicted human image, mask image, and human keypoint image. These keypoint weighted loss parameters can then be used to supervise the first portrait segmentation network model in learning the correlation of whole-body information.
[0079] As one implementation method, the product of the absolute value of the difference between the second predicted human image and the mask image and the human keypoint image can be used as the keypoint weighted loss parameter. The calculation method of the keypoint weighted loss parameter is expressed as follows:
[0080] loss_keypoint = |B2-C|*D / 255
[0081] Where loss_keypoint represents the keypoint weighted loss parameter, B2 represents the second predicted human image, C represents the mask image, D represents the human keypoint image, and 255 represents the gray value. In actual implementation, it can also be other gray values.
[0082] Step S380: The sum of the first loss parameter and the key point weighted loss parameter is used as the second loss parameter, and the model obtained when the second loss parameter converges is used as the second human face segmentation network model. The attention weight of the second human face segmentation network model to human face key points is higher than that of the first human face segmentation network model to human face key points.
[0083] In one implementation, the sum of the first loss parameter and the keypoint-weighted loss parameter can be used as the second loss parameter. This second loss parameter is then used to supervise the accuracy of the prediction results of the first portrait segmentation network model. That is, the second loss parameter is used as a loss constraint to train the first portrait segmentation network model until the function of the second loss parameter converges. The model obtained when the second loss parameter converges is then used as the second portrait segmentation network model. In at least one embodiment, the matting accuracy of the second portrait segmentation network model is higher than that of the first portrait segmentation network model.
[0084] In a specific application scenario, such as Figure 4 , 5 As shown in Figures 6 and 7, the human image (A) in the human keypoint dataset, the mask image corresponding to the human image (C) in the dataset, and the human keypoint image corresponding to the human image (D) in the dataset are shown respectively. During the first stage of training the human image segmentation network model, the following can be used: Figure 4 and Figure 5 Input the image segmentation network model to be trained, and then train the image segmentation network model; in the second stage, it can be... Figure 4 , Figure 5 as well as Figure 6 All inputs are the first portrait segmentation network model obtained from the first stage of training, in order to train the first portrait segmentation network model. Other intermediate processes are similar to the above-mentioned related descriptions and will not be repeated here.
[0085] This application provides a data processing method that involves acquiring a human keypoint dataset, the dataset including a human image, a mask image corresponding to the human image, and human keypoint images corresponding to the human image; inputting the human image and the mask image into a human image segmentation network model to train the model; obtaining a first predicted human image output by the human image segmentation network model based on the human image and the mask image; obtaining a first loss parameter based on the first predicted human image and the mask image, and using the model obtained when the first loss parameter converges as a first human image segmentation network model; and inputting the human image and the mask image into a human image segmentation network model to train the human image segmentation network model. The first portrait segmentation network model is trained by inputting the human keypoint image into it. A second predicted human image is obtained from the first portrait segmentation network model based on the human image and the mask image. Keypoint weighted loss parameters are obtained based on the second predicted human image, the mask image, and the human keypoint image. The sum of the first loss parameter and the keypoint weighted loss parameter is used as the second loss parameter, and the model obtained when the second loss parameter converges is used as the second portrait segmentation network model. The attention weight of the second portrait segmentation network model for human keypoints is higher than that of the first portrait segmentation network model. This method allows for the addition of human keypoint information as a loss function constraint during the training of the portrait segmentation network model, thereby directing network attention to the keypoint information of the human body, learning the semantic relationships between keypoints, and ultimately achieving accurate and detailed portrait segmentation, thus improving the image matting accuracy of the portrait segmentation algorithm.
[0086] Please see Figure 7 Another embodiment of this application provides a data processing method that can be applied to electronic devices, the method comprising:
[0087] Step S410: Obtain a human body image and a mask image corresponding to the human body image.
[0088] The source of the human body image is not limited, and the mask image corresponding to the human body image can be understood as the ground truth image of the human body image annotated manually (the annotated positions are the key points of the human skeleton).
[0089] Step S420: Input the human body image into the human body skeleton key point detection algorithm to obtain the human body key point image corresponding to the human body image output by the human body skeleton key point detection algorithm.
[0090] One implementation method is to input a human image into a human skeletal keypoint detection algorithm to obtain the human keypoint image corresponding to the human image output by the algorithm. Alternatively, an open-source human skeletal keypoint detection algorithm can be used to batch obtain the skeletal keypoints of a human segmentation dataset (which can be from the internet and contains multiple human images) and then obtain the human keypoint image corresponding to the human image. The specific type of human skeletal keypoint detection algorithm is not limited; for example, it can be the MediaPipe algorithm package or any other well-known human skeletal keypoint detection algorithm.
[0091] The obtained human keypoint image can be stored as a binary image, for example, the keypoint can be stored as a circle with a radius of 3 pixels and a gray value of 255, so as to facilitate the subsequent calculation of the keypoint weighted loss parameters.
[0092] In at least one embodiment, in order to improve the accuracy of the dataset and thus the reliability of training the human image segmentation network model, the obtained human key point images can be cleaned and filtered. For those that fail the filter, they can be manually labeled and corrected (using a drawing tool to draw points at the correct key point positions).
[0093] In one specific implementation, please refer to Figure 8 This document illustrates a visualization image of key human body points provided in an embodiment of this application. As one approach, the human body image from the foregoing embodiments ( Figure 4 ) and human key point images ( Figure 6 ) are synthesized to obtain Figure 8 The visualization of key human body points shown allows users to intuitively filter out key points that need correction and directly annotate and correct them on the visualization image.
[0094] Step S430: Use the human body image, the mask image, and the human body key point image as a human body key point dataset.
[0095] In this embodiment of the application, the human body image, the mask image, and the corrected human body key point image can be stored in the form of an array, and the array can be used as a human body key point dataset.
[0096] Step S440: Input the human body image and the mask image into the human image segmentation network model to be trained, so as to train the human image segmentation network model.
[0097] Step S450: Obtain the first loss parameter output by the human image segmentation network model to be trained, and use the model obtained when the first loss parameter converges as the first human image segmentation network model.
[0098] Step S460: Input the human body image, the mask image, and the human body key point image into the first human image segmentation network model to train the first human image segmentation network model.
[0099] Step S470: Obtain the second loss parameter output by the first portrait segmentation network model, and use the model obtained when the second loss parameter converges as the second portrait segmentation network model. The attention weight of the second portrait segmentation network model to the human body key points is higher than that of the first portrait segmentation network model to the human body key points.
[0100] This application provides a data processing method that involves: acquiring a human image and a mask image corresponding to the human image; inputting the human image into a human skeletal keypoint detection algorithm to obtain human keypoint images corresponding to the human image output by the algorithm; using the human image, the mask image, and the human keypoint images as a human keypoint dataset; inputting the human image and the mask image into a human image segmentation network model to be trained; obtaining a first loss parameter output by the human image segmentation network model to be trained, and using the model obtained when the first loss parameter converges as a first human image segmentation network model; inputting the human image, the mask image, and the human keypoint images into the first human image segmentation network model to train it; obtaining a second loss parameter output by the first human image segmentation network model, and using the model obtained when the second loss parameter converges as a second human image segmentation network model, wherein the attention weight of the second human image segmentation network model for human keypoints is higher than the attention weight of the first human image segmentation network model for human keypoints. Thus, by using the above method, we can add human key point information as a loss function constraint when training the human image segmentation network model, thereby enabling the network to focus its attention on the key point information of the human body, learn the semantic relationships between key points, and achieve accurate and detailed segmentation of the human image, that is, improve the image matting accuracy of the human image segmentation algorithm.
[0101] By cleaning and filtering human keypoint images, the accuracy of human keypoint datasets can be improved, thereby increasing the reliability of training human image segmentation network models using human keypoint datasets.
[0102] Please see Figure 9 This application provides a data processing device 500 that can operate on an electronic device. The device 500 includes:
[0103] The dataset acquisition module 510 is used to acquire a human key point dataset, which includes a human image, a mask image corresponding to the human image, and human key point images corresponding to the human image.
[0104] As one implementation, the dataset acquisition module 510 can be used to acquire a human body image and a mask image corresponding to the human body image; input the human body image into a human portrait skeleton key point detection algorithm to acquire the human body key point image corresponding to the human body image output by the human portrait skeleton key point detection algorithm; and use the human body image, the mask image, and the human body key point image as a human body key point dataset.
[0105] The first training module 520 is used to input the human body image and the mask image into the human image segmentation network model to be trained, so as to train the human image segmentation network model.
[0106] The first processing module 530 is used to obtain the first loss parameters output by the portrait segmentation network model to be trained, and to use the model obtained when the first loss parameters converge as the first portrait segmentation network model.
[0107] In one implementation, the first processing module 530 can be used to obtain a first predicted human image output by the human image segmentation network model to be trained based on the human image and the mask image; and to obtain a first loss parameter based on the first predicted human image and the mask image.
[0108] In at least one embodiment, the first processing module 530 may specifically be used to use the absolute value of the difference between the first predicted human image and the mask image as the first loss parameter.
[0109] The second training module 540 is used to input the human body image, the mask image, and the human body key point image into the first human image segmentation network model to train the first human image segmentation network model.
[0110] The second processing module 550 is used to obtain the second loss parameter output by the first portrait segmentation network model, and use the model obtained when the second loss parameter converges as the second portrait segmentation network model. The attention weight of the second portrait segmentation network model to the human body key points is higher than that of the first portrait segmentation network model to the human body key points.
[0111] In one implementation, the second processing module 550 can be used to obtain a second predicted human image output by the first human image segmentation network model based on the human image and the mask image; obtain key point weighted loss parameters based on the second predicted human image, the mask image and the human key point image; and use the sum of the first loss parameter and the key point weighted loss parameter as the second loss parameter.
[0112] In at least one embodiment, the second processing module 550 may specifically be used to multiply the absolute value of the difference between the second predicted human image and the mask image with the human keypoint image as the keypoint weighted loss parameter.
[0113] In this embodiment of the application, the apparatus 500 may further include an image segmentation module, which is used to perform image segmentation on the image data to be processed based on the second image segmentation network model after the model obtained when the second loss parameter converges is used as the second image segmentation network model.
[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0115] In the several embodiments provided in this application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or other forms.
[0116] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0117] Please see Figure 10 Based on the data processing method and apparatus described above, this application also provides an electronic device 100 capable of executing the aforementioned data processing method. The electronic device 100 includes a memory 102 and one or more (only one shown in the figure) processors 104 coupled to each other, with a communication line connecting the memory 102 and the processors 104. The memory 102 stores a program capable of executing the contents of the aforementioned embodiments, and the processors 104 can execute the program stored in the memory 102.
[0118] The processor 104 may include one or more processing cores. The processor 104 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 102, and by calling data stored in the memory 102. Optionally, the processor 104 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 104 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 104 and may be implemented separately using a communication chip.
[0119] The memory 102 may include random access memory (RAM) or read-only memory (ROM). The memory 102 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 102 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the aforementioned embodiments. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).
[0120] Please refer to Figure 11 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 600 stores program code that can be called by a processor to execute the methods described in the above method embodiments.
[0121] The computer-readable storage medium 600 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 600 has storage space for program code 610 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 610 may be compressed, for example, in a suitable form.
[0122] In summary, the data processing method, apparatus, electronic device, and storage medium provided in this application acquire a human keypoint dataset, which includes a human image, a mask image corresponding to the human image, and human keypoint images corresponding to the human image; input the human image and the mask image into a human image segmentation network model to be trained; obtain a first loss parameter output by the human image segmentation network model to be trained, and use the model obtained when the first loss parameter converges as a first human image segmentation network model; input the human image, the mask image, and the human keypoint images into the first human image segmentation network model to train the first human image segmentation network model; obtain a second loss parameter output by the first human image segmentation network model, and use the model obtained when the second loss parameter converges as a second human image segmentation network model, wherein the attention weight of the human keypoints in the second human image segmentation network model is higher than the attention weight of the human keypoints in the first human image segmentation network model. Thus, by using the above method, we can add human key point information as a loss function constraint when training the human image segmentation network model, thereby enabling the network to focus its attention on the key point information of the human body, learn the semantic relationships between key points, and achieve accurate and detailed segmentation of the human image, that is, improve the image matting accuracy of the human image segmentation algorithm.
[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A data processing method, characterized in that, The method includes: Obtain a human key point dataset, which includes a human image, a mask image corresponding to the human image, and human key point images corresponding to the human image; The human body image and the mask image are input into the human image segmentation network model to be trained; Obtain the first predicted human image output by the human image and the mask image from the human image segmentation network model to be trained; obtain the first loss parameter based on the first predicted human image and the mask image; and use the model obtained when the first loss parameter converges as the first human image segmentation network model. The human body image, the mask image, and the human body key point image are input into the first human image segmentation network model to train the first human image segmentation network model. A second predicted human image is obtained from the first human image segmentation network model based on the human image and the mask image. Keypoint weighted loss parameters are obtained based on the second predicted human image, the mask image, and the human keypoint image. The sum of the first loss parameter and the keypoint weighted loss parameter is used as the second loss parameter. The model obtained when the second loss parameter converges is used as the second human image segmentation network model. The attention weight of the second human image segmentation network model to the human keypoints is higher than that of the first human image segmentation network model to the human keypoints.
2. The method according to claim 1, characterized in that, The step of obtaining the first loss parameter based on the first predicted human image and the mask image includes: The absolute value of the difference between the first predicted human image and the mask image is used as the first loss parameter.
3. The method according to claim 1, characterized in that, The step of obtaining keypoint weighted loss parameters based on the second predicted human image, the mask image, and the human keypoint image includes: The absolute value of the difference between the second predicted human image and the mask image is multiplied by the human keypoint image as the keypoint weighted loss parameter.
4. The method according to claim 1, characterized in that, The acquisition of the human body key point dataset includes: Acquire a human body image and a mask image corresponding to the human body image; The human body image is input into the human body skeleton key point detection algorithm to obtain the human body key point image corresponding to the human body image output by the human body skeleton key point detection algorithm. The human body image, the mask image, and the human body key point image are used as the human body key point dataset.
5. The method according to any one of claims 1-4, characterized in that, After using the model obtained when the second loss parameter converges as the second portrait segmentation network model, the method further includes: Image segmentation is performed on the image data to be processed based on the second portrait segmentation network model.
6. A data processing apparatus, characterized in that, The device includes: The dataset acquisition module is used to acquire a human key point dataset, which includes a human image, a mask image corresponding to the human image, and human key point images corresponding to the human image. The first training module is used to input the human body image and the mask image into the human image segmentation network model to be trained, so as to train the human image segmentation network model. The first processing module is used to obtain the first predicted human image output by the human image and the mask image by the human image segmentation network model to be trained, obtain the first loss parameter based on the first predicted human image and the mask image, and use the model obtained when the first loss parameter converges as the first human image segmentation network model. The second training module is used to input the human body image, the mask image, and the human body key point image into the first human image segmentation network model to train the first human image segmentation network model. The second processing module is used to obtain a second predicted human image output by the first human image segmentation network model based on the human image and the mask image, obtain key point weighted loss parameters based on the second predicted human image, the mask image and the human key point image, use the sum of the first loss parameter and the key point weighted loss parameter as the second loss parameter, and use the model obtained when the second loss parameter converges as the second human image segmentation network model. The attention weight of the second human image segmentation network model to the human key points is higher than the attention weight of the first human image segmentation network model to the human key points.
7. An electronic device, characterized in that, Includes one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method according to any one of claims 1-5.
Citation Information
Patent Citations
Instance segmentation method based on key points
CN111507334A