Image Processing Method, Apparatus, Electronic Device, and Storage Medium

Abstract

The present disclosure provides an image processing method, apparatus, electronic device, and storage medium, which relate to technical fields such as machine learning and artificial intelligence, and particularly relate to the technical field of image processing. The specific implementation solution is as follows: obtaining an instance segmentation result and a skin segmentation result corresponding to an image to be processed, where the instance segmentation result includes multiple human instance regions, and the skin segmentation result includes multiple skin connected regions; for each skin connected region in the skin segmentation result, traversing each human instance region in the instance segmentation result, calculating the first intersection area between the current skin connected region and each human instance region, and calculating the ratio between the first intersection area and the area of the current skin connected region to obtain a first proportion result; updating the human instance region and the skin connected region according to the first proportion result to obtain an updated instance segmentation result and an updated skin segmentation result. The present disclosure can improve the accuracy of instance segmentation result classification.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of machine learning, artificial intelligence, etc., and specifically relates to the field of image processing technology, and particularly relates to an image processing method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of deep learning, computer vision technology has been widely applied. Among them, instance segmentation technology, as a relatively basic task in visual tasks, is mainly used for pixel-level segmentation of object instances in images, so as to identify and track each object instance in the image to determine its category.

[0003] Existing instance segmentation technology mainly optimizes at the instance segmentation model level (i.e., aspects such as network structure and loss function). However, optimizing at the model level cannot solve the problems of re-detection or missed detection in instance segmentation results. Summary of the Invention

[0004] The present disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, including:

[0006] Obtaining an instance segmentation result and a skin segmentation result corresponding to an image to be processed, where the instance segmentation result includes multiple human instance regions, and the skin segmentation result includes multiple skin connected regions;

[0007] For each skin connected region in the skin segmentation result, traversing each human instance region in the instance segmentation result, calculating a first intersection area between the current skin connected region and each human instance region, and calculating a ratio between the first intersection area and the area of the current skin connected region to obtain a first proportion result;

[0008] Updating the human instance region and the skin connected region according to the first proportion result to obtain an updated instance segmentation result and an updated skin segmentation result.

[0009] According to another aspect of the present disclosure, there is provided an image processing apparatus, including:

[0010] An obtaining module, configured to obtain an instance segmentation result and a skin segmentation result corresponding to an image to be processed, where the instance segmentation result includes multiple human instance regions, and the skin segmentation result includes multiple skin connected regions;

[0011] A traversal module, configured to traverse each skin connected region in the skin segmentation result, traverse each human instance region in the instance segmentation result, calculate a first intersection area between the current skin connected region and each human instance region, and calculate a ratio between the first intersection area and the area of the current skin connected region to obtain a first ratio result;

[0012] An update module, configured to update the human instance region and the skin connected region according to the first ratio result to obtain an updated instance segmentation result and an updated skin segmentation result.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the above technical solutions.

[0017] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in any one of the above technical solutions.

[0018] According to a fifth aspect of the present disclosure, there is provided a computer program product, including a computer program, where the computer program implements the method described in any one of the above technical solutions when executed by a processor.

[0019] The present disclosure provides an image processing method, apparatus, device, and storage medium. After obtaining the instance segmentation result, the present disclosure corrects the undetected part of the instance segmentation result by using the skin segmentation result. Specifically, the skin connected regions in the skin segmentation result are compared with the human instance regions in the instance segmentation result, and the human instance regions and the skin connected regions are updated according to the comparison result, so as to obtain an updated instance segmentation result and an updated skin segmentation result. In this way, the undetected part in the instance segmentation result is filled by the skin connected regions, so that the undetected situation in the instance segmentation result can be corrected, and further the accuracy of the instance segmentation result classification can be improved.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0021] The drawings are used to better understand the present solution and do not limit the present disclosure. Among them:

[0022] Figure 1 is a schematic diagram of the steps of the image processing method in the embodiments of the present disclosure;

[0023] Figure 2 is a schematic flowchart corresponding to the image processing method in an embodiment of the present disclosure;

[0024] Figure 3 is a schematic flowchart corresponding to the image processing method in another embodiment of the present disclosure;

[0025] Figure 4 The principle block diagram of the image processing device in the embodiments of the present disclosure;

[0026] Figure 5 is a block diagram of an electronic device for implementing the image processing method of the embodiments of the present disclosure. Detailed Embodiments

[0027] The following describes exemplary embodiments of the present disclosure with reference to the drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0028] The present disclosure provides an image processing method. Referring to Figure 1 as shown, it includes:

[0029] Step S101, obtaining an instance segmentation result and a skin segmentation result corresponding to the image to be processed. The instance segmentation result includes multiple human instance regions, and the skin segmentation result includes multiple skin connected regions.

[0030] Specifically, when obtaining the instance segmentation result and the skin segmentation result corresponding to the image to be processed, the image to be processed is input into the instance segmentation model, and the instance segmentation result corresponding to the image to be processed can be obtained through the instance segmentation model. For the skin segmentation result, the image to be processed is input into the skin semantic segmentation model, and the skin segmentation result corresponding to the image to be processed is obtained through the skin semantic segmentation model. Since the instance segmentation model and the skin semantic segmentation model are existing models, the specific principles of these two models are not described herein.

[0031] Since the instance segmentation result includes multiple person instances contained in the image to be processed, and each person instance occupies a certain area, it can be considered that the instance segmentation result contains multiple person instances. For the skin segmentation result, the skin segmentation result contains multiple skin patches, and the areas contained in each skin patch are connected, so it can be considered that the skin segmentation result contains multiple skin connected regions. It should be noted that all the skin in the skin segmentation result output by the skin semantic segmentation model, and at this stage, the skin connected regions in the skin segmentation result do not know which person instance the skin belongs to. Since there may be missed detections in the instance segmentation result obtained by the instance segmentation model, that is, some areas of the obtained person instance regions are incomplete, the solution of the embodiment of the present disclosure uses the skin connected regions in the skin segmentation result to fill in the incomplete parts in the instance segmentation result, so as to correct the instance segmentation result, and thus is beneficial to improving the accuracy of the instance segmentation result classification.

[0032] Step S102: For each skin connected region in the skin segmentation result, traverse each person instance region in the instance segmentation result, calculate the first intersection area between the current skin connected region and each person instance region, and calculate the ratio between the first intersection area and the area of the current skin connected region to obtain the first ratio result.

[0033] Specifically, since the skin segmentation result contains multiple skin connected regions, and the instance segmentation result contains multiple person instance regions, when processing the skin segmentation result and the instance segmentation result, not only each skin connected region in the skin segmentation result is traversed, but also each person instance region in the instance segmentation result is traversed. During the traversal process, first take a skin connected region in the skin segmentation result as an example. This skin connected region is called the current skin connected region. Taking the current skin connected region as a fixed point, calculate the first intersection area between the current skin connected region and each person instance region respectively. At this time, there are multiple numbers of the obtained first intersection areas. Then, calculate the ratios of the multiple first intersection areas to the area of the current skin connected region respectively, so that multiple numbers of first ratio results can be obtained.

[0034] To facilitate the understanding of the solutions of the embodiments of the present disclosure, the following is an example. Assume that the multiple skin connected regions included in the skin segmentation result are respectively a, b, c... n, and the person instance regions included in the instance segmentation result are respectively A, B, C,... N. Taking the current skin connected region a as an example, at this time, first calculate the first intersection areas between a and A, B, C,... N respectively to obtain the first intersection area set {a1, a2, a3... an}, and then calculate the ratios of the first intersection area set to the area of the current skin connected region a respectively. Assume that the area of the current skin connected region a is P, then the first ratio result can be obtained as {a1 / P, a2 / P, a3 / P... an / P}. For other skin connected regions, such as b, c... n, they are processed in a similar manner, which will not be elaborated here. In this way, by obtaining the first ratio result, it is beneficial to determine how to update the person instance region and the skin connected region.

[0035] Step S103: Update the person instance region and the skin connected region according to the first ratio result to obtain the updated instance segmentation result and the updated skin segmentation result.

[0036] Specifically, in the obtained first ratio result, when updating the person instance region and the skin connected region according to the first ratio result, that is, making up for the undetected part in the instance segmentation result through the skin connected region, so as to correct the undetected situation in the instance segmentation result and avoid the occurrence of undetected situations in the instance segmentation result, thereby improving the accuracy of the instance segmentation result classification.

[0037] The present disclosure provides an image processing method, device, equipment and storage medium. After obtaining the instance segmentation result, the present disclosure corrects the undetected part of the instance segmentation result by using the skin segmentation result. Specifically, it compares the skin connected domains in the skin segmentation result with the person instance regions in the instance segmentation result, and updates the person instance region and the skin connected region according to the comparison result, so as to obtain the updated instance segmentation result and the updated skin segmentation result. In this way, making up for the undetected part in the instance segmentation result through the skin connected region can correct the undetected situation in the instance segmentation result, and further improve the accuracy rate of the instance segmentation result classification.

[0038] In some optional embodiments, refer to Figure 2 , Figure 2It is a schematic flowchart corresponding to an image processing method in an embodiment of the present disclosure. Before step S102, for each skin connected region in the skin segmentation result, traversing each person instance region in the instance segmentation result, and calculating the first intersection area between the current skin connected region and each person instance region, the method further includes S201, finding each connected domain in the skin segmentation result to obtain multiple skin connected regions, and then numbering each skin connected region as 0, 1, 2,... N. In this way, by finding each connected domain in the skin segmentation result to obtain multiple skin connected regions included in the skin segmentation result, and then numbering each skin connected region, it is beneficial to subsequently orderly use the skin connected regions to fill in the incomplete parts in the instance segmentation result and avoid missed detections.

[0039] In some optional embodiments, taking one of the skin connected regions in the skin segmentation result as an example, step S102, for each skin connected region in the skin segmentation result, traversing each person instance region in the instance segmentation result, calculating the first intersection area between the current skin connected region and each person instance region, and calculating the ratio between the first intersection area and the area of the current skin connected region to obtain the first proportion result, includes the following steps:

[0040] Step S202, calculating the first intersection area between the current skin connected region and the current person instance region.

[0041] Specifically, during the traversal, first take one of the skin connected regions in the skin segmentation result as an example, and this skin connected region is called the current skin connected region. Then select one person instance region from the instance segmentation result, and this person instance region is called the current person instance region. Next, calculate the intersection area between the current skin connected region and the current person instance region to obtain the first intersection area.

[0042] Step S203, calculating the ratio between the first intersection area and the area of the current skin connected region to obtain the first proportion.

[0043] Specifically, after obtaining the first intersection area between the current skin connected region and the current person instance region previously, calculate the proportion of the first intersection area to the area of the current skin connected region, so as to obtain the first proportion.

[0044] Step S204, judging whether the serial number of the current person instance region is less than the number of person instances.

[0045] Step S205: If the serial number of the current person instance area is less than the number of person instances, increment the serial number of the traversed person instance area by 1, use the person instance area with the next serial number as the current person instance area, and repeat the above steps until the serial number of the current person instance area is greater than or equal to the number of person instances, thereby obtaining the first ratio result.

[0046] In this way, by calculating the first intersection area between the current skin connected area and each person instance area, and calculating the ratio between the first intersection area and the area of the current skin connected area, the first ratio result is obtained, which is beneficial for subsequent determination of how to update the person instance area and the skin connected area.

[0047] In some optional embodiments, in step S103, updating the person instance area and the skin connected area according to the first ratio result to obtain the updated instance segmentation result includes:

[0048] Step S206: Select the person instance area corresponding to the maximum ratio from the first ratio result.

[0049] Step S207: Determine whether the maximum ratio is greater than the set threshold.

[0050] Step S208: If the maximum ratio is greater than or equal to the set threshold, merge the person instance area corresponding to the maximum ratio with the current skin connected area into a new person instance area to obtain the updated instance segmentation result and the updated skin segmentation result.

[0051] Specifically, after obtaining the first ratio result, there are multiple first ratio results. In the embodiments of the present disclosure, the person instance area corresponding to the maximum ratio is selected from the first ratio results, and then the maximum ratio is compared with the set threshold. Since the person instance area corresponding to the maximum ratio is the person instance with the largest ratio of the first intersection area to the current skin connected area, if the maximum ratio is greater than the set threshold, it indicates that the skin connected area and the person instance area are the most matched. At this time, the person instance area corresponding to the maximum ratio is merged with the current skin connected area into a new person instance area, obtaining the updated instance segmentation result. The embodiments of the present disclosure further determine whether they are matched by comparing the skin connected area with the person instance area. If they are matched, it can be considered that there is an incomplete part in the current person instance area. At this time, the incomplete part is filled by using the matched skin connected area, thereby correcting the missed detection situation in the instance segmentation result, and then obtaining the updated instance segmentation result to improve the classification accuracy of the instance segmentation result. Among them, for the setting of the set threshold, those skilled in the art can set it according to actual needs and are not limited herein.

[0052] In this way, by comparing the skin connected region with the person instance region, it is further determined whether there are incomplete parts in the current person instance region, that is, there is a missed detection situation. At this time, the matching skin connected region is used to complement the incomplete part, so as to correct the missed detection situation in the instance segmentation result, and then an updated instance segmentation result can be obtained to improve the classification accuracy of the instance segmentation result.

[0053] Since the above embodiment takes one of the skin connected regions in the skin segmentation result as an example, next, the remaining skin connected regions in the skin segmentation result are processed. If the traversal serial number of the skin connected region is less than the number of skin connected regions, the traversal serial number of the skin connected region is incremented by 1, and the person instance region with the next serial number is used as the current person instance region, and the skin connected region is processed in the above manner until the traversal serial number of the skin connected region is greater than or equal to the number of skin connected regions. Since the processing method for each skin connected region is similar to that of the skin connected region in the above embodiment, it will not be elaborated here.

[0054] In some optional embodiments, after all the skin connected regions are traversed, that is, after updating the person instance region and the skin connected region according to the first ratio result to obtain the updated instance segmentation result and the updated skin segmentation result, the image processing method further includes:

[0055] Step S209, obtaining the updated instance segmentation result and the updated skin segmentation result;

[0056] Step S210, calculating the difference set between the updated instance segmentation result and the updated skin segmentation result;

[0057] Step S211, for each skin connected domain included in the difference set, traversing each person instance region in the updated instance segmentation result to update the person instance region and the skin connected region again.

[0058] Specifically, after updating the person instance area and the skin connected area according to the first proportion result, there are still some skin connected areas that have not been assigned to the corresponding person instance area, that is, there will still be missed detection cases. At this time, further judgment is needed. When making further judgment, first obtain the updated instance segmentation result and the updated skin segmentation result. Among them, the updated instance segmentation result not only includes the updated person instance area but also the unupdated person instance area. Similarly, for the updated skin segmentation result, it also not only includes the updated skin connected area but also the unupdated skin connected area. After obtaining the updated instance segmentation result and the updated skin segmentation result, calculate the difference set between the two. The so-called difference set is actually the skin connected area that remains unmerged into the person instance area. For the skin connected area that has not been merged into the person instance area, then traverse each person instance area in the updated instance segmentation result to update the person instance area and the skin connected area again.

[0059] In this way, by calculating the difference set between the updated instance segmentation result and the updated skin segmentation result, it is beneficial to determine which parts in the instance segmentation result are still not filled. At this time, through secondary update judgment, the unfilled parts can be filled with the matching skin connected areas, thereby improving the missed detection situation in the instance segmentation result.

[0060] In some alternative embodiments, obtaining the updated instance segmentation result and the updated skin segmentation result includes:

[0061] Calculate the union of the updated person instance areas to obtain the updated instance segmentation result;

[0062] Calculate the union of the updated skin connected areas to obtain the updated skin segmentation result.

[0063] Specifically, calculating the union of the updated person instance areas means obtaining all person instance areas. All person instance areas include the updated person instance areas and the remaining original person instance areas that have not been updated, so as to obtain the updated instance segmentation result. Similarly, calculating the union of the updated skin connected areas means obtaining all skin connected areas. All skin connected areas include the updated skin connected areas and the remaining original skin connected areas that have not been updated, so as to obtain the updated skin segmentation result.

[0064] In this way, by obtaining the updated instance segmentation result and the updated skin segmentation result, it is beneficial to determine the difference set between the updated instance segmentation result and the updated skin segmentation result subsequently.

[0065] In some alternative embodiments, in step S211, before traversing each person instance region in the updated instance segmentation result for each skin connected region included in the difference set to update the person instance region and the skin connected region again, the method further includes S2111, finding each connected region in the above difference set to obtain a plurality of skin connected regions, and then numbering each skin connected region as 0, 1, 2,... N. In this way, by finding each connected region in the difference set to obtain a plurality of skin connected regions included in the skin segmentation result, and then numbering each skin connected region, it is beneficial to subsequently orderly use the skin connected regions in to supplement the incomplete parts in the instance segmentation result and avoid missed detections.

[0066] In some alternative embodiments, taking one of the skin connected regions in the difference set as an example, step S211, for each skin connected region included in the difference set, traversing each person instance region in the updated instance segmentation result to update the person instance region and the skin connected region again, includes:

[0067] Step S2112, for each skin connected region included in the difference set, traversing each person instance region in the updated instance segmentation result, calculating the second intersection area between the current skin connected region and each person instance region, and calculating the ratio between the second intersection area and the area of the current skin connected region to obtain a second proportion result.

[0068] Specifically, during the traversal, first take one skin connected region in the difference set as an example, and this skin connected region is called the current skin connected region. Then select one person instance region from the instance segmentation result, and this person instance region is called the current person instance region. Next, calculate the second intersection area between the current skin connected region and the current person instance region. After obtaining the second intersection area between the current skin connected region and the current person instance region, calculate the proportion of the second intersection area to the area of the current skin connected region, so as to obtain a second proportion. Then, determine whether the serial number of the current person instance region is less than the number of person instances. If the serial number of the current person instance region is less than the number of person instances, increment the serial number of the traversed person instance region by 1, and use the person instance region with the next serial number as the current person instance region, and repeat the above steps until the serial number of the current person instance region is greater than or equal to the number of person instances, and then a second proportion result can be obtained. In this way, by calculating the second intersection area between the current skin connected region and each person instance region, and calculating the ratio between the second intersection area and the area of the current skin connected region to obtain a second proportion result, it is beneficial to determine how to update the person instance region and the skin connected region.

[0069] Step S2113: Select the person instance region corresponding to the maximum ratio from the second ratio result.

[0070] Step S2114: Merge the person instance region corresponding to the maximum ratio with the current skin connected region into a new person instance region.

[0071] Specifically, after obtaining the second ratio result, since there are multiple second ratio results, select the person instance region corresponding to the maximum ratio from the second ratio results and record the corresponding person instance region. Since the person instance region corresponding to the maximum ratio is also the one with the largest ratio of the second intersection area to the skin connected region, it indicates that this skin connected region best matches this person instance region. At this time, merge the person instance region corresponding to the maximum ratio with the current skin connected region into a new person instance region, and use the skin connected region in the skin segmentation result to fill in the incomplete part to further improve the situation of missed detection.

[0072] In this way, through secondary judgment, use the skin connected region in the skin segmentation result to fill in the incomplete part in the instance segmentation result to correct the missed detection part in the instance segmentation result, which is conducive to improving the accuracy of instance segmentation result classification.

[0073] In some optional embodiments, refer to Figure 3 , Figure 3 is the schematic flowchart corresponding to the image processing method in another embodiment of the present disclosure. After obtaining the updated person instance regions in the above embodiment, first number each person instance region, and the corresponding serial numbers of each person instance region are 0, 1, 2, 3,..., N respectively. This image processing method further includes:

[0074] Obtain a first person instance region and a second person instance region from the updated person instance regions, and calculate the overlapping area between the first person instance region and the second person instance region. Refer to step S301, where the first person instance region is called instance i, and the second person instance region is called instance j. Calculating the overlapping area between the first person instance region and the second person instance region is to calculate the intersection of instance i and instance j.

[0075] After calculating the overlapping area between the first person instance region and the second person instance region, perform a dilation operation on the overlapping area to obtain the dilated overlapping area. Refer to Figure 3 , after calculating the intersection of instance i and instance j in step S301, next perform step S302 to perform a dilation operation on the intersection, where the dilation radius is r.

[0076] Next, based on the overlapping area after dilation processing, update the first person instance area and the second person instance area. After step S302, step S303 is then executed. To facilitate understanding of the solution of the embodiment of the present application, the following is an example. Still taking the first person instance area as instance i and the second person instance area as instance j, after calculating the intersection of instance i and instance j, perform a dilation operation on the intersection, calculate the intersection of the dilated area and instance i to obtain Ai, and calculate the intersection of the dilated area and instance j to obtain Aj. Next, determine whether Ai is greater than or equal to Aj. If Ai is greater than or equal to Aj, update instance j to the union of instance j and the intersection area. If Ai is less than Aj, update instance i to the union of instance i and the intersection area. Finally, the updated first person instance area and the updated second person instance area can be obtained. It should be noted that here only instance i is taken as the first person instance area and j as the second person instance area as an example, and similar processing is also performed on other person instance areas until the instance traversal serial number i is greater than or equal to the number of instances and the instance traversal serial number j is greater than or equal to the number of instances, then the process ends.

[0077] Specifically, after detecting the missed detection situation in the instance segmentation result, that is, after filling in the skin connected areas to the missed detected person instance areas, there will be an overlapping situation between different person instance areas. To solve this problem, in this embodiment, the overlapping area between pairwise person instance areas is calculated. Taking the first person instance area and the second person instance area as an example, obtain the first person instance area and the second person instance area from the updated person instance areas, and then calculate the overlapping area between the two. Then, perform a dilation operation on the overlapping area to obtain the overlapping area after dilation processing. Based on the overlapping area after dilation processing, update the first person instance area and the second person instance area, so that the overlapping situation between two person instance areas can be avoided.

[0078] In this way, by performing a dilation operation on the overlapping area between the first person instance area and the second person instance area, and updating the first person instance area and the second person instance area according to the overlapping area after dilation processing, the overlapping situation between the first person instance area and the second person instance area can be avoided.

[0079] In some optional embodiments, updating the first person instance area and the second person instance area according to the overlapping area after dilation processing includes:

[0080] Calculate the ratio between the overlapping area after dilation processing and the first person instance area to obtain the first occupancy ratio;

[0081] Calculate the ratio between the overlapping area after dilation processing and the second person instance area to obtain the second occupancy ratio;

[0082] If the first occupancy ratio is greater than the second occupancy ratio, divide the overlapping area between the first person instance area and the second person instance area into the first person instance area to obtain the updated first person instance area and the updated second person instance area.

[0083] Specifically, the so-called dilation processing means expanding the overlapping area. For example, it can be expanded by one pixel. Calculate the ratio between the overlapping area after dilation processing and the second person instance area to obtain the second occupancy ratio. Then compare the first occupancy ratio and the second occupancy ratio. If the first occupancy ratio is greater than the second occupancy ratio, divide the overlapping area between the first person instance area and the second person instance area into the first person instance area to obtain the updated first person instance area and the updated second person instance area. If the second occupancy ratio is greater than the first occupancy ratio, divide the overlapping area between the first person instance area and the second person instance area into the second person instance area to obtain the updated first person instance area and the updated second person instance area.

[0084] In this way, by performing dilation processing on the overlapping area between the first person instance area and the second person instance area, and updating the first person instance area and the second person instance area according to the overlapping area after dilation processing, the situation of overlap between the first person instance area and the second person instance area can be avoided.

[0085] In summary, the embodiments of the present disclosure process the instance segmentation results output by the instance segmentation model, and further correct the recheck and missed detection situations in the instance segmentation results in a multi-person scenario through the skin segmentation results, which can not only not increase the model inference time, but also achieve the segmentation accuracy of optimizing the segmentation results. This method can improve the accuracy of the instance segmentation model, optimize the person instance segmentation effect of the AI (Artificial Intelligence) photo retouching client, and make the effect of person-level instance photo retouching more fitting and the edge transition more natural.

[0086] The following introduces the device embodiments of the present application, which can be used to execute the image processing method in the above embodiments of the present application. For the details not disclosed in the device embodiments of the present application, please refer to the embodiments of the above image processing method of the present application.

[0087] The present disclosure also provides an image processing device 400, as Figure 4 shown, including:

[0088] An acquisition module 401, configured to acquire an instance segmentation result and a skin segmentation result corresponding to an image to be processed, where the instance segmentation result includes multiple human instance regions, and the skin segmentation result includes multiple skin connected regions;

[0089] A traversal module 402, configured to traverse each human instance region in the instance segmentation result for each skin connected region in the skin segmentation result, calculate a first intersection area between the current skin connected region and each human instance region, and calculate a ratio between the first intersection area and the area of the current skin connected region to obtain a first ratio result;

[0090] An update module 403, configured to update the human instance region and the skin connected region according to the first ratio result to obtain an updated instance segmentation result and an updated skin segmentation result.

[0091] In some alternative embodiments, the update module 403 updates the human instance region and the skin connected region according to the first ratio result to obtain an updated instance segmentation result and an updated skin segmentation result, including:

[0092] Select the human instance region corresponding to the maximum ratio from the first ratio result;

[0093] If the maximum ratio is greater than or equal to a set threshold, merge the human instance region corresponding to the maximum ratio and the current skin connected region into a new human instance region to obtain an updated instance segmentation result and an updated skin segmentation result.

[0094] In some alternative embodiments, after the update module 403 updates the human instance region and the skin connected region according to the first ratio result to obtain an updated instance segmentation result and an updated skin segmentation result, it is further configured to:

[0095] Acquire the updated instance segmentation result and the updated skin segmentation result;

[0096] Calculate the difference set between the updated instance segmentation result and the updated skin segmentation result;

[0097] For each skin connected domain included in the difference set, traverse each human instance region in the updated instance segmentation result to update the human instance region and the skin connected region again.

[0098] In some alternative embodiments, the update module 403 acquires the updated instance segmentation result and the updated skin segmentation result, including:

[0099] Calculate the union of the updated human instance regions to obtain the updated instance segmentation result;

[0100] Calculate the union of the updated skin connected regions to obtain the updated skin segmentation result.

[0101] In some alternative embodiments, for each skin connected region included in the difference set, the update module 403 traverses each person instance region in the updated instance segmentation result to further update the person instance region and the skin connected region, including:

[0102] For each skin connected region included in the difference set, traverse each person instance region in the updated instance segmentation result, calculate the second intersection area between the current skin connected region and each person instance region, and calculate the ratio between the second intersection area and the area of the current skin connected region to obtain the second ratio result;

[0103] Select the person instance region corresponding to the maximum ratio from the second ratio result;

[0104] Merge the person instance region corresponding to the maximum ratio and the current skin connected region into a new person instance region.

[0105] In some alternative embodiments, the update module 403 is further configured to obtain a first person instance region and a second person instance region from the updated person instance regions;

[0106] Calculate the overlapping area between the first person instance region and the second person instance region;

[0107] Perform a dilation operation on the overlapping area to obtain the dilated overlapping area;

[0108] Update the first person instance region and the second person instance region according to the dilated overlapping area.

[0109] In some alternative embodiments, the update module 403 updates the first person instance region and the second person instance region according to the dilated overlapping area, including:

[0110] Calculate the ratio between the dilated overlapping area and the first person instance region to obtain the first ratio value;

[0111] Calculate the ratio between the dilated overlapping area and the second person instance region to obtain the second ratio value;

[0112] If the first ratio value is greater than the second ratio value, divide the overlapping area between the first person instance region and the second person instance region into the first person instance region to obtain the updated first person instance region and the updated second person instance region.

[0113] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0114] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0115] Figure 5 FIG. shows a schematic block diagram of an exemplary electronic device 500 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0116] As Figure 5 shown, the electronic device 500 includes a computing unit 501 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0117] A plurality of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 executes the various methods and processes described above, such as the image processing method. For example, in some embodiments, the image processing method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the small program distribution described above can be executed. Alternatively, in other embodiments, the computing unit 501 can be configured to execute the image processing method in any other suitable manner (e.g., by means of firmware).

[0119] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, can also be a server of a distributed system, or a server incorporating a blockchain.

[0125] It should be understood that the various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and no limitation is imposed herein.

[0126] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Obtaining an instance segmentation result and a skin segmentation result corresponding to an image to be processed, wherein the instance segmentation result includes multiple human instance regions, and the skin segmentation result includes multiple skin connected regions; For each skin connected region in the skin segmentation result, traversing each human instance region in the instance segmentation result, calculating the first intersection area between the current skin connected region and each human instance region, and calculating the ratio between the first intersection area and the area of the current skin connected region to obtain a first proportion result; Selecting the human instance region corresponding to the maximum ratio from the first proportion result; If the maximum ratio is greater than or equal to a set threshold, merging the human instance region corresponding to the maximum ratio and the current skin connected region into a new human instance region to obtain an updated instance segmentation result and an updated skin segmentation result.

2. The method according to claim 1, wherein, The method further includes: Obtaining the updated instance segmentation result and the updated skin segmentation result; Calculating the difference set between the updated instance segmentation result and the updated skin segmentation result; For each skin connected domain included in the difference set, traversing each human instance region in the updated instance segmentation result to re-update the human instance region and the skin connected region.

3. The method according to claim 2, wherein, The obtaining the updated instance segmentation result and the updated skin segmentation result includes: Calculating the union of the updated human instance regions to obtain the updated instance segmentation result; Calculating the union of the updated skin connected regions to obtain the updated skin segmentation result.

4. The method according to claim 2, wherein The for each skin connected domain included in the difference set, traversing each human instance region in the updated instance segmentation result to re-update the human instance region and the skin connected region includes: For each skin connected domain included in the difference set, traversing each human instance region in the updated instance segmentation result, calculating the second intersection area between the current skin connected region and each human instance region, and calculating the ratio between the second intersection area and the area of the current skin connected region to obtain a second proportion result; Selecting the human instance region corresponding to the maximum ratio from the second proportion result; Merging the human instance region corresponding to the maximum ratio and the current skin connected region into a new human instance region.

5. The method according to claim 1, wherein, The method further includes: Obtaining a first human instance region and a second human instance region from the updated human instance regions; Calculating the overlapping area between the first human instance region and the second human instance region; Performing a dilation operation on the overlapping area to obtain a dilated overlapping area; Updating the first human instance region and the second human instance region according to the dilated overlapping area.

6. The method according to claim 5, wherein, The updating the first human instance region and the second human instance region according to the dilated overlapping area includes: Calculating the ratio between the dilated overlapping area and the first human instance region to obtain a first proportion value; Calculate the ratio between the overlapping area after dilation processing and the second person instance area to obtain the second occupancy ratio; If the first occupancy ratio is greater than the second occupancy ratio, divide the overlapping area between the first person instance area and the second person instance area into the first person instance area to obtain the updated first person instance area and the updated second person instance area.

7. An image processing device, comprising: An acquisition module, configured to acquire an instance segmentation result and a skin segmentation result corresponding to an image to be processed, where the instance segmentation result includes multiple person instance areas, and the skin segmentation result includes multiple skin connected areas; A traversal module, configured to, for each skin connected area in the skin segmentation result, traverse each person instance area in the instance segmentation result, calculate the first intersection area between the current skin connected area and each person instance area, and calculate the ratio between the first intersection area and the area of the current skin connected area to obtain a first occupancy result; An update module, configured to select the person instance area corresponding to the maximum ratio from the first occupancy result; If the maximum ratio is greater than or equal to a set threshold, merge the person instance area corresponding to the maximum ratio and the current skin connected area into a new person instance area to obtain an updated instance segmentation result and an updated skin segmentation result.

8. The apparatus according to claim 7, wherein, The update module is further configured to: Acquire the updated instance segmentation result and the updated skin segmentation result; Calculate the difference set between the updated instance segmentation result and the updated skin segmentation result; For each skin connected domain included in the difference set, traverse each person instance area in the updated instance segmentation result to update the person instance area and the skin connected area again.

9. The apparatus according to claim 8, wherein The update module acquires the updated instance segmentation result and the updated skin segmentation result, including: Calculating the union of the updated person instance areas to obtain the updated instance segmentation result; Calculating the union of the updated skin connected areas to obtain the updated skin segmentation result.

10. The device according to claim 8, wherein, The update module traverses each person instance area in the updated instance segmentation result for each skin connected domain included in the difference set to update the person instance area and the skin connected area again, including: For each skin connected domain included in the difference set, traverse each person instance area in the updated instance segmentation result, calculate the second intersection area between the current skin connected area and each person instance area, and calculate the ratio between the second intersection area and the area of the current skin connected area to obtain a second occupancy result; Select the person instance area corresponding to the maximum ratio from the second occupancy result; Merge the person instance area corresponding to the maximum ratio and the current skin connected area into a new person instance area.

11. The device according to claim 7, wherein, The update module is further configured to obtain a first person instance area and a second person instance area from the updated person instance areas; Calculate the overlapping area between the first person instance area and the second person instance area; Perform a dilation operation on the overlapping area to obtain the overlapping area after dilation processing; Update the first person instance area and the second person instance area according to the overlapping area after dilation processing.

12. The device according to claim 11, wherein, The updating module updates the first person instance area and the second person instance area according to the overlapping area after dilation processing, including: Calculate the ratio between the overlapping area after dilation processing and the first person instance area to obtain a first occupancy ratio; Calculate the ratio between the overlapping area after dilation processing and the second person instance area to obtain a second occupancy ratio; If the first occupancy ratio is greater than the second occupancy ratio, divide the overlapping area between the first person instance area and the second person instance area into the first person instance area to obtain the updated first person instance area and the updated second person instance area.

13. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.

Images