Image segmentation method, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image segmentation method, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a to-be-segmented image; carrying out portrait segmentation on the to-be-segmented image by adopting the portrait segmentation model to obtain a portrait segmentation result; obtaining a screened mask pattern, wherein the screened mask pattern is a mask pattern screened based on other segmentation results; combining the screened mask pattern and the portrait segmentation result to obtain a target object segmentation result; based on the target object segmentation result, performing shadow segmentation on the to-be-segmented image by using a shadow segmentation model to obtain a shadow segmentation result; and fusing the shadow segmentation result and the target object segmentation result to obtain a fused segmentation result. According to the method, full-scene segmentation can be realized, the image segmentation accuracy is improved, the image generation quality is further improved, and the user experience is improved.

Description

[0001] This application claims priority to Chinese Patent Application No. 202410855058.8, filed on June 27, 2024, entitled "Image Segmentation Method, Electronic Device and Computer-Readable Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, specifically to an image segmentation method, an electronic device, and a computer-readable storage medium. Background Technology

[0003] With the development of image processing technology, people are able to process images in various ways through electronic devices to generate images that meet their needs.

[0004] For example, when a user takes a photo using an electronic device, if the photo contains an unwanted object, such as an inadvertently captured passerby, the photo will not meet the user's expectations. In this case, the user can operate the electronic device to remove the inadvertently captured passerby portion from the photo, obtaining a photo that meets the user's expectations. Similarly, users can delete other unwanted objects from photos or other images to obtain an image that satisfies their viewing needs. Specifically, the electronic device can, under user operation, segment the object selected by the user to be removed from the original image, retaining the rest of the image, and then fill in the segmented area to form a complete image, thus achieving the operation of removing unwanted objects from the image.

[0005] However, traditional segmentation methods have low segmentation accuracy and are prone to missegmentation, resulting in poor image quality after subsequent filling and affecting user experience. Summary of the Invention

[0006] This application provides an image segmentation method, apparatus, chip, electronic device, computer-readable storage medium, and computer program product that can improve segmentation accuracy.

[0007] In a first aspect, an image segmentation method is provided, comprising: acquiring an image to be segmented; performing image segmentation on the image to be segmented using a human image segmentation model to obtain an image segmentation result; acquiring a filtering mask image, wherein the filtering mask image is a mask image filtered based on other segmentation results, and the other segmentation results are the results of segmenting the image to be segmented; merging the filtering mask image and the human image segmentation result to obtain a target object segmentation result; performing shadow segmentation on the image to be segmented using a shadow segmentation model based on the target object segmentation result to obtain a shadow segmentation result; and fusing the shadow segmentation result and the target object segmentation result to obtain a fused segmentation result.

[0008] Optionally, the image segmentation method in this application can be executed on the cloud side, on the device side, or a combination of cloud and device side.

[0009] The image to be segmented can be an image after the terminal device has resized the initial image to be segmented, or it can be an image without resizing, as long as it meets the size requirements of the portrait segmentation model. Optionally, the portrait segmentation model can also resize the input image of any size to the required size before segmentation.

[0010] The terminal device can send the image to be segmented to a cloud server, which then uses a human face segmentation model to perform human face segmentation on the image, obtaining the segmentation result. Optionally, the human face segmentation result can be output as one or more human face mask images. Each human face mask image can correspond to one person. Optionally, the region of a person in a human face mask image can be a complete shape of a person or an incomplete part of the shape of a person.

[0011] Optionally, if there is no human figure in the image to be segmented, the human figure segmentation result is empty.

[0012] The filtering mask is the result of segmenting the image to be segmented using another model by the cloud testing server. Optionally, the other model can be a preset SAM model or a subject matting model. Correspondingly, the filtering mask can be the segmentation result output by the preset SAM model, the subject segmentation result output by the subject matting model, or a selection mask. The filtering mask can also be the result selected from the outputs of the preset SAM model and the subject matting model, or a selection mask, or a fusion of some or all of the outputs of these two models. There is no limitation here, as long as it is different from the portrait segmentation result.

[0013] This method integrates multiple segmentation results, enabling segmentation across the entire scene, improving the accuracy of image segmentation, thereby enhancing the quality of image generation and improving the user experience.

[0014] In some possible implementations, a human image segmentation model is used to perform human image segmentation on the image to be segmented, and the human image segmentation result is obtained. This includes: using a human image segmentation model to perform human image segmentation on the image to be segmented, obtaining at least one initial human image mask; and performing connected component filtering and validity verification on the at least one initial human image mask to obtain the human image segmentation result.

[0015] Optionally, the cloud-side server can pre-train a portrait segmentation model to obtain a pre-trained portrait segmentation model. This model can accurately segment a portrait mask. The cloud-side server first performs portrait segmentation on the image to be segmented using the portrait segmentation model, obtaining at least one initial portrait mask. Then, the cloud-side server performs connected component filtering on each of the at least one initial portrait mask, thereby filtering out invalid connected components such as small fragments. The cloud-side server also performs validity verification on each of the at least one initial portrait mask, thereby deleting invalid initial portrait masks and retaining valid initial portrait masks as the portrait segmentation result. Through the above connected component filtering and validity verification, the output portrait segmentation result can be ensured to be more accurate.

[0016] In some possible implementations, connected component filtering and validity verification are performed on at least one initial portrait mask to obtain a portrait segmentation result, including: performing connected component filtering on the first initial portrait mask to obtain at least one filtered portrait mask, wherein the first initial portrait mask is any one of the initial portrait masks; and performing validity verification on the first filtered portrait mask to obtain a portrait segmentation result, wherein the first filtered portrait mask is any one of the at least one filtered portrait mask.

[0017] The cloud-based server can perform connected component filtering on each initial portrait mask image. Then, it can perform validity verification on each filtered portrait mask image that has undergone connected component filtering, thereby deleting invalid initial portrait mask images and obtaining the retained portrait mask images as the portrait segmentation result.

[0018] For details on performing connected component filtering and validity verification on the first initial human image mask, please refer to the fourth post-processing and fourth validity verification, which will not be repeated here.

[0019] The connected component filtering operation described above can filter out invalid connected components such as small fragments in the initial portrait mask image. Validity verification removes invalid initial portrait masks, retaining only valid ones, ensuring more accurate portrait segmentation results.

[0020] In some possible implementations, before obtaining the filtering mask image, the method further includes: obtaining a selection mask image, which is a mask image generated based on the selection trajectory input by the user; based on the selection mask image, segmenting the image to be segmented using a preset SAM segmentation model to obtain SAM segmentation results, which are used to characterize the region where the segmented object is located in the image to be segmented; cropping the image to be segmented according to the SAM segmentation results to obtain a main body segmentation image, which includes the region corresponding to the target object segmentation results, and the area corresponding to the main body segmentation image is larger than the area corresponding to the target object segmentation results; segmenting the main body segmentation image using a main body matting model to obtain a main body segmentation result, which is used to characterize the region where the main object is located in the main body segmentation image.

[0021] A selection mask is a mask generated by closing the selection path input by the user. The selection mask can represent the user's selected area.

[0022] The cloud server can use a pre-trained SAM segmentation model to segment the image and obtain the SAM segmentation result. The objects that this pre-trained SAM segmentation model can segment are not limited to people; they can also be other objects.

[0023] The cloud-based server can determine the cropping ratio based on the SAM segmentation results, and then crop the image to be segmented according to the cropping ratio to obtain the main segmented image.

[0024] Optionally, if the SAM segmentation result is empty, the image to be segmented is cropped according to a larger first scaling factor, so that the main segmentation image may include more cropped objects, thus obtaining the main segmentation image; if the SAM segmentation result is not empty, the image to be segmented is cropped according to a smaller second scaling factor, and it is also necessary to ensure that there are segmentable objects in the cropped image, thus obtaining the main segmentation image.

[0025] Afterwards, the cloud server can use a subject matting model to segment the cropped image. The subject matting model is a pre-trained segmentation model capable of recognizing various types of subjects and segmenting the main objects in the subject segmentation image. The main object refers to the clearly dominant object in the subject segmentation image. The main objects that the subject matting model can segment include various categories of objects, such as obvious cabinets, windows, signs, and people.

[0026] The above target object segmentation result is the result of merging the filtered mask image and the portrait segmentation result. Regardless of whether the first ratio or the second ratio is used to crop the image to be segmented, the region corresponding to the obtained subject segmentation image is larger than the region corresponding to the target object segmentation result; that is, the area corresponding to the subject segmentation image is larger than the area corresponding to the target object segmentation result.

[0027] Cropping the main segmentation image based on the SAM segmentation results can reasonably divide the size of the main segmentation image, which ensures that the mask image of the main object can be segmented with a high probability in the main segmentation image, and can also avoid cropping too large a size, resulting in cropping too large an invalid region, which is more reasonable.

[0028] In some possible implementations, before segmenting the image to be segmented using a pre-defined SAM segmentation model based on the selected mask image, the method further includes: performing text detection on the image to be segmented using a text detection model to obtain text detection results.

[0029] The cloud server can first use a text detection model to detect text in the image to be segmented and obtain the text detection results. Then, the cloud server uses a preset SAM segmentation model to segment the image to be segmented, which can ensure that the text is segmented even when the user selects the text region.

[0030] In some possible implementations, the method also includes: if the fusion segmentation result is empty, then determine whether the text detection result is empty; if the text detection result is not empty, then output the text detection result.

[0031] If the above fusion and segmentation result is empty, it means that the area selected by the user likely does not contain any non-text objects that need to be segmented. In this case, the cloud server can check whether the text detection result is empty. If the text detection result is not empty, it means that the user likely needs to segment text, and the text detection result can be output as the segmentation result to facilitate subsequent text segmentation, which matches the user's segmentation intent.

[0032] Optionally, if the text detection result is also empty, it means that no valid object to be segmented was identified in the area selected by the user, and no further segmentation process needs to be performed.

[0033] In some possible implementations, based on the selected mask image, a preset SAM segmentation model is used to segment the image to be segmented to obtain the SAM segmentation result. This includes: inputting the selected mask image and the image to be segmented into the preset SAM segmentation model to obtain at least one initial SAM segmentation mask image; deleting at least one initial SAM segmentation mask image whose predicted score is lower than a first preset score threshold to obtain at least one filtered initial SAM segmentation mask image; and performing connected component filtering and validity verification operations on each of the at least one filtered initial SAM segmentation mask image to obtain the SAM segmentation result.

[0034] The cloud-based server can input the selected mask image and the image to be segmented into a preset SAM segmentation model to obtain at least one initial SAM segmentation mask image. When outputting the initial SAM segmentation mask images, the preset SAM segmentation model also outputs a prediction score for each initial SAM segmentation mask image. This prediction score represents the accuracy of the corresponding initial SAM segmentation mask image (i.e., the confidence level output by the preset SAM segmentation model). A higher prediction score indicates that the preset SAM segmentation model is more confident in the initial SAM segmentation mask image; a lower prediction score indicates that the preset SAM segmentation model is less confident in the initial SAM segmentation mask image.

[0035] The cloud server first deletes at least one initial SAM segmentation mask that has a predicted score lower than a first preset score threshold (threshold B-1), resulting in at least one filtered initial SAM segmentation mask. Then, the cloud server performs connected component filtering and validity verification operations on each of the at least one filtered initial SAM segmentation mask to obtain the SAM segmentation result.

[0036] Optionally, the specific process of the cloud-side server performing connected component filtering operation on each of the at least one filtered initial SAM segmentation mask can be found in the detailed description of the other steps in the second post-processing, except for the step of filtering the initial SAM segmentation mask based on the IOU prediction score, and will not be repeated here.

[0037] The cloud-side server performs validity verification on each image in at least one filtered initial SAM segmentation mask image. For a detailed description of the second validity verification, please refer to the description of the second validity verification. It will not be repeated here.

[0038] The connected component filtering operation described above can filter out invalid connected components such as small fragments in the initial SAM segmentation mask. After validity verification, invalid initial SAM segmentation masks are removed, retaining only valid ones, ensuring more accurate SAM segmentation results in the output.

[0039] In some possible implementations, a subject matting model is used to segment the subject image to obtain the subject segmentation result. This includes: inputting the subject segmentation image into the subject matting model for segmentation to obtain at least one initial subject segmentation mask; and performing connected component filtering and validity verification operations on each of the at least one initial subject segmentation mask to obtain the subject segmentation result.

[0040] The cloud-based server uses a subject matting model to segment the subject image, obtaining at least one initial subject segmentation mask. Each initial subject segmentation mask corresponds to a main object. For details on the connected component filtering and validity verification operations performed by the cloud-based server on each of the at least one initial subject segmentation mask, please refer to the third post-processing operation and the third validity verification operation.

[0041] The above connected component filtering operation can filter out invalid connected components such as small fragments in the initial subject segmentation mask. After validity verification, invalid initial subject segmentation masks are deleted, retaining valid ones, ensuring more accurate subject segmentation results in the output.

[0042] In some possible implementations, the filter mask image is obtained by selecting the filter mask image from the SAM segmentation results, the main body segmentation results, and the selection mask image.

[0043] Optionally, the cloud testing server can select any one from the SAM segmentation result, the main body segmentation result, and the selection mask image as the filter mask image. Optionally, the cloud testing server can also select one of the valid ones one by one according to a certain priority as the filter mask image.

[0044] In some possible implementations, if the subject segmentation result is not empty, the filter mask is the subject segmentation result; if the subject segmentation result is empty and the SAM segmentation result is not empty, the filter mask is the SAM segmentation result; if the subject segmentation result is empty and the SAM segmentation result is empty, the filter mask is the selection mask.

[0045] The cloud-based server can use the main segmentation result as a filter mask when it is not empty, and use the SAM segmentation result as a filter mask when the main segmentation result is empty. If both the main segmentation result and the SAM segmentation result are empty, the selected mask can be used as a fallback filter mask to avoid outputting invalid results when both the main segmentation and SAM segmentation results are empty, ensuring that subsequent processes proceed normally.

[0046] Optionally, the cloud server can also perform validity verification on the filter mask image to obtain a filtered filter mask image, making the filter mask image more accurate. For details, please refer to the detailed description of the fifth validity verification section, which will not be repeated here.

[0047] In some possible implementations, the image to be segmented is cropped according to the SAM segmentation result to obtain the main body segmentation image, including: if the SAM segmentation result is empty, the image to be segmented is cropped according to a first scaling factor to obtain the main body segmentation image; if the SAM segmentation result is not empty, the image to be segmented is cropped according to a second scaling factor to obtain the main body segmentation image; the first scaling factor is greater than the second scaling factor, and both the first scaling factor and the second scaling factor are greater than 1.

[0048] Optionally, if the SAM segmentation result is empty, the image to be segmented is cropped according to a larger first scaling factor, so that the main segmentation image may include more cropped objects, thus obtaining the main segmentation image; if the SAM segmentation result is not empty, the image to be segmented is cropped according to a smaller second scaling factor, and it is also necessary to ensure that there are segmentable objects in the cropped image, thus obtaining the main segmentation image.

[0049] Cropping the main segmentation image based on the SAM segmentation results can reasonably divide the size of the main segmentation image, which ensures that the mask image of the main object can be segmented with a high probability in the main segmentation image, and can also avoid cropping too large a size, resulting in cropping too large an invalid region, which is more reasonable.

[0050] In some possible implementations, the screening mask image and the portrait segmentation result are merged to obtain the target object segmentation result. This includes: if the portrait segmentation result is empty, then the screening mask image is determined to be the target object segmentation result; based on the target object segmentation result, a shadow segmentation model is used to perform shadow segmentation on the image to be segmented to obtain the shadow segmentation result. This includes: inputting the minimum bounding rectangle of the object in the screening mask image and the image to be segmented into the shadow segmentation model for segmentation to obtain the shadow segmentation result.

[0051] In some possible implementations, merging the filter mask image and the portrait segmentation result to obtain the target object segmentation result includes: if the portrait segmentation result is not empty, then fusing the filter mask image and the portrait segmentation result to obtain the target object segmentation result; based on the target object segmentation result, using a shadow segmentation model to perform shadow segmentation on the image to be segmented to obtain the shadow segmentation result, then includes: inputting the portrait mask image in the target object segmentation result and the image to be segmented into the shadow segmentation model to perform shadow segmentation to obtain the shadow segmentation result.

[0052] If there are no people in the image to be segmented, the image segmentation result is likely to be empty. In this case, the cloud server can use the filter mask as the target object segmentation result. Then, the cloud server can use the minimum bounding rectangle of the object in the filter mask as a cue word, and input it along with the image to be segmented into the shadow segmentation model for segmentation to obtain the shadow segmentation result.

[0053] If the image to be segmented contains a person, the portrait segmentation result is unlikely to be empty. In this case, the cloud server can fuse the filtered mask image and the portrait segmentation result to obtain the target object segmentation result. Then, the cloud server can use the minimum bounding rectangle of the portrait in the target object segmentation result as a cue word, and input it along with the image to be segmented into the shadow segmentation model for segmentation, obtaining the shadow segmentation result. This shadow segmentation result can include the negative values ​​corresponding to the portrait, ensuring the accuracy of the segmentation result.

[0054] In some possible implementations, based on the target object segmentation result, a shadow segmentation model is used to perform shadow segmentation on the image to be segmented to obtain the shadow segmentation result. This includes: inputting the target object segmentation result and the image to be segmented into the shadow segmentation model to obtain an initial shadow segmentation result, which includes at least one initial shadow mask image; deleting at least one initial shadow mask image whose predicted score is lower than a second preset score threshold to obtain at least one filtered initial shadow mask image; and performing a connected component filtering operation on each of the at least one filtered initial shadow mask image to obtain the shadow segmentation result.

[0055] A detailed description of performing connected component filtering on each of at least one of the initial filtered shadow mask images can be found in the description of the fifth post-processing, which will not be repeated here.

[0056] The above connected component filtering operation can filter out invalid connected components such as small fragments in the initial shadow segmentation mask. After validity verification, invalid initial shadow segmentation masks are removed, while valid ones are retained, ensuring more accurate shadow segmentation results in the output.

[0057] In some possible implementations, the connected component filtering operation includes: acquiring a plurality of connected components to be filtered in a first mask image to be filtered, wherein the first mask image to be filtered is any one of at least one mask image to be filtered; determining whether a first ratio of the area of ​​the first connected component to be filtered to the area of ​​the mask image to be filtered is greater than or equal to a first threshold, wherein the first connected component to be filtered is any one of the plurality of connected components to be filtered; if the first ratio is greater than or equal to the first threshold, then retaining the first connected component to be filtered; if the first ratio is less than the first threshold, then deleting the first connected component to be filtered.

[0058] Obtain the first largest connected component with the largest area among multiple connected components to be filtered; determine whether the second ratio of the area of ​​the second connected component to be filtered to the area of ​​the first largest connected component is greater than or equal to a second threshold, wherein the second connected component to be filtered is any one of the multiple connected components to be filtered that is retained; if the second ratio is greater than or equal to the second threshold, the second connected component to be filtered is retained; if the second ratio is less than the second threshold, the second connected component to be filtered is deleted.

[0059] Obtain the sum of the areas of the connected components retained in the first mask to be verified; determine whether the third ratio of the sum of the areas of the retained connected components to the area of ​​the first mask to be verified is greater than or equal to a third threshold; if the third ratio is greater than or equal to the third threshold, retain the first mask to be verified; if the third ratio is less than the third threshold, delete the first mask to be verified.

[0060] Obtain the width and height of the first connected component in the second mask image to be verified. The second mask image to be verified is any one of the mask images to be verified retained from at least one mask image to be filtered. If the fourth ratio of the width of the first connected component to the width of the second mask image to be verified is greater than or equal to a fourth threshold, and / or the fifth ratio of the height of the first connected component to the height of the second mask image to be verified is greater than or equal to a fifth threshold, then the second mask image to be verified is retained. If the fourth ratio of the width of the first connected component to the width of the second mask image to be verified is less than a fourth threshold, and the fifth ratio of the height of the first connected component to the height of the second mask image to be verified is less than a fifth threshold, then the second mask image to be verified is deleted.

[0061] The aforementioned mask image to be filtered can be any of the mask images for which connected component filtering operations need to be performed, such as the initial SAM segmentation mask image, the initial shadow segmentation mask image, or the initial subject segmentation mask image. The aforementioned threshold can also be different values ​​when the mask image to be filtered is a different segmentation mask image; this will not be elaborated upon here.

[0062] The above connected component filtering operation can filter out small invalid connected components and retain the valid connected components, making the segmentation results more accurate.

[0063] In some possible implementations, the validity verification operation includes: obtaining a first intersection of multiple connected components to be verified and a selected region in a first mask image to be verified, wherein the first mask image to be verified is any one of at least one mask image to be verified; calculating a sixth ratio of the first intersection to the sum of the areas of the multiple connected components to be verified, and calculating a seventh ratio of the first intersection to the area of ​​the selected region; if the sixth ratio is greater than or equal to a sixth threshold, and / or the seventh ratio is greater than or equal to a seventh threshold, then retaining the first mask image to be verified; if the sixth ratio is less than the sixth threshold, and the seventh ratio is less than the seventh threshold, then deleting the first mask image to be verified.

[0064] The aforementioned mask image to be verified can be any mask image that requires validity verification, such as any one of the following: the initial SAM segmentation mask image after connected component filtering, the initial shadow segmentation mask image, or the initial subject segmentation mask image. The threshold value can also be different depending on whether the mask image to be verified is a different segmentation mask image; this will not be elaborated upon here.

[0065] After the above validity verification operation, invalid segmentation mask images can be deleted, and valid segmentation mask images can be retained, making the segmentation results more accurate.

[0066] In some possible implementations, the shadow segmentation result and the target object segmentation result are fused to obtain a fused segmentation result, including: obtaining a second intersection of a first target segmentation mask and a first shadow segmentation mask, where the first target segmentation mask is any mask in the target object segmentation result and the first shadow segmentation mask is any mask in the shadow segmentation result; obtaining a third ratio of the area of ​​the second intersection to the area of ​​the target object in the first target segmentation mask; if the third ratio is greater than or equal to a third threshold, then the union of the first target segmentation mask and the first shadow segmentation mask is determined to be the fused segmentation result; if the third ratio is less than the third threshold, then the first target segmentation mask is determined to be the fused segmentation result.

[0067] The fusion process can be found in the description of step S713, and will not be repeated here. The cloud server uses this method to fuse the shadow segmentation results and the target object segmentation results. It can delete shadow segmentation mask images with low shadow area matching degree, retain the shadow segmentation mask images with high shadow area matching degree, and fuse them with the target object segmentation results, so that the fused segmentation results are more accurate.

[0068] In some possible implementations, the method also includes: post-processing the fused segmentation result to obtain an output segmentation result, wherein in the output segmentation result, there are no pixels with a pixel value of 0 inside the region where the pixel value of 1 is distributed.

[0069] The cloud-based server can then perform post-processing on the fused segmentation results, namely hole filling, to obtain a distribution area where there are no pixels with a value of 0, i.e., a mask map without holes. This avoids subsequent invalid processing steps caused by fragmented holes and saves computing resources.

[0070] In some possible implementations, the fused segmentation result is post-processed to obtain the output segmentation result, including: if the fused segmentation result is not empty, then the fused segmentation result is post-processed to obtain the output segmentation result.

[0071] If the cloud-side server determines that the fusion and segmentation result is not empty, it will perform post-processing on the fusion and segmentation result. Otherwise, if the fusion and segmentation result is empty, no post-processing is required, avoiding invalid post-processing operations and saving computing resources.

[0072] In some possible implementations, the fused segmentation result is post-processed to obtain the output segmentation result, including: obtaining the outer contour of the region where the pixel with a pixel value of 1 is located in the fused segmentation result; updating the pixel value of the pixel with a pixel value of 0 within the outer contour to 1, and obtaining the output segmentation result.

[0073] Specifically, the method for hole filling can be as follows: calculate the outline of the mask image of the fused segmentation result, and update the pixel values ​​of the mask image with a pixel value of 0 to 1 for the holes in the outline, that is, the areas in the mask image with a pixel value of 1, so as to obtain a mask image without holes.

[0074] In some possible implementations, the method further includes: receiving a selection operation input by the user, wherein the selection operation is the operation of selecting the target segmentation object in the initial image to be segmented, and the initial image to be segmented is the image before the size transformation of the image to be segmented; in response to the selection operation, displaying an animation effect in the area where the target segmentation object and the target shadow are located in the initial image to be segmented, wherein the target shadow is the shadow generated by the target segmentation object in the initial image to be segmented.

[0075] When a user selects an area, the terminal device can recognize the target object to be segmented and the corresponding target shadow, and display animation effects in the corresponding area to indicate to the user that the segmentation process is in progress. This allows the user to have a clear perception of the segmentation process and improves the user experience.

[0076] In a second aspect, an image segmentation apparatus is provided, comprising a unit consisting of software and / or hardware, the unit being used to perform any one of the methods in the technical solution of the first aspect.

[0077] Thirdly, embodiments of this application provide a chip including a processor; the processor is used to read and execute a computer program stored in a memory to perform any one of the methods described in the first aspect.

[0078] Optionally, the chip further includes a memory, which is connected to the processor via a circuit or wire.

[0079] Alternatively, the chip may further include a communication interface.

[0080] Fourthly, an electronic device is provided, comprising: a processor, a memory, and an interface; the processor, memory, and interface cooperate with each other to enable the electronic device to perform any one of the methods described in the first aspect.

[0081] Alternatively, the electronic device may be a terminal device or a server.

[0082] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the processor performs any one of the methods described in the first aspect.

[0083] In a sixth aspect, a computer program product is provided, the computer program product comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform any one of the methods described in the first aspect.

[0084] In a seventh aspect, a system is provided, the system comprising a cloud-side server and a terminal device, the cloud-side server being used to execute the method executed by the cloud-side server in the first aspect, and the terminal device being used to execute the method executed by the terminal device in the first aspect. Attached Figure Description

[0085] Figure 1 This is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of this application;

[0086] Figure 2 This is a software structure block diagram of the terminal device 100 provided in the embodiments of this application;

[0087] Figure 3 This is a schematic diagram of an example gallery interface provided in an embodiment of this application;

[0088] Figure 4 This is a schematic diagram illustrating an example of entering the editing interface from the image preview interface provided in an embodiment of this application;

[0089] Figure 5 This is a schematic diagram of the interface and gestures of a user selecting a target in an intelligent selection mode provided in an embodiment of this application;

[0090] Figure 6 This is a comparison image before and after intelligent elimination provided in an embodiment of this application;

[0091] Figure 7 This is a schematic diagram of the overall process of image segmentation provided in an embodiment of this application;

[0092] Figure 8 This is a comparison diagram of the selected trajectory before and after closure, provided in an embodiment of this application;

[0093] Figure 9 This is an example of a selection mask image provided in an embodiment of this application;

[0094] Figure 10 This is a schematic diagram illustrating an example of image resizing and filling provided in an embodiment of this application;

[0095] Figure 11 This is a schematic diagram illustrating the recognition process of a text mask image provided in an embodiment of this application;

[0096] Figure 12 This is another example of a text mask image recognition process provided in the embodiments of this application;

[0097] Figure 13 This is a schematic diagram of the minimum bounding rectangle of a selected area provided in an embodiment of this application;

[0098] Figure 14 This is a schematic diagram of an example of a clipping area provided in an embodiment of this application;

[0099] Figure 15 This is a schematic diagram illustrating a cropped main body segmentation image provided in an embodiment of this application;

[0100] Figure 16 This is a schematic diagram of the minimum bounding rectangle of a selected area provided in an embodiment of this application;

[0101] Figure 17 This is a schematic diagram of an example of a human image segmentation mask provided in an embodiment of this application;

[0102] Figure 18 This is a schematic diagram of a shadow segmentation mask image obtained by segmentation according to an embodiment of this application;

[0103] Figure 19 This is another schematic diagram of obtaining a shadow segmentation mask image through segmentation, provided in an embodiment of this application;

[0104] Figure 20 This is a flowchart of an image segmentation method provided in an embodiment of this application;

[0105] Figure 21 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of this application. Detailed Implementation

[0106] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0107] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0108] With the development of image processing technology, people can use electronic devices to perform various processing operations on images, generating images that meet their needs. For example, when a user takes a photo using an electronic device, if the photo contains an unwanted object, such as a passerby accidentally captured during the shooting process, the photo will not meet the user's expectations. In this case, the user can operate the electronic device to remove the accidentally captured part of the passerby from the photo, obtaining a photo that meets the user's expectations. Similarly, users can also delete other unwanted objects from photos or other images to obtain an image that meets their viewing requirements. The electronic device can, under user operation, segment the object selected by the user to be removed from the original image, retaining the rest of the image. Then, the electronic device fills in the area containing the segmented object, forming a complete image, thus achieving the operation of removing unwanted objects from the image. However, traditional segmentation methods have low segmentation accuracy and are prone to missegmentation, resulting in poor image quality after subsequent filling.

[0109] The image segmentation method provided in this application can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.

[0110] For example, Figure 1This is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of this application. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0111] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0112] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0113] The software system of terminal device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of terminal device 100.

[0114] Figure 2This is a software structure block diagram of the terminal device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0115] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0116] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0117] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0118] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0119] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0120] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0121] The phone manager is used to provide communication functions for terminal device 100. For example, it manages call status (including connection, hang-up, etc.).

[0122] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0123] The notification manager allows applications to display notification information in the status bar. It can be used to convey informational messages and can disappear automatically after a short time without user interaction.

[0124] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0125] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0126] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0127] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0128] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0129] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0130] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0131] A 2D graphics engine is a graphics engine for 2D drawing.

[0132] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0133] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 1 and Figure 2 Taking the terminal device with the structure shown as an example, and in conjunction with the accompanying drawings and application scenarios, the image segmentation method provided in this application embodiment will be specifically described.

[0134] After the user unlocks the terminal device, the terminal device can display as follows: Figure 3 The desktop shown in Figure a is an example. The desktop includes icons for multiple applications. Users can click different icons to open different applications. It should be noted that the desktop also includes a gallery icon. Users can... Figure 3As shown in Figure a, clicking the gallery icon opens the gallery. In response to the user opening the gallery, the terminal device can display something like... Figure 3 Figure b shows the gallery interface. The gallery interface includes thumbnails of multiple images.

[0135] Users can click on the thumbnail of the image they want to view in the gallery interface. See below for details. Figure 3 The operation is shown in diagram b. It responds to the user's... Figure 3 In the image gallery interface shown in Figure b, when selecting an image, the terminal device can display something like... Figure 4 Figure a shows the preview interface of the image. The image preview interface can also include multiple operation controls, such as: share control, favorite control, edit control, delete control, and menu control (e.g., more).

[0136] When users are like Figure 4 In the preview interface shown in Figure a, clicking the share control allows you to share the image; clicking the favorite control adds the image to your favorites; clicking the delete control deletes the current image; and clicking more controls displays a menu bar, allowing you to perform other operations on the image—details omitted here.

[0137] When users are like Figure 4 When the edit control is clicked in the preview interface shown in Figure a, the terminal device can enter the edit state and display the edit interface. The display is as follows: Figure 4 The editing interface is shown in Figure b. See also... Figure 4 Image b in the image includes several operation controls in the editing interface, such as: doodle control, text control, smart removal control, mosaic control, and watermark control. Users can click on different operation controls in the editing interface to perform different operations. When the user clicks the removal control (AI removal) in the editing interface, the terminal device displays something like this. Figure 4 The elimination interface is shown in Figure b. The elimination interface also includes various elimination controls, such as a smart selection control and a manual smear control.

[0138] In the elimination interface, if the user clicks the manual smudge control, the terminal device enters manual smudge mode, where the user can use their finger to smudge back and forth on the interface to eliminate the areas that need to be erased from the image. If the user clicks the smart selection control, the terminal device enters smart selection mode (also known as the elimination pen). In smart selection mode, the user can circle the object to be eliminated on the screen, and the terminal device can intelligently recognize the circled object and eliminate it.

[0139] Users can also... Figure 5 In the elimination interface shown in Figure a, clicking the "Back" control cancels the previous operation, while clicking the "Next" control executes the canceled operation. After the user completes the selection, they can click the "Save" control to save the newly generated image after the elimination.

[0140] This application primarily describes the process of segmenting objects selected by the user in intelligent selection mode. In intelligent selection mode, the user can draw circles on the screen to select objects to be eliminated; for details, please refer to... Figure 5 The operation is shown in Figure a. In Figure 5 In the interface shown in Figure a, the user can draw a circle in the direction indicated by the dotted arrow, and the trajectory of the user's circle can also be seen as shown by the dotted line. When the user completes the circle drawing operation (which can also be called a selection operation), the trajectory of the user's finger can be called the selection trajectory. For example, the selection trajectory can be seen as follows... Figure 5 The dashed line in diagram b is shown in the figure.

[0141] Once the user completes the selection, the terminal device responds by segmenting the selected object from the original image. The terminal device then fills in the segmented area to form a complete image. The object previously selected by the user will no longer be displayed in the filled, complete image.

[0142] When users take photos in complex environments, such as scenic spots, the surrounding environment and numerous tourists often interfere with the main people or scenery they want to capture, resulting in photos containing many unwanted objects. In such cases, the types of objects the user needs to remove become more complex. For example, the photo might include unwanted pedestrians, vehicles, signs, trash cans, buildings, and so on. To effectively remove these complex types of objects, the user-selected objects need to be precisely segmented.

[0143] This application provides an image segmentation method that can process selected regions according to different scenes and fuse the segmentation results obtained from different scenes. Therefore, regardless of whether the region selected by the user contains a person or other objects, the terminal device can perform accurate segmentation. After segmentation, the terminal device can also fill in the regions where the eliminated objects were located. The more accurate the segmentation of the object, the higher the quality of the filled image.

[0144] After the user completes the selection operation, the terminal device can display animations of the objects to be segmented and eliminated while recognizing and removing the selected objects. Optionally, the animations displayed by the terminal device could highlight the objects circled by the user, or flash the objects circled by the user, for example... Figure 6 As shown in Figure a, the animation of the eliminated object serves to indicate to the user which object has been selected and what content is to be eliminated. This allows the user to perceive the elimination operation and alleviates the user's waiting time, improving the user experience. The terminal device can also display prompts such as "Eliminating..." on the interface to indicate that elimination is in progress. This application embodiment does not limit the display method of the animation of the eliminated object. After elimination is completed, the terminal device can display as follows: Figure 6 The new image shown in Figure b no longer contains the object selected by the user. Optionally, after removing the object selected by the user, the terminal device can also fill the area where the removed object was located based on the texture of the background around the object, thereby generating a new complete background image, making the image after removing the selected object more complete and realistic, and improving the image quality.

[0145] Next, this application embodiment will describe in detail the process of segmenting objects within a user's selection trajectory. For details, please refer to... Figure 7 As shown, it includes:

[0146] S701. Obtain the initial selection trajectory input by the user.

[0147] In intelligent selection mode, when a user circles the object to be eliminated on the screen, that is, when the user performs a selection operation, the terminal device responds to the user's input selection operation and recognizes the trajectory of the user's finger swipe (also known as the initial selection trajectory).

[0148] S702. Close the initial selection trajectory to form a closed selection trajectory.

[0149] Optionally, if the initial selection trajectory of the user's finger swipe is an open curve, for example... Figure 8 As shown in Figure a, the terminal device can perform a closing operation on an unclosed initial selection trajectory. Optionally, the closing operation can be performed by the terminal device connecting the two endpoints of the unclosed initial selection trajectory with line segments to obtain a closed selection trajectory. Figure 8 After the initial selection trajectory shown in Figure a is closed, the corresponding closed selection trajectory can be found in Figure 1. Figure 8 As shown in Figure b.

[0150] Alternatively, if the initial selection trajectory is a closed curve, the terminal device may also directly use the initial selection trajectory as the closed selection trajectory without performing a closing operation on the initial selection trajectory.

[0151] S703. Fill the closed selection trajectory to obtain the initial selection mask image.

[0152] Next, the terminal device can fill the closed selection trajectory to generate an initial selection mask image (also called an initial selection mask image). It should be noted that this initial selection mask image is a black and white mask image. In this initial selection mask image, within the region inside the closed selection trajectory, each pixel has a pixel value of 1 (corresponding to white, also called foreground pixels), while pixels in other regions have a pixel value of 0 (corresponding to black, also called background pixels). For example, based on the above... Figure 8 Based on the image shown in Figure b and the initial selection trajectory, the terminal device can generate, for example, [the following]. Figure 9 The initial selection mask image shown.

[0153] Alternatively, in some cases, the pixel value of the white foreground pixel can also be 255, and the pixel value of the black background pixel can be 0.

[0154] S704. Perform size transformation on the initial image to be segmented and the initial selection mask image to obtain the size-transformed image to be segmented and the size-transformed selection mask image.

[0155] For ease of understanding, the original image of the object to be eliminated can be called the initial image to be segmented. When the size of the initial image to be segmented does not meet the input conditions of the subsequent model, the terminal device can first perform a size transformation operation (also called a resize operation) on the initial image to be segmented to generate an image of the size required by the subsequent model. The following example, using an image size requirement of 1024*1024, illustrates the specific implementation process of the size transformation operation:

[0156] The terminal device can proportionally scale the initial image to be segmented to an image with a long side size of 1024, thus generating a resized image to be segmented. This resized image is then transmitted to a cloud server, where it performs further processing.

[0157] For example, when the initial image to be segmented is a rectangle with dimensions of 2048*768, that is, the longer side of the initial image to be segmented is 2048 and the shorter side is 768, see [link to documentation]. Figure 10 As shown in Figure a ( Figure 10(Only the image size change is shown as an example; the content of the image is not shown again.) The terminal device needs to reduce the size of the long side from 2048 to 1024, which means the long side of the image to be segmented is reduced by a factor of 0.5. The terminal device can proportionally reduce the initial image to be segmented by a factor of 0.5. The size of the long side of the reduced image to be segmented is 1024, and the short side is also proportionally reduced by a factor of 0.5, that is, the short side is reduced from 768 to 384. It can be seen that the size of the proportionally reduced image to be segmented is 1024*384, for example... Figure 10 As shown in Figure b.

[0158] It should be noted that if the proportionally scaled-down image to be segmented is not square, when the model requires a square image as input, the proportionally scaled-down image can be filled along the shorter side to form a square before being input into the model. Optionally, the proportionally scaled image (i.e., the image with a length of 1024) can be sent from the terminal device to the cloud server. The cloud server can then resize the image again before inputting it into different models, proportionally scaling the longer side from 1024 to the size required by the input model (e.g., 836), and then filling along the shorter side to form the square size required by the model. If the required size of the input model is 1024*1024, the cloud server does not need to scale the proportionally scaled image to be segmented sent by the terminal device again; it only needs to fill along the distribution direction of the shorter side. For example, Figure 10 Image b in the image is an image with a long side of 1024 and a short side of 384. The cloud server can pad the short side from 384 to 1024, resulting in a 1024*1024 image. Figure 10 As shown in Figure c.

[0159] It should be noted that, in the embodiments of this application, the operation of padding the short side of the image to 1024 can also be performed after other operations, such as padding the short side of the image after post-processing the segmentation result, which will not be elaborated here.

[0160] Optionally, when the cloud server fills the image, it can fill it along both directions of the short side, or it can fill it along one of the directions of the short side. This application embodiment does not limit this. Figure 10 The c-graph in the example is filled in one direction along the shorter side.

[0161] For example, when the initial image to be segmented is a rectangle with dimensions of 512*256, meaning the longer side is 512 and the shorter side is 256, the terminal device needs to enlarge the longer side from 512 to 1024, effectively doubling the size of the longer side. The terminal device can proportionally enlarge the initial image to be segmented to twice its original size. The enlarged image will then have a longer side of 1024, and the shorter side will also be proportionally doubled, from 256 to 512. In other words, the enlarged image will be 1024*512. This enlarged image can then be sent from the terminal device to the cloud server. However, since the enlarged image does not meet the 1024*1024 size requirement, the cloud server can then pad along the shorter side of the image, increasing the shorter side from 512 to 1024, resulting in a 1024*1024 image. The padded image is called the preprocessed image to be segmented, which can be used as input data for the model. The cloud server can also resize and padded the input image again according to the size requirements of the model to be input. For details, please refer to the relevant description above, which will not be repeated here.

[0162] Terminal devices and cloud servers can also perform size transformation operations on the initial selection mask image to obtain a size-transformed selection mask image. Specific size transformation methods can be found in the previous description of size transformation operations on the image to be segmented, and will not be repeated here.

[0163] S705. Use a text detection model to perform text detection on the image to be segmented and obtain the text detection results.

[0164] Optionally, the text detection model requires an input image size of 960*960. After receiving the image to be segmented from the terminal device, the cloud-side server can perform a first preprocessing step. The first preprocessing step includes: scaling the entire image proportionally to a scale where the longer side is transformed to 960, and then padding to obtain a 960*960 image.

[0165] The cloud server normalizes the pre-processed image to be segmented (960*960 pixels) to obtain a normalized image to be segmented. This image normalization can be performed by normalizing the pixel value of each pixel in the image. Then, the cloud server inputs the normalized image to be segmented into a text detection model for text detection.

[0166] Optionally, the cloud server does not restrict the order in which the image to be segmented is normalized, resized, and filled; these three steps can be rearranged arbitrarily, which will not be elaborated here.

[0167] Optionally, this section uses a single pixel as an example to describe the specific process of pixel value normalization: the pixel value of this pixel is denoted as `resize_image`. The method by which the cloud server normalizes the pixel color value of this pixel is expressed by the following formula:

[0168] resize_image-1=(resize_image-mean) / std.

[0169] Where std = [S1] is the standard deviation of the pixel values ​​of the pixels in the image; mean = [M1] is the average value of the pixel values.

[0170] The cloud-side server calculates resize_image-1 = (P1-M1) / S1.

[0171] Optionally, image normalization can also involve normalizing the three-channel color values ​​of each pixel. Specifically, the cloud server normalizes the three-channel color values ​​of the resized and padded image to be segmented (960*960 pixels) to obtain a normalized image to be segmented. This normalized image is then input into a text detection model for text detection.

[0172] Optionally, this section uses a single pixel as an example to describe the specific process of normalizing the three-channel color values ​​of a pixel:

[0173] The three-channel color values ​​of this pixel are denoted as resize_image = [P1, P2, P3]. The cloud server normalizes the three-channel color values ​​of this pixel, expressed by the following formula:

[0174] resize_image-1=(resize_image-mean) / std.

[0175] Where std = [S1, S2, S3], mean = [M1, M2, M3], and std and mean are preset values.

[0176] The resize_image-1 calculated by the cloud server is [(P1-M1) / S1, (P2-M2) / S2, (P3-M3) / S1].

[0177] Optionally, S1 can be 59.286, S2 can be 58.12, S3 can be 58.7, M1 can be 124.785, M2 can be 117.38, and M3 can be 184.589. std and mean are preset values ​​and can also be other values, which are not limited here.

[0178] Optionally, the text detection model can be an Optical Character Recognition (OCR) model. The text detection model can detect text regions in the normalized image to be segmented, obtaining an initial text line mask. Figure 11 For example, the cloud server will Figure 11 The image to be segmented, shown in Figure a, after being resized and normalized, is input into a text detection model for text detection, and the output can be as follows: Figure 11 The initial text line mask is shown in Figure b. It can be seen that the brand name on the person's clothing and the text label for attraction B can both be detected as corresponding text lines, and are reflected in the initial text line mask image. Optionally, the text detection model can detect various fonts, including handwritten text and printed text, including Chinese characters, English letters, and numbers.

[0179] Next, the cloud server can perform a first post-processing step on the initial text line mask image. This first post-processing involves extracting connected components. Specifically, the cloud server performs polygon fitting on the connected regions of the initial text line mask image to obtain the final text line mask image. Figure 11 After performing polygon fitting on the initial text line mask image shown in Figure b, we can obtain the following: Figure 11 The text line mask diagram shown in Figure c is shown in the figure. Figure 11 In diagram c, the outer contour of the text line mask is composed of multiple short line segments, not as shown. Figure 11 In the image b, the outer contour of the text lines is formed by curves. It should be noted that if the text in the image to be segmented has multiple lines, then there will be multiple white areas in the detected text line mask image, meaning that each line of text corresponds to a connected component.

[0180] Next, the cloud server can perform a first validity verification on the text line mask image obtained after the first post-processing. Before this, the cloud server resizes the received circled mask image again, for example, to a size of 960*960. The cloud server performs a first validity verification on the connected components corresponding to each line of text. Taking one connected component as an example, denoted as connected component 1, the cloud server calculates the intersection of the area of ​​connected component 1 and the area of ​​the circled region in the circled mask image (the circled mask image after size transformation, for example, 960*960), and calculates the ratio of the intersection to the area of ​​connected component 1, denoted as ratio A-1. If ratio A-1 is greater than or equal to a preset threshold A-1, it means that connected component 1 is valid and can be retained; if ratio A-1 is less than the preset threshold A-1, it means that connected component 1 is invalid and can be discarded. Optionally, the threshold A-1 can be a value such as 0.7, 0.8, or other values.

[0181] When the user inputs the selection trajectory as follows Figure 12 When the curve in graph a is shown, the corresponding selected mask graph can be found as follows: Figure 12 As shown in Figure c. Figure 12 The white area in graph c is the selected area. Taking a threshold A-1 of 0.7 as an example, in... Figure 12 In the text line mask diagram shown in Figure b, the ratio of the area of ​​the intersection of connected component 1-1 and the selected mask to the area of ​​connected component 1-1 is 0.2 (less than 0.7), indicating that only 20% of the area of ​​connected component 1-1 falls within the selected mask. The probability that connected component 1-1 matches the user's selection intention is low, so the cloud server discards connected component 1-1.

[0182] The ratio of the intersection area of ​​connected region 1-2 and the selected mask to the area of ​​connected region 1-2 is 0.8 (greater than 0.7), indicating that 80% of the connected region falls within the selected mask. Connected region 1-2 is likely to match the user's selection intent, so the cloud server retains connected region 1-2.

[0183] The ratio of the area of ​​the intersection of connected region 1-3 and the selected mask to the area of ​​connected region 1-3 is 0.75 (greater than 0.7), indicating that 75% of the area of ​​the third connected region falls within the selected mask. Connected region 1-3 has a high probability of matching the user's selection intention, so the cloud server retains connected region 1-3.

[0184] Next, the cloud server can count all the remaining connected components (e.g., connected component 1-2 and connected component 1-3) (e.g.) Figure 12 The number of pixels in the intersection A (the overlapping part of the connected component and the selected region) of the region (as shown in Figure d) is denoted as Quantity 1. The cloud server calculates the ratio of Quantity 1 to the number of pixels in the selected region, denoted as Ratio A-2. It should be noted that the number of pixels in a connected component can be represented by the area of ​​that connected component. Therefore, Ratio A-2 can also be described as the ratio of the area of ​​intersection A to the area of ​​the selected region.

[0185] When the ratio A-2 is greater than or equal to the threshold A-2, it indicates that the user's intention is to select all the text corresponding to the remaining connected components, and thus all the remaining connected components are retained. Optionally, the threshold A-2 can be a value such as 0.25, 0.35, 0.4, or other values.

[0186] When the ratio A-2 is less than the threshold A-3, it indicates that the user's intention is not to select the text corresponding to all the remaining connected components. Therefore, this detection is invalid, and all the remaining connected components can be deleted. In other words, if the text detection result is empty or invalid, and the text lines represented by the remaining connected components are not the objects the user intended to select, then all the remaining connected components can be deleted. Optionally, the threshold A-3 can be a value less than the threshold A-2, such as 0.1, 0.12, 0.15, or other values.

[0187] Optionally, if the text detection result is valid, meaning that the aforementioned retained connected components exist, the terminal device determines the text detection result as the final segmentation result. If the text detection result is invalid, meaning that no retained connected components exist, i.e., no text is recognized in the user-selected area, the subsequent process can be executed.

[0188] S706. Based on the selected mask image, the SAM model is used to segment the image to be segmented, and the first segmentation result is obtained.

[0189] Specifically, based on the selected mask image, the process of segmenting the image to be segmented using the SAM model can include four processes: second preprocessing, first inference processing, second postprocessing, and second validity verification.

[0190] Second preprocessing:

[0191] The cloud-based server can normalize the image to be segmented, obtaining a normalized image. The method for normalizing the image is as follows: the cloud-based server normalizes the pixel color value of each pixel, expressed by the following formula:

[0192] resize_image-1=(resize_image-mean) / std.

[0193] Where std = [S1] is the standard deviation of the pixel values ​​of the pixels in the image; mean = [M1] is the average value of the pixel values.

[0194] The resize_image-1 obtained by the cloud server is (P1-M1) / S1. The normalization process can be found in the relevant description of the first preprocessing step, and will not be repeated here.

[0195] Optionally, the SAM model requires an input image size of 1024*1024. After receiving the image to be segmented (with its longer side already transformed to length 1024) and the selection mask image, the cloud server fills both the image to be segmented (with its longer side transformed to length 1024) and the selection mask image along their respective shorter sides to a size of 1024*1024, thus performing a second size transformation. Next, the cloud server can obtain the minimum bounding rectangle of the selection region in the second-sized selection mask image. For example, for... Figure 9 For the selected mask diagram shown, the case where the cloud server obtains the minimum bounding rectangle of the selected area can be found in the example below. Figure 13 As shown.

[0196] First reasoning process:

[0197] The cloud-side server inputs the pre-processed image to be segmented into the trained SAM model, using the minimum bounding rectangle of the selected region as a cue word for inference, to obtain a preliminary inference result (i.e., the initial SAM segmentation result). Specifically, the cloud-side server uses the image encoder in the SAM model to encode the pre-processed image to be segmented, obtaining the encoded image result, also known as the feature vector of the image to be segmented. The cloud-side server uses the cue encoder to encode the coordinates of the diagonal of the minimum bounding rectangle of the selected region in the selected mask image, obtaining the cue encoding result (or cue vector).

[0198] After encoding, the cloud-based server can input the encoded prompts and the encoded image to be segmented into the decoder in the SAM model for decoding, thereby generating preliminary inference results. It should be noted that the preliminary inference results can be output as multiple prediction mask images. These multiple prediction mask images represent inference results at different levels. For example, one prediction mask image represents the segmentation result of the entire body of a person, one represents the segmentation result of the head, one represents the segmentation result of the glasses, and another represents the segmentation result of an object on the person's body.

[0199] In the process of outputting the predicted mask map, the SAM model also outputs the IOU prediction score (iou_score) for each predicted mask map. The higher the IOU prediction score, the more confident the SAM model is in its output prediction results.

[0200] It should be noted that the SAM model can be obtained by inputting multiple sets of sample pairs into an initial SAM model for training. Each sample pair includes a training image and a corresponding labeled mask image. Within the same sample pair, the labeled mask image is the precisely labeled mask image of the object to be segmented in its corresponding training image.

[0201] Second post-processing:

[0202] First, the cloud server can filter out prediction mask images with IOU prediction scores below threshold B-1, obtaining other prediction mask images with IOU prediction scores greater than or equal to threshold B-1. Optionally, threshold B-1 can be a value such as 0.68, 0.75, 0.8, or other values.

[0203] Taking one of the remaining predicted mask images after filtering—the first predicted mask image—as an example, the cloud server converts the pixel values ​​in the first predicted mask image to be between 1 and 255, and performs binarization based on the converted pixel values ​​according to a threshold B-2. For example, after the pixel values ​​in the first predicted mask image are converted to 1-255, if the pixel value of a pixel is greater than or equal to the threshold B-2, then the pixel value of that pixel is set to 1; if the pixel value of a pixel is less than the threshold B-2, then the pixel value of that pixel is set to 0, thus obtaining the binarized image corresponding to the first predicted mask image. Optionally, the threshold B-2 can be a value such as 50, 52, 54, or other values; this embodiment of the application does not limit this.

[0204] In this binarized image, multiple adjacent pixels with a value of 1 form a connected component. Optionally, multiple connected components may exist in the binarized image.

[0205] Next, the cloud server filters the prediction results filtered according to threshold B-1 (i.e., the remaining prediction mask images after filtering according to threshold B-1) according to conditions A-1 to A-4. The following explanation uses any second prediction mask image from the remaining prediction mask images after filtering according to threshold B-1 as an example.

[0206] Condition A-1:

[0207] The second prediction mask map may be the same as the first prediction mask map, or it may be a different prediction mask map. The second prediction mask map includes one or more connected components. Taking one of these connected components, B-1, as an example, the cloud server calculates the ratio of the area of ​​connected component B-1 to the area of ​​the image to be segmented, denoted as the ratio B-1. Optionally, when the size of the image to be segmented input to the SAM model is 1024*1024, the area of ​​the image to be segmented is 1024*1024=1048576.

[0208] The cloud-side server also uses a threshold B-3 to filter whether to retain connected component B-1. If the ratio B-1 is greater than or equal to the threshold B-3, it means that the area of ​​connected component B-1 is comparable to the area of ​​the entire image to be segmented, and connected component B-1 can be retained; if the ratio B-1 is less than the threshold B-3, it means that the area of ​​connected component B-1 is not comparable to the area of ​​the entire image to be segmented, and connected component B-1 is filtered out (i.e., connected component B-1 is deleted). Optionally, the threshold B-3 can be a value such as 0.001, 0.002, 0.003, or other values.

[0209] Condition A-2:

[0210] The cloud-side server calculates the area of ​​each connected component retained in the second predicted mask graph and selects the connected component with the largest area, denoted as the largest connected component B-1. The cloud-side server then filters the connected components retained through condition A-1 as follows, taking one connected component B-2 as an example: the cloud-side server calculates the ratio of the area of ​​connected component B-2 to the area of ​​the largest connected component B-1, denoted as the ratio B-2. The cloud-side server further filters whether to retain connected component B-2 based on a threshold B-4. If the ratio B-2 is greater than or equal to the threshold B-4, connected component B-2 is retained; if the ratio B-2 is less than the threshold B-4, connected component B-2 is filtered out. Optionally, the threshold B-4 can be a value such as 0.18, 0.22, 0.25, or other values.

[0211] Condition A-3:

[0212] The cloud-side server processes the connected components retained after filtering under condition A-2 as follows: It calculates the ratio of the sum of the areas of all remaining connected components in the second predicted mask image to the area of ​​the image to be segmented, denoted as ratio B-3. The cloud-side server also uses a threshold B-5 to filter whether to retain the second predicted mask image. If ratio B-3 is greater than or equal to threshold B-5, the second predicted mask image is retained; if ratio B-3 is less than threshold B-5, the second predicted mask image is filtered out. Filtering out the second predicted mask image means setting all pixel values ​​of the second predicted mask image to 0, resulting in a completely black image. Optionally, threshold B-5 can be a value such as 0.0022, 0.0024, 0.0026, or other values.

[0213] Condition A-4:

[0214] The cloud-side server processes each remaining predicted mask image after filtering according to condition A-3, following the steps outlined in condition A-4. Taking any third predicted mask image as an example: For the binarized image corresponding to the third predicted mask image, the cloud-side server performs the following calculations for each remaining connected component. Let's take one of the remaining connected components, denoted as connected component B-3, as an example. The cloud-side server calculates the ratio of the width of the minimum bounding rectangle of connected component B-3 to the width of the image to be segmented, denoted as ratio B-4. The cloud-side server also calculates the ratio of the height of the minimum bounding rectangle of connected component B-3 to the height of the image to be segmented, denoted as ratio B-5. The cloud-side server further filters whether to retain the third predicted mask image based on a threshold B-6. If ratio B-4 and / or ratio B-5 are greater than or equal to threshold B-6, the third predicted mask image is retained; if both ratio B-4 and ratio B-5 are less than threshold B-6, the third predicted mask image is filtered out. Filtering out the third predicted mask image means setting all pixel values ​​of the third predicted mask image to 0, resulting in a completely black image. Optionally, the threshold B-6 can be a value such as 0.048, 0.052, 0.06, or other values.

[0215] Optionally, the order in which the cloud-side server filters connected components according to conditions A-1 and A-2 can be interchanged. For example, it can filter connected components first according to condition A-2, and then filter the remaining connected components according to condition A-1. Similarly, the order in which the cloud-side server filters the predicted mask graph according to conditions A-3 and A-4 can also be interchanged. For example, it can filter the predicted mask graph first according to condition A-4, and then filter the remaining predicted mask graph according to condition A-3. It should be noted that during the second post-processing, the cloud-side server only needs to filter connected components according to conditions A-1 and A-2 first, and then filter the predicted mask graph according to conditions A-3 and A-4.

[0216] It should be noted that each predicted mask image obtained through inference on the cloud server can undergo the operations described in conditions A-1 to A-4. Optionally, the number of predicted mask images may be reduced after the second post-processing.

[0217] Second validity verification:

[0218] The cloud-side server can verify the validity of each prediction mask image retained from the second post-processing step as follows:

[0219] Taking one of the prediction mask graphs as an example, denoted as the fourth prediction mask graph, the cloud server calculates the area of ​​each connected component in the fourth prediction mask graph and obtains the connected component with the largest area, denoted as the largest connected component B-2. Then, the cloud server performs the following processing on each connected component in the fourth prediction mask graph, taking one connected component B-4 as an example: The cloud server calculates the intersection of the area of ​​connected component B-4 and the area of ​​the selected region, denoted as intersection B. The cloud server then calculates the ratio of intersection B to the area of ​​the largest connected component B-2, denoted as ratio B-6. Finally, the cloud server calculates the ratio of the area of ​​intersection B to the area of ​​the selected region, denoted as ratio B-7.

[0220] The cloud-side server also uses thresholds B-7 and B-8 to filter whether to retain connected component B-4: if the ratio B-6 is greater than or equal to threshold B-7, and / or if the ratio B-7 is greater than or equal to threshold B-8, then connected component B-4 is retained; if the ratio B-6 is less than threshold B-7, and all ratios B-7 are less than threshold B-8, then connected component B-4 is filtered out, thus obtaining a segmentation result after validity verification. Optionally, threshold B-7 can be a value such as 0.75, 0.82, 0.85, or other values. Optionally, threshold B-8 can be a value such as 0.45, 0.52, 0.55, or other values.

[0221] After undergoing the aforementioned second preprocessing, first inference processing, second postprocessing, and second validity verification, the initial SAM segmentation result is output by the cloud server as the first segmentation result.

[0222] S707. Based on the first segmentation result (SAM segmentation result), the main body segmentation image is cropped, and the main body segmentation image is segmented using the main body matting model to obtain the second segmentation result (main body segmentation result).

[0223] Specifically, the process of obtaining the cropped subject segmentation image based on the first segmentation result is as follows:

[0224] It should be noted that when the cloud server crops the image to be segmented, it uses the smallest bounding rectangle of the selected area as the reference. That is, the width and height of the smallest bounding rectangle of the selected area are multiplied by the corresponding scaling factor to obtain the cropping size. Then, the cloud server crops the image to be segmented according to the cropping size, with the smallest bounding rectangle of the selected area as the center, to obtain the main segmented image.

[0225] Optionally, when the first segmentation result is invalid, for example, after the second post-processing and second validity verification, no predicted mask image is retained, then the first segmentation result is empty. In this case, the cloud-side server can crop the image to be segmented according to the first scaling factor to obtain the main body segmentation image; if the first segmentation result is valid, for example, after the second post-processing and second validity verification, at least one predicted mask image is retained, then the cloud-side server crops the image to be segmented according to the second scaling factor to obtain the main body segmentation image. Optionally, the first scaling factor can be greater than the second scaling factor. Optionally, the first scaling factor can be a value such as 1.55, 1.6, or 1.65, or other values; the second scaling factor can be a value such as 1.2, 1.28, or 1.3, or other values.

[0226] like Figure 14 As shown, if the width of the minimum bounding rectangle of the selected area, multiplied by the scaling factor, exceeds the width of the image to be segmented, then the width of the image to be segmented is used as the cropping width; if the height of the minimum bounding rectangle of the selected area, multiplied by the scaling factor, exceeds the height of the image to be segmented, then the height of the image to be segmented is used as the cropping height.

[0227] After obtaining the cropping size, the cloud server can crop the image to be segmented according to the cropping size to obtain the subject segmentation image. Next, the cloud server uses a subject matting model to segment the subject segmentation image. This segmentation process can include four steps: third preprocessing, saliency detection, third postprocessing, and third validity verification.

[0228] The third preprocessing includes:

[0229] Optionally, the image size required for the subject cutout model is 832*832. The cloud server can scale the subject segmentation image by transforming the longer side to 832, achieving a second size transformation.

[0230] The cloud server normalizes the resized and padded subject segmentation image to obtain a normalized image to be segmented. This normalization refers to pixel value normalization, specifically dividing the pixel value of each pixel by 255 to obtain a new pixel value. Then, the cloud server padded the normalized subject segmentation image to a size of 832*832 and input it into the subject matting model for detection.

[0231] Optionally, the order of scaling, pixel value normalization, and padding in the third preprocessing step is not limited.

[0232] The process of significance detection includes:

[0233] The cloud-based server uses a subject matting model to detect the subject segmentation image after the third preprocessing step, obtaining the initial subject matting result. This initial subject matting result can be visualized as a mask image of the main object in the subject segmentation image, called the initial subject mask image.

[0234] Optionally, the aforementioned subject matting model is a saliency detection model, which can detect the most salient or easily noticeable objects. It should be noted that this subject matting model is trained using training images labeled with various subjects (also known as objects). In other words, the more diverse the types of labeled subjects in the training images, the more accurate the subject matting model's detection results will be. These subjects may include, but are not limited to, people, buildings, trees, and items (boxes, cars, etc.).

[0235] Next, the cloud server can resize the initial subject mask image, scaling it back to the size of the subject segmentation image. Then, the cloud server can paste the resized initial subject mask image back into the cropped area to obtain the subject mask image. Specifically, the cloud server resizes the initial subject mask image to obtain a resized mask image. Then, the cloud server creates a new image with all zero pixels, the same size as the image in the first segmentation result, and pastes the resized mask image back into the image with all zero pixels, resulting in a new mask image, called the subject mask image. For example, the initial subject mask image could be like this: Figure 15 Figure a in the diagram, after being resized, can be obtained as follows: Figure 15 Figure b shows the initial subject mask image after size transformation. After the cloud server pastes the initial subject mask image back into an image with all pixel values ​​of 0, it can obtain... Figure 15 The main mask diagram shown in Figure c is shown in the figure.

[0236] Third post-processing operation:

[0237] First, the cloud server can convert the pixel values ​​in the initial subject mask image to a range of 1 to 255, and then binarize them according to a threshold C-1. For example, after converting the pixel values ​​in the initial subject mask image to 1-255, taking one pixel as an example, if the pixel value of that pixel is greater than or equal to the threshold C-1, then the pixel value of that pixel is set to 1; if the pixel value of that pixel is less than the threshold C-1, then the pixel value of that pixel is set to 0, thus obtaining the binarized image corresponding to the initial subject mask image. Optionally, the threshold C-1 can be a value such as 50, 52, 54, or other values.

[0238] In this binarized image, adjacent pixels with a value of 1 form a connected component. Optionally, there may be multiple connected components in this binarized image.

[0239] Next, the cloud server filters the results obtained in the aforementioned process according to the following conditions B-1 to B-4.

[0240] Condition B-1:

[0241] Taking one of the connected components C-1 in the binarized image as an example, the cloud server calculates the ratio of the area of ​​connected component C-1 to the area of ​​the main segmented image, denoted as the ratio C-1.

[0242] The cloud-side server also determines whether to retain the connected component C-1 based on a threshold C-2. If the ratio C-1 is greater than or equal to the threshold C-2, it means that the area of ​​the connected component C-1 is comparable to the area of ​​the entire subject segmentation image, and the cloud-side server can retain the connected component C-1. If the ratio C-1 is less than the threshold C-2, it means that the area of ​​the connected component C-1 is not comparable to the area of ​​the entire subject segmentation image, and the cloud-side server filters out the connected component C-1. Optionally, the threshold C-2 can be a value such as 0.008, 0.009, 0.012, or other values.

[0243] Condition B-2:

[0244] For each binarized graph, the cloud server calculates the area of ​​each connected component and selects the connected component with the largest area, denoted as the largest connected component C-1. The cloud server then filters the remaining connected components that pass condition B-1 as follows, taking one connected component C-2 as an example: the cloud server calculates the ratio of the area of ​​connected component C-2 to the area of ​​the largest connected component C-1, denoted as the ratio C-2. The cloud server also uses a threshold C-3 to determine whether to retain connected component C-2. If the ratio C-2 is greater than or equal to the threshold C-3, the cloud server retains connected component C-2; if the ratio C-2 is less than the threshold C-3, the cloud server filters out connected component C-2. Optionally, the threshold C-3 can be a value such as 0.12, 0.14, 0.16, or other values.

[0245] Condition B-3:

[0246] The cloud-side server processes the connected components retained after filtering by condition B-2 as follows: It calculates the ratio of the sum of the areas of all remaining connected components to the area of ​​the subject segmentation image, denoted as ratio C-3. The cloud-side server also uses a threshold C-4 to filter whether to retain the initial subject mask image corresponding to the binarized image. If the ratio C-3 is greater than or equal to the threshold C-4, the initial subject mask image corresponding to the binarized image is retained; if the ratio C-3 is less than the threshold C-4, the initial subject mask image corresponding to the binarized image is filtered out. Filtering out a mask image means setting all pixel values ​​of that mask image to 0, resulting in a completely black image. Optionally, the threshold C-4 can be a value such as 0.028, 0.032, 0.035, or other values.

[0247] Condition B-4:

[0248] The cloud-side server processes the initial subject mask image retained after filtering under condition B-3 as follows: For each remaining connected component in the binarized image of the initial subject mask image, the cloud-side server performs the following calculations. Taking one of the remaining connected components as an example, denoted as connected component C-3, the cloud-side server calculates the ratio of the width of the minimum bounding rectangle of connected component C-3 to the width of the subject segmentation image, denoted as ratio C-4; the cloud-side server also calculates the ratio of the height of the minimum bounding rectangle of connected component C-3 to the height of the subject segmentation image, denoted as ratio C-5. The cloud-side server further filters whether to retain the initial subject mask image based on the threshold C-5. If ratio C-4 and / or ratio C-5 are greater than or equal to the threshold C-5, the initial subject mask image is retained; if both ratio C-4 and ratio C-5 are less than the threshold C-5, the initial subject mask image is filtered out, that is, the entire initial subject mask image is set to 0, becoming a completely black image. Optionally, the threshold C-5 can be a value such as 0.048, 0.052, 0.06, or other values.

[0249] Optionally, the filtering order of conditions B-1 and B-2 can be interchanged, and the filtering order of conditions B-3 and B-4 can be interchanged, as long as conditions B-1 and B-2 need to be filtered a few times before conditions B-3 and B-4.

[0250] It should be noted that the operations described in conditions B-1 to B-4 above can filter out small fragments. Optionally, invalid initial subject mask images may also be filtered out, indicating that the result output by the subject matting model is invalid.

[0251] Third validity verification:

[0252] The cloud server verifies the validity of each initial subject mask image retained from the third post-processing step as follows:

[0253] Taking one of the initial main body mask diagrams as an example, the cloud server calculates the area of ​​each connected component in the initial main body mask diagram. Then, for each connected component, the cloud server performs the following processing. Taking one connected component C-4 as an example: the cloud server calculates the intersection of the area of ​​connected component C-4 and the area of ​​the selected region, denoted as intersection C. The ratio of intersection C to connected component C-4 is denoted as ratio C-6. The cloud server also calculates the ratio of the area of ​​intersection C to the area of ​​the selected region, denoted as ratio C-7.

[0254] The cloud-side server also uses thresholds C-6 and C-7 to filter whether to retain connected component C-4. If the ratio C-6 is greater than or equal to threshold C-6, and / or if the ratio C-7 is greater than or equal to threshold C-7, then connected component C-4 is retained; if the ratio C-6 is less than threshold C-6, and the ratio C-7 is less than threshold C-7, then connected component C-4 is filtered out, thus obtaining the final main body mask graph. Optionally, threshold C-6 is greater than threshold C-7. Optionally, threshold C-6 can be a value such as 0.75, 0.82, 0.85, or other values. Optionally, threshold C-7 can be a value such as 0.45, 0.52, 0.55, or other values.

[0255] After the above-mentioned third preprocessing, saliency detection, third postprocessing and third validity verification process, the subject matting model outputs the second segmentation result (subject segmentation result), which includes the final subject mask image that is retained.

[0256] S708. Based on the first segmentation result and the second segmentation result, determine the filter mask image.

[0257] It should be noted that after the aforementioned third preprocessing, saliency detection, third post-processing, and third validity verification, if a retained main body mask image still exists, and the retained main body mask image contains valid connected components (meaning the second segmentation result is not empty), the cloud server can use the second segmentation result as the output filtering mask image and pass it to subsequent processes. If the second segmentation result is empty, but the first segmentation result is valid (not empty), the cloud server will use the first segmentation result as the output filtering mask image and pass it to subsequent processes. If both the first and second segmentation results are empty, the selected mask image will be used as the output filtering mask image and passed to subsequent processes. The filtering mask image output to subsequent processes here can be merged with the results of the portrait instantiation described later.

[0258] Next, we will introduce the specific process of instantiating a portrait.

[0259] S709. Using a human image segmentation model, perform human image segmentation on the image to be segmented to obtain the human image segmentation result.

[0260] Specifically, the cloud server performs four processes on the image to be segmented: fourth preprocessing, human image segmentation, fourth postprocessing, and fourth validity verification.

[0261] Fourth preprocessing:

[0262] The cloud-based server can normalize the image to be segmented by dividing the pixel value of each pixel by 255 to obtain a new pixel value. Then, the cloud-based server fills the normalized image to the size required by the model, for example, 1024*1024. Finally, the cloud-based server inputs the filled image into the human segmentation model for human detection.

[0263] Human face segmentation processing:

[0264] The cloud-based server uses a human face segmentation model to detect human faces in the normalized and resized images to be segmented, obtaining an initial human face mask image. It should be noted that if multiple human faces exist in the image, a corresponding mask image can be obtained for each human face. For example, the human face segmentation model can be used to detect human faces in images such as... Figure 16 After performing face detection on the image to be segmented (containing two figures) shown in Figure a, the results are as follows: Figure 16 The images b and c show the two initial portrait mask images. Optionally, if the person in the image to be segmented is occluded by other people or objects, the foreground shape in the resulting initial portrait mask image may be a discontinuous shape of the person, for example... Figure 16 As shown in Figure d.

[0265] Optionally, if the number of initial human image mask images is 0, it means that no human figures were detected in the images. In this case, there is no need to perform the fourth post-processing and fourth validity verification, and the subsequent steps can be skipped. For example, step S712 can be executed directly.

[0266] Fourth post-processing:

[0267] The cloud server performs a fourth post-processing on each initial portrait mask image. Taking one of the initial portrait mask images as an example, let's call it the first initial portrait mask image:

[0268] First, the cloud server can convert the pixel values ​​in the initial portrait mask image to a value between 1 and 255, and then binarize them according to a threshold D-1. For example, after converting the pixel values ​​in the initial portrait mask image to 1-255, taking one pixel as an example, if the pixel value of that pixel is greater than or equal to the threshold D-1, then the pixel value of that pixel is set to 1; if the pixel value of that pixel is less than the threshold D-1, then the pixel value of that pixel is set to 0, thus obtaining the binarized image of the initial portrait mask image. Optionally, the threshold D-1 can be a value such as 50, 52, 54, or other values.

[0269] In this binarized image, adjacent pixels with a value of 1 form a connected component. Optionally, there may be multiple connected components in this binarized image.

[0270] Next, the cloud server filters the results after the fourth post-processing step according to the following conditions:

[0271] Condition C-1:

[0272] Taking the binarized graph of one of the initial human face mask graphs as an example, the cloud server calculates the area of ​​each connected component and selects the connected component with the largest area, which is denoted as the largest connected component D-1.

[0273] Taking one of the connected components D-1 in a binary graph as an example, the cloud server calculates the ratio of the area of ​​connected component D-1 to the area of ​​the image input to the portrait segmentation model (e.g., the area of ​​a 1024*1024 image), and denoted as the ratio D-1.

[0274] Here, the image input to the portrait segmentation model is referred to as the portrait segmentation image. The cloud server also uses a threshold D-2 to filter whether to retain the connected component D-1. If the ratio D-1 is greater than or equal to the threshold D-2, it means that the area of ​​the connected component D-1 is comparable to the area of ​​the portrait segmentation image, and the connected component D-1 can be retained; if the ratio D-1 is less than the threshold D-2, it means that the area of ​​the connected component D-1 is not comparable to the area of ​​the entire portrait segmentation image, and the connected component D-1 is filtered out. Optionally, the threshold D-2 can be a value such as 0.0001, 0.0002, 0.0003, or other values. After filtering by condition C-1, some individual parts of the human body, such as individual fingers or feet, can be filtered out.

[0275] Condition C-2:

[0276] After filtering under condition C-1, the cloud server performs the following filtering on the remaining connected components, taking one of them, D-2, as an example: The cloud server calculates the ratio of the area of ​​connected component D-2 to the area of ​​the largest connected component D-1, denoted as the ratio D-2. The cloud server also uses a threshold D-3 to filter whether to retain connected component D-2. If the ratio D-2 is greater than or equal to the threshold D-3, the cloud server retains connected component D-2; if the ratio D-2 is less than the threshold D-3, the cloud server filters out connected component D-2. Connected components D-2 and D-1 can be the same or different. Optionally, the threshold D-3 can be a value such as 0.04, 0.06, 0.07, or other values.

[0277] Condition C-3:

[0278] The cloud-side server processes the connected components retained after filtering under condition C-2 as follows: It calculates the ratio of the sum of the areas of all remaining connected components to the area of ​​the entire portrait segmentation image, denoted as ratio D-3. The cloud-side server also uses a threshold D-4 to filter whether to retain the initial portrait mask image. Taking the second initial portrait mask image as an example, the second initial portrait mask image is a non-empty initial portrait mask image after filtering under conditions C-1 and C-2. This second initial portrait mask image may be the same as or different from the first initial portrait mask image. If the ratio D-3 is greater than or equal to the threshold D-4, the cloud-side server retains the second initial portrait mask image; if the ratio D-3 is less than the threshold D-4, the cloud-side server filters out the second initial portrait mask image, that is, sets the entire second initial portrait mask image to 0, making it a completely black image. Optionally, the threshold D-4 can be a value such as 0.0009, 0.0012, 0.0015, or other values.

[0279] Condition C-4:

[0280] The cloud-side server processes each initial portrait mask image retained after filtering by condition C-3 as follows. Taking one initial portrait mask image as an example, denoted as the third initial portrait mask image: The cloud-side server performs the following calculations for each remaining connected component in the third initial portrait mask image. Taking one of the remaining connected components as an example, denoted as connected component D-3: The cloud-side server calculates the ratio of the width of the minimum bounding rectangle of connected component D-3 to the width of the portrait segmentation image, denoted as ratio D-4; the cloud-side server also calculates the ratio of the height of the minimum bounding rectangle of connected component D-3 to the height of the portrait segmentation image, denoted as ratio D-5. The cloud-side server also uses the threshold D-5 to filter whether to retain the third initial portrait mask image. If the ratio D-4 and / or the ratio D-5 are greater than or equal to the threshold D-5, the cloud server retains the third initial portrait mask image; if both the ratio D-4 and the ratio D-5 are less than the threshold D-5, the cloud server filters out the third initial portrait mask image, that is, sets the entire third initial portrait mask image to 0, making it a completely black image. The third initial portrait mask image may be the same as or different from the second initial portrait mask image. Optionally, the threshold D-5 can be a value such as 0.0025, 0.0032, 0.0035, or other values.

[0281] Optionally, the filtering order of conditions C-1 and C-2 can be interchanged, and the filtering order of conditions C-3 and C-4 can be interchanged, as long as the filtering process of conditions C-1 and C-2 is performed before conditions C-3 and C-4.

[0282] Fourth validity verification:

[0283] After the fourth post-processing described above, the cloud server performs the following operations on the retained non-empty initial portrait mask images. Taking one of the portrait mask images as an example, let's call it the fourth initial portrait mask image. The cloud server calculates the intersection of the area of ​​the portrait region and the selected area in the fourth initial portrait mask image, denoted as intersection D. The cloud server calculates the ratio of intersection D to the area of ​​the portrait region, denoted as ratio D-6; the cloud server calculates the ratio of intersection D to the area of ​​the selected area, denoted as ratio D-7.

[0284] The cloud-side server also uses thresholds D-6 and D-7 to filter whether to retain the fourth initial human image mask. If the ratio D-6 is greater than or equal to the threshold D-6, and / or if the ratio D-7 is greater than or equal to the threshold D-7, the cloud-side server retains the fourth initial human image mask; if the ratio D-6 is less than the threshold D-6, and the ratio D-7 is less than the threshold D-7, the cloud-side server filters out the fourth initial human image mask. Optionally, the threshold D-6 is greater than the threshold D-7. Optionally, the threshold D-6 can be a value such as 0.75, 0.82, 0.85, or other values. Optionally, the threshold D-7 can be a value such as 0.45, 0.52, 0.55, or other values.

[0285] After the above-mentioned fourth preprocessing, human image segmentation, fourth postprocessing and fourth validity verification processes, the cloud server will perform the first merging of the retained initial human image mask (effectHumanMask) to form the merged human image mask, which can also be called the human image segmentation result.

[0286] S710. Validate the effectiveness of the filtered mask image to obtain the filtered mask image.

[0287] After obtaining the filter mask image, the cloud-side server can perform a fifth validity verification on the filter mask image.

[0288] Specifically, the fifth validity verification includes: the cloud server calculating the area of ​​each connected component in the filter mask graph. Taking one connected component as an example, denoted as connected component E-1. The cloud server calculates the intersection of the areas of connected component E-1 and the selected region, denoted as intersection E. The cloud server calculates the ratio of the area of ​​intersection E to the area of ​​connected component E-1, denoted as ratio E-1; the cloud server calculates the ratio of the area of ​​connected component E-1 to the area of ​​the selected region, denoted as ratio E-2.

[0289] The cloud-side server also uses thresholds E-1 and E-2 to filter whether to retain the connected component E-1. If the ratio E-1 is greater than or equal to threshold E-1, and / or if the ratio E-2 is greater than or equal to threshold E-2, the cloud-side server retains the connected component E-1; if the ratio E-1 is less than threshold E-1, and the ratio E-2 is less than threshold E-2, the cloud-side server filters out the connected component E-1. Optionally, threshold E-1 is greater than threshold E-2. Optionally, threshold E-1 can be a value such as 0.75, 0.82, 0.85, or other values. Optionally, threshold E-2 can be a value such as 0.45, 0.52, 0.55, or other values.

[0290] After the fifth validity verification, the cloud server filters out some connected components to obtain the filtered mask graph.

[0291] S711. Merge the filtered mask image and the human image segmentation results to obtain the target object segmentation results.

[0292] The cloud-based server can perform a second merging of the portrait segmentation results and the filtered mask image. That is, the cloud-based server will merge the connected components that have passed the fifth validity verification into the portrait segmentation results, obtaining the merged result. This second merged result can be called the target object segmentation result.

[0293] Optionally, if the initial number of human face mask images is 0, the cloud server will filter the mask images as the target object segmentation result and continue to execute subsequent processes, such as executing S712.

[0294] S712. Based on the target object segmentation result, a shadow segmentation model is used to perform shadow segmentation on the image to be segmented, and the shadow segmentation result is obtained.

[0295] Specifically, the process can include three steps: the fifth preprocessing step, the shadow segmentation step, and the fifth postprocessing step.

[0296] Fifth preprocessing:

[0297] The cloud server normalizes the pixel values ​​of the image to be segmented by dividing the pixel value of each pixel by 255 to obtain a new pixel value. Then, the cloud server fills the normalized image to be segmented to a size of 1024*1024 and inputs it into the shadow segmentation model for shadow detection.

[0298] The cloud-side server also inputs the target object segmentation result as a prompt message into the shadow segmentation model. Optionally, the prompt message can be a dialog box. Specifically, the cloud-side server extracts the minimum bounding rectangle of the objects in the target object segmentation result. Optionally, this example assumes the target object segmentation result includes two cases:

[0299] Scenario 1: The initial number of human face mask images is 0.

[0300] In scenario one, since the initial number of human face mask images is 0, it means there are no human faces in the image. The cloud server can then extract the minimum bounding rectangle for the valid regions in the filter mask image. Taking the filter mask image as a selection mask image as an example, the cloud server can extract the minimum bounding rectangle of the selected area in the selection mask image as a prompt box.

[0301] Scenario 2: The number of initial human face mask images is not 0.

[0302] In scenario two, the cloud server extracts the minimum bounding rectangle of each initial portrait mask image and uses this minimum bounding rectangle as the tooltip. When there are multiple initial portrait masks, there are also multiple tooltips.

[0303] Shadow segmentation process:

[0304] The cloud-based server inputs the prompt box and the model to be segmented into the shadow segmentation model to perform shadow segmentation, and can obtain the shadow segmentation result.

[0305] It should be noted that the shadow segmentation model can also be a type of SAM model, which is a model obtained by training on a training map that is labeled with objects with shadows and shadows.

[0306] During shadow segmentation, the segmentation results of the target object are used as cue information (also known as cue words) and input into the shadow segmentation model. The cue encoder in the shadow segmentation model encodes the diagonal coordinates of the cue box to obtain cue-encoded data; the image encoder in the shadow segmentation model encodes the normalized image to be segmented to obtain the image-encoded result. Then, the decoder in the shadow segmentation model decodes the cue-encoded data and the image-encoded result to generate the initial shadow prediction result.

[0307] Optionally, the initial shadow prediction result can be output as a shadow mask image. Optionally, the shadow mask image corresponding to the shadow segmentation result can be one or more images. Optionally, the shadow mask image can be a mask image containing only the shadow region, or it can be a mask image including the object and the shadow region of the object; this embodiment of the application does not limit this.

[0308] Optionally, the shadow segmentation model can also output a corresponding IOU prediction score for each shadow mask image. The higher the IOU prediction, the more confident the shadow segmentation model is in its output prediction results.

[0309] Fifth post-processing:

[0310] First, the cloud server can filter out shadow mask images with IOU prediction scores below a threshold F-1, obtaining other shadow mask images with IOU prediction scores greater than or equal to the threshold F-1. Optionally, the threshold F-1 can be a value such as 0.9, 0.92, 0.96, or other values.

[0311] Taking one of the remaining shadow mask images, the first shadow mask image, as an example, the cloud server converts the pixel values ​​of the pixels in the first shadow mask image to between 1 and 255, and then performs binarization according to a threshold F-2. For example, after converting the pixel values ​​in the first shadow mask image to 1-255, taking one pixel as an example: if the pixel value of the pixel is greater than or equal to the threshold F-2, then the pixel value of the pixel is set to 1; if the pixel value of the pixel is less than the threshold F-2, then the pixel value of the pixel is set to 0, thus obtaining the binarized image corresponding to the first shadow mask image. Optionally, the threshold F-2 can be a value such as 50, 52, 54, or other values.

[0312] In this binarized image, adjacent pixels with a value of 1 form a connected component. Optionally, multiple connected components may exist in the binarized image.

[0313] Next, the cloud server filters the output of the fifth post-processing step according to the following conditions D-1 to D-4:

[0314] Condition D-1:

[0315] Taking one of the multiple connected components, F-1, as an example, the cloud server calculates the ratio of the area of ​​connected component F-1 to the area of ​​the image to be segmented, denoted as the ratio F-1. Optionally, when the size of the image to be segmented input to the shadow segmentation model is 1024*1024, the area of ​​the image to be segmented can be denoted as 1024*1024.

[0316] The cloud-side server also uses a threshold F-3 to filter whether to retain the connected component F-1. If the ratio F-1 is greater than or equal to the threshold F-3, it means that the area of ​​the connected component F-1 is comparable to the area of ​​the entire image to be segmented (e.g., 1024*1024), and the cloud-side server can retain the connected component F-1. If the ratio F-1 is less than the threshold F-3, it means that the area of ​​the connected component F-1 is not comparable to the area of ​​the entire image to be segmented, and the cloud-side server filters out the connected component F-1. Optionally, the threshold F-3 can be a value such as 0.001, 0.002, 0.003, or other values.

[0317] Condition D-2:

[0318] The cloud server calculates the area of ​​each connected component and selects the connected component with the largest area, denoted as the largest connected component F-1. The cloud server then filters the connected components retained through condition D-1 as follows, taking one connected component F-2 as an example: the cloud server calculates the ratio of the area of ​​connected component F-2 to the area of ​​the largest connected component F-1, denoted as the ratio F-2. The cloud server also uses a threshold F-4 to determine whether to retain connected component F-2. If the ratio F-2 is greater than or equal to the threshold F-4, the cloud server retains connected component F-2; if the ratio F-2 is less than the threshold F-4, the cloud server filters out connected component F-2. Optionally, the threshold F-4 can be a value such as 0.18, 0.22, 0.25, or other values.

[0319] Condition D-3:

[0320] The cloud-side server processes the connected components retained after filtering by condition D-2 as follows: It calculates the ratio of the sum of the areas of all remaining connected components to the area of ​​the image to be segmented, denoted as ratio F-3. The cloud-side server also uses a threshold F-5 to filter whether to retain the first shadow mask image. If the ratio F-3 is greater than or equal to the threshold F-5, the cloud-side server retains the first shadow mask image; if the ratio F-3 is less than the threshold F-5, the cloud-side server filters out the first shadow mask image, effectively setting the entire first shadow mask image to 0, making it a completely black image. Optionally, the threshold F-5 can be a value such as 0.0022, 0.0024, 0.0026, or other values.

[0321] Condition D-4:

[0322] The cloud-side server processes the remaining shadow mask images after filtering by condition D-3 as follows. Taking one of the second shadow mask images as an example, the cloud-side server performs the following calculations for each connected component in the binarized graph of the second shadow mask image. Let's take one of the remaining connected components as an example, denoted as connected component F-3. The second shadow mask image may be the same as or different from the first shadow mask image. The cloud-side server calculates the ratio of the width of the minimum bounding rectangle of connected component F-3 to the width of the image to be segmented, denoted as ratio F-4; the cloud-side server also calculates the ratio of the height of the minimum bounding rectangle of connected component F-3 to the height of the image to be segmented, denoted as ratio F-5. The cloud-side server further filters whether to retain the second shadow mask image based on a threshold F-6. If the ratio F-4 and / or the ratio F-5 are greater than or equal to the threshold F-6, the cloud server retains the second shadow mask image; if both the ratio F-4 and the ratio F-5 are less than the threshold F-6, the cloud server filters out the second shadow mask image, that is, sets the entire second shadow mask image to 0, making it a completely black image. Optionally, the threshold F-6 can be a value such as 0.048, 0.052, 0.06, or other values.

[0323] Optionally, the filtering order of conditions F-1 and F-2 can be interchanged, and the filtering order of conditions F-3 and F-4 can be interchanged, as long as conditions F-1 and F-2 need to be filtered before conditions F-3 and F-4.

[0324] It should be noted that each shadow mask image obtained through inference can undergo the operations described in conditions F-1 to F-4. Optionally, after the fifth post-processing step, the number of shadow mask images may be reduced.

[0325] After the above three processes—the fifth preprocessing, the shadow segmentation process, and the fifth postprocessing—the cloud server can obtain the shadow segmentation result.

[0326] Figure 17 In the image to be segmented shown in Figure a, it can be seen that the figure in the background has a shadow. A shadow segmentation model is used to segment the image. Figure 17 After segmenting graph a in the image, we can obtain the following: Figure 17 The shadow segmentation results are shown in Figure b or Figure c. Figure 17 In the image b, the prediction result is the target (object) plus its shadow. Figure 17 In the equation, c represents the prediction result for only the shadow.

[0327] S713. Merge the shadow segmentation result and the target object segmentation result to obtain the merged segmentation result.

[0328] Specifically, the target object segmentation results also exhibit the following situations:

[0329] Scenario 3: The initial number of human face mask images is 0.

[0330] In scenario three, the initial number of human face mask images is 0, indicating that there are no human faces in the images. The cloud server can then use the filtered mask images as the segmentation result for the target object.

[0331] Case 4: The number of initial human face mask images is not 0.

[0332] In scenario four, when the number of initial human face mask images is not zero, the target object segmentation result may include multiple initial human face mask images. Optionally, it may also include mask images corresponding to objects detected by the SAM model and / or the subject matting model.

[0333] Taking one of the predicted masks in the target object segmentation results as an example, denoted as the fifth predicted mask image. The cloud-side server calculates the intersection of the target object in the fifth predicted mask image and the target shadow in the corresponding shadow mask image, denoted as the intersection G. It should be noted that after the image to be segmented is detected by the shadow segmentation model, it can output the shadow mask image of the object with shadow, as described in step S712, which will not be repeated here. The cloud-side server calculates the ratio of the area of ​​the intersection G to the area of ​​the target object in the fifth predicted mask image, denoted as the ratio G-1.

[0334] When the ratio G-1 is greater than or equal to the threshold G-1, the cloud-side server fuses the fifth predicted mask image and the shadow mask image corresponding to the target object to obtain a fused segmentation result. Specifically, the process of fusing the two mask images by the cloud-side server can be to take their union, that is, to merge the shadow mask image corresponding to the target object into the target object segmentation result to obtain the fused segmentation result. If the ratio G-1 is less than the threshold G-1, the cloud-side server does not need to fuse the shadow mask image. Based on this, the cloud-side server can fuse the shadow segmentation result and the target object segmentation result to obtain a fused segmentation result. Optionally, the threshold G-1 can be a value close to 1, such as 0.95, 0.92, 0.9, etc., or other values.

[0335] Figure 18 Figure a in the diagram is an example of an image to be segmented. After being segmented by a shadow segmentation model, the image is obtained as shown in Figure a. Figure 18 The shadow mask image shown in figure b above. The target segmentation image mentioned above may include, for example... Figure 18The image shown in Figure c is a portrait mask. The cloud server determines the intersection G of the target object and the target shadow in the corresponding shadow mask image, and calculates whether the ratio G-1 is greater than or equal to the threshold G-1. If the ratio G-1 is greater than or equal to the threshold G-1, the image can be fused. Figure 18 From graphs a and b, we obtain the following: Figure 18 The fusion segmentation result is shown in Figure d. It should be noted that when there are multiple people in the image to be segmented, the fusion segmentation result also includes the segmentation regions corresponding to the other people. Optionally, when multiple people or objects in the image to be segmented have shadows, the fusion segmentation result also includes the segmentation regions of the shadows of these people or objects.

[0336] S714. Post-process the fused segmentation results to obtain the final segmentation result.

[0337] Alternatively, it is also possible that after the aforementioned processing and validity verification, the fused segmentation result is empty. In this case, the cloud server can output the text detection result as the segmentation result.

[0338] If the fused segmentation result is not empty, the cloud server can perform post-processing on the fused segmentation result, namely hole filling. Specifically, the cloud server calculates the outline of the mask image of the fused segmentation result and fills the holes within the outline (the black areas enclosed in the white areas of the mask image) to obtain a mask image without holes. Afterwards, the cloud server restores the size of the mask image without holes to the original image size of the initial image to be segmented.

[0339] Afterwards, the cloud-side server can output the resized mask image as the final segmentation result. For example, the cloud-side server can send the final segmentation result to the terminal device, where the terminal device will fill in the background; or, in the cloud, the cloud-side server can fill in the background of the region corresponding to the final segmentation result.

[0340] S715. Data point analysis, optimization of post-processing strategies and / or validity judgment strategies.

[0341] Optionally, the cloud server can also be configured with a tracking system to count the number of connected components and mask images filtered out during the various post-processing steps. When the number of filtering steps is relatively large compared to the number of images being processed—for example, in ten segmentation steps involving ten images, if small fragments are filtered out in five or more of eight images (e.g., in eight segmentation steps)—this indicates insufficient model accuracy or inaccurate post-processing parameters leading to frequent post-processing operations. The cloud server can then adjust the corresponding parameters in the model and various thresholds to reduce the generation of small fragments (also known as burrs) or invalid mask images, thereby reducing the number of filtering steps in post-processing and further improving segmentation accuracy and efficiency.

[0342] Optionally, the data logging system can also continuously log new data and optimize and adjust the various models mentioned above to improve model accuracy.

[0343] Optionally, the above Figure 7 In the embodiments, each step can be executed entirely by the cloud server, entirely by the terminal device, or partially by the cloud server and partially by the terminal device, thus achieving end-to-cloud integration.

[0344] For example, steps S701-S704 are executed by the terminal device, and steps S705-S715 are executed by the cloud server.

[0345] To fully describe the process of the embodiments of this application, the following is combined with... Figure 19 This paper provides a comprehensive description of image segmentation methods across all scenarios. For details, please refer to [reference needed]. Figure 19As shown: The terminal device can acquire an initial image to be segmented, which can be the image of the object to be eliminated selected by the user. The terminal device can receive the user-input selection trajectory, perform a closure operation to generate a closed trajectory, and generate a selection mask image based on the closed trajectory. The terminal device can then resize the selection mask image and the initial image to be segmented before sending them to the cloud server. The cloud server can preprocess the selection mask image and the resized initial image to be segmented, and then input the preprocessed image to be segmented into the model for detection. Specifically, the cloud server can first input the preprocessed image to be segmented into the text detection model for text detection and then verify the validity of the text detection results. If the text detection result is valid, the text region can be segmented to obtain the text detection result. Then, the cloud server can perform post-processing on the text detection result to obtain the final output segmentation result (in this case, the segmented object is a text object). If the text detection result is invalid, the pre-processed image to be segmented can be input into the SAM segmentation model for inference, and the inference result can be post-processed by SAM segmentation, as well as validated for validity, and then cropped to obtain the main segmented image.

[0346] The segmented image is input into the segmentation model for inference, and the inference results are post-processed for segmentation and their validity is verified. A filter mask image is then obtained through selection.

[0347] The cloud-based server also inputs the image to be segmented into the portrait segmentation model for inference, performs portrait segmentation post-processing on the inference results, and verifies their validity. Then, it merges the result with the selected mask image to obtain the merged result.

[0348] The cloud-side server also inputs the image to be segmented into the shadow segmentation model for inference, performs shadow segmentation post-processing on the inference results, then merges them with the merging results again, and performs post-processing on the fused results to obtain the final output segmentation result.

[0349] Figure 19 The specific implementation details in the embodiments can be found in the relevant descriptions above, and will not be repeated here.

[0350] Optionally, the segmentation process (or detection process) for different objects in each of the above steps can be executed separately to obtain the corresponding results. For example, the cloud server can use a text detection model to perform text detection on the image to be segmented and obtain text detection results; the cloud server can also use a SAM segmentation model to segment the image to be segmented and obtain SAM segmentation results; the cloud server can also use a subject matting model to segment the image to be segmented and obtain subject segmentation results; the cloud server can also use a portrait segmentation model to perform portrait segmentation on the image to be segmented and obtain portrait segmentation results; the cloud server can also use a shadow segmentation model to perform shadow segmentation on the image to be segmented and obtain shadow segmentation results. Optionally, the cloud server can also combine any two or more separately executed segmentation processes to achieve diversified and multi-scene image segmentation purposes. Optionally, when multiple separately executed segmentation processes are combined, the size transformation operation and preprocessing operation of the image to be processed and the selected mask image can be executed only once, or selectively executed according to the input requirements of different models, and is not required for each separate segmentation.

[0351] Figure 20 A flowchart of an image segmentation method provided in an embodiment of this application includes:

[0352] S2001. Obtain the image to be segmented.

[0353] S2002. Using a human image segmentation model, perform human image segmentation on the image to be segmented to obtain the human image segmentation result. The human image segmentation result is used to characterize the region where the human object is located in the image to be segmented.

[0354] S2003. Obtain the filter mask image. The filter mask image is a mask image obtained based on other segmentation results. Other segmentation results are the results of segmenting the image to be segmented. Other segmentation results are different from the portrait segmentation results.

[0355] S2004. Merge the filtered mask image and the portrait segmentation results to obtain the target object segmentation results.

[0356] S2005. Based on the target object segmentation results, a shadow segmentation model is used to perform shadow segmentation on the image to be segmented, and the shadow segmentation results are used to characterize the region where the shadow is located in the image to be segmented.

[0357] S2006. Merge the shadow segmentation result and the target object segmentation result to obtain the merged segmentation result.

[0358] The image to be segmented can be an image after the terminal device has resized the initial image to be segmented, or it can be an image without resizing, as long as it meets the size requirements of the portrait segmentation model. Optionally, the portrait segmentation model can also resize the input image of any size to the required size before segmentation.

[0359] The terminal device can send the image to be segmented to a cloud server, which then uses a human face segmentation model to perform human face segmentation on the image, obtaining the segmentation result. Optionally, the human face segmentation result can be output as one or more human face mask images. Each human face mask image can correspond to one person. Optionally, the region of a person in a human face mask image can be a complete shape of a person or an incomplete part of the shape of a person.

[0360] Optionally, if there are no people in the image to be segmented, the image segmentation result is empty.

[0361] The filtering mask is the result of segmenting the image to be segmented using another model by the cloud testing server. Optionally, the other model can be a preset SAM model or a subject matting model. Correspondingly, the filtering mask can be the segmentation result output by the preset SAM model, the subject segmentation result output by the subject matting model, or a selection mask. The filtering mask can also be the result selected from the outputs of the preset SAM model and the subject matting model, or a selection mask, or a fusion of some or all of the outputs of these two models. There is no limitation here, as long as it is different from the portrait segmentation result.

[0362] This method integrates multiple segmentation results, enabling segmentation across the entire scene, improving the accuracy of image segmentation, thereby enhancing the quality of image generation and improving the user experience.

[0363] The foregoing has detailed examples of the methods provided in this application. It is understood that the corresponding apparatus, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0364] This application can divide the image segmentation apparatus into functional modules based on the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0365] Figure 21 A schematic diagram of an image segmentation apparatus provided in this application is shown. The apparatus 2100 includes:

[0366] The acquisition module 2101 is used to acquire the image to be segmented.

[0367] The portrait segmentation module 2102 is used to perform portrait segmentation on the image to be segmented using a portrait segmentation model, and obtain the portrait segmentation result. The portrait segmentation result is used to characterize the region where the human object is located in the image to be segmented.

[0368] The filtering module 2103 is also used to obtain a filtering mask image, which is a mask image obtained based on other segmentation results. Other segmentation results are the results of segmenting the image to be segmented, and the other segmentation results are different from the portrait segmentation results.

[0369] The compositing module 2104 is used to merge the filtered mask image and the human image segmentation result to obtain the target object segmentation result.

[0370] The shadow segmentation module 2105 is used to perform shadow segmentation on the image to be segmented based on the target object segmentation result and using a shadow segmentation model to obtain the shadow segmentation result. The shadow segmentation result is used to characterize the region where the shadow is located in the image to be segmented.

[0371] The fusion module 2106 is used to fuse the shadow segmentation result and the target object segmentation result to obtain the fused segmentation result.

[0372] Optionally, the modules in device 2100 are also used to perform the steps in the various method embodiments described above.

[0373] The specific manner in which the device 2100 performs the image segmentation method and the beneficial effects thereof can be found in the relevant descriptions in the method embodiments, and will not be repeated here.

[0374] This application also provides an electronic device, including the processor described above. The electronic device provided in this embodiment may be... Figure 1The terminal device 100 shown is used to execute the image segmentation method described above. When using integrated units, the terminal device may include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the actions of the terminal device; for example, it can support the terminal device in executing the steps performed by the display unit, detection unit, and processing unit. The storage module can support the terminal device in storing program code and data. The communication module can support communication between the terminal device and other devices.

[0375] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other terminal devices.

[0376] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device involved in this embodiment can be a device having... Figure 1 The device with the structure shown.

[0377] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the image segmentation method described in any of the above embodiments.

[0378] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image segmentation method described above.

[0379] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0380] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units. The replaced units may or may not be physically separate. The component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0381] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0382] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0383] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image segmentation method characterized by, The method comprises the following steps: obtaining an image to be segmented; performing portrait segmentation on the image to be segmented by using a portrait segmentation model to obtain a portrait segmentation result, wherein the portrait segmentation result is used to represent a region where a human object in the image to be segmented is located; obtaining a screening mask, wherein the screening mask is a mask obtained based on other segmentation results, the other segmentation results are results of segmentation on the image to be segmented, and the other segmentation results are different from the portrait segmentation result; merging the screening mask and the portrait segmentation result to obtain a target object segmentation result; performing shadow segmentation on the image to be segmented by using a shadow segmentation model based on the target object segmentation result to obtain a shadow segmentation result, wherein the shadow segmentation result is used to represent a region where a shadow in the image to be segmented is located; fusing the shadow segmentation result and the target object segmentation result to obtain a fused segmentation result.

2. The method of claim 1, wherein, The method comprises the following steps: performing portrait segmentation on the image to be segmented by using the portrait segmentation model to obtain a portrait segmentation result, wherein the portrait segmentation result is used to represent a region where a human object in the image to be segmented is located. The method comprises the following steps:

3. The method of claim 2, wherein, performing portrait segmentation on the image to be segmented by using the portrait segmentation model to obtain at least one initial portrait mask; performing connected domain filtering and validity verification on the at least one initial portrait mask to obtain the portrait segmentation result. The method comprises the following steps:

4. The method according to any one of claims 1 to 3, characterized in that, performing a connected domain filtering operation on a first initial portrait mask to obtain at least one filtered portrait mask, wherein the first initial portrait mask is any one of the initial portrait masks; performing a validity verification operation on a first filtered portrait mask to obtain the portrait segmentation result, wherein the first filtered portrait mask is any one of the at least one filtered portrait mask. Before the step of obtaining the screening mask, the method further comprises the following steps: obtaining a lasso mask, wherein the lasso mask is a mask generated based on a lasso trajectory input by a user; performing segmentation on the image to be segmented by using a preset SAM segmentation model based on the lasso mask to obtain a SAM segmentation result, wherein the SAM segmentation result is used to represent a region where a segmented object in the image to be segmented is located; 5. The method of claim 4, wherein, cropping the image to be segmented according to the SAM segmentation result to obtain a main body segmentation image, wherein the main body segmentation image includes a region corresponding to the target object segmentation result, and an area of a region corresponding to the main body segmentation image is larger than an area of the region corresponding to the target object segmentation result; performing segmentation on the main body segmentation image by using a main body matting model to obtain a main body segmentation result, wherein the main body segmentation result is used to represent a region where a main object in the main body segmentation image is located.

6. The method of claim 5, wherein, Before the step of performing segmentation on the image to be segmented by using the preset SAM segmentation model based on the lasso mask, the method further comprises the following step: performing text detection on the image to be segmented by using a text detection model to obtain a text detection result. The method further comprises the following steps: if the fused segmentation result is empty, determining whether the text detection result is empty. If the text detection result is not empty, output the text detection result.

7. The method according to any one of claims 4 to 6, characterized in that, The SAM segmentation result is obtained by segmenting the to-be-segmented image based on the circle selection mask map using a preset SAM segmentation model, including: The circle selection mask map and the to-be-segmented image are input into the preset SAM segmentation model to obtain at least one initial SAM segmentation mask map; The initial SAM segmentation mask map with a predicted score lower than a first preset score threshold is deleted from the at least one initial SAM segmentation mask map to obtain at least one filtered initial SAM segmentation mask map; The connected domain filtering operation and the validity verification operation are performed on each of the at least one filtered initial SAM segmentation mask map to obtain the SAM segmentation result.

8. The method according to any one of claims 4 to 7, characterized in that, The main body segmentation result is obtained by segmenting the main body segmentation image using a main body matting model, including: The main body segmentation image is input into the main body matting model for segmentation to obtain at least one initial main body segmentation mask map; The connected domain filtering operation and the validity verification operation are performed on each of the at least one initial main body segmentation mask map to obtain the main body segmentation result.

9. The method according to any one of claims 4 to 8, characterized in that, The screening mask map is obtained, including: The screening mask map is screened from the SAM segmentation result, the main body segmentation result, and the circle selection mask map.

10. The method of claim 9, wherein, If the main body segmentation result is not empty, the screening mask map is the main body segmentation result; If the main body segmentation result is empty and the SAM segmentation result is not empty, the screening mask map is the SAM segmentation result; If the main body segmentation result is empty and the SAM segmentation result is empty, the screening mask map is the circle selection mask map.

11. The method according to any one of claims 4 to 10, characterized in that, The main body segmentation image is obtained by cropping the to-be-segmented image according to the SAM segmentation result, including: If the SAM segmentation result is empty, the to-be-segmented image is cropped according to a first proportion coefficient to obtain the main body segmentation image; If the SAM segmentation result is not empty, the to-be-segmented image is cropped according to a second proportion coefficient to obtain the main body segmentation image; The first proportion coefficient is greater than the second proportion coefficient, and both the first proportion coefficient and the second proportion coefficient are greater than 1.

12. The method of claim 2, wherein, The target object segmentation result is obtained by merging the screening mask map and the portrait segmentation result, including: If the portrait segmentation result is empty, the screening mask map is determined as the target object segmentation result; The shadow segmentation result is obtained by performing shadow segmentation on the to-be-segmented image based on the target object segmentation result using a shadow segmentation model, including: The minimum circumscribed rectangle of the object in the screening mask map and the to-be-segmented image are input into the shadow segmentation model for segmentation to obtain the shadow segmentation result.

13. The method of claim 2, wherein, The target object segmentation result is obtained by merging the screening mask map and the portrait segmentation result, including: If the portrait segmentation result is not empty, the screening mask map and the portrait segmentation result are fused to obtain the target object segmentation result; The shadow segmentation model is used for shadow segmentation on the to-be-segmented image based on the target object segmentation result, and a shadow segmentation result is obtained. The portrait mask graph in the target object segmentation result is input into the shadow segmentation model for shadow segmentation, and the shadow segmentation result is obtained.

14. The method of claim 2, 12 or 13, wherein, The shadow segmentation model is used for shadow segmentation on the to-be-segmented image based on the target object segmentation result, and a shadow segmentation result is obtained. The target object segmentation result and the to-be-segmented image are input into the shadow segmentation model to obtain an initial shadow segmentation result, and the initial shadow segmentation result includes at least one initial shadow mask graph. An initial shadow mask graph with a prediction score lower than a second preset score threshold is deleted from the at least one initial shadow mask graph to obtain at least one filtered initial shadow mask graph. A connected domain filtering operation is performed on each of the at least one filtered initial shadow mask graph to obtain the shadow segmentation result.

15. The method of any one of claims 7, 8, or 14, wherein, The connected domain filtering operation includes: A plurality of to-be-filtered connected domains in a first to-be-filtered mask image are obtained, the first to-be-filtered mask image being any one of at least one to-be-filtered mask image. It is determined whether a first ratio of an area of a first to-be-filtered connected domain to an area of the to-be-filtered mask image is greater than or equal to a first threshold value, the first to-be-filtered connected domain being any one of the plurality of to-be-filtered connected domains. If the first ratio is greater than or equal to the first threshold value, the first to-be-filtered connected domain is retained. If the first ratio is less than the first threshold value, the first to-be-filtered connected domain is deleted. A first maximum connected domain with a maximum area in the plurality of to-be-filtered connected domains is obtained. It is determined whether a second ratio of an area of a second to-be-filtered connected domain to an area of the first maximum connected domain is greater than or equal to a second threshold value, the second to-be-filtered connected domain being any one of the retained to-be-filtered connected domains. If the second ratio is greater than or equal to the second threshold value, the second to-be-filtered connected domain is retained. If the second ratio is less than the second threshold value, the second to-be-filtered connected domain is deleted. A sum of areas of the retained connected domains in the first to-be-verified mask graph is obtained. It is determined whether a third ratio of the sum of the areas of the retained connected domains to an area of the first to-be-verified mask graph is greater than or equal to a third threshold value. If the third ratio is greater than or equal to the third threshold value, the first to-be-verified mask graph is retained. If the third ratio is less than the third threshold value, the first to-be-verified mask graph is deleted. A width of a first connected domain and a height of the first connected domain in a second to-be-verified mask graph are obtained, the second to-be-verified mask graph being any one of the retained to-be-filtered mask images. If a fourth ratio of the width of the first connected domain to a width of the second to-be-verified mask graph is greater than or equal to a fourth threshold value, and / or a fifth ratio of the height of the first connected domain to a height of the second to-be-verified mask graph is greater than or equal to a fifth threshold value, the second to-be-verified mask graph is retained. If a fourth ratio of a width of the first connected domain to a width of the second to-be-verified mask image is less than a fourth threshold, and a fifth ratio of a height of the first connected domain to a height of the second to-be-verified mask image is less than a fifth threshold, the second to-be-verified mask image is deleted.

16. The method of any one of claims 7, 8, 14, or 15, wherein, The validity verification operation comprises: obtaining a first intersection of a plurality of to-be-verified connected domains in a first to-be-verified mask image and a selection region, the first to-be-verified mask image being any one of at least one to-be-verified mask image; calculating a sixth ratio of the first intersection to a sum of areas of the plurality of to-be-verified connected domains, and calculating a seventh ratio of the first intersection to an area of the selection region; if the sixth ratio is greater than or equal to a sixth threshold, and / or the seventh ratio is greater than or equal to a seventh threshold, the first to-be-verified mask image is retained; if the sixth ratio is less than the sixth threshold, and the seventh ratio is less than the seventh threshold, the first to-be-verified mask image is deleted.

17. The method of any one of claims 1 to 16, wherein, The fusion of the shadow segmentation result and the target object segmentation result comprises: obtaining a second intersection of a first target segmentation mask image and a first shadow segmentation mask image, the first target segmentation mask image being any one of the target object segmentation results, and the first shadow segmentation mask image being any one of the shadow segmentation results; obtaining a third ratio of an area of the second intersection to an area of a target object in the first target segmentation mask image; if the third ratio is greater than or equal to a third threshold, a union of the first target segmentation mask image and the first shadow segmentation mask image is determined as the fusion segmentation result; if the third ratio is less than the third threshold, the first target segmentation mask image is determined as the fusion segmentation result.

18. The method of any one of claims 1 to 17, wherein, The method further comprises: post-processing the fusion segmentation result to obtain an output segmentation result, in which there is no pixel point with a pixel value of 0 inside a region in which pixel points with a pixel value of 1 are distributed.

19. The method of claim 18, wherein, The post-processing of the fusion segmentation result to obtain an output segmentation result comprises: if the fusion segmentation result is not empty, post-processing the fusion segmentation result to obtain the output segmentation result.

20. The method of claim 18 or 19, wherein, The post-processing of the fusion segmentation result to obtain an output segmentation result comprises: obtaining an external contour of a region in which pixel points with a pixel value of 1 are located in the fusion segmentation result; updating a pixel value of a pixel point with a pixel value of 0 in the external contour to 1 to obtain an output segmentation result.

21. The method of claim 1, wherein, The method further comprises: receiving a selection operation input by a user, the selection operation being an operation of selecting a target segmentation object in an initial to-be-segmented image, the initial to-be-segmented image being an image before a size transformation of the to-be-segmented image; in response to the selection operation, displaying an animation effect in a region in which the target segmentation object and a target shadow are located in the initial to-be-segmented image, the target shadow being a shadow generated by the target segmentation object in the initial to-be-segmented image.

22. An electronic device, comprising: comprise: a processor, a memory, and an interface; The processor, the memory, and the interface cooperate such that the electronic device performs the method of any one of claims 1 to 21.

23. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored therein a computer program which, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 21.

Images