Intelligent matting method based on Segment Anything model

Abstract

The invention relates to the technical field of image processing, and discloses an intelligent matting method based on a Segment Anything model, and the method comprises the steps: obtaining a to-be-processed sample image, inputting the sample image into an Image Segment model, carrying out the segmentation, obtaining a preliminary foreground and background mask, and making a sample data set; a Segment Anything model is called, the sample data set is input into the Segment Anything model, and an object in the target image is segmented and recognized; obtaining an operation instruction of a user, matching the identified object mask according to a smearing operation of the user, and updating the mask; and carrying out optimization processing on the updated foreground mask edge, and outputting a segmentation result. The method can accurately identify and extract the target object in the image, intelligently delete unnecessary parts or increase missing areas, improve the accuracy of the matting result, quickly meet individual requirements of different users, quickly obtain the high-quality matting result, reduce the time of manual adjustment and trimming editing, and greatly improve the working efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an intelligent image cutting method based on a SegmentAnything model. Background Art

[0002] With the rapid development of image processing technology in recent years, intelligent cutouts have become a crucial application in image editing. Traditional cutout methods rely on manual editing tools, often requiring significant time and effort to adjust, and often producing unsatisfactory results. Especially when processing complex images, traditional methods struggle to accurately separate the user's target object from the background, resulting in significant variations in cutout quality. Summary of the Invention

[0003] The purpose of the present invention is to solve the above problems and design an intelligent cutout method based on the SegmentAnything model to solve the existing problems.

[0004] The present invention provides an intelligent cutout method based on a SegmentAnything model, the intelligent cutout method based on the SegmentAnything model comprising the following steps:

[0005] Obtain the sample image to be processed, input the sample image into the Image Segmentation model for segmentation, obtain preliminary foreground and background masks, and make a sample data set;

[0006] Calling the SegmentAnything model, inputting the sample data set into the SegmentAnything model, and segmenting and identifying objects in the target image;

[0007] Obtain the user's operation instructions, match the identified object mask according to the user's smear operation, and update the mask;

[0008] Optimize the updated foreground mask edge and output the segmentation result.

[0009] Optionally, in a first implementation of the present invention, obtaining a sample image to be processed, inputting the sample image into an Image Segmentation model for segmentation, obtaining preliminary foreground and background masks, and producing a sample dataset includes:

[0010] Get the image uploaded by the user to the server, adjust the image size to match the model input requirements, and obtain the sample image to be processed;

[0011] Input the sample image to be processed into the Image Segmentation model, and output a feature map with the same size as the input image, where each pixel value represents the probability of the pixel belonging to the foreground or background;

[0012] Use CNN-based convolutional neural network for pixel-level classification to generate foreground and background masks;

[0013] Apply a threshold to convert the probability map into a binary mask, set the threshold to 0.5, and regard pixels above the threshold as foreground and pixels below the threshold as background;

[0014] Overlay the generated mask with the original image to visualize the segmentation results.

[0015] Optionally, in a second implementation of the present invention, calling the SegmentAnything model, inputting the sample data set into the SegmentAnything model, and segmenting and identifying objects in the target image include:

[0016] Calling the SegmentAnything model and inputting the sample data set into the SegmentAnything model;

[0017] Extract features from images through the backbone network CNN and feature pyramid network FPN, and integrate features of different sizes;

[0018] The output consists of a detection branch and a segmentation branch;

[0019] The detection branch is responsible for identifying the object category and location in the image; generating the category label and bounding box coordinates of each object through classification and regression;

[0020] The segmentation branch is responsible for generating a detailed mask for each detected object; it processes the high-resolution feature map and outputs a mask coefficient for each pixel, which is used to define the probability that each pixel belongs to a different category, and then multiplied and added with the prototype to form the final segmentation mask.

[0021] Optionally, in a third implementation of the present invention, obtaining the user's operation instruction, matching the identified object mask according to the user's smearing operation, and updating the mask includes:

[0022] Obtain the user's operation instruction and determine whether the user performs a smearing operation;

[0023] If the user does not smudge, the preliminary segmentation result is output directly;

[0024] If the user smears, the overlap between the smeared area and all identified object masks is calculated by matching the overlapping area ratios A and B. If A > 0.2 and B > 0.1, the match is considered successful and the object mask is merged into the matching mask.

[0025] According to the user's operation mode, the matching mask is added or subtracted from the mask of the current foreground object to obtain an updated foreground mask.

[0026] Optionally, in a fourth implementation of the present invention, the step of optimizing the updated foreground mask edge and outputting the segmentation result includes the following steps:

[0027] Obtain an updated foreground mask, perform morphological operations and edge smoothing on the edges to trim and optimize the mask edges and improve the mask quality and consistency;

[0028] Finally, the optimized image segmentation results are output.

[0029] Optionally, in a fifth implementation of the present invention, the matching by calculating the overlapping area proportions A and B includes:

[0030] A is the area of ​​the overlap between the smeared area and the object mask / the area of ​​the smeared area;

[0031] B is the area of ​​the overlap between the smeared area and the object mask / the area of ​​the object mask.

[0032] The second aspect of the present invention provides an intelligent cutout device based on the SegmentAnything model, characterized in that the intelligent cutout device based on the SegmentAnything model includes an image segmentation module, an image recognition module, a smear update module and an optimization output module, wherein:

[0033] The image segmentation module is used to obtain the sample image to be processed, input the sample image into the ImageSegmentation model for segmentation, obtain preliminary foreground and background masks, and make a sample data set;

[0034] An image recognition module is used to call the SegmentAnything model, input the sample data set into the SegmentAnything model, and segment and recognize objects in the target image;

[0035] The smear update module is used to obtain the user's operation instructions, match the identified object mask according to the user's smear operation, and update the mask;

[0036] The optimization output module is used to optimize the updated foreground mask edge and output the segmentation result.

[0037] The third aspect of the present invention provides an intelligent cutout device based on the SegmentAnything model, characterized in that the intelligent cutout device based on the SegmentAnything model includes: a memory and at least one processor, the memory storing instructions; at least one of the processors calls the instructions in the memory to enable the intelligent cutout device based on the SegmentAnything model to execute each step of the intelligent cutout method based on the SegmentAnything model described above.

[0038] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed by a processor, implement the various steps of any of the above-mentioned intelligent cutout methods based on the SegmentAnything model.

[0039] The technical solution provided by the present invention obtains a sample image to be processed, inputs the sample image into the ImageSegmentation model for segmentation, obtains preliminary foreground and background masks, and creates a sample dataset. The SegmentAnything model is then called and the sample dataset is input into the SegmentAnything model to segment and identify objects in the target image. The user's operation instructions are obtained, and the identified object mask is matched and updated based on the user's smearing operation. The edges of the updated foreground mask are optimized and the segmentation result is output. This solution can accurately identify and extract target objects in images, intelligently delete unnecessary parts or add missing areas, improve the accuracy of the cutout results, quickly meet the personalized needs of different users, quickly obtain high-quality cutout results, reduce the time spent on manual adjustment and trimming and editing, and greatly improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Various other advantages and benefits will become apparent to those skilled in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0041] Figure 1 A schematic diagram of a first embodiment of intelligent cutout based on the SegmentAnything model provided in an embodiment of the present invention;

[0042] Figure 2 A schematic structural diagram of an intelligent cutout device based on the SegmentAnything model provided in an embodiment of the present invention;

[0043] Figure 3A schematic structural diagram of an intelligent cutout device based on the SegmentAnything model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, apparatus, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0045] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A first embodiment of an intelligent image cutout method based on a SegmentAnything model provided by an embodiment of the present invention is shown in FIG. 1 , and the method specifically includes the following steps:

[0046] Step 101: Obtain a sample image to be processed, input the sample image into the Image Segmentation model for segmentation, obtain preliminary foreground and background masks, and create a sample dataset;

[0047] Step 102: Call the SegmentAnything model, input the sample data set into the SegmentAnything model, and segment and identify the objects in the target image;

[0048] Step 103: Obtain the user's operation instruction, match the identified object mask according to the user's smear operation, and update the mask;

[0049] Step 104: Optimize the edges of the updated foreground mask and output the segmentation result.

[0050] A second embodiment of the intelligent cutout method based on the SegmentAnything model provided by an embodiment of the present invention includes:

[0051] Get the image uploaded by the user to the server, adjust the image size to match the model input requirements, and obtain the sample image to be processed;

[0052] Input the sample image to be processed into the Image Segmentation model, and output a feature map with the same size as the input image. Each pixel value represents the probability of the pixel belonging to the foreground or background.

[0053] Use CNN-based convolutional neural network for pixel-level classification to generate foreground and background masks;

[0054] Apply a threshold to convert the probability map into a binary mask. Set the threshold to 0.5, and consider pixels above the threshold as foreground and pixels below the threshold as background.

[0055] Overlay the generated mask with the original image to visualize the segmentation results.

[0056] A third embodiment of the intelligent cutout method based on the SegmentAnything model provided by an embodiment of the present invention includes:

[0057] Call the SegmentAnything model and input the sample dataset into the SegmentAnything model;

[0058] Extract features from images through the backbone network CNN and feature pyramid network FPN, and integrate features of different sizes;

[0059] The output consists of a detection branch and a segmentation branch;

[0060] The detection branch is responsible for identifying the object category and location in the image; it generates the category label and bounding box coordinates of each object through classification and regression;

[0061] The segmentation branch is responsible for generating detailed masks for each detected object; it processes the high-resolution feature map and outputs the mask coefficients for each pixel, which are used to define the probability of each pixel belonging to different categories, and then multiplied and added with the prototype to form the final segmentation mask.

[0062] A fourth embodiment of the intelligent cutout method based on the SegmentAnything model provided by an embodiment of the present invention includes:

[0063] According to the user's smear operation, the identified object mask is matched and the mask is updated, including:

[0064] Obtain the user's operation instruction and determine whether the user performs a smearing operation;

[0065] If the user does not smudge, the preliminary segmentation result is output directly;

[0066] If the user smears, the overlap between the smeared area and all the identified object masks is calculated by matching the overlapped area ratios A and B, where A is the area of ​​the overlapped area between the smeared area and the object mask / the area of ​​the smeared area.

[0067] B is the area of ​​the overlap between the smeared area and the object mask / the area of ​​the object mask;

[0068] If A>0.2 and B>0.1, the match is considered successful and the object mask is merged into the matching mask;

[0069] According to the user's operation mode, the matching mask is added or subtracted from the mask of the current foreground object to obtain an updated foreground mask.

[0070] A fifth embodiment of the intelligent cutout method based on the SegmentAnything model provided by an embodiment of the present invention includes:

[0071] Obtain an updated foreground mask, perform morphological operations and edge smoothing on the edges to trim and optimize the mask edges and improve the mask quality and consistency;

[0072] Finally, the optimized image segmentation results are output.

[0073] The above describes the intelligent cutout method based on the SegmentAnything model in the embodiment of the present invention. The following describes the intelligent cutout device based on the SegmentAnything model in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, an intelligent image cutting device includes:

[0074] Image segmentation module 201, used to obtain a sample image to be processed, input the sample image into the ImageSegmentation model for segmentation, obtain preliminary foreground and background masks, and make a sample data set;

[0075] Image recognition module 202, used to call the SegmentAnything model, input the sample data set into the SegmentAnything model, and segment and recognize objects in the target image;

[0076] The smear update module 203 is used to obtain the user's operation instruction, match the identified object mask according to the user's smear operation, and update the mask;

[0077] The optimization output module 204 is used to optimize the edges of the updated foreground mask and output the segmentation result.

[0078] The following is a detailed description of the intelligent cutout device based on the SegmentAnything model in an embodiment of the present invention from the perspective of hardware processing.

[0079] Figure 3 It is a structural diagram of an intelligent cutout device based on the SegmentAnything model provided by an embodiment of the present invention. The intelligent cutout device 600 based on the SegmentAnything model may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more massive storage devices) for storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be short-term storage or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the intelligent cutout device 600 based on the SegmentAnything model. Furthermore, the processor 610 can be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the intelligent cutout device 600 based on the SegmentAnything model to implement the steps of the intelligent cutout method based on the SegmentAnything model provided in the above-mentioned method embodiments.

[0080] The intelligent cutout device 600 based on the SegmentAnything model may further include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the structure of the intelligent cutout device based on the SegmentAnything model does not limit the intelligent cutout device based on the SegmentAnything model, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0081] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of an intelligent clipping method based on a SegmentAnything model.

[0082] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0084] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent cutout method based on Segment Anything model, characterized in that: The intelligent cutout method based on the SegmentAnything model comprises the following steps: Get the sample image to be processed, input the sample image into the Image Segmentation model for segmentation, obtain the preliminary foreground and background masks, and make a sample data set; Calling the Segment Anything model, inputting the sample data set into the Segment Anything model, and segmenting and identifying objects in the target image; Obtain the user's operation instruction, match the identified object mask according to the user's smearing operation, and update the mask; The updated foreground mask edge is optimized and the segmentation result is output.

2. The intelligent cutout method based on the Segment Anything model as claimed in claim 1, characterized in that: The method of obtaining a sample image to be processed, inputting the sample image into the Image Segmentation model for segmentation, obtaining preliminary foreground and background masks, and making a sample data set includes: Get the image uploaded by the user to the server, adjust the image size to match the model input requirements, and obtain the sample image to be processed; Input the sample image to be processed into the Image Segmentation model, and output a feature map with the same size as the input image, where each pixel value represents the probability of the pixel belonging to the foreground or the background; CNN-based convolutional neural network is used for pixel-level classification to generate foreground and background masks; Apply a threshold to convert the probability map into a binary mask, set the threshold to 0.5, and consider pixels above the threshold as foreground and pixels below the threshold as background; Overlay the generated mask with the original image to visualize the segmentation results.

3. The intelligent cutout method based on the Segment Anything model as claimed in claim 1, characterized in that: The calling of the Segment Anything model, inputting the sample data set into the Segment Anything model, and segmenting and identifying objects in the target image include: Calling the Segment Anything model, and inputting the sample data set into the Segment Anything model; Extract features from images through the backbone network CNN and feature pyramid network FPN, and integrate features of different sizes; The output consists of a detection branch and a segmentation branch; The detection branch is responsible for identifying the object category and location in the image; generating the category label and bounding box coordinates of each object through classification and regression; The segmentation branch is responsible for generating detailed masks for each detected object; it processes the high-resolution feature map and outputs the mask coefficients for each pixel, which are used to define the probability that each pixel belongs to different categories, and then multiplied and added with the prototype to form the final segmentation mask.

4. The intelligent cutout method based on the Segment Anything model as claimed in claim 1, characterized in that: The obtaining of the user's operation instruction, matching the identified object mask according to the user's smearing operation, and updating the mask includes: Obtain the user's operation instruction and determine whether the user performs a smearing operation; If the user does not smear, the preliminary segmentation result is directly output; If the user smears, the overlap between the smeared area and all the identified object masks is calculated by matching the overlapping area ratios A and B. If A>0.2 and B>0.1, the match is considered successful and the object mask is merged into the matching mask. According to the user's operation mode, the matching mask is added or subtracted from the mask of the current foreground object to obtain an updated foreground mask.

5. The intelligent cutout method based on the Segment Anything model as claimed in claim 1, characterized in that: The step of optimizing the edge of the updated foreground mask and outputting the segmentation result comprises the following steps: Obtain an updated foreground mask, perform morphological operations and edge smoothing on the edges to trim and optimize the mask edges and improve the mask quality and consistency; Finally, the optimized image segmentation result is output.

6. The intelligent cutout method based on the Segment Anything model as claimed in claim 4, characterized in that: The matching by calculating the overlapping area proportions A and B includes: A is the area of ​​the overlap between the smeared area and the object mask / the area of ​​the smeared area; B is the area of ​​the overlap between the smeared area and the object mask / the area of ​​the object mask.

7. An intelligent image cutting device based on the Segment Anything model, characterized in that: The intelligent cutout device based on the SegmentAnything model includes an image segmentation module, an image recognition module, a smear update module and an optimization output module, wherein: The image segmentation module is used to obtain the sample image to be processed, input the sample image into the Image Segmentation model for segmentation, obtain the preliminary foreground and background masks, and make a sample data set; An image recognition module is used to call the Segment Anything model, input the sample data set into the SegmentAnything model, and segment and recognize objects in the target image; The smear update module is used to obtain the user's operation instruction, match the identified object mask according to the user's smear operation, and update the mask; The optimization output module is used to optimize the edge of the updated foreground mask and output the segmentation result.

8. An intelligent image cutting device based on the Segment Anything model, characterized in that: The intelligent cutout device based on the SegmentAnything model includes: a memory and at least one processor, wherein instructions are stored in the memory; At least one of the processors calls the instructions in the memory so that the intelligent cutout device based on the SegmentAnything model executes the various steps of the intelligent cutout method based on the SegmentAnything model as described in any one of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the intelligent cutout method based on the Segment Anything model as described in any one of claims 1 to 6 are implemented.

Images