Image segmentation method, device, electronic device and storage medium

By identifying the main object key points of the reference image, automatically determining the segmentation prompt information and performing image segmentation, the problem of user manual input dependence in the prior art is solved, and the efficiency and accuracy of image segmentation are improved.

CN117788486BActive Publication Date: 2024-11-05SHUXING TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311816313.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-26
Publication Date
2024-11-05
Estimated Expiration
2043-12-26

AI Technical Summary

Technical Problem

The existing image segmentation technology requires users to manually input segmentation prompt information, resulting in segmentation accuracy dependent on user operations, and efficiency and accuracy are limited.

Method used

By identifying the key points of the subject object by the reference image to be segmented, the segmentation prompt information is automatically determined, and inputting it to the image segmentation model for segmentation, avoiding manual operations.

Benefits of technology

This improves the efficiency and accuracy of image segmentation, reduces dependence on user operations, and realizes an automated image segmentation process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117788486B_ABST
    Figure CN117788486B_ABST
Patent Text Reader

Abstract

The embodiments of the present application disclose an image segmentation method, device, electronic device and storage medium. The method includes: obtaining an image to be segmented, and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image; identifying the key points of the subject object for the reference image, and obtaining the position information of the key points of the subject object in the reference image; determining the first segmentation prompt information for indicating the position of the subject object in the image to be segmented according to the position information of the key points; inputting the image to be segmented and the first segmentation prompt information into an image segmentation model; performing subject object segmentation on the image to be segmented according to the first segmentation prompt information through the image segmentation model, and obtaining the first subject mask corresponding to the subject object in the image to be segmented. When segmenting the image, there is no need to manually input the segmentation prompt information, which can improve the efficiency and accuracy of image segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image segmentation method, device, electronic device and storage medium. Background Art

[0002] At present, the application of image segmentation technology is becoming more and more extensive, and the requirements for image segmentation are becoming higher and higher. In the related technology, a model can be used for image segmentation processing, but the user needs to manually input segmentation prompt information to achieve the segmentation of the main object in the image.

[0003] The problem with the related technology is that the user is required to manually input segmentation prompt information for each image, and the segmentation effect depends on the accuracy of the segmentation prompt information input by the user, that is, the segmentation accuracy is affected by the user's manual operation. Therefore, it is not conducive to improving the efficiency and accuracy of image segmentation. Summary of the invention

[0004] The embodiments of the present application provide an image segmentation method, device, electronic device and storage medium. When performing image segmentation, the key points of the main object are automatically identified for a reference image corresponding to the image to be segmented, so as to automatically determine the first segmentation prompt information according to the position information of the key points to perform image segmentation without manual operation, which is conducive to improving the efficiency and accuracy of image segmentation.

[0005] A first aspect of an embodiment of the present application provides an image segmentation method, the method comprising:

[0006] Acquire an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image;

[0007] Recognize the key points of the subject object in the reference image to obtain the position information of the key points of the subject object in the reference image;

[0008] Determine first segmentation prompt information for indicating the position of the main object in the image to be segmented according to the position information of the key point;

[0009] Inputting the image to be segmented and the first segmentation prompt information into an image segmentation model;

[0010] The image to be segmented is segmented into a main object by using the image segmentation model according to the first segmentation prompt information, so as to obtain a first main object mask corresponding to the main object in the image to be segmented.

[0011] A second aspect of an embodiment of the present application provides an image segmentation device, the device comprising:

[0012] An image acquisition module, used to acquire an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image;

[0013] A key point recognition module is used to recognize key points of the subject object in the reference image and obtain position information of the key points of the subject object in the reference image;

[0014] A prompt information generating module, used for determining first segmentation prompt information for indicating the position of the main object in the image to be segmented according to the position information of the key point;

[0015] A data transmission module, used for inputting the image to be segmented and the first segmentation prompt information into an image segmentation model;

[0016] The image segmentation module is used to segment the main object of the image to be segmented according to the first segmentation prompt information through the image segmentation model, and obtain a first main object mask corresponding to the main object in the image to be segmented.

[0017] In some optional embodiments, the prompt information generating module includes:

[0018] A first information acquisition unit, configured to use the position information of the key point as second segmentation prompt information;

[0019] A first segmentation unit is used to segment the main object of the image to be segmented according to the image segmentation model and the second segmentation prompt information, and obtain a second main object mask corresponding to the main object in the image to be segmented;

[0020] A sampling unit, used to sample pixel points within the second main body mask range to obtain position information of the sampling points;

[0021] The second information acquisition unit is used to determine the first segmentation prompt information according to the position information of the sampling points and the position information of the key points.

[0022] In some optional embodiments, the prompt information generating module further includes an image erosion unit, which is used to:

[0023] Get image corrosion control parameters;

[0024] The second main body mask is subjected to image corrosion processing according to the image corrosion control parameters to reduce the range corresponding to the second main body mask, thereby obtaining the processed second main body mask.

[0025] In some optional embodiments, the sampling unit is specifically used for:

[0026] Determine the sampling step size according to the image size of the image to be segmented;

[0027] Uniformly sampling the pixel points of the image to be segmented according to the sampling step length to obtain a plurality of sampling pixel points and position information of the sampling pixel points;

[0028] According to the position information of the sampling pixel points, the sampling pixel points within the second main body mask range are used as sampling points, and the position information corresponding to the sampling points is obtained.

[0029] In some optional embodiments, the sampling unit is specifically used for:

[0030] Obtain the minimum bounding box corresponding to the second subject mask;

[0031] Determine the sampling step size according to the size of the minimum bounding box;

[0032] The pixel points within the second main body mask range are uniformly sampled according to the sampling step length to obtain a plurality of sampling points and position information of the sampling points.

[0033] In some optional embodiments, the second information acquisition unit is specifically used to:

[0034] Determine the target distance threshold according to the sampling step size and the position information of the key points;

[0035] According to the position information of the key point, determine the target key point corresponding to the key point in the image to be segmented, and obtain the position information of the target key point;

[0036] Determine a target sampling point from the sampling points according to the target distance threshold, the position information of the sampling points in the image to be segmented, and the position information of the target key points, wherein the distance between the target sampling point and each target key point is greater than the target distance threshold;

[0037] The first segmentation prompt information is constructed according to the position information of the target sampling point and the position information of the key point.

[0038] In some optional embodiments, the second information acquisition unit is further specifically used for:

[0039] Calculate the minimum distance between the key points based on the position information of the key points;

[0040] The smaller one between the sampling step size and the minimum distance value is used as the target distance threshold.

[0041] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in the image segmentation method provided in the first aspect of the embodiment of the present application.

[0042] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the steps in the image segmentation method provided in the first aspect of the embodiment of the present application.

[0043] By adopting the scheme of the embodiment of the present application, an image to be segmented and a reference image corresponding to the image to be segmented can be obtained, wherein the image to be segmented is generated by a diffusion model based on the reference image; the key points of the main object are identified for the reference image to obtain the position information of the key points of the main object in the reference image; the first segmentation prompt information for indicating the position of the main object in the image to be segmented is determined based on the position information of the key points; the image to be segmented and the first segmentation prompt information are input into an image segmentation model; the image to be segmented is segmented for the main object according to the first segmentation prompt information through the image segmentation model to obtain the first main mask corresponding to the main object in the image to be segmented.

[0044] In this way, when performing image segmentation, the key points of the main object are automatically identified for the reference image corresponding to the image to be segmented, so that the first segmentation prompt information is automatically determined according to the position information of the key points. After the first segmentation prompt information and the image to be processed are input into the image segmentation model, the image segmentation model automatically performs main object segmentation on the image to be segmented according to the first segmentation prompt information. No manual operation is required, and the segmentation accuracy is not affected by the user's manual operation, which is conducive to improving the efficiency and accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is a schematic diagram of the timing of data processing for image segmentation by an image segmentation device provided in an embodiment of the present application;

[0047] Figure 2 It is a flowchart of an image segmentation method provided in an embodiment of the present application;

[0048] Figure 3This is an interactive schematic diagram of an image segmentation process provided by an embodiment of the present application;

[0049] Figure 4 is a structural block diagram of an image segmentation device provided in an embodiment of the present application;

[0050] Figure 5 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0052] The embodiments of the present application provide an image segmentation method, an apparatus, an electronic device, and a storage medium. Specifically, the image segmentation method of the embodiments of the present application can be performed by a computer device, wherein the computer device can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a personal computer (PC, Personal Computer), a personal digital assistant (PDA, Personal Digital Assistant), etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0053] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0054] Figure 1 FIG. 1 is a schematic diagram of a timing sequence of data processing for image segmentation performed by an image segmentation device provided in an embodiment of the present application. Figure 1 As shown, in the embodiment of the present application, image segmentation processing is performed according to the process of steps S0 to S5. Among them, step S0 is the process of pre-generating the image to be segmented.

[0055] It should be noted that in the embodiment of the present application, the image to be segmented is an image generated by a diffusion model. In actual use, the image to be segmented may also be an image obtained by other means based on a reference image, which is not specifically limited here.

[0056] Specifically, in the field of generative artificial intelligence (AICG, Artificial Intelligence Generated Content), diffusion models are usually used for image generation. Specifically, in the embodiment of the present application, the diffusion model used to generate the image to be segmented can be a StableDiffusion model. In actual use, other models can also be used, which are not specifically limited here. When the diffusion model generates an image, a reference image A and guide prompts are input, the guide prompts are text-encoded, the reference image A is self-encoded and Gaussian noise is added, and then the StableDiffusion denoising process is used to obtain a style virtual character image B that conforms to the text description of the guide prompts and the character reference image A. In the embodiment of the present application, image B is used as the image to be segmented, that is, the area where the main body of the character in image B is located needs to be segmented.

[0057] At the same time, image segmentation models such as the general segmentation model (SAM, Segment anything model) can be used for image segmentation. However, when performing character segmentation on an image generated by a diffusion model, the character image generated by the diffusion model may be a real person, a virtual person, or a person of different styles, or a person generated by superimposing different styles. Therefore, the segmentation effect is not good, and it is difficult to accurately segment the area where the character is located. In addition, for the SAM model, the user needs to manually input the segmentation prompt information prompt, and the segmentation prompt information cannot be automatically obtained, which affects the efficiency of image segmentation.

[0058] In the image segmentation method provided in the embodiment of the present application, the first segmentation prompt information can be automatically generated, thereby improving the efficiency of image segmentation.

[0059] The first aspect of the present application embodiment provides an image segmentation method. Figure 2 , Figure 2 : is a flow chart of an image segmentation method provided in an embodiment of the present application. The specific flow of the image segmentation method can be as follows:

[0060] 201. Obtain an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image.

[0061] In the embodiment of the present application, image segmentation is performed on the image to be segmented that is generated in advance by the diffusion model. It should be noted that the image to be segmented generated based on the reference image has the same image size as the reference image.

[0062] 202. Perform key point recognition of the subject object on the reference image to obtain position information of the key points of the subject object in the reference image.

[0063] It should be noted that the image style of the image to be segmented may be relatively complex, and it is difficult to determine the location of the main object (i.e., the human subject). However, the image style of the reference image used by the diffusion model is usually relatively simple, and the image to be segmented is generated with reference to the reference image. Therefore, in the embodiment of the present application, the location information of the key points of the main object in the reference image can be obtained by identifying the key points of the main object in the reference image. The location information of the key points in the reference image can be used to roughly indicate the location of the main object in the image to be segmented.

[0064] In some embodiments of the present application, a preset two-dimensional human key point recognition model is used to input a reference image (i.e., image A) for character key point recognition. It should be noted that the two-dimensional human key point recognition model can be selected and adjusted according to actual needs. For example, an openpose model or RTMPose can be used, which is not specifically limited here.

[0065] Further, multiple key points can be identified through the two-dimensional human key point recognition model. In the embodiment of the present application, some of the key points are selected, for example, the coordinates of the key points of the nose, left and right eyes, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles can be selected to construct the position information of the key points of the subject object in the reference image Point2d = [[x1, y1], [x2, y2], ..., [x15, y15]], that is, including a total of 15 two-dimensional coordinates. In actual use, other key points can also be used, which are not specifically limited here.

[0066] 203. Determine first segmentation prompt information for indicating the position of the main object in the image to be segmented according to the position information of the key point.

[0067] It should be noted that the segmentation prompt information (including the first segmentation prompt information and the second segmentation prompt information) used in the embodiments of the present application can be the position of a point in the image, or a detection box, a segmentation mask, etc. used to indicate the range of a certain area in the image. The segmentation prompt information is taken as the position of the point (i.e., the coordinates) as an example for explanation in the present application, but it is not a specific limitation.

[0068] Because the image size of the reference image and the image to be segmented is the same, the pixels in the reference image and the image to be segmented also correspond one to one. Usually, the area where the main object in the image to be segmented is located is relatively close to the area where the main object in the reference image is located.

[0069] In one application scenario, the position information of the key point can be directly used as the first segmentation prompt information. At this time, the position information Point2d of the key point of the main object in the reference image is used as the first segmentation prompt information, and the first segmentation prompt information and the image to be segmented (i.e., image B) are input into the image segmentation model to obtain the first main body mask indicating the area where the main object is located in the image to be segmented.

[0070] In another application scenario, the position information of the key points may be further processed to obtain more accurate first segmentation prompt information to improve the accuracy of image segmentation.

[0071] Specifically, the first segmentation prompt information for indicating the position of the main object in the image to be segmented is determined according to the position information of the key point, including:

[0072] Using the position information of the key points as the second segmentation prompt information;

[0073] Performing subject object segmentation on the image to be segmented according to the image segmentation model and the second segmentation prompt information, and obtaining a second subject mask corresponding to the subject object in the image to be segmented;

[0074] Sampling pixel points within the second main body mask range to obtain position information of the sampling points;

[0075] The first segmentation prompt information is determined according to the position information of the sampling points and the position information of the key points.

[0076] At this time, the position information Point2d of the key point of the main object in the reference image is used as the second segmentation prompt information, and the second segmentation prompt information and the image to be segmented (i.e., image B) are input into the image segmentation model to obtain a second main body mask indicating the area where the main object is located in the image to be segmented. The pixel points in the second main body mask are sampled to generate more accurate first segmentation prompt information.

[0077] Specifically, the above-mentioned image segmentation model can be selected according to actual needs. For example, the SAM vit-l model is used as the image segmentation model in the embodiment of the present application. The SAM vit-l model is a general segmentation model with the Vision Transformer Large Model (ViT-Large) as the basic model.

[0078] In the embodiment of the present application, the second segmentation prompt information and the image to be segmented (ie, image B) are input into the image segmentation model to obtain the second subject mask Mask_B. Then, pixel sampling is performed according to the second subject mask to generate more accurate first segmentation prompt information.

[0079] It should be noted that in order to avoid the sampling points exceeding the area range of the second main body mask or falling on the boundary of the second main body mask and affecting the image segmentation result, in the embodiment of the present application, the second main body mask is first eroded and then the pixel sampling is performed.

[0080] Specifically, before sampling the pixel points within the second main body mask range to obtain the position information of the sampling points, the method further includes:

[0081] Get image corrosion control parameters;

[0082] The second main body mask is subjected to image corrosion processing according to the image corrosion control parameters to reduce the range corresponding to the second main body mask, thereby obtaining the processed second main body mask.

[0083] The above-mentioned image corrosion control parameters are parameters for controlling the range of image corrosion, which can be preset or adjusted according to actual needs and are not specifically limited here.

[0084] In the embodiment of the present application, the image erosion parameters are set to kernel=5, iterations=6, that is, 5 pixels are reduced each time, and the iterative process is performed 6 times. Specifically, according to the image erosion control parameters of kernel=5, iterations=6, the image erosion process is performed on Mask_B to obtain a second main body mask Mask_B_erode with a reduced range, and pixel sampling is performed based on the processed second main body mask.

[0085] In an application scenario, sampling the pixel points within the second subject mask range to obtain the position information of the sampling points includes:

[0086] Determine the sampling step size according to the image size of the image to be segmented;

[0087] Uniformly sampling the pixel points of the image to be segmented according to the sampling step length to obtain a plurality of sampling pixel points and position information of the sampling pixel points;

[0088] According to the position information of the sampling pixel points, the sampling pixel points within the second main body mask range are used as sampling points, and the position information corresponding to the sampling points is obtained.

[0089] Specifically, the image size of the image to be segmented is width_B, height_B, and the sampling step lengths in the width and height directions are determined according to a preset sampling step control parameter. The sampling step control parameter can be set and adjusted according to actual needs, for example, it can be set to 12, and is not specifically limited here.

[0090] Furthermore, the corresponding sampling step size is int(width_B / 12) in the width direction and int(height_B / 12) in the height direction. Uniform sampling is performed along the width and height directions to obtain 169 (169=13x13) sampling pixels, and the position information of the sampling pixels is obtained: Point_Sample=[[0,0],[0,12],[12,0],…,[int(width_B / 12)*12,int(height_B / 12)*12]].

[0091] At this time, sampling is performed on the entire image to be segmented, so the sampling pixel points do not necessarily fall within the second main body mask range. In the embodiment of the present application, only the sampling pixel points within the second main body mask range are used as sampling points.

[0092] In another application scenario, the step of sampling the pixel points within the second main body mask range to obtain the position information of the sampling points includes:

[0093] Obtain the minimum bounding box corresponding to the second subject mask;

[0094] Determine the sampling step size according to the size of the minimum bounding box;

[0095] The pixel points within the second main body mask range are uniformly sampled according to the sampling step length to obtain a plurality of sampling points and position information of the sampling points.

[0096] Specifically, when determining the sampling step length according to the size of the minimum bounding box, the width and height of the minimum bounding box are divided by the above sampling step control parameters to determine the corresponding sampling step lengths in the width and height directions. The specific sampling process is similar to the above description and will not be repeated here.

[0097] In this way, only the points within the minimum bounding box area corresponding to the second main body mask are sampled, which can reduce the amount of data processing and further improve the processing efficiency.

[0098] The determining of the first segmentation prompt information according to the position information of the sampling points and the position information of the key points includes:

[0099] Determine the target distance threshold according to the sampling step size and the position information of the key points;

[0100] According to the position information of the key point, determine the target key point corresponding to the key point in the image to be segmented, and obtain the position information of the target key point;

[0101] Determine a target sampling point from the sampling points according to the target distance threshold, the position information of the sampling points in the image to be segmented, and the position information of the target key points, wherein the distance between the target sampling point and each target key point is greater than the target distance threshold; the distance between the target sampling point and the target key point is determined according to the position information of the target sampling point and the position information of the target key point;

[0102] The first segmentation prompt information is constructed according to the position information of the target sampling point and the position information of the key point.

[0103] It should be noted that, for each key point or sampling point for constructing the first segmentation prompt information, if the distance is too close, it will affect the accuracy of image segmentation, and too many points will affect the image segmentation efficiency. Therefore, in the embodiment of the present application, the sampling points are further screened to obtain the target sampling points therein to ensure that the distance between each point is greater than the target distance threshold.

[0104] Among them, the size of the image to be segmented and the reference image are the same, and the pixels are also one-to-one corresponding. Therefore, by overlapping the image to be segmented and the reference image, or establishing the same coordinate system for the two, the position of the target key point in the image to be segmented can be determined.

[0105] In the embodiment of the present application, an example is taken to establish the same coordinate system for the image to be segmented and the reference image. Then, the coordinates of a corresponding set of key points and the target key points are the same (that is, the position information is the same), and the position information of the key points can be directly used as the position information of the target key points.

[0106] Specifically, the target distance threshold is determined according to the sampling step and the position information of the key point, including:

[0107] Calculate the minimum distance between the key points based on the position information of the key points;

[0108] The smaller one between the sampling step size and the minimum distance value is used as the target distance threshold.

[0109] It should be noted that the target distance threshold may also be pre-set and adjusted according to actual needs, and is not specifically limited here.

[0110] In an application scenario, for a sampling pixel point, if the sampling pixel point meets the following two conditions, it is used as a target sampling point. The first condition is that the coordinates of the sampling pixel point are located in the segmentation area of ​​the second main body mask Mask_B_erode after the erosion process; the second condition is that the distance from any point in Point2d exceeds the threshold threshold, where the threshold hreshold = min (int(width_B / 12), int(height_B / 12), the minimum distance between any two points in Point2D). Construct the point set Point_Mask corresponding to the target sampling point that meets the above two conditions.

[0111] The two point sets Point2d and Point_Mask are merged to obtain a new point set Point2D_New, and the position information stored in Point2D_New is used as the first segmentation prompt information.

[0112] 204. Input the image to be segmented and the first segmentation prompt information into an image segmentation model.

[0113] 205 . Perform subject object segmentation on the image to be segmented according to the first segmentation prompt information by using the image segmentation model, and obtain a first subject mask corresponding to the subject object in the image to be segmented.

[0114] In an embodiment of the present application, the location information stored in the above Point2D_New is used as the first segmentation prompt information (i.e., guidance information), and is input into the decoder stage (i.e., decoding stage) of the image segmentation model SAM vit-1 that has performed image segmentation once according to the second segmentation prompt information, and is calculated again to obtain the final segmentation result Mask as the first subject mask. Among them, the above first subject mask indicates the area where the subject object is located in the image to be segmented. In an application scenario, the subject object segmentation processing can also be performed on the image to be segmented based on the above first subject mask to obtain an image containing only the subject object, which is not specifically limited here.

[0115] Figure 3 is an interactive schematic diagram of an image segmentation process provided by an embodiment of the present application, such as Figure 3As shown, the user inputs the image to be segmented and its reference image generated in advance by the diffusion model, and then the key points of the subject object are identified on the reference image to obtain the position information of the key points in the subject object, and the first segmentation prompt information is automatically determined based on this, and the image segmentation model is used to perform image segmentation processing to obtain the first subject mask. In this way, an automatic prompt (segmentation prompt information) extraction method based on SAM is provided, which does not require manual setting of prompts. The user only needs to input the corresponding image to automatically complete the image segmentation, which can improve the processing efficiency and can better segment the human image generated by the diffusion model.

[0116] By adopting the scheme of the embodiment of the present application, an image to be segmented and a reference image corresponding to the image to be segmented can be obtained, wherein the image to be segmented is generated by a diffusion model based on the reference image; the key points of the main object are identified for the reference image to obtain the position information of the key points of the main object in the reference image; the first segmentation prompt information for indicating the position of the main object in the image to be segmented is determined based on the position information of the key points; the image to be segmented and the first segmentation prompt information are input into an image segmentation model; the image to be segmented is segmented for the main object according to the first segmentation prompt information through the image segmentation model to obtain the first main mask corresponding to the main object in the image to be segmented.

[0117] In this way, when performing image segmentation, the key points of the main object are automatically identified for the reference image corresponding to the image to be segmented, so that the first segmentation prompt information is automatically determined according to the position information of the key points. After the first segmentation prompt information and the image to be processed are input into the image segmentation model, the image segmentation model automatically performs main object segmentation on the image to be segmented according to the first segmentation prompt information. No manual operation is required, and the segmentation accuracy is not affected by the user's manual operation, which is conducive to improving the efficiency and accuracy of image segmentation.

[0118] With respect to the image segmentation method provided in the first aspect of the embodiment of the present application, the second aspect of the embodiment of the present application provides an image segmentation device, see Figure 4 , Figure 4 : is a structural block diagram of an image segmentation device provided in an embodiment of the present application, the device comprising:

[0119] An image acquisition module 401 is used to acquire an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image;

[0120] A key point recognition module 402 is used to recognize key points of the subject object in the reference image and obtain position information of the key points of the subject object in the reference image;

[0121] A prompt information generating module 403 is used to determine first segmentation prompt information for indicating the position of the main object in the image to be segmented according to the position information of the key point;

[0122] A data transmission module 404 is used to input the image to be segmented and the first segmentation prompt information into an image segmentation model;

[0123] The image segmentation module 405 is used to segment the main object of the image to be segmented according to the first segmentation prompt information by using the image segmentation model to obtain a first main object mask corresponding to the main object in the image to be segmented.

[0124] In some optional embodiments, the prompt information generating module 403 includes:

[0125] A first information acquisition unit, configured to use the position information of the key point as second segmentation prompt information;

[0126] A first segmentation unit is used to segment the main object of the image to be segmented according to the image segmentation model and the second segmentation prompt information, and obtain a second main object mask corresponding to the main object in the image to be segmented;

[0127] A sampling unit, used to sample pixel points within the second main body mask range to obtain position information of the sampling points;

[0128] The second information acquisition unit is used to determine the first segmentation prompt information according to the position information of the sampling points and the position information of the key points.

[0129] In some optional embodiments, the prompt information generating module 403 further includes an image erosion unit, which is used to:

[0130] Get image corrosion control parameters;

[0131] The second main body mask is subjected to image corrosion processing according to the image corrosion control parameters to reduce the range corresponding to the second main body mask, thereby obtaining the processed second main body mask.

[0132] In some optional embodiments, the sampling unit is specifically used for:

[0133] Determine the sampling step size according to the image size of the image to be segmented;

[0134] Uniformly sampling the pixel points of the image to be segmented according to the sampling step length to obtain a plurality of sampling pixel points and position information of the sampling pixel points;

[0135] According to the position information of the sampling pixel points, the sampling pixel points within the second main body mask range are used as sampling points, and the position information corresponding to the sampling points is obtained.

[0136] In some optional embodiments, the sampling unit is specifically used for:

[0137] Obtain the minimum bounding box corresponding to the second subject mask;

[0138] Determine the sampling step size according to the size of the minimum bounding box;

[0139] The pixel points within the second main body mask range are uniformly sampled according to the sampling step length to obtain a plurality of sampling points and position information of the sampling points.

[0140] In some optional embodiments, the second information acquisition unit is specifically used to:

[0141] Determine the target distance threshold according to the sampling step size and the position information of the key points;

[0142] According to the position information of the key point, determine the target key point corresponding to the key point in the image to be segmented, and obtain the position information of the target key point;

[0143] Determine a target sampling point from the sampling points according to the target distance threshold, the position information of the sampling points in the image to be segmented, and the position information of the target key points, wherein the distance between the target sampling point and each target key point is greater than the target distance threshold;

[0144] The first segmentation prompt information is constructed according to the position information of the target sampling point and the position information of the key point.

[0145] In some optional embodiments, the second information acquisition unit is further specifically used for:

[0146] Calculate the minimum distance between the key points based on the position information of the key points;

[0147] The smaller one between the sampling step size and the minimum distance value is used as the target distance threshold.

[0148] The embodiment of the present application discloses an image segmentation device, which acquires an image to be segmented and a reference image corresponding to the image to be segmented through an image acquisition module 401, wherein the image to be segmented is generated by a diffusion model based on the reference image; a key point recognition module 402 performs subject object key point recognition on the reference image to obtain position information of the key points of the subject object in the reference image; a prompt information generation module 403 determines first segmentation prompt information for indicating the position of the subject object in the image to be segmented according to the position information of the key points; a data transmission module 404 inputs the image to be segmented and the first segmentation prompt information into an image segmentation model; an image segmentation module 405 performs subject object segmentation on the image to be segmented according to the first segmentation prompt information through the image segmentation model, and obtains a first subject mask corresponding to the subject object in the image to be segmented.

[0149] In this way, when performing image segmentation, the key points of the main object are automatically identified for the reference image corresponding to the image to be segmented, so that the first segmentation prompt information is automatically determined according to the position information of the key points. After the first segmentation prompt information and the image to be processed are input into the image segmentation model, the image segmentation model automatically performs main object segmentation on the image to be segmented according to the first segmentation prompt information. No manual operation is required, and the segmentation accuracy is not affected by the user's manual operation, which is conducive to improving the efficiency and accuracy of image segmentation.

[0150] It should be noted that the division of each module in the above device can be determined according to actual needs and is not specifically limited here.

[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] Accordingly, the embodiment of the present application also provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), or other terminal devices. Figure 5 As shown, Figure 5 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer-readable storage media, and a computer program stored in the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. Those skilled in the art can understand that Figure 5The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include Figure 5 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0153] The processor 501 is the control center of the electronic device 500, and uses various interfaces and lines to connect various parts of the entire electronic device 500. By running or loading software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, the processor 501 executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device 500 as a whole. The processor 501 can be a central processing unit CPU, a graphics processing unit GPU, a network processor (NP, Network Processor), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0154] In the embodiment of the present application, the processor 501 in the electronic device 500 will load instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 will run the application programs stored in the memory 502 to implement various functions, such as:

[0155] Acquire an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image;

[0156] Recognize the key points of the subject object in the reference image to obtain the position information of the key points of the subject object in the reference image;

[0157] Determine first segmentation prompt information for indicating the position of the main object in the image to be segmented according to the position information of the key point;

[0158] Inputting the image to be segmented and the first segmentation prompt information into an image segmentation model;

[0159] The image to be segmented is segmented into a main object by using the image segmentation model according to the first segmentation prompt information, so as to obtain a first main object mask corresponding to the main object in the image to be segmented.

[0160] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0161] Optional, such as Figure 5As shown, the electronic device 500 further includes: a touch screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507, respectively. Those skilled in the art can understand that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include Figure 5 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0162] The touch display screen 503 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 503 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 501, and can receive the command sent by the processor 501 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 501 to determine the type of touch event, and then the processor 501 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 503 to realize the input and output functions. However, in some embodiments, the touch panel and the display panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 503 can also be used as a part of the input unit 506 to realize the input function.

[0163] The radio frequency circuit 504 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0164] The audio circuit 505 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 505 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 505 and converted into audio data, and then the audio data is output to the processor 501 for processing, and then sent to another electronic device through the radio frequency circuit 504, or the audio data is output to the memory 502 for further processing. The audio circuit 505 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0165] The input unit 506 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0166] The power supply 507 is used to supply power to various components of the electronic device 500. Optionally, the power supply 507 can be logically connected to the processor 501 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 507 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0167] although Figure 5 Not shown, the electronic device 500 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0169] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0170] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any one of the image segmentation methods provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0171] Acquire an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image;

[0172] Recognize the key points of the subject object in the reference image to obtain the position information of the key points of the subject object in the reference image;

[0173] Determine first segmentation prompt information for indicating the position of the main object in the image to be segmented according to the position information of the key point;

[0174] Inputting the image to be segmented and the first segmentation prompt information into an image segmentation model;

[0175] The image to be segmented is segmented into a main object by using the image segmentation model according to the first segmentation prompt information, so as to obtain a first main object mask corresponding to the main object in the image to be segmented.

[0176] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0177] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0178] Since the computer program stored in the storage medium can execute the steps in any image segmentation method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image segmentation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0179] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0180] The image segmentation method, device, electronic device and storage medium provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image segmentation method, characterized in that: include: Acquire an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image; Performing subject object key point recognition on the reference image to obtain position information of the key points of the subject object in the reference image; Using the position information of the key point as second segmentation prompt information; Performing subject object segmentation on the image to be segmented according to the image segmentation model and the second segmentation prompt information, and obtaining a second subject mask corresponding to the subject object in the image to be segmented; Sampling pixel points within the mask range of the second subject to obtain position information of the sampling points; Determine first segmentation prompt information according to the position information of the sampling point and the position information of the key point; Inputting the image to be segmented and the first segmentation prompt information into an image segmentation model; The image to be segmented is subjected to subject object segmentation by using the image segmentation model according to the first segmentation prompt information, and a first subject mask corresponding to the subject object in the image to be segmented is obtained.

2. The image segmentation method according to claim 1, characterized in that: Before sampling the pixel points within the second main body mask range to obtain the position information of the sampling points, the method further includes: Get image corrosion control parameters; The second main body mask is subjected to image corrosion processing according to the image corrosion control parameter to reduce the range corresponding to the second main body mask, and a processed second main body mask is obtained.

3. The image segmentation method according to claim 1, characterized in that: The sampling of the pixel points within the second main body mask range to obtain the position information of the sampling points includes: Determine the sampling step size according to the image size of the image to be segmented; Uniformly sampling the pixel points of the image to be segmented according to the sampling step length to obtain a plurality of sampling pixel points and position information of the sampling pixel points; According to the position information of the sampling pixel points, the sampling pixel points within the second main body mask range are used as sampling points, and the position information corresponding to the sampling points is obtained.

4. The image segmentation method according to claim 1, characterized in that: The sampling of the pixel points within the second main body mask range to obtain the position information of the sampling points includes: Obtaining a minimum bounding box corresponding to the second subject mask; Determine the sampling step length according to the size of the minimum bounding box; The pixel points within the second main body mask range are uniformly sampled according to the sampling step size to obtain a plurality of sampling points and position information of the sampling points.

5. The image segmentation method according to claim 3 or 4, characterized in that: The determining the first segmentation prompt information according to the position information of the sampling point and the position information of the key point includes: Determine a target distance threshold according to the sampling step and the position information of the key point; According to the position information of the key point, determine the target key point corresponding to the key point in the image to be segmented, and obtain the position information of the target key point; Determine a target sampling point from the sampling points according to the target distance threshold, the position information of the sampling points in the image to be segmented, and the position information of the target key points, wherein the distance between the target sampling point and each target key point is greater than the target distance threshold; The first segmentation prompt information is constructed according to the position information of the target sampling point and the position information of the key point.

6. The image segmentation method according to claim 5, characterized in that: The determining the target distance threshold according to the sampling step and the position information of the key point includes: Calculate the minimum distance between the key points according to the position information of the key points; The smaller one between the sampling step size and the minimum distance value is used as the target distance threshold.

7. An image segmentation device, characterized in that: include: An image acquisition module, used for acquiring an image to be segmented and a reference image corresponding to the image to be segmented, wherein the image to be segmented is generated by a diffusion model according to the reference image; A key point recognition module, used to recognize key points of the subject object in the reference image and obtain position information of the key points of the subject object in the reference image; a prompt information generating module, configured to use the position information of the key point as the second segmentation prompt information; segment the image to be segmented into a main object according to the image segmentation model and the second segmentation prompt information, and obtain a second main object mask corresponding to the main object in the image to be segmented; sample the pixel points within the range of the second main object mask to obtain the position information of the sampling points; and determine the first segmentation prompt information according to the position information of the sampling points and the position information of the key point; A data transmission module, used for inputting the image to be segmented and the first segmentation prompt information into an image segmentation model; The image segmentation module is used to perform subject object segmentation on the image to be segmented according to the first segmentation prompt information through the image segmentation model, and obtain a first subject mask corresponding to the subject object in the image to be segmented.

8. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the image segmentation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image segmentation method according to any one of claims 1 to 6.