Image processing method and system, electronic equipment, storage medium and program product
By point sampling and quad-tree algorithm segmentation of the target image, selectable areas are displayed, and user interactively selecting the cutout area, solving the problem of multiple cutouts or missing cutouts in the existing technology, achieving a more efficient cutout effect.
Patent Information
- Application Number
- CN202410957465.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
AI Technical Summary
The existing cutting technology is prone to multiple objects and complex scenes when processing multiple objects in images, which is difficult to meet the actual needs of users.
By point sampling of the target image, the sampling point information is obtained, and the full image segmentation is combined with the quad-tree algorithm to display multiple optional areas. Users can interactively select the cutout area and use the preset full image segmentation model to cut out. Diagram processing.
It improves the fineness and accuracy of picture cutting, avoids too much or missed, and provides a better user interaction experience, so that users can complete picture cutting without professional knowledge.
Smart Images

Figure CN118918125A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image cutting technology, and in particular to an image processing method, system, electronic device, storage medium and program product. Background Art
[0002] In many scenarios, it is necessary to cut out an image in order to extract the area or object of interest to the user in the image and form a separate image. Currently, among the technologies related to realizing cutouts, one is to use offline cutout tools to realize cutouts. The use of cutout tools requires high professionalism from the user and requires manual cutouts, which is slow and inefficient. Another method is to realize automatic cutouts through the automatic cutout function provided online. However, when the image contains multiple objects and is complex, it is easy to have multiple cutouts or missed cutouts, making the cutout results unable to meet the actual cutout needs of the user. Therefore, it is urgent to provide an automatic cutout technology solution that can meet the actual cutout needs of the user. Summary of the invention
[0003] In view of the above problems mentioned in the background technology, the present application provides an image processing method, system, electronic device, storage medium and program product that solve the above problems or at least partially solve the above problems, so as to clarify the user's actual cutout intention through interaction with the user and improve the quality of automated cutout.
[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:
[0005] In combination with the image content of the target image, point sampling is performed on the target image to obtain sampling point information;
[0006] Based on the sampling point information, the target image is segmented in its entirety;
[0007] Displaying a segmentation map corresponding to the target image on a client interface, wherein the segmentation map includes a plurality of segmented optional areas;
[0008] In response to the area selection operation triggered for the multiple optional areas, the target image is cut out according to at least one selected optional area to obtain a cutout image.
[0009] In a second aspect, an embodiment of the present application provides another image processing method. The method includes:
[0010] Acquiring segmentation prompt information of the target image according to color information of the target image;
[0011] Using the segmentation prompt information and the target image, triggering a preset full-image segmentation model to perform full-image segmentation on the target image;
[0012] On the client interface, a segmentation map corresponding to the target image is displayed, wherein the segmentation map includes a plurality of segmented optional areas;
[0013] In response to the area selection operation triggered for the multiple optional areas, the target image is cut out according to at least one selected optional area to obtain a cutout image;
[0014] The cutout image is displayed on the client interface.
[0015] In a third aspect, the present application provides another image processing method. The method includes:
[0016] Get the product image to be cut out;
[0017] Based on the image content of the commodity image, point sampling is performed on the commodity image to obtain sampling point information;
[0018] Based on the sampling point information, the product image is segmented into its entirety;
[0019] On the client interface, a segmentation map corresponding to the product map is displayed, wherein the segmentation map includes a plurality of segmented object areas;
[0020] In response to the area selection operation triggered for the plurality of object areas, a target object body is separated from the commodity image according to at least one selected object area to obtain a display image of the target object body;
[0021] A display diagram showing the main body of the target object.
[0022] In a fourth aspect, an embodiment of the present application provides an image processing system. The system includes:
[0023] The client is used to obtain the target image to be cut out;
[0024] The server is used to implement the steps in the image processing method provided in the above-mentioned embodiments of the present application.
[0025] In a fifth aspect, an embodiment of the present application provides an electronic device. The electronic device includes a memory and a processor, wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in each method embodiment provided in the present application;
[0026] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program or instruction; when the computer program or instruction is executed by a processor, the steps in the above-mentioned method embodiments provided in the present application can be implemented.
[0027] In a seventh aspect, an embodiment of the present application provides a computer program product. The computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments provided in the present application.
[0028] The technical solution provided by the embodiment of the present application is to obtain sampling point information by sampling the target image based on the image content of the target image (such as the color information of the target image), so as to perform full-image segmentation on the target image based on the sampling point information. This full-image segmentation method enables the segmentation to be adaptively changed according to the image content information of the target image, which can improve the full-image area division effect of the target image and balance the function time consumption. For example, the sampling density can be increased for the color-rich and complex areas in the target image to achieve more refined segmentation, while the sampling density can be reduced for the areas with similar colors to reduce the redundant calculation amount of segmentation. Furthermore, the segmentation map corresponding to the target image will be displayed on the client interface, and the segmentation map includes multiple optional areas segmented out, and the user can select areas for the multiple optional areas, so that the target image can be cut out according to at least one optional area selected by the user to obtain the corresponding cutout image. The above-mentioned target image can be a commodity image, and the obtained cutout image can be a display image of the target object body separated from the commodity image. This solution provides an interactive cutout solution. By interacting with the user, the user's actual intention can be clarified based on the user's selection, avoiding excessive or missed cutouts, and providing users with better cutout services. Moreover, the user can complete the selection in a relatively simple way (such as clicking) without the need for professional knowledge. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0030] Figure 1 A flowchart of an image processing method provided in one embodiment of the present application;
[0031] Figure 2a A schematic diagram of a cutout provided in accordance with an embodiment of the present application;
[0032] Figure 2b An example diagram showing a cutout image provided by an embodiment of the present application;
[0033] Figure 3 A schematic diagram of the results of the full-image segmentation model provided in one embodiment of the present application;
[0034] Figure 4 and Figure 5 A schematic diagram of the principle of cutting out provided in other embodiments of the present application;
[0035] Figure 6 and Figure 7 A schematic diagram of the flow of image processing methods provided in the other two embodiments of the present application;
[0036] Figures 8 to 10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;
[0037] Fig.11 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0038] Cutting out images mainly extracts the target object that the user needs from an image. It is widely used in image editing, creative synthesis, and other fields. For example, for merchants on e-commerce platforms, whether it is the main image or secondary image of a product, or a creative image of a product used for advertising, it is necessary to extract the main image of the product from the product image through cutting out. At present, automatic cutting out tools are mainly used to achieve cutting out images. However, when there are multiple and complex objects in the image to be cut out, it is inevitable that there will be multiple or missed cuts when using existing automatic cutting out tools, or manual cutting out is required.
[0039] For example, various existing photo beautification tools all provide a brush cutout function, but the brush cutout function requires manual cutout, which places high demands on the user's professionalism.
[0040] For another example, all the existing photo beautification tools include a one-click cutout function; or, with the release and rapid development of technologies such as GPT (Generative Pre-Training, pre-trained language model), SAM (Segment Anything Model, image segmentation model) and image generation, some platforms have also launched automated cutout functions in merchant assistant products. Among them, the image-related functions launched by merchant assistant products include not only automated cutouts, but also other image functions such as making white background images and scene images. The automated cutout function is the core basic capability, which is basically involved in other image function processes. The aforementioned one-click cutout function or automated cutout function, with the help of image understanding, detection, segmentation and other modules, allows users to directly obtain the image cutout results output by the algorithm without interaction costs. However, combined with the actual usage of online users, it is found that although the automated cutout solution can solve the cutout of most input images, there are still some unsolvable cutout difficulties. For example, when the input original image contains multiple object subjects, it is difficult to identify the user's actual intention due to the lack of interactive input, and it is impossible to determine whether the user wants to extract all object subjects in the image or only one of them. In addition, when the scene of the original image is complex, the automatic cutout is relatively subjective as to whether to extract matching items, packaging boxes, icons, logos, etc. in the image. All of the above may cause the final cutout result to have multiple cutouts or missing cutouts, making the cutout result incomprehensible and inconsistent with the user's actual intention.
[0041] In order to solve the problems existing in the above-mentioned cutout scheme, the present application provides an image processing technology solution, in which the image is mainly divided into full-image segments, and then the divided areas are numbered and provided to the front end (client) for display for the user to select areas, and then the areas selected by the user are merged and post-processed to realize the cutout function. Among them, in the full-image segmentation process, the quadtree algorithm is combined with the full-image segmentation model to make the full-image area division of the image more reasonable and precise. The quadtree algorithm is mainly used to sample the image, and the obtained sampling points are used as prompt points to guide the detection of the object content in the image. The quadtree algorithm is described in detail below and will not be described in detail here.
[0042] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0043] In some processes described in the specification, claims and the above-mentioned figures of this application, multiple operations appearing in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to different types. The term "or / and" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example: A or / and B, indicating that A can exist alone, A and B can exist at the same time, and B can exist alone; the character " / " in this application generally indicates that the front and back associated objects are an "or" relationship. It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such product or system. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the product or system including the elements. In addition, the following embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.
[0044] The technical solutions provided by each embodiment of the present application are introduced and explained below.
[0045] The execution subject of each method embodiment provided in the following application may be an electronic device with a logic operation function, and the electronic device may be a server. The server may be a single server, a service cluster composed of multiple servers, a cloud server or a virtual server, etc., and the embodiments of the application do not specifically limit this.
[0046] Figure 1 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 1 As shown, the image processing method comprises the following steps:
[0047] 101. Based on the image content of the target image, perform point sampling on the target image to obtain sampling point information;
[0048] 102. Perform full image segmentation on the target image based on the sampling point information;
[0049] 103. Displaying a segmentation map corresponding to the target image on a client interface, wherein the segmentation map includes a plurality of segmented optional areas;
[0050] 104. In response to the area selection operation triggered for the multiple optional areas, the target image is cut out according to at least one selected optional area to obtain a cutout image.
[0051] In the above 101, the target image may be, but is not limited to, a product image, which may be input by a user. In specific implementation, the target image may be selected by a user from a gallery, which may be stored locally by the execution subject described in this embodiment (such as an album), or stored on other devices on the network side, which is not limited in this embodiment.
[0052] For example, see Figure 2a As shown, the merchant opens the merchant business assistant page provided by the e-commerce platform through the client 1, and clicks the "Smart Cutout" control on the merchant business assistant page. At this time, the smart cutout page 10 will be displayed on the client interface. The user can operate the corresponding image selection control on the smart cutout page 10 to input the target image to be cut out, such as operating the "Select image from the network side" control or the "Select local image" control to input the target image to be cut out. The input target image (such as product image P) can be displayed in area Z and can also be obtained by the server 2 for subsequent processing.
[0053] From the above content, that is, before executing the above step 101, the method provided in this embodiment further includes:
[0054] 100. Obtain the target image to be cut out.
[0055] In a specific implementation, the above step 100 of "obtaining the target image to be cut out" includes:
[0056] 1001. Displaying a cutout service page (such as smart cutout page 10) on the client;
[0057] 1002. In response to an input operation of the user through the cutout service page, the target image input by the user is obtained.
[0058] The client described in this embodiment may be, but is not limited to, a smart phone, a tablet, a laptop computer, a desktop computer, etc.
[0059] In addition to obtaining the target image, considering that the target object may have multiple object bodies and the scene is complex, in order to enable the automated cutout to better meet the actual needs of users, this embodiment will first perform full-image segmentation on the target image to predict and segment all possible objects in the target image for interaction with the user, thereby clarifying the user's intention through interaction and providing the user with better cutout services.
[0060] At present, the implementation of full-image segmentation is basically achieved with the help of corresponding full-image segmentation models, such as the SAM model. The SAM model can receive inputs such as text, coordinate points, coordinate boxes, etc. as prompts (instructions), and then identify and segment every possible object element in the image based on the auxiliary information provided by the prompt (which is a fuzzy prompt). The existing SAM model uses the sampling points obtained by uniform sampling as prompts (instructions) when implementing automatic full-image segmentation. Uniform sampling refers to dividing an image into M*N networks with equal spacing in the horizontal and vertical directions, and using the center point of each network as the sampling point (such as Figure 4 256 points shown in ). Since the uniform sampling method does not combine the characteristics of the image itself, there are often problems such as insufficient sampling of complex images, which leads to insufficient extraction of fine structures in the image during the final segmentation and unreasonable image area division. In addition, the uniform sampling method will also introduce more unnecessary sampling points during sampling, resulting in a large amount of redundant calculations during subsequent segmentation. In order to improve the rationality of image area division during full image segmentation and reduce redundant calculations, the solution provided in this embodiment is to optimize sampling in combination with the image content of the image itself (such as color information (also pixel value information)), so as to use the sampling point set obtained by sampling as the prompt of the full image segmentation model such as the SAM model.
[0061] Therefore, in a specific achievable solution, the image content of the target image includes color information; and the above 101 "combining the image content of the target image, performing point sampling on the target image to obtain sampling point information" can be specifically: based on the color information of the target image, performing point sampling on the target image to obtain sampling point information. In a achievable technical solution, the quadtree algorithm can be used but not limited to implement the point sampling of the target image based on the color information of the target image. Therefore, a specific implementation solution of step 101 may include:
[0062] 10111. Recursively divide the target image based on the color information of the target image using a quadtree algorithm, so as to determine the center point of each sub-region obtained by the division as a sampling point.
[0063] The above quadtree algorithm mainly recursively divides the image area according to the color information of the image until a certain division termination condition is met. In specific implementation, the specific implementation process of using the quadtree algorithm to divide and sample the target image can be as follows:
[0064] 1) The entire target image is considered as a single rectangular area, which is used as the root node N0 of the quadtree.
[0065] 2) Divide the rectangular area of the root node N0 into four equal sub-areas in the horizontal and vertical directions, forming four child nodes N1, N2, N3 and N4; if the area represented by a child node among the child nodes N1, N2, N3 and N4 does not meet the set division termination condition, the area can be further divided into four equal sub-areas in the horizontal and vertical directions. The image area hierarchical division is performed in this way.
[0066] When performing hierarchical division of image regions, the division termination conditions set may include:
[0067] Homogeneity condition: when the color richness of the current area is lower than a preset first threshold (eg, the pixel values of all pixels in the area are substantially the same), no further division is performed.
[0068] Tree depth condition: when the depth of the divided tree is greater than a preset second threshold, the division will not continue.
[0069] Region area threshold condition: when the area of a sub-region is smaller than the preset third threshold, no further division is performed.
[0070] 3) The center point of each divided sub-area is determined as the sampling point.
[0071] This embodiment uses a quadtree algorithm to divide the target image into regions for sampling. The divided regions can be adaptively changed according to the complexity of the color information of the input image, and the set division termination condition can avoid over-fine division. After the division is completed, the center point of each divided sub-region can be used as a sampling point. Since regions with similar colors may belong to the same object, the sampling point information obtained based on the quadtree algorithm is helpful in guiding the subsequent full-image segmentation of the target image. The sampling point information includes each sampling point (such as Figure 4 The coordinate information of the 157 points shown in the figure is used to guide the full-image segmentation model to understand and parse different parts of the image and detect and explore all possible foreground objects in the image.
[0072] Compared with the uniform sampling method, this embodiment samples the target image using a quadtree algorithm, so that the sampling point density can be adaptively changed according to the image information (such as color information) of the target image. For example, the sampling density can be reduced in areas with similar colors, which is conducive to reducing the amount of subsequent redundant calculations and reducing the time consumption of the cutout function service. In addition, more refined division can be achieved for complex areas with rich colors in the image to increase the sampling point density.
[0073] Accordingly, the above step 102 of "segmenting the target image in its entirety based on the sampling point information" may include the following steps:
[0074] 1021. Input the sampling point information and the target image into a full-image segmentation model, execute the full-image segmentation model to output a plurality of mask images; the mask images are used to indicate the area where the corresponding object is located in the target image;
[0075] 1022. Perform full-image segmentation on the target image according to the multiple mask images to divide the target image into the multiple optional areas.
[0076] In the above 1021, the preset full-image segmentation model may be, but is not limited to, a SAM model. Inputting the sampling point information (segmentation prompt information) into the full-image segmentation model can be used to guide the model to complete the corresponding segmentation task.
[0077] For example, see Figure 3 The structural diagram of the full-image segmentation model shown in the figure includes a first encoder, a second encoder and a mask decoder (Mask Encode). Among them, the first encoder is an image encoder (ImageEncode) and the second encoder is a prompt encoder (Prompt Encode). After the target image and the segmentation prompt information are input into the full-image segmentation model, the first encoder will encode the target image to generate a first feature vector (that is, an image embedding vector used to describe the characteristics of the image), and the second encoder will encode the segmentation prompt information to generate a second feature vector (that is, a prompt embedding vector used to describe the location characteristics of the sampling point). The mask decoder will analyze and predict all possible objects in the target image based on the first feature vector and the second feature vector, thereby outputting multiple corresponding mask images. The mask image is a binary image whose size matches the size of the target image and is used to indicate the area where the corresponding object is located in the target image.
[0078] Taking into account that the full-image segmentation model often has repeated detections in the process of detecting objects based on segmentation prompt information and target images, some of the predicted masks are repeated, which are redundant masks. In response to this problem, the full-image segmentation model described in this embodiment will perform post-processing on the predicted mask map to remove duplicates, wherein the post-processing method used for deduplication can be the NMS (Non-Maximum Suppression) algorithm, whose main function is to eliminate redundant masks and retain better masks that may indicate objects. For a detailed description of deduplication using the NMS algorithm, please refer to the existing relevant content. Finally, the full-image segmentation model will output multiple mask maps obtained after deduplication processing.
[0079] In the above 1022, when the target image is segmented according to the multiple mask images, the target image is divided into multiple optional areas, and different optional areas can be marked with different colors (such as Figure 2a The image P' shown in FIG. 1 , and different optional regions represent different objects or parts in the target image.
[0080] In specific implementation, during the division, the target image can be further divided according to the connectivity of the non-mask area in the target image. This is done for the convenience of subsequent display and interactive selection on the one hand, and on the other hand, since the unconnected area in the non-mask area may not belong to the background, it is better to make a finer area division based on the connectivity of the non-mask area to avoid missed divisions.
[0081] That is, a specific implementation scheme of the above 1022 “segmenting the entire target image according to the multiple mask images to divide the target image into the multiple optional areas” may include:
[0082] 10221. Determine a non-mask area of the target image according to the multiple mask images;
[0083] 10222. Divide the target image into multiple optional areas according to the multiple mask images and connectivity information of the non-mask areas.
[0084] In addition, for the multiple optional areas divided, each optional area can also be numbered according to the area size of the mask image of each optional area. In specific implementation, the mask images of each optional area can be sorted in descending order first, and then the optional areas can be numbered from 0 or 1 according to the sorting result. Numbering each area based on the sorting result obtained by using the descending sorting method will make the area number of the optional area smaller as the area of the mask image of the optional area is larger. Of course, in addition to the sorting method described above, other methods can also be used to sort the optional areas, for example: the mask images of each optional area can also be sorted in ascending order, and then the optional areas can be numbered from 0 or 1 according to the sorting result, and this embodiment does not impose specific restrictions on this. Among them, descending sorting is to arrange the optional areas in order from large to small in area, and ascending sorting is to arrange the optional areas in order from small to large in area.
[0085] The area of the mask image is the area of the object region in the mask image. When numbering, if there are at least two overlapping optional regions, the overlapping portion can be marked according to the largest region number among the region numbers of the at least two optional regions. Afterwards, the region numbers of the optional regions can be sent to the client, so that the client can associate the region numbers of the optional regions with the corresponding optional regions when displaying the target image after the full image segmentation, so that the user can clearly know which are the selectable regions.
[0086] Therefore, the method provided in this embodiment may also include the following steps:
[0087] 105. Determine area numbers of the respective optional areas according to areas of the mask images corresponding to the respective optional areas, so as to associate the respective optional areas with the area numbers of the respective optional areas and display them on a client interface.
[0088] When at least two of the multiple optional areas partially overlap, the largest area number among the area numbers of the at least two optional areas is determined as the area number corresponding to the overlapping area of the at least two optional areas.
[0089] For example, the area number may be displayed at a corner of the corresponding optional area, etc., which is not specifically limited here.
[0090] Furthermore, by executing the above step 103, the segmentation map corresponding to the target image can be displayed on the client interface, such as displayed on Figure 2aIn the selection page 11 shown in FIG. 1 , when the segmentation map P" corresponding to the target image is displayed in the selection page, the corresponding area number ( Figure 2a not fully shown).
[0091] The user can select multiple optional areas included in the segmentation map corresponding to the displayed target image by using the interactive method provided by the client (such as mouse, voice, touch, etc.).
[0092] For example, see Figure 2a , if the user clicks on the optional area with area number ① once, the optional area will be highlighted (such as changing the color of this area to blue), indicating that this optional area is in a selected state. Later, when the user clicks on the area with area number ① again, the selection of the optional area can be canceled. Correspondingly, the optional area will restore its original color, indicating that this optional area is in an unselected state. Similarly, the user can also click or cancel the selection of other optional areas. After the user completes the selection of at least one optional area according to his or her own cutout requirements, the user can click the "Confirm Selection" control. In response to the user's operation on the "Confirm Selection" control, the server will trigger the execution of "cutout the target image according to at least one selected optional area" in the above step 104. In addition, the user can also click the "Cancel" control to cancel all selected areas with one click.
[0093] In addition to selecting the selectable area by clicking, the user can also select the area by other interactive methods. For example, if the selection page 11 provides interactive methods such as text or voice, the user can also enter the corresponding area number information to select the area. This embodiment does not specifically limit the specific implementation method of the selection.
[0094] Based on this example, in a specific implementable solution, the method described in this embodiment may further include the following steps:
[0095] In response to the point selection operation, the at least one optional area in a selected state is the at least one optional area selected; or
[0096] In response to the area number input operation, the optional area corresponding to the input area number is the selected optional area.
[0097] From the above content, this embodiment divides the whole image and displays the optional areas divided by the whole image, so that users can select areas through corresponding interactive methods (such as clicking, inputting area numbers, etc.), so that the user's actual intention can be clarified according to the optional areas selected by the user, and the problems of over-cutting or missing cuts caused by unclear cutout targets can be effectively solved, and more flexible and simple automated cutout services can be provided to users, thereby optimizing the user's interactive cutout service experience.
[0098] In the above 104, the mask images corresponding to the at least one selected area may be merged, and the merged mask image obtained by erosion and dilation, edge optimization, etc. may be performed to cut out the target image. That is, in a specific achievable technical solution, in the above 104, "cut out the target image according to the at least one selected optional area to obtain a cutout image" may include:
[0099] 1031. Merge the mask images corresponding to the at least one selected region to obtain a merged mask image;
[0100] 1032. Process the merged mask image to obtain a target mask image; the processing includes corrosion expansion and edge optimization;
[0101] 1033. Based on the target mask image, perform cutout on the target image to obtain a cutout image.
[0102] In the above, the purpose of performing corrosion and dilation processing on the obtained merged mask image is to construct a three-color map (trimap, also called a three-part map).
[0103] For example, the white area in the merged mask image represents the target area to be extracted, and the pixel value in the white area is 255. The merged mask image is eroded to obtain an erosion map, wherein the pixel values of the areas in the erosion map where the pixel values are greater than 127 are defined as 255, and this area is definitely determined to be the foreground area. The merged mask image is expanded to obtain an expansion map, wherein the pixel values of the areas in the expansion map where the pixel values are greater than 127 and do not overlap with the aforementioned foreground area are defined as 127 (gray, indicating that this area is a pending area), and the pixel values of the areas where the pixel values are less than or equal to 127 are defined as 0 (black, indicating that this area is the background area). A Trimap image can be generated based on the erosion map and expansion map obtained above, such as Figure 5 The trimap diagram shown in (b) uses three numerical values to identify the following three types of areas: definitely foreground (white), definitely background (black), and uncertain (gray).
[0104] Furthermore, the above-mentioned trimap image and the target image are input into a corresponding matting model such as a matting model, and the corresponding matting result (including the matting image) is outputted when the matting model is executed. Figure 5 As shown in (b) in the figure, after the trimap image and the target image are input into the corresponding cutout model, the cutout model will perform edge optimization processing on the trimap image to obtain the target mask image; then, the target area is separated from the target image according to the target mask image, and the cutout result is output (as shown in FIG. Figure 5 In addition, the target mask image can be output. That is, the cutout model provided in this embodiment can be used to achieve edge optimization and cutout, and the algorithm used for edge optimization can be, but is not limited to, an edge sharpening algorithm based on a specific grayscale distribution, an optimization algorithm based on line and surface features, etc., which are not specifically limited here. Among them, the edge optimization of the trimap image by the cutout model is reflected in the prediction of a value between 0 and 255 for the edge area, rather than direct binarization.
[0105] Figure 5 (a) in FIG. 1 shows the corresponding matting result of matting the target image directly according to the obtained merged mask image, and the matting result is as follows: Figure 5 From the cutout image shown in (a), it can be clearly seen that the edges of the aforementioned merged mask image and the corresponding cutout image are relatively defective, such as jagged edges.
[0106] From the above, the quality of the cutout can be effectively improved through various post-processing such as corrosion expansion, edge optimization, etc.
[0107] The cutout image obtained above can be used in subsequent processes such as white background image, scene image generation, and image editing.
[0108] Furthermore, the method provided in this embodiment may also include the following steps:
[0109] 106. Display the cutout image on the client interface.
[0110] In specific implementation, the cutout image and the original target image can be displayed on the same client interface (such as Figure 2b As shown); or, the cutout image can be displayed separately on a page. The purpose of displaying the cutout image is to facilitate the user to review and confirm whether to proceed with the subsequent process, such as uploading the cutout image as a white background image to the corresponding publishing page, or downloading it locally, or editing the cutout image to generate a corresponding scene graph and publish it, etc.
[0111] In summary, the technical solution provided by this embodiment is to segment the target image in its entirety based on the image content of the target image (such as the color information of the target image) using a preset full-image segmentation model. This full-image segmentation method enables segmentation to be adaptively changed according to the image content information of the target image, improves the full-image area division effect of the target image, and balances the function time consumption. For example, the sampling density can be increased for the color-rich and complex areas in the target image to achieve a more refined segmentation, while the sampling density can be reduced for the areas with similar colors to reduce the redundant calculation amount of segmentation. Furthermore, the target object after full-image segmentation will be displayed on the client interface. The target image after full-image segmentation is divided into multiple selectable areas, and the user can select areas for the multiple areas, so that the target image can be cut out according to at least one area selected by the user to obtain a corresponding cutout interface. This solution provides an interactive cutout solution, which can clarify the user's actual intention according to the user's selection through interaction with the user, avoid multiple cutouts or missed cutouts, and provide users with better cutout services; moreover, the user can complete the selection in a relatively simple way (such as clicking) without professional knowledge.
[0112] This application also provides several other image processing method embodiments. The details are as follows:
[0113] Figure 6 FIG. 2 shows an image processing method provided by another embodiment of the present application. Figure 6 , the image processing method comprises the following steps:
[0114] 201. Acquire segmentation prompt information of the target image according to color information of the target image;
[0115] 202. Using the segmentation prompt information and the target image, triggering a preset full-image segmentation model to perform full-image segmentation on the target image;
[0116] 203. Displaying a segmentation map corresponding to the target image on a client interface, wherein the segmentation map includes a plurality of segmented optional areas;
[0117] 204. In response to the area selection operation triggered for the multiple optional areas, perform a cutout process on the target image according to at least one selected optional area to obtain a cutout image;
[0118] 205. Display the cutout image on the client interface.
[0119] In the above 201, the segmentation prompt information is also the sampling point information described in other embodiments of the present application.
[0120] In an achievable technical solution, the above 201 “obtaining segmentation hint information of the target image based on color information of the target image” may include:
[0121] 2011. Recursively divide the target image based on the color information of the target image by using a quadtree algorithm, so as to determine the center point of each sub-region obtained by the division as a sampling point;
[0122] The segmentation prompt information is the determined sampling point information.
[0123] For the specific implementation description of the above step 2011 and the specific implementation description of the above steps 202 to 204, please refer to the relevant content in other embodiments.
[0124] In the above 205, the cutout image and the original target image can be displayed on the same client interface (eg Figure 2b The purpose of displaying the cutout image is to facilitate the user to review and confirm whether to proceed with the subsequent process, such as uploading the cutout image as a white background image to the corresponding publishing page, or downloading it locally, or editing the cutout image to generate a corresponding scene graph and publish it, etc.
[0125] In addition to the above steps, the method provided in the embodiment of the present application may also include other steps. For other steps that may be included and specific implementation descriptions, please refer to the relevant content in other embodiments and will not be repeated here.
[0126] Figure 7 FIG. 2 is a flow chart of an image processing method provided by another embodiment of the present application. Figure 7 As shown, the image processing method includes:
[0127] 301. Obtain the product image to be cut out;
[0128] 302. Perform point sampling on the product image in combination with the image content of the product image to obtain sampling point information;
[0129] 303. Segment the entire product image based on the sampling point information;
[0130] 304. Displaying a segmentation map corresponding to the product map on a client interface, wherein the segmentation map includes a plurality of segmented object areas;
[0131] 305. In response to the area selection operation triggered on the multiple object areas, separate the target object body from the product image according to the at least one selected object area to obtain a display image of the target object body;
[0132] 306. Display a display diagram of the target object body.
[0133] For the specific implementation description of the above steps 301 to 306, please refer to the relevant content in other embodiments. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other steps. For the other steps that may be included and the specific implementation description, please refer to the relevant content in other embodiments, which will not be repeated here.
[0134] Furthermore, the present application also provides embodiments of image processing systems and image processing devices corresponding to the above-mentioned method embodiments. Specifically, they are as follows:
[0135] See Figure 2a As shown, the image processing system provided in the embodiment of the present application includes: a client 1 and a server 2. Among them,
[0136] In one example, the functions of client 1 and server 2 are as follows:
[0137] Client 1 is used to obtain a target image to be cut out, and send the target image to a server; receive and display a segmentation map of the target image fed back by the server, wherein the segmentation map includes a plurality of segmented optional areas; respond to a selection operation triggered for the plurality of optional areas, cut out the target image according to at least one selected optional area, and obtain and display the cutout image;
[0138] Server 2 is used to perform point sampling on the target image in combination with the image content of the target image to obtain sampling point information; based on the sampling point information, perform full-image segmentation on the target image to obtain a segmentation map; or, server 2 is used to use the segmentation prompt information and the target image to trigger a preset full-image segmentation model to perform full-image segmentation on the target image to obtain a segmentation map.
[0139] The above-mentioned “cutting out the target image according to the selected at least one optional area to obtain a cut-out image” can also be executed by the server. The client sends the selected at least one optional area to the server. The server cuts out the target image according to the selected at least one area to obtain a cut-out image, and sends the cut-out image to the client 2.
[0140] In another example, the functions of the client and the server are as follows:
[0141] Client 1 is used to obtain a product image to be cut out and send the product image to a server; receive and display a segmentation image fed back by the server, wherein the segmentation image includes multiple segmented object areas; respond to a selection operation triggered for the multiple object areas, separate a target object body from the product image according to at least one selected object area, and obtain a display image of the target object body; and display the display image of the target object body.
[0142] The server 2 is used to perform point sampling on the product in combination with the image content of the product image to obtain sampling point information, and to perform full image segmentation on the product image based on the sampling point information to obtain a segmentation image corresponding to the product image.
[0143] Similarly, the above-mentioned "separating the target object body from the product image" can be executed by the server. The client sends the selected at least one object area to the server, and the server separates the target object body from the product image based on the selected at least one object area to obtain a display image of the target object body; and feeds the display image back to the client.
[0144] For detailed description of the functions of each terminal in the system provided in this embodiment, please refer to the relevant content in other embodiments.
[0145] Figure 8 FIG. 1 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present application. Figure 8 The image processing device includes: a sampling and segmentation module 41, a display module 42 and a cutout module 43.
[0146] The sampling and segmentation module 41 is used to perform point sampling on the target image in combination with the image content of the target image to obtain sampling point information, and perform full image segmentation on the target image based on the sampling point information;
[0147] A display module 42, used to display the segmentation map corresponding to the target image on the client interface; wherein the segmentation map includes a plurality of segmented optional areas;
[0148] The cutout module 43 is configured to, in response to a selection operation triggered by a user on the multiple optional areas, cutout the target image according to at least one selected optional area to obtain a cutout image.
[0149] Optionally, the above-mentioned sampling segmentation module 41, when used to perform full-image segmentation on the target image based on the sampling point information, can be specifically used to: take the sampling point information and the target image as input, execute a full-image segmentation model to output multiple mask images; the mask images are used to represent the areas where corresponding objects are located in the target image; and perform full-image segmentation on the target image according to the multiple mask images to divide the target image into the multiple optional areas.
[0150] Optionally, the image content includes color information. And, the sampling and segmentation module 41, when used to perform point sampling on the target image in combination with the image content of the target image to obtain sampling point information, can be specifically used to: recursively divide the target image based on the color information of the target image using a quadtree algorithm to obtain multiple sub-regions; and determine the center point of each of the multiple sub-regions as a sampling point.
[0151] Optionally, the above-mentioned sampling segmentation module 41, when used to perform full-image segmentation on the target image according to the multiple mask images to divide the target image into the multiple optional areas, can be specifically used to: determine the non-mask area of the target image according to the multiple mask images; divide the target image into multiple optional areas according to the connectivity information of the multiple mask images and the non-mask areas.
[0152] Optionally, the device also includes: a determination module, used to determine the area number of the optional area based on the area of the mask image corresponding to the optional area, so as to associate the optional area and the area number of the optional area with each other and display them on the client interface; when there is partial overlap among multiple optional areas, the largest area number among the area numbers is determined as the area number corresponding to the overlapping area.
[0153] Optionally, the above-mentioned cutout module 43, when used to determine at least one selected optional area in response to a selection operation triggered by a user for the multiple optional areas, can be specifically used to: in response to a point selection operation, determine at least one optional area in a selected state as at least one selected optional area; or, in response to an area number input operation, determine at least one selected optional area according to the input area number information.
[0154] Optionally, the above-mentioned cutout module 43, when used to cutout the target image according to at least one selected optional area to obtain a cutout image, can be specifically used to: merge the mask images corresponding to the at least one selected optional area to obtain a merged mask image; process the merged mask image to obtain a target mask image; wherein the processing includes corrosion expansion and edge optimization; based on the target mask image, cutout the target image to obtain a cutout image.
[0155] It should be noted here that: the image processing device provided in the above embodiment can implement the technical solution described in the above corresponding method embodiment. The specific implementation principles of the above modules or units can refer to the relevant contents in the above corresponding method embodiment, and will not be described in detail here.
[0156] Fig. 9 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by another embodiment of the present application. Fig. 9 The image processing device includes: an acquisition module 51, a segmentation module 52, a display module 53, a cutout module 54 and a display module 55.
[0157] An acquisition module 51 is used to acquire segmentation prompt information of the target image according to color information of the target image;
[0158] A segmentation module 52, used to trigger a preset full-image segmentation model to perform full-image segmentation on the target image using the segmentation prompt information and the target image;
[0159] A display module 53 is used to display the segmentation map corresponding to the target image on the client interface; wherein the segmentation map includes a plurality of segmented optional areas;
[0160] A cutout module 54 is configured to, in response to a selection operation triggered by a user on the plurality of selectable areas, perform a cutout process on the target image according to at least one selected selectable area to obtain a cutout image;
[0161] The display module 55 is used to display the cutout image on the client interface.
[0162] It should be noted here that: the image processing device provided in the above embodiment can implement the technical solution described in the above corresponding method embodiment. The specific implementation principles of the above modules or units can refer to the relevant contents in the above corresponding method embodiment, and will not be described in detail here.
[0163] Fig.10 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by another embodiment of the present application. Fig.10 The image processing device includes: an acquisition module 61, a segmentation and display module 62, a separation module 63 and a display module 64.
[0164] An acquisition segmentation module 61 is used to acquire a product image to be cut out;
[0165] The sampling, segmentation and display module 62 is used to perform point sampling on the product image based on the image content of the product image to obtain sampling point information; perform full image segmentation on the product image based on the sampling point information; and display the segmentation map corresponding to the product image on the client interface; wherein the segmentation map includes a plurality of segmented object areas (which are optional object areas);
[0166] A separation module 63 is configured to, in response to a selection operation triggered by a user on the plurality of object regions, separate a target object body from the commodity image according to at least one selected object region, and obtain a display image of the target object body;
[0167] The display module 64 is used to display the display image of the target object body on the client interface so that the user can review and confirm whether it is displayed in the corresponding product page.
[0168] It should be noted here that: the image processing device provided in the above embodiment can implement the technical solution described in the above corresponding method embodiment. The specific implementation principles of the above modules or units can refer to the relevant contents in the above corresponding method embodiment, and will not be described in detail here.
[0169] Fig.11 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Fig.11 As shown, the electronic device includes: a memory 71 and a processor 72. The memory 71 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Specifically,
[0170] The memory 71 is used to store programs;
[0171] The processor 72 is coupled to the memory 71 and is used to execute the program stored in the memory for the steps or functions in the methods provided in the embodiments of the present application.
[0172] Further, if Fig.11 The electronic device also includes: a communication component 73, a power component 74, an audio component 75 and other components. Fig.11 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.11 The electronic device may be a server.
[0173] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the method steps provided in the above embodiments can be implemented.
[0174] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the processor is enabled to implement the method steps or functions provided in the above embodiments.
[0175] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An image processing method, characterized in that: include: In combination with the image content of the target image, point sampling is performed on the target image to obtain sampling point information; Based on the sampling point information, the target image is segmented in its entirety; Displaying a segmentation map corresponding to the target image on a client interface, wherein the segmentation map includes a plurality of segmented optional areas; In response to the area selection operation triggered for the multiple optional areas, the target image is cut out according to at least one selected optional area to obtain a cutout image.
2. The method according to claim 1, characterized in that Based on the sampling point information, the target image is segmented in its entirety, including: Taking the sampling point information and the target image as input, executing the full image segmentation model to output a plurality of mask images; the mask images are used to represent the area where the corresponding object is located in the target image; The target image is segmented in its entirety according to the multiple mask images, so as to divide the target image into the multiple optional areas.
3. The method according to claim 2, characterized in that The image content includes color information; and combining the image content of the target image, performing point sampling on the target image to obtain sampling point information, including: Using a quadtree algorithm, recursively divide the target image based on color information of the target image to obtain a plurality of sub-regions; The center point of each sub-region in the multiple sub-regions is determined as a sampling point.
4. The method according to claim 2, characterized in that: Performing full image segmentation on the target image according to the multiple mask images to divide the target image into the multiple optional areas includes: Determining a non-mask area of the target image according to the multiple mask images; The target image is divided into the multiple optional areas according to the multiple mask images and the connectivity information of the non-mask areas.
5. The method according to any one of claims 2 to 4, characterized in that Also includes: Determine the area number of the optional area according to the area of the mask image corresponding to the optional area, so as to associate the optional area with the area number of the optional area and display it on the client interface; When there is partial overlap among multiple optional areas, the largest area number among the area numbers is determined as the area number corresponding to the overlapping area.
6. The method according to claim 5, characterized in that Also includes: In response to the click operation, the at least one optional area in a selected state is the at least one optional area selected; or In response to the area number input operation, the optional area corresponding to the input area number is the selected optional area.
7. The method according to any one of claims 2 to 4, characterized in that Cutting out the target image according to the selected at least one optional area to obtain a cutout image, comprising: Merging the mask images corresponding to the at least one selected area to obtain a merged mask image; Processing the merged mask image to obtain a target mask image; wherein the processing includes corrosion expansion and edge optimization; Based on the target mask image, the target image is cut out to obtain a cutout image.
8. An image processing method, characterized in that: include: Acquiring segmentation prompt information of the target image according to color information of the target image; Using the segmentation prompt information and the target image, triggering a preset full-image segmentation model to perform full-image segmentation on the target image; On the client interface, a segmentation map corresponding to the target image is displayed, wherein the segmentation map includes a plurality of segmented optional areas; In response to the area selection operation triggered for the multiple optional areas, the target image is cut out according to at least one selected optional area to obtain a cutout image; The cutout image is displayed on the client interface.
9. An image processing method, characterized in that: include: Based on the image content of the product image, point sampling is performed on the product image to obtain sampling point information; Based on the sampling point information, the product image is segmented into its entirety; On the client interface, a segmentation map corresponding to the product image is displayed, wherein the segmentation map includes a plurality of segmented object areas; In response to the area selection operation triggered for the multiple object areas, a target object body is separated from the commodity image according to at least one selected object area to obtain a display image of the target object body.
10. An image processing system, characterized in that: include: The client is used to obtain the target image to be cut out; The server is used to implement the method described in any one of claims 1 to 9.
11. An electronic device, characterized in that: include: A memory and a processor; wherein, The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the image processing method described in any one of claims 1 to 7, or to implement the steps in the image processing method described in claim 8, or to implement the steps in the image processing method described in claim 9.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instructions; when the computer program or instructions are executed by a processor, it can implement the steps in the image processing method described in any one of claims 1 to 7, or implement the steps in the image processing method described in claim 8, or implement the steps in the image processing method described in claim 9.
13. A computer program product, characterized in that The computer program product includes a computer program or instructions, which, when executed by a processor, enables the processor to perform the steps in the image processing method described in any one of claims 1 to 7, or the steps in the image processing method described in claim 8, or the steps in the image processing method described in claim 9.
Citation Information
Patent Citations
Adaptive downsampling depth map compression method based on visual saliency
CN108200430A
Interactive matting system, method and device based on superpixel segmentation
CN110969629A
Image processing method and device, electronic equipment and storage medium
CN117132608A
Cited By
Ultra-high-resolution original picture image layered segmentation method and device, medium and product
CN119810122A
A method, device, medium and product for layering and segmenting a high-resolution original picture image
CN119810122B
Image processing method, system, electronic device, storage medium, and program product
WO2026016679A1
