Image processing method and device, equipment, medium and product

Abstract

The invention provides an image processing method and device, equipment, a medium and a product, and the method comprises the steps: obtaining initial mask data of a target object in response to a first interaction operation of the target object in a to-be-matted original image, compressing the initial mask data, and obtaining compressed mask data corresponding to the target object; the pixel values of the pixel points on the edge contour line and the pixel points on the non-edge contour line of the target object in the compressed mask data are distinguished and marked, and the matting result corresponding to the target object in the original image is generated based on the compressed mask data, so that the matting efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, medium and product. Background Technology

[0002] Interactive image cutout is an advanced image processing technique that uses computer vision and machine learning algorithms to enable users to interactively cut out desired objects, such as people, objects, or backgrounds, from complex images, thereby achieving fast and accurate cutout of target objects in images.

[0003] Currently, image processing tools are commonly used for image cutout. However, these tools are complex to use, making the cutout process time-consuming and thus affecting efficiency. Summary of the Invention

[0004] Based on the above-mentioned technological status, this application proposes an image processing method, apparatus, device, medium, and product that can reduce the time spent on image matting and improve the efficiency of image matting.

[0005] To achieve the above-mentioned technical objectives, this application proposes the following technical solution:

[0006] The first aspect of this application proposes an image processing method, comprising: in response to a first interactive operation on a target object in an original image to be matted, acquiring initial mask data of the target object; compressing the initial mask data to obtain compressed mask data corresponding to the target object, wherein the pixel values ​​of pixels on the edge contour line and pixels on non-edge contour lines of the target object are distinguished and marked in the compressed mask data; and generating a matting result corresponding to the target object in the original image based on the compressed mask data.

[0007] In some implementations, in response to a loading operation on the original image, the original mask data generated by the image segmentation model running on the server for the original image is obtained, wherein the mask values ​​of the original mask data corresponding to each pixel in the original image are within a first numerical range; an ONNX model running on the web is used to determine the initial mask data corresponding to the target object based on the original mask data and the position coordinates corresponding to the first interaction operation, wherein the initial mask data includes the mask values ​​corresponding to each pixel in the original image, the mask values ​​corresponding to the pixels of the target object are within the first numerical range, and the mask values ​​corresponding to the pixels outside the target object are within a second numerical range.

[0008] In some implementations, the step of compressing the initial mask to obtain compressed mask data corresponding to the target object includes: counting the number of consecutive mask values ​​within a first numerical range and the number of consecutive mask values ​​within a second numerical range in the initial mask data, and then compressing the initial mask data to obtain compressed mask data.

[0009] In some implementations, the compressed mask data is in image format. Generating the cutout result corresponding to the target object in the original image based on the compressed mask data includes: determining the contour edge of the target object based on the compressed mask data, and parsing the position coordinates of pixels located on the edge contour of the target object in the compressed mask data to obtain SVG data corresponding to the target object; highlighting the edge contour of the target object based on the SVG data to obtain the cutout result corresponding to the target object in the original image.

[0010] In some implementations, highlighting the edge contour of the target object based on the SVG data to obtain the cutout result corresponding to the target object in the original image includes: drawing lines based on the SVG data using line drawing instructions to generate multiple line segments corresponding to the edge contour of the target object; connecting the multiple line segments using move instructions to obtain the edge contour corresponding to the target object; and highlighting the edge contour of the target object to obtain the cutout result corresponding to the target object.

[0011] In some implementations, the method further includes: setting the remaining image regions in the original image, excluding the target object, to a color different from the color of the target object; and / or, separating the cutout result corresponding to the target object from the target object and displaying it separately.

[0012] In some implementations, the method further includes: in response to a second interactive operation on the target object in the original image to be cut out, generating a hover effect image of the target object using an ONNX model running on the web, wherein the target object is highlighted in the hover effect image.

[0013] In some implementations, the method further includes: in response to an undo operation, regenerating the matting result based on the saved historical point selection coordinates of the target object and the initial mask data.

[0014] A second aspect of this application provides an image processing apparatus, comprising: an acquisition unit, configured to acquire initial mask data of the target object in response to a first interactive operation on a target object in an original image to be matted; a compression unit, configured to compress the initial mask data to obtain compressed mask data corresponding to the target object, wherein the pixel values ​​of pixels on the edge contour lines and pixels on non-edge contour lines of the target object are distinguished in the compressed mask data; and a generation unit, configured to generate a matting result corresponding to the target object in the original image based on the compressed mask data.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the image processing method described in the first aspect and any of its implementations by running the program in the memory.

[0016] The fourth aspect of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the image processing method described in the first aspect and any of its implementations.

[0017] The fifth aspect of this application provides a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to implement the image processing method described in the first aspect and any of the implementations of the first aspect.

[0018] The image processing method, apparatus, device, medium, and product proposed in this application involve obtaining initial mask data of the target object in response to a first interactive operation on the target object in the original image to be matted, and compressing the initial mask data to obtain compressed mask data corresponding to the target object. In this compressed mask data, the pixel values ​​of pixels on the edge contour lines and pixels on non-edge contour lines of the target object are distinguished and marked. Based on the compressed mask data, a matting result corresponding to the target object in the original image is generated. Because the initial mask data of the target object is compressed, the computational load of redundant information from pixels on non-edge contour lines during the matting process can be reduced, thereby reducing matting time, improving matting efficiency, and achieving real-time matting effects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0021] Figure 2 A schematic diagram illustrating the image cutout effect provided in the embodiments of this application;

[0022] Figure 3 A schematic diagram illustrating the hovering effect provided in an embodiment of this application;

[0023] Figure 4 A schematic diagram of the dotted markers provided in the embodiments of this application;

[0024] Figure 5 A schematic diagram of the image cutout interface provided in an embodiment of this application;

[0025] Figure 6 A framework diagram of a real-time point-and-click image matting scheme provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solution proposed in this application is applicable to any scenario requiring image matting, such as in generative artificial intelligence (AIGC) applications, where a user generates a target image using AIGC and then needs to extract target objects such as people and animals from the image. Another example is when a user downloads an image from the internet and needs to extract target objects such as people and animals from it. In such cases, the technical solution proposed in this application can be used to improve image matting efficiency.

[0029] Images contain a wealth of information and are a primary source of information, widely used in daily life. However, users may sometimes only be interested in a portion of an image, requiring image cutout techniques to extract the part of interest and apply it to meet their specific needs.

[0030] Currently, the most widely used image cutout technique is Photoshop, a powerful image processing tool. However, its cutout process is complex and time-consuming. Furthermore, objects with subtle, complex outlines or semi-transparent effects may require even more time and specialized skills to achieve accurate cutout. This inherently increases the complexity for users, resulting in lower efficiency.

[0031] In view of the above-mentioned technical status, this application proposes a new image processing method. By performing image matting on the original image, the matting time can be reduced, thereby improving the matting efficiency.

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] This application first proposes an image processing method that can be applied to the web; see [link to relevant documentation]. Figure 1 As shown, the method includes steps S101 to S103:

[0034] S101, In response to the first interactive operation of the target object in the original image to be cut out, obtain the initial mask data of the target object.

[0035] The original image is the user-uploaded image to be cut out. The original image contains multiple objects in different locations. The target object can be any one of these objects. For example, if an original image includes people, animals, and plants, the target object can be any one of them, or even a part of a person, animal, or plant, such as a person's face, an animal's head, or a plant's flower.

[0036] The first interactive operation is the action that triggers the image cutout process. This first interactive operation can be a point-and-click operation. For example, when the mouse pointer is detected clicking on a target object in the original image within the cutout interface, this can be identified as the first interactive operation performed on the target object in the original image to be cut out. Here, a point-and-click operation refers to the action of selecting an object by clicking.

[0037] The initial mask data of the target object can be understood as the mask data obtained by adding a mask to the target object. Obtaining the initial mask data of the target object includes steps a1 and a2:

[0038] a1. In response to the loading operation of the original image, obtain the original mask data generated by the image segmentation model running on the server for the original image.

[0039] Image segmentation model (SAM) can directly segment objects within a unified framework by specifying a point and a bounding box, thereby generating the original mask data of the original image.

[0040] The SAM model comprises an image encoder, a cue encoder, and a mask decoder. The image encoder encodes the original image to obtain an image-encoded result. The cue encoder encodes the cue information to obtain a cue-encoded result. The mask decoder outputs the original mask data based on the image-encoded result and the cue-encoded result.

[0041] In this context, the original mask data can be understood as mask data obtained by adding a mask to the original image. The original mask data can accurately indicate specific regions in the original image, such as which pixels belong to a specific object or background.

[0042] Image segmentation models typically run on a server. After receiving the original image, the web client needs to send it to the server so that the server can generate original mask data using the image segmentation model and return the original mask data to the web client. It can be seen that there is a transmission time involved in this process, which affects the matting time and thus the matting efficiency. To reduce matting time and improve efficiency, this embodiment can send the original image to the server as soon as the web client detects the loading operation of the original image. The server then uses the image segmentation model to generate original mask data and returns it to the web client, where it is stored locally. By sending the original image to the server immediately upon receipt to obtain the original mask data, matting time can be reduced and efficiency improved, thus achieving real-time matting.

[0043] Once the original mask data is stored on the web, it can be reused in subsequent image matting operations on that original image without needing to retrieve it from the server frequently, thus reducing the time spent on matting. In other words, for the same original image, step a1 only needs to be performed once, and the original mask data can be reused in subsequent matting operations.

[0044] In this context, the mask value of each pixel in the original image falls within a first numerical range in the original mask data. This first numerical range can be any range of values ​​greater than 0.

[0045] a2. Using the ONNX model running on the web, the initial mask data is determined based on the original mask data and the position coordinates corresponding to the first interactive operation.

[0046] The ONNX model is a lightweight version of the SAM model. By exporting the lightweight mask decoder of the image segmentation model into ONNX format and deploying the ONNX model on the web, the SAM model can be migrated to the web, thereby reducing transmission time between the web client and the server and improving image matting efficiency.

[0047] The first interactive operation includes the position coordinates corresponding to the point selection operation. The initial mask data can be obtained by inferring from the original mask data and the position coordinates corresponding to the point selection operation using the Onnx model.

[0048] The initial mask data includes the mask values ​​corresponding to each pixel in the original image. The mask values ​​corresponding to the pixels of the target object are within a first numerical range, and the mask values ​​corresponding to the pixels outside the target object are within a second numerical range. The second numerical range can be a range of values ​​less than 0.

[0049] For example, the original image includes a person wearing a hat. All elements in the original mask data have values ​​greater than 0. When the coordinates of the selected location fall within the image area containing the hat, new mask data is generated—the initial mask data. In the initial mask data, the mask data corresponding to the hat has values ​​greater than 0, while the mask data corresponding to the image area outside the hat has values ​​less than 0.

[0050] Continue reading Figure 1 After step S101, the image processing method of this embodiment may further include step S102.

[0051] S102. Compress the initial mask data to obtain the compressed mask data corresponding to the target object.

[0052] The initial mask data is quite large. Directly generating the matting result based on the initial mask data would result in too much redundant information during the matting process, leading to a large computational load and affecting matting efficiency. To improve matting efficiency, this embodiment compresses the initial mask data to reduce the computational load, and then generates the matting result based on the compressed mask data, thus improving matting efficiency.

[0053] In some embodiments, step S102 specifically includes: counting the number of consecutive mask values ​​within a first numerical range and the number of consecutive mask values ​​within a second numerical range in the initial mask data, and compressing the initial mask data to obtain compressed mask data.

[0054] Specifically, the initial mask data can be compressed by counting the number of consecutive greater than 0 mask values ​​and consecutive less than 0 mask values ​​in the initial mask data, following a left-to-right and top-to-bottom order. For example, given the initial mask data: [[-0.3, -0.3, -0.1, 1.2, 1.5, 0.4, 0.1, -0.5], [-0.3, -0.3, -0.1, 1.2, 1.5, 0.4, 0.1, -0.5]]. By counting the number of consecutive greater than 0 mask values ​​and consecutive less than 0 mask values ​​in the initial mask data, the compressed mask data can be obtained as: [[3, 4, 1], [3, 4, 1]].

[0055] S103. Based on the compressed mask data, generate the matting result corresponding to the target object in the original image.

[0056] Compressed mask data reduces redundant information compared to the initial mask data. Generating matting results for the target object based on compressed mask data can improve matting efficiency and achieve real-time matting effects.

[0057] Specifically, step S103 includes steps c1 and c2:

[0058] c1. Determine the outline edge of the target object based on the compressed mask data, and parse the position coordinates of the pixels located on the edge outline of the target object in the compressed mask data to obtain the SVG data corresponding to the target object.

[0059] Specifically, the edge contour of the target object is determined based on the mask values ​​greater than 0 and less than 0 in the compressed mask data.

[0060] For example, consider any two rows in the initial mask data: [[-0.3, -0.3, -0.1, 1.2, 1.5, 0.4, 0.1, -0.5], [-0.3, -0.3, -0.1, 1.2, 1.5, 0.4, 0.1, -0.5]]. By counting the number of consecutive mask values ​​greater than 0 and the number of consecutive mask values ​​less than 0 in the initial mask data, the compressed mask data can be obtained as: [[3, 4, 1], [3, 4, 1]]. It can then be determined that the first 3 pixels in these two rows of initial mask data correspond to the mask values ​​for the image region outside the target object, the 4th to 7th pixels correspond to the mask values ​​for the image region inside the target object, and the 8th pixel corresponds to the mask value for the image region outside the target object. Therefore, the 4th and 7th pixels in these two rows can be determined to be pixels on the contour edge of the target object.

[0061] By repeating the above process, the pixel coordinates corresponding to the outline edge of the target object can be obtained.

[0062] c2. Highlight the edge contour of the target object based on the SVG data to obtain the cutout result of the target object in the original image.

[0063] SVG data can be understood as vector image data corresponding to the edge contours of a target object. Based on SVG data, the path data corresponding to the edge contours of the target object can be obtained, which is the path of the cutout edge of the target object corresponding to the initial mask data.

[0064] Specifically, the path data corresponding to the edge contour of the target object is obtained based on the SVG data, including steps d1 to d3:

[0065] d1. Using line drawing commands, lines are drawn based on SVG data to generate multiple line segments corresponding to the edge contour of the target object.

[0066] Specifically, based on the position coordinates of each first pixel value, the adjacent pixels of each first pixel value can be determined, and line drawing instructions can be used to connect the pixels corresponding to each first pixel value and their adjacent pixels to obtain multiple line segments corresponding to the edge contour of the target object.

[0067] The line drawing commands include L-line drawing commands, horizontal line drawing commands, vertical line drawing commands, curve drawing commands, and ellipse drawing commands. In this step, the L-line drawing command can be used to obtain multiple line segments.

[0068] One way to obtain path data is by using the SVG's `path` attribute. For example, setting the `path` attribute as `pathd = "M100, 100L200, 200L200, 400"` means drawing a straight line from (100, 100) to (200, 200) and another straight line from (200, 200) to (200, 400), starting from the coordinates (100, 100). This allows you to obtain multiple line segments corresponding to the edge contour of the target object.

[0069] d2. Connect multiple line segments using the move command to obtain the path data corresponding to the edge contour of the target object.

[0070] d3. Based on the path data corresponding to the edge contour of the target object, highlight the edge contour of the target object to obtain the cutout result corresponding to the target object.

[0071] Highlighting the edge outline of the target object can be achieved by highlighting the edge outline of the target object.

[0072] In some embodiments, the method of this embodiment may further include: setting the remaining image regions in the original image, excluding the target object, to a color different from the color of the target object; and / or, separating the cutout result corresponding to the target object from the target object and displaying it separately.

[0073] This includes separating the cutout result corresponding to the target object from the target object for separate display, including: displaying the cutout result obtained from the original image and the target object separately through a two-column page; or displaying the cutout target object in the background area of ​​the original image.

[0074] Figure 2 This is a schematic diagram illustrating the image cutout effect provided in an embodiment of this application. For example... Figure 2 As shown, when the mouse clicks on the hat, a highlighted edge appears around the hat, and the outer edge of the hat is set to gray. At the same time, the click effect can be seen in real time on the right side of the two-column page.

[0075] In some embodiments, the method of this embodiment may further include: in response to a second interactive operation of a target object in the original image to be cut out, generating a hover effect image of the target object using an ONNX model running on the web, wherein the target object is highlighted in the hover effect image.

[0076] The second interactive operation can be a hover operation. A hover operation refers to an action where the mouse pointer is located at the position of the target object, but no click operation is performed. For example, if it is detected that the mouse pointer is pointing at the position of the target object in the original image of the cutout interface without performing a click operation, and the hovering time at the position of the target object exceeds a preset time, it can be identified as performing a second interactive operation on the target object in the original image to be cut out.

[0077] At this point, the ONNX model running on the web determines the target object corresponding to the second interactive action based on its position coordinates, and highlights the entire target object to obtain the hover effect. This highlighting can be achieved by simply showing the highlighted area.

[0078] Figure 3 This is a schematic diagram illustrating the hovering effect provided in an embodiment of this application. Figure 3 As shown, when the mouse pointer hovers over the location of the hat, the entire hat becomes white and highlighted.

[0079] In some embodiments, to provide users with a clearer and more intuitive experience of image cutout, a selection marker can be added at the location corresponding to the selection operation when a selection operation on the target object is detected.

[0080] Figure 4 This is a schematic diagram of the dotted markers provided in an embodiment of this application. For example... Figure 4 As shown in the image, the small dots are selection markers. When a user selects multiple target objects consecutively, such as selecting a hat, a face, and a body, selection markers will be added at the locations of the hat, face, and body, respectively.

[0081] This embodiment highlights the edge contour of the target object to be cut out and transforms the user's operation during the selection process into a visual cutout operation, which is rendered to the user in real time, allowing the user to clearly understand the cutout process.

[0082] In some embodiments, when a user performs an undo operation, in order to avoid the time-consuming process of repeatedly running the ONNX model, the method may further include: in response to the undo operation, regenerating the matting result based on the saved historical point selection coordinates of the target object and the initial mask data.

[0083] Specifically, an undo option can be set on the cutout interface. When the mouse pointer is detected to be selected by the undo option, it can be recognized as an undo operation. At this time, the cutout result can be regenerated based on the saved historical point coordinates of the target object and the initial mask data.

[0084] Figure 5 This is a schematic diagram of the image cutout interface provided in an embodiment of this application. Figure 5 As shown, the cutout interface is a two-column page. The left page is the real-time editing page, and the right page is the effect preview page. The real-time editing page has options such as keep, remove, undo, and redo. The two-column page also has options such as "download image," "save to my materials," and "done."

[0085] Specifically, when the mouse pointer is selected by the "Remove" option, the corresponding image region of the selected target object can also be removed. For example, if a user selects a hat and then clicks the "Remove" option, the selected hat will be removed. If a user consecutively selects a hat, a face, and a body, and then selects at least one of these three objects again and clicks the "Remove" option, at least one of these three objects will be removed.

[0086] When the mouse pointer is detected to have the "Redo" option selected, the current cutout result is cleared. This is equivalent to re-cutting out the original image.

[0087] Figure 6 This is a framework diagram of a real-time point-and-click image matting scheme provided in an embodiment of this application. Figure 6 As shown, this real-time point-and-click image cutout solution includes:

[0088] (1) Export the SAM model running on the server as ONNX format and initialize it through ONNXRuntime-Web to deploy the ONNX model on the web.

[0089] (2) When a user uploads the original image, the original mask data is generated by the matting service function of the SAM model running on the server, and the original mask data is converted into tensor data, a data format that the Onnx model can process, and returned to the Web client.

[0090] (3) When an interactive operation is detected, the Onnx model performs real-time calculations based on the position coordinates of the interactive operation and the original mask data to obtain the initial mask data of the target object of the interactive operation.

[0091] (4) If it is determined that it is not a point selection operation, i.e. a hover operation, generate a hover effect image based on the initial mask data.

[0092] (5) When the operation is determined to be a point selection, the initial mask data is compressed to obtain compressed mask data. The position coordinates corresponding to the edge contour of the target object are obtained based on the compressed mask data. The path attribute is used to generate line segments based on the position coordinates corresponding to the edge contour of the target object. The generated line segments form path data. Then, the path data is highlighted and the inside of the edge contour is configured as transparent to obtain the cutout result.

[0093] The above-described image cutout process can be achieved through an interactive editor. The interactive editor includes components for Konva's underlying rendering, SVG to Canvas rendering, adding icons by selection, removing blank edges from cutouts, recreating, downloading cutouts, and undoing operations.

[0094] Among them, the Konva underlying rendering component is used to provide edge highlighting effects for target objects.

[0095] This is an SVG to canvas rendering component used to convert SVG data into a canvas (HTML5 canvas) for rendering and manipulation in a browser.

[0096] Click on the icon to add a component, which provides a selection marker function to remind the user that the current area has been selected.

[0097] The cutout blank edge removal component is used to set the image area outside the cutout target object as a blank area in the cutout effect preview interface to highlight the cutout effect.

[0098] The Reset Operation component provides the function to clear the current cutout result.

[0099] This is a cutout download component, which provides the function of downloading cutout results.

[0100] The undo component provides undo functionality.

[0101] In some embodiments, when an SVG with a DOM structure cannot be exported as an image, a new Konva can be created, layers such as selection markers can be removed, and the cutout result of the target object (i.e., the edge outline of the target object is highlighted) can be used to export the cutout effect, thus avoiding recalculation of the data.

[0102] In some embodiments, the coordinates of each point selection can be saved to obtain a point selection coordinate stack, and the initial mask data obtained each time can be saved to obtain a mask data stack, so as to support the undo operation and avoid having to run the model repeatedly for each undo.

[0103] The interactive editor in this embodiment can support user click behavior in complex visual images, providing functions such as adding or removing selected areas, as well as undo and redo operations. In addition, it can also preview the cutout effect in real time.

[0104] As can be seen from the above description, the image processing method proposed in this application reduces the amount of computation of redundant information of pixels on non-edge contour lines during the matting process by compressing the initial mask data of the target object to be matted and performing matting based on the compressed mask data. This reduces the matting time, improves the matting efficiency, and achieves real-time matting effect.

[0105] Corresponding to the image processing method described above, embodiments of this application also propose an image processing apparatus, such as... Figure 7 As shown, the device includes: an acquisition unit 801, configured to acquire initial mask data of the target object in response to a first interactive operation of the target object in the original image to be cut out; a compression unit 802, configured to compress the initial mask data to obtain compressed mask data corresponding to the target object, wherein the pixel values ​​of pixels on the edge contour line and pixels on non-edge contour lines of the target object are distinguished in the compressed mask data; and a generation unit 803, configured to generate a cutout result corresponding to the target object in the original image based on the compressed mask data.

[0106] In some embodiments, the step of obtaining initial mask data of the target object in response to a first interactive operation on the original image to be cut out includes: in response to a loading operation on the original image, obtaining original mask data generated by an image segmentation model running on a server for the original image, wherein the mask values ​​of the original mask data corresponding to each pixel in the original image are within a first numerical range; and using an ONNX model running on the web to determine the initial mask data corresponding to the target object based on the original mask data and the position coordinates corresponding to the first interactive operation, wherein the initial mask data includes the mask values ​​corresponding to each pixel in the original image, the mask values ​​corresponding to the pixels of the target object are within the first numerical range, and the mask values ​​corresponding to the pixels outside the target object are within a second numerical range.

[0107] In some embodiments, the compression unit 802 compresses the initial mask to obtain compressed mask data corresponding to the target object, including: counting the number of consecutive mask values ​​within a first numerical range and the number of consecutive mask values ​​within a second numerical range in the initial mask data, and compressing the initial mask data to obtain compressed mask data.

[0108] In some embodiments, the compressed mask data is in image format, and the generation unit 803 generates a cutout result corresponding to the target object in the original image based on the compressed mask data, including: determining the contour edge of the target object based on the compressed mask data, and parsing the position coordinates of the pixels located on the edge contour of the target object in the compressed mask data.

[0109] In some embodiments, the generation unit 803 highlights the edge contour of the target object based on the SVG data to obtain a cutout result corresponding to the target object in the original image, including: drawing lines based on the SVG data using line drawing instructions to generate multiple line segments corresponding to the edge contour of the target object; connecting the multiple line segments using move instructions to obtain the edge contour corresponding to the target object; and highlighting the edge contour of the target object to obtain the cutout result corresponding to the target object.

[0110] In some embodiments, the generation unit 803 is further configured to: set the remaining image regions in the original image, excluding the target object, to a color different from the color of the target object; and / or, separate the cutout result corresponding to the target object from the target object and display it separately.

[0111] In some embodiments, the generation unit 803 is further configured to: in response to a second interactive operation of a target object in the original image to be cut out, generate a hover effect image of the target object using an ONNX model running on the web, wherein the target object is highlighted in the hover effect image.

[0112] In some embodiments, the generation unit 803 is further configured to: regenerate the matting result based on the saved historical point selection coordinates of the target object and the initial mask data in response to the undo operation.

[0113] The image processing apparatus provided in this embodiment belongs to the same concept as the image processing method provided in the above embodiments of this application. It can execute the image processing method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the image processing method. Technical details not described in detail in this embodiment can be found in the specific processing content of the image processing method provided in the above embodiments of this application, and will not be repeated here.

[0114] The functions implemented by each unit in the above image processing device can be implemented by the same or different processors, and the embodiments of this application do not limit this.

[0115] It should be understood that the units in the above image processing device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through a configuration file to implement the functions of some or all of the above units. All units of the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0116] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0117] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0118] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0119] This application also proposes a control device, which includes a processor and an interface circuit. The processor in the control device is connected to a data input component through the interface circuit of the control device.

[0120] The data input component specifically refers to a functional component that can input or collect voice data, such as a microphone, etc.

[0121] The aforementioned interface circuit can be any interface circuit capable of implementing data communication functions, such as a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIe circuit, etc.

[0122] The processor in this control device is also a circuit with signal processing capabilities, which executes the image processing method described in the above embodiments. For specific implementation details of this processor, please refer to the processor implementation methods described above; this application does not impose strict limitations on the embodiments.

[0123] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 8As shown, the device includes:

[0124] Memory 200 and processor 210;

[0125] The memory 200 is connected to the processor 210 and is used to store programs;

[0126] The processor 210 is configured to implement the image processing method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0127] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0128] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0129] A bus can include a pathway for transmitting information between various components of a computer system.

[0130] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0131] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0132] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0133] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0134] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0135] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0136] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement various steps of any of the image processing methods provided in the above embodiments of this application.

[0137] This application also proposes a chip including a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the image processing method described in any of the above embodiments. For details of the processing and its beneficial effects, please refer to the embodiments of the above image processing method.

[0138] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the image processing methods described in any of the above embodiments of this specification.

[0139] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0140] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor of the steps in the image processing method described in any of the above embodiments of this specification.

[0141] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0142] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0143] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0144] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0145] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0146] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0147] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0148] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0149] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0151] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: In response to a first interactive operation on the target object in the original image to be cut out, the initial mask data of the target object is obtained; The initial mask data is compressed to obtain compressed mask data corresponding to the target object. The pixel values ​​of the pixels on the edge contour line and the pixels on the non-edge contour line of the target object are distinguished and marked in the compressed mask data. Based on the compressed mask data, the matting result corresponding to the target object in the original image is generated.

2. The method according to claim 1, characterized in that, The first interactive operation in response to the target object in the original image to be cut out, obtaining the initial mask data of the target object, includes: In response to the loading operation of the original image, the original mask data generated by the image segmentation model running on the server for the original image is obtained, wherein the mask value of the original mask data corresponding to each pixel in the original image is within a first numerical range; The ONNX model running on the web is used to determine the initial mask data corresponding to the target object based on the original mask data and the position coordinates corresponding to the first interactive operation. The initial mask data includes the mask value corresponding to each pixel in the original image. The mask value corresponding to the pixel of the target object is within a first numerical range, and the mask value corresponding to the pixel outside the target object is within a second numerical range.

3. The method according to claim 2, characterized in that, The step of compressing the initial mask to obtain compressed mask data corresponding to the target object includes: The number of consecutive mask values ​​within the first numerical range and the number of consecutive mask values ​​within the second numerical range in the initial mask data are counted. The initial mask data is then compressed to obtain compressed mask data.

4. The method according to claim 3, characterized in that, The compressed mask data is in image format. The step of generating a cutout result corresponding to the target object in the original image based on the compressed mask data includes: The contour edges of the target object are determined based on the compressed mask data, and the position coordinates of the pixels located on the edge contour of the target object in the compressed mask data are analyzed to obtain the SVG data corresponding to the target object. The edge contour of the target object is highlighted based on the SVG data to obtain the cutout result corresponding to the target object in the original image.

5. The method according to claim 4, characterized in that, The step of highlighting the edge contour of the target object based on the SVG data to obtain the cutout result corresponding to the target object in the original image includes: Line drawing is performed based on the SVG data using line drawing instructions to generate multiple line segments corresponding to the edge contour of the target object. The multiple line segments are connected by the move command to obtain the edge contour corresponding to the target object; The edge contour of the target object is highlighted to obtain the cutout result corresponding to the target object.

6. The method according to claim 4, characterized in that, The method further includes: Set the remaining image regions in the original image, excluding the target object, to a color different from the color of the target object; And / or, The cutout result corresponding to the target object is separated from the target object and displayed separately.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to a second interactive operation on the target object in the original image to be cut out, an ONNX model running on the web is used to generate a hover effect image of the target object, in which the target object is highlighted.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to the undo operation, the matting result is regenerated based on the saved historical point selection coordinates of the target object and the initial mask data.

9. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the initial mask data of the target object in response to a first interactive operation of the target object in the original image to be cut out; A compression unit is used to compress the initial mask data to obtain compressed mask data corresponding to the target object. The pixel values ​​of pixels on the edge contour line and pixels on non-edge contour lines of the target object are distinguished and marked in the compressed mask data. The generation unit is used to generate the matting result corresponding to the target object in the original image based on the compressed mask data.

10. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method as described in any one of claims 1 to 8 by running a program in the memory.

11. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 8.

Images