Image segmentation method, device, electronic device and storage medium

Abstract

The present disclosure provides an image segmentation method, device, electronic device and storage medium. The image segmentation method includes: receiving an interactive operation on an image; cutting out a corresponding regional image from the image according to the scope of the interactive operation; obtaining a result mask of the interactive operation according to a first image mask obtained in a previous interactive operation of the interactive operation, the regional image and the mask of the interactive operation; updating the first image mask to a second image mask using the result mask of the interactive operation, and obtaining a segmented image based on the second image mask.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for interactive image segmentation. Background Art

[0002] Image segmentation technology is a very important computer vision task, which has many applications in image retrieval, image editing and film and television production. As a subtask in the field of image segmentation, interactive segmentation aims to distinguish objects of interest from backgrounds with minimal user input and inference time, and achieve the best segmentation accuracy. Due to the diversity of user input methods (such as clicks, smears, bounding boxes, etc.), the interactive segmentation method provides users with great flexibility and can effectively adjust the current segmentation results according to user guidance.

[0003] The interactive segmentation technology in related technologies relies on two interactive operations: click and scribble. In each interactive process, segmentation is performed based on the entire image. The disadvantage of this method is that the effective resolution of the interactive model network is reduced based on the entire image, so it has great limitations for the segmentation of details. Summary of the invention

[0004] According to a first aspect of the present disclosure, there is provided an image segmentation method, characterized in that it includes: receiving an interactive operation on an image; cutting out a corresponding regional image from the image according to the scope of the interactive operation; obtaining a result mask of the interactive operation according to a first image mask obtained in a previous interactive operation of the interactive operation, the regional image and the mask of the interactive operation; updating the first image mask to a second image mask using the result mask of the interactive operation, and obtaining a segmented image based on the second image mask.

[0005] According to the first aspect of the present disclosure, the interactive operation includes a smear and / or click operation on the image.

[0006] According to a first aspect of the present disclosure, a corresponding regional image is cut out from an image according to a range of the interactive operation, including: determining an interactive area corresponding to a trajectory of the interactive operation; and cutting out the regional image corresponding to the interactive area from the image.

[0007] According to a first aspect of the present disclosure, the interaction area is a circular area, and the step of cutting out the area image corresponding to the interaction area from the image includes: cutting out the area image corresponding to the rectangular area surrounding the circular area from the image; or cutting out a partial image corresponding to the circular area from the image, and processing the partial image into a rectangular image to obtain the area image.

[0008] According to a first aspect of the present disclosure, the interaction area is a rectangular area, and cutting out the area image corresponding to the interaction area from the image includes: cutting out the area image corresponding to the rectangular area from the image.

[0009] According to a first aspect of the present disclosure, clipping out a regional image corresponding to the rectangular area from an image includes: expanding the rectangular area at a predetermined ratio, and clipping out the regional image corresponding to the expanded rectangular area from the image.

[0010] According to the first aspect of the present disclosure, the predetermined ratio is negatively correlated with the size of the rectangular area, and / or the predetermined ratio is negatively correlated with the degree to which the image is magnified.

[0011] According to a first aspect of the present disclosure, expanding the rectangular area at a predetermined ratio includes: enlarging the length of the rectangular area at a first ratio and enlarging the width of the rectangular area at a second ratio.

[0012] According to a first aspect of the present disclosure, when the ratio of the size of the rectangular area to the size of the image is less than a first threshold, the predetermined ratio is determined as a first value; when the ratio of the size of the rectangular area to the size of the image is greater than the first threshold, the predetermined ratio is determined as a second value, wherein the first value is greater than the second value, and the first value and the second value are negatively correlated with the multiple by which the image is magnified.

[0013] According to a first aspect of the present disclosure, when the ratio of the size of the regional image to the size of the image is less than a second threshold, the regional image is expanded to a regional image of a first preset size, wherein the second threshold is less than the first threshold and is negatively correlated with the magnification factor of the image.

[0014] According to the first aspect of the present disclosure, the method further includes: adjusting the size of the region image to a second preset size, and acquiring a result mask of the interactive operation based on the adjusted region image.

[0015] According to a second aspect of the present disclosure, an image segmentation device is provided, comprising: an interaction unit, configured to receive an interaction operation on an image; a cropping unit, configured to crop a corresponding region image from the image according to a range of the interaction operation; an inference unit, configured to obtain a result mask of the interaction operation according to a first image mask obtained in a previous interaction operation of the interaction operation, the region image and a mask of the interaction operation; and a mask updating unit, configured to update the first image mask to a second image mask using the result mask of the interaction operation, and obtain a segmented image based on the second image mask.

[0016] According to a second aspect of the present disclosure, the interactive operation includes smearing and / or clicking operations on the image.

[0017] According to a second aspect of the present disclosure, the cropping unit is configured to: determine an interaction area corresponding to the trajectory of the interaction operation; and crop the area image corresponding to the interaction area from the image.

[0018] According to the second aspect of the present disclosure, the interaction area is a circular area, and the cropping unit is configured to: crop from the image the area image corresponding to the rectangular area surrounding the circular area; or crop from the image the partial image corresponding to the circular area, and process the partial image into a rectangular image to obtain the area image.

[0019] According to a second aspect of the present disclosure, the interaction area is a rectangular area, and the cropping unit is configured to: crop a regional image corresponding to the rectangular area from the image.

[0020] According to a second aspect of the present disclosure, the cropping unit is configured to: expand the rectangular area at a predetermined ratio, and crop the area image corresponding to the expanded rectangular area from the image.

[0021] According to a second aspect of the present disclosure, the predetermined ratio is negatively correlated with the size of the rectangular area, and / or the predetermined ratio is negatively correlated with the degree to which the image is magnified.

[0022] According to a second aspect of the present disclosure, the cropping unit is configured to: enlarge the length of the rectangular area at a first ratio, and enlarge the width of the rectangular area at a second ratio.

[0023] According to a second aspect of the present disclosure, the cropping unit is configured to: when the ratio of the size of the rectangular area to the size of the image is less than a first threshold, determine the predetermined ratio as a first value; when the ratio of the size of the rectangular area to the size of the image is greater than the first threshold, determine the predetermined ratio as a second value, wherein the first value is greater than the second value, and the first value and the second value are negatively correlated with the multiple by which the image is magnified.

[0024] According to a second aspect of the present disclosure, the cropping unit is configured to: when the ratio of the size of the regional image to the size of the image is less than a second threshold, expand the regional image into a regional image with a first preset size, wherein the second threshold is less than the first threshold and is negatively correlated with the multiple by which the image is magnified.

[0025] According to the second aspect of the present disclosure, the inference unit is configured to: adjust the size of the area image to a second preset size, and based on the adjusted area image, obtain a result mask of the interactive operation. According to the third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the image segmentation method as described above.

[0026] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is enabled to perform the image segmentation method as described above.

[0027] According to a fifth aspect of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor to perform the image segmentation method as described above.

[0028] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: the image segmentation method according to the exemplary embodiment of the present disclosure can adaptively perform local cropping of the image according to the scope of the user operation in each interaction, and the interactive operations input to the interactive segmentation network are reduced. Therefore, the image segmentation method according to the exemplary embodiment of the present disclosure is simple, and even if some erroneous operations occur in the user interactive operation, resulting in an unsatisfactory segmentation result of a certain interactive operation, it will not affect the subsequent prediction process, and has a strong fault tolerance rate, which greatly facilitates the user to perform segmentation and extraction of objects of interest. At the same time, according to the embodiment of the present disclosure, the method of cropping the regional image according to the user interactive operation can obtain a finer local area, so it can help the user extract a high-precision mask of the object of interest, and then obtain a more accurate segmented image.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0031] Figure 1 is a flowchart illustrating an image segmentation method according to an exemplary embodiment of the present disclosure.

[0032] Figure 2 is a schematic diagram illustrating interactive operations in an image segmentation method according to an exemplary embodiment of the present disclosure.

[0033] Figure 3 is a schematic diagram illustrating a local cropping process in an image segmentation method according to an exemplary embodiment of the present disclosure.

[0034] Figure 4 is a block diagram illustrating an image segmentation apparatus according to an exemplary embodiment of the present disclosure.

[0035] Figure 5 is a schematic diagram illustrating an electronic device for image segmentation according to an exemplary embodiment of the present disclosure.

[0036] Figure 6 is a schematic diagram illustrating an electronic device for image segmentation according to another exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0039] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0040] Figure 1It is a flowchart of an image segmentation method according to an exemplary embodiment of the present disclosure. It should be understood that the image segmentation scheme according to the exemplary embodiment of the present disclosure can be implemented in any device with image processing capabilities. For example, the method can be performed in a terminal device with image processing capabilities. Here, the terminal device can be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a netbook, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device. An application can be run on the terminal device, and image segmentation processing can be implemented in the application. The image segmentation method can also be performed on a server, for example. The exemplary embodiments of the present disclosure do not limit it.

[0041] like Figure 1 As shown, in step S110, an interactive operation on an image is received.

[0042] Here, the image may be an image in a predetermined format (e.g., an RGB image) displayed to the user through the screen of the electronic device. In the image, there is a target area to be segmented that the user is interested in and a background area other than the target. The user can segment the image of the target area from the image through multiple interactive operations. According to an exemplary embodiment of the present disclosure, for each user interactive operation, the following steps may be performed as described in reference to FIG. Figure 1 The corresponding processing described.

[0043] According to an exemplary embodiment of the present disclosure, the interactive operation may include a smear operation and / or a click operation. For example, a user may perform a smear operation by moving an input device (e.g., a stylus, a finger, etc.) in a certain trajectory in an image displayed on a touch screen, and the electronic device may receive the user's interactive operation by processing the signal received by the sensor. Generally, a smear operation performed in a target area of ​​interest may be referred to as positive smearing, and a smear operation performed in a background area may be referred to as negative smearing. In an interactive task, in order to obtain a result that the user is more satisfied with, it is usually necessary to go back and forth multiple times for positive and negative interactions.

[0044] Figure 2 FIG. 1 is a schematic diagram showing an interactive operation according to an exemplary embodiment of the present disclosure. Figure 2 As shown on the left in , the user wants to segment the target area of ​​interest of the tennis player from the image, so the user can draw a trajectory as shown by the line on the player's body to perform positive painting. Figure 2 In the interactive heat map on the right shown, the pixel value of the smeared position can be set to 255 (i.e., Figure 2The pixel values ​​of the remaining positions are set to 0 (ie, the areas other than the lines are all black), so that a mask of the track painted by the user can be obtained. The width of the line can be adjusted according to the application form of the specific product.

[0045] In step S120, a corresponding regional image is cut out from the image according to the range of the interactive operation.

[0046] According to an exemplary embodiment of the present disclosure, cutting out a corresponding regional image from an image according to a range of the interactive operation may include: determining an interactive area corresponding to a trajectory of the interactive operation; and cutting out the regional image corresponding to the interactive area from the image.

[0047] According to an exemplary embodiment of the present disclosure, the interaction area may be a circular area. For example, the area of ​​the minimum enclosing circle surrounding the interaction trajectory may be determined as the interaction area. In one example, the maximum length of the coordinates of the connection trajectory may be determined, and the minimum enclosing circle may be obtained with the maximum length as the diameter. The step of clipping the area image corresponding to the interaction area from the image may include: clipping the area image corresponding to the rectangular area surrounding the circular area from the image. In one example, pixels of 0 or average value may be filled around the circular area to obtain the minimum rectangular area surrounding the circular area, and then the area image corresponding to the minimum rectangular area may be clipped from the image. Alternatively, a partial image corresponding to the circular area may be clipped from the image, and the partial image may be processed into a rectangular image to obtain the area image. For example, pixels of 0 or average value may be added around the area image of the circular area to obtain an image of a rectangular area.

[0048] According to an exemplary embodiment of the present disclosure, the interaction area may be a rectangular area, and cutting out the area image corresponding to the interaction area from the image includes: cutting out the area image corresponding to the rectangular area from the image.

[0049] It should be understood that the shape of the interaction area is not limited here, and any shape of area covering the trajectory of the user interaction operation can be used, and the interaction area can be processed into a rectangular image in various ways for subsequent segmentation processing.

[0050] According to an exemplary embodiment of the present disclosure, the scope of the interactive operation may be the smallest rectangular area covering the track of the user's interactive operation, that is, the smallest rectangular area is the interactive area corresponding to the track of the interactive operation. Figure 3As shown, the rectangular area shown in 310 is the minimum rectangular area covering the user's smearing track. The minimum rectangular area can be determined by the maximum and minimum values ​​of the coordinates of the track on the screen coordinate axis. In addition, according to an exemplary embodiment of the present disclosure, the area image obtained through the user interaction operation can be expanded to obtain better context information to accurately determine the target area of ​​interest to the user. According to an exemplary embodiment of the present disclosure, in step S120, the rectangular area obtained according to the range of the user interaction operation can be expanded at a predetermined ratio. Figure 3 As shown, the rectangular area 310 covered by the user interaction can be adaptively expanded according to a predetermined expansion ratio, thereby obtaining a more accurate segmented area 320. The expansion ratio here refers to the ratio of the length of the expanded part to the original length.

[0051] According to an exemplary embodiment of the present disclosure, the predetermined ratio for performing the expansion is negatively correlated with the size of the rectangular area covered by the user interaction operation, and / or the predetermined ratio is negatively correlated with the degree to which the image is magnified. That is, the expansion ratio according to the type embodiment of the present disclosure is related to two factors, namely, the size ratio of the rectangular area covered by the user interaction operation to the original image and the multiple by which the user magnifies the original image during the interaction process. The larger the size of the rectangular area covered by the user interaction operation to the original image, the smaller the expansion ratio can be, and the larger the multiple by which the user magnifies the original image during the interaction operation, the smaller the expansion ratio can be. In this way, it can be ensured that a relatively suitable clipping area can be obtained for different user interaction operations, and the clipping area can contain enough information for subsequent target segmentation operations, and will not contain too much information to affect the processing speed.

[0052] According to an exemplary embodiment of the present disclosure, expanding the rectangular area at a predetermined ratio includes: enlarging the length of the rectangular area at a first ratio, and enlarging the width of the rectangular area at a second ratio. The first ratio and the second ratio may be different or the same. In other words, the length and width of the rectangular area may be expanded at the same ratio, or at different ratios, or the first ratio and the second ratio may have a preset relationship, which is not limited here.

[0053] In addition, according to an exemplary embodiment of the present disclosure, a cropped regional image can be determined according to a minimum rectangular area in a piecewise manner. When the ratio of the size of the rectangular area to the size of the image is less than a first threshold, the predetermined ratio is determined as a first value; when the ratio of the size of the rectangular area to the size of the image is greater than the first threshold, the predetermined ratio is determined as a second value, wherein the first value is greater than the second value, and the first value and the second value are negatively correlated with the multiple of the image magnified by the user. For example, when the ratio of the size of the rectangular area to the size of the image is less than the first threshold, the predetermined ratio ER is determined as ER= / ; when the ratio of the size of the rectangular area to the size of the image is greater than the first threshold, the predetermined ratio ER is determined as ER=β / z, wherein α>β, z represents the multiple of the user's magnification of the image, and α and β are pre-set according to the business scenario, for example, α=2.0, β=1.5. Here, the first threshold may be a value between 0-1, for example, it may be 0.3. Therefore, the piecewise function may be set as the following equation 1:

[0054]

[0055] Among them, Area(R min ) represents the size of the minimum rectangular area, Area(I RGB ) indicates the size of the original image.

[0056] According to an exemplary embodiment of the present disclosure, when the ratio of the size of the cropped area image to the size of the image is less than a second threshold, the cropped area image is expanded to an area with a preset size, wherein the second threshold is less than the first threshold and is negatively correlated with the multiple by which the user magnifies the image. Here, the area image of the preset size is, for example, an area image whose area is a predetermined ratio (for example, 0.2) of the original image. The ratio here can be adjusted according to actual conditions. That is, after cropping with the expanded minimum rectangular area as described above, there may still be a situation where the cropped area image is small and the target of interest to the user cannot be quickly segmented. Therefore, a cropping area Area (R expand ) is the lower limit value, when Area(R expand ) / Area(I RGB ) is less than a certain hyperparameter threshold τ (i.e., the second threshold), R can be further expanded expand The area, τ, is defined as follows:

[0057] τ=γ / z

[0058] When z is larger, τ will be smaller accordingly. According to an exemplary embodiment of the present disclosure, the default value of γ may be 0.34. Through the above segmented expansion and the lower limit, the local area image can be better refined.

[0059] It should be understood that the above manner of expanding the scope of user interaction operations is only illustrative, and those skilled in the art may make various changes according to the needs of actual business scenarios.

[0060] In other embodiments of the present disclosure, the interactive area may also be other geometric shapes besides a circle or a rectangle, such as an ellipse, or the outline of a target targeted by the interactive operation. There are many implementation methods for cropping the area image:

[0061] In one example, the regional image corresponding to the rectangular area surrounding the interactive area is clipped from the image. For example, pixels of 0 or average value may be filled around the interactive area to obtain the minimum rectangular area surrounding the interactive area, and then the regional image corresponding to the minimum rectangular area is clipped from the image. It should be understood that the minimum rectangular area is only an example, and the rectangular area surrounding the interactive area may not be the minimum rectangular area. Those skilled in the art may make various changes according to the needs of actual business scenarios.

[0062] In another example, a portion of the image corresponding to the interactive area is cut out from the image, and the portion of the image is processed into a rectangular image to obtain the area image. For example, pixels of 0 or average value may be added around the area image of the interactive area to obtain the rectangular area image.

[0063] It can be seen that the obtained area image is a rectangular image.

[0064] In step S130, a result mask of the interactive operation is obtained according to the first image mask obtained in the previous interactive operation, the region image and the mask of the interactive operation. The result mask output here is a segmentation mask corresponding to the cropped region image.

[0065] As described above, in each interactive operation of the user, the image mask obtained after the last interaction, the regional image cropped in step S120, and the mask of this interactive operation will be used. These will be input into the trained interactive image segmentation reasoning model to output the result mask of this interaction. The reasoning model here can be a convolutional neural network (CNN) model for interactive image segmentation obtained through mask-guided recovery interaction training. The model can generate a predicted mask result for the input image based on the input image, user guidance, and the mask result of the last interactive operation through the neural network. It should be understood that the present application can also use other trained forward feedback form reasoning models to perform reasoning to obtain the corresponding mask.

[0066] According to an exemplary embodiment of the present disclosure, the size of the area clipped by the interactive operation is adjusted to a predetermined size before being input into the interactive image segmentation reasoning model. This is because the model usually requires an input of a fixed size. Therefore, the clipped area can be adjusted to a fixed size to facilitate the use of the model.

[0067] In step S140, the first image mask is updated to a second image mask using the result mask of the interactive operation, and a segmented image is obtained based on the second image mask. After obtaining the result mask corresponding to the cropped area, the result mask is updated to the corresponding area of ​​the full-image mask after the last interactive operation on the image, thereby obtaining the full-image mask after this interaction. In other words, the position of the result mask of the interactive operation can be recorded, and the obtained result mask can be updated to the full-image mask obtained in the last interactive operation according to the position of the result mask, so that a new full-image mask can be obtained. After obtaining the new full-image mask, the image of the target area of ​​interest to the user can be obtained through the full-image mask as the final image segmentation result. It should be understood that if there is no last interactive operation before the current interactive operation (that is, the current interactive operation is the first interactive operation), the first image mask is the initial image mask, and all pixel values ​​of the initial image mask can be 0.

[0068] For example, if the user's interactive input is positive smearing (i.e., smearing of the object of interest), the output result mask can be updated to the corresponding area of ​​the full-image mask with a pixel value of 255. If the user's input is negative smearing (i.e., smearing of the background area), the output result mask can be updated to the corresponding area of ​​the full-image mask with a pixel value of 0.

[0069] According to the image segmentation method of the exemplary embodiment of the present disclosure, the image can be locally cropped according to the scope of the user operation in each interaction, and then sent to the interactive segmentation model to generate the user's target segmentation mask. In addition, in the related art, the results of all interactive operations before the current interactive operation need to be input into the interactive segmentation model for processing, so if an error occurs in an interactive operation, it will interfere with the final image segmentation result, and the error tolerance rate is small. According to the image segmentation method of the exemplary embodiment of the present disclosure, the interactive operations input to the interactive segmentation network are reduced. Therefore, the image segmentation method according to the exemplary embodiment of the present disclosure is simple, even if some erroneous operations occur in the user interactive operation, resulting in an unsatisfactory segmentation result of a certain interactive operation, it will not affect the subsequent prediction process, has a strong error tolerance rate, and greatly facilitates the user to perform segmentation and extraction of objects of interest. At the same time, according to the embodiment of the present disclosure, the method of cropping the regional image according to the user interactive operation can obtain a more refined local area, so it can help the user extract a high-precision object of interest mask, and then obtain a more accurate segmented image.

[0070] Figure 4 is a block diagram showing an image segmentation apparatus according to an exemplary embodiment of the present disclosure. It should be understood that the image segmentation apparatus can be implemented in an electronic device with image processing in a software / hardware or software / hardware combination manner.

[0071] like Figure 4 As shown, the image segmentation apparatus 400 according to an exemplary embodiment of the present disclosure may include an interaction unit 410 , a cropping unit 420 , an inference unit 430 , and a mask updating unit 440 .

[0072] The interaction unit 410 is configured to receive an interaction operation on an image. The cropping unit 420 is configured to crop a corresponding region image from the image according to the range of the interaction operation.

[0073] The reasoning unit 430 is configured to obtain a result mask of the interactive operation according to the first image mask obtained in the previous interactive operation of the interactive operation, the region image and the mask of the interactive operation.

[0074] The mask updating unit 440 is configured to update the first image mask to a second image mask using the result mask of the interaction operation, and obtain a segmented image based on the second image mask.

[0075] According to an exemplary embodiment of the present disclosure, the interactive operation includes a smear and / or click operation on the image.

[0076] According to an exemplary embodiment of the present disclosure, the cropping unit is configured to: determine an interaction area corresponding to the trajectory of the interaction operation; and crop the area image corresponding to the interaction area from the image.

[0077] According to an exemplary embodiment of the present disclosure, the interaction area is a circular area, and the cropping unit is configured to: crop from the image the area image corresponding to the rectangular area surrounding the circular area; or crop from the image the partial image corresponding to the circular area, and process the partial image into a rectangular image to obtain the area image.

[0078] According to an exemplary embodiment of the present disclosure, the interaction area is a rectangular area, and the cropping unit is configured to: crop a regional image corresponding to the rectangular area from the image.

[0079] According to an exemplary embodiment of the present disclosure, the cropping unit 420 may be configured to: expand the rectangular area at a predetermined ratio, and crop the area image corresponding to the expanded rectangular area from the image.

[0080] According to an exemplary embodiment of the present disclosure, the predetermined ratio is negatively correlated with the size of the rectangular area, and / or the predetermined ratio is negatively correlated with the degree to which the image is enlarged in the interactive operation.

[0081] According to an exemplary embodiment of the present disclosure, the cropping unit 420 may be configured to: enlarge the length of the rectangular area at a first ratio, and enlarge the width of the rectangular area at a second ratio. The first ratio and the second ratio may be independent of each other.

[0082] According to an exemplary embodiment of the present disclosure, the cropping unit 420 may be configured to: when the ratio of the size of the rectangular area to the size of the image is less than a first threshold, determine the predetermined ratio as a first value; when the ratio of the size of the rectangular area to the size of the image is greater than the first threshold, determine the predetermined ratio as a second value, wherein the first value is greater than the second value, and the first value and the second value are negatively correlated with the multiple by which the image is magnified by the user.

[0083] According to an exemplary embodiment of the present disclosure, the cropping unit 420 may be configured to expand the regional image into an area having a first preset size when the ratio of the size of the regional image to the size of the image is less than a second threshold, wherein the second threshold is less than the first threshold and is negatively correlated with the multiple by which the image is magnified.

[0084] According to an exemplary embodiment of the present disclosure, the reasoning unit 430 may be configured to: adjust the size of the region image to a second preset size, and acquire a result mask of the interaction operation based on the adjusted region image.

[0085] The above has been referred to Figure 1 The image segmentation method according to the exemplary embodiment of the present disclosure is described, wherein each step is as follows Figure 4 The various units of the image segmentation device shown correspond to each other, so they will not be described repeatedly.

[0086] Figure 5 1 is a block diagram showing a structure of an electronic device 500 for video processing for image segmentation according to an exemplary embodiment of the present disclosure. The electronic device 500 may be, for example, a smart phone, a tablet computer, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. The electronic device 500 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0087] Typically, the electronic device 500 includes a processor 501 and a memory 502 .

[0088] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor, wherein the main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); and the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In an exemplary embodiment of the present disclosure, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0089] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement the image segmentation method of the exemplary embodiment of the present disclosure.

[0090] In some embodiments, the electronic device 500 may further optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504, a touch display screen 505, a camera 506, an audio circuit 507, a positioning component 508 and a power supply 509.

[0091] The peripheral device interface 503 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502. In some embodiments, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0092] The radio frequency circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 504 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 504 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0093] The display screen 505 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to collect touch signals on the surface or above the surface of the display screen 505. The touch signal can be input to the processor 501 as a control signal for processing. At this time, the display screen 505 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 505 can be one, arranged on the front panel of the electronic device 500; in other embodiments, the display screen 505 can be at least two, respectively arranged on different surfaces of the terminal 500 or in a folding design; in some embodiments, the display screen 505 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 500. Even, the display screen 505 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 505 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0094] The camera assembly 506 is used to capture images or videos. Optionally, the camera assembly 506 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0095] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 501 for processing, or input them into the radio frequency circuit 504 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 500. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 507 may also include a headphone jack.

[0096] Positioning component 508 is used to locate the current geographic location of electronic device 500 to implement navigation or LBS (Location Based Service). Positioning component 508 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0097] The power supply 509 is used to power various components in the electronic device 500. The power supply 509 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 509 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0098] In some embodiments, the electronic device 500 further includes one or more sensors 510 , including but not limited to: an acceleration sensor 511 , a gyroscope sensor 512 , a pressure sensor 513 , a fingerprint sensor 514 , an optical sensor 515 , and a proximity sensor 516 .

[0099] The acceleration sensor 311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 500. For example, the acceleration sensor 511 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 501 can control the touch display screen 505 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 511. The acceleration sensor 511 can also be used for collecting game or user motion data.

[0100] The gyro sensor 512 can detect the body direction and rotation angle of the terminal 500, and the gyro sensor 512 can cooperate with the acceleration sensor 511 to collect the user's 3D actions on the terminal 500. The processor 501 can implement the following functions based on the data collected by the gyro sensor 512: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0101] The pressure sensor 513 can be set on the side frame of the terminal 500 and / or the lower layer of the touch display screen 505. When the pressure sensor 513 is set on the side frame of the terminal 500, it can detect the user's holding signal of the terminal 500, and the processor 501 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 513. When the pressure sensor 513 is set on the lower layer of the touch display screen 505, the processor 501 controls the operability controls on the UI according to the user's pressure operation on the touch display screen 505. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0102] The fingerprint sensor 514 is used to collect the user's fingerprint, and the processor 501 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 501 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 514 can be set on the front, back or side of the electronic device 500. When a physical button or a manufacturer logo is set on the electronic device 500, the fingerprint sensor 514 can be integrated with the physical button or the manufacturer logo.

[0103] The optical sensor 515 is used to collect the ambient light intensity. In one embodiment, the processor 501 can control the display brightness of the touch display screen 505 according to the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 505 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 505 is reduced. In another embodiment, the processor 501 can also dynamically adjust the shooting parameters of the camera assembly 506 according to the ambient light intensity collected by the optical sensor 515.

[0104] The proximity sensor 516, also called a distance sensor, is usually disposed on the front panel of the electronic device 500. The proximity sensor 516 is used to collect the distance between the user and the front of the electronic device 500. In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the terminal 500 is gradually decreasing, the processor 501 controls the touch display screen 505 to switch from the screen-on state to the screen-off state; when the proximity sensor 516 detects that the distance between the user and the front of the electronic device 500 is gradually increasing, the processor 501 controls the touch display screen 505 to switch from the screen-off state to the screen-on state.

[0105] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the electronic device 500, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0106] Figure 6 FIG. 6 is a block diagram of another electronic device 600. For example, the electronic device 600 may be provided as a server. Figure 6 , the electronic device 600 includes one or more processing processors 610 and a memory 620. The memory 620 may include one or more programs for executing the above image segmentation method. The electronic device 600 may also include a power supply component 630 configured to perform power management of the electronic device 600, a wired or wireless network interface 640 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 650. The electronic device 600 may operate based on an operating system stored in the memory 620, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0107] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform the image segmentation method according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0108] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the above-mentioned image segmentation method.

[0109] The image segmentation method according to the exemplary embodiment of the present disclosure can adaptively crop the image locally according to the range of the user operation in each interaction, reduce the interactive operations input to the interactive segmentation network, and have a strong fault tolerance rate. A finer local area can be obtained, and a more accurate segmented image can be obtained.

[0110] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the schemes disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0111] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image segmentation method, characterized in that: include: receiving interactive operations on the image; Cutting out a corresponding regional image from the image according to the range of the interactive operation; Acquire a result mask of the interactive operation according to a first image mask obtained in a previous interactive operation of the interactive operation, the region image, and a mask of the interactive operation; Using the result mask of the interaction operation to update the first image mask to a second image mask, and obtaining a segmented image based on the second image mask, The step of cutting out a corresponding regional image from the image according to the range of the interactive operation includes: Determining an interactive area corresponding to the trajectory of the interactive operation; Cut out the area image corresponding to the interactive area from the image, where the area image is a rectangular area; The rectangular area is expanded at a predetermined ratio, and the area image corresponding to the expanded rectangular area is cut out from the image.

2. The method according to claim 1, characterized in that The interactive operation includes painting and / or clicking operations on the image.

3. The method according to claim 1, characterized in that The interactive area is a circular area, and the step of cutting out the area image corresponding to the interactive area from the image includes: Cut out the area image corresponding to the rectangular area surrounding the circular area from the image; or A partial image corresponding to the circular area is cut out from the image, and the partial image is processed into a rectangular image to obtain the area image.

4. The method according to claim 1, characterized in that The predetermined ratio is negatively correlated with the size of the rectangular area, and / or The predetermined ratio is inversely correlated to the degree to which the image is magnified.

5. The method according to claim 1, characterized in that Expanding the rectangular area at a predetermined ratio includes: enlarging the length of the rectangular area at a first ratio, and enlarging the width of the rectangular area at a second ratio.

6. The method according to claim 4, characterized in that When the ratio of the size of the rectangular area to the size of the image is less than a first threshold, determining the predetermined ratio as a first value; When the ratio of the size of the rectangular area to the size of the image is greater than a first threshold, the predetermined ratio is determined as a second value, The first value is greater than the second value, and the first value and the second value are negatively correlated with the magnification factor of the image.

7. The method according to claim 6, characterized in that When the ratio of the size of the regional image to the size of the image is less than a second threshold, the regional image is expanded to a regional image of a first preset size, wherein the second threshold is less than the first threshold and is negatively correlated with the magnification factor of the image.

8. The method according to claim 1, characterized in that Also includes: The size of the region image is adjusted to a second preset size, and based on the adjusted region image, a result mask of the interactive operation is obtained.

9. An image segmentation device, comprising: An interaction unit, configured to receive an interaction operation for an image; A clipping unit, configured to clip a corresponding region image from the image according to the range of the interactive operation; an inference unit configured to obtain a result mask of the interactive operation according to a first image mask obtained in a previous interactive operation of the interactive operation, the region image, and a mask of the interactive operation; a mask updating unit configured to update the first image mask to a second image mask using the result mask of the interaction operation, and obtain a segmented image based on the second image mask, The trimming unit is configured as follows: Determining an interactive area corresponding to the trajectory of the interactive operation; Cut out the area image corresponding to the interactive area from the image, where the area image is a rectangular area; The rectangular area is expanded at a predetermined ratio, and the area image corresponding to the expanded rectangular area is cut out from the image.

10. The device according to claim 9, characterized in that The interactive operation includes a smear and / or click operation on the image.

11. The device according to claim 9, characterized in that The interaction area is a circular area, and the clipping unit is configured as follows: Cut out the area image corresponding to the rectangular area surrounding the circular area from the image; or A partial image corresponding to the circular area is cut out from the image, and the partial image is processed into a rectangular image to obtain the area image.

12. The device according to claim 9, characterized in that The predetermined ratio is negatively correlated with the size of the rectangular area, and / or The predetermined ratio is inversely correlated to the degree to which the image is magnified.

13. The device according to claim 9, characterized in that The cropping unit is configured to: enlarge the length of the rectangular area at a first ratio, and enlarge the width of the rectangular area at a second ratio.

14. The device according to claim 12, characterized in that The trimming unit is configured as: When the ratio of the size of the rectangular area to the size of the image is less than a first threshold, determining the predetermined ratio as a first value; When the ratio of the size of the rectangular area to the size of the image is greater than a first threshold, the predetermined ratio is determined as a second value, The first value is greater than the second value, and the first value and the second value are negatively correlated with the magnification factor of the image.

15. The device according to claim 14, characterized in that The cropping unit is configured to expand the regional image into a regional image with a first preset size when the ratio of the size of the regional image to the size of the image is less than a second threshold, wherein the second threshold is less than the first threshold and is negatively correlated with the magnification factor of the image.

16. The device according to claim 9, characterized in that The reasoning unit is configured to: adjust the size of the region image to a second preset size, and obtain a result mask of the interactive operation based on the adjusted region image.

17. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method of any one of claims 1 to 8.

18. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by at least one processor, enables the at least one processor to perform the method according to any one of claims 1 to 8.

Images