Image segmentation method and device

Abstract

An embodiment of the present specification provides an image segmentation method, including: in response to a click operation made by a user on an image area, determining a first coding map representing a click position; respectively cutting out parts corresponding to the first area from the original image, the first coding map and the existing mask map, and inputting them into a first semantic segmentation model to obtain a first prediction mask and a semantic representation of the first area; wherein the first area is a local area in the image area that contains an existing mask; by comparing the first prediction mask with the existing mask, determining a second area containing a difference mask; respectively cutting out parts corresponding to the second area from the original image, the first coding map, the first prediction mask and the semantic representation, and inputting them into a second semantic segmentation model to obtain a second prediction mask for updating the existing mask.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of machine learning technology, and more particularly, to an image segmentation method and device. Background Art

[0002] Image segmentation technology is an important research direction in the field of computer vision. It refers to the process of dividing an image into several regions with similar properties.

[0003] With the rise of machine learning technology, interactive image segmentation based on deep learning is proposed. Users only need to make a few clicks on the foreground or background, and the model can predict the complete image mask to achieve the target cutout. Figure 1 , the user clicks on the image foreground, with the cross symbol 101 indicating the clicked position, based on which the deep learning model predicts the foreground mask, indicated by the diagonal shadow 102, thereby completing the cutout of the apple area 103. It should be understood that Figure 1 The dashed boxes are only used for auxiliary illustration.

[0004] However, the current interactive segmentation solutions based on deep learning are difficult to meet the needs of practical applications. Therefore, a solution is needed that can meet the needs of practical applications, including improving the computing speed to reduce the user's waiting time, or reducing the requirements for computing resources. Summary of the invention

[0005] One or more embodiments of the present specification describe an image segmentation method and device, which effectively reduces the amount of calculation and achieves rapid cutout by means of local rough segmentation and fine modification while ensuring the accuracy of cutout.

[0006] According to a first aspect, an image segmentation method is provided, comprising: in response to a click operation made by a user on an image area, determining a first coding map representing a click position; respectively cutting out parts corresponding to the first area from the original image, the first coding map and the existing mask map, and inputting them into a first semantic segmentation model to obtain a first prediction mask, and a semantic representation of the first area; wherein the first area is a local area in the image area that contains an existing mask; by comparing the first prediction mask with the existing mask, determining a second area containing a difference mask; respectively cutting out parts corresponding to the second area from the original image, the first coding map, the first prediction mask and the semantic representation, and inputting them into a second semantic segmentation model to obtain a second prediction mask for updating the existing mask.

[0007] In one embodiment, the first coding map includes two coding maps for the foreground and the background; wherein, determining the first coding map representing the click position includes: encoding the pixel blocks corresponding to the click position and other positions in the image area into different code values ​​to obtain a coding map corresponding to the foreground or background targeted by the click operation; and encoding the pixel blocks in the image area into the same code value to obtain another coding map.

[0008] In one embodiment, before respectively cutting out parts corresponding to the first area from the original image, the first coding image and the existing mask image, the method further includes: determining a minimum area in the image area that contains the existing mask and has a preset shape; and enlarging the minimum area by a preset multiple as the first area.

[0009] In one embodiment, before respectively cutting out the parts corresponding to the first area from the original image, the first coding image and the existing mask image, the method further comprises: when the existing mask is absent, determining the image area as the first area.

[0010] In one embodiment, before inputting it into the first semantic model, the method further comprises: extracting a portion corresponding to the first region from a coding matrix representing the historical click positions of the image region.

[0011] In one embodiment, the first semantic segmentation model includes a representation layer and a prediction layer; wherein, inputting it into the first semantic segmentation model to obtain a first prediction mask and a semantic representation of the first region includes: using the representation layer to process the input of the first semantic segmentation model to obtain the semantic representation; using the prediction layer to process the semantic representation to obtain the first prediction mask.

[0012] In one embodiment, determining a second area including a difference mask by comparing the first predicted mask with an existing mask comprises: determining a difference mask including a plurality of scattered masks by comparing the first predicted mask with an existing mask; determining a target scattered mask among the plurality of scattered masks that intersects with the click position and has the largest area; and determining the second area based on the target scattered mask.

[0013] In a specific embodiment, determining the second area based on the target scatter mask includes: determining a minimum area in the first area that contains the target scatter mask and has a preset shape; and magnifying the minimum area by a preset multiple to serve as the second area.

[0014] In one embodiment, the semantic representation includes multiple representation matrices corresponding to multiple channels; wherein, extracting a portion corresponding to a second area from the semantic representation includes: reducing the second area according to a reduction ratio of the representation matrix compared to the original image to obtain a reduced area; respectively extracting portions corresponding to the reduced area from the multiple representation matrices to obtain multiple sub-matrices; and respectively performing pooling or interpolation processing on the multiple sub-matrices according to preset dimensions to obtain a portion of the semantic representation corresponding to the second area.

[0015] In one embodiment, the second semantic segmentation model includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a fusion layer; wherein, inputting it into the second semantic segmentation model to obtain a second prediction mask includes: using the first convolutional layer to process the part intercepted from the semantic representation to obtain a first convolutional representation; using the second convolutional layer to process the other parts corresponding to the second area intercepted to obtain a second convolutional representation; using the fusion layer to fuse the first convolutional representation and the second convolutional representation to obtain a fused representation; using the third convolutional layer to process the fused representation to obtain the second prediction mask.

[0016] In one embodiment, after obtaining the second prediction mask, the method further includes: merging the existing mask and the second prediction mask to obtain a merged mask; and updating the existing mask to the merged mask.

[0017] In one embodiment, the method further includes: displaying an updated existing mask to a user based on the image area.

[0018] According to a second aspect, an image segmentation device is provided, comprising: an encoding unit, configured to determine a first encoding map representing a click position in response to a click operation made by a user on an image area; a first interception unit, configured to intercept a portion corresponding to a first area from the original image, the first encoding map and the existing mask map, respectively, wherein the first area is a local area in the image area containing an existing mask; a first prediction unit, configured to input the intercepted portion corresponding to the first area into a first semantic segmentation model to obtain a first prediction mask and a semantic representation of the first area; a comparison unit, configured to determine a second area containing a difference mask by comparing the first prediction mask with the existing mask; a second interception unit, configured to intercept a portion corresponding to the second area from the original image, the first encoding map, the first prediction mask and the semantic representation, respectively; and a second prediction unit, configured to input the intercepted portion corresponding to the second area into a second semantic segmentation model to obtain a second prediction mask for updating the existing mask.

[0019] In one embodiment, the first coding map includes two coding maps for the foreground and the background; the coding unit is configured to: encode the pixel blocks corresponding to the click position and other positions in the image area into different code values ​​to obtain a coding map corresponding to the foreground or background targeted by the click operation; and encode the pixel blocks in the image area into the same code value to obtain another coding map.

[0020] In one embodiment, the device further includes: a first area determination unit configured to determine a minimum area in the image area that contains the existing mask and has a preset shape; and enlarge the minimum area by a preset multiple to serve as the first area.

[0021] In one embodiment, the apparatus further includes: a first region determining unit configured to determine the image region as the first region when the existing mask is absent.

[0022] In one embodiment, the first interception unit is further configured to intercept a portion corresponding to the first area from a coding matrix representing historical click positions of the image area.

[0023] In one embodiment, the first semantic segmentation model includes a representation layer and a prediction layer; the first prediction unit is specifically configured to: use the representation layer to process the input of the first semantic segmentation model to obtain the semantic representation; use the prediction layer to process the semantic representation to obtain the first prediction mask.

[0024] In one embodiment, the comparison unit is specifically configured as follows: by comparing the first predicted mask and the existing mask, a difference mask is determined, which includes multiple scattered masks; a target scattered mask among the multiple scattered masks is determined, which intersects with the click position and has the largest area; and based on the target scattered mask, the second area is determined.

[0025] In a specific embodiment, the comparison unit is further configured to: determine a minimum area in the first area that contains the target scatter mask and has a preset shape; and magnify the minimum area by a preset multiple to serve as the second area.

[0026] In one embodiment, the semantic representation includes multiple representation matrices corresponding to multiple channels; the second interception unit is specifically configured as follows: according to the reduction ratio of the representation matrix compared to the original image, the second area is reduced to obtain a reduced area; parts corresponding to the reduced area are respectively intercepted from the multiple representation matrices to obtain multiple sub-matrices; according to a preset dimension, the multiple sub-matrices are respectively pooled or interpolated to obtain the part of the semantic representation corresponding to the second area.

[0027] In one embodiment, the second semantic segmentation model includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a fusion layer; the second prediction unit is specifically configured as follows: using the first convolutional layer to process the part intercepted from the semantic representation to obtain a first convolutional representation; using the second convolutional layer to process the other parts corresponding to the second area intercepted to obtain a second convolutional representation; using the fusion layer to fuse the first convolutional representation and the second convolutional representation to obtain a fused representation; using the third convolutional layer to process the fused representation to obtain the second prediction mask.

[0028] In one embodiment, the apparatus further includes: a mask updating unit configured to merge the existing mask and the second prediction mask to obtain a merged mask; and update the existing mask to the merged mask.

[0029] In one embodiment, the device further includes: a mask display unit configured to display an updated existing mask to a user based on the image area.

[0030] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0031] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0032] By using the method and device provided in the embodiments of this specification, the time-consuming full-image prediction is broken down into two steps of fast local coarse segmentation and refinement, thereby significantly reducing the amount of calculation, improving the running speed, and further decoupling the dependence on high-performance hardware environment, fully improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A schematic diagram showing changes in the interface of image segmentation interaction according to an example;

[0035] Figure 2 A schematic diagram showing an implementation architecture of the image segmentation solution disclosed in the embodiments of this specification;

[0036] Figure 3A schematic diagram showing the flow of the method for image segmentation disclosed in the embodiments of this specification;

[0037] Figure 4 A schematic diagram showing changes in the interface of image segmentation interaction disclosed in the embodiments of this specification;

[0038] Figure 5 A schematic diagram showing the model structure of the second semantic segmentation model disclosed in the embodiments of this specification is shown;

[0039] Figure 6 A schematic diagram showing a mask update display disclosed in an embodiment of this specification;

[0040] Figure 7 A schematic diagram of the structure of an image segmentation device disclosed in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0041] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0042] At present, the implementation of interactive image segmentation based on deep learning usually adopts the following paradigm: for a given original image, the user is required to make several clicks on the foreground and background areas. After each click, the model takes the complete image together with the click encoding and mask information corresponding to the complete image area as input to re-predict the mask.

[0043] The inventors found that since the segmentation target and the area to be modified each time may account for a very small proportion of the entire image, re-predicting the entire image area each time causes redundant calculations; in addition, since the entire image is re-predicted each time, there is a risk that the previously marked details will be destroyed.

[0044] Based on the above observations and analysis, the inventors proposed an interactive image segmentation solution, which breaks down the time-consuming full-image prediction into two steps of fast local coarse segmentation and further refinement, thereby significantly reducing the amount of calculation, improving the running speed, and further decoupling the dependence on high-performance hardware environment, fully improving the user experience.

[0045] Figure 2 FIG. 2 is a schematic diagram showing an implementation architecture of the image segmentation solution disclosed in the embodiments of this specification. Figure 2As shown, the implementation of the scheme can be divided into three stages: input preparation, rough segmentation and refinement. In the input preparation stage, the following input items are prepared: original image, encoding map of the user click position, existing mask map, etc.; then, in the rough segmentation stage, each input item is partially intercepted from the first area containing the existing mask, and it is input into the first semantic segmentation model to obtain the first predicted mask and the semantic representation of the first area; then, in the refinement stage, by comparing the first predicted mask with the existing mask, the second area containing the difference mask is determined, and the prepared input items and the items obtained by the rough segmentation are intercepted corresponding to the second area, and the interception results are input into the second semantic segmentation model to obtain the second predicted mask for updating the existing mask. In this way, there is no need to calculate the entire image information, but to calculate the determined local modification area, thereby effectively reducing the amount of calculation and avoiding damage to the marked mask area.

[0046] The implementation steps of the above solution proposed by the inventor are introduced below in conjunction with specific embodiments. Figure 3 The following is a flow chart of the method for image segmentation disclosed in the embodiment of this specification. The execution subject of the method is any device, platform and equipment cluster with computing and processing capabilities. Figure 3 As shown, the method comprises the following steps:

[0047] Step S310, in response to a click operation made by a user on an image area, determine a first coding map representing a click position; step S320, respectively cut out parts corresponding to the first area from the original image, the first coding map and the existing mask map, wherein the first area is a local area in the image area containing an existing mask; step S330, input the cut out part corresponding to the first area into a first semantic segmentation model to obtain a first prediction mask and a semantic representation of the first area; step S340, by comparing the first prediction mask with the existing mask, determine a second area containing a difference mask; step S350, respectively cut out parts corresponding to the second area from the original image, the first coding map, the first prediction mask and the semantic representation; step S360, input the cut out part corresponding to the second area into a second semantic segmentation model to obtain a second prediction mask for updating the existing mask.

[0048] The above steps are expanded as follows:

[0049] First, in step S310, in response to a click operation made by the user on the image area, a first coding map representing the click position is determined. It should be noted that the image area refers to the area corresponding to the original image (usually provided by the user); the click operation is directed to the foreground or background of the image, the foreground refers to the target area of ​​interest to the user in the image, and the background refers to other areas outside the foreground. In addition, the above-mentioned click position usually includes several pixel blocks covered by the interface cursor when clicking, or a pixel area with a predetermined radius (such as 5 pixels) can be expanded outward with any pixel block among the several pixel blocks as the center as the click position.

[0050] In one implementation scenario, the system supports the user's independent choice to click on the foreground or background, for example, supporting the left mouse button to trigger the foreground click, and the right mouse button to trigger the background click. At this time, the first coding map may include two coding maps, one for the foreground and the other for the background. Further, in one embodiment, the pixel blocks corresponding to the click position and other positions in the image area are encoded into different code values ​​(for example, 1 and 0) to obtain a coding map corresponding to the foreground or background targeted by the click operation; and the pixel blocks in the image area are encoded into the same code value (for example, 0) to obtain another coding map. Exemplarily, as Figure 4 As shown, in response to a click operation on the foreground of the image, a corresponding click area 401 is determined, thereby determining a click information coding map 402 for the foreground and a click information coding map 403 for the background.

[0051] In another implementation scenario, the system only supports users to click on the foreground, and accordingly, the first information encoding diagram is for the image foreground. In yet another implementation scenario, the system only supports users to click on the background, and accordingly, the first information encoding diagram is for the image background.

[0052] From the above, the first coding map representing the click position can be determined. Then, in step S320, portions corresponding to the first area are respectively intercepted from the original image, the first coding map and the existing mask map, wherein the first area is a local area in the image area that contains an existing mask. It should be noted that a mask is a pattern or object used to cover an area of ​​interest, such as pure red or pure green. The mask may be a foreground mask (a mask used to cover the foreground) or a background mask (a mask used to cover the background). For the sake of clarity, in the embodiments of this specification, the mask is mainly described as a foreground mask. Accordingly, the above-mentioned existing mask corresponds to a part of the foreground area that has been determined. In one scenario, before this click operation, there is a historical click operation for the image area, and accordingly, there is a mask predicted based on this historical click operation; in another scenario, before this operation, there is a mask manually marked using a stroking tool.

[0053] The existing mask map is a position coding map of the above-mentioned existing mask in the image area; illustratively, the position coding value corresponding to the pixel block where the existing mask is located is a first value (such as 1), and the position coding values ​​corresponding to other pixel blocks are second values ​​(such as 0). On the other hand, for the above-mentioned first area, it at least includes the area corresponding to the existing mask. Usually, the first area is the area obtained by expanding the existing mask area. In one embodiment, the minimum area area in the image area that contains the existing mask and has a preset shape (such as a rectangle, a circle, etc.) can be first determined, and then the minimum area area is enlarged by a preset multiple as the first area. Exemplarily, the minimum circumscribed rectangle of the existing mask is first determined, and then the minimum circumscribed rectangle is enlarged by 1.4 times as the first area. In addition, in one embodiment, if the existing mask is not present, the complete image area can be used as the first area.

[0054] Further, based on the determined first area, the original image, the first coding image and the existing mask image can be intercepted in the corresponding area to obtain a first intercepted portion. It should be understood that the intercepted objects are usually organized in a matrix form, and accordingly, the first intercepted portion includes sub-matrices intercepted from each object matrix.

[0055] In one embodiment, the original image corresponds to a pixel matrix of RGB three channels, and in this case, three pixel sub-matrices can be obtained by intercepting the original image. In another embodiment, the original image is a black and white image, which corresponds to a pixel matrix of a single channel, and in this case, one pixel sub-matrix can be intercepted.

[0056] In one embodiment, a coding sub-matrix may be obtained by cutting out a portion corresponding to the first region from the first coding image.

[0057] In one embodiment, the existing mask image is intercepted corresponding to the first region to obtain a mask sub-matrix.

[0058] On the other hand, in one embodiment, before the current click operation, there is a historical click operation on the above-mentioned image area. Accordingly, the corresponding first area can also be intercepted from the coding matrix representing the historical click position of the historical click operation to obtain a historical coding sub-matrix.

[0059] From the above, we can obtain multiple sub-matrices corresponding to the first region, denoted as Wherein H1 and W1 represent the height and width of the submatrix respectively, and n represents the number of submatrices. For example, assuming that the RGB three-channel image of the original image, the foreground click coding map, the background click coding map and the existing mask map are intercepted, 6 submatrices can be obtained, that is, n=6.

[0060] Furthermore, in one embodiment, multiple sub-matrices may be sampled respectively to obtain a sampling matrix of a preset size, denoted as It should be understood that the sampling size can be pre-set to a smaller value to further reduce the amount of model calculation during coarse segmentation. For example, the sampling size can be H ′ 1*W1 ′ =128*128.

[0061] From the above, the portion corresponding to the first area can be intercepted. Then, in step S330, the portion corresponding to the first area (such as X1 or X2) is intercepted. ′ 1), input a first semantic segmentation model to obtain a first prediction mask and a semantic representation of the first region. It should be understood that semantic segmentation is a classification at the pixel level, and pixels belonging to the same category are classified into one category, for example, the foreground is classified into one category and the background is classified into another category. Exemplarily, corresponding to the above-mentioned existing mask, the first prediction mask here corresponds to the prediction area for the foreground of the image.

[0062] In one embodiment, the first semantic segmentation model includes a representation layer and a prediction layer. Accordingly, this step includes: using the representation layer to process the input of the first semantic segmentation model to obtain a semantic representation; using the prediction layer to process the semantic representation to obtain a first prediction mask. It can be understood that the semantic representation contains deep image features in the first region. Exemplarily, the above-mentioned representation layer can be implemented as multiple convolutional layers, and the prediction layer can be implemented as a fully connected layer. On the other hand, in one embodiment, the first semantic segmentation model can adopt a SegFormer model, an HRNet model, or other existing image semantic segmentation models.

[0063] From the above, a rough segmentation result can be obtained, that is, a first predicted mask located in the first region, and a semantic representation of the first region can be obtained. Then, in step S340, by comparing the first predicted mask with the existing mask, a second region containing a difference mask is determined. Specifically, the difference mask is first determined by comparison, and then the second region is determined based on the difference mask. It should be understood that the difference mask here may include the mask that is extra and the mask that is missing from the existing mask compared to the first predicted mask.

[0064] In one embodiment, the area where the difference mask is located can be directly determined as the second area. In another embodiment, the difference mask includes multiple scattered masks, and accordingly, the target scattered mask that intersects with the above-mentioned click position and has the largest area among the multiple scattered masks can be determined, so as to determine the above-mentioned second area based on this target scattered mask. Further, in a specific embodiment, the area where the target scattered mask is located can be determined as the second area. In another specific embodiment, the minimum area area in the above-mentioned first area that contains the target scattered mask and has a preset shape (such as a circle or rectangle, etc.) can be determined, and then the minimum area area is enlarged by a preset multiple (such as 1.2 or 1.5, etc.) as the second area. In this way, the determined second area can be closer to the user's actual intention and effectively reduce the subsequent calculation amount.

[0065] In the above, the second region containing the difference mask can be determined. Then, in step S350, the portion corresponding to the second region is respectively intercepted from the original image, the first coding map, the first prediction mask and the semantic representation. It should be understood that the first three are usually of the same size as the original image region, so the interception can be performed directly according to the second region. Furthermore, the intercepted portions corresponding to the first three can also be upsampled to enrich the detail information.

[0066] The semantic representation usually includes multiple representation matrices corresponding to multiple channels, wherein the size of each representation matrix (or the dimension, number of rows and columns of the matrix) is pre-set, and may be the same as or different from the original image area. Under the same circumstances, the corresponding second area can be directly intercepted; under different interceptions, the second area can be scaled according to the scaling ratio of the representation matrix compared to the original image to obtain a scaled matrix, and then the parts corresponding to the scaled area are respectively intercepted from the multiple representation matrices to obtain multiple representation sub-matrices, so as to determine the intercepted part of the second area corresponding to the semantic representation based on the multiple representation sub-matrices. Furthermore, in one embodiment, the multiple representation sub-matrices can be directly determined as the intercepted part of the second area corresponding to the semantic representation. In another embodiment, each representation sub-matrix can also be pooled or interpolated according to the preset dimension, and the processed result can be used as the intercepted part of the second area corresponding to the semantic representation.

[0067] From the above, the clipped portion corresponding to the second region can be obtained. Thus, in step S360, the clipped portion corresponding to the second region is input into the second semantic segmentation model to obtain a second prediction mask.

[0068] It should be understood that the portion corresponding to the second area obtained by interception already includes semantic representations with deep semantic information and other image features with rich detail information. Therefore, the second semantic segmentation model in this step can adopt a simpler and lighter model than the above-mentioned first semantic segmentation model, including fewer model parameters, etc.

[0069] Figure 5 A schematic diagram of the model structure of the second semantic segmentation model disclosed in the embodiment of this specification is shown, such as Figure 5 As shown, the second semantic segmentation model includes a first convolution layer 501, a second convolution layer 502, a third convolution layer 503 and a fusion layer 504. Based on this, this step may include: using the first convolution layer 501 to process the part intercepted from the semantic representation ( Figure 5 , and obtain the first convolutional representation; and use the second convolutional layer 502 to process the other parts ( Figure 5 The first convolutional representation and the second convolutional representation are fused by the fusion layer 504 to obtain a fused representation; the fused representation is processed by the third convolutional layer 503 to obtain a second prediction mask. It should be noted that the number of convolutional layers included in the first convolutional layer can be set according to actual needs, such as 3, and the second convolutional layer is similar to the third convolutional layer. In one embodiment, there are multiple ways to perform fusion processing in the fusion layer 504, such as addition, averaging, or splicing, etc.

[0070] It should be understood that the second semantic segmentation model can also be implemented as other model structures.

[0071] From the above, a refined result can be obtained, including the second predicted mask. According to another embodiment, after obtaining the second predicted mask, the method further includes: merging the existing mask and the second predicted mask to obtain a merged mask; updating the existing mask to the merged mask. Further, it also includes: displaying the updated existing mask to the user based on the image area, which can be seen in Figure 6 It should be understood that visually, it is the addition or removal of the existing mask before the operation. Generally, if the current click operation is for the foreground, you will see the added mask, and if the current click operation is for the background, you will see the existing mask reduced.

[0072] In summary, by adopting the image segmentation method disclosed in the embodiments of this specification, the time-consuming full-image prediction is broken down into two steps of fast local coarse segmentation and refinement, thereby significantly reducing the amount of calculation, improving the running speed, and further decoupling the dependence on high-performance hardware environment, fully improving the user experience.

[0073] Corresponding to the above-mentioned image segmentation method, the embodiment of this specification also discloses an image segmentation device. It should be noted that this device can be integrated in any platform, client or terminal device with computing and processing capabilities. For example, this device can be implemented as mapping software, and accordingly, users can download, install and use it through the terminal; for another example, this device can be implemented as a plug-in in an e-commerce platform or e-commerce client, and accordingly, merchants can use this plug-in to process product images, so as to select the processed images for publication when publishing products. Figure 7 A schematic diagram of the structure of the image segmentation device disclosed in the embodiment of this specification is shown as follows: Figure 7 As shown, the device 700 includes:

[0074] The encoding unit 710 is configured to determine a first encoding map representing a click position in response to a click operation made by a user on an image area; the first interception unit 720 is configured to intercept a portion corresponding to the first area from the original image, the first encoding map and the existing mask map, respectively, wherein the first area is a local area in the image area containing an existing mask; the first prediction unit 730 is configured to input the intercepted portion corresponding to the first area into a first semantic segmentation model to obtain a first prediction mask and a semantic representation of the first area; the comparison unit 740 is configured to determine a second area containing a difference mask by comparing the first prediction mask with the existing mask; the second interception unit 750 is configured to intercept a portion corresponding to the second area from the original image, the first encoding map, the first prediction mask and the semantic representation, respectively; the second prediction unit 760 is configured to input the intercepted portion corresponding to the second area into a second semantic segmentation model to obtain a second prediction mask for updating the existing mask.

[0075] In one embodiment, the first coding map includes two coding maps for the foreground and the background; the coding unit 710 is specifically configured to: encode the pixel blocks corresponding to the click position and other positions in the image area into different code values ​​to obtain a coding map corresponding to the foreground or background targeted by the click operation; and encode the pixel blocks in the image area into the same code value to obtain another coding map.

[0076] In one embodiment, the device 700 further includes: a first area determination unit 770, configured to determine a minimum area in the image area that contains the existing mask and has a preset shape; and enlarge the minimum area by a preset multiple to serve as the first area.

[0077] In one embodiment, the apparatus further includes: a first region determining unit 770 configured to determine the image region as the first region when the existing mask is absent.

[0078] In one embodiment, the first interception unit 720 is further configured to intercept a portion corresponding to the first area from a coding matrix representing the historical click positions of the image area.

[0079] In one embodiment, the first semantic segmentation model includes a representation layer and a prediction layer; the first prediction unit 730 is specifically configured to: use the representation layer to process the input of the first semantic segmentation model to obtain the semantic representation; use the prediction layer to process the semantic representation to obtain the first prediction mask.

[0080] In one embodiment, the comparison unit 740 is specifically configured as follows: by comparing the first predicted mask and the existing mask, a difference mask is determined, which includes multiple scattered masks; a target scattered mask among the multiple scattered masks is determined, which intersects with the click position and has the largest area; and based on the target scattered mask, the second area is determined.

[0081] In a specific embodiment, the comparison unit 740 is further configured to: determine a minimum area in the first area that contains the target scatter mask and has a preset shape; and magnify the minimum area by a preset multiple to serve as the second area.

[0082] In one embodiment, the semantic representation includes multiple representation matrices corresponding to multiple channels; the second interception unit 750 is specifically configured as follows: according to the reduction ratio of the representation matrix compared to the original image, the second area is reduced to obtain a reduced area; parts corresponding to the reduced area are respectively intercepted from the multiple representation matrices to obtain multiple sub-matrices; according to a preset dimension, the multiple sub-matrices are respectively pooled or interpolated to obtain the part of the semantic representation corresponding to the second area.

[0083] In one embodiment, the second semantic segmentation model includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a fusion layer; the second prediction unit 760 is specifically configured as follows: using the first convolutional layer to process the part intercepted from the semantic representation to obtain a first convolutional representation; using the second convolutional layer to process the other parts corresponding to the second area intercepted to obtain a second convolutional representation; using the fusion layer to fuse the first convolutional representation and the second convolutional representation to obtain a fused representation; using the third convolutional layer to process the fused representation to obtain the second prediction mask.

[0084] In one embodiment, the apparatus 700 further includes: a mask updating unit 780 configured to merge the existing mask and the second prediction mask to obtain a merged mask; and update the existing mask to the merged mask.

[0085] In one embodiment, the apparatus further includes: a mask display unit 790 configured to display an updated existing mask to a user based on the image region.

[0086] In summary, by adopting the image segmentation device disclosed in the embodiments of this specification, the time-consuming full-image prediction is broken down into two steps of fast local coarse segmentation and refinement, thereby significantly reducing the amount of calculation, improving the running speed, and further decoupling the dependence on high-performance hardware environment, fully improving the user experience.

[0087] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 3 The method described.

[0088] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 3 The method described.

[0089] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0090] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image segmentation method, comprising: In response to a click operation made by a user on an image area, determining a first coding map representing a click position; Cut out parts corresponding to the first region from the original image, the first coding image and the existing mask image respectively, and input them into a first semantic segmentation model to obtain a first predicted mask and a semantic representation of the first region; wherein the first region is a local region in the image region that contains the existing mask; Determining a second region including a difference mask by comparing the first predicted mask with an existing mask; Parts corresponding to the second area are respectively intercepted from the original image, the first coding image, the first prediction mask and the semantic representation, and are input into a second semantic segmentation model to obtain a second prediction mask for updating the existing mask.

2. The method according to claim 1, wherein: The first coding map includes two coding maps for the foreground and the background; wherein determining the first coding map representing the click position includes: The pixel blocks corresponding to the click position and other positions in the image area are encoded into different code values ​​to obtain a coding map corresponding to the foreground or background targeted by the click operation; and the pixel blocks in the image area are encoded into the same code value to obtain another coding map.

3. The method according to claim 1, wherein: Before respectively cutting out the portion corresponding to the first area from the original image, the first coding image and the existing mask image, the method further includes: Determine a minimum area in the image area that contains the existing mask and has a preset shape; The minimum area region is enlarged by a preset multiple to serve as the first area.

4. The method according to claim 1, wherein: Before respectively cutting out the portion corresponding to the first area from the original image, the first coding image and the existing mask image, the method further includes: When the existing mask is absent, the image area is determined as the first area.

5. The method according to claim 1, wherein: Before inputting it into the first semantic model, the method further comprises: A portion corresponding to the first area is cut out from a coding matrix representing historical click positions of the image area.

6. The method according to claim 1, wherein: The first semantic segmentation model includes a representation layer and a prediction layer; wherein, the first prediction mask and the semantic representation of the first region are obtained by inputting the first semantic segmentation model into the first semantic segmentation model, including: Processing the input of the first semantic segmentation model using the representation layer to obtain the semantic representation; The semantic representation is processed using the prediction layer to obtain the first prediction mask.

7. The method according to claim 1, wherein: By comparing the first predicted mask with the existing mask, determining a second region containing a difference mask includes: By comparing the first predicted mask with the existing mask, a difference mask is determined, which includes a plurality of scattered masks; Determine a target scatter mask among the multiple scatter masks that intersects with the click position and has the largest area; Based on the target scatter mask, the second region is determined.

8. The method according to claim 7, wherein: Determining the second area based on the target scatter mask includes: Determine a minimum area in the first area that contains the target scatter mask and has a preset shape; The minimum area region is enlarged by a preset multiple to serve as the second area.

9. The method according to claim 1, wherein: The semantic representation includes a plurality of representation matrices corresponding to a plurality of channels; wherein extracting a portion corresponding to the second area from the semantic representation includes: According to the reduction ratio of the representation matrix compared with the original image, the second region is reduced to obtain a reduced region; Cutting out parts corresponding to the reduced area from the multiple representation matrices respectively to obtain multiple sub-matrices; According to a preset dimension, pooling processing or interpolation processing is performed on the multiple sub-matrices respectively to obtain a part of the semantic representation corresponding to the second area.

10. The method according to claim 1, wherein: The second semantic segmentation model includes a first convolution layer, a second convolution layer, a third convolution layer and a fusion layer; wherein, inputting it into the second semantic segmentation model to obtain a second prediction mask includes: Processing the portion intercepted from the semantic representation using the first convolutional layer to obtain a first convolutional representation; Using the second convolutional layer to process the other portion of the intercepted image corresponding to the second region to obtain a second convolutional representation; Using the fusion layer to fuse the first convolutional representation and the second convolutional representation to obtain a fused representation; The fused representation is processed using the third convolutional layer to obtain the second prediction mask.

11. The method according to claim 1, wherein: After obtaining the second prediction mask, the method further includes: Merging the existing mask and the second predicted mask to obtain a merged mask; The existing mask is updated to the merged mask.

12. An image segmentation device, comprising: An encoding unit, configured to determine a first encoding map representing a click position in response to a click operation made by a user on an image region; A first interception unit is configured to intercept a portion corresponding to a first area from the original image, the first coding image, and the existing mask image, respectively, wherein the first area is a local area in the image area that includes the existing mask; A first prediction unit is configured to input the intercepted portion corresponding to the first region into a first semantic segmentation model to obtain a first prediction mask and a semantic representation of the first region; a comparing unit, configured to determine a second region including a difference mask by comparing the first predicted mask with an existing mask; A second interception unit is configured to intercept a portion corresponding to the second area from the original image, the first coding map, the first prediction mask and the semantic representation respectively; The second prediction unit is configured to input the intercepted portion corresponding to the second area into the second semantic segmentation model to obtain a second prediction mask for updating the existing mask.

Images