Interactive image segmentation method, system, device and storage medium

Abstract

Embodiments of the present application provide an interactive image segmentation method, system, device, and storage medium. In this interactive image segmentation method, after obtaining a guidance map based on instance segmentation axes drawn by a user on an image, a neural network-based approach is used to predict the mask of the instance selected by the instance segmentation axes, guided by the guidance map. In this approach, the neural network model can effectively extract deep semantic features from the image, which helps improve the accuracy of instance segmentation results.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an interactive image segmentation method, system, device and storage medium. Background Art

[0002] Instance segmentation is widely used in image-related applications. Instance segmentation involves segmenting a specific instance from an image and outputting a mask representing that instance. For example, we can segment the partial image corresponding to a single zebra from a group of zebras in an image and output a mask representing that zebra's partial image.

[0003] However, the segmentation results of existing instance segmentation methods are less reliable. Therefore, a new solution is needed. Summary of the Invention

[0004] Various aspects of the present application provide an interactive image segmentation method, system, device, and storage medium for improving the reliability of a predicted mask of a first instance of an image.

[0005] An embodiment of the present application provides an interactive image segmentation system, comprising: a terminal device and a server; wherein the terminal device is configured to: display a first image and a first mask; in response to a user's operation of drawing an instance segmentation guide axis on the first image, determine a first guidance map corresponding to the instance segmentation guide axis, and send the first image and the first guidance map to the server; and receive a second mask corresponding to the instance segmentation guide axis returned by the server, and highlight the local image area corresponding to the second mask on the first image; the server is configured to: receive the first image and the first guidance map sent by the terminal device; input the first image, the first mask and the first guidance map into a neural network model, the neural network model being trained based on guidance map samples generated by simulating the user's segmentation guidance interactive operation; in the neural network model, extract semantic features of the first image and the first guidance map; predict the second mask corresponding to the instance segmentation guide axis based on the semantic features, and return the predicted second mask to the terminal device.

[0006] An embodiment of the present application also provides an interactive image segmentation method, including: displaying a first image and a first mask; in response to a user's operation of drawing an instance segmentation guide axis on the first image, determining a first guidance map corresponding to the instance segmentation guide axis; inputting the first image, the first mask, and the first guidance map into a neural network model, the neural network model being trained based on guidance map samples generated by simulating the user's segmentation guidance interactive operation; in the neural network model, extracting semantic features of the first image and the first guidance map, and predicting a second mask corresponding to the instance segmentation guide axis based on the semantic features; and highlighting a local image area corresponding to the second mask on the first image.

[0007] An embodiment of the present application also provides an interactive image segmentation method, suitable for a terminal device, including: displaying a first image and a first mask; responding to a user's operation of drawing an instance segmentation guide axis on the first image, determining a first guidance map corresponding to the instance segmentation guide axis; sending the first image, the first mask, and the first guidance map to a server, so that the server uses a neural network model to predict a second mask corresponding to the instance segmentation guide axis; receiving the second mask returned by the server, and highlighting the local image area corresponding to the second mask on the first image.

[0008] An embodiment of the present application also provides an interactive image segmentation method, which is applicable to a server, including: receiving a first image, a first mask and a first guidance map sent by a terminal device, wherein the first guidance map is generated based on an instance segmentation guide axis drawn by a user on the first image; inputting the first image, the first mask and the first guidance map into a neural network model, and the neural network model is trained based on guidance map samples generated by simulating the user's segmentation guidance interactive operation; in the neural network model, extracting semantic features of the first image and the first guidance map, and predicting a second mask corresponding to the instance segmentation guide axis based on the semantic features; and returning the predicted second mask to the terminal device for display.

[0009] An embodiment of the present application also provides a terminal device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the interactive image segmentation method provided in an embodiment of the present application.

[0010] The embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the interactive image segmentation method provided in the embodiment of the present application.

[0011] In this embodiment, after obtaining a guidance map based on the instance segmentation axis drawn by the user on the image, a neural network-based approach is used to predict the mask of the instance selected by the instance segmentation axis under the guidance of the guidance map. In this way, the neural network model can effectively extract deep semantic features in the image, which is conducive to improving the accuracy of instance segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 A schematic structural diagram of an interactive image segmentation system provided by an exemplary embodiment of the present application;

[0014] Figure 2 A schematic diagram of a guide map provided for an exemplary embodiment of the present application;

[0015] Figure 3 A schematic diagram of the training process of a neural network model provided for an exemplary embodiment of the present application;

[0016] Figure 4 A schematic diagram of an interactive image segmentation method provided by an exemplary embodiment of the present application;

[0017] Figure 5 A schematic diagram of an interactive image segmentation method provided by another exemplary embodiment of the present application;

[0018] Figure 6 A schematic diagram of an interactive image segmentation method provided by yet another exemplary embodiment of the present application;

[0019] Figure 7 A schematic diagram of the structure of a terminal device provided by an exemplary embodiment of the present application;

[0020] Figure 8 A schematic diagram of the structure of a server provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] Instance segmentation has a wide range of applications in image-related fields, such as image editing based on instance segmentation, image enhancement based on instance segmentation, anomaly detection based on images and instance segmentation, and medical diagnosis based on medical imaging and instance segmentation.

[0023] Existing instance segmentation methods are usually implemented based on lasso-type cutout tools and "magic wand-type" cutout tools.

[0024] Lasso-type cutout tools require users to manually draw the edges of an instance, which then form the selection area. However, using lasso-type cutout tools for instances with complex boundaries requires longer interaction times and is more dependent on the level of precision of the operation.

[0025] Magic wand-type cutout tools require users to click a pixel on an image. The tool then calculates the pixels extending outward from that pixel based on its color, transparency, and other information, adding pixels of the same color or transparency to the selected area. However, when the texture of an instance is complex, these magic wand-type cutout tools may not produce accurate segmentation results.

[0026] In response to the technical problems existing in the above technical solutions, an embodiment of the present application provides an interactive image segmentation system, which will be described below with reference to the accompanying drawings.

[0027] Figure 1 A schematic diagram of the structure of an interactive image segmentation system provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the interactive image segmentation system 100 includes: a terminal device 101 and a server 102.

[0028] In the interactive image segmentation system 100, the terminal device 101 can provide the user with instance segmentation operations for images and provide the user with a better instance segmentation result through at least one round of interactive operations with the user. Typically, the terminal device 101 can be implemented as a user's mobile phone, computer, tablet computer, etc., and this embodiment does not impose any limitation.

[0029] In the interactive image segmentation system 100, the server 102 can provide data and computing support for the terminal device 101. The server 102 can be implemented as a conventional server, a cloud server, a cloud host, a virtual center, or other server device, and this embodiment does not limit this. The server device primarily comprises a processor, a hard disk, memory, a system bus, and the like, similar to a general computer architecture, and will not be further described.

[0030] In this embodiment, the terminal device 101 may display an image to be segmented and a current mask. For ease of description and distinction, the image to be segmented is described below as a first image, and the current mask is described below as a first mask. The user may draw an instance segmentation guide axis on the first image based on instance segmentation requirements. The instance segmentation guide axis is commonly known as the "skeleton" of the instance. In response to the user's operation of drawing the instance segmentation guide axis on the first image, the terminal device 101 may determine a guidance map corresponding to the instance segmentation guide axis.

[0031] The current mask may be the predicted mask returned by the server 102 during the previous round of interaction. When the current round of interaction is the first round of interaction, the value in the current mask may be set to a null value.

[0032] The guide image is another image drawn by the user, which has the same resolution as the first image to be segmented and is used to indicate instance segmentation. A typical guide image is Figure 2 To facilitate computer interpretation, the value of each pixel in the guidance map indicates one of three situations: (a) the pixel belongs to the instance; (b) the pixel does not belong to the instance; (c) it is uncertain whether the pixel belongs to the instance. In other words, the guidance map can be considered a "ternary image" with the same resolution as the first image.

[0033] For ease of description and distinction, the guidance map generated based on the instance segmentation guidance axis drawn by the user is described as a first guidance map. After acquiring the first guidance map, the terminal device 101 may send the first image, the first mask, and the first guidance map to the server 102.

[0034] After receiving the first image, first mask, and first guidance map from terminal device 101, server 102 may input the first image, first mask, and first guidance map into a neural network model. The neural network model is trained based on the automatically generated guidance map samples. The training process of the neural network model will be described in subsequent embodiments and is not detailed here.

[0035] In some embodiments, the server 102 includes multiple server devices, such as multiple ECS (Elastic Compute Service, cloud servers). In this scenario, when the user needs multiple interactions to complete instance segmentation, the multiple rounds of interaction between the terminal device 101 and the server 102 can be implemented as interactions between the terminal device and different ECSs. Based on this, when the terminal device 101 sends the instance segmentation request to the server 102, the image to be segmented, the current mask, and the guide map generated by this interaction are sent to the server of this interaction. This is beneficial for the server of this interaction to quickly obtain the data required for this instance segmentation when the server of this interaction is different from the server of the previous interaction.

[0036] When the terminal device 101 interacts with only one server, it can be assumed that the server already has the image to be segmented sent by the terminal device 101 during the previous interaction, as well as the mask predicted by the server for the terminal device during the previous interaction. In this case, when the terminal device 101 sends the current instance segmentation request to the server 102, it can simply send the guidance map generated by the current interaction to the server, thereby improving interaction efficiency.

[0037] In the neural network model, semantic features of the first image and the first guide map can be extracted. After the semantic features are extracted, a mask corresponding to the instance segmentation guide axis can be predicted based on the semantic features.

[0038] Semantic features extracted from images are deep features. They encompass not only low-level visual features like color, texture, and shape, but also semantic representations of the objects in the image and their relationships, as well as semantic descriptions derived from high-level reasoning about objects and scenes. Mask prediction based on semantic features fully leverages diverse image information to achieve highly accurate instance segmentation results.

[0039] For ease of description and distinction, the mask predicted by the neural network model based on the first image, the first mask, and the first guide map is described as the second mask. After the server 102 obtains the second mask predicted and output by the neural network model, it can return the second mask to the terminal device 101.

[0040] After receiving the second mask returned by the server 102, the terminal device 101 may highlight the local image area corresponding to the second mask on the first image. The local image area is the image area selected by the instance segmentation guide axis drawn by the user.

[0041] In some cases, the local image area is the entire instance to which the instance segmentation guide axis drawn by the user belongs. In this case, the user does not need to draw the instance segmentation guide axis again on the first image. In other cases, the local image area is a portion of the instance to which the instance segmentation guide axis drawn by the user belongs. In this case, the user can draw the instance segmentation guide axis again on the first image based on the actual instance segmentation requirements. After the user draws the instance segmentation guide axis again, the terminal device 101 can repeat the aforementioned embodiment based on the new instance segmentation guide axis, and no further details will be given.

[0042] In the interactive image segmentation system 100, the communication between the terminal device 101 and the server 102 can be carried out by wired communication or wireless communication. Among them, the wireless communication method includes short-range communication methods such as Bluetooth, ZigBee, infrared, WiFi (WIreless-Fidelity, wireless fidelity technology), and also includes long-range wireless communication methods such as LORA, and can also include wireless communication methods based on mobile networks. Among them, when connected through mobile network communication, the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax, etc., and this embodiment does not impose any restrictions.

[0043] In this embodiment, after obtaining a guidance map based on the instance segmentation axis drawn by the user on the image, a neural network-based approach is used to predict the mask of the instance selected by the instance segmentation axis under the guidance of the guidance map. In this way, the neural network model can effectively extract deep semantic features in the image, which is conducive to improving the accuracy of instance segmentation results.

[0044] The following will be combined Figure 3 And some optional embodiments are provided to illustrate the implementation methods of training neural network models.

[0045] Among them, the goal of training the neural network model is to enable the neural network model to learn based on the input picture I, the guidance map G provided by the user's current round of interaction, and the prediction mask M' generated by the previous round of user interaction, and output the prediction mask M* generated by this round of interaction.

[0046] In some optional embodiments, the training data required for training the neural network model includes: a paired sample image I and a true mask M of any instance on the sample image. For ease of distinction, any instance on the sample image is described as an instance sample below.

[0047] After obtaining the sample image I and the real mask M of the instance sample in the sample image I, multiple rounds of user interaction operations on the sample image can be simulated. When simulating each round of interaction operations, the real mask M of the instance sample and the predicted mask M input in each round of interaction operations can be used. i `, simulating the user's operation of drawing the instance segmentation guide axis on the sample image I. Among them, simulating a round of interactive operation, a guide map sample G can be obtained i , simulating multiple rounds of interactive operations, multiple guide graph samples {G i}, i = 1, 2, 3…N, N represents the number of simulations, and N is a positive integer.

[0048] Get the corresponding guidance graph samples G for multiple rounds of interactive operations i After that, we can use the sample image I, the real mask M, and multiple guide map samples G i And the prediction mask M of each round of interactive operation input i `, and obtain the training data corresponding to each round of interactive operations. For the i-th round of interactive operations, the corresponding training data includes: a set of data consisting of sample images, real masks, guide map samples generated by the i-th round of interactive operations, and predicted masks input by the i-th round of interactive operations, that is, (I,G i ,M i `,M).

[0049] Optionally, in the above-mentioned process of simulating the user's interactive operation, for the i-th round of simulation operation, if i=1, the prediction mask M of the i-th round of interactive operation input i `, can be initialized to a null value. If i ≥ 2, then the prediction mask M of the i-th round interactive operation input i `, which can be the prediction mask M output by the i-1th layer i-2 *.

[0050] Optionally, taking the interactive operation of the i-th round in the multi-round simulation operation as an example, when simulating the interactive operation of the i-th round, the sample image and the guidance graph G generated by the interactive operation of the i-th round are combined. i And the predicted mask M of the i-th round interactive operation input i `, as the input parameter of the set mask prediction algorithm f(), calculate the predicted mask M output by the interactive operation of the i-th round i *.

[0051] Next, determine the prediction mask M of the interactive operation output of round i i * Whether the difference between the real mask M is greater than the first threshold; if the difference is greater than the first threshold and the number of interactions i is less than the second threshold, the predicted mask M output by the i-th round of interaction operation can be i *The prediction mask as the input of the interactive operation in round i+1, i.e. M i+1 `=M i *.

[0052] On the other hand, if the difference between the predicted mask and the true mask outputted by the interaction operation in round i is less than or equal to the first threshold, or the number of interactions is greater than or equal to the second threshold, the simulation operation may be terminated. The second threshold may be N, i.e., the total number of interactions.

[0053] Next, the neural network model can be trained according to the training data corresponding to each of the multiple rounds of interactive operations, so as to optimize the parameters of the neural network model under the supervision of the true mask M.

[0054] Taking a set of training data generated by the i-th round of interactive operation as an example, when training the neural network model, the sample image I and the guide map sample G generated by the i-th round of interactive operation can be i And the predicted mask M of the i-th round interactive operation input i `As the input data of the neural network model, the neural network model is trained to calculate the prediction mask M based on the input data i *, and make the prediction mask M i *The ability to approach the true mask M. That is, according to each group (I,G i ,M i `,M), take M as the true value, train the neural network model to calculate M i *=f(I,G i ,M i `), and make M i *Close to M's capabilities.

[0055] In the above embodiments, the real mask M according to the example sample and the predicted mask M input in each round of interactive operation are recorded. i `, a technical solution for simulating the operation of a user drawing an instance segmentation guide axis on a sample image I, which will be further illustrated below by taking the simulation of the i-th round of operation as an example.

[0056] Optionally, when simulating the interaction operation of the i-th round, at least one connected region may be determined on the sample image, wherein the value of the pixel in each connected region in the real mask M is consistent with the predicted mask M input in the interaction operation of the i-th round. i The values ​​in ` are different.

[0057] When determining the connected area, for each pixel on the first image, the corresponding value of the pixel in the real mask M and the predicted mask M input in the interactive operation in the i-th round can be determined. i The values ​​in ` are the same. If different, the value of the pixel can be set to the specified value. After performing the above judgment on each pixel in the first image, multiple pixels with the specified value can be obtained. Based on the distribution of the multiple pixels with the specified value, at least one connected region can be obtained.

[0058] Next, one of the connected regions may be selected from the at least one connected region. For ease of description and distinction, the selected connected region is described as a first connected region.

[0059] Optionally, the first region may be selected from the at least one connected region according to a probability of random selection, wherein the probability of random selection is related to the area of ​​the at least one connected region. The larger the area of ​​the connected region, the greater the probability of the connected region being selected.

[0060] Optionally, based on the area of ​​the at least one connected region, a connected region whose area satisfies a first condition may be selected from the at least one connected region as the first connected region. The first condition may include: having the largest area, or having an area greater than a set threshold, which is not limited in this embodiment.

[0061] Next, a first path located in the instance sample can be selected from the first connected region. Optionally, the operation of selecting the first path from the first connected region can be implemented by thinning the first connected region (commonly known as "skeleton extraction"). This is exemplified below.

[0062] Optionally, the first connected region may be subjected to a medial axis process to extract at least one path in the first connected region. Next, a path is selected from the at least one path as the first path.

[0063] Optionally, the path length of each of the at least one path may be calculated, and based on the path length of each of the at least one central axis paths, a path whose path length satisfies a second condition may be selected from the at least one central axis path as the first path. The second condition may include: the path length being the longest, or the path length being greater than a certain length threshold.

[0064] Optionally, after obtaining the first path, the first path can be perturbed. The first path is used to simulate the instance segmentation axis drawn during user interaction. Therefore, perturbing the first path can simulate the randomness of the instance segmentation axis drawn by the user and enhance the robustness of the model training results.

[0065] Optionally, when perturbing the first path, a predetermined curve fitting method may be used to fit the first path. Next, the control point positions on the fitted curve are randomly perturbed, and the first path is updated based on the result of the random perturbation of the curve.

[0066] The set curve fitting method may include: Bezier curve fitting method, such as a third-order Bezier curve, a second-order Bezier curve, etc., which is not limited in this embodiment.

[0067] Next, a guidance map sample for the i-th interactive operation input with the same resolution as the sample image can be determined. When i = 1, the value of each pixel position in the guidance map sample for the i-th interactive operation input can be initialized to a null value. When i ≥ 2, the guidance map sample for the i-th interactive operation input is the guidance map sample generated for the i-1-th interactive operation round.

[0068] Next, within the guidance graph sample for the i-th round of interactive input, the first path is drawn, and the values ​​of the pixels corresponding to the positions of the pixels in the first connected region are determined within the guidance graph sample for the i-th round of interactive input, thereby generating a guidance graph sample corresponding to the i-th round of interactive operation. The guidance graph sample corresponding to the i-th round of interactive operation can be considered as a guidance graph generated by simulating the i-th user's operation of drawing the "skeleton" of the instance.

[0069] Among them, the following will take any pixel in the guidance map sample of the i-th round of interactive operation input as an example to exemplify the optional implementation method of determining the value of the pixel corresponding to the position of the pixel in the first connected area in the guidance map sample of the i-th round of interactive operation input.

[0070] Optionally, for any pixel in the guidance map sample of the i-th round of interactive operation input that has a corresponding position with the first connected area, determine whether the position corresponding to the pixel in the real mask M belongs to the instance sample; if the position corresponding to the pixel in the real mask M belongs to the instance sample, then determine that the pixel belongs to the instance sample in the guidance map sample of the i-th round of interactive operation input; if the position corresponding to the pixel in the real mask M does not belong to the instance sample, it can be determined that the pixel does not belong to the instance sample in the guidance map sample of the i-th round of interactive operation input.

[0071] Based on the above embodiment, N user interaction operations can be simulated and N sets of training data (I, G i ,M i `,M), where i=1,2,3…N. Next, N groups of training data can be used to train the neural network model, and the true mask M can be used as a supervisory signal to obtain the loss function of the neural network model, and optimize the parameters of the neural network model until the loss function converges to a specified range. In the above and following embodiments of the present application, the neural network model can be implemented as a convolutional neural network model (CNN). In some embodiments, CNN can be implemented as a U-Net (a neural network that uses a fully convolutional network for semantic segmentation) built on the basis of ResNet50. Of course, in other embodiments, CNN can also have other implementation forms, which are not limited in this embodiment.

[0072] In CNN, based on the convolutional layer, features can be extracted from the input image, the predicted mask generated by the previous round of interaction, and the guidance map of the current round of interaction to obtain the deep semantic features of each image data. Based on the deep semantic features, a more reliable mask prediction result can be obtained to improve the efficiency of instance segmentation.

[0073] It is worth noting that in the embodiments of the present application, sample images corresponding to a variety of different categories of subjects can be obtained, and a neural network model can be trained based on the model training method described in the aforementioned embodiments, so that the resulting model can segment any instance in the image without limiting the subject category. The different categories of subjects may include portraits, animals, commodities, human or animal tissue radiography images, etc., and this embodiment includes but is not limited to these. The resulting model trained using the model training method provided in this embodiment can achieve highly accurate and reliable instance segmentation results in a variety of different fields, meeting diverse image processing needs.

[0074] It should be understood that in this embodiment, the use of "first" and "second" to limit the objects being described is only for the convenience of description and distinction, and does not represent information such as order, level, and quantity.

[0075] Figure 4 A flow chart of an interactive image segmentation method provided by an exemplary embodiment of the present application is shown in FIG. Figure 4 As shown, when the method is executed on the terminal device side, it includes:

[0076] Step 401: Display a first image and a first mask.

[0077] Step 402: In response to a user operation of drawing an instance segmentation guiding axis on the first image, a first guidance map corresponding to the instance segmentation guiding axis is determined.

[0078] Step 403: Send the first image and the first guiding map to the server. Send the first image, the first mask, and the first guiding map to the server, so that the server uses a neural network model to predict the second mask corresponding to the instance segmentation guiding axis.

[0079] Step 404: Receive the second mask returned by the server, and highlight the local image area corresponding to the second mask on the first image.

[0080] In this embodiment, when users segment an image, they can draw instance segmentation guide axes (i.e., "skeletons") rather than instance edges, which reduces the difficulty of user interaction. After obtaining a guidance map based on the instance segmentation axes drawn by the user on the image, a neural network-based approach is used to predict the mask of the instance selected by the instance segmentation axes under the guidance of the guidance map. In this way, the neural network model can effectively extract deep semantic features in the image, which is conducive to improving the accuracy of instance segmentation results.

[0081] Figure 5 A flow chart of an interactive image segmentation method provided by another exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, when the method is executed on the server side, it includes:

[0082] Step 501: Receive a first image, a first mask, and a first guide map sent by a terminal device, where the first guide map is generated according to an instance segmentation guide axis drawn by a user on the first image.

[0083] Step 502: Input the first image, the first mask, and the first guidance map into a neural network model, wherein the neural network model is trained based on guidance map samples generated by simulating a user's segmentation guidance interaction operation.

[0084] Step 503: In the neural network model, extract semantic features of the first image and the first guide map, and predict a second mask corresponding to the instance segmentation guide axis based on the semantic features.

[0085] Step 504: Return the predicted second mask to the terminal device for display.

[0086] In some exemplary embodiments, the method further includes: obtaining a sample image and a real mask of an instance sample in the sample image; simulating multiple rounds of segmentation guidance interaction operations of the user, and in each round of interaction operations, simulating the user's operation of drawing an instance segmentation guidance axis on the sample image based on the real mask of the instance sample and the predicted mask input of each round of interaction operations to obtain multiple guidance map samples; obtaining training data corresponding to each of the multiple rounds of interaction operations based on the sample image, the real mask, the multiple guidance map samples and the predicted mask input of each round of interaction operations; training a neural network model based on the training data corresponding to each of the multiple rounds of interaction operations to optimize the parameters of the neural network model under the supervision of the real mask.

[0087] In some exemplary embodiments, the method further includes: when simulating the interaction operation of the i-th round, using the sample image, the guidance map generated by the interaction operation of the i-th round, and the predicted mask of the interaction operation input of the i-th round as input parameters of the set mask prediction algorithm, and calculating the predicted mask output by the interaction operation of the i-th round; judging whether the difference between the predicted mask output by the interaction operation of the i-th round and the real mask is greater than a first threshold; if the difference is greater than the first threshold and the number of interactions is less than a second threshold, using the predicted mask output by the interaction operation of the i-th round as the predicted mask input by the interaction operation of the i+1-th round.

[0088] In some exemplary embodiments, based on the real mask of the instance sample and the predicted mask of each round of interactive operation input, a method of simulating the user's operation of drawing the instance segmentation guide axis on the sample image to obtain multiple guide map samples may include: when simulating the interaction operation of the i-th round, determining a first connected area on the sample image; the values ​​of the pixels in the first connected area in the real mask are different from the values ​​in the predicted mask of the interaction operation input in the i-th round; selecting a first path located in the instance sample from the first connected area; drawing the first path in the guide map sample of the i-th round of interactive operation input with the same resolution as the sample image; determining the values ​​of the pixels corresponding to the positions of the pixels in the first connected area in the guide map sample of the i-th round of interactive operation input, and obtaining the guide map sample generated by the i-th round of interactive operation.

[0089] In some exemplary embodiments, a method of determining a first connected area on the sample image may include: determining at least one connected area on the sample image; the value of the pixel in each connected area in the real mask is different from the value in the predicted mask of the interactive operation input in the i-th round; and selecting the first connected area from the at least one connected area based on the area of ​​the at least one connected area.

[0090] In some exemplary embodiments, a method of selecting a first path located in the instance sample from the first connected area may include: performing a medial axis processing on the first connected area to extract at least one path in the first connected area; calculating the path length of each of the at least one path; and selecting the first path from the at least one medial axis path based on the path length of each of the at least one medial axis paths.

[0091] In some exemplary embodiments, after selecting a first path located in the instance sample from the first connected area, the method further includes: fitting the first path using a set curve fitting method; randomly perturbing the positions of control points on the fitted curve; and updating the first path based on the result of the random perturbation of the curve.

[0092] In some exemplary embodiments, a method for determining the value of a pixel corresponding to the position of a pixel in the first connected area in the guidance map sample of the i-th round of interactive operation input to obtain the guidance map sample generated by the i-th round of interactive operation may include: for any pixel in the guidance map sample of the i-th round of interactive operation input that has a corresponding position with the first connected area, judging whether the position corresponding to the pixel in the real mask belongs to the instance sample; if the position corresponding to the pixel in the real mask belongs to the instance sample, determining that the pixel belongs to the instance sample in the guidance map sample of the i-th round of interactive operation input; if the position corresponding to the pixel in the real mask does not belong to the instance sample, determining that the pixel does not belong to the instance sample in the guidance map sample of the i-th round of interactive operation input.

[0093] In this embodiment, after obtaining a guidance map based on the instance segmentation axis drawn by the user on the image, a neural network-based approach is used to predict the mask of the instance selected by the instance segmentation axis, guided by the guidance map. In this way, the neural network model can effectively extract deep semantic features in the image, which is conducive to improving the accuracy of instance segmentation results.

[0094] It should be understood that in some optional embodiments, the image instance segmentation process based on the neural network can be implemented as "remote computing" as described in the above embodiment. When the server is implemented as a cloud server, the above computing process can be described as "cloud computing". In other optional embodiments, the image instance segmentation process based on the neural network model can also be implemented as "edge computing" to provide near-end services to users. Figure 6 An exemplary description is given.

[0095] Figure 6 A flowchart of an interactive image segmentation method provided in yet another exemplary embodiment of the present application may include:

[0096] Step 601: Display a first image and a first mask.

[0097] Step 602: In response to the user drawing an instance segmentation guiding axis on the first image, determine a first guidance map corresponding to the instance segmentation guiding axis.

[0098] Step 603: Input the first image, the first mask, and the first guidance map into a neural network model, wherein the neural network model is trained based on guidance map samples generated by simulating a user's segmentation guidance interaction operation.

[0099] Step 604: In the neural network model, extract semantic features of the first image and the first guide map, and predict a second mask corresponding to the instance segmentation guide axis based on the semantic features.

[0100] Step 605: Highlight the local image area corresponding to the second mask on the first image.

[0101] This embodiment can be executed by a terminal device, which may include a computer, tablet computer, smartphone, PDA, or other device. The neural network model can be trained by the server, and the resulting model can be sent to the terminal device. For alternative implementations of server training of the neural network model, reference can be made to the description of the aforementioned embodiment, which will not be repeated here. When the neural network model is updated, the server can push the updated model to the terminal device in real time.

[0102] In order to facilitate the operation of the neural network model on the terminal device, the server can further perform pruning, quantization, compression and other operations on the neural network model before sending it to the terminal device to reduce the size of the neural network model. This will not be elaborated here.

[0103] In this embodiment, the neural network model resides on the terminal device. Each time the terminal device interacts with a user, the data generated by the interaction is input into the neural network model on the terminal device to obtain image instance segmentation results through the neural network model. This embodiment allows the terminal device to perform image instance segmentation offline, which not only speeds up the segmentation process but also reduces the data transmission bandwidth and costs.

[0104] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 501 to 503 can be device A; for another example, the execution entity of steps 501 and 502 can be device A, and the execution entity of step 503 can be device B; and so on.

[0105] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations, such as 501, 502, etc., are merely used to distinguish between different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0106] It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to different types.

[0107] The interactive image segmentation method provided in the embodiments of the present application can be applied to a variety of scenarios, such as disease diagnosis scenarios based on medical images, poster design scenarios, image enhancement scenarios, etc. The following will be exemplified using the disease diagnosis scenario based on medical images as an example.

[0108] After the user obtains a medical image, such as a color ultrasound image, an X-ray image, or a computer tomography (CT) image, the user can use the client provided by the terminal device to open the medical image. If there is a need to segment a local tissue in the medical image (such as a lung angiography image), the user can draw the "skeleton" of the local tissue (i.e., the instance segmentation guide axis) on the local tissue using an external device such as a mouse. After obtaining the "skeleton" of the local tissue, the client can generate a guide map and send the medical image, the guide map, and the current mask to the server. After the server obtains the above data sent by the client, it inputs the above data into a pre-trained neural network model and obtains the predicted mask output by the neural network model. Next, the server can return the predicted mask to the client for display.

[0109] After the client receives the predicted mask, it highlights the local area corresponding to the predicted mask on the medical image. If the local area is not the entire area where the local tissue that the user wants to extract is located, the user can draw a "skeleton" on the local tissue again. The position of the skeleton can be different from the position drawn last time. After the client obtains the skeleton drawn by the user again, it can generate a new guidance map and send the new guidance map, the current predicted mask and the medical image to the server, so that the server can return a new prediction mask based on the neural network model again. The above interactive process can be repeated until the user obtains a relatively satisfactory instance segmentation result.

[0110] Figure 7 This is a schematic diagram of the structure of a terminal device provided by an exemplary embodiment of the present application. Figure 7 As shown, the terminal device includes: a memory 701, a processor 702, a communication component 703 and a display component 704.

[0111] Memory 701 is used to store computer programs and can be configured to store various other data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc.

[0112] Among them, the memory 701 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0113] The processor 702 is coupled to the memory 701 and is used to execute the computer program in the memory 701, so as to: display a first image and a first mask through the display component 704; determine a first guidance map corresponding to the instance segmentation guide axis in response to a user's operation of drawing an instance segmentation guide axis on the first image; send the first image and the first guidance map to the server through the communication component 703, so that the server uses a neural network model to predict a second mask corresponding to the instance segmentation guide axis; receive the second mask returned by the server through the communication component 703, and highlight the local image area corresponding to the second mask on the first image.

[0114] Further, if Figure 7 As shown, the terminal device also includes: a power supply component 705, an audio component 706 and other components. Figure 7 Only some components are shown schematically, which does not mean that the terminal equipment only includes Figure 7 Components shown.

[0115] The display component 704 includes a screen, which may include a liquid crystal display component (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0116] In this embodiment, after obtaining a guidance map based on the instance segmentation axis drawn by the user on the image, a neural network-based approach is used to predict the mask of the instance selected by the instance segmentation axis, guided by the guidance map. In this way, the neural network model can effectively extract deep semantic features in the image, which is conducive to improving the accuracy of instance segmentation results.

[0117] In addition to the above method logic, Figure 7The terminal device shown can also execute the following method logic to achieve interactive image segmentation: display a first image and a first mask; in response to a user's operation of drawing an instance segmentation guide axis on the first image, determine a first guidance map corresponding to the instance segmentation guide axis; input the first image, the first mask, and the first guidance map into a neural network model, and the neural network model is trained based on guidance map samples generated by simulating the user's segmentation guidance interaction operation; in the neural network model, extract semantic features of the first image and the first guidance map, and predict a second mask corresponding to the instance segmentation guide axis based on the semantic features; highlight the local image area corresponding to the second mask on the first image. In this embodiment, the neural network model is located on the terminal device, realizing edge computing, which is conducive to improving the speed of image instance segmentation and reducing the traffic cost required for instance segmentation.

[0118] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the terminal device in the above method embodiment.

[0119] Figure 8 This is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. Figure 8 As shown, the server includes: a memory 801 , a processor 802 , a communication component 803 and a display component 804 .

[0120] Memory 801 is used to store computer programs and can be configured to store various other data to support operations on the server. Examples of such data include instructions for any application or method operating on the server, contact data, phone book data, messages, pictures, videos, etc.

[0121] Among them, the memory 801 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0122] The processor 802 is coupled to the memory 801 and is used to execute the computer program in the memory 801, so as to: receive a first image, a first mask and a first guidance map sent by a terminal device, wherein the first guidance map is generated according to an instance segmentation guide axis drawn by a user on the first image; input the first image, the first mask and the first guidance map into a neural network model, and the neural network model is trained based on guidance map samples generated by simulating the user's segmentation guidance interaction operation; in the neural network model, extract semantic features of the first image and the first guidance map, and predict a second mask corresponding to the instance segmentation guide axis according to the semantic features; and return the predicted second mask to the terminal device for display.

[0123] In some exemplary embodiments, the processor 802 is further used to: obtain a sample image and a real mask of an instance sample in the sample image; simulate multiple rounds of segmentation guidance interaction operations by the user, and in each round of interaction operations, simulate the user's operation of drawing an instance segmentation guidance axis on the sample image based on the real mask of the instance sample and the predicted mask input of each round of interaction operations to obtain multiple guidance map samples; obtain training data corresponding to each of the multiple rounds of interaction operations based on the sample image, the real mask, the multiple guidance map samples and the predicted mask input of each round of interaction operations; train a neural network model based on the training data corresponding to each of the multiple rounds of interaction operations to optimize the parameters of the neural network model under the supervision of the real mask.

[0124] In some exemplary embodiments, the processor 802 is also used to: when simulating the interaction operation of the i-th round, use the sample image, the guidance map generated by the interaction operation of the i-th round, and the predicted mask of the interaction operation input of the i-th round as input parameters of the set mask prediction algorithm, and calculate the predicted mask output by the interaction operation of the i-th round; determine whether the difference between the predicted mask output by the interaction operation of the i-th round and the real mask is greater than a first threshold; if the difference is greater than the first threshold and the number of interactions is less than the second threshold, then use the predicted mask output by the interaction operation of the i-th round as the predicted mask input of the interaction operation of the i+1-th round.

[0125] In some exemplary embodiments, when the processor 802 simulates the user's operation of drawing the instance segmentation guide axis on the sample image based on the real mask of the instance sample and the predicted mask of each round of interactive operation input to obtain multiple guide map samples, it is specifically used to: when simulating the interaction operation of the i-th round, determine a first connected area on the sample image; the values ​​of the pixels in the first connected area in the real mask are different from the values ​​in the predicted mask of the interaction operation input in the i-th round; select a first path located in the instance sample from the first connected area; draw the first path in the guide map sample of the i-th round of interactive operation input with the same resolution as the sample image; determine the values ​​of the pixels corresponding to the positions of the pixels in the first connected area in the guide map sample of the i-th round of interactive operation input, and obtain the guide map sample generated by the i-th round of interactive operation.

[0126] In some exemplary embodiments, when the processor 802 determines the first connected area on the sample image, it is specifically used to: determine at least one connected area on the sample image; the value of the pixel in each connected area in the real mask is different from the value in the predicted mask of the interactive operation input in the i-th round; and select the first connected area from the at least one connected area based on the area of ​​the at least one connected area.

[0127] In some exemplary embodiments, when the processor 802 selects the first path located in the instance sample from the first connected area, it is specifically used to: perform central axis processing on the first connected area to extract at least one path in the first connected area; calculate the path length of each of the at least one path; and select the first path from the at least one central axis path based on the path length of each of the at least one central axis paths.

[0128] In some exemplary embodiments, after selecting the first path located in the instance sample from the first connected area, the processor 802 is further used to: fit the first path using a set curve fitting method; randomly perturb the positions of the control points on the fitted curve; and update the first path based on the result of the random perturbation of the curve.

[0129] In some exemplary embodiments, when the processor 802 determines the value of the pixel corresponding to the position of the pixel in the first connected area in the guidance map sample of the i-th round of interactive operation input, and obtains the guidance map sample generated by the i-th round of interactive operation, it is specifically used to: for any pixel in the guidance map sample of the i-th round of interactive operation input that has a corresponding position with the first connected area, determine whether the position corresponding to the pixel in the real mask belongs to the instance sample; if the position corresponding to the pixel in the real mask belongs to the instance sample, determine that the pixel belongs to the instance sample in the guidance map sample of the i-th round of interactive operation input; if the position corresponding to the pixel in the real mask does not belong to the instance sample, determine that the pixel does not belong to the instance sample in the guidance map sample of the i-th round of interactive operation input.

[0130] Further, if Figure 8 As shown, the terminal device also includes: a power supply component 804 and other components. Figure 8 Only some components are shown schematically, which does not mean that the terminal equipment only includes Figure 8 Components shown.

[0131] In this embodiment, after obtaining a guidance map based on the instance segmentation axis drawn by the user on the image, a neural network-based approach is used to predict the mask of the instance selected by the instance segmentation axis, guided by the guidance map. In this way, the neural network model can effectively extract deep semantic features in the image, which is conducive to improving the accuracy of instance segmentation results.

[0132] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the server in the above method embodiment.

[0133] In the above Figure 7 as well as Figure 8 In the embodiment, the communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0134] In the above Figure 7 as well as Figure 8In the present invention, the power supply component is used to provide power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0135] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0139] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0140] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0141] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0142] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0143] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An interactive image segmentation system, characterized in that: include: Terminal devices and servers; The terminal device is configured to: display a first image and a first mask; in response to a user drawing an instance segmentation guide axis on the first image, generate a first guidance map corresponding to the instance segmentation guide axis, and send the first image and the first guidance map to the server; and receive a second mask corresponding to the instance segmentation guide axis returned by the server, and highlight the local image area corresponding to the second mask on the first image; the first guide image has the same resolution as the first image and is used to indicate instance segmentation; The server is used to: receive a first image and a first guidance map sent by the terminal device; input the first image, the first mask and the first guidance map into a neural network model, wherein the neural network model is trained based on guidance map samples generated by simulating a user's segmentation guidance interaction operation; extract semantic features of the first image and the first guidance map in the neural network model; predict a second mask corresponding to the instance segmentation guide axis based on the semantic features, and return the predicted second mask to the terminal device.

2. An interactive image segmentation method, characterized in that: include: displaying a first image and a first mask; In response to a user's operation of drawing an instance segmentation guiding axis on the first image, generating a first guidance map corresponding to the instance segmentation guiding axis; The first guide image has the same resolution as the first image and is used to indicate instance segmentation; Inputting the first image, the first mask, and the first guidance map into a neural network model, wherein the neural network model is trained based on guidance map samples generated by simulating a user's segmentation guidance interaction operation; In the neural network model, semantic features of the first image and the first guide map are extracted, and a second mask corresponding to the instance segmentation guide axis is predicted based on the semantic features; The local image area corresponding to the second mask is highlighted on the first image.

3. An interactive image segmentation method, applicable to a terminal device, characterized in that: include: displaying a first image and a first mask; In response to a user's operation of drawing an instance segmentation guiding axis on the first image, generating a first guidance map corresponding to the instance segmentation guiding axis; The first guide image has the same resolution as the first image and is used to indicate instance segmentation; Sending the first image, the first mask, and the first guidance map to a server, so that the server uses a neural network model to predict a second mask corresponding to the instance segmentation guide axis; The second mask returned by the server is received, and a local image area corresponding to the second mask is highlighted on the first image.

4. An interactive image segmentation method, applicable to a server, characterized in that: include: Receiving a first image, a first mask, and a first guide map sent by a terminal device, where the first guide map is generated based on instance segmentation guide axes drawn by a user on the first image; the first guide image has the same resolution as the first image and is used to indicate instance segmentation; Inputting the first image, the first mask, and the first guidance map into a neural network model, wherein the neural network model is trained based on guidance map samples generated by simulating a user's segmentation guidance interaction operation; In the neural network model, semantic features of the first image and the first guide map are extracted, and a second mask corresponding to the instance segmentation guide axis is predicted based on the semantic features; The predicted second mask is returned to the terminal device for display.

5. The method according to claim 4, characterized in that Also includes: Obtaining a sample image and a true mask of an instance sample in the sample image; Simulating multiple rounds of segmentation guidance interaction operations by a user. In each round of interaction operations, based on the actual mask of the instance sample and the predicted mask input in each round of interaction operations, the user is simulated to draw an instance segmentation guidance axis on the sample image to obtain multiple guidance map samples. Obtaining training data corresponding to each of the multiple rounds of segmentation guidance interaction operations according to the sample image, the true mask, the multiple guide map samples, and the predicted mask input in each round of interaction operation; The neural network model is trained according to the training data corresponding to each of the multiple rounds of interactive operations, so as to optimize the parameters of the neural network model under the supervision of the true mask.

6. The method according to claim 5, Its features include: When simulating the interactive operation of the i-th round, the sample image, the guidance map generated by the interactive operation of the i-th round, and the prediction mask input by the interactive operation of the i-th round are used as input parameters of the set mask prediction algorithm to calculate the prediction mask output by the interactive operation of the i-th round, where i is a positive integer; Determine whether a difference between a predicted mask outputted by the interactive operation of the i-th round and the real mask is greater than a first threshold; If the difference is greater than the first threshold and the number of interactions is less than the second threshold, the prediction mask output by the interaction operation in the i-th round is used as the prediction mask input by the interaction operation in the (i+1)-th round.

7. The method according to claim 5, characterized in that Based on the real mask of the instance sample and the predicted mask input in each round of interactive operation, simulate the user's operation of drawing the instance segmentation guide axis on the sample image to obtain multiple guidance map samples, including: When simulating the interactive operation of the i-th round, determining a first connected region on the sample image; values ​​of pixels in the first connected region in the real mask are different from values ​​in the predicted mask input in the interactive operation of the i-th round; Selecting a first path located in the instance sample from the first connected area; Drawing the first path in the guide graph sample inputted in the i-th round of interactive operation having the same resolution as the sample image; The values ​​of pixels corresponding to the positions of the pixels in the first connected area are determined in the guidance map samples inputted by the i-th round of interactive operation to obtain the guidance map samples generated by the i-th round of interactive operation.

8. The method according to claim 7, characterized in that Determining a first connected region on the sample image includes: On the sample image, at least one connected region is determined; a value of a pixel in each connected region in the true mask is different from a value in the predicted mask input in the i-th round of interactive operation; The first connected region is selected from the at least one connected region according to an area of ​​the at least one connected region.

9. The method according to claim 7, characterized in that Selecting a first path located in the instance sample from the first connected region includes: Performing medial axis processing on the first connected region to extract at least one path in the first connected region; calculating a path length of each of the at least one path; The first path is selected from the at least one path according to the path length of each of the at least one path.

10. The method according to any one of claims 7 to 9, characterized in that: After selecting a first path located in the instance sample from the first connected region, the method further includes: Using a set curve fitting method to fit the first path; Randomly perturb the control point positions on the fitted curve; The first path is updated according to a result of randomly perturbing the curve.

11. The method according to any one of claims 7 to 9, characterized in that: Determining, in the guide map sample inputted by the i-th round of interactive operation, a value of a pixel corresponding to a position of a pixel in the first connected region, to obtain the guide map sample generated by the i-th round of interactive operation, comprising: For any pixel in the guide graph sample input in the i-th round of interactive operation that has a corresponding position with the first connected area, determining whether the position corresponding to the pixel in the real mask belongs to the instance sample; If the position corresponding to the pixel in the real mask belongs to the instance sample, determining that the pixel belongs to the instance sample in the guidance image sample input in the i-th round of interactive operation; If the position corresponding to the pixel in the real mask does not belong to the instance sample, it is determined that the pixel does not belong to the instance sample in the guidance image sample input in the i-th round of interactive operation.

12. A terminal device, characterized in that: include: memory, processor communication components, and display components; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to perform the steps of the method according to claim 2 or 3.

13. A terminal device, characterized in that: include: memory, processor communication components; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to perform the steps of the method according to any one of claims 4 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 2 or 3 or the steps of any one of claims 4 to 11 can be implemented.

Images