Interactive image segmentation method and apparatus based on click point influence enhancement

Abstract

This application discloses an interactive image segmentation method and apparatus based on click point influence enhancement, which is used to improve the influence range of enhanced click points and improve image segmentation accuracy. The method includes: determining the predicted mask image of the Nth round as the initial mask image of the N+1th round, and generating a click image of the N+1th round based on the click operations of the previous N+1 rounds; inputting the original image corresponding to the predicted mask image obtained in the Nth round, the click image of the N+1th round, and the initial mask image of the N+1th round into a pre-trained target image segmentation model to obtain a predicted mask image of the N+1th round. The target image segmentation model includes a target click point enhancement layer for enhancing the influence range of target click points in the N+1th round click image. Target click points include click points in the N+1th round click image corresponding to the click operations of the N+1th round; and performing image segmentation operations on the original image based on the predicted mask image of the N+1th round in response to a segmentation confirmation command for the predicted mask image of the N+1th round.

Description

Technical Field

[0001] This application relates to the field of image segmentation, and more particularly to an interactive image segmentation method and apparatus based on click point influence enhancement. Background Technology

[0002] Interactive segmentation aims to segment target objects in images using limited user interaction information. Due to the addition of extra user annotation information, interactive segmentation can often achieve higher accuracy than traditional segmentation methods in complex scenes.

[0003] Existing technical solutions use Gaussian click maps to model click points. This limits the validity of click points to a small range with a fixed variance, meaning user-annotated click points only have an impact within a limited scope. Given this limited influence, users can only maintain segmentation accuracy to some extent by increasing the number of click points, thus increasing annotation costs. Summary of the Invention

[0004] This application provides an interactive image segmentation method and apparatus based on click point influence enhancement, which is used to increase the influence range of enhanced click points and improve image segmentation accuracy.

[0005] The first aspect of this application provides an interactive image segmentation method based on local region similarity click point influence enhancement, including:

[0006] In response to the user's click operation in round N+1, the predicted mask image obtained in round N is determined as the initial mask image in round N+1, and the click image in round N+1 is generated based on the click operations in the previous rounds.

[0007] The original image corresponding to the predicted mask image obtained in the Nth round, the click image in the N+1th round, and the initial mask image in the N+1th round are input into a pre-trained target image segmentation model to obtain the predicted mask image in the N+1th round. The target image segmentation model includes a target click point enhancement layer, which is used to enhance the influence range of the target click points in the click image in the N+1th round. The target click points include the click points in the click image in the N+1th round that correspond to the click operation in the N+1th round.

[0008] In response to the segmentation confirmation instruction for the predicted mask map of the N+1th round, an image segmentation operation is performed on the original image based on the predicted mask map of the N+1th round.

[0009] In one specific implementation, the method further includes:

[0010] Obtain multiple sets of sample pairs, each set of sample pairs containing a sample image and the expected mask image corresponding to the sample image;

[0011] Each pair of samples is determined as the target sample pair in turn, and the sample images contained in the target sample pair are the target sample images;

[0012] Based on the target sample image, the expected mask image corresponding to the target sample image, and the initial image segmentation model, a preprocessed click image and a preprocessed mask image are determined.

[0013] The target sample image, the preprocessed click map, and the preprocessed mask map are input into the initial image segmentation model to obtain the training mask map;

[0014] The loss value is calculated based on the preset loss function, the training mask image, and the expected mask image corresponding to the target sample image;

[0015] If the loss value satisfies the preset convergence condition, then the initial image segmentation model is determined to be the target image segmentation model;

[0016] If the loss value does not meet the preset convergence condition, the parameters of the initial image segmentation model are adjusted to obtain a new initial image segmentation model.

[0017] In one specific implementation, determining the preprocessed click map and the preprocessed mask map based on the target sample image and the initial image segmentation model includes:

[0018] Based on the expected mask image corresponding to the target sample image, a first click image is randomly generated;

[0019] The target sample image, the first click image, and the first mask image corresponding to the target sample image are input into the initial image segmentation model to obtain a preprocessed mask image, wherein the first mask image does not contain the region of interest.

[0020] Based on the preprocessed mask image and the expected mask image, the point of maximum prediction difference is determined;

[0021] Based on the expected mask image, the click point type of the maximum predicted difference point is determined, and the maximum predicted difference point is added to the first click image to obtain a preprocessed click image.

[0022] In one specific implementation, the initial image segmentation model includes an initial feature extraction layer, an initial attention layer, and an initial click point enhancement layer;

[0023] The step of inputting the target sample image, the preprocessed click map, and the preprocessed mask map into the initial image segmentation model to obtain a training mask map includes:

[0024] The target sample image is divided into multiple sub-images according to a preset division rule, the preprocessed click image is divided into multiple sub-click images according to a preset division rule, and the preprocessed mask image is divided into multiple sub-mask images according to a preset division rule, wherein each sub-click image corresponds one-to-one with a sub-image, and each sub-mask image corresponds one-to-one with a sub-image.

[0025] Each sub-image is then input into the initial feature extraction layer to obtain the sub-image features corresponding to each sub-image.

[0026] Each sub-image feature, each sub-click map, and each sub-mask map are input into the initial attention layer to obtain the first attention corresponding to each sub-image.

[0027] The first attention corresponding to each sub-image is input into the initial click point enhancement layer to obtain the second attention corresponding to each sub-image output by the initial click point enhancement layer. The second attention corresponding to each sub-image is the product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image. The feature similarity corresponding to each sub-image includes the sub-image features of each sub-image and the similarity between the sub-image features of the sub-image where each target click point is located.

[0028] The product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image is determined as the second attention corresponding to each sub-image;

[0029] The second attention corresponding to each sub-image is input into the initial mask layer to obtain the training mask image.

[0030] In one specific implementation, the initial click point enhancement layer further includes an initial mapping function; the initial click point enhancement layer is also used to map the features of each sub-image to an initial space according to the initial mapping function, and calculate the feature similarity between each click sub-image and each other sub-image in the initial space, wherein the click sub-image is a sub-image containing any positive click point, and each other sub-image is an image other than the click sub-image among the plurality of sub-images.

[0031] In one specific implementation, calculating the loss value based on a preset loss function, the training mask image, and the expected mask image corresponding to the target sample image includes:

[0032] Based on the preset correlation formula, the second attention of each sub-image, and the mask value corresponding to each sub-image in the training mask image, calculate the correlation value between the second attention and the mask value in the training mask image;

[0033] The loss value is calculated based on the correlation value between the second attention and the mask value in the training mask image, the training mask image, and the expected mask image corresponding to the target sample image.

[0034] A second aspect of this application provides a computer device, including:

[0035] The response unit is used to respond to the user's click operation in the (N+1)th round, determine the predicted mask image obtained in the Nth round as the initial mask image in the (N+1)th round, and generate the click image in the (N+1)th round based on the click operations in the previous N+1 rounds.

[0036] The prediction unit is used to input the original image corresponding to the prediction mask image obtained in the Nth round, the click image in the N+1th round, and the initial mask image in the N+1th round into a pre-trained target image segmentation model to obtain the prediction mask image in the N+1th round. The target image segmentation model includes a target click point enhancement layer, which is used to enhance the influence range of the target click points in the click image in the N+1th round. The target click points include the click points in the click image in the N+1th round that correspond to the click operation in the N+1th round.

[0037] A segmentation unit is configured to perform an image segmentation operation on the original image based on the prediction mask map of the (N+1)th round in response to a segmentation confirmation instruction for the prediction mask map of the (N+1)th round.

[0038] In one specific implementation, the device further includes: an acquisition unit, a training unit, and a computing unit;

[0039] The process involves acquiring multiple sets of sample pairs, each set of sample pairs containing a sample image and the expected mask image corresponding to the sample image.

[0040] The determining unit is also used to sequentially determine each pair of samples as a target sample pair, wherein the sample image contained in the target sample pair is the target sample image;

[0041] The determining unit is further configured to determine the preprocessed click map and the preprocessed mask map based on the target sample image, the expected mask map corresponding to the target sample image, and the initial image segmentation model;

[0042] The training unit is used to input the target sample image, the preprocessed click map, and the preprocessed mask map into the initial image segmentation model to obtain a training mask map;

[0043] The calculation unit is used to calculate the loss value based on the preset loss function, the training mask image, and the expected mask image corresponding to the target sample image;

[0044] The determining unit is further configured to determine the initial image segmentation model as the target image segmentation model if the loss value satisfies a preset convergence condition.

[0045] The training unit is further configured to adjust the parameters of the initial image segmentation model to obtain a new initial image segmentation model if the loss value does not meet the preset convergence condition.

[0046] In one specific implementation, the determining unit is specifically used to randomly generate a first click image based on the expected mask image corresponding to the target sample image;

[0047] The target sample image, the first click image, and the first mask image corresponding to the target sample image are input into the initial image segmentation model to obtain a preprocessed mask image, wherein the first mask image does not contain the region of interest.

[0048] Based on the preprocessed mask image and the expected mask image, the point of maximum prediction difference is determined;

[0049] Based on the expected mask image, the click point type of the maximum predicted difference point is determined, and the maximum predicted difference point is added to the first click image to obtain a preprocessed click image.

[0050] In one specific implementation, the initial image segmentation model includes an initial feature extraction layer, an initial attention layer, and an initial click point enhancement layer;

[0051] The training unit is specifically used to segment the target sample image into multiple sub-images according to a preset segmentation rule, segment the preprocessed click image into multiple sub-click images according to a preset segmentation rule, and segment the preprocessed mask image into multiple sub-mask images according to a preset segmentation rule, wherein each sub-click image corresponds one-to-one with a sub-image, and each sub-mask image corresponds one-to-one with a sub-image.

[0052] Each sub-image is then input into the initial feature extraction layer to obtain the sub-image features corresponding to each sub-image.

[0053] Each sub-image feature, each sub-click map, and each sub-mask map are input into the initial attention layer to obtain the first attention corresponding to each sub-image.

[0054] The first attention corresponding to each sub-image is input into the initial click point enhancement layer to obtain the second attention corresponding to each sub-image output by the initial click point enhancement layer. The second attention corresponding to each sub-image is the product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image. The feature similarity corresponding to each sub-image includes the sub-image features of each sub-image and the similarity between the sub-image features of the sub-image where each target click point is located.

[0055] The product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image is determined as the second attention corresponding to each sub-image;

[0056] The second attention corresponding to each sub-image is input into the initial mask layer to obtain the training mask image.

[0057] In one specific implementation, the initial click point enhancement layer further includes an initial mapping function; the initial click point enhancement layer is also used to map the features of each sub-image to an initial space according to the initial mapping function, and calculate the feature similarity between each click sub-image and each other sub-image in the initial space, wherein the click sub-image is a sub-image containing any positive click point, and each other sub-image is an image other than the click sub-image among the plurality of sub-images.

[0058] In one specific implementation, the computing unit is specifically used to calculate the correlation value between the second attention and the mask value in the training mask image based on a preset correlation formula, the second attention of each sub-image, and the mask value corresponding to each sub-image in the training mask image;

[0059] The loss value is calculated based on the correlation value between the second attention and the mask value in the training mask image, the training mask image, and the expected mask image corresponding to the target sample image.

[0060] A third aspect of this application provides a computer device, including:

[0061] Central processing unit, memory, and input / output interfaces;

[0062] The memory is either a short-term storage memory or a persistent storage memory;

[0063] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.

[0064] A fourth aspect of this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect.

[0065] A fifth aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0066] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: In response to the user's click operation in the (N+1)th round, the predicted mask image obtained in the Nth round is determined as the initial mask image in the (N+1)th round, and the click image in the (N+1)th round is generated based on the click operations in the previous (N+1)th rounds; the original image corresponding to the predicted mask image obtained in the Nth round, the click image in the (N+1)th round, and the initial mask image in the (N+1)th round are input into a pre-trained target image segmentation model to obtain the predicted mask image in the (N+1)th round. The target image segmentation model includes a target click point enhancement layer, which is used to enhance the influence range of the target click points in the (N+1)th round click image. The target click points include the click points in the (N+1)th round click image that correspond to the click operation in the (N+1)th round; In response to the segmentation confirmation instruction for the predicted mask image in the (N+1)th round, an image segmentation operation is performed on the original image based on the predicted mask image in the (N+1)th round. By constructing a click point enhancement layer (target click point enhancement layer) in the model (target image segmentation model), the influence range of each click point annotated by the user is expanded, so that users can obtain high image segmentation accuracy with only a low annotation cost, thereby improving the user experience. Attached Figure Description

[0067] Figure 1 This is a schematic diagram illustrating the positive and negative click attention coupling of the existing interactive image segmentation method disclosed in the embodiments of this application;

[0068] Figure 2 This is a schematic flowchart of an interactive image segmentation method disclosed in an embodiment of this application;

[0069] Figure 3 This is a schematic diagram of the second attention calculation disclosed in an embodiment of this application;

[0070] Figure 4 A schematic diagram illustrating the affinity loss calculation disclosed in the embodiments of this application;

[0071] Figure 5 This is another schematic flowchart of the interactive image segmentation method disclosed in the embodiments of this application;

[0072] Figure 6 This is a schematic diagram illustrating the effectiveness of the interactive image segmentation method disclosed in the embodiments of this application;

[0073] Figure 7This is a schematic diagram illustrating the relationship between the number of clicks and the desired accuracy as disclosed in the embodiments of this application;

[0074] Figure 8 This is a schematic diagram of the structure of a computer device disclosed in an embodiment of this application;

[0075] Figure 9 This is another schematic diagram of the computer device disclosed in the embodiments of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] Please see Figure 1 , Figure 1 This is a schematic diagram of positive and negative click attention coupling in existing sampling techniques, where... Figure 1 The included positive and negative click area attention distribution maps show that ellipses indicate a small range of influence for the corresponding click point, while squares indicate that attention is activated in unrelated areas.

[0078] Current interactive image segmentation methods based on click points typically use a click map to represent each user-annotated click point, then concatenate the click map with the corresponding original image and input it into a model to predict the mask of the target object. Method 1 uses a Gaussian click map to model click points, which results in the click map having only effective values ​​within a small range of fixed variance, thus narrowing the influence range of the click points. Considering the limitations of Method 1, Method 2 utilizes prior information about the hierarchical structure of the image to improve segmentation performance. Meanwhile, Method 3 proposes a post-fusion method that utilizes the semantic features of click points to obtain the prediction result, and then further optimizes the target mask using the click points. However, the click information constructed by the above methods is still a low-level feature that is sensitive to the scene.

[0079] Furthermore, to achieve efficient differentiation between positive and negative clicks, Method 4 proposes constructing a two-channel click graph to distinguish between positive and negative clicks, a strategy subsequently adopted by numerous related technologies. However, Method 4's strategy exacerbates the sparsity of click information, making it difficult for the network to learn a clear distinction between positive and negative clicks.

[0080] In summary, the current modeling approach for click points focuses only on the local area near the click point, leading to an increase in the number of clicks. Furthermore, the current approach lacks effective differentiation between positive and negative clicks, causing interference between them and reducing the segmentation accuracy after multiple clicks.

[0081] To address these issues, this application's embodiments divide the input (original image, initial mask image, and initial click image) into multiple local region blocks. It then utilizes the similarity between the local region block containing the click point and other local region blocks to enhance the influence range of the click point, reduce the number of user clicks, and guide the network to achieve accurate segmentation. Furthermore, this application's embodiments propose an affinity loss based on the correlation of local region blocks to decouple positive and negative clicks, guiding positive clicks to focus more on the target object and negative clicks to focus more on the background, thus improving segmentation accuracy. Overall, this invention demonstrates advanced performance on existing evaluation standards while requiring fewer parameters.

[0082] Please see Figure 2 This application provides an interactive image segmentation method based on click point influence enhancement, comprising the following steps:

[0083] 201. In response to the user's click operation in round N+1, determine the predicted mask image obtained in round N as the initial mask image in round N+1, and generate the click image in round N+1 based on the click operations in the previous rounds.

[0084] Similar to existing interactive image segmentation methods, the interactive image segmentation in this embodiment is also implemented iteratively. Each image segmentation cycle (i.e., completing multiple iterations to segment the user's desired region of interest) typically involves multiple iterations. Before each iteration, the user needs to annotate different types of click points in the original image to inform the model: the user's region of interest (the location marked by a positive click point) and the region of non-interest (the location marked by a negative click point). The input to the image segmentation model is the original image, the click map, and the mask image output by the previous iteration. Therefore, before using the pre-trained target image segmentation model, we need to determine the model input for this iteration, namely the original image, the click map of the Nth iteration, and the initial mask image of the N+1th iteration. It is important to note that before the current image segmentation is completed (i.e., before performing image segmentation on the original image), the original image input for each iteration is the same; except for the initial mask image in the first iteration, which is a default mask image (i.e., the mask image does not include the region of interest), the initial mask image in subsequent iterations is the predicted mask image output by the previous iteration.

[0085] In practical applications, a click operation can be a user marking a click point, or a user selecting a click point and then confirming the click with a control; no specific limitations are made here.

[0086] It should be noted that the click map for the N+1th round is generated based on the click operations of the previous N+1 rounds. This means that the click points marked in the original image by the user in the current round and every round before it are recorded in a single click map (i.e., the click map for this round). Each iteration's click operations may include one or more click points, and the type of each click point may be different. Furthermore, each click point in each round's click map can be in the same channel (i.e., the click map is single-channel), or each click point in each round's click map can belong to either the positive or negative channel depending on its type (i.e., the click map is dual-channel), but this is not limited here.

[0087] 202. Input the original image corresponding to the predicted mask image obtained in the Nth round, the click image in the N+1th round, and the initial mask image in the N+1th round into the pre-trained target image segmentation model to obtain the predicted mask image in the N+1th round. The target image segmentation model includes a target click point enhancement layer, which is used to enhance the influence range of the target click points in the click image in the N+1th round. The target click points include the click points in the click image in the N+1th round that correspond to the click operation in the N+1th round.

[0088] After determining the initial mask image and the click image for round N+1 in step 201, the corresponding original image (the predicted mask image for each iteration of this image segmentation corresponds to the same original image), the click image for round N+1, and the initial mask image for round N+1 can be input into the pre-trained target image segmentation model. Finally, the predicted mask image for round N+1 is obtained from the model output. In other words, interactive image segmentation relies on user-annotated click points, which is particularly evident in the first few iterations. As the number of user-annotated click points increases, the obtained predicted mask image will be closer to the user's segmentation requirements.

[0089] It should be noted that, unlike the image segmentation models in the prior art, the target image segmentation model in this application includes a click point enhancement layer, which is used to solve the problem of the small influence range of click points caused by the model in the prior art, and to enhance the attention influence range of the target click points in the click image.

[0090] In practical applications, the target click point can be each newly added click point in this round (here, the N+1th round), or the target click point can also be each click point in this round and every round before this round (here, the 1st round to the N+1th round) (i.e., each click point included in the click map of the N+1th round). This embodiment does not limit this.

[0091] 203. In response to the segmentation confirmation instruction for the prediction mask map of round N+1, perform image segmentation operation on the original image based on the prediction mask map of round N+1.

[0092] After obtaining the predicted mask map of the N+1th round output by the model in step 202, the user can determine whether the predicted mask map meets their requirements. If it does, the user can initiate a segmentation confirmation command for the predicted mask map of the N+1th round. Then, the computer device can segment the user's region of interest from the original image based on the predicted mask map, that is, perform image segmentation operation on the original image based on the predicted mask map of the N+1th round.

[0093] In addition to user-initiated segmentation confirmation commands, after determining that the predicted mask image in the (N+1)th round is the predicted mask image obtained in the latest round iteration, the computer device can determine the time difference between the time when the predicted mask image was generated and the current time. When the time difference exceeds a preset time threshold (such as 5 minutes or 10 minutes, which can be determined as needed), the computer device can automatically trigger the operation of generating a segmentation confirmation command and performing image segmentation operations. This embodiment does not specifically limit the method of generating segmentation confirmation commands.

[0094] In this embodiment of the application, by constructing a click point enhancement layer (target click point enhancement layer) in the model (target image segmentation model), the influence range of each click point annotated by the user is expanded, so that the user can obtain higher image segmentation accuracy with only a lower annotation cost, thereby improving the user experience.

[0095] In some specific implementations, the target image segmentation model of this application embodiment can be trained in the following way: Multiple sets of sample pairs are acquired, each set containing a sample image and a corresponding expected mask image; each set of sample pairs is sequentially determined as a target sample pair, and the sample image contained in the target sample pair is the target sample image; based on the target sample image, the corresponding expected mask image, and the initial image segmentation model, a preprocessed click image and a preprocessed mask image are determined; the target sample image, the preprocessed click image, and the preprocessed mask image are input into the initial image segmentation model to obtain a training mask image; based on a preset loss function, the training mask image, and the corresponding expected mask image of the target sample image, a loss value is calculated; if the loss value meets a preset convergence condition, the initial image segmentation model is determined as the target image segmentation model; if the loss value does not meet the preset convergence condition, the parameters of the initial image segmentation model are adjusted to obtain a new initial image segmentation model.

[0096] Specifically, a single sample image may contain multiple regions of interest with pre-labeled masks. That is, if any two sample pairs contain identical sample images, then the expected mask images contained in those two sample pairs will definitely be different; if any two sample pairs contain identical sample images, then the expected mask images contained in those two sample pairs will be different. Figure 1If the two sample pairs mentioned above are identical, then the sample images they contain are definitely not the same; that is, the sample image and the expected mask image corresponding to different samples cannot be completely identical. Furthermore, as can be seen from the aforementioned related embodiments, the inputs to the image segmentation model (initial image segmentation model and target image segmentation model) in this application embodiment are the image, the click image, and the previous round mask image. Therefore, it is first necessary to generate a preprocessed click image and a preprocessed mask image through the initial image segmentation model. It should be noted that during the generation of the preprocessed click image and the preprocessed mask image, the initial image segmentation model does not perform parameter adjustments or loss calculations. Then, after inputting the target sample image, the preprocessed click image, and the preprocessed mask image into the initial image segmentation model, the corresponding training mask image can be obtained. Based on the preset loss function, the training mask image, and the expected mask image corresponding to the target sample image, the loss value is calculated. The loss value is used to characterize the deviation between the predicted mask image of the current model and the corresponding expected mask image. Therefore, based on whether the loss value meets the preset convergence condition, it can be determined whether to continue training or consider the current model to meet the requirements. For example, if the loss value meets the preset convergence condition, the initial image segmentation model can be considered to meet the user's needs, and the initial image segmentation model is determined as the target image segmentation model; if the loss value does not meet the preset convergence condition, the parameters of the initial image segmentation model are adjusted to obtain a new initial image segmentation model, and then other sample pairs not used for training are used to train the new initial image segmentation model until the loss value after a certain round of training meets the preset convergence condition, and the training ends.

[0097] The preset convergence condition can be that the loss value is less than a preset loss threshold, and the specific preset loss threshold can be set by the user as needed.

[0098] In some specific implementations, the aforementioned preprocessed click map and preprocessed mask map can be generated in the following ways: A first click map is randomly generated based on the expected mask map corresponding to the target sample image; the target sample image, the first click map, and the first mask map corresponding to the target sample image are input into an initial image segmentation model to obtain a preprocessed mask map, where the first mask map does not contain the region of interest; the maximum predicted difference point is determined based on the preprocessed mask map and the expected mask map; the click point type of the maximum predicted difference point is determined based on the expected mask map, and the maximum predicted difference point is added to the first click map to obtain the preprocessed click map.

[0099] Specifically, a second click map is first generated randomly, with both the location and number of click points being random. Then, based on the expected mask map (i.e., the region of interest in the target sample image) corresponding to the target sample image, the type (positive or negative click point) of each click point in the second click map is determined. Finally, a first click map, categorized by type and channel, is generated based on the second click map. The first click map contains two channels. It's important to note that, to simulate the randomness of the number of click points during real-world model usage, the number of click points in the randomly generated first click map is variable.

[0100] After determining the first click map, the target sample image, the first click map, and the first mask map corresponding to the target sample image are input into the initial image segmentation model to obtain a preprocessed mask map. The first mask map is similar to the mask map input to the first iteration of the image segmentation model in actual use. Since there is no corresponding mask map output by the previous model, it is assumed that the first mask map here should not contain any region of interest.

[0101] Next, by combining the preprocessed mask image and the expected mask image, the point with the largest prediction difference is determined. This means identifying the pixel with the largest prediction deviation between the preprocessed and expected mask images. This pixel is then added as a new click point and, according to its type, to the corresponding channel of the first click image, thus obtaining the preprocessed click image. The point with the largest prediction difference may be the pixel with the largest difference between the attention value of each pixel when obtaining the preprocessed mask image and the attention value of each pixel when obtaining the expected mask image. It is understood that the point with the largest prediction difference can also be a pixel whose difference between the two values ​​exceeds a preset difference threshold; no specific restrictions are imposed here.

[0102] The above embodiments only describe the method of obtaining a preprocessed click map and a preprocessed mask map through a single prediction. In practice, to improve the reliability or practicality of the preprocessed click map and the preprocessed mask map, multiple iterations can be performed according to the above method to obtain a preprocessed click map (generated based on the click map input in the last iteration and the maximum prediction difference point calculated in the last iteration) and a preprocessed mask map (based on the last iteration). No specific limit is made here on the number of iterations required to obtain the preprocessed click map and the preprocessed mask map.

[0103] Furthermore, the initial image segmentation model in this embodiment includes an initial feature extraction layer, an initial attention layer, and an initial click point enhancement layer. Based on this, the processing performed by each layer in the image segmentation model of this embodiment can be referred to the following: The target sample image is segmented into multiple sub-images according to a preset segmentation rule; the preprocessed click map is segmented into multiple sub-click maps according to a preset segmentation rule; and the preprocessed mask map is segmented into multiple sub-mask maps according to a preset segmentation rule, wherein each sub-click map corresponds one-to-one with a sub-image, and each sub-mask map corresponds one-to-one with a sub-image; each sub-image is input into the initial feature extraction layer to obtain the sub-image features corresponding to each sub-image; and each sub-image feature, each sub-click map, and each sub-mask map are input into the initial feature extraction layer. The first attention for each sub-image is obtained by inputting the initial attention layer into the initial click point enhancement layer. The second attention for each sub-image is then input into the initial click point enhancement layer to obtain the second attention for each sub-image output by the initial click point enhancement layer. The second attention for each sub-image is the product of the first attention for each sub-image and the feature similarity for each sub-image. The feature similarity for each sub-image includes the sub-image features of each sub-image and the similarity between the sub-image features of each target click point and the sub-image features of the sub-image containing the target click point. The first attention for each sub-image and the product of the feature similarity for each sub-image are then used to determine the second attention for each sub-image. The second attention for each sub-image is then input into the initial mask layer to obtain the training mask image.

[0104] Specifically, since the image segmentation model in this embodiment receives images of a fixed size, it is also necessary to ensure that the sizes of each sub-image, each sub-click image, and each sub-mask image obtained according to the preset segmentation rules are consistent. Specifically, the preset segmentation rules can be the sizes of the sub-images, sub-click images, and sub-masks obtained from the segmentation, or the number of sub-images, sub-click images, or sub-masks obtained from the segmentation. Except for ensuring that the same preset segmentation rules apply during model training and model application (obtaining the predicted mask image based on the target image segmentation model) and requiring the same input image size, no other restrictions are imposed here. It is understood that after segmenting into multiple sub-images, multiple sub-click images, and multiple sub-masks, each sub-image corresponds to a unique sub-click image and a unique sub-mask image.

[0105] Then, considering the similarity between the sub-image features of any sub-image and the sub-image features of any click point sub-image, it indicates to some extent the probability that any sub-image belongs to the region of interest or region of non-interest to which the aforementioned click point sub-image belongs. Therefore, by calculating the feature similarity corresponding to each sub-image and weighting the first attention corresponding to each sub-image to obtain the second attention, the attention influence range of each click point can be effectively expanded.

[0106] Furthermore, when the sub-image features extracted by the initial feature extraction layer are primarily used for segmentation and not suitable for directly calculating feature similarity, an initial mapping function can be configured in the initial click point enhancement layer. This mapping function can map the sub-image features to other spaces, making the mapped features suitable for calculating feature similarity. See the following: The initial click point enhancement layer also includes an initial mapping function; the initial click point enhancement layer is also used to map each sub-image feature to an initial space according to the initial mapping function, and calculate the feature similarity between each click sub-image and each other sub-image in the initial space. A click sub-image is a sub-image containing any positive click point, and each other sub-image is an image other than the click sub-image among multiple sub-images.

[0107] Specifically, as mentioned above, the number of feature similarities corresponding to each other sub-image is the same as the number of positive clicks in the target click point. Therefore, the feature similarity corresponding to each other sub-image can be equal to the average of the sum of its feature similarities with the positive clicks in each target click point. Furthermore, since the influence of each positive click on its corresponding click sub-image should be the greatest, the feature similarity of each positive click's corresponding click sub-image is directly set to 1 (no calculation required). Then, each first attention is weighted according to its corresponding feature similarity to obtain the second attention.

[0108] The following describes a click point impact enhancement scheme according to an embodiment of this application in a specific scenario.

[0109] First, in this embodiment, the entire input (i.e., the image, the click image, and the press image) is divided into L×L sub-blocks (sub-image blocks, sub-click image blocks, and sub-mask image blocks) by default, and the embedded features (or sub-image features) of all sub-image blocks are extracted. Features f of different sub-image patches p i p They are often different, and they characterize the features of local regions of an image.

[0110] In an entire image, an object to be segmented (i.e., a region of interest) often contains multiple sub-image patches, which represent the object. Observation reveals that a user's positive click typically lands only on a limited number of sub-image patches contained within the object to be segmented, while paying less attention to other sub-image patches. In extreme cases, this leads to the user needing to click multiple times until a click point is found on all sub-image patches before achieving ideal segmentation accuracy.

[0111] Inspired by this, this embodiment utilizes the embedded features of the sub-image patch where the target click point is located. Similarity calculation of embedded features with other local regions of the image Where corr represents the cross-correlation operation, N k This represents the number of sub-images where the user is currently clicking. By considering N... k The average of the similarity results is used to obtain the final feature similarity as shown in equation (2).

[0112]

[0113] It should be noted that because the features of the sub-images corresponding to the sub-image where the click is located are consistent, in order to... Figure 3 In the region of interest (ROI) example, a motorcycle tire has similar texture properties across different parts, so this characteristic is used to calculate weights (i.e., feature similarity). However, the sub-image corresponding to the negative click location lacks consistency in its features; the background (i.e., the region of non-ROI) is cluttered with objects such as people and scenery, offering no discernible pattern.

[0114] However, the sub-image features of different sub-images are used for segmentation and are not suitable for directly calculating similarity. Therefore, we propose a nonlinear mapping function. The features f of each sub-image i Mapped to another space Similarity calculation is performed, and the robustness of the algorithm is improved by decoupling the similarity task from the image segmentation task. Its structure is as follows: Figure 3 As shown, Figure 3 The sub-image features corresponding to the positive click point, as well as the sub-image features of other sub-images, are mapped to other spaces, and similarity calculations have been completed. Figure 3 As shown in the rightmost click attention distribution diagram, although the user only marked the positive click point on one tire of the motorcycle, the other tire of the motorcycle also received high attention due to the similarity between the two tires, thus enhancing the influence range of the click point. It should be noted that... Figure 3 Each small block in the upper middle section identifies the sub-image features of other sub-images, the three small blocks in the lower middle section identify the sub-image features of the click point sub-image, and the second to last grid diagram represents the second attention for each sub-image.

[0115] Based on this nonlinear mapping φ, segmentation mask labels y are used during model training. i (i.e., the mask value in the corresponding expected mask image) is used as the supervision information for its similarity calculation, and the mean square error is used as the loss function for training, as shown in Equation (3).

[0116]

[0117] Where, N i This indicates the number of downsampling stages involved in the calculation. The robust feature similarity s is then obtained.i Then, the similarity results s i As the first attention The bias guides the network to focus on the target area clicked by the user, as shown in equation (4). Additionally, the similarity learning loss function shown in equation (3) can also be the mean squared error loss, cross-entropy loss, or... Losses are not limited here.

[0118]

[0119] Here, ⊙ represents multiplying corresponding elements. For second attention, It is the first point of attention.

[0120] Based on the above, it can be seen that the attention to the region with higher similarity to the clicked sub-image in the first attention of the sub-image will be enhanced, while the region with low similarity to the clicked sub-image will be suppressed, thereby guiding the network to strengthen its attention to the clicked instance.

[0121] In practical applications, due to the semantic complexity of local regions in negative click images, the aforementioned click point enhancement only considers the correlation between the clicked sub-image and the global sub-image. This may lead the network (or model) to incorrectly focus on similar objects in the background. Specifically, the attention weight matrix... Attention scores between sub-images were recorded during the forward propagation process, reflecting the guidance of different sub-images on the network's overall region of interest. However, due to the lack of clear distinction between the categories (regions of interest or regions of non-interest) of sub-images, the local image regions corresponding to positive and negative clicks may struggle to learn clear discriminative relationships, leading to the learning of unnecessary contextual dependencies, such as... Figure 4 As shown.

[0122] To address this issue, this application proposes a discriminative affinity loss based on similarity measurement, guiding the network to fully utilize positive and negative clicks to suppress erroneous attention in the background. The overall algorithm structure is as follows: Figure 4 As shown. Specifically, the following steps can be followed: Calculate the correlation value between the second attention and the mask value in the training mask image based on the preset correlation formula, the second attention of each sub-image, and the mask value corresponding to each sub-image in the training mask image; calculate the loss value based on the correlation value between the second attention and the mask value in the training mask image, the training mask image, and the expected mask image corresponding to the target sample image.

[0123] Specifically, in order to guide the model to make full use of positive and negative click points to suppress erroneous attention in the background, the correlation value between the second attention and the mask value in the training mask image can be introduced when calculating the loss value. Finally, the loss value is obtained by combining the training mask image and the expected mask image corresponding to the target sample image.

[0124] The following is based on Figure 4 For example, the input image is divided into L×L local regions (or sub-images) and processed by a segmentation network to output the corresponding instance segmentation result x. P The corresponding background segmentation result is represented as follows: The attention in the diagram corresponds to the i-th downsampling stage ∈ {1,2,3}. The j-th row represents the attention relationship between the j-th local image region corresponding to the c-th click point and other local image regions. By using this attention and the positive / negative click prediction results to calculate the corresponding affinity loss, the network can be guided to learn discriminative attentional responses. That is, positive click attention is focused on the instance part (i.e., the region of interest), while negative click attention is focused on the background part (i.e., the region of non-interest). Figure 4 The target prediction probability shown is the training mask image s. Therefore, the corresponding background prediction probability is the mask image whose values ​​are opposite to those of each sub-block in the training mask image, i.e., 1-s. In the figure, P represents different sub-images.

[0125] Specifically, stacked It exhibits global dependencies between local image regions. This is achieved by aggregating multi-head and multi-layer attention weight distributions A during the model training phase. i,l Attention distribution map A for each stage was established. i As shown in equation (5).

[0126]

[0127] in, Let represent the mean of the attention weight matrix in the i-th stage. The attention corresponding to all local regions of the image where the positive click point is located is . Simultaneously, the attention corresponding to the local region of all negative click images (the sub-image where the negative click point is located) is... Where 1-s represents the weight of the local region of the image corresponding to the negative click, and ⊙ represents the weight of the region in image A. i Multiply the corresponding elements in the second dimension.

[0128] To improve the ability to distinguish between positive and negative click attention by utilizing affinity loss, I used positive and negative click attention... With the segmentation network output x P , The affinity loss is calculated as shown in Equation (6).

[0129]

[0130] Where, ‖·‖1 represents Loss, N sIndicates the number of stages involved in the calculation. By x P , Interpolation is obtained. x P ⊙s∈R B×m×w×h ,x P ⊙(1-s)∈R B×n×w×h This indicates the predicted output for the local region of the image corresponding to the clicked point. and All of these represent the correlation between attention and prediction results, as shown in Equation (7).

[0131]

[0132] in, This represents the vector dot product. It characterizes the probability distribution of instances during the forward propagation process. It represents the probability distribution of background positions during forward propagation. By using Equation (6) to constrain these distributions, the network can make full use of interactive information to understand the user's interactive intent and reduce the number of clicks.

[0133] It is understood that the click attention in the affinity loss calculation described above can also be replaced by discrete positive and negative click region attention for affinity loss calculation; the affinity loss calculation method of this application embodiment can also be used for affinity loss calculation under other click point differentiation methods besides positive and negative differentiation.

[0134] Please see Figure 5 The following is an overall framework diagram of an embodiment of this application. The three images on the left, from top to bottom, are: the image, the click image, and the mask image (from the previous round). The black background image on the right is the mask image output by the model.

[0135] The following explanation will be provided in a specific scenario.

[0136] In standard interactive segmentation tasks, the model input x includes the input image. Click the image The previous segmentation result (i.e., the predicted mask image). Where, x c The 2 in the image represents the two channels where the positive and negative click points are located, x P x represents the distribution of target object locations predicted by the network. m This represents the final segmentation mask.

[0137] In this embodiment, the input x is divided into L 4×4 local image patches, which are then fed into a 4-level Transformer encoder to obtain features with scales of [1 / 4, 1 / 8, 1 / 16, 1 / 32]. Where B represents the number of samples in the batch. i C represents different downsampling stages. i This represents the feature dimension of a local image patch in the i-th downsampling level. Finally, these features from different scales are upsampled to the same scale and the prediction result is obtained using an MLP decoder. Where N cls =1. Each downsampling layer contains multiple multi-head self-attention layers. To achieve efficient self-attention computation, the SRA self-attention algorithm proposed by Wenhai W et al. is adopted, as shown in Equation (1).

[0138]

[0139] Where Q = W q ·x,K=W k ·SR(x),V=W v ·SR(x), h represents the attention weight matrix of the l-th layer in the i-th downsampling layer. i R represents the number of detector heads in the i-th downsampling stage. i The spatial downsampling operation SR(·) reduces the proportion of local region image patch sequences. Represents the query matrix. Let W represent the key matrix and the value matrix, respectively. q W k W v The parameter represents the linear mapping. In this scheme, the attention corresponding to the local image patch (i.e., sub-image patch) where the positive click point is located is called positive click attention, and the attention corresponding to the local image patch (i.e., sub-image patch) where the negative click point is located is called negative click attention.

[0140] In terms of increasing the influence range of click points and decoupling the mutual interference between positive and negative clicks, the embodiments of this application show better performance than the benchmark algorithm, such as... Figure 6 As shown. By Figure 6 As can be seen, the similarity-based click attention in this invention can effectively alleviate the problems of insufficient click influence range and positive-negative click coupling. Furthermore, the affinity loss algorithm in this invention can effectively decouple the coupling between positive and negative click attention. Compared with other algorithms, the proposed scheme can converge to the expected accuracy with fewer clicks, such as... Figure 7 As shown, the present invention outperforms existing algorithms in both click numbers and average accuracy (IOU), where Ours-B3-S2 is the algorithm of the present invention.

[0141] To verify the effectiveness of the present invention, it was evaluated on multiple benchmarks, and the experimental results are shown in Table 1.

[0142] Table 1

[0143]

[0144]

[0145] In Table 1, the top row of gray-background content represents different training sets; the NOC in Table 1 represents the number of clicks required for a user to achieve the preset accuracy, and NOC 90 represents the number of clicks required for 90% of users to achieve the preset accuracy; the leftmost column in Table 1 represents different image segmentation models; the architectures of this invention-B0S2 and this invention-B3S2 are similar, the difference being that the number of model parameters is different, with one having more Transformer layers than the other.

[0146] As shown in Table 1, the embodiments of this application exhibit the best performance among the compared algorithms. By adopting the solution of this invention, the efficiency of interactive segmentation is effectively improved. Overall, the interactive segmentation algorithm proposed in the embodiments of this application achieves advanced performance.

[0147] Please see Figure 8 This application provides a computer device, comprising:

[0148] The response unit 801 is used to respond to the user's click operation in the (N+1)th round, determine the predicted mask image obtained in the Nth round as the initial mask image in the (N+1)th round, and generate the click image in the (N+1)th round based on the click operations in the previous N+1 rounds.

[0149] The prediction unit 802 is used to input the original image corresponding to the prediction mask image obtained in the Nth round, the click image in the N+1th round, and the initial mask image in the N+1th round into a pre-trained target image segmentation model to obtain the prediction mask image in the N+1th round. The target image segmentation model includes a target click point enhancement layer, which is used to enhance the influence range of the target click points in the click image in the N+1th round. The target click points include the click points in the click image in the N+1th round that correspond to the click operation in the N+1th round.

[0150] The segmentation unit 803 is configured to perform image segmentation on the original image based on the prediction mask of the N+1th round in response to a segmentation confirmation instruction for the prediction mask of the N+1th round.

[0151] In one specific implementation, the device further includes: an acquisition unit, a training unit, and a computing unit;

[0152] Obtain multiple sets of sample pairs, each set of sample pairs containing a sample image and a corresponding expected mask image of the sample image;

[0153] The determining unit is also used to sequentially determine each pair of samples as the target sample pair, and the sample image contained in the target sample pair is the target sample image;

[0154] The determining unit is also used to determine the preprocessed click map and the preprocessed mask map based on the target sample image, the expected mask map corresponding to the target sample image, and the initial image segmentation model;

[0155] The training unit is used to input the target sample image, the preprocessed click map, and the preprocessed mask map into the initial image segmentation model to obtain the training mask map;

[0156] The calculation unit is used to calculate the loss value based on the preset loss function, the training mask image, and the expected mask image corresponding to the target sample image;

[0157] The determining unit is also used to determine the initial image segmentation model as the target image segmentation model if the loss value meets the preset convergence condition;

[0158] The training unit is also used to adjust the parameters of the initial image segmentation model to obtain a new initial image segmentation model if the loss value does not meet the preset convergence condition.

[0159] In one specific implementation, the determining unit is specifically used to randomly generate a first click image based on the expected mask image corresponding to the target sample image;

[0160] The target sample image, the first click image, and the first mask image corresponding to the target sample image are input into the initial image segmentation model to obtain a preprocessed mask image. The first mask image does not contain the region of interest.

[0161] Based on the preprocessed mask image and the expected mask image, the point of maximum prediction difference is determined;

[0162] The click point type with the largest predicted difference is determined based on the expected mask image, and the largest predicted difference is added to the first click image to obtain a preprocessed click image.

[0163] In one specific implementation, the initial image segmentation model includes an initial feature extraction layer, an initial attention layer, and an initial click point enhancement layer;

[0164] The training unit is specifically used to segment the target sample image into multiple sub-images according to the preset segmentation rules, segment the preprocessed click image into multiple sub-click images according to the preset segmentation rules, and segment the preprocessed mask image into multiple sub-mask images according to the preset segmentation rules, wherein each sub-click image corresponds one-to-one with a sub-image, and each sub-mask image corresponds one-to-one with a sub-image.

[0165] Each sub-image is then input into the initial feature extraction layer to obtain the sub-image features corresponding to each sub-image.

[0166] Input each sub-image feature, each sub-click map, and each sub-mask map into the initial attention layer to obtain the first attention corresponding to each sub-image;

[0167] The first attention corresponding to each sub-image is input into the initial click point enhancement layer to obtain the second attention corresponding to each sub-image output by the initial click point enhancement layer. The second attention corresponding to each sub-image is the product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image. The feature similarity corresponding to each sub-image includes the sub-image features of each sub-image and the similarity between the sub-image features of each sub-image and the sub-image features of the sub-image where each target click point is located.

[0168] The product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image is determined as the second attention corresponding to each sub-image.

[0169] The second attention corresponding to each sub-image is input into the initial mask layer to obtain the training mask image.

[0170] In one specific implementation, the initial click point enhancement layer further includes an initial mapping function; the initial click point enhancement layer is also used to map the features of each sub-image to an initial space according to the initial mapping function, and to calculate the feature similarity between each click sub-image and each other sub-image in the initial space, wherein the click sub-image is a sub-image containing any positive click point, and each other sub-image is an image other than the click sub-image among multiple sub-images.

[0171] In one specific implementation, the computing unit is specifically used to calculate the correlation value between the second attention and the mask value in the training mask image based on the preset correlation formula, the second attention of each sub-image, and the mask value corresponding to each sub-image in the training mask image.

[0172] The loss value is calculated based on the correlation between the second attention and the mask values ​​in the training mask image, the training mask image, and the expected mask image corresponding to the target sample image.

[0173] Figure 9 This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 900 may include one or more central processing units (CPUs) 901 and a memory 905, in which one or more application programs or data are stored.

[0174] The memory 905 can be volatile or persistent storage. The program stored in the memory 905 can include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the central processing unit 901 can be configured to communicate with the memory 905 and execute the series of instruction operations stored in the memory 905 on the computer device 900.

[0175] The computer device 900 may also include one or more power supplies 902, one or more wired or wireless network interfaces 903, one or more input / output interfaces 904, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0176] The central processing unit 901 can perform the aforementioned... Figures 1 to 8 The specific operations performed by the computer device in the illustrated embodiment will not be described in detail here.

[0177] It should be noted that although the steps in the flowcharts of the various embodiments are drawn sequentially according to the arrows, unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the various embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the interactive image segmentation method based on the enhanced influence range of click points as described above.

Claims

1. An interactive image segmentation method based on click point influence enhancement, characterized in that, include: In response to the user's click operation in round N+1, the predicted mask image obtained in round N is determined as the initial mask image in round N+1, and the click image in round N+1 is generated based on the click operations in the previous rounds. The original image corresponding to the predicted mask image obtained in the Nth round, the click image in the (N+1)th round, and the initial mask image in the (N+1)th round are input into a pre-trained target image segmentation model to obtain the predicted mask image in the (N+1)th round. The target image segmentation model includes a target click point enhancement layer, which is used to enhance the attention influence range of the target click points in the (N+1)th round click image. The target click points include the click points in the (N+1)th round click image corresponding to the click operation in the (N+1)th round. The process of obtaining the predicted mask image in the (N+1)th round... The prediction mask image for round 1 includes: segmenting the original image into multiple sub-images according to a preset segmentation rule; calculating the feature similarity between the sub-image features of each sub-image and the sub-image features of the sub-image containing each target click point in the click image of round N+1, to obtain the feature similarity corresponding to each sub-image; multiplying the first attention of each sub-image with the feature similarity corresponding to each sub-image to obtain the second attention of each sub-image; and obtaining the prediction mask image for round N+1 through the second attention of each sub-image. In response to the segmentation confirmation instruction for the predicted mask map of the N+1th round, an image segmentation operation is performed on the original image based on the predicted mask map of the N+1th round.

2. The method according to claim 1, characterized in that, The method further includes: Obtain multiple sets of sample pairs, each set of sample pairs containing a sample image and the expected mask image corresponding to the sample image; Each pair of samples is determined as the target sample pair in turn, and the sample image contained in the target sample pair is the target sample image; Based on the target sample image, the expected mask image corresponding to the target sample image, and the initial image segmentation model, the preprocessed click image and the preprocessed mask image are determined. The target sample image, the preprocessed click map, and the preprocessed mask map are input into the initial image segmentation model to obtain the training mask map; The loss value is calculated based on the preset loss function, the training mask image, and the expected mask image corresponding to the target sample image; If the loss value satisfies the preset convergence condition, then the initial image segmentation model is determined to be the target image segmentation model; If the loss value does not meet the preset convergence condition, the parameters of the initial image segmentation model are adjusted to obtain a new initial image segmentation model.

3. The method according to claim 2, characterized in that, The step of determining the preprocessed click map and the preprocessed mask map based on the target sample image, the expected mask map corresponding to the target sample image, and the initial image segmentation model includes: Based on the expected mask image corresponding to the target sample image, a first click image is randomly generated; The target sample image, the first click image, and the first mask image corresponding to the target sample image are input into the initial image segmentation model to obtain a preprocessed mask image, wherein the first mask image does not contain the region of interest. Based on the preprocessed mask image and the expected mask image, the point of maximum prediction difference is determined; Based on the expected mask image, the click point type of the maximum predicted difference point is determined, and the maximum predicted difference point is added to the first click image to obtain a preprocessed click image.

4. The method according to claim 2, characterized in that, The initial image segmentation model includes an initial feature extraction layer, an initial attention layer, an initial click point enhancement layer, and an initial mask layer; The step of inputting the target sample image, the preprocessed click map, and the preprocessed mask map into the initial image segmentation model to obtain a training mask map includes: The target sample image is divided into multiple sub-images according to a preset division rule, the preprocessed click image is divided into multiple sub-click images according to a preset division rule, and the preprocessed mask image is divided into multiple sub-mask images according to a preset division rule, wherein each sub-click image corresponds one-to-one with a sub-image, and each sub-mask image corresponds one-to-one with a sub-image. Each sub-image is then input into the initial feature extraction layer to obtain the sub-image features corresponding to each sub-image. Each sub-image feature, each sub-click map, and each sub-mask map are input into the initial attention layer to obtain the first attention corresponding to each sub-image. The first attention corresponding to each sub-image is input into the initial click point enhancement layer to obtain the second attention corresponding to each sub-image output by the initial click point enhancement layer. The second attention corresponding to each sub-image is the product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image. The feature similarity corresponding to each sub-image includes the sub-image features of each sub-image and the similarity between the sub-image features of the sub-image where each target click point is located. The product of the first attention corresponding to each sub-image and the feature similarity corresponding to each sub-image is determined as the second attention corresponding to each sub-image; The second attention corresponding to each sub-image is input into the initial mask layer to obtain the training mask image.

5. The method according to claim 4, characterized in that, The initial click point enhancement layer further includes an initial mapping function; the initial click point enhancement layer is also used to map the features of each sub-image to an initial space according to the initial mapping function, and to calculate the feature similarity between each click sub-image and each other sub-image in the initial space, wherein the click sub-image is a sub-image containing any positive click point, and each other sub-image is an image other than the click sub-image among the plurality of sub-images.

6. The method according to any one of claims 4-5, characterized in that, The step of calculating the loss value based on the preset loss function, the training mask image, and the expected mask image corresponding to the target sample image includes: Based on the preset correlation formula, the second attention of each sub-image, and the mask value corresponding to each sub-image in the training mask image, calculate the correlation value between the second attention and the mask value in the training mask image; The loss value is calculated based on the correlation value between the second attention and the mask value in the training mask image, the training mask image, and the expected mask image corresponding to the target sample image.

7. A computer device, characterized in that, include: The response unit is used to respond to the user's click operation in the (N+1)th round, determine the predicted mask image obtained in the Nth round as the initial mask image in the (N+1)th round, and generate the click image in the (N+1)th round based on the click operations in the previous N+1 rounds. The prediction unit is used to input the original image corresponding to the prediction mask image obtained in the Nth round, the click image in the (N+1)th round, and the initial mask image in the (N+1)th round into a pre-trained target image segmentation model to obtain the prediction mask image in the (N+1)th round. The target image segmentation model includes a target click point enhancement layer, which is used to enhance the attention influence range of the target click points in the (N+1)th round click image. The target click points include the click points in the (N+1)th round click image that correspond to the click operation in the (N+1)th round. Obtaining the prediction mask image for the (N+1)th round includes: segmenting the original image into multiple sub-images according to a preset segmentation rule; calculating the feature similarity between the sub-image features of each sub-image and the sub-image features of the sub-image containing each target click point in the click image of the (N+1)th round, to obtain the feature similarity corresponding to each sub-image; multiplying the first attention of each sub-image with the feature similarity corresponding to each sub-image to obtain the second attention of each sub-image; and obtaining the prediction mask image for the (N+1)th round through the second attention of each sub-image. A segmentation unit is configured to perform an image segmentation operation on the original image based on the prediction mask map of the (N+1)th round in response to a segmentation confirmation instruction for the prediction mask map of the (N+1)th round.

8. The computer device according to claim 7, characterized in that, The device further includes: an acquisition unit, a determination unit, a training unit, and a calculation unit; The process involves acquiring multiple sets of sample pairs, each set of sample pairs containing a sample image and the expected mask image corresponding to the sample image. The determining unit is also used to sequentially determine each pair of samples as a target sample pair, wherein the sample image contained in the target sample pair is a target sample image; The determining unit is further configured to determine the preprocessed click map and the preprocessed mask map based on the target sample image, the expected mask map corresponding to the target sample image, and the initial image segmentation model; The training unit is used to input the target sample image, the preprocessed click map, and the preprocessed mask map into the initial image segmentation model to obtain a training mask map; The calculation unit is used to calculate the loss value based on the preset loss function, the training mask image, and the expected mask image corresponding to the target sample image; The determining unit is further configured to determine the initial image segmentation model as the target image segmentation model if the loss value satisfies a preset convergence condition. The training unit is further configured to adjust the parameters of the initial image segmentation model to obtain a new initial image segmentation model if the loss value does not meet the preset convergence condition.

9. A computer device, characterized in that, include: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method of any one of claims 1-6.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1-6.

Images