Model optimization method and device, electronic equipment and storage medium

Abstract

The application provides a model optimization method and device, electronic equipment and a storage medium, which relates to the technical field of artificial intelligence, and comprises the following steps: based on at least one image in a first data set, a first target image corresponding to each image is constructed; the number of channels of the first target image is more than that of the image; the first target image is input into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is obtained by training a second target image corresponding to a sample image in a second data set; based on the first feature data and the first segmentation result, pseudo-label data corresponding to the first target image is determined; and based on the pseudo-label data and true label data, the interactive segmentation model is optimized. The method provided by the application realizes optimization of the interactive segmentation model and improves the accuracy of the interactive segmentation model in object instance segmentation of an image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model optimization method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image segmentation is an important and classic computer vision task with wide applications in fields such as autonomous driving, video analysis, and remote sensing monitoring. Interactive segmentation requires users to select specific object instances in an image based on their interests in order to segment the corresponding objects. The key research focus is on how to efficiently select interaction strategies and learn the content contained in the interaction information within the network model.

[0003] In related technologies, traditional interactive segmentation often relies on large amounts of finely labeled datasets to train the network model; for datasets containing domain differences, the network model needs to be retrained. This poses a challenge to the practical application of interactive segmentation models in open environments, resulting in low image segmentation accuracy. Summary of the Invention

[0004] This invention provides a model optimization method, apparatus, electronic device, and storage medium to address the low accuracy of image segmentation in existing technologies. It enables continuous optimization of the interactive segmentation model based on pseudo-label data, thereby improving the accuracy of the interactive segmentation model in segmenting object instances in images.

[0005] This invention provides a model optimization method, comprising:

[0006] Based on at least one image in the first dataset, a first target image is constructed corresponding to each of the images; the number of channels in the first target image is greater than the number of channels in the image.

[0007] The first target image is input into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on the second target image corresponding to the sample image in the second dataset, and the interactive segmentation model is used to segment object instances in the second target image, the second target image in the second dataset includes true label data;

[0008] Based on the first feature data and the first segmentation result, the pseudo-label data corresponding to the first target image is determined;

[0009] The interactive segmentation model is optimized based on the pseudo-label data and the real label data.

[0010] According to a model optimization method provided by the present invention, determining the pseudo-label data corresponding to the first target image based on the first feature data and the first segmentation result includes:

[0011] Based on the first feature data and the first segmentation result, at least one prototype feature of the object instance is determined;

[0012] Based on the aforementioned prototype features, the pseudo-label data corresponding to the first target image is determined.

[0013] According to a model optimization method provided by the present invention, the at least one prototype feature includes a global prototype feature, a central prototype feature, and a peripheral prototype feature;

[0014] Determining at least one prototype feature of the object instance based on the first feature data and the first segmentation result includes:

[0015] Based on the first feature data and the first segmentation result, determine the second feature data corresponding to the pixels in the first feature data that belong to the foreground point position in the first segmentation result;

[0016] The sums of multiple second feature data are calculated, and the sums are averaged to obtain the global prototype feature of the object instance.

[0017] Randomly select x first foreground locations from the multiple foreground locations corresponding to the first segmentation result; x is a positive integer.

[0018] Based on the x first foreground point locations, the second foreground point location corresponding to the first target image, and the first feature data, the third feature data corresponding to the pixels of the first foreground point location and the second foreground point location are determined respectively.

[0019] The central prototype feature of the object instance is obtained by summing multiple third feature data and averaging the sums.

[0020] The first segmentation result is eroded to obtain the edge position corresponding to the first segmentation result;

[0021] The fourth feature data corresponding to the pixels at the edge positions are summed, and the sums are averaged to obtain the edge prototype features of the object instance.

[0022] According to a model optimization method provided by the present invention, determining the pseudo-label data corresponding to the first target image based on each of the prototype features includes:

[0023] Based on the global prototype feature, the central prototype feature, and the edge prototype feature, the squared Euclidean distance corresponding to the feature vector of each pixel is calculated respectively;

[0024] Based on the squared Euclidean distances, at least one confidence matrix is ​​determined; the confidence matrix includes at least one of the following: global confidence matrix, center confidence matrix, and marginal confidence matrix;

[0025] The center confidence matrix and the edge confidence matrix are weighted and summed to obtain the weighted sum confidence matrix;

[0026] Based on the global confidence matrix and the weighted summation confidence matrix, determine the first confidence value corresponding to the global confidence matrix and the second confidence value corresponding to the weighted summation confidence matrix;

[0027] Based on the first confidence value and the second confidence value, determine the second segmentation result corresponding to the first confidence value and the third segmentation result corresponding to the second confidence value;

[0028] Based on the first segmentation result, the second segmentation result, and the third segmentation result, the pseudo-label data corresponding to the first target image is determined.

[0029] According to a model optimization method provided by the present invention, determining the pseudo-label data corresponding to the first target image based on the first segmentation result, the second segmentation result, and the third segmentation result includes:

[0030] Based on the first segmentation result, the second segmentation result, and the third segmentation result, determine whether the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold;

[0031] If the sum of the values ​​corresponding to the position of each pixel is greater than the preset threshold, the pixel is a foreground pixel;

[0032] If the sum of the values ​​corresponding to the position of each pixel is not greater than the preset threshold, the pixel is a background pixel.

[0033] According to a model optimization method provided by the present invention, optimizing the interactive segmentation model based on the pseudo-label data and the true label data includes:

[0034] Based on the pseudo-label data and the true label data, sample the third target image from the first dataset;

[0035] Based on the third target image, determine the fourth target image in the second dataset;

[0036] The interactive segmentation model is optimized based on the third target image and the fourth target image.

[0037] According to a model optimization method provided by the present invention, the interactive segmentation model is optimized based on the third target image and the fourth target image. This includes:

[0038] Based on the third target image and the fourth target image, the interactive segmentation model is optimized using formulas (1) and (2);

[0039]

[0040]

[0041] Among them, L feat Let denot be the feature alignment loss function, ca represent the global prototype feature corresponding to the third target image, cb represent the global prototype feature corresponding to the fourth target image, and p represent the feature alignment loss function. i Let Fbi represent the global confidence matrix corresponding to the fourth target image, and let L represent the feature data corresponding to the i-th pixel in the fourth target image. max Let H represent the maximum squared error loss function, H represent the length of the fourth target image, W represent the width of the fourth target image, c represent the category corresponding to the pseudo-label, C represent the total number of categories, and pi ,c This represents the probability that the i-th pixel in the fourth image belongs to category c.

[0042] The present invention also provides a model optimization apparatus, comprising:

[0043] A construction module is used to construct a first target image corresponding to each of the first images based on at least one image in the first dataset; the number of channels of the first target image is greater than the number of channels of the image.

[0044] The segmentation module is used to input the first target image into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on the second target image corresponding to the sample image in the second dataset, and the interactive segmentation model is used to segment object instances in the target image, wherein the second target image in the second dataset includes true label data;

[0045] The determining module is used to determine the pseudo-label data corresponding to the first target image based on the first feature data and the first segmentation result;

[0046] An optimization module is used to optimize the interactive segmentation model based on the pseudo-label data and the real label data.

[0047] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the model optimization method as described above.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model optimization method as described above.

[0049] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model optimization method as described above.

[0050] The model optimization method, apparatus, electronic device, and storage medium provided by this invention construct a first target image corresponding to each image based on at least one image in a first dataset; the number of channels in the first target image is greater than the number of channels in the image; the first target image is input into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on second target images corresponding to sample images in a second dataset, and the interactive segmentation model is used to segment object instances in the second target image, the second target image in the second dataset including true label data; based on the first feature data and the first segmentation result, pseudo label data corresponding to the first target image is determined; based on the pseudo label data and the true label data, the interactive segmentation model is optimized. The method provided by this invention determines the pseudo label data corresponding to the first target image through an interactive segmentation model, and then continuously trains and optimizes the interactive segmentation model using the images corresponding to the pseudo label data and the true label data, thereby optimizing the interactive segmentation model and improving the accuracy of the interactive segmentation model in segmenting object instances in images. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is one of the flowcharts illustrating the model optimization method provided by the present invention;

[0053] Figure 2 This is a schematic diagram of the construction of five-channel input data provided by the present invention;

[0054] Figure 3 This is the second flowchart illustrating the model optimization method provided by the present invention;

[0055] Figure 4 This is a schematic diagram of the optimization process of the interactive segmentation model provided by the present invention;

[0056] Figure 5 This is a schematic diagram of the processing procedure for multiple datasets provided by the present invention;

[0057] Figure 6 This is the third flowchart illustrating the model optimization method provided by this invention;

[0058] Figure 7 This is a schematic diagram of the model optimization device provided by the present invention;

[0059] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0061] The model optimization method provided by the present invention will be described in detail below with reference to the accompanying drawings, through some embodiments and application scenarios.

[0062] This invention provides a model optimization method applicable to image segmentation scenarios. The method involves constructing a first target image corresponding to each image based on at least one image in a first dataset; the number of channels in the first target image is greater than the number of channels in the image; inputting the first target image into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on second target images corresponding to sample images in a second dataset, and is used to segment object instances in the second target images, which include true label data; based on the first feature data and the first segmentation result, pseudo label data corresponding to the first target image is determined; and based on the pseudo label data and the true label data, the interactive segmentation model is optimized. The method provided by this invention determines the pseudo label data corresponding to the first target image through an interactive segmentation model, and then continuously trains and optimizes the interactive segmentation model using the images corresponding to the pseudo label data and the true label data, thereby optimizing the interactive segmentation model and improving its accuracy in segmenting object instances in images.

[0063] The following is combined Figures 1-6 The model optimization method of the present invention is described.

[0064] Figure 1 This is one of the flowcharts illustrating the model optimization method provided by this invention, such as... Figure 1 As shown, the method includes steps 101-104, wherein:

[0065] Step 101: Based on at least one image in the first dataset, construct a first target image corresponding to each of the images; the number of channels of the first target image is greater than the number of channels of the image.

[0066] It should be noted that the model optimization method provided by this invention is applicable to image segmentation scenarios. The execution entity of this method can be a model optimization device, such as an electronic device, or a control module within that model optimization device for executing the model optimization method.

[0067] Specifically, the first dataset includes multiple images, which are RGB color images. Each pixel contains three channels and has no true label data, meaning the images in the first dataset have no annotation information. Based on the multiple images in the first dataset, and the interaction information of each object instance corresponding to each RGB color image, and each RGB color image, five-channel input data can be constructed. That is, each image corresponds to a five-channel first target image. Therefore, the number of channels in the first target image is greater than the number of channels in each image.

[0068] It should be noted that the interaction information of each object instance consists of two diagonal background points and one foreground point, which form a point pair simulating user interaction. A Gaussian map of the background points can be generated based on the two diagonal background points, and a Gaussian map of the foreground point can be generated based on the foreground point.

[0069] Figure 2 This is a schematic diagram of the construction of five-channel input data provided by the present invention, as shown below. Figure 2 As shown, Figure 2 Image a is the RGB image of the object, image b is a Gaussian map of the foreground point generated from a foreground point, image c is a Gaussian map of the background points generated from two diagonal background points, and image d is the constructed five-channel input data, which is the first target image corresponding to image a.

[0070] Step 102: Input the first target image into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on the second target image corresponding to the sample image in the second dataset, and the interactive segmentation model is used to segment object instances in the second target image, the second target image in the second dataset includes true label data.

[0071] Specifically, the first target image with five-channel input data is input into the interactive segmentation model, and the interactive segmentation model is used to segment the first target image to obtain the first feature data and the first segmentation result output by the interactive segmentation model, namely the first feature data F output by the backbone network in the interactive segmentation model and the preliminary first segmentation result M.

[0072] It should be noted that before the interactive segmentation model outputs the first segmentation result, the output result of the interactive segmentation model is binarized to obtain the first segmentation result, that is, the value of each pixel in the first segmentation result is 0 or 1.

[0073] In practice, the interactive segmentation model is trained on the second target image corresponding to the sample images in the second dataset. The interactive segmentation model is used to segment object instances in the second target image, which includes true label data. Before training the interactive segmentation model using the second target image corresponding to the sample images in the second dataset, the sample images in the second dataset are RGB color images, each pixel contains three channels and corresponds to one true label data, meaning that the sample images in the second dataset have annotation information.

[0074] Based on multiple sample images in the second dataset, five-channel input data is constructed for each object instance corresponding to each RGB color sample image, along with the interaction information and the RGB color sample image itself. Each sample image corresponds to a five-channel second target image, with the second target image having more channels than the sample images. The interaction information for each object instance also consists of a pair of points simulating user interaction, composed of two diagonal background points and one foreground point. A Gaussian map of the background points is generated based on the two diagonal background points, and a Gaussian map of the foreground point is generated based on the foreground point.

[0075] After the constructed second target image with five-channel input data undergoes data augmentation processing such as rotation, cutting, and scaling, it is normalized to a uniform size, for example, a 5×512×512 pixel image. The data-augmented second target image is then input into a multi-neural network framework, namely a residual network (ResNet). ResNet is used to perform supervised training on the interactive segmentation model to obtain the segmentation result. During the training process, the loss value of the loss function is calculated to determine whether the training has converged. Training ends when the convergence condition is met.

[0076] Step 103: Based on the first feature data and the first segmentation result, determine the pseudo-label data corresponding to the first target image.

[0077] Specifically, based on the first feature data F and the first segmentation result M output by the interactive segmentation model, the pseudo-label data corresponding to the first target image can be determined.

[0078] Step 104: Optimize the interactive segmentation model based on the pseudo-label data and the real label data.

[0079] Specifically, based on the pseudo-label data corresponding to the first target image in the first dataset and the real label data corresponding to the second target image in the second dataset, the interactive segmentation model can be further optimized, so that the interactive segmentation model can achieve accurate segmentation results for input images without real label data even without real label supervision.

[0080] The model optimization method provided by this invention constructs a first target image corresponding to each image based on at least one image in a first dataset; the number of channels in the first target image is greater than the number of channels in the image; the first target image is input into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on second target images corresponding to sample images in a second dataset, and the interactive segmentation model is used to segment object instances in the second target images, which include true label data; based on the first feature data and the first segmentation result, pseudo label data corresponding to the first target image is determined; based on the pseudo label data and the true label data, the interactive segmentation model is optimized. The method provided by this invention determines the pseudo label data corresponding to the first target image through an interactive segmentation model, and then continuously trains and optimizes the interactive segmentation model using the images corresponding to the pseudo label data and the true label data, thereby optimizing the interactive segmentation model and improving the accuracy of the interactive segmentation model in segmenting object instances in images.

[0081] Figure 3 This is the second flowchart illustrating the model optimization method provided by this invention, as shown below. Figure 3 As shown, the method includes steps 301-305, wherein:

[0082] Step 301: Based on at least one image in the first dataset, construct a first target image corresponding to each of the images; the number of channels of the first target image is greater than the number of channels of the image.

[0083] Step 302: Input the first target image into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on the second target image corresponding to the sample image in the second dataset, and the interactive segmentation model is used to segment object instances in the second target image, the second target image in the second dataset includes true label data.

[0084] Step 303: Based on the first feature data and the first segmentation result, determine at least one prototype feature of the object instance.

[0085] Specifically, based on the first feature data and the first segmentation result output by the interactive segmentation model, at least one prototype feature of the object instance in the first target image can be determined.

[0086] Step 304: Based on each of the prototype features, determine the pseudo-label data corresponding to the first target image.

[0087] Specifically, the prototype features of the object instance can determine the pseudo-label data corresponding to the first target image.

[0088] Step 305: Optimize the interactive segmentation model based on the pseudo-label data and the real label data.

[0089] The model optimization method provided by this invention determines at least one prototype feature of an object instance in a first target image based on first feature data and first segmentation results. Then, based on each prototype feature, it determines the pseudo-label data corresponding to the first target image. Subsequently, it optimizes the interactive segmentation model based on the pseudo-label data and the true label data, thereby improving the accuracy of the interactive segmentation model in segmenting object instances in the image.

[0090] Optionally, the at least one prototype feature includes a global prototype feature, a central prototype feature, and a peripheral prototype feature; the specific implementation of step 303 above includes the following steps:

[0091] Step 1) Based on the first feature data and the first segmentation result, determine the second feature data corresponding to the pixels in the first feature data that belong to the foreground position in the first segmentation result.

[0092] Specifically, based on the first feature data F and the first segmentation result M, the pixel value of the foreground point position in the first segmentation result M is 1. That is, by corresponding the position of the foreground point with the pixel value of 1 in the first segmentation result M with the position of each pixel in the first feature data F, the pixels in the first feature data F that belong to the foreground point position in the first segmentation result M can be determined respectively, and the second feature data corresponding to each pixel can be determined based on the determined multiple pixels.

[0093] Step 2) Summing multiple second feature data and averaging the sums to obtain the global prototype feature of the object instance.

[0094] Specifically, multiple second feature data are summed, and the sums are averaged to obtain the global prototype feature c of the object instance. g .

[0095] Step 3) Randomly select x first foreground locations from the multiple foreground locations corresponding to the first segmentation result; x is a positive integer.

[0096] Specifically, for each pixel with a value of 1 in the first segmentation result M, x first foreground point positions are randomly selected from the foreground point positions corresponding to each pixel, where x is a positive integer.

[0097] Step 4) Based on the x first foreground point locations, the second foreground point location corresponding to the first target image, and the first feature data, determine the third feature data corresponding to the pixels of the first foreground point location and the second foreground point location, respectively.

[0098] Specifically, the second foreground point position corresponding to the first target image is the foreground point of the object instance in the foreground interaction information provided during user interaction when constructing the five-channel input data. Based on x randomly selected first foreground point positions, the second foreground point position corresponding to the first target image, and the first feature data F, the feature data corresponding to the pixels belonging to the x first foreground point positions in the first feature data F and the feature data corresponding to the pixels belonging to the second foreground point positions in the first feature data F are respectively used as third feature data, thereby determining the third feature data corresponding to the pixels at the first and second foreground point positions.

[0099] Step 5) Summing up multiple third feature data and averaging the sums to obtain the central prototype feature of the object instance.

[0100] Specifically, by summing multiple third feature data and averaging the sums, the central prototype feature c of the object instance can be obtained. c .

[0101] Step 6) Erode the first segmentation result to obtain the edge position corresponding to the first segmentation result.

[0102] Specifically, by performing an erosion operation in computer graphics on the first segmentation result M, the position of each pixel in the edge region can be obtained, that is, the edge position corresponding to the first segmentation result M can be obtained.

[0103] Step 7) Sum the fourth feature data corresponding to the pixels at the edge position, and average the sum to obtain the edge prototype feature of the object instance.

[0104] Specifically, the feature data corresponding to the pixels at the positions of each pixel in the edge region in the first feature data F are taken as the fourth feature data. Then, the fourth feature data are summed, and the sums are averaged to obtain the edge prototype feature c of the object instance. e .

[0105] The model optimization method provided by this invention determines the second feature data corresponding to the pixels belonging to the foreground position in the first segmentation result using first feature data and a first segmentation result; sums multiple second feature data and averages the sums to obtain the global prototype feature of the object instance; randomly selects x first foreground positions from multiple foreground positions corresponding to the first segmentation result; x is a positive integer; based on the x first foreground positions, the second foreground position corresponding to the first target image, and the first feature data, determines the third feature data corresponding to the pixels of the first and second foreground positions respectively; sums multiple third feature data and averages the sums to obtain the central prototype feature of the object instance; erodes the first segmentation result to obtain the edge position corresponding to the first segmentation result; sums the fourth feature data corresponding to the pixels of the edge position and averages the sums to obtain the edge prototype feature of the object instance. This method realizes the calculation of three prototype features of the object instance in the first target image. Using three prototype features can provide more accurate pseudo-label data, thereby further optimizing the interactive segmentation model based on pseudo-label data and true label data, improving the accuracy of the interactive segmentation model in segmenting object instances in the image.

[0106] Optionally, the specific implementation of step 304 above includes the following steps:

[0107] Step 1) Based on the global prototype feature, the central prototype feature, and the edge prototype feature, calculate the squared Euclidean distance corresponding to the feature vector of each pixel.

[0108] Specifically, based on the global prototype feature c g Central prototype feature c c and edge prototype features c e Calculate the feature vector of each pixel in the first feature data F and the global prototype feature c respectively. g Central prototype feature c c and edge prototype features c e The square Euclidean distance d(c,F) i This allows us to obtain the global prototype feature c. g The corresponding squared Euclidean distance and central prototype feature c c The corresponding squared Euclidean distance and edge prototype feature c e The corresponding squared Euclidean distance is calculated using the following formula (3), where:

[0109] d(c,F i )=||cF i || 2 (3)

[0110] Where c represents one of the global prototype feature, the central prototype feature, and the peripheral prototype feature, F i Let d(c,F) represent the feature vector of pixel i. i ) represents the squared Euclidean distance.

[0111] Step 2) Based on each of the squared Euclidean distances, determine at least one confidence matrix; the confidence matrix includes at least one of the following: global confidence matrix, central confidence matrix, and marginal confidence matrix.

[0112] Specifically, based on each squared Euclidean distance, using the following formula (4), at least one confidence matrix can be determined, thus obtaining the global confidence matrix P. g , central confidence matrix P c Marginal confidence matrix P e ,in:

[0113]

[0114] Where c represents one of the global prototype feature, the central prototype feature, and the peripheral prototype feature, F i Let d(c,F) represent the feature vector of pixel i. i ) represents the squared Euclidean distance. This represents the prototype feature obtained when the position of pixel i is inverted.

[0115] Step 3) Perform a weighted summation of the center confidence matrix and the edge confidence matrix to obtain a weighted summation confidence matrix; This represents the prototype feature obtained when the position of pixel i is inverted.

[0116] Specifically, since the foreground point has already been represented in the interactive segmentation model as one channel input data of the first target image, only the center confidence matrix P is used here. c and marginal confidence matrix P e We perform a weighted summation to obtain the weighted summation confidence matrix P. m Where is the marginal confidence matrix P e The assigned weights are compared to the center confidence matrix P. c The high weight values ​​assigned make the interactive segmentation model pay more attention to the feature data at the edge positions, and the center confidence matrix P c and marginal confidence matrix P e The closer the value in the matrix is ​​to 1, the closer the feature data of that pixel is to the center confidence matrix P. c and marginal confidence matrix P e The corresponding central prototype features c c and edge prototype features c eThe more similar the pixels are, the more likely they should be considered as foreground pixels.

[0117] Step 4) Based on the global confidence matrix and the weighted summation confidence matrix, determine the first confidence value corresponding to the global confidence matrix and the second confidence value corresponding to the weighted summation confidence matrix.

[0118] Specifically, based on the global confidence matrix P g The weighted summation confidence matrix P m The global confidence matrix P is respectively... g The weighted summation confidence matrix P m The values ​​corresponding to the positions of each pixel in the first segmentation result that are foreground points are summed, and the sums are averaged to obtain the global confidence matrix P. g The corresponding first confidence value and weighted summation confidence matrix P m The corresponding second confidence value.

[0119] Step 5) Based on the first confidence value and the second confidence value, determine the second segmentation result corresponding to the first confidence value and the third segmentation result corresponding to the second confidence value.

[0120] Specifically, based on the first confidence value and the second confidence value, each of the first and second confidence values ​​is multiplied by a parameter to obtain a threshold value, which is then applied to the global confidence matrix P. g The weighted summation confidence matrix P m The values ​​in the data are filtered, with values ​​greater than the threshold being 1 and values ​​less than the threshold being 0, thus obtaining the second segmentation result corresponding to the first confidence value and the third segmentation result corresponding to the second confidence value.

[0121] Step 6) Based on the first segmentation result, the second segmentation result, and the third segmentation result, determine the pseudo-label data corresponding to the first target image.

[0122] Specifically, based on the first segmentation result M and the second segmentation result M g and the third segmentation result M m This allows us to determine the pseudo-label data corresponding to the first target image.

[0123] The model optimization method provided by this invention calculates the squared Euclidean distance corresponding to the feature vector of each pixel using global prototype features, central prototype features, and edge prototype features. Based on these squared Euclidean distances, it determines the global confidence matrix, central confidence matrix, and edge confidence matrix. The central confidence matrix and edge confidence matrices are then weighted and summed to obtain a weighted confidence matrix. Based on the global confidence matrix and the weighted confidence matrix, a first confidence value corresponding to the global confidence matrix and a second confidence value corresponding to the weighted confidence matrix are determined. Finally, based on the first and second confidence values... The system determines the second segmentation result corresponding to the first confidence value and the third segmentation result corresponding to the second confidence value. Based on the first, second, and third segmentation results, it determines the pseudo-label data corresponding to the first target image. This achieves the determination of pseudo-label data corresponding to the first target image based on global prototype features, central prototype features, and edge prototype features, thus optimizing the first segmentation result and making the obtained pseudo-label data more accurate. Therefore, the interactive segmentation model can be further optimized based on the pseudo-label data and the true label data, thereby improving the accuracy of the interactive segmentation model in segmenting object instances in the image.

[0124] Optionally, the specific implementation of step 6) above includes the following steps:

[0125] Step 6-1) Based on the first segmentation result, the second segmentation result and the third segmentation result, determine whether the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold.

[0126] Specifically, based on the first segmentation result M and the second segmentation result M g and the third segmentation result M m A voting mechanism is used to determine whether the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold, for example, the preset threshold is 2.

[0127] Step 6-2) If the sum of the values ​​corresponding to the position of each pixel is greater than the preset threshold, the pixel is a foreground pixel.

[0128] Specifically, if the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold, that is, if there are two or more matrix values ​​of 1 corresponding to the position of each pixel, then the pixel at that position is considered to be a foreground pixel.

[0129] (Step 6-3) If the sum of the values ​​corresponding to the position of each pixel is not greater than the preset threshold, the pixel is a background pixel.

[0130] Specifically, if the sum of the values ​​corresponding to each pixel position is not greater than a preset threshold, that is, if there is only one or no matrix value of 1 for each pixel position, then the pixel at that position is considered to be a background pixel, thus obtaining the optimized fourth segmentation result M'. The fourth segmentation result M' is used as the output result of the pseudo-label optimization module, that is, the pseudo-label data corresponding to the first target image.

[0131] The model optimization method provided by this invention uses a voting mechanism based on the first, second, and third segmentation results to determine whether the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold. If the sum of the values ​​corresponding to the position of each pixel is greater than the preset threshold, the pixel is a foreground pixel; if the sum of the values ​​corresponding to the position of each pixel is not greater than the preset threshold, the pixel is a background pixel. This yields pseudo-label data corresponding to the first target image. The obtained pseudo-label data has higher accuracy, and the interactive segmentation model can be further optimized based on the pseudo-label data and the true label data, thereby improving the accuracy of the interactive segmentation model in segmenting object instances in the image.

[0132] Optionally, the specific implementation of step 305 above includes the following steps:

[0133] Step 1) Based on the pseudo-label data and the real label data, sample the third target image in the first dataset.

[0134] Specifically, based on the pseudo-label data of the first target image, a classifier is used to sample the third target image in the first dataset.

[0135] 2) Based on the third target image and the true label data, determine the fourth target image in the second dataset.

[0136] Specifically, after the classifier samples the third target image in the first dataset, the classifier determines the category of the third target image, searches for a fourth target image in the second dataset whose true label data has the same category as the third target image, samples the fourth target image, and thus determines the fourth target image in the second dataset.

[0137] Step 3) Optimize the interactive segmentation model based on the third target image and the fourth target image.

[0138] Specifically, the interactive segmentation model is trained in the next stage based on the third and fourth target images to further optimize the interactive segmentation model.

[0139] Optionally, based on the third target image and the fourth target image, the interactive segmentation model is optimized using formulas (1) and (2);

[0140]

[0141]

[0142] Among them, L feat c represents the feature alignment loss function. a c represents the global prototype feature corresponding to the third target image. b p represents the global prototype feature corresponding to the fourth target image. i F represents the global confidence matrix corresponding to the fourth target image. bi L represents the feature data corresponding to the i-th pixel in the fourth target image. max Let H represent the maximum squared error loss function, H represent the length of the fourth target image, W represent the width of the fourth target image, c represent the category corresponding to the pseudo-label, C represent the total number of categories, and p represent the maximum squared error loss function. i,c This represents the probability that the i-th pixel in the fourth image belongs to category c.

[0143] Specifically, the third and fourth target images are input into the interactive segmentation model, respectively, to obtain the fifth feature data and fifth segmentation result corresponding to the third target image, and the sixth feature data and sixth segmentation result corresponding to the fourth target image. Then, based on the fifth feature data and fifth segmentation result corresponding to the third target image, the global prototype feature of the third target image is calculated. Similarly, based on the sixth feature data and sixth segmentation result corresponding to the fourth target image, the global prototype feature of the fourth target image is calculated. For example, the global prototype feature c... a and global prototype feature c b .

[0144] Based on the obtained global prototype feature c a and global prototype feature c b For global prototype feature c a and global prototype feature c b Alignment is performed, and the loss values ​​of the feature alignment loss function and the maximum squared error loss function are calculated according to the above formulas (1) and (2), respectively. Then, the parameters of the interactive segmentation model are updated according to the calculated loss values ​​of the feature alignment loss function and the maximum squared error loss function until the interactive segmentation model converges.

[0145] The interactive segmentation model was continuously trained on multiple unlabeled images in the first dataset. After training, the segmentation accuracy of the interactive segmentation model on the images in the first dataset was significantly improved, while maintaining the model's performance on images in the second dataset. This allowed the interactive segmentation model to evolve from the initialization of the dataset including the real-labeled dataset to the unlabeled dataset. In the case of the unlabeled dataset, the interactive segmentation model could accurately segment the object instances in the image.

[0146] The model optimization method provided by this invention samples the third target image in the first dataset using pseudo-labeled data; then, based on the third target image and the true label data, it determines the fourth target image in the second dataset; and then, based on the third target image and the fourth target image, it continuously optimizes the interactive segmentation model, thereby enabling the interactive segmentation model to evolve from an initial dataset including a true label dataset to an unlabeled dataset. In the case of an unlabeled dataset, the interactive segmentation model can accurately segment object instances in the image, improving the accuracy of the interactive segmentation model in segmenting object instances in the image.

[0147] Figure 4 This is a schematic diagram of the optimization process of the interactive segmentation model provided by the present invention, as shown below. Figure 4 As shown, in the initialization phase, a five-channel image corresponding to the input data is constructed for each object instance in dataset A. The interaction information of each object instance consists of two diagonal background points and one foreground point, forming a point pair that simulates user interaction. A background point Gaussian map is generated based on the two diagonal background points, and a foreground point Gaussian map is generated based on one foreground point, thus obtaining the second target image corresponding to each image in dataset A. After performing data augmentation processing such as rotation, cutting, and scaling on the second target image corresponding to the five-channel input data, it is input into the interactive segmentation model, and the interactive segmentation model is trained in a supervised manner.

[0148] After training, the pseudo-label optimization module based on prototype features uses the trained interactive segmentation model to predict and segment object instances in each image of dataset B. Specifically, it constructs a five-channel input image for each object instance in dataset B and inputs it into the interactive segmentation model to obtain the first feature data and the first segmentation result (coarse mask). Then, based on the first feature data and the first segmentation result, it obtains global prototype features, center prototype features, and edge prototype features. Next, it calculates the global confidence matrix, center confidence matrix, and edge confidence matrix based on the global prototype features, center prototype features, and edge prototype features. It then performs a weighted summation of the center confidence matrix and the edge confidence matrix to obtain a weighted summation confidence matrix. Finally, based on the weighted summation confidence matrix and the global confidence matrix, it determines the segmentation result M. g and segmentation result Mm Then, a voting mechanism is used to determine the pseudo-label data corresponding to each image in dataset B.

[0149] In the next training phase, namely the training module based on image feature alignment, the interactive segmentation model is trained using images from dataset A (including true-label data) and dataset B (including pseudo-label data) sampled by the classifier. This yields the feature data and segmentation results output by the interactive segmentation model. Then, based on the feature data and segmentation results, the global prototype feature c corresponding to the image in dataset A is calculated. a The global prototype feature c corresponding to the image in data B b For the global prototype features c corresponding to the images in dataset A a The global prototype feature c corresponding to the image in data B b Alignment is performed, and the feature alignment loss function and the maximum squared error loss function are calculated to optimize the interactive segmentation model.

[0150] Figure 5 This is a schematic diagram illustrating the processing procedure for multiple datasets provided by the present invention, such as... Figure 5 As shown, in the initialization phase, the interactive segmentation model is trained using dataset A, which includes true labeled data. Then, images from dataset B, which does not include true labeled data, are input into the interactive segmentation model to obtain the feature data and segmentation results output by the interactive segmentation model. Based on the feature data and segmentation results, prototype features (global prototype features, central prototype features, and edge prototype features) are obtained, thus yielding the pseudo-label data for dataset B. Next, images of the same category from datasets A and B are sampled to optimize the interactive segmentation model using an inter-category sampling strategy. This allows the interactive segmentation model to evolve from dataset A, which includes true labeled data, to dataset B, which contains unlabeled data. In the case of unlabeled datasets, it can accurately segment object instances in images within the unlabeled dataset.

[0151] Figure 6 This is the third flowchart of the model optimization method provided by the present invention, as shown below. Figure 6 As shown, the process includes the following steps 601-610:

[0152] Step 601: Construct an image with five-channel input data. For multiple sample images in the second dataset, construct a second target image with five-channel input data, where the second target image in the second dataset includes true label data.

[0153] Step 602: Initialize the interactive segmentation model. Input the second target image into the interactive segmentation model, perform supervised training on the interactive segmentation model, and obtain the segmentation result.

[0154] Step 603, segment the target dataset. Input the first target images corresponding to multiple images in the first dataset into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model.

[0155] Step 604: Calculate prototype features. Based on the first feature data and the first segmentation result, calculate the global prototype features, central prototype features, and edge prototype features of the object instance.

[0156] Step 605: Obtain the corresponding confidence matrix. Based on the global prototype features, central prototype features, and edge prototype features of the object instance, calculate the global confidence matrix, central confidence matrix, and edge confidence matrix for each pixel.

[0157] Step 606: Obtain additional auxiliary coarse mask. The center confidence matrix and the edge confidence matrix are weighted and summed to obtain a weighted sum confidence matrix. Based on the weighted sum confidence matrix and the global confidence matrix, the confidence values ​​corresponding to the weighted sum confidence matrix and the global confidence matrix are obtained respectively. These confidence values ​​are multiplied by a parameter to serve as thresholds for filtering, resulting in a second segmentation result and a third segmentation result with the same size as the first segmentation result. The second and third segmentation results are the auxiliary coarse mask.

[0158] Step 607: Use a voting mechanism. Based on the first segmentation result, the second segmentation result, and the third segmentation result, use a voting mechanism to determine whether the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold.

[0159] Step 608: Obtain the pseudo-label data corresponding to the first target image. If the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold, the pixel is determined to be a foreground pixel; if the sum of the values ​​corresponding to the position of each pixel is not greater than the preset threshold, the pixel is determined to be a background pixel.

[0160] Step 609: Sample the third and fourth target images that have the same category as those in the first and second datasets.

[0161] Step 610: Optimize the interactive segmentation model. Sample third and fourth target images of the same category from the first and second datasets to further train the interactive segmentation model. During training, the interactive segmentation model is continuously optimized using the feature alignment loss function and the maximum squared error loss function. The trained interactive segmentation model can significantly improve the segmentation accuracy of images in the first dataset while maintaining its performance on images in the second dataset.

[0162] The model optimization method provided by this invention constructs an image of five-channel input data, initializes an interactive segmentation model using the image, segments the target dataset using the interactive segmentation model, calculates prototype features based on the first feature data output by the interactive segmentation model and the first segmentation result, calculates the corresponding confidence matrix based on the prototype features, obtains an additional auxiliary coarse mask based on the confidence matrix, obtains pseudo-label data corresponding to the first target image using a voting mechanism, and optimizes the interactive segmentation model by sampling third and fourth target images of the same category from the first and second datasets. This optimization of the interactive segmentation model enables it to accurately segment object instances in images even with unlabeled datasets, improving the accuracy of object instance segmentation in images.

[0163] The model optimization apparatus provided by the present invention is described below. The model optimization apparatus described below and the model optimization method described above can be referred to in correspondence.

[0164] Figure 7 This is a schematic diagram of the model optimization device provided by the present invention, as shown below. Figure 7 As shown, the model optimization device 700 includes: a construction module 701, a segmentation module 702, a determination module 703, and an optimization module 704; wherein,

[0165] The construction module 701 is used to construct a first target image corresponding to each of the first images based on at least one image in the first dataset; the number of channels of the first target image is greater than the number of channels of the image.

[0166] The segmentation module 702 is used to input the first target image into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on the second target image corresponding to the sample image in the second dataset, and the interactive segmentation model is used to segment object instances in the target image, wherein the second target image in the second dataset includes true label data;

[0167] The determining module 703 is used to determine the pseudo-label data corresponding to the first target image based on the first feature data and the first segmentation result;

[0168] The optimization module 704 is used to optimize the interactive segmentation model based on the pseudo-label data and the real label data.

[0169] The model optimization apparatus provided by this invention constructs a first target image corresponding to each image based on at least one image in a first dataset; the number of channels in the first target image is greater than the number of channels in the image; the first target image is input into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on second target images corresponding to sample images in a second dataset, and the interactive segmentation model is used to segment object instances in the second target images, the second target images in the second dataset including true label data; based on the first feature data and the first segmentation result, pseudo label data corresponding to the first target image is determined; based on the pseudo label data and the true label data, the interactive segmentation model is optimized. The method provided by this invention determines the pseudo label data corresponding to the first target image through an interactive segmentation model, and then continuously trains and optimizes the interactive segmentation model using the images corresponding to the pseudo label data and the true label data, thereby optimizing the interactive segmentation model and improving the accuracy of the interactive segmentation model in segmenting object instances in images.

[0170] Optionally, the determining module 703 is specifically used for:

[0171] Based on the first feature data and the first segmentation result, at least one prototype feature of the object instance is determined;

[0172] Based on the aforementioned prototype features, the pseudo-label data corresponding to the first target image is determined.

[0173] Optionally, the at least one prototype feature includes global prototype features, central prototype features, and peripheral prototype features; the determining module 703 is specifically used for:

[0174] Based on the first feature data and the first segmentation result, determine the second feature data corresponding to the pixels in the first feature data that belong to the foreground point position in the first segmentation result;

[0175] The sums of multiple second feature data are calculated, and the sums are averaged to obtain the global prototype feature of the object instance.

[0176] Randomly select x first foreground locations from the multiple foreground locations corresponding to the first segmentation result; x is a positive integer.

[0177] Based on the x first foreground point locations, the second foreground point location corresponding to the first target image, and the first feature data, the third feature data corresponding to the pixels of the first foreground point location and the second foreground point location are determined respectively.

[0178] The central prototype feature of the object instance is obtained by summing multiple third feature data and averaging the sums.

[0179] The first segmentation result is eroded to obtain the edge position corresponding to the first segmentation result;

[0180] The fourth feature data corresponding to the pixels at the edge positions are summed, and the sums are averaged to obtain the edge prototype features of the object instance.

[0181] Optionally, the determining module 703 is specifically used for:

[0182] Based on the global prototype feature, the central prototype feature, and the edge prototype feature, the squared Euclidean distance corresponding to the feature vector of each pixel is calculated respectively;

[0183] Based on the squared Euclidean distances, at least one confidence matrix is ​​determined; the confidence matrix includes at least one of the following: global confidence matrix, center confidence matrix, and marginal confidence matrix;

[0184] The center confidence matrix and the edge confidence matrix are weighted and summed to obtain the weighted sum confidence matrix;

[0185] Based on the global confidence matrix and the weighted summation confidence matrix, determine the first confidence value corresponding to the global confidence matrix and the second confidence value corresponding to the weighted summation confidence matrix;

[0186] Based on the first confidence value and the second confidence value, determine the second segmentation result corresponding to the first confidence value and the third segmentation result corresponding to the second confidence value;

[0187] Based on the first segmentation result, the second segmentation result, and the third segmentation result, the pseudo-label data corresponding to the first target image is determined.

[0188] Optionally, the determining module 703 is specifically used for:

[0189] Based on the first segmentation result, the second segmentation result, and the third segmentation result, determine whether the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold;

[0190] If the sum of the values ​​corresponding to the position of each pixel is greater than the preset threshold, the pixel is a foreground pixel;

[0191] If the sum of the values ​​corresponding to the position of each pixel is not greater than the preset threshold, the pixel is a background pixel.

[0192] Optionally, the optimization module 704 is specifically used for:

[0193] Based on the pseudo-label data, sample the third target image from the first dataset;

[0194] Based on the third target image and the true label data, determine the fourth target image in the second dataset;

[0195] The interactive segmentation model is optimized based on the third target image and the fourth target image.

[0196] Optionally, the optimization module 704 is specifically used for:

[0197] Based on the third target image and the fourth target image, the interactive segmentation model is optimized using formulas (1) and (2);

[0198]

[0199]

[0200] Among them, L feat c represents the feature alignment loss function. a c represents the global prototype feature corresponding to the third target image. b p represents the global prototype feature corresponding to the fourth target image. i F represents the global confidence matrix corresponding to the fourth target image. bi L represents the feature data corresponding to the i-th pixel in the fourth target image. max Let H represent the maximum squared error loss function, H represent the length of the fourth target image, W represent the width of the fourth target image, c represent the category corresponding to the pseudo-label, C represent the total number of categories, and p represent the maximum squared error loss function. i,c This represents the probability that the i-th pixel in the fourth image belongs to category c.

[0201] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 8As shown, the electronic device 800 may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a model optimization method, which includes: constructing a first target image corresponding to each image based on at least one image in a first dataset; the first target image has more channels than the number of channels in the image; inputting the first target image into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on second target images corresponding to sample images in a second dataset, and the interactive segmentation model is used to segment object instances in the second target image, the second target image in the second dataset including true label data; determining pseudo label data corresponding to the first target image based on the first feature data and the first segmentation result; and optimizing the interactive segmentation model based on the pseudo label data and the true label data.

[0202] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model optimization method provided by the above methods. The method includes: constructing a first target image corresponding to each of the images based on at least one image in a first dataset; the number of channels of the first target image is greater than the number of channels of the image; inputting the first target image into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on a second target image corresponding to a sample image in a second dataset, the interactive segmentation model being used to segment object instances in the second target image, the second target image in the second dataset including true label data; determining pseudo label data corresponding to the first target image based on the first feature data and the first segmentation result; and optimizing the interactive segmentation model based on the pseudo label data and the true label data.

[0204] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the model optimization method provided by the above methods. The method includes: constructing a first target image corresponding to each of the images based on at least one image in a first dataset; the number of channels in the first target image is greater than the number of channels in the image; inputting the first target image into an interactive segmentation model to obtain first feature data and a first segmentation result output by the interactive segmentation model; the interactive segmentation model is trained based on a second target image corresponding to a sample image in a second dataset, the interactive segmentation model being used to segment object instances in the second target image, the second target image in the second dataset including true label data; determining pseudo label data corresponding to the first target image based on the first feature data and the first segmentation result; and optimizing the interactive segmentation model based on the pseudo label data and the true label data.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model optimization method, characterized in that, include: Based on at least one image in the first dataset, construct a first target image corresponding to each of the images; The first target image has more channels than the image has. The first target image is input into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model. The interactive segmentation model is trained based on the second target image corresponding to the sample images in the second dataset. The interactive segmentation model is used to segment object instances in the second target image. The second target image in the second dataset includes true label data. Based on the first feature data and the first segmentation result, the pseudo-label data corresponding to the first target image is determined; Based on the pseudo-label data and the real label data, optimize the interactive segmentation model; The step of determining the pseudo-label data corresponding to the first target image based on the first feature data and the first segmentation result includes: Based on the first feature data and the first segmentation result, at least one prototype feature of the object instance is determined; Based on the aforementioned prototype features, the pseudo-label data corresponding to the first target image is determined; The at least one prototype feature includes a global prototype feature, a central prototype feature, and a peripheral prototype feature; Determining at least one prototype feature of the object instance based on the first feature data and the first segmentation result includes: Based on the first feature data and the first segmentation result, determine the second feature data corresponding to the pixels in the first feature data that belong to the foreground point position in the first segmentation result; The sums of multiple second feature data are calculated, and the sums are averaged to obtain the global prototype feature of the object instance. Randomly select x first foreground locations from the multiple foreground locations corresponding to the first segmentation result; x is a positive integer. Based on the x first foreground point locations, the second foreground point location corresponding to the first target image, and the first feature data, the third feature data corresponding to the pixels of the first foreground point location and the second foreground point location are determined respectively. The central prototype feature of the object instance is obtained by summing multiple third feature data and averaging the sums. The first segmentation result is eroded to obtain the edge position corresponding to the first segmentation result; The fourth feature data corresponding to the pixels at the edge positions are summed, and the sums are averaged to obtain the edge prototype features of the object instance. The step of determining the pseudo-label data corresponding to the first target image based on each of the prototype features includes: Based on the global prototype feature, the central prototype feature, and the edge prototype feature, the squared Euclidean distance corresponding to the feature vector of each pixel is calculated respectively; Based on the squared Euclidean distances, at least one confidence matrix is ​​determined; the confidence matrix includes at least one of the following: global confidence matrix, center confidence matrix, and marginal confidence matrix; The center confidence matrix and the edge confidence matrix are weighted and summed to obtain the weighted sum confidence matrix; Based on the global confidence matrix and the weighted summation confidence matrix, determine the first confidence value corresponding to the global confidence matrix and the second confidence value corresponding to the weighted summation confidence matrix; Based on the first confidence value and the second confidence value, determine the second segmentation result corresponding to the first confidence value and the third segmentation result corresponding to the second confidence value; Based on the first segmentation result, the second segmentation result, and the third segmentation result, the pseudo-label data corresponding to the first target image is determined; The step of optimizing the interactive segmentation model based on the pseudo-label data and the real label data includes: Based on the pseudo-label data, sample the third target image from the first dataset; Based on the third target image and the true label data, determine the fourth target image in the second dataset; The interactive segmentation model is optimized based on the third target image and the fourth target image.

2. The model optimization method according to claim 1, characterized in that, The step of determining the pseudo-label data corresponding to the first target image based on the first segmentation result, the second segmentation result, and the third segmentation result includes: Based on the first segmentation result, the second segmentation result, and the third segmentation result, determine whether the sum of the values ​​corresponding to the position of each pixel is greater than a preset threshold; If the sum of the values ​​corresponding to the position of each pixel is greater than the preset threshold, the pixel is a foreground pixel; If the sum of the values ​​corresponding to the position of each pixel is not greater than the preset threshold, the pixel is a background pixel.

3. The model optimization method according to claim 1, characterized in that, The step of optimizing the interactive segmentation model based on the third target image and the fourth target image includes: Based on the third target image and the fourth target image, the interactive segmentation model is optimized using formulas (1) and (2); （1） （2） in, This represents the feature alignment loss function. This represents the global prototype features corresponding to the third target image. This represents the global prototype features corresponding to the fourth target image. This represents the global confidence matrix corresponding to the fourth target image. This represents the feature data corresponding to the i-th pixel in the fourth target image. Let H represent the maximum squared error loss function, H represent the length of the fourth target image, W represent the width of the fourth target image, c represent the category corresponding to the pseudo-label, and C represent the total number of categories. This represents the probability that the i-th pixel in the fourth target image belongs to category c.

4. A model optimization device, characterized in that, include: A construction module is used to construct a first target image corresponding to each of the images based on at least one image in the first dataset. The first target image has more channels than the image has. The segmentation module is used to input the first target image into the interactive segmentation model to obtain the first feature data and the first segmentation result output by the interactive segmentation model. The interactive segmentation model is trained based on the second target image corresponding to the sample image in the second dataset. The interactive segmentation model is used to segment object instances in the target image. The second target image in the second dataset includes true label data. The determining module is used to determine the pseudo-label data corresponding to the first target image based on the first feature data and the first segmentation result; An optimization module is used to optimize the interactive segmentation model based on the pseudo-label data and the real label data; The determining module is specifically used for: Based on the first feature data and the first segmentation result, determine the second feature data corresponding to the pixels in the first feature data that belong to the foreground point position in the first segmentation result; Summing multiple second feature data and averaging the sums yields the global prototype features of the object instance. Randomly select x first foreground locations from the multiple foreground locations corresponding to the first segmentation result; x is a positive integer. Based on the x first foreground point locations, the second foreground point location corresponding to the first target image, and the first feature data, the third feature data corresponding to the pixels of the first foreground point location and the second foreground point location are determined respectively. The central prototype feature of the object instance is obtained by summing multiple third feature data and averaging the sums. The first segmentation result is eroded to obtain the edge position corresponding to the first segmentation result; The fourth feature data corresponding to the pixels at the edge positions are summed, and the sums are averaged to obtain the edge prototype features of the object instance. Based on the global prototype features, the central prototype features, and the edge prototype features, the squared Euclidean distance corresponding to the feature vector of each pixel is calculated respectively. Based on each of the squared Euclidean distances, at least one confidence matrix is ​​determined; the confidence matrix includes at least one of the following: global confidence matrix, center confidence matrix, and edge confidence matrix; The center confidence matrix and the edge confidence matrix are weighted and summed to obtain the weighted sum confidence matrix; Based on the global confidence matrix and the weighted summation confidence matrix, determine the first confidence value corresponding to the global confidence matrix and the second confidence value corresponding to the weighted summation confidence matrix; Based on the first confidence value and the second confidence value, determine the second segmentation result corresponding to the first confidence value and the third segmentation result corresponding to the second confidence value; Based on the first segmentation result, the second segmentation result, and the third segmentation result, the pseudo-label data corresponding to the first target image is determined; The optimization module is specifically used for: Based on the pseudo-label data, sample the third target image from the first dataset; Based on the third target image and the true label data, determine the fourth target image in the second dataset; The interactive segmentation model is optimized based on the third target image and the fourth target image.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the model optimization method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model optimization method as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the model optimization method as described in any one of claims 1 to 3.

Images