A training method of an image segmentation model and an image segmentation method

Abstract

The application provides a training method of an image segmentation model and an image segmentation method. Through various improvements, the application can enable the model to better learn features in the alignment object local, object and scene containing the object in the feature learning stage, low-cost and efficient improve the object recognition ability of the model under different scene noises, and help to improve the performance of the downstream image segmentation model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, more specifically to the field of neural network technology, and more specifically to a training method for an image segmentation model and an image segmentation method. Background Technology

[0002] Supervised image segmentation techniques have achieved great success in fields such as medical image processing[1] and autonomous driving[2]. However, the huge annotation cost limits the expansion of data scale. Self-supervised learning methods learn pixel-level general representations from data without annotation, which solves the problem of data annotation very well.

[0003] Existing methods [3] demonstrate that visual Transformer models can learn patch-level representations of objects well by aligning different viewpoints on image-centric datasets. Building on this, a two-stage approach [4] obtains fine-grained information about objects through further fine-tuning. Specifically, the two-stage approach first pre-trains the model on an image-centric dataset, and then retrains (fine-tunes) the model on scene-centric data by aligning randomly enhanced viewpoints, thereby obtaining a tighter and more semantically relevant feature space.

[0004] While two-stage methods have achieved good results in downstream image segmentation tasks, the underlying knowledge behind their training process remains unclear. Some existing methods first train the model's feature extractor to learn object-related features in the first stage, and then fine-tune the model's predictor to learn image segmentation based on these object-related features in the second stage. Another class of methods studies generating viewpoints on salient regions of the image (based on gradient heatmaps, self-attention maps, etc.) to enable the model to learn object-related features; however, the inventors found that this approach does not improve performance during the fine-tuning process in two-stage training.

[0005] In conclusion, researching how to improve model performance and segmentation accuracy in image segmentation models at low cost during two-stage learning is an urgent problem to be solved.

[0006] The following are references mentioned in the background section:

[0007] [1]Asgari Taghanaki S,Abhishek K,Cohen JP,et al.Deep semantic segmentation of natural and medical images:a review[J].ArtificialIntelligence Review,2021,54:137-178.

[0008] [2]Asgari Taghanaki S,Abhishek K,Cohen JP,et al.Deep semantic segmentation of natural and medical images:a review[J].ArtificialIntelligence Review,2021,54:137-178.

[0009] [3]Caron M,Touvron H,Misra I,et al.Emerging properties in self-supervised vision transformers[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision.2021:9650-9660.

[0010] [4]Asgari Taghanaki S,Abhishek K,Cohen JP,et al.Deep semantic segmentation of natural and medical images:a review[J].ArtificialIntelligence Review,2021,54:137-178.

[0011] [5]Caron M,Misra I,Mairal J,et al.Unsupervised learning of visualfeatures by contrasting cluster assignments[J].Advances in neural informationprocessing systems,2020,33:9912-9924.

[0012] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0013] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a training method for an image segmentation model and an image segmentation method.

[0014] The objective of this invention is achieved through the following technical solution:

[0015] According to a first aspect of the present invention, a training method for an image segmentation model is provided, comprising the steps of: S1, obtaining a first training set, the first training set including a set of viewpoint pairs corresponding to each image in a variety of images, the set of viewpoint pairs containing two types of viewpoint pairs, the first type of viewpoint pair including an ancestor viewpoint and its sub-viewpoints, the second type of viewpoint pair including two scene views of the same image, wherein the scene viewpoint can be used as an ancestor viewpoint of an object viewpoint or a local viewpoint, the object viewpoint can be used as an ancestor viewpoint of a local viewpoint, and the sub-viewpoints originate from the ancestor viewpoints; S2, iteratively updating a teacher model and a student model multiple times according to a preset twin training method, wherein both the student model and the teacher model adopt a visual Transformer, wherein each update includes: obtaining all views of a batch of images from the first training set, wherein each ancestor viewpoint is input into the teacher model to extract a first image representation, and each scene viewpoint and each... The student model extracts the second image representation and obtains the self-attention map generated by the student model for the input viewpoint; each first image representation and second image representation is mapped to obtain the first predictive code and the second predictive code; based on the deviation between the first predictive code and the second predictive code at the common sub-block under each viewpoint of the batch of images, the norm and variance of the matrix corresponding to the self-attention map of each viewpoint, the total loss is determined, and the parameters of the student model are updated according to the total loss. The parameters of the teacher model are updated according to the weighted values ​​of the current teacher model parameters and the updated student model parameters; S3, a segmentation predictor is added to the teacher model or student model obtained in step S2 to form an image segmentation model. The image segmentation model is trained using a second training set with image segmentation labels to perform image segmentation according to the input image, and only the parameters of the segmentation predictor are updated during training.

[0016] Optionally, the first training set is obtained as follows: An image set comprising multiple images is acquired; a corresponding viewpoint set is constructed for each image; the viewpoint set includes at least two scene views that intersect with each other, cropped from the image; one or more object views constructed from sub-blocks cropped from each scene view; and one or more local views of objects constructed from sub-blocks cropped from each object view; a viewpoint pair set is constructed for each image, wherein each ancestor viewpoint and its sub-viewpoints constitute a first viewpoint pair, and every two scene views constitute a second viewpoint pair, including the first viewpoint pair composed of each ancestor viewpoint and its sub-viewpoints; a scene viewpoint can serve as the ancestor viewpoint of a corresponding object viewpoint or local viewpoint, and an object viewpoint can serve as the ancestor viewpoint of a corresponding local viewpoint; all viewpoint pairs of the multiple images constitute the first training set.

[0017] Optionally, the total loss is determined as follows: A sub-loss for each image in the current batch is determined, wherein the sub-loss for each image is a weighted sum of alignment loss and attention loss, where the alignment loss is the sum of the deviations between the first and second predicted codes for all viewpoint pairs of each image, and the attention loss is the mean of the sum of the norms and variances of the matrices corresponding to each attention head in the self-attention maps generated by the student model for each viewpoint; the total loss is obtained by averaging the sub-losses of all images in the current batch.

[0018] Optionally, the deviation between the first and second predictive codes for any viewpoint pair is determined as follows: when the viewpoint pair is a first type of viewpoint pair, the local value of the position corresponding to the second predictive code is extracted from the first predictive code of the viewpoint pair, the local value is interpolated to make it consistent with the size of the second predictive code, and the cross-entropy loss between the interpolated local value and the second predictive code is calculated as the deviation; when the viewpoint pair is a second type of viewpoint pair, the intersection position of the two views under the viewpoint pair on the corresponding image is determined, and the cross-entropy loss between the values ​​of the first and second predictive codes corresponding to the intersection position is used as the deviation.

[0019] Optionally, the sub-loss for each image is calculated as follows:

[0020]

[0021] in, Indicates alignment loss. Let α represent the attention loss, β represent the weight of the alignment loss, and β represent the weight of the attention loss.

[0022] Optionally, the alignment loss can be calculated in the following ways:

[0023]

[0024] in, This represents the deviation between the first predicted code of the first perspective for the lower ancestor perspective x′ and the second predicted code of the sub-perspective x. This represents a scene perspective from a second viewpoint. First predictive coding and another scene perspective The deviation between the second predictive codes, x p Indicates local perspective, x l Indicates the object's perspective. This represents the ancestor view of child view x.

[0025] Optionally, the calculation method for the attention loss includes:

[0026]

[0027] Where n represents the total number of viewpoints, n h A represents the number of attention heads in the visual Transformer. i,j The self-attention map generated by the j-th self-attention head at the i-th viewpoint in the student model, ||·|| F This indicates taking the F-norm, Var(·) represents the variance, and λ1 and λ2 are adjustable hyperparameters.

[0028] According to a second aspect of the present invention, an image segmentation method is provided, comprising: acquiring an image to be processed; acquiring a trained image segmentation model obtained by the method described in the first aspect; and using the trained image segmentation model to perform image segmentation on the image to be processed.

[0029] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the methods of the first aspect and / or the second aspect by executing the executable instructions. Attached Figure Description

[0030] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0031] Figure 1 This is a flowchart illustrating a training method for an image segmentation model according to an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0033] As mentioned in the background section, improving model performance and segmentation accuracy in image segmentation models at low cost is a pressing issue in two-stage learning. However, the inventors found that this method does not improve performance during fine-tuning in two-stage training. Based on this finding and the inventors' research, this invention proposes that the main purpose of two-stage training is to improve the scene invariance of the model representation. Therefore, the inventors made improvements from three aspects. First, the training set in the first stage is improved by using each image to construct a scene view, an object view, and a local view of the object. The scene view can serve as the ancestor view of the object view or the local view, and the object view can serve as the ancestor view of the local view. Sub-views originate from ancestor views. This setup allows the model to learn the basis of scene invariance. Second, the operations for inputting various viewpoints into the model are defined. Each ancestor viewpoint is input into the teacher model to extract the first image representation, and each scene viewpoint and each sub-viewpoint are input into the student model to extract the second image representation. The self-attention map generated by the student model for the input viewpoint is also obtained, which serves as the basis for calculating the total loss. Thirdly, the calculation method of the total loss in the first stage is improved. This invention determines the total loss based on the deviation between the first and second predicted codes for each viewpoint pair in a batch of images, and the norm and variance of the self-attention map matrix corresponding to each viewpoint. The parameters of the student model are updated according to the total loss, and the parameters of the teacher model are updated based on the weighted values ​​of the current teacher model parameters and the updated student model parameters. Thus, based on the deviation between the first and second predicted codes at common sub-blocks for each viewpoint pair in a batch of images, the model can learn to align features of the same region under different levels of viewpoint pairs (scene view-object view, object view-local view, scene view-local view, scene view-scene view). This deviation drives the model to extract scene-invariant features from the same sub-block with different background noise, improving the performance of subsequent image segmentation. Furthermore, by incorporating the influence of the norm and variance of the self-attention map matrix corresponding to each viewpoint into the total loss, the model's attention to noisy backgrounds is penalized, allowing the model to better learn object representations after introducing scene information. In summary, through various improvements, this invention enables the model to better learn and align features of local objects, objects, and scenes containing objects during the feature learning stage. This improves the model's ability to distinguish objects under different scene noise conditions in a low-cost and efficient manner, and helps to enhance the performance of downstream image segmentation models.

[0034] First, the process by which the inventors proposed the concept of scenario invariance is presented. During the inventors' research, the following observations were made that violated domain consensus:

[0035] Using the following method to generate viewpoints based on self-attention maps for model training will lead to a decrease in model performance:

[0036]

[0037] Where I represents the input image, w and h represent the width and height of the viewpoint, respectively, and A I This is the self-attention map of the image after passing through the Visual Transformer, where p represents the probability of generating the viewpoint with coordinates (x, y) as the top-left corner, and sum(A I ) represents A I sum.

[0038] Furthermore, it was observed that when the model was trained on scene-centric data, its attention was distracted by background noise.

[0039] Based on the above two observations, this invention assumes that the purpose of two-stage training is to ensure that the model's representation of the same object is as similar as possible under different backgrounds. This invention calls this property scene invariance.

[0040] Secondly, a brief illustrative implementation process of the present invention will be given.

[0041] (1) Constructing the training set

[0042] (2) Construct a two-stage model

[0043] (3) Constructing the loss function

[0044] (4) The training process of training the model using the training set

[0045] (5) Application scenarios, namely the process of using trained models for image segmentation.

[0046] To better understand the present invention, the following is a schematic description of each part of the above implementation process.

[0047] (1) Constructing the training set

[0048] The training set is divided into the first training set and the second training set, which will be explained below.

[0049] According to one embodiment of the present invention, the first training set is the basis for learning scene invariance, with the aim of constructing multiple levels of viewpoint pairs. This is not limited to objects and their local views, but also includes broader scene views. Optionally, the first training set is obtained as follows: An image set comprising multiple images is acquired; a corresponding viewpoint set is constructed for each image, the viewpoint set including at least two overlapping scene views cropped from the image, one or more object views constructed from sub-blocks cropped from each scene view, and one or more local views of objects constructed from sub-blocks cropped from each object view; a viewpoint pair set is constructed for each image, wherein each ancestor viewpoint and its sub-viewpoints constitute a first viewpoint pair, and every two scene views constitute a second viewpoint pair, including the first viewpoint pair composed of each ancestor viewpoint and its sub-viewpoints, where a scene viewpoint can serve as the ancestor viewpoint of a corresponding object viewpoint or local viewpoint, and an object viewpoint can serve as the ancestor viewpoint of a corresponding local viewpoint; all viewpoint pairs of the multiple images constitute the first training set. For example, the implementer could set an algorithm: randomly select two intersecting scene views from an image; then extract sub-blocks from each scene view to construct an object view; and finally extract sub-blocks from the object view to construct a local view of the object. Of course, other methods can also be used to construct these views. For example, first extract n smaller bounding boxes from the image. p A smaller perspective Representing a local viewpoint of an object; then combining local viewpoints in pairs to form multiple combinations, and for each combination, constructing a viewpoint that encompasses that combination, forming n. l A medium-sized perspective Represents the object's perspective; finally, construct n. s A larger perspective The representation of an object in a scene is defined as follows: each of the multiple scene perspectives encompasses at least one object perspective, and these scene perspectives intersect with each other. Subsequently, ancestor and sub-perspectives are determined in the aforementioned manner. If perspective A contains a sub-block corresponding to perspective B, then perspective A is an ancestor perspective of perspective B, meaning perspective B originates from perspective A. It should be noted that the sizes of the various scene perspectives or sub-blocks can be fixed to directly construct the first training set. Alternatively, the sizes of the various scene perspectives or sub-blocks can be random, and resizing can be used to adjust all scene perspectives to the same size, all object perspectives to the same size, and all local perspectives to the same size before constructing the first training set. The technical solution of this embodiment can achieve at least the following beneficial technical effects: The first training set of the present invention constructs multiple levels of view pairs, such as scene view-object view, object view-local view, scene view-local view, and scene view-scene view. Each view pair has a corresponding position region. Different position regions can be regarded as added noise. In the future, the model can learn to align the features of this position region in the presence of scene noise at low cost, thereby better learning the features of scene invariance.

[0050] According to one embodiment of the present invention, the second training set may be a subset of a self-made or existing image segmentation dataset. The image segmentation dataset includes images and segmentation labels, where the segmentation labels indicate the category of each pixel in the image. Existing image segmentation datasets include, for example, the COCO-Thing dataset, the COCO-Stuff dataset, the Pascal VOC dataset, or the ADE20k dataset. For example, a subset may be extracted from the COCO-Thing dataset, the COCO-Stuff dataset, the Pascal VOC dataset, or the ADE20k dataset as the second training set.

[0051] (2) Construct a two-stage model

[0052] The overall image segmentation model includes a feature extractor and a segmentation predictor. The feature extractor is used to extract image representations of the input image (generally speaking, the viewpoint is also an image), and the segmentation predictor is used to perform segmentation processing based on the image representations to obtain the category of each pixel in the input image.

[0053] The feature extractor can be either the teacher model or the student model. In practice, after training either the teacher or student model on the first training set, their performance can be observed, and the better model can be selected as the feature extractor. Both the teacher and student models can use a visual Transformer.

[0054] (3) Constructing the loss function

[0055] This invention constructs viewpoint pairs between an object or a part of an object and its ancestor viewpoints, thereby forcing the model to align the representation of the same object under different viewpoints.

[0056] According to one embodiment of the present invention, the total loss is determined in the following manner:

[0057] Determine the sub-loss for each image in the current batch. The sub-loss for each image is a weighted sum of the alignment loss and the attention loss. The alignment loss is the sum of the deviations between the first and second predicted codes for all viewpoint pairs of each image. The attention loss (or attention regularization loss) is the mean of the sum of the norms and variances of the matrices corresponding to each attention head in the self-attention maps generated by the student model for each viewpoint. The total loss is obtained by averaging the sub-losses of all images in the current batch.

[0058] Optionally, the sub-loss for each image can be calculated as follows:

[0059]

[0060] in, Indicates alignment loss. The values ​​represent attention loss, α represents the weight of alignment loss, and β represents the weight of attention loss. The values ​​of α and β can be set according to the implementer's needs. For example, α can be set to 0.9, 1, or 1.1; and β can be set to 0.01, 0.03, 0.1, 0.3, or 1.0.

[0061] For the alignment loss, the original representation can be:

[0062]

[0063] in, This represents the total alignment loss. This represents the sub-alignment loss between scene views. This represents the sub-alignment loss between the scene view and the object view. This represents the sub-alignment loss between the object's viewpoint and its local viewpoint. This represents the sub-alignment loss between the scene view and the object's local view.

[0064] After simple rearrangement, we can obtain:

[0065]

[0066] in, This represents the deviation between the first predicted code of the first perspective for the lower ancestor perspective x′ and the second predicted code of the sub-perspective x. This represents a scene perspective from a second viewpoint. First predictive coding and another scene perspective The deviation between the second predictive codes, x p Indicates local perspective, x l Indicates the object's perspective. This represents the ancestor view of the child view x. Since the time of each scene can be used as... or Therefore, the last item corresponds to the combination of various scene perspectives. Sum.

[0067] This invention uses a visual Transformer as the model backbone (feature extractor). The teacher and student models are denoted as f, respectively. t and f s An additional linear layer g is added after the output layer of the visual Transformer. t ,g s It is used to perform mapping to project features into a more semantically relevant feature space.

[0068] According to one embodiment of the present invention, since there are two viewpoint pairs, the calculation method of the deviation needs to be considered separately. Optionally, the deviation between the first and second predictive codes under any viewpoint pair is determined as follows: when the viewpoint pair is a first type of viewpoint pair, the local value of the position corresponding to the second predictive code is extracted from the first predictive code under the viewpoint pair, the local value is interpolated to make it consistent with the size of the second predictive code, and the cross-entropy loss between the interpolated local value and the second predictive code is calculated as the deviation; when the viewpoint pair is a second type of viewpoint pair, the intersection position of the two views under the viewpoint pair on the corresponding image is determined, and the cross-entropy loss between the values ​​of the first and second predictive codes corresponding to the intersection position is used as the deviation. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the present invention uses two methods to determine the deviation between predictive codes under different viewpoint pairs, which can better realize the calculation of alignment deviation and improve the performance of the subsequently trained model.

[0069] Indicatively, for a pair of perspectives (x1, x2), it can be assumed without loss of generality that x1 is the ancestor perspective of x2. First, inputting x1 into the teacher model and x2 into the student model yields the corresponding representations. Furthermore, the similarity between the image representation and the K cluster centers is calculated, denoted as . Next, an optimal transmission algorithm (e.g., the Sinkhorn algorithm) is used to sharpen the output of the teacher model, and a Softmax function with a temperature coefficient is applied to the output of the student model to obtain the predicted codes for the teacher and student models, respectively.

[0070] Q t (x1)=Sk(g t (f t (x1))C T ),

[0071] Q s (x2) = Softmax τ (g s (f s (x2))C T ).

[0072] Where Sk(·) represents the Sinkhorn algorithm and Softmax τ (·) denotes the Softmax function with a temperature coefficient, and T denotes transpose. Since x1 is amplified from x2, the value of the position corresponding to x2 in the predictive code of x1 (that is, the position corresponding to the region where x2 is located in the first predictive code of x1) can be extracted, and then transformed to the same size as the predictive code of x2 through an interpolation algorithm. Finally, the cross-entropy loss between the two is calculated:

[0073]

[0074] Where R(·) represents the interpolation algorithm and CE(·) represents the cross-entropy loss function.

[0075] For a viewpoint pair consisting of two scene perspectives, if there is an intersection of the same size, the cross-entropy loss can be calculated by taking the values ​​of the corresponding intersection portions from the first and second predictive codes, and used as...

[0076] The above alignment framework constructs the alignment method of objects in different scenes. However, how to deal with another observed phenomenon, namely, the model's attention being interfered with by noise in the background, is still a problem to be solved. It is a widely accepted understanding that the self-attention map of the visual Transformer is related to the semantics of the object [3]. Based on this assumption, this invention constrains the self-attention map. Specifically, the F-norm of the self-attention map is added to the loss function as a regularization term, so that the model can focus its limited attention on the most critical information.

[0077] However, imposing constraints solely on the self-attention map can lead to training collapse, i.e., the existence of a solution where the loss is zero, causing the model to no longer apply attention to any objects and become overly pessimistic. To avoid this phenomenon, this invention adds an additional variance term to protect the most critical information from decaying too rapidly during training. According to one embodiment of the invention, the attention loss is calculated as follows:

[0078]

[0079] Where n represents the total number of viewpoints, n h A represents the number of attention heads in the visual Transformer. i,j The self-attention map generated by the j-th self-attention head at the i-th viewpoint in the student model, ||·|| F This indicates the use of the F-norm, Var(·) represents the variance, and λ1 and λ2 are adjustable hyperparameters. λ1 and λ2 can be set and adjusted according to the implementer's needs or experience. For example, λ1 can be set to 0.9 or 1, and λ2 can be set to 0.1, 0.3, 1.0, 3.0, or 10.0. The technical solution of this embodiment can achieve at least the following beneficial technical effects: This formula reflects a trade-off; on the one hand, it aims to obtain more scene information so that the model can better learn scene invariance; on the other hand, it also avoids the model overfitting scene noise, leading to distraction; thereby improving the performance of the model.

[0080] (4) The training process of training the model using the training set

[0081] Based on the above training set, the model is trained to obtain the final image segmentation model.

[0082] According to an embodiment of the present invention, a method for training an image segmentation model is provided, comprising steps S1, S2, and S3. Each step is described below.

[0083] S1: Obtain the first training set, which includes viewpoint pairs corresponding to each image in multiple images. The viewpoint pairs contain two types of viewpoint pairs: the first type of viewpoint pair includes the ancestor viewpoint and its sub-viewpoints, and the second type of viewpoint pair includes two scene views of the same image.

[0084] S2. The teacher model and student model are iteratively updated multiple times according to the preset twin training method. Both the student model and the teacher model adopt the visual Transformer. Each update includes: obtaining all viewpoints of a batch of images from the first training set, wherein each ancestor viewpoint is input into the teacher model to extract the first image representation, each scene viewpoint and each sub-viewpoint are input into the student model to extract the second image representation, and the self-attention map generated by the student model for the input viewpoint is obtained; each first image representation and the second image representation are mapped to obtain the first prediction code and the second prediction code; based on the deviation between the first prediction code and the second prediction code under each viewpoint of the batch of images at the common sub-block, and the norm and variance of the matrix corresponding to the self-attention map of each viewpoint, the total loss is determined; the parameters of the student model are updated according to the total loss, and the parameters of the teacher model are updated according to the weighted value of the current parameters of the teacher model and the updated parameters of the student model.

[0085] According to one embodiment of the present invention, each ancestor view is input into the teacher model to extract the first image representation corresponding to the ancestor view, and then the first image representation is mapped using the Sinkhorn algorithm to obtain the first predictive code. Each scene view and each sub-view are input into the student model to extract the second image representation, and then the Softmax function with temperature coefficient is used to map to obtain the second predictive code. Then, based on the view pair and the aforementioned total loss function, the total loss corresponding to a batch of images can be calculated, the gradient is calculated based on the total loss, and the gradient is backpropagated to update the parameters of the student model. As for the teacher model, it can be updated according to the weighted values ​​of the current teacher model parameters and the updated student model parameters. Illustratively, it can be based on the twin training method in reference [3] or other similar twin training methods.

[0086] S3. After obtaining the teacher model or student model in step S2, add a segmentation predictor to form an image segmentation model. Use the second training set with image segmentation labels to train the image segmentation model to perform image segmentation based on the input image, and only update the parameters of the segmentation predictor during training.

[0087] According to one embodiment of the present invention, in step S3, the implementer can first compare the performance of the teacher model or the student model, and select the model with better performance as the feature extractor. Then, a segmentation predictor is added to form an initial image segmentation model. Next, the initial image segmentation model is trained (fine-tuned) using a second training set. During training, the image is input to the feature extractor to extract image representations. The image representations are input to the segmentation predictor to obtain the category prediction value for each pixel of the input image. The gradient is calculated based on the cross-entropy loss calculated using the category prediction values ​​and segmentation labels, and the parameters of the segmentation predictor are updated via backpropagation. After training is completed in step S3, the trained image segmentation model is obtained.

[0088] (5) Application Scenarios

[0089] According to an embodiment of the present invention, an image segmentation method is provided, the method comprising: acquiring an image to be processed; acquiring a trained image segmentation model obtained by the method described in the foregoing embodiments; and using the trained image segmentation model to perform image segmentation on the image to be processed.

[0090] To verify the effectiveness of the invention, the inventors also conducted experiments.

[0091] In the experiments, multiple datasets were used for validation: the COCO-Thing dataset, the COCO-Stuff dataset, the Pascal VOC dataset, and the ADE20k dataset. The experimental results are shown in the table below:

[0092] Table 1 Experimental Results

[0093]

[0094]

[0095] In the diagram, the bolded results represent the optimal results, the underlined results represent the suboptimal results, mIoU represents the mean intersection-over-union ratio, and acc represents the accuracy. It can be seen that the model obtained using the method of this invention achieves the best results in all comparisons, indicating that the segmentation performance of the model trained using this method is superior and significantly improves upon existing methods.

[0096] The literature from which the above existing methods are derived is as follows:

[0097] [MoCo-v2]Chen X,Fan H,Girshick R,et al.Improved baselines with momentum contrastive learning[J].arXiv preprint arXiv:2003.04297,2020.

[0098] [SwAV]Caron M,Misra I,Mairal J,et al.Unsupervised learning of visualfeatures by contrasting cluster assignments[J].Advances in neural informationprocessing systems,2020,33:9912-9924.

[0099] [SlotCon]Wen X, Zhao B, Zheng A, et al.Self-supervised visual representation learning with semantic grouping[J].Advances in neuralinformation processing systems,2022,35:16423-16438.

[0100] [DINO]Caron M,Touvron H,Misra I,et al.Emerging properties in self-supervised vision transformers[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision.2021:9650-9660.

[0101] [STEGO]Hamilton M, Zhang Z, Hariharan B, et al.Unsupervised semanticsegmentation by distilling feature correspondences[J].arXiv preprint arXiv:2203.08414,2022.

[0102] [Leopart]Ziegler A,Asano Y M.Self-supervised learning of object parts for semantic segmentation[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2022:14502-14511.

[0103] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0104] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0105] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0106] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for an image segmentation model, characterized in that, Including the following steps: S1. Obtain the first training set. The first training set includes viewpoint pairs corresponding to each image in multiple images. The viewpoint pairs contain two types of viewpoint pairs. The first type of viewpoint pair includes an ancestor viewpoint and its sub-viewpoints. The second type of viewpoint pair includes two scene views of the same image. The scene viewpoint can be used as the ancestor viewpoint of an object viewpoint or a local viewpoint. The object viewpoint can be used as the ancestor viewpoint of a local viewpoint. The sub-viewpoints originate from the ancestor viewpoints. S2. Iteratively update the teacher and student models multiple times according to the preset twin training method. Both the student and teacher models use the visual Transformer. Each update includes: All viewpoints of a batch of images are obtained from the first training set. The ancestor viewpoints are input into the teacher model to extract the first image representation. The scene viewpoints and sub-viewpoints are input into the student model to extract the second image representation. The self-attention map generated by the student model for the input viewpoints is also obtained. Each first image representation and second image representation is mapped to obtain a first prediction code and a second prediction code; Based on the deviation between the first and second predictive codes at common sub-blocks for each viewpoint pair of the batch of images, and the norm and variance of the matrix corresponding to the self-attention map for each viewpoint, the total loss is determined, and the parameters of the student model are updated according to the total loss. The parameters of the teacher model are updated according to the weighted values ​​of the current teacher model parameters and the updated student model parameters. S3. After obtaining the teacher model or student model in step S2, add a segmentation predictor to form an image segmentation model. Use the second training set with image segmentation labels to train the image segmentation model to perform image segmentation based on the input image, and only update the parameters of the segmentation predictor during training.

2. The method according to claim 1, characterized in that, The first training set was obtained in the following manner: Obtain an image set including multiple images, and construct a corresponding view set for each image. The view set includes at least two scene views that intersect with each other, cropped from the image, one or more object views constructed from sub-blocks cropped from each scene view, and one or more local views of objects constructed from sub-blocks cropped from each object view. Construct a set of viewpoint pairs for each of the images, wherein each ancestor viewpoint and its sub-viewpoints constitute a first viewpoint pair, and every two scene views constitute a second viewpoint pair, including the first viewpoint pair composed of each ancestor viewpoint and its sub-viewpoints. A scene viewpoint can serve as the ancestor viewpoint of a corresponding object viewpoint or a local viewpoint, and an object viewpoint can serve as the ancestor viewpoint of a corresponding local viewpoint. All viewpoint pairs of the multiple images constitute the first training set.

3. The method according to claim 2, characterized in that, The total loss is determined in the following manner: Determine the sub-loss for each image in the current batch, wherein the sub-loss for each image is a weighted sum of alignment loss and attention loss, wherein the alignment loss is the sum of the deviations between the first and second predicted codes under all viewpoint pairs for each image, and the attention loss is the mean of the sum of the norms and variances of the matrices corresponding to each attention head in the self-attention maps generated by the student model for each viewpoint. The total loss is obtained by averaging the sub-losses of all images in the current batch.

4. The method according to claim 3, characterized in that, The deviation between the first and second predictive codes for any given viewpoint is determined as follows: When the viewpoint pair is the first viewpoint pair, the local value of the position corresponding to the second predictive code is extracted from the first predictive code under the viewpoint pair, the local value is interpolated to make it consistent with the size of the second predictive code, and the cross-entropy loss between the interpolated local value and the second predictive code is calculated as the deviation. When the viewpoint pair is the second type of viewpoint pair, determine the intersection position of the two viewpoints under the viewpoint pair on the corresponding image, and use the cross-entropy loss between the values ​​of the first predictive code and the second predictive code corresponding to the intersection position as the deviation.

5. The method according to claim 3 or 4, characterized in that, The sub-loss for each image is calculated as follows: in, Indicates alignment loss. Let α represent the attention loss, β represent the weight of the alignment loss, and β represent the weight of the attention loss.

6. The method according to claim 5, characterized in that, The alignment loss is calculated in the following ways: in, This represents the deviation between the first predicted code of the first perspective for the lower ancestor perspective x′ and the second predicted code of the sub-perspective x. This represents a scene perspective from a second viewpoint. First predictive coding and another scene perspective The deviation between the second predictive codes, x p Indicates local perspective, x l Indicates the object's perspective. This represents the ancestor view of child view x.

7. The method according to claim 5, characterized in that, The calculation methods for attention loss include: Where n represents the total number of viewpoints, n h A represents the number of attention heads in the visual Transformer. i,j The self-attention map generated by the j-th self-attention head at the i-th viewpoint in the student model, ||·|| F This indicates taking the F-norm, Var(·) represents the variance, and λ1 and λ2 are adjustable hyperparameters.

8. An image segmentation method, characterized in that, The method includes: Obtain the image to be processed; Obtain a trained image segmentation model obtained by the method according to any one of claims 1-7, and use the trained image segmentation model to perform image segmentation on the image to be processed.

9. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 8 by executing the executable instructions.

Images