A training method and device of an image segmentation model, an apparatus, and a storage medium

Abstract

Embodiments of the present application provide a kind of training method, device and equipment of image segmentation model and storage medium, for reducing the data heterogeneity influence of third party unlabeled data, improve the accuracy of image segmentation. Including: obtaining training sample set, initial student model and initial teacher model;Supervised training is carried out to the initial student model to obtain supervision loss;Based on the initial teacher model, the local unlabeled feature prototype of the local unlabeled sample and the third party unlabeled feature prototype of the third party unlabeled sample are obtained;According to the local unlabeled feature prototype, the third party unlabeled feature prototype, the first feature map and the real label, unlabeled loss is obtained;According to the supervision loss and the unlabeled loss, the initial student model is trained to obtain the image segmentation model. The technical scheme provided in the present application can be applied to the fields of artificial intelligence, computer vision and cloud technology.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a training method, apparatus, device, and storage medium for an image segmentation model. Background Technology

[0002] In recent years, deep learning-based methods have greatly improved the accuracy and efficiency of medical image segmentation. However, the success of deep learning-based methods usually requires a large amount of labeled medical data to train the image segmentation model, and labeling medical data requires specialized knowledge, making it very expensive.

[0003] Semi-supervised learning, which can utilize both limited labeled data and abundant unlabeled data, is widely used in training medical image segmentation models. However, the effectiveness of semi-supervised learning largely depends on the quantity and quality of unlabeled data. In practice, due to limitations in medical centers' image acquisition capabilities and patient numbers, their local unlabeled data is also limited, necessitating the inclusion of medical images from other medical centers. Since medical images from other centers differ from local images, data heterogeneity is typically introduced into semi-supervised learning, thus affecting the quality of image segmentation.

[0004] Therefore, there is an urgent need for a solution that can address the data heterogeneity problem introduced by unlabeled third-party data and improve the accuracy of image segmentation. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for an image segmentation model, which reduces the impact of data heterogeneity from unlabeled third-party data and improves the accuracy of image segmentation.

[0006] In view of this, this application provides a training method for an image segmentation model, comprising: obtaining a training sample set, an initial student model, and an initial teacher model, wherein the training sample set includes locally labeled samples, locally unlabeled samples, and third-party unlabeled samples, and the locally labeled samples also include ground truth labels; the initial student model and the initial teacher model have the same network structure; obtaining a first predicted segmentation image and a first feature map of the locally labeled samples based on the initial student model, and obtaining the supervision loss based on the first predicted segmentation image and the ground truth labels; and obtaining the locally unlabeled samples corresponding to the locally unlabeled samples based on the initial teacher model. The model is used to annotate the feature prototype and obtain the third-party unlabeled feature prototype corresponding to the third-party unlabeled sample using the initial teacher model; a first prototype propagation prediction is obtained based on the local unlabeled feature prototype and the first feature map, and a second prototype propagation prediction is obtained based on the third-party unlabeled feature prototype and the first feature map; a first loss value is obtained based on the first prototype propagation prediction and the ground truth label, and a second loss value is obtained based on the second prototype propagation prediction and the ground truth label, with the first loss value and the second loss value serving as the unlabeled loss; the initial student model is trained based on the supervised loss and the unlabeled loss to obtain the image segmentation model.

[0007] Another aspect of this application provides a training apparatus for an image segmentation model, comprising:

[0008] The acquisition module is used to acquire a training sample set, an initial student model, and an initial teacher model. The training sample set includes locally labeled samples, locally unlabeled samples, and third-party unlabeled samples. The locally labeled samples also include real labels. The initial student model and the initial teacher model have the same network structure.

[0009] The processing module is configured to obtain a first predicted segmentation image and a first feature map of the locally labeled sample based on the initial student model, and obtain the supervised loss based on the first predicted segmentation image and the ground truth label; obtain the local unlabeled feature prototype corresponding to the local unlabeled sample based on the initial teacher model, and obtain the third-party unlabeled feature prototype corresponding to the third-party unlabeled sample using the initial teacher model; predict a first prototype propagation prediction based on the local unlabeled feature prototype and the first feature map, and predict a second prototype propagation prediction based on the third-party unlabeled feature prototype and the first feature map; obtain a first loss value based on the first prototype propagation prediction and the ground truth label, and obtain a second loss value based on the second prototype propagation prediction and the ground truth label, wherein the first loss value and the second loss value are used as the unlabeled loss;

[0010] The training module is used to train the initial student model based on the supervised loss and the unlabeled loss to obtain the image segmentation model.

[0011] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to obtain a second predicted segmentation image of the local unlabeled sample based on the initial student model, and to obtain a first pseudo-label of the local unlabeled sample based on the initial teacher model;

[0012] A third loss value is obtained by processing the second predicted segmentation image and the first pseudo-label, and the third loss value is included in the unlabeled loss.

[0013] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain a first prediction probability map of the second predicted segmentation image;

[0014] The target pixel set is determined based on the first predicted probability map and the first threshold, and each pixel in the target pixel set is used to determine the third loss value;

[0015] The probability value of each pixel in the target pixel set is processed with the first pseudo-label to obtain the third loss value.

[0016] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to divide the pixels in the target pixel set into K regions of the same size, and divide the first pseudo-label into K regions of the same size in a way that corresponds one-to-one with the target pixel set, where K is greater than or equal to 2;

[0017] K loss values ​​are obtained from K regions in the target pixel set and K regions of the first pseudo-label, respectively.

[0018] The third loss value is obtained by averaging the K loss values.

[0019] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to obtain the local annotation feature prototype of the local annotation sample based on the initial student model;

[0020] The fourth loss value is obtained by performing categorical feature regularization on the local labeled feature prototype, the local unlabeled feature prototype, and the third-party unlabeled feature prototype. This fourth loss value is included in the unlabeled loss.

[0021] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain the first foreground feature prototype and the first background feature prototype of the local labeled feature prototype, the second foreground feature prototype and the second background feature prototype of the local unlabeled feature prototype, and the third foreground feature prototype and the third background feature prototype of the third-party unlabeled feature prototype.

[0022] Obtain the first distance function between the third foreground feature prototype and the second foreground feature prototype and the first foreground feature prototype, and obtain the second distance function between the third background feature prototype and the second background feature prototype and the first background feature prototype;

[0023] The fourth loss value is obtained by weighting and summing the first and second distance functions.

[0024] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to preprocess the local unlabeled sample and the third-party unlabeled sample to obtain the first perturbation sample and the second perturbation sample.

[0025] Based on the initial student model, a third predicted segmentation image of the first perturbation sample is obtained, and based on the initial teacher model, a fourth predicted segmentation image of the second perturbation sample is obtained.

[0026] A fifth loss value is obtained by determining the distance between each pixel in the third and fourth predicted segmentation images, and this fifth loss value is included in the unlabeled loss.

[0027] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain a first pseudo-label, a first predicted probability map, and a second feature map of the local unlabeled sample based on the initial teacher model;

[0028] The local unlabeled feature prototype is obtained by performing confidence-weighted masking average pooling on the first pseudo-label, the first predicted probability map, and the second feature map.

[0029] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to perform pixel-by-pixel cosine similarity processing on the local unlabeled feature prototype and the first feature map to obtain the first prototype propagation prediction.

[0030] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain a second pseudo-label, a second predicted probability map, and a third feature map of the third-party unlabeled sample based on the initial teacher model;

[0031] The second pseudo-label, the second predicted probability map, and the third feature map are subjected to confidence-weighted masking average pooling to obtain the prototype of the third-party unlabeled feature.

[0032] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to perform pixel-by-pixel cosine similarity processing on the third-party unlabeled feature prototype and the first feature map to obtain the second prototype propagation prediction.

[0033] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to adjust the network parameters of the initial teacher model according to the network parameters of the initial student model.

[0034] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain the first network parameters in the previous iteration and the second network parameters in the current iteration of the initial student model;

[0035] The third network parameter is obtained by performing a moving average process based on the first network parameter and the second network parameter;

[0036] Update the third network parameter to the network parameter of the initial teacher model.

[0037] In one possible design, in another implementation of another aspect of the embodiments of this application, the student model and the teacher model can be a U-Net network, a DenseNet network, or a ResNet network.

[0038] In one possible design, in another implementation of another aspect of the embodiments of this application, the locally labeled sample, the locally unlabeled sample, and the third-party unlabeled sample can be either two-dimensional images or three-dimensional images.

[0039] This application also provides an image segmentation method, including:

[0040] Obtain the image to be processed; obtain a segmented image of the image to be processed based on the image segmentation model, which is trained based on the above training method; output the segmented image.

[0041] This application also provides an image segmentation apparatus, comprising:

[0042] The acquisition module is used to acquire the image to be processed;

[0043] The processing module is used to obtain a segmented image of the image to be processed based on the image segmentation model, which is trained based on the above training method; the output module is used to output the segmented image.

[0044] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to obtain a second predicted segmentation image of the local unlabeled sample based on the initial student model, and to obtain a first pseudo-label of the local unlabeled sample based on the initial teacher model;

[0045] A third loss value is obtained by processing the second predicted segmentation image and the first pseudo-label, and the third loss value is included in the unlabeled loss.

[0046] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain a first prediction probability map of the second predicted segmentation image;

[0047] The target pixel set is determined based on the first predicted probability map and the first threshold, and each pixel in the target pixel set is used to determine the third loss value;

[0048] The probability value of each pixel in the target pixel set is processed with the first pseudo-label to obtain the third loss value.

[0049] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to divide the pixels in the target pixel set into K regions of the same size, and divide the first pseudo-label into K regions of the same size in a way that corresponds one-to-one with the target pixel set, where K is greater than or equal to 2;

[0050] K loss values ​​are obtained from K regions in the target pixel set and K regions of the first pseudo-label, respectively.

[0051] The third loss value is obtained by averaging the K loss values.

[0052] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to obtain the local annotation feature prototype of the local annotation sample based on the initial student model;

[0053] The fourth loss value is obtained by performing categorical feature regularization on the local labeled feature prototype, the local unlabeled feature prototype, and the third-party unlabeled feature prototype. This fourth loss value is included in the unlabeled loss.

[0054] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain the first foreground feature prototype and the first background feature prototype of the local labeled feature prototype, the second foreground feature prototype and the second background feature prototype of the local unlabeled feature prototype, and the third foreground feature prototype and the third background feature prototype of the third-party unlabeled feature prototype.

[0055] Obtain the first distance function between the third foreground feature prototype and the second foreground feature prototype and the first foreground feature prototype, and obtain the second distance function between the third background feature prototype and the second background feature prototype and the first background feature prototype;

[0056] The fourth loss value is obtained by weighting and summing the first and second distance functions.

[0057] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to preprocess the local unlabeled sample and the third-party unlabeled sample to obtain the first perturbation sample and the second perturbation sample.

[0058] Based on the initial student model, a third predicted segmentation image of the first perturbation sample is obtained, and based on the initial teacher model, a fourth predicted segmentation image of the second perturbation sample is obtained.

[0059] A fifth loss value is obtained by determining the distance between each pixel in the third and fourth predicted segmentation images, and this fifth loss value is included in the unlabeled loss.

[0060] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain a first pseudo-label, a first predicted probability map, and a second feature map of the local unlabeled sample based on the initial teacher model;

[0061] The local unlabeled feature prototype is obtained by performing confidence-weighted masking average pooling on the first pseudo-label, the first predicted probability map, and the second feature map.

[0062] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to perform pixel-by-pixel cosine similarity processing on the local unlabeled feature prototype and the first feature map to obtain the first prototype propagation prediction.

[0063] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain a second pseudo-label, a second predicted probability map, and a third feature map of the third-party unlabeled sample based on the initial teacher model;

[0064] The second pseudo-label, the second predicted probability map, and the third feature map are subjected to confidence-weighted masking average pooling to obtain the prototype of the third-party unlabeled feature.

[0065] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to perform pixel-by-pixel cosine similarity processing on the third-party unlabeled feature prototype and the first feature map to obtain the second prototype propagation prediction.

[0066] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is further configured to adjust the network parameters of the initial teacher model according to the network parameters of the initial student model.

[0067] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is specifically used to obtain the first network parameters in the previous iteration and the second network parameters in the current iteration of the initial student model;

[0068] The third network parameter is obtained by performing a moving average process based on the first network parameter and the second network parameter;

[0069] Update the third network parameter to the network parameter of the initial teacher model.

[0070] In one possible design, in another implementation of another aspect of the embodiments of this application, the student model and the teacher model can be a U-Net network, a DenseNet network, or a ResNet network.

[0071] In one possible design, in another implementation of another aspect of the embodiments of this application, the locally labeled sample, the locally unlabeled sample, and the third-party unlabeled sample can be either two-dimensional images or three-dimensional images. Another aspect of this application provides a computer device, including: a memory, a processor, and a bus system;

[0072] The memory is used to store programs;

[0073] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0074] Bus systems are used to connect memory and processor to enable communication between them.

[0075] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0076] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0077] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: In the training process of the image segmentation model, a teacher-student model is used to train both locally labeled samples and unlabeled samples. The locally labeled samples are trained by the student model to obtain supervised loss, while the unlabeled samples are trained by comparing class prototypes with pixel features to obtain prototype propagation predictions of the locally labeled samples. This enables the unlabeled samples to achieve supervised learning with labels, thereby making the features of the unlabeled samples closer to the features of the labeled samples, thus reducing the impact of data heterogeneity of third-party unlabeled data and improving the accuracy of image segmentation. Attached Figure Description

[0078] Figure 1a This is a schematic diagram illustrating an application scenario of the image segmentation model in this application embodiment;

[0079] Figure 1b This is a schematic diagram of an application architecture for training an image segmentation model in an embodiment of this application;

[0080] Figure 2 This is a schematic diagram of the training framework of the image segmentation model in an embodiment of this application;

[0081] Figure 3 This is a schematic diagram of an embodiment of the training method for the image segmentation model in this application.

[0082] Figure 4 This is a schematic diagram of another embodiment of the training method for the image segmentation model in this application;

[0083] Figure 5 This is a schematic diagram of another embodiment of the training method for the image segmentation model in this application;

[0084] Figure 6 This is a schematic diagram of another embodiment of the training method for the image segmentation model in this application;

[0085] Figure 7 This is a schematic diagram of one embodiment of the image segmentation method in this application;

[0086] Figure 7a This is a schematic diagram of the visualization results of the experiments in the embodiments of this application;

[0087] Figure 8 This is a schematic diagram of one embodiment of the training device for the image segmentation model in this application.

[0088] Figure 9 This is a schematic diagram of one embodiment of the image segmentation device in this application;

[0089] Figure 9a This is a schematic diagram of another embodiment of the image segmentation device in this application;

[0090] Figure 10 This is a schematic diagram of another embodiment of the image segmentation device or the image segmentation model training device in the embodiments of this application;

[0091] Figure 11 This is a schematic diagram of another embodiment of the image segmentation device or the training device for the image segmentation model in the present application. Detailed Implementation

[0092] This application provides a training method, apparatus, device, and storage medium for an image segmentation model, which reduces the impact of data heterogeneity from unlabeled third-party data and improves the accuracy of image segmentation.

[0093] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0094] In recent years, deep learning-based methods have significantly improved the accuracy and efficiency of medical image segmentation. However, the success of deep learning-based methods typically requires a large amount of labeled medical data to train the image segmentation model, and labeling medical data requires specialized knowledge, making it very expensive. Semi-supervised learning, on the other hand, can utilize both limited labeled data and abundant unlabeled data, and therefore has been widely applied to the training of medical image segmentation models. However, the effectiveness of semi-supervised learning largely depends on the quantity and quality of unlabeled data. In practice, however, medical centers are limited by their ability to acquire medical images and the number of patients, resulting in a limited amount of unlabeled data locally. Therefore, it is necessary to incorporate medical images from other medical centers. Since medical images from other medical centers differ from local images, data heterogeneity is often introduced into semi-supervised learning, affecting the quality of image segmentation. Therefore, there is an urgent need for a solution to address the data heterogeneity problem introduced by third-party unlabeled data and improve the accuracy of image segmentation.

[0095] To address this technical problem, this application provides the following technical solution: A training sample set, an initial student model, and an initial teacher model are obtained. The training sample set includes locally labeled samples, locally unlabeled samples, and third-party unlabeled samples. The locally labeled samples also include ground truth labels. The initial student model and the initial teacher model have the same network structure. Based on the initial student model, a first predicted segmentation image and a first feature map of the locally labeled samples are obtained, and a supervised loss is obtained based on the first predicted segmentation image and the ground truth labels. Based on the initial teacher model, a locally unlabeled feature prototype corresponding to the locally unlabeled samples is obtained, and a third-party unlabeled feature prototype corresponding to the third-party unlabeled samples is obtained using the initial teacher model. A first prototype propagation prediction is obtained based on the locally unlabeled feature prototype and the first feature map, and a second prototype propagation prediction is obtained based on the third-party unlabeled feature prototype and the first feature map. A first loss value is obtained based on the first prototype propagation prediction and the ground truth labels, and a second loss value is obtained based on the second prototype propagation prediction and the ground truth labels. The first loss value and the second loss value serve as the unlabeled loss. The initial student model is trained based on the supervised loss and the unlabeled loss to obtain the image segmentation model. In this way, during the training process of the image segmentation model, a teacher-student model is used for training both locally labeled samples and unlabeled samples. The locally labeled samples are trained through supervised learning by the student model to obtain supervised loss, while the unlabeled samples are trained by comparing class prototypes with pixel features to obtain prototype propagation predictions of the locally labeled samples. This allows the unlabeled samples to achieve supervised learning with labeled samples, thereby making the features of the unlabeled samples closer to the features of the labeled samples. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0096] The training method or image segmentation method of the image segmentation model in the various optional embodiments of this application is based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions.

[0097] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0098] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0099] This application also relates to cloud technology. Cloud technology refers to a managed technology that unifies hardware, software, network, and other system resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing.

[0100] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology applied to cloud computing business models. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of internet behavior, every item may possess its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing. The cloud technology involved in this application mainly refers to image recognition and other similar processes that may occur between terminal devices or servers via the "cloud."

[0101] For ease of understanding, some terms used in this application are explained below.

[0102] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence (AI) and the fundamental way to endow computers with intelligence; its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. With the research and advancement of AI technology, it is being researched and applied in multiple fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with technological development, AI technology will be applied in even more fields and play an increasingly important role.

[0103] Neural Networks: Artificial Neural Networks (ANNs) are composed of numerous neurons connected by adjustable connection weights. They are characterized by large-scale parallel processing, distributed information storage, and good self-organization and self-learning capabilities.

[0104] A convolutional layer (Conv) is a layered structure in a convolutional neural network (CNN) consisting of several convolutional units. A CNN is a type of feedforward neural network that includes at least two neural network layers. Each neural network layer contains several neurons, which are arranged in layers. There are no connections between neurons in the same layer, and information is transmitted between layers in only one direction.

[0105] Backpropagation: Forward propagation refers to the feedforward processing of the model, while backpropagation is the opposite of forward propagation, referring to updating the weight parameters of each layer of the model based on the model's output. For example, if the model includes an input layer, hidden layers, and an output layer, then forward propagation means processing the data in the order of input layer-hidden layer-output layer, while backpropagation means updating the weight parameters of each layer sequentially in the order of output layer-hidden layer-input layer.

[0106] U-Net Network: U-Net is a variant of a fully convolutional network. Therefore, the U-Net network structure is similar to that of the FCN network structure, consisting of a downsampling stage and an upsampling stage. The network structure only contains convolutional layers and pooling layers, without fully connected layers. The shallower, high-resolution layers in the network are used to solve the pixel localization problem, while the deeper layers are used to solve the pixel classification problem, thereby enabling semantic-level segmentation of images.

[0107] DenseNet Network: DenseNet is a classification network that followed ResNet. Its network structure is initially similar to ResNet, starting with a large-scale convolution followed by a pooling layer; then several sub-modules (DenseBlock and TransitionLayer); and finally a pooling layer and a fully connected layer. Taking DenseNet-121 as an example, let's look at its network structure: DenseNet-121 consists of 121 weight layers, including 4 Denseblocks, totaling 2×(6+12+24+16)=116 weight layers, plus 1 convolutional layer for the initial input + 3 transition layers + the final fully connected layer for the output, for a total of 121 layers; during training, the DenseNet-BC structure is used, with a compression factor of 0.5 and a growth rate k=32; the initial convolutional layer has 2k channels, which reduces the 224×224 input image to 112×112 after 7×7 convolutions; Denseblocks are composed of stacked layers, all with the same size: two conv layers of 1×1 + 3×3 (each conv layer = BN + ReLU + Conv); transition layers are formed between Denseblocks, which reduce the number of channels using 1×1 convolutional layers and use average pooling layers with a stride of 2 to halve the height and width. Finally, the output is obtained through 1000 softmax passes of global average pooling + fully connected layers.

[0108] ResNet network: Its network structure typically involves a large-scale convolution followed by a pooling layer; then several consecutive sub-modules (DenseBlock and TransitionLayer); finally, a pooling layer and a fully connected layer. Taking ResNet101 as an example, its network structure is as follows: ResNet101 has 3+4+23+3=33 building blocks, each block has 3 layers, so there are 33x3=99 layers. Adding the first convolutional layer (conv1) and the final fully connected layer (for classification), the total is 99+1+1=101 layers.

[0109] Supervised learning is a training method in machine learning that involves adjusting the parameters of a classifier using a set of samples with known classes to achieve the desired performance. It is also known as supervised training or teacher-guided learning, and is a machine learning task that infers a function from labeled training data. Supervised learning models are generally classified into two types: First, by model form: probabilistic models and non-probabilistic models; second, by whether they model the distribution of observed variables: discriminative models and generative models.

[0110] Semi-supervised learning: A training method in machine learning. Its basic idea is to train on labeled data, then predict unlabeled data, take the sample with the highest prediction confidence and directly label it, and then include this type of sample in the current training sample to continue training until the model's prediction results no longer change.

[0111] This application provides a training method, apparatus, device, and storage medium for an image segmentation model, which can reduce the impact of data heterogeneity from unlabeled third-party data and improve the accuracy of image segmentation. The following describes exemplary applications of the electronic devices provided in this application. These electronic devices can be implemented as various types of user terminals or as servers.

[0112] Electronic devices, by running the training method of the image segmentation model provided in this application embodiment, reduce the impact of data heterogeneity from unlabeled third-party data and improve the accuracy of image segmentation. In other words, this improves the image segmentation accuracy of electronic devices equipped with this image segmentation model.

[0113] See Figure 1a , Figure 1a This is a schematic diagram illustrating an application scenario of the image segmentation scheme provided in this application embodiment. This application scenario includes a front-end A, a back-end device, and a front-end B. Front-end A acquires the image to be processed and uploads it to the back-end device. The back-end device deploys the image segmentation model. After receiving the image to be processed, the back-end device runs the image segmentation model to obtain a segmented image of the image to be processed. The back-end device then sends the segmented image to front-end B. It is understood that front-end A, the back-end device, and front-end B can be the same device or separate, independent devices; no specific limitation is made here.

[0114] See Figure 1b , Figure 1bThis is a schematic diagram of an optional architecture for an image segmentation model training scheme provided in this application embodiment. To support the training scheme of an image segmentation model, the terminal device 100 connects to the server 300 via the network 200, and the server 300 connects to the database 400. The network 200 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The client for implementing the image segmentation model training scheme is deployed on the terminal device 100. The client can run on the terminal device 100 via a browser or as a standalone application (APP). The specific form of the client is not limited here. The server 300 involved in this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal device 100 can be a smartphone, tablet, laptop, PDA, personal computer, smart TV, smartwatch, in-vehicle device, wearable device, etc., but is not limited to these. Terminal device 100 and server 300 can be directly or indirectly connected to network 200 via wired or wireless communication, which is not limited in this application. The number of server 300 and terminal device 100 is also not limited. The solution provided in this application can be completed independently by terminal device 100, independently by server 300, or jointly by terminal device 100 and server 300, which is not specifically limited in this application. In short, database 400 can be regarded as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data in the files. A "database" is a collection of data stored together in a certain way, which can be shared by multiple users, has the lowest possible redundancy, and is independent of applications. Database Management System (DBMS) is a computer software system designed for managing databases, and generally has basic functions such as storage, retrieval, security, and backup.Database management systems can be categorized based on the database model they support, such as relational or Extensible Markup Language (XML); or based on the type of computer they support, such as server clusters or mobile phones; or based on the query language used, such as Structured Query Language (SQL) or XQuery; or based on performance priorities, such as maximum scale or highest operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​simultaneously. In this application, database 400 can be used to store training samples or images to be processed. Of course, the storage location of training samples or images to be processed is not limited to the database; for example, it can also be stored in terminal device 100, a blockchain, or the distributed file system of server 300, etc.

[0115] In some embodiments, both the server 300 and the terminal device 100 can execute the training method of the image segmentation model and the image segmentation method provided in the embodiments of this application.

[0116] In this embodiment, the specific process can be as follows: The terminal device 100 acquires a training sample set, an initial student model, and an initial teacher model; then, the terminal device 100 stores the training sample set in the database 400 or in the memory of the terminal device 100; the server 300 acquires the training sample set from the database 400 or the terminal device 100; then, it trains the initial student model based on the training sample set to obtain the image segmentation model. Finally, the server 300 can deploy the image segmentation model to the terminal device 100 so that the terminal device 100 can call the image segmentation model to perform image segmentation of the image to be processed; or, the server 300 can deploy the image segmentation model on the image processing server so that the image processing server can call the image segmentation model to perform image segmentation of the image to be processed.

[0117] In another embodiment, the terminal device 100 independently executes the training method of the image segmentation model provided in this application embodiment. In this embodiment, the specific process can be as follows: the terminal device 100 obtains a training sample set, an initial student model, and an initial teacher model; then, the terminal device 100 trains the initial student model based on the training sample set to obtain the image segmentation model. Finally, the terminal device 100 can deploy the image segmentation model to itself, so that the terminal device 100 can call the image segmentation model to perform image segmentation of the image to be processed; or, the terminal device 100 can deploy the image segmentation model on the image processing server, so that the image processing server can call the image segmentation model to perform image segmentation of the image to be processed.

[0118] In this application, the training framework of the image segmentation model can be as follows: Figure 2 As shown. In this training framework, the training sample set includes the local sample set. (from tagged subsets) (i.e., locally labeled samples) and unlabeled subsets (i.e., composed of local unlabeled samples) and external unlabeled datasets. (i.e., unlabeled samples by third parties), among which This refers to the number of third-party centers. Specifically, locally labeled samples can be represented as: ,in This represents the number of locally labeled samples. Unlabeled samples can be represented as: ,in This is the number of unlabeled samples in the local sample. Indicates the image height is Width is ,depth The image, and express The tag (i.e., the real tag). Similarly, the first The unlabeled third-party samples are represented as follows: ,in This refers to the number of unlabeled samples from third parties.

[0119] The training framework includes a student model and a teacher model, wherein the student model... The model is optimized through conventional loss backpropagation, while this student model... The network parameters of the teacher model can be obtained by processing with a moving average. The network parameters. In an exemplary scheme, during iteration... middle, Updated to ,in Represents the network parameters of the student model. Represents the network parameters of the teacher model, where This is the decay rate of the Exponential Moving Average (EMA), which can be empirically set to 0.99. Meanwhile, this teacher model... With the student model They have the same network structure. For example, this teacher model. With the student model They are all U-Net networks, DenseNet networks, or ResNet networks.

[0120] In this training framework, the locally labeled sample is used for supervised learning through the student model to obtain the supervised loss. Meanwhile, the local unlabeled sample and the third-party unlabeled sample are used by the teacher model of the student model to obtain the unlabeled loss through prototype propagation prediction and class-supervised learning with pseudo-labels. Then, the initial student model is trained by backpropagation based on the supervised loss and the unlabeled loss until the network parameters of the initial student model converge or the number of learning iterations reaches a threshold. The final student model, also known as the image segmentation model, is then output.

[0121] It is understood that in the specific implementation of this application, data such as the image to be processed and the training sample set are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0122] Based on the above introduction, the training method for the image segmentation model using a server as the execution entity in this application will be described below. Please refer to [link / reference needed]. Figure 3 One embodiment of the training method for the image segmentation model in this application includes:

[0123] 301. Obtain the training sample set, the initial student model, and the initial teacher model. The training sample set includes the locally labeled samples, the locally unlabeled samples, and the third-party unlabeled samples. The locally labeled samples also include real labels. The initial student model and the initial teacher model have the same network structure.

[0124] The server receives locally labeled samples and unlabeled samples uploaded by local front-end devices, as well as unlabeled samples provided by third parties. It uses these locally labeled samples, unlabeled samples, and unlabeled samples from third parties as the training sample set. The labels of the locally labeled samples can also be referred to as ground truth labels. In an exemplary scenario, when the image segmentation model is applied to medical image segmentation, the third party can be other medical centers, and the unlabeled samples from third parties can be unlabeled medical images used by those other medical centers.

[0125] Simultaneously, the server constructs a teacher-student model framework, including the initial student model and the initial teacher model. In this embodiment, the student model... The model is optimized through conventional loss backpropagation, while this student model... The network parameters of the teacher model can be obtained by processing with a moving average. The network parameters. In an exemplary scheme, during iteration... middle, Updated to ,in Represents the network parameters of the student model. Represents the network parameters of the teacher model, where This is the decay rate of the Exponential Moving Average (EMA), which can be empirically set to 0.99. Meanwhile, this teacher model... With the student model They have the same network structure. For example, this teacher model. With the student model They are all U-Net networks, DenseNet networks, or ResNet networks.

[0126] 302. Based on the initial student model, the first predicted segmentation image and the first feature map of the local labeled sample are obtained, and the supervised loss is obtained by processing the first predicted segmentation image and the real label.

[0127] During training, the server transmits the locally labeled sample to the initial student model via a corresponding interface. After the locally labeled sample passes through the initial student model, it outputs the first predicted segmentation image and the first feature map of the locally labeled sample. Then, the server performs loss processing on the first predicted segmentation image and the ground truth label to obtain the supervised loss. This loss processing involves a differential comparison between the first predicted segmentation image and the ground truth label to obtain the loss value of the first predicted segmentation image relative to the ground truth label. The smaller the loss value of the first predicted segmentation image relative to the ground truth label, the closer the first predicted segmentation image is to the ground truth label. In one exemplary scheme, the server can use the Dice loss function or the cross-entropy loss function, or other loss functions, when performing the differential comparison; no specific limitation is made here. Alternatively, the supervised loss can be obtained by the server using the Dice loss function or the cross-entropy loss function, or other loss functions; no specific limitation is made here.

[0128] The first feature map is the locally labeled image before the penultimate convolution of the initial student model. The feature map is represented as .Notice, Upsampled using bilinear interpolation to the same level as Same size, but with There are 1 channel. Therefore, the first feature map has all the features of the locally labeled image.

[0129] 303. Based on the initial teacher model, obtain the prototype of the local unlabeled feature of the local unlabeled sample.

[0130] The server transmits the local unlabeled sample to the initial teacher model through the corresponding interface. After the local unlabeled sample passes through the initial teacher model, it outputs the first predicted probability map, the first pseudo-label, and the second feature map corresponding to the local unlabeled sample. Then, it performs confidence-weighted masking average pooling on the first pseudo-label, the first predicted probability map, and the second feature map to obtain the prototype of the local unlabeled feature.

[0131] In this embodiment, the first prediction probability map is used to indicate the probability value of each pixel in the local unlabeled sample being predicted as the target segmentation object. In an exemplary scheme, the first prediction probability map can be represented as a set of probability values, where each probability value corresponds to a pixel. For example, the first prediction probability map can be represented as follows: (pixel 1, 0), (pixel 2, 0.1), (pixel 3, 0.1), (pixel 4, 0.5), (pixel 5, 0.6), (pixel 7, 0.7), (pixel 8, 0.75), (pixel 8, 0.8), (pixel 9, 0.8), (pixel 10, 0.9), ..., (pixel n, 0).

[0132] The first pseudo-label can be understood as the segmented image predicted by the initial teacher model for the local unlabeled sample.

[0133] The second feature map is the local unlabeled image preceding the penultimate convolution of the initial teacher model. The feature map is represented as .Notice, Upsampled using bilinear interpolation to the same level as Same size, but with Each channel. Therefore, the second feature map contains all the features of the local unlabeled sample.

[0134] In this embodiment, the process by which the server obtains the local unlabeled feature prototype through confidence-weighted masking average pooling of the first pseudo-label, the first predicted probability map, and the second feature map can be understood as follows: for each channel, the first pseudo-label is used to select the second feature map pixel by pixel, and the first predicted probability map is used to set the weight of the prototype processing at that position and sum them up. Simultaneously, the first pseudo-label and the first predicted probability map are summed pixel by pixel, and finally, the two are divided to obtain the average value for each channel, thereby generating a foreground prototype with a length equal to the number of channels. In an exemplary scheme, the specific processing method can adopt the following formula:

[0135]

[0136] Among them, the Used to indicate the foreground prototype of the local unlabeled sample, and the Used to indicate the first pseudo-label, Used to indicate the first predicted probability map, Used to indicate the second feature map.

[0137] Similarly, the background prototype of the unlabeled local sample can be determined in the same way. At this point, simply inverting the value of the first predicted probability map is sufficient. For example, when determining the foreground, pixels in the first predicted probability map that are greater than a preset threshold are defined as the target object to be segmented, while other pixels are other objects besides the target object. Similarly, when determining the background, pixels in the first predicted probability map that are greater than a preset threshold are defined as other objects besides the target object, while other pixels are the target object. For example, in an MRI image where the unlabeled local sample is the pancreas, the foreground is the pancreas, and the background is other objects besides the pancreas.

[0138] In this embodiment, the foreground prototype can be understood as the prototype of the target object to be segmented, and the background prototype can be understood as the prototype of other objects in the sample besides the target object to be segmented. For example, when the local MRI image is not labeled as a pancreas, the foreground prototype is the prototype of the pancreas, and the background prototype is the prototype of other objects besides the pancreas.

[0139] 304. The propagation prediction of the first prototype is obtained based on the local unlabeled feature prototype and the first feature map.

[0140] The server determines the first prototype propagation prediction by pixel-by-pixel determining the probability value between the local unlabeled feature prototype and the first feature map. Specifically, for each spatial location (i.e., each pixel) in the feature map, the similarity between the foreground and background prototypes and the predicted prototype is processed, where T is a temperature function. Then, the softmax function is used to process the predicted probability for each spatial location, thus obtaining the second prototype propagation prediction.

[0141] In one exemplary solution, the following formula can be used:

[0142]

[0143] Among them, the Used to indicate the propagation prediction of the first prototype, Used to indicate the prototype of the local unlabeled feature. Used to indicate the first feature map, Used to indicate cosine similarity, Used to indicate temperature; Used to indicate the foreground prototype of a local unlabeled feature prototype. Used to indicate the background prototype of the local unlabeled feature prototype.

[0144] 305. The first loss value is obtained based on the first prototype propagation prediction and the real label processing.

[0145] In this embodiment, the server uses the first prototype propagation prediction and the real label to process the data according to the corresponding loss processing method to obtain the first loss value. That is, it can be understood as comparing the first prototype propagation prediction with the real label to obtain the loss value between the pixels segmented in the first prototype propagation prediction and the real label.

[0146] Optionally, in order to obtain a high-quality loss value, the server can filter the target pixel set according to the first predicted probability map and the first threshold, wherein only the pixels in the target pixel set participate in determining the first loss value; and then determine the first loss value based on each pixel in the target pixel set and the true label.

[0147] To better learn the local differences of unlabeled samples, the server can divide each pixel in the target pixel set into K regions of the same size, and also divide the real label into corresponding K regions. Then, it can use the corresponding loss processing method to process the K regions in the target pixel set and the K regions in the real label to obtain K loss values ​​respectively. The average of the K loss values ​​is then used to obtain the first loss value.

[0148] It is understood that this loss processing method includes, but is not limited to, the Dice loss function or the cross-entropy loss function. In an exemplary scheme, if the loss processing method is the Dice loss function, then the above loss processing method can be expressed by the following formula:

[0149]

[0150] 306. Based on the initial teacher model, obtain the prototype of the unlabeled features of the third party for the unlabeled samples.

[0151] The server transmits the third-party unlabeled sample to the initial teacher model through the corresponding interface. After the third-party unlabeled sample passes through the initial teacher model, it outputs the second predicted probability map, the second pseudo-label, and the first feature map corresponding to the predicted third-party unlabeled sample. Then, it performs confidence-weighted masking average pooling on the second pseudo-label, the second predicted probability map, and the third feature map to obtain the prototype of the third-party unlabeled feature.

[0152] In this embodiment, the second prediction probability map is used to indicate the probability value of each pixel in the third-party unlabeled sample being predicted as the target segmentation object. In an exemplary scheme, the second prediction probability map can be represented as a set of probability values, where each probability value corresponds to a pixel. For example, the second prediction probability map can be represented as follows: (pixel 1, 0), (pixel 2, 0.1), (pixel 3, 0.1), (pixel 4, 0.5), (pixel 5, 0.6), (pixel 7, 0.7), (pixel 8, 0.75), (pixel 8, 0.8), (pixel 9, 0.8), (pixel 10, 0.9), ..., (pixel n, 0).

[0153] The second pseudo-label can be understood as the segmented image predicted by the initial teacher model for the unlabeled sample from the third party.

[0154] The third feature map consists of unlabeled third-party samples before the penultimate convolution of the initial teacher model. The feature map is represented as .Notice, Upsampled using bilinear interpolation to the same level as Same size, but with There are 1 channel. Therefore, this third feature map contains all the features of the unlabeled samples from this third party.

[0155] In this embodiment, the process by which the server obtains the third-party unlabeled feature prototype through confidence-weighted masking average pooling of the second pseudo-label, the second predicted probability map, and the third feature map can be understood as follows: For each channel, the second pseudo-label is used to select pixels from the third feature map, and the second predicted probability map is used to set the weight of the prototype processing at that position and sum them up. Simultaneously, the second pseudo-label and the second predicted probability map are summed pixel by pixel, and finally, the two are divided to obtain the average value of each channel, thereby generating a foreground prototype with a length equal to the number of channels. In an exemplary scheme, the specific processing method can adopt the following formula:

[0156]

[0157] Among them, the Used to indicate the foreground prototype of the unlabeled sample from the third party, and the Used to indicate the second pseudo-label, Used to indicate the second predicted probability map, Used to indicate the third feature map.

[0158] Similarly, the background prototype of the unlabeled third-party sample can be obtained in the same way. At this point, simply inverting the value of the second prediction probability map is sufficient. For example, when determining the foreground, pixels in the second prediction probability map that exceed a preset threshold are defined as the target object to be segmented, while other pixels are other objects besides the target object. Similarly, when determining the background, pixels in the second prediction probability map that exceed a preset threshold are defined as other objects besides the target object, while other pixels are the target object. For example, in an MRI image where the unlabeled third-party sample is a pancreas, the foreground is the pancreas, and the background is other objects besides the pancreas.

[0159] In this embodiment, the foreground prototype can be understood as the prototype of the target object to be segmented, and the background prototype can be understood as the prototype of other objects in the sample besides the target object to be segmented. For example, when the third-party unlabeled MRI image is of the pancreas, the foreground prototype is the prototype of the pancreas, and the background prototype is the prototype of other objects besides the pancreas.

[0160] 307. The propagation prediction of the second prototype is obtained based on the unlabeled feature prototype of the third party and the prediction of the first feature map.

[0161] The server performs pixel-by-pixel processing on the unlabeled third-party feature prototype and the first feature map to determine the probability value and obtain the second prototype propagation prediction. Specifically, for each spatial location (i.e., each pixel) in the feature map, it processes the similarity between the foreground prototype and the background prototype, where T is a temperature function. Then, it uses a softmax function to process the predicted probability for each spatial location, thereby obtaining the first prototype propagation prediction.

[0162] In one exemplary solution, the following formula can be used:

[0163]

[0164] Among them, the Used to indicate the propagation prediction of the second prototype, Used to indicate that the third party did not annotate the feature prototype. Used to indicate the first feature map, Used to indicate the processing of cosine similarity, Used to indicate temperature; Used to indicate a foreground prototype for a third party without labeled features. Used to indicate the background prototype of a third party without labeled features.

[0165] 308. A second loss value is obtained based on the second prototype propagation prediction and the real label processing. The first loss value and the second loss value are used as the unlabeled loss.

[0166] In this embodiment, the server uses the second prototype propagation prediction and the real label to determine the second loss value according to the corresponding loss processing method. That is, it can be understood as comparing the second prototype propagation prediction with the real label to obtain the loss value between the pixels segmented in the second prototype propagation prediction and the real label.

[0167] Optionally, in order to obtain a high-quality loss value, the server can filter the target pixel set based on the second predicted probability map and the first threshold, wherein only the pixels in the target pixel set participate in determining the second loss value; and then determine the second loss value based on each pixel in the target pixel set and the true label.

[0168] To better learn the local differences of unlabeled samples, the server can divide each pixel in the target pixel set into K regions of the same size, and also divide the real label into corresponding K regions. Then, it can use the corresponding loss processing method to process the K regions in the target pixel set and the K regions in the real label to obtain K loss values ​​respectively. The average of the K loss values ​​is then used to obtain the second loss value.

[0169] It is understood that this loss processing method includes, but is not limited to, the Dice loss function or the cross-entropy loss function. In an exemplary scheme, if the loss processing method is the Dice loss function, then the above loss processing method can be expressed by the following formula:

[0170]

[0171] In this embodiment, the first loss value and the second loss value are used as the unlabeled loss, therefore the unlabeled loss can be expressed as:

[0172]

[0173] It is understandable that the time order of steps 303 to 305 and steps 306 to 308 is not limited. That is, steps 306 to 308 can be executed first and then steps 303 to 305 can be executed, or steps 306 to 308 and steps 303 to 305 can be executed in parallel.

[0174] 309. Train the initial student model based on the supervised loss and the unlabeled loss to obtain the image segmentation model.

[0175] The server sums the supervised loss and the unlabeled loss to obtain the total loss of the initial student model, and then backpropagates the initial student model based on the total loss to obtain the image segmentation model.

[0176] Understandably, when summing the supervised loss and the unlabeled loss, appropriate weight values ​​can be added. In an exemplary scheme, the formula for summing the supervised loss and the unlabeled loss to obtain the total loss can be as follows:

[0177]

[0178] in It is the supervised loss of locally labeled samples. This is indicated as unmarked loss. It typically consists of a Gaussian function that slopes upwards with increasing training time: To avoid being dominated by meaningless predictions in the early training phase, among which and These are the maximum consistency weight and the training step size, respectively.

[0179] It is understandable that, in the above training method, the local unlabeled samples can still undergo pseudo-label-based supervised learning. Based on the above scheme, such as Figure 4 As shown, another embodiment of the training method for the image segmentation model in this application includes:

[0180] Steps 401 to 408 are the same as steps 301 to 308, and will not be described in detail here.

[0181] 409. Based on the initial student model, obtain the second predicted segmentation image of the local unlabeled sample, and based on the initial teacher model, obtain the first pseudo-label of the local unlabeled sample.

[0182] The server transmits the local unlabeled sample to the initial student model through the corresponding interface, and after the local unlabeled sample passes through the initial student model, it outputs the second predicted segmentation image and the second feature map of the local unlabeled sample. The server also transmits the local unlabeled sample to the initial teacher model through the corresponding interface, and after the local unlabeled sample passes through the initial teacher model, it outputs the first pseudo label.

[0183] In this embodiment, the first pseudo-label can be understood as the predicted segmentation image obtained by the local unlabeled sample through the initial teacher model.

[0184] 410. A third loss value is obtained by processing the second predicted segmentation image and the first pseudo-label, and the third loss value is included in the unlabeled loss.

[0185] The server uses the second predicted segmentation image and the first pseudo-label to process the data according to the corresponding loss processing method to obtain the first loss value.

[0186] Optionally, in order to obtain a high-quality loss value, the server can filter the target pixel set based on the first predicted probability map corresponding to the second predicted segmentation image and a first threshold, wherein only the pixels in the target pixel set participate in determining the third loss value; and then determine the third loss value based on each pixel in the target pixel set and the true label.

[0187] To better learn the local differences of unlabeled samples, the server can divide each pixel in the target pixel set into K regions of the same size, and also divide the real label into corresponding K regions. Then, it can use the corresponding loss processing method to process the K regions in the target pixel set and the K regions in the real label to obtain K loss values ​​respectively. The average of the K loss values ​​is then used to obtain the third loss value.

[0188] It is understood that this loss processing method includes, but is not limited to, the Dice loss function or the cross-entropy loss function. In an exemplary scheme, if the loss processing method is the Dice loss function, then the above loss processing method can be expressed by the following formula:

[0189]

[0190] In this embodiment, the first loss value, the second loss value, and the third loss value are considered as the unlabeled loss. Therefore, the unlabeled loss can be expressed as:

[0191]

[0192] 411. The image segmentation model is obtained by training the initial student model based on the supervised operation and the unlabeled loss.

[0193] The server sums the supervised loss and the unlabeled loss to obtain the total loss of the initial student model, and then backpropagates the initial student model based on the total loss to obtain the image segmentation model.

[0194] It is understandable that, in the above training methods, during the training process of the image segmentation model, class feature regularization loss can also be introduced. Based on the above scheme, such as... Figure 5 As shown, another embodiment of the training method for the image segmentation model in this application includes:

[0195] Steps 501 to 510 are the same as steps 401 to 410, and will not be described in detail here.

[0196] 511. Based on the initial student model, the prototype of the local annotation features of the local annotation sample is obtained.

[0197] The server transmits the local labeled sample to the initial student model through the corresponding interface, and outputs the first feature map after the local labeled sample passes through the initial student model; then, the local labeled feature prototype is obtained by performing confidence-weighted masking average pooling on the real label and the first feature map.

[0198] In this embodiment, the probability map corresponding to the real label can be represented as a set of probability values, where each probability value corresponds to a pixel. For example, the second predicted probability map can be represented as follows: (pixel 1, 0), (pixel 2, 0), (pixel 3, 0), (pixel 4, 0), (pixel 5, 0), (pixel 7, 0), (pixel 8, 0), (pixel 8, 1), (pixel 9, 1), (pixel 10, 1), ..., (pixel n, 0).

[0199] The first feature map is the locally labeled image before the penultimate convolution of the initial student model. The feature map is represented as .Notice, Upsampled using bilinear interpolation to the same level as Same size, but with There are 1 channel. Therefore, the first feature map contains all the features of the locally labeled sample.

[0200] In this embodiment, the server's process of obtaining the local labeled feature prototype by performing confidence-weighted masking average pooling on the real label and the first feature map can be understood as follows: For each channel, the real label is used to select pixels from the first feature map, and the probability map corresponding to the real label is used to set the weight of the position for prototype processing and summed. Simultaneously, the real label and its corresponding probability map are summed pixel by pixel, and finally, the two are divided to obtain the average value of each channel, thereby generating a foreground prototype with a length equal to the number of channels. In an exemplary scheme, the specific processing method can adopt the following formula:

[0201]

[0202] Among them, the Used to indicate the foreground prototype of the local labeled sample, and the Used to indicate the true label, This is used to indicate the predicted probability map corresponding to the true label. Used to indicate the first feature map.

[0203] Similarly, the background prototype of the unlabeled local sample can be obtained in the same way. At this point, simply invert the value of the probability map corresponding to the ground truth label. For example, when determining the foreground, pixels with a probability map value of 1 corresponding to the ground truth label are designated as the target object to be segmented, while other pixels are other objects besides the target object. Similarly, when determining the background, pixels with a probability map value of 1 corresponding to the ground truth label are designated as other objects besides the target object, while other pixels are the target object itself. For example, in an unlabeled MRI image of a pancreas, the foreground represents the pancreas, and the background represents other objects besides the pancreas.

[0204] In this embodiment, the foreground prototype can be understood as the prototype of the target object to be segmented, and the background prototype can be understood as the prototype of other objects in the sample besides the target object to be segmented. For example, when the local MRI image is not labeled as a pancreas, the foreground prototype is the prototype of the pancreas, and the background prototype is the prototype of other objects besides the pancreas.

[0205] 512. Based on the local labeled feature prototype, the local unlabeled feature prototype and the third-party unlabeled feature prototype, perform category feature regularization to obtain the fourth loss value, which is included in the unlabeled loss.

[0206] The server obtains the first foreground feature prototype and the first background feature prototype of the locally labeled feature prototype, the second foreground feature prototype and the second background feature prototype of the locally unlabeled feature prototype, and the third foreground feature prototype and the third background feature prototype of the third party unlabeled feature prototype; determines the first distance function between the third foreground feature prototype and the second foreground feature prototype and the first foreground feature prototype, and determines the second distance function between the third background feature prototype and the second background feature prototype and the first background feature prototype; and performs a weighted summation of the first distance function and the second distance function to obtain the fourth loss value.

[0207] Understandably, this category feature regularization refers to processing the similarity between foreground prototypes and the similarity between background prototypes for each spatial location among the unlabeled third-party feature prototypes, the locally labeled feature prototypes, and the unlabeled local feature prototypes; then, weighted summation is used to obtain the final loss value. In an exemplary scheme, the category feature regularization formula can be as follows:

[0208]

[0209] in, Used to indicate the foreground prototype of the locally labeled sample. Used to indicate the foreground prototype of the unlabeled sample from the third party. Used to indicate the foreground prototype of the local unlabeled sample. Used to indicate the background prototype of the local labeled sample Used to indicate the background prototype of the unlabeled sample from the third party. Used to indicate the background prototype of the local unlabeled sample.

[0210] In this embodiment, mean square error can be used as the distance function. Other errors can also be used as the distance function; specific methods are not limited here. When setting the weights, since foreground information is more important than background information, the weight of the foreground can be set greater than the weight of the background. It is understood that the weights in the above formula are only one possible implementation, and specific implementation methods are not limited here.

[0211] In this embodiment, the first loss value, the second loss value, the third loss value, and the fourth loss value are considered as the unlabeled loss. Therefore, the unlabeled loss can be expressed as:

[0212]

[0213] 513. The image segmentation model is obtained by training the initial student model based on the supervised operation and the unlabeled loss.

[0214] The server sums the supervised loss and the unlabeled loss to obtain the total loss of the initial student model, and then backpropagates the initial student model based on the total loss to obtain the image segmentation model.

[0215] It is understandable that, in the above training methods, perturbation learning loss can be introduced during the training process of the image segmentation model. Based on the above scheme, such as... Figure 6 As shown, another embodiment of the training method for the image segmentation model in this application includes:

[0216] Steps 601 to 612 are the same as steps 501 to 512, and will not be described in detail here.

[0217] 613. Preprocess the local unlabeled sample and the third-party unlabeled sample to obtain the first perturbation sample and the second perturbation sample.

[0218] In this embodiment, the server can use the following possible preprocessing methods to obtain the first perturbation sample and the second perturbation sample from the local unlabeled sample and the third-party unlabeled sample:

[0219] One possible implementation involves adding two different types of Gaussian noise to both the local unlabeled sample and the third-party unlabeled sample.

[0220] In another possible implementation, image enhancement is performed using different intensities on the local unlabeled sample and the third-party unlabeled sample.

[0221] In another possible implementation, the local unlabeled sample and the third-party unlabeled sample are rotated using different rotation angles.

[0222] 614. Based on the initial student model, obtain the third predicted segmentation image of the first perturbation sample, and based on the initial teacher model, obtain the fourth predicted segmentation image of the second perturbation sample.

[0223] In this embodiment, the server transmits the first perturbation sample to the initial student model through a corresponding interface. After the first perturbation sample passes through the initial student model, it outputs the predicted third segmentation image. Similarly, the server transmits the second perturbation sample to the initial teacher model through a corresponding interface. After the second perturbation sample passes through the initial teacher model, it outputs the predicted fourth segmentation image. It is understood that the third and fourth predicted segmentation images are predicted segmentation images of the same sample.

[0224] 615. Determine the distance between each pixel in the third predicted segmentation image and the fourth predicted segmentation image to obtain the fifth loss value, which is included in the unlabeled loss.

[0225] In this embodiment, the server performs pixel-by-pixel mean square error processing on the third predicted segmentation image and the fourth predicted segmentation image to obtain the distance function, which is used as the fifth loss value.

[0226] At this point, the fifth loss value can be expressed as follows:

[0227]

[0228] Among them, the Used to indicate the fourth predicted segmentation image, Used to indicate the third predicted segmentation image.

[0229] In this embodiment, the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value are considered as the unlabeled loss. Therefore, the unlabeled loss can be expressed as:

[0230]

[0231] Overall, this unlabeled loss can also be represented as follows:

[0232]

[0233] 616. The image segmentation model is obtained by training the initial student model based on the supervised operation and the unlabeled loss.

[0234] The server sums the supervised loss and the unlabeled loss to obtain the total loss of the initial student model, and then backpropagates the initial student model based on the total loss to obtain the image segmentation model.

[0235] It is understandable that the above Figures 4 to 6 Only the class feature regularization loss, pseudo-label class supervised learning loss, and perturbation learning loss are described, which can be related to this. Figure 3 The training method shown is a partial combination scheme. In practical applications, the aforementioned class feature regularization loss, pseudo-label class supervised learning loss, and perturbation learning loss can be combined with this... Figure 3 The training methods shown can be combined in any way, and no specific restrictions are imposed here.

[0236] Based on the image segmentation model trained using the above methods, the following description uses a terminal device as the execution subject to illustrate this image segmentation method. Please refer to [link / reference needed] for details. Figure 7 As shown, one embodiment of the image segmentation method includes:

[0237] 701. Obtain the image to be processed.

[0238] In this embodiment, the terminal device receives an image sent by a third party or an image captured by the terminal device itself as the image to be processed.

[0239] Optionally, the type of the image to be processed should be consistent with the type of the training samples in the image segmentation model to achieve better segmentation results. For example, if the image segmentation model is trained using two-dimensional images, then the image to be processed is a two-dimensional image or a two-dimensional image obtained after processing; if the image segmentation model is trained using three-dimensional images, then the image to be processed is a three-dimensional image or a three-dimensional image obtained after processing.

[0240] 702. Obtain the segmented image of the image to be processed based on the image segmentation model.

[0241] The image to be processed is transmitted to the input network of the image segmentation model through the corresponding interface, and then the segmented image of the image to be processed is obtained through the image segmentation model.

[0242] It is understood that the image segmentation model in this embodiment adopts the above-mentioned... Figures 3 to 6 The technical solution shown in any one of the embodiments is trained, and the specific details will not be repeated here.

[0243] 703. Output the segmented image.

[0244] The segmented image is output through the output network layer of the image segmentation model, and then the segmented image is output to the display module of the terminal device through the corresponding interface or to a third-party device through a network connection.

[0245] The beneficial effects of the technical solution provided in this application are illustrated below with specific experimental data:

[0246] I. Experimental Data

[0247] In this study, T2-weighted magnetic resonance (MR) images of the prostate from six different clinical centers (C1-6) were selected for retrospective evaluation. The data are summarized in Table 1.

[0248] Table 1

[0249]

[0250] The heterogeneity of the data from different centers primarily stemmed from differences in scanner, field strength, coil type, disease, and in-plane / in-plane resolution. Compared to C1 and C2, scans from C3 to C6, originating from prostate cancer patients and used for detection or staging purposes, resulted in inherent semantic differences in the prostate region, further exacerbating the heterogeneity. As a preprocessing step, each scan was cropped to retain only slices of the prostate region, then resized to 384x384 pixels in the axial plane, and the images were normalized to a variance of 1 and a mean of 0. Using C1 or C2 as local target centers, 30 MRI images from them were randomly divided into 18, 3, and 9 samples, respectively, serving as the training, validation, and test sets.

[0251] II. Experimental Details and Evaluation Indicators

[0252] This framework is implemented using PyTorch on an NVIDIA GeForce RTX 3090 GPU. Considering the significant variability in the thickness of different center slices, we employ a 2D architecture. Specifically, a 2D U-Net is used as the backbone network for both the student and teacher models. The input block size is set to 384x384, and the batch size is set to 36, comprising 12 labeled local slices, 12 unlabeled local slices, and 12 unlabeled external slices. Supervision loss... Includes Dice loss and cross-entropy loss. Maximum consistency weights. Set it to 0.1. Set it to 20000. Empirically set to 2. Train the network using the SGD optimizer with an initial learning rate of 0.01, multiplied by... To mitigate this, we employed data augmentation techniques, including random flipping and rotation. We used the Dice similarity coefficient (DSC) and Jaccard as evaluation metrics.

[0253] Baseline comparison method:

[0254] The technical solution provided in this application is compared with fully supervised baselines and some semi-supervised medical image segmentation methods, including: MT, UA-MT, Interpolation Consistency Training (ICT), Cross Consistency Training (CCT), CPCL, CPS, and SSNet. It is also compared with AHDC, which uses heterogeneous data.

[0255] Experimental results:

[0256] Table 2 shows the performance of the method provided in this application and other state-of-the-art semi-supervised methods with 6 or 8 labeled data sets, respectively, when C1 or C2 is used as the local target center. For example, if C1 is used as the local center, then the remaining data C2-C6 are used as unlabeled data to assist C1. Under the same partitioning protocol, the technical solution provided in this application achieves a significant improvement compared to existing technologies, demonstrating the superiority and robustness of the technical solution provided in this application. In this experiment, Table 2 provides a quantitative comparison. express (The technical solution provided in this application is compared with other methods using the Wilcoxon signed-rank test). Standard deviation is shown in parentheses.

[0257] Table 2

[0258]

[0259]

[0260] The visualization results can be found in [reference]. Figure 7a As shown, by Figure 7a It is evident that the solution provided in this application represents a significant improvement over other solutions.

[0261] The training apparatus for the image segmentation model in this application is described in detail below. Please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the training apparatus for the image segmentation model in this application. The training apparatus 20 for the image segmentation model includes:

[0262] The acquisition module 201 is used to acquire a training sample set, an initial student model and an initial teacher model. The training sample set includes locally labeled samples, locally unlabeled samples and third-party unlabeled samples. The locally labeled samples also include real labels. The initial student model and the initial teacher model have the same network structure.

[0263] Processing module 202 is configured to obtain a first predicted segmentation image and a first feature map of the locally labeled sample based on the initial student model, and obtain the supervised loss based on the first predicted segmentation image and the ground truth label; obtain the local unlabeled feature prototype corresponding to the local unlabeled sample based on the initial teacher model, and obtain the third-party unlabeled feature prototype corresponding to the third-party unlabeled sample using the initial teacher model; predict a first prototype propagation prediction based on the local unlabeled feature prototype and the first feature map, and predict a second prototype propagation prediction based on the third-party unlabeled feature prototype and the first feature map; obtain a first loss value based on the first prototype propagation prediction and the ground truth label, and obtain a second loss value based on the second prototype propagation prediction and the ground truth label, wherein the first loss value and the second loss value are used as the unlabeled loss;

[0264] Training module 203 is used to train the initial student model based on the supervised loss and the unlabeled loss to obtain the image segmentation model.

[0265] This application provides a training apparatus for an image segmentation model. Using this apparatus, a teacher-student model is employed to train both locally labeled and unlabeled samples. The locally labeled samples undergo supervised learning through the student model to obtain a supervised loss, while the unlabeled samples obtain prototype propagation predictions by comparing class prototypes with pixel features. This allows the unlabeled samples to achieve supervised learning similar to labeled samples, thereby making the features of the unlabeled samples more similar to those of the labeled samples. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0266] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0267] The processing module is also used to obtain a second predicted segmentation image of the local unlabeled sample based on the initial student model, and to obtain a first pseudo-label of the local unlabeled sample based on the initial teacher model;

[0268] A third loss value is obtained by processing the second predicted segmentation image and the first pseudo-label, and the third loss value is included in the unlabeled loss.

[0269] This application provides a training apparatus for an image segmentation model. Using this apparatus, pseudo-labels are learned from the teacher model for unlabeled local samples. These pseudo-labels are then used for supervised learning, enabling the learning model to better fit the local distribution and preventing overfitting of labeled local samples during supervised learning.

[0270] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0271] The processing module 202 is specifically used to obtain the first prediction probability map of the second predicted segmentation image;

[0272] The target pixel set is determined based on the first predicted probability map and the first threshold, and each pixel in the target pixel set is used to determine the third loss value;

[0273] The probability value of each pixel in the target pixel set is processed with the first pseudo-label to obtain the third loss value.

[0274] This application provides a training apparatus for an image segmentation model. Using this apparatus, the pixels of the unlabeled local samples are filtered using the predicted probability map obtained from the initial teacher model and a given threshold, thereby obtaining higher-quality pseudo-labels. This results in better-quality processed images and ultimately improves the segmentation accuracy of the image segmentation model.

[0275] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0276] The processing module 202 is specifically used to divide the pixels in the target pixel set into K regions of the same size, and to divide the first pseudo-label into K regions of the same size in a way that corresponds one-to-one with the target pixel set, where K is greater than or equal to 2;

[0277] K loss values ​​are obtained from K regions in the target pixel set and K regions of the first pseudo-label, respectively.

[0278] The third loss value is obtained by averaging the K loss values.

[0279] This application provides a training apparatus for an image segmentation model. Using this apparatus, a local unlabeled image is processed by region-based loss values, and the final loss value is obtained by averaging these values. This helps the model better perceive local differences, thereby improving the accuracy of the image segmentation model in image segmentation.

[0280] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application, the processing module 202 is further used to obtain the local annotation feature prototype of the local annotation sample based on the initial student model;

[0281] The fourth loss value is obtained by performing categorical feature regularization on the local labeled feature prototype, the local unlabeled feature prototype, and the third-party unlabeled feature prototype. This fourth loss value is included in the unlabeled loss.

[0282] This application provides a training apparatus for an image segmentation model. Using this apparatus, after learning the prototypes of local unlabeled samples and the prototypes of third-party unlabeled samples, the prototypes of local labeled samples are then learned. Class feature regularization is then performed based on the prototypes of the local labeled samples, the prototypes of the local unlabeled samples, and the prototypes of the third-party unlabeled samples. This allows the unlabeled samples to align with the class prototypes of the local labeled samples, making the features of the unlabeled samples closer to the features of the local labeled samples. This reduces the impact of data heterogeneity of unlabeled samples and improves the image segmentation accuracy of the image segmentation model.

[0283] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application, the processing module 202 is specifically used to obtain the first foreground feature prototype and the first background feature prototype of the locally labeled feature prototype, the second foreground feature prototype and the second background feature prototype of the locally unlabeled feature prototype, and the third foreground feature prototype and the third background feature prototype of the third party unlabeled feature prototype.

[0284] Obtain the first distance function between the third foreground feature prototype and the second foreground feature prototype and the first foreground feature prototype, and obtain the second distance function between the third background feature prototype and the second background feature prototype and the first background feature prototype;

[0285] The fourth loss value is obtained by weighting and summing the first and second distance functions.

[0286] This application provides a training apparatus for an image segmentation model. Using this apparatus, foreground and background prototypes are processed separately, and a loss function for feature regularization is obtained by processing the distance function between each pixel. This allows the features of the unlabeled sample to more closely resemble the features of the locally labeled sample through adjustment of the distance function, thereby reducing the impact of data heterogeneity of the unlabeled sample and improving the image segmentation accuracy of the image segmentation model.

[0287] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0288] The processing module 202 is also used to preprocess the local unlabeled sample and the third-party unlabeled sample to obtain the first perturbation sample and the second perturbation sample;

[0289] The third predicted segmentation image of the first perturbation sample is obtained based on the initial student model, and the fourth predicted segmentation image of the second perturbation sample is obtained based on the initial teacher model.

[0290] A fifth loss value is obtained by determining the distance between each pixel in the third and fourth predicted segmentation images, and this fifth loss value is included in the unlabeled loss.

[0291] This application provides a training apparatus for an image segmentation model. Using this apparatus, artificial noise perturbations are added to unlabeled samples, which is considered to increase the simulation of heterogeneity of the unlabeled samples, thereby improving the generalization ability of the image segmentation model and ultimately enhancing its segmentation accuracy.

[0292] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0293] The processing module 202 is specifically used to obtain, based on the initial teacher model, the first pseudo-label, the first predicted probability map, and the second feature map of the local unlabeled sample;

[0294] The local unlabeled feature prototype is obtained by performing confidence-weighted masking average pooling on the first pseudo-label, the first predicted probability map, and the second feature map.

[0295] This application provides a training apparatus for an image segmentation model. By using the real labels of locally labeled samples as guidance, the propagation of erroneous prototypes caused by false labels generated from unlabeled samples is prevented, thus ensuring the correctness of the prototype propagation prediction for locally labeled samples. This, in turn, makes the features of the unlabeled sample closer to the features of the labeled sample, improving the training effect and accuracy of the image segmentation model.

[0296] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0297] The processing module 202 is specifically used to perform pixel-by-pixel cosine similarity processing on the local unlabeled feature prototype and the first feature map to obtain the propagation prediction of the first prototype.

[0298] In this embodiment, a training apparatus for an image segmentation model is provided. Using this apparatus, the unlabeled sample obtains the prototype propagation prediction of the locally labeled sample by comparing the distance between the class prototype and pixel features. This enables the unlabeled sample to achieve supervised learning with labeled data, thereby making the features of the unlabeled sample closer to the features of the labeled sample. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0299] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0300] The processing module 202 is specifically used to obtain the second pseudo-label, the second predicted probability map, and the third feature map of the third-party unlabeled sample based on the initial teacher model.

[0301] The second pseudo-label, the second predicted probability map, and the third feature map are subjected to confidence-weighted masking average pooling to obtain the prototype of the third-party unlabeled feature.

[0302] This application provides a training apparatus for an image segmentation model. By using the real labels of locally labeled samples as guidance, the propagation of erroneous prototypes caused by false labels generated from unlabeled samples is prevented, thus ensuring the correctness of the prototype propagation prediction for locally labeled samples. This, in turn, makes the features of the unlabeled sample closer to the features of the labeled sample, improving the training effect and accuracy of the image segmentation model.

[0303] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application,

[0304] The processing module 202 is specifically used to perform pixel-by-pixel cosine similarity processing on the third-party unlabeled feature prototype and the first feature map to obtain the second prototype propagation prediction.

[0305] In this embodiment, a training apparatus for an image segmentation model is provided. Using this apparatus, the unlabeled sample obtains the prototype propagation prediction of the locally labeled sample by comparing the distance between the class prototype and pixel features. This enables the unlabeled sample to achieve supervised learning with labeled data, thereby making the features of the unlabeled sample closer to the features of the labeled sample. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0306] Optionally, in the above Figure 8Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application, the processing module 202 is further configured to adjust the network parameters of the initial teacher model according to the network parameters of the initial student model.

[0307] This application provides a training apparatus for an image segmentation model. Using this apparatus, the network parameters of the student model are adjusted to adjust the network parameters of the teacher model. This ensures that the student model and the teacher model maintain a connection, preventing significant abrupt changes in the teacher model that could negatively impact the overall training performance of the image segmentation model.

[0308] Optionally, in the above Figure 8 Based on the corresponding embodiments, in another embodiment of the image segmentation model training device 20 provided in this application, the processing module 202 is specifically used to obtain the first network parameters in the previous iteration and the second network parameters in the current iteration of the initial student model;

[0309] The third network parameter is obtained by performing a moving average process based on the first network parameter and the second network parameter;

[0310] Update the third network parameter to the network parameter of the initial teacher model.

[0311] In this embodiment, an image segmentation apparatus is provided. Using this apparatus, the network parameters of the student model are processed by a moving average to obtain the network parameters of the teacher model. This ensures the connection between the student model and the teacher model while increasing the self-integration capability of the teacher model, thereby smoothing noise and preventing abrupt changes in prediction results.

[0312] The image segmentation apparatus of this application is described in detail below. Please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of one embodiment of the image segmentation apparatus in this application. The image segmentation apparatus 30 includes:

[0313] The acquisition module 301 is used to acquire the image to be processed;

[0314] Processing module 302 is used to obtain a segmented image of the image to be processed based on an image segmentation model, wherein the image segmentation model is based on... Figures 3 to 6 The training method shown is used to obtain the results;

[0315] Output module 303 is used to output the segmented image.

[0316] This application provides an image segmentation apparatus. Using this apparatus, a teacher-student model is used to train both locally labeled samples and unlabeled samples. The locally labeled samples undergo supervised learning through the student model to obtain a supervised loss, while the unlabeled samples obtain prototype propagation predictions by comparing class prototypes with pixel features. This allows the unlabeled samples to achieve supervised learning similar to labeled samples, thereby making the features of the unlabeled samples more similar to those of the labeled samples. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0317] Optionally, in the above Figure 9 Based on the corresponding embodiments, such as Figure 9a As shown, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0318] The acquisition module 301 is used to acquire a training sample set, an initial student model and an initial teacher model. The training sample set includes locally labeled samples, locally unlabeled samples and third-party unlabeled samples. The locally labeled samples also include real labels. The initial student model and the initial teacher model have the same network structure.

[0319] Processing module 302 is configured to obtain a first predicted segmentation image and a first feature map of the locally labeled sample based on the initial student model, and obtain the supervised loss based on the first predicted segmentation image and the ground truth label; obtain the local unlabeled feature prototype corresponding to the local unlabeled sample based on the initial teacher model, and obtain the third-party unlabeled feature prototype corresponding to the third-party unlabeled sample using the initial teacher model; predict a first prototype propagation prediction based on the local unlabeled feature prototype and the first feature map, and predict a second prototype propagation prediction based on the third-party unlabeled feature prototype and the first feature map; obtain a first loss value based on the first prototype propagation prediction and the ground truth label, and obtain a second loss value based on the second prototype propagation prediction and the ground truth label, wherein the first loss value and the second loss value are used as the unlabeled loss;

[0320] Training module 304 is used to train the initial student model based on the supervised loss and the unlabeled loss to obtain the image segmentation model.

[0321] This application provides an image segmentation apparatus. Using this apparatus, a teacher-student model is used to train both locally labeled samples and unlabeled samples. The locally labeled samples undergo supervised learning through the student model to obtain a supervised loss, while the unlabeled samples obtain prototype propagation predictions by comparing class prototypes with pixel features. This allows the unlabeled samples to achieve supervised learning similar to labeled samples, thereby making the features of the unlabeled samples more similar to those of the labeled samples. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0322] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application, the processing module 302 is further configured to obtain a second predicted segmentation image of the local unlabeled sample based on the initial student model, and to obtain a first pseudo-label of the local unlabeled sample based on the initial teacher model;

[0323] A third loss value is obtained by processing the second predicted segmentation image and the first pseudo-label, and the third loss value is included in the unlabeled loss.

[0324] This application provides an image segmentation apparatus. Using this apparatus, pseudo-labels are learned from the teacher model for unlabeled local samples. These pseudo-labels are then used for supervised learning, enabling the learning model to better fit the local distribution and preventing overfitting of labeled local samples during supervised learning.

[0325] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0326] The processing module 302 is specifically used to obtain the first prediction probability map of the second predicted segmentation image;

[0327] The target pixel set is determined based on the first predicted probability map and the first threshold, and each pixel in the target pixel set is used to determine the third loss value;

[0328] The probability value of each pixel in the target pixel set is processed with the first pseudo-label to obtain the third loss value.

[0329] In this embodiment, an image segmentation apparatus is provided. Using this apparatus, the pixels of the local unlabeled sample are filtered using the predicted probability map obtained from the initial teacher model and a given threshold, thereby obtaining higher quality pseudo-labels. This results in better quality processed images and ultimately improves the segmentation accuracy of the image segmentation model.

[0330] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0331] The processing module 302 is specifically used to divide the pixels in the target pixel set into K regions of the same size, and to divide the first pseudo-label into K regions of the same size in a way that corresponds one-to-one with the target pixel set, where K is greater than or equal to 2;

[0332] K loss values ​​are obtained from K regions in the target pixel set and K regions of the first pseudo-label, respectively.

[0333] The third loss value is obtained by averaging the K loss values.

[0334] This application provides an image segmentation apparatus. Using this apparatus, a local unlabeled image is processed by region-based loss values, and the final loss value is obtained by averaging these values. This helps the model better perceive local differences, thereby improving the accuracy of the image segmentation model in image segmentation.

[0335] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application, the processing module 302 is further used to obtain the local annotation feature prototype of the local annotation sample based on the initial student model;

[0336] The fourth loss value is obtained by performing categorical feature regularization on the local labeled feature prototype, the local unlabeled feature prototype, and the third-party unlabeled feature prototype. This fourth loss value is included in the unlabeled loss.

[0337] This application provides an image segmentation apparatus. Using this apparatus, after learning the prototypes of local unlabeled samples and the prototypes of third-party unlabeled samples, the apparatus then learns the prototypes of local labeled samples. Furthermore, it performs class feature regularization processing based on the prototypes of the local labeled samples, the prototypes of the local unlabeled samples, and the prototypes of the third-party unlabeled samples. This allows the unlabeled samples to align with the class prototypes of the local labeled samples, making the features of the unlabeled samples closer to the features of the local labeled samples. This reduces the impact of data heterogeneity of unlabeled samples and improves the image segmentation accuracy of the image segmentation model.

[0338] Optionally, in the above Figure 9aBased on the corresponding embodiments, in another embodiment of the image segmentation device 30 provided in this application, the processing module 302 is specifically used to obtain the first foreground feature prototype and the first background feature prototype of the locally labeled feature prototype, the second foreground feature prototype and the second background feature prototype of the locally unlabeled feature prototype, and the third foreground feature prototype and the third background feature prototype of the third party unlabeled feature prototype.

[0339] Obtain the first distance function between the third foreground feature prototype and the second foreground feature prototype and the first foreground feature prototype, and obtain the second distance function between the third background feature prototype and the second background feature prototype and the first background feature prototype;

[0340] The fourth loss value is obtained by weighting and summing the first and second distance functions.

[0341] This application provides an image segmentation apparatus. Using this apparatus, foreground and background prototypes are processed separately, and a loss function for feature regularization is obtained by processing the distance function between each pixel. This allows the features of the unlabeled sample to more closely resemble the features of the locally labeled sample through adjustment of the distance function, thereby reducing the impact of data heterogeneity of the unlabeled sample and improving the image segmentation accuracy of the image segmentation model.

[0342] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0343] The processing module 302 is also used to preprocess the local unlabeled sample and the third-party unlabeled sample to obtain the first perturbation sample and the second perturbation sample;

[0344] The third predicted segmentation image of the first perturbation sample is obtained based on the initial student model, and the fourth predicted segmentation image of the second perturbation sample is obtained based on the initial teacher model.

[0345] A fifth loss value is obtained by determining the distance between each pixel in the third and fourth predicted segmentation images, and this fifth loss value is included in the unlabeled loss.

[0346] This application provides an image segmentation apparatus. By using this apparatus, artificial noise perturbation is added to unlabeled samples. This is considered to increase the simulation of heterogeneity of unlabeled samples, enabling the image segmentation model to achieve better generalization and thus improving the segmentation accuracy of the image segmentation model.

[0347] Optionally, in the above Figure 9aBased on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0348] The processing module 302 is specifically used to obtain, based on the initial teacher model, the first pseudo-label, the first predicted probability map, and the second feature map of the local unlabeled sample.

[0349] The first pseudo-label, the first predicted probability map, and the second feature map are subjected to confidence-weighted masked average pooling to obtain the local unlabeled feature prototype. In this embodiment, an image segmentation apparatus is provided. Using the above apparatus, the true labels of locally labeled samples are used as guidance to prevent the propagation of erroneous prototypes caused by pseudo-labels generated from unlabeled samples, thereby ensuring the correctness of the prototype propagation prediction of locally labeled samples. This makes the features of the unlabeled sample closer to the features of the labeled sample, improving the training effect of the image segmentation model and the accuracy of image segmentation.

[0350] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0351] The processing module 302 is specifically used to perform pixel-by-pixel cosine similarity processing on the local unlabeled feature prototype and the first feature map to obtain the propagation prediction of the first prototype.

[0352] In this embodiment, an image segmentation apparatus is provided. Using this apparatus, the unlabeled sample obtains the prototype propagation prediction of the locally labeled sample by comparing the distance between the class prototype and pixel features. This enables the unlabeled sample to achieve supervised learning with labeled data, thereby making the features of the unlabeled sample closer to the features of the labeled sample. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0353] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0354] The processing module 302 is specifically used to obtain the second pseudo-label, the second predicted probability map, and the third feature map of the third-party unlabeled sample based on the initial teacher model.

[0355] The second pseudo-label, the second predicted probability map, and the third feature map are subjected to confidence-weighted masking average pooling to obtain the prototype of the third-party unlabeled feature.

[0356] This application provides an image segmentation apparatus. By using the actual labels of locally labeled samples as guidance, the propagation of erroneous prototypes caused by false labels generated by unlabeled samples is prevented, thus ensuring the correctness of the prototype propagation prediction for locally labeled samples. This makes the features of the unlabeled sample closer to the features of the labeled sample, improving the training effect of the image segmentation model and the accuracy of image segmentation.

[0357] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application,

[0358] The processing module 302 is specifically used to perform pixel-by-pixel cosine similarity processing on the third-party unlabeled feature prototype and the first feature map to obtain the second prototype propagation prediction.

[0359] In this embodiment, an image segmentation apparatus is provided. Using this apparatus, the unlabeled sample obtains the prototype propagation prediction of the locally labeled sample by comparing the distance between the class prototype and pixel features. This enables the unlabeled sample to achieve supervised learning with labeled data, thereby making the features of the unlabeled sample closer to the features of the labeled sample. This reduces the impact of data heterogeneity from third-party unlabeled data and improves the accuracy of image segmentation.

[0360] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation apparatus 30 provided in this application, the processing module 302 is further configured to adjust the network parameters of the initial teacher model according to the network parameters of the initial student model.

[0361] This application provides an image segmentation apparatus. Using this apparatus, the network parameters of the student model are adjusted to adjust the network parameters of the teacher model. This ensures that the student model and the teacher model maintain a connection, preventing significant abrupt changes in the teacher model that could negatively impact the overall training performance of the image segmentation model.

[0362] Optionally, in the above Figure 9a Based on the corresponding embodiments, in another embodiment of the image segmentation device 30 provided in this application, the processing module 302 is specifically used to obtain the first network parameters in the previous iteration and the second network parameters in the current iteration of the initial student model;

[0363] The third network parameter is obtained by performing a moving average process based on the first network parameter and the second network parameter;

[0364] Update the third network parameter to the network parameter of the initial teacher model.

[0365] In this embodiment, an image segmentation apparatus is provided. Using this apparatus, the network parameters of the student model are processed by a moving average to obtain the network parameters of the teacher model. This ensures the connection between the student model and the teacher model while increasing the self-integration capability of the teacher model, thereby smoothing noise and preventing abrupt changes in prediction results.

[0366] The image segmentation apparatus and the training apparatus for the image segmentation model provided in this application can be servers. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations stored in the storage media 330 on the server 300.

[0367] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0368] The steps performed by the image segmentation device and the training device of the image segmentation model in the above embodiments can be based on this... Figure 10 The server structure shown.

[0369] The image segmentation apparatus and the training device for the image segmentation model provided in this application can be terminal devices. Please refer to [link / reference]. Figure 11 For ease of explanation, only the parts relevant to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0370] Figure 11 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 11 The smartphone includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will understand that... Figure 11 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0371] The following is combined with Figure 11 A detailed introduction to the various components of a smartphone:

[0372] RF circuit 410 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 480; additionally, it transmits uplink data to the base station. Typically, RF circuit 410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 410 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

[0373] The memory 420 can be used to store software programs and modules. The processor 480 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0374] The input unit 430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 480, and can receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0375] Display unit 440 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 440 may include a display panel 441, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 431 may cover display panel 441. When touch panel 431 detects a touch operation on or near it, it transmits the information to processor 480 to determine the type of touch event. Subsequently, processor 480 provides corresponding visual output on display panel 441 based on the type of touch event. Although in Figure 11 In this embodiment, the touch panel 431 and the display panel 441 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 431 and the display panel 441 can be integrated to realize the input and output functions of the smartphone.

[0376] Smartphones may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 441 according to the ambient light level, and the proximity sensor can turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes) and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that smartphones may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0377] Audio circuit 460, speaker 461, and microphone 462 provide an audio interface between the user and the smartphone. Audio circuit 460 converts received audio data into electrical signals and transmits them to speaker 461, where speaker 461 converts them into sound signals for output. On the other hand, microphone 462 converts collected sound signals into electrical signals, which are received by audio circuit 460, converted into audio data, and then processed by processor 480 before being transmitted via RF circuit 410 to, for example, another smartphone, or the audio data can be output to memory 420 for further processing.

[0378] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi modules (470), can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 11 WiFi module 470 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0379] The processor 480 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, thereby providing overall monitoring of the smartphone. Optionally, the processor 480 may include one or more processing units; optionally, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 480.

[0380] The smartphone also includes a power supply 490 (such as a battery) that powers various components. Optionally, the power supply can be logically connected to the processor 480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0381] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0382] The steps performed by the image segmentation device and the training device of the image segmentation model in the above embodiments can be based on this... Figure 11 The terminal device structure is shown.

[0383] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0384] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0385] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0386] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0387] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0388] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0389] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0390] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for an image segmentation model, characterized in that, include: Obtain a training sample set, an initial student model, and an initial teacher model. The training sample set includes locally labeled samples, locally unlabeled samples, and third-party unlabeled samples. The locally labeled samples also include real labels. The initial student model and the initial teacher model have the same network structure. Based on the initial student model, the first predicted segmentation image and the first feature map of the local labeled sample are obtained, and the supervised loss is obtained by processing the first predicted segmentation image and the real label. Based on the initial teacher model, the first pseudo-label, the first predicted probability map, and the second feature map of the local unlabeled samples are obtained. The first pseudo-label, the first predicted probability map, and the second feature map are subjected to confidence-weighted masking average pooling to obtain the local unlabeled feature prototype. Based on the initial teacher model, the second pseudo-label, the second predicted probability map, and the third feature map of the third-party unlabeled samples are obtained. The second pseudo-label, the second predicted probability map, and the third feature map are subjected to confidence-weighted masking average pooling to obtain the third-party unlabeled feature prototype. The first prototype propagation prediction is obtained by performing pixel-by-pixel cosine similarity processing between the local unlabeled feature prototype and the first feature map. The second prototype propagation prediction is obtained by performing pixel-by-pixel cosine similarity processing between the third-party unlabeled feature prototype and the first feature map. A first loss value is obtained based on the first prototype propagation prediction and the real label processing, and a second loss value is obtained based on the second prototype propagation prediction and the real label processing. The first loss value and the second loss value are used as the unlabeled loss. The image segmentation model is obtained by training the initial student model based on the supervised loss and the unlabeled loss.

2. The method according to claim 1, characterized in that, The method further includes: The second predicted segmentation image of the local unlabeled sample is obtained based on the initial student model, and the first pseudo-label of the local unlabeled sample is obtained based on the initial teacher model; A third loss value is obtained based on the second predicted segmentation image and the first pseudo-label, and the third loss value is included in the unlabeled loss.

3. The method according to claim 2, characterized in that, The step of obtaining the third loss value based on the second predicted segmented image and the first pseudo-label includes: Obtain the first prediction probability map of the second predicted segmentation image; A target pixel set is determined based on the first predicted probability map and the first threshold, and each pixel in the target pixel set is used to determine the third loss value; The probability value of each pixel in the target pixel set is processed with the first pseudo-label to obtain the third loss value.

4. The method according to claim 3, characterized in that, The step of processing the probability values ​​of each pixel in the target pixel set with the first pseudo-label to obtain the third loss value includes: The pixels in the target pixel set are divided into K regions of the same size, and the first pseudo-label is divided into K regions of the same size in a way that corresponds one-to-one with the target pixel set, where K is greater than or equal to 2; K loss values ​​are obtained for K regions in the target pixel set and K regions of the first pseudo-label, respectively. The third loss value is obtained by averaging the K loss values.

5. The method according to claim 2, characterized in that, The method further includes: Based on the initial student model, the local annotation feature prototype corresponding to the local annotation sample is obtained; A fourth loss value is obtained by performing categorical feature regularization processing based on the local labeled feature prototype, the local unlabeled feature prototype, and the third-party unlabeled feature prototype. The fourth loss value is included in the unlabeled loss.

6. The method according to claim 5, characterized in that, The step of performing categorical feature regularization based on the local labeled feature prototype, the local unlabeled feature prototype, and the third-party unlabeled feature prototype to obtain the fourth loss value includes: Obtain the first foreground feature prototype and the first background feature prototype of the local labeled feature prototype, the second foreground feature prototype and the second background feature prototype of the local unlabeled feature prototype, and the third foreground feature prototype and the third background feature prototype of the third-party unlabeled feature prototype; Obtain a first distance function between the third foreground feature prototype and the second foreground feature prototype and the first foreground feature prototype, and obtain a second distance function between the third background feature prototype and the second background feature prototype and the first background feature prototype; The fourth loss value is obtained by weighting and summing the first distance function and the second distance function.

7. The method according to claim 5, characterized in that, The method further includes: The local unlabeled samples and the third-party unlabeled samples are preprocessed to obtain the first perturbation sample and the second perturbation sample; The third predicted segmentation image of the first perturbation sample is obtained based on the initial student model, and the fourth predicted segmentation image of the second perturbation sample is obtained based on the initial teacher model. The distance between each pixel in the third predicted segmentation image and the fourth predicted segmentation image is determined to obtain a fifth loss value, which is included in the unlabeled loss.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Adjust the network parameters of the initial teacher model based on the network parameters of the initial student model.

9. The method according to claim 8, characterized in that, The step of adjusting the network parameters of the initial teacher model based on the network parameters of the initial student model includes: Obtain the first network parameters from the previous iteration and the second network parameters from the current iteration of the initial student model; The third network parameter is obtained by performing a moving average process based on the first network parameter and the second network parameter; The third network parameters are updated to the network parameters of the initial teacher model.

10. An image segmentation method, characterized in that, it should include: Obtain the image to be processed; The image segmentation model is used to obtain a segmented image of the image to be processed, wherein the image segmentation model is trained based on any one of claims 1 to 9 above; Output the segmented image.

11. A training device for an image segmentation model, characterized in that, include: The acquisition module is used to acquire a training sample set, an initial student model, and an initial teacher model. The training sample set includes locally labeled samples, locally unlabeled samples, and third-party unlabeled samples. The locally labeled samples also include real labels. The initial student model and the initial teacher model have the same network structure. The processing module is used to obtain a first predicted segmentation image and a first feature map of the locally labeled sample based on the initial student model, and to obtain a supervised loss based on the first predicted segmentation image and the ground truth label; to obtain a first pseudo label, a first predicted probability map and a second feature map of the locally unlabeled sample based on the initial teacher model; and to perform confidence-weighted masking average pooling on the first pseudo label, the first predicted probability map and the second feature map to obtain the local unlabeled feature prototype. Based on the initial teacher model, a second pseudo-label, a second predicted probability map, and a third feature map are obtained for the third-party unlabeled samples. Confidence-weighted masked average pooling is applied to the second pseudo-label, the second predicted probability map, and the third feature map to obtain third-party unlabeled feature prototypes. Pixel-wise cosine similarity processing is performed between the local unlabeled feature prototypes and the first feature map to obtain a first prototype propagation prediction. Pixel-wise cosine similarity processing is also performed between the third-party unlabeled feature prototypes and the first feature map to obtain a second prototype propagation prediction. A first loss value is obtained based on the first prototype propagation prediction and the real label, and a second loss value is obtained based on the second prototype propagation prediction and the real label. The first loss value and the second loss value are used as the unlabeled loss. The training module is used to train the initial student model based on the supervised loss and the unlabeled loss to obtain the image segmentation model.

12. An image segmentation apparatus, characterized in that, include: The acquisition module is used to acquire the image to be processed; The processing module is used to obtain a segmented image of the image to be processed based on an image segmentation model, wherein the image segmentation model is trained based on any one of claims 1 to 9 above; The output module is used to output the segmented image.

13. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the method of any one of claims 1 to 9 or claim 10 according to the instructions in the program code; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

14. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 9 or 10.

15. A computer program product, characterized in that, The computer program product includes computer instructions, which are executed by a processor of a computer device to cause the computer device to perform the method as claimed in any one of claims 1 to 9 or 10.

Images