A semi-supervised segmentation model construction and image analysis method, device and system

Abstract

The application relates to a semi-supervised segmentation model construction and image analysis method, system and equipment. The method comprises the following steps: acquiring an image to be processed; performing consistency learning on the image based on a constructed semi-supervised segmentation model trained, to obtain an image segmentation result; wherein the consistency learning is based on the consistency of the feature extraction capability of an encoder in the semi-supervised segmentation model trained on the input image and the cooperation of a plurality of auxiliary decoders and a main decoder in completing the prediction result training. The method aims to construct a semi-supervised segmentation model based on a consistency learning semantic segmentation network, realize image analysis, and explore the analysis capability and potential application value of the semi-supervised segmentation model in image data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and image analysis, and more particularly, to a semi-supervised segmentation model construction and image analysis method, device, system, computer readable storage medium and application thereof. BACKGROUND

[0002] With the combination and development of computer vision and computer technology, the method of using computer technology to assist in processing various graphic data and models has become one of the hotspots in recent years. In the field of vision, there are a large number of images that need to be recognized and processed. In the past, artificial methods were used, and experienced staff analyzed the images comprehensively, but the analysis speed was slow, and the evaluation standard and analysis accuracy varied from person to person. For example, electroencephalogram data can only obtain a small amount of diagnostic information by manual observation, or like laboratory work to count cells or calculate the area of each part of the tissue section, which is time-consuming and laborious. For medical image data of different characteristics, many different recognition algorithms have been developed, but further solutions are still needed based on the different needs of researchers.

[0003] At present, the resolution of the photographed image is as high as 6000x4000, and there are many experimental sample images in each group. The labeling process is tedious and time-consuming, and occupies a lot of time in scientific research work. Moreover, due to the constraints of the number of experimental samples and individual differences, the number of sample images that need to be analyzed is large, and the differences between the images of individuals in the group are large. It is still a challenge to effectively analyze experimental data from various reliable and accurate sources.

[0004] In recent years, with the development of the field of computer vision, related algorithms have been designed using computer technology to complete the same work more quickly and accurately. Based on the development of artificial intelligence technology in the field of image processing, it is possible to apply it to solve and simplify manual labeling and statistical work. SUMMARY

[0005] The purpose of the present application is to provide a semi-supervised segmentation model construction method, and a corresponding image analysis method, system, device, computer readable storage medium and application thereof, which aims to effectively solve the segmentation and classification prediction problem of images, save the manpower and time cost of data processing, improve the acquisition speed and segmentation statistical accuracy of experimental data, based on deep learning algorithm and image processing technology, a semi-supervised segmentation model construction method is provided, a new semi-supervised network intra-consistency constraint and network inter-consistency constraint combined semantic segmentation model is designed, which can more accurately predict the segmentation result of the image, and further improve the segmentation effect and segmentation speed, to provide more sufficient support and potential application value in image analysis and prediction.

[0006] According to a first aspect of the present application, an embodiment of the present application provides a semi-supervised segmentation model construction method, which comprises:

[0007] Step a: obtaining a weakly labeled data set, a pseudo-labeled data set and an unlabeled data set;

[0008] Step b: inputting images in the weakly labeled data set into a full convolution network for supervised training, and training a trained full convolution network model based on a supervised loss;

[0009] Step c: predicting weakly labeled semantic segmentation information of images in the pseudo-labeled data set based on the trained full convolution network model, and obtaining pseudo-labeled images;

[0010] Step d: processing the weakly labeled data set and the pseudo-labeled images using a pseudo-label generation algorithm to generate pseudo-labeled images with all target class labels;

[0011] Step e: supervising and training the segmentation network based on the generated pseudo-labeled images, optimizing the segmentation network by calculating a supervised loss value, and updating parameters of a first encoder, a first main decoder and a first auxiliary decoder in the segmentation network;

[0012] Step f: optimizing the segmentation network based on unlabeled images in the unlabeled data set by calculating an unsupervised loss value of an unsupervised consistency loss, and updating parameters of a second encoder, a second main decoder and a second auxiliary decoder in the segmentation network;

[0013] Step g: optimizing and calculating an unsupervised loss value by a cross-pseudo-supervised loss function based on the pseudo-labeled images and the unlabeled images, and then calculating a total loss value of model training, simultaneously updating parameters according to the total loss value, and outputting a trained semi-supervised segmentation model; wherein the total loss value is calculated as follows:

[0014]

[0015] Wherein, λ1 and λ2 are weights of the unsupervised consistency loss and the cross-pseudo-supervised loss, L s represents the supervised loss value, L c represents the unsupervised loss value calculated by the unsupervised consistency loss, L p represents the unsupervised loss value calculated by the cross-pseudo-supervised loss function.

[0016] Further, the cross-pseudo-supervised loss function is based on a pseudo-label cross-self-supervised consistency training structure of the semi-supervised segmentation model, one of the predicted segmentation images of two different initialized parameter networks is used as a pseudo-label to guide the training of the other branch network, to optimize and calculate the unsupervised loss value L p .

[0017] Further, preferably, the unsupervised loss value L p is calculated as follows:

[0018]

[0019] wherein, denotes all images in the training set, W and H are the height and width of the image X in the input network, H(·) denotes the cross-entropy function, and are the confidence information generated by the main decoder corresponding to the two branch networks a and b respectively, and are the one-hot encoding information of the corresponding main decoder.

[0020] Further, the construction of the pseudo-label cross self-supervised consistency training structure further includes supervision of a supervised loss value L s , and preferably, the supervised loss value L s is calculated as follows:

[0021]

[0022] wherein, H(·) denotes the cross-entropy function, y i denotes the pseudo-label corresponding to the pixel x i , and x i denotes the pixel with index i in the input image X, and denote the pseudo-label prediction images generated by the two branch networks a and b respectively, W and H are the height and width of the image X in the input network, and X and Y denote the obtained heart-stained slice images to be predicted, denotes a weakly labeled dataset, denotes a pseudo-labeled dataset.

[0023] Further, the construction of the semi-supervised segmentation model further includes adding a feature disturbance module between the encoder and the auxiliary decoder, the feature disturbance module adds different types of noise to the feature information in the encoded feature image generated by the encoder based on the input image and inputs the noise to the main decoder, and the prediction result of the main decoder is used to guide the prediction result learning of the auxiliary decoder, and an unsupervised loss value L c is calculated by an unsupervised consistency loss function to realize unsupervised consistency training.

[0024] Specifically, preferably, the unsupervised loss value L c is calculated as follows:

[0025]

[0026]

[0027]

[0028]

[0029] Where a and b are two branch networks in the semi-supervised segmentation model, x i For image The pixel at index i in the middle, This represents an unlabeled dataset. and These represent the predicted images generated by branches a and b, respectively. h represents the predicted image generated by the auxiliary decoder. a Represents encoders a and g a Indicates the main decoder a, h b Represents encoders b and g b Indicates the main decoder b, g k ∈{g1,......g 2K It contains 2K auxiliary decoders, ξ represents the feature perturbation module in the network, and MSE(·) represents the mean square error function.

[0030] Furthermore, optionally, the different types of noise are generated through various different functional perturbations, specifically:

[0031] (1) Set the feature information discard probability to 0.3, randomly discard the channel dimension of the feature tensor, and set the feature value of the discarded dimension to zero;

[0032] (2) Calculate the mean of the feature information channel dimension and normalize it to calculate the attention matrix. Sample a random threshold uniformly in the range of 0.7 to 0.9. Output 1 for points in the attention matrix that are greater than the threshold, and output 0 otherwise. Generate the corresponding mask matrix. Multiply the feature matrix and the mask matrix to obtain the final feature information map.

[0033] (3) Use uniform sampling within the range of 0.8 to 1.2 to generate a noise tensor with the same shape as the encoded feature image. Multiply the noise tensor with the encoded feature image to obtain the injected noise.

[0034] (4) Based on the encoded feature image, randomly erase and set some regions to zero, and randomly select a rectangular region of a fixed size to erase and set to zero.

[0035] Furthermore, in some embodiments, step a, obtaining the weakly labeled dataset, the pseudo-labeled dataset, and the unlabeled dataset, further includes preprocessing the obtained weakly labeled dataset, pseudo-labeled dataset, and unlabeled dataset. The preprocessing is implemented by any one or more of the following methods: flipping, rotating, scaling, cropping, mirroring, contrast adjustment, brightness adjustment, and grayscale conversion. The images in the weakly labeled dataset contain coarse classification labels for both the background and the foreground, and the images in the pseudo-labeled dataset contain fine classification labels for different target regions.

[0036] Furthermore, in some embodiments, the segmentation network includes an encoder, a main decoder, and an auxiliary decoder. The encoder generates a corresponding encoded feature image for the input image through an encoder structure. The main decoder decodes the feature information in the encoded feature image generated by the encoder into a semantic segmentation image corresponding to the input image. The auxiliary decoder cooperates with the main decoder during network training to complete the training of the unsupervised part.

[0037] Optionally, the encoder is constructed based on any one or more of the following network models: convolutional neural network, deep neural network, LSTM / GRU;

[0038] Preferably, the encoding process of the encoder is as follows: an encoder structure is constructed using a convolutional neural network as the backbone network; a feature map is obtained from the input image; the feature map is passed through a finite number of adaptive average pooling layers with different output scales in the pyramid pooling module; at the same time, feature maps with different scales containing local and global contextual feature information are obtained after adaptive average pooling; the feature maps with different scales are resampled to obtain the final encoded feature image with the same size as the feature map before the input pyramid pooling module.

[0039] Preferably, the main decoder consists of a 1x1 convolutional layer and a finite number of transposed convolutions.

[0040] According to a second aspect of this application, one embodiment of this application provides an image analysis method, which includes:

[0041] Obtain the image to be processed;

[0042] Image segmentation results are obtained by performing consistency learning on the constructed and trained semi-supervised segmentation model. The consistency learning is based on the feature extraction capability of the encoder of the trained semi-supervised segmentation model and the cooperation of several auxiliary decoders with the main decoder to achieve consistency of the prediction results.

[0043] Furthermore, the constructed and trained semi-supervised segmentation model is implemented based on the semi-supervised segmentation model construction method provided in this application.

[0044] According to a third aspect of this application, one embodiment of this application provides an image analysis system, which includes:

[0045] The acquisition module is used to acquire the image to be processed;

[0046] The prediction module performs image consistency learning based on the constructed and trained semi-supervised segmentation model to obtain the image segmentation result.

[0047] Furthermore, the consistency learning is based on the feature extraction capability of the encoder in the trained semi-supervised segmentation model for the input image and the cooperation of several auxiliary decoders with the main decoder, thereby achieving consistency in the prediction results.

[0048] Furthermore, the trained semi-supervised segmentation model is implemented based on the semi-supervised segmentation model construction method provided in this application.

[0049] In some embodiments, the encoder is constructed based on one or more of the following network models: convolutional neural network, deep neural network, LSTM / GRU;

[0050] Preferably, the main decoder consists of a 1x1 convolutional layer and a finite number of transposed convolutions.

[0051] According to a fourth aspect of this application, one embodiment of this application provides an image analysis device, which includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, to implement the above-described image analysis method or the semi-supervised segmentation model construction method.

[0052] According to a fifth aspect of this application, one embodiment of this application provides a computer-readable storage medium storing a computer program for image analysis based on a constructed semi-supervised segmentation model. When the computer program is executed by a processor, it implements the above-described image analysis method or semi-supervised segmentation model construction method.

[0053] According to a sixth aspect of this application, an embodiment of this application provides its related applications, mainly including:

[0054] Application of the aforementioned devices or systems in image target segmentation and classification tasks;

[0055] The aforementioned devices or systems are used for intelligent prediction and image analysis; optionally, the application calculates the area of ​​different target regions to perform corresponding effect prediction or intelligent analysis.

[0056] The above-mentioned equipment or system is used in assisting disease diagnosis, prognosis, and efficacy prediction. Optionally, the disease diagnosis, prognosis, and efficacy prediction include consistency learning of the pathological slide image to be predicted based on the semi-supervised segmentation model provided in this application to obtain relevant indicators, which have a positive impact and promoting effect on prognosis and efficacy prediction.

[0057] This invention presents a semi-supervised segmentation model based on consistent semantic segmentation. Utilizing deep learning, semantic segmentation, and pseudo-label generation technologies, the model is constructed and trained using a large amount of unlabeled data and a small amount of weakly labeled data to segment target regions in the image to be processed, thereby obtaining prediction results. This significantly reduces the annotation work required by researchers, overcomes the limitations of manual segmentation and prediction, and assists in the automatic segmentation of images and the intelligent prediction of the entire sample image. It is highly innovative and will positively promote the analysis and research of image data segmentation and classification.

[0058] Advantages of this application:

[0059] 1. This application innovatively discloses an effective semi-supervised segmentation model based on a pseudo-label generation method and semi-supervised consistency training. By combining intra-network consistency constraints and inter-network consistency constraints, it can automatically segment unlabeled image data, greatly reducing the annotation work of researchers, more accurately predicting the segmentation results of images, and objectively improving the accuracy and depth of data analysis.

[0060] 2. This application innovatively constructs a semi-supervised segmentation model based on an encoder, a master decoder, an auxiliary decoder, and a feature perturbation module. The feature perturbation module in its network structure is located between the encoder and the auxiliary decoder. It perturbs the features based on the encoded feature image generated by the encoder. By adding different types of noise to the feature information generated by the encoded feature image, it inputs it into the master decoder. The prediction results of the master decoder guide the learning and automatic analysis of the prediction results of the auxiliary decoder, thereby achieving unsupervised consistent training with significant time efficiency.

[0061] 3. This application creatively discloses a method for generating pseudo-labels that can effectively generate reliable pseudo-labels for training. By combining encoder, decoder and feature perturbation module, semi-supervised training can be performed more effectively. The use of cross pseudo-supervision and consistency training loss helps the model converge faster and perform better, making this application more accurately applicable to segmentation and classification tasks related to image data analysis. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a schematic diagram illustrating an image analysis scenario based on a semi-supervised segmentation model provided in an embodiment of the present invention.

[0064] Figure 2 This is a flowchart of an image analysis method based on a semi-supervised segmentation model provided in an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of a method for constructing a semi-supervised segmentation model provided in an embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram of the construction structure of a semi-supervised segmentation model provided in an embodiment of the present invention;

[0067] Figure 5 This is a graph showing the statistical results of the loss values ​​of the semi-supervised segmentation model provided in this embodiment of the invention.

[0068] Figure 6 This is a schematic diagram of an image analysis device based on consistent semantic segmentation provided in an embodiment of the present invention. Detailed Implementation

[0069] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0070] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be performed in the order they appear herein, or may be performed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel.

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] This application provides a semi-supervised segmentation model construction method, image analysis method, system, computer device, and computer-readable storage medium. The disease risk prediction model training device can be integrated into the computer device, which can be a terminal or server. The terminal can be a smartphone, tablet, laptop, personal computer, or other similar device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0073] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an image analysis scenario based on a semi-supervised segmentation model provided in an embodiment of the present invention. Specifically, as shown in the diagram... Figure 1 The computer device shown is a terminal device. This terminal device can acquire the image to be processed, and then perform image consistency learning based on a pre-trained semi-supervised segmentation model to obtain the image segmentation result. The consistency learning is based on the encoder's feature extraction capability of the input image in the pre-trained semi-supervised segmentation model and the consistency of the prediction results achieved by several auxiliary decoders working in conjunction with the main decoder.

[0074] Figure 2 This is a flowchart of an image analysis method based on a semi-supervised segmentation model provided in an embodiment of the present invention. Specifically, the image analysis method includes the following steps:

[0075] S101: Obtain the image to be processed.

[0076] In one embodiment, the acquired image to be processed is primarily used for segmentation or classification tasks.

[0077] S102: Based on the constructed and trained semi-supervised segmentation model, perform consistency learning on the image to obtain the image segmentation result.

[0078] Specifically, a method and process for constructing a semi-supervised segmentation model, such as... Figure 3 As shown:

[0079] S301: Obtain weakly labeled datasets, pseudo-labeled datasets, and unlabeled datasets.

[0080] In one embodiment, the collected images are divided into three sets: a weakly labeled dataset. Pseudo-labeled datasets and unlabeled datasets Of the data, 5% were labeled, 20% were pseudo-labeled, and 75% were unlabeled.

[0081] Furthermore, preliminary coarse annotations are performed on the images in the weakly labeled dataset. The slice edges in the images are divided and the images are labeled as two parts based on the edge information: the background and the foreground containing the slice images. The background is labeled with category number 0 and the foreground is labeled with category number 1. After the above annotations, the annotation information of the weakly labeled dataset is obtained.

[0082] In a more specific embodiment, acquiring a weakly labeled dataset, a pseudo-labeled dataset, and an unlabeled dataset further includes preprocessing the acquired weakly labeled dataset, pseudo-labeled dataset, and unlabeled dataset. The preprocessing is implemented using any one or more of the following methods: flipping, rotating, scaling, cropping, mirroring, contrast adjustment, brightness adjustment, and grayscale conversion. The images in the weakly labeled dataset contain coarse classification labels for both background and foreground, while the images in the pseudo-labeled dataset contain fine classification labels for different target regions.

[0083] S302: Input the images from the weakly labeled dataset into the fully convolutional network for supervised training, and obtain the trained fully convolutional network model based on the supervised loss.

[0084] Specifically, a fully convolutional network is trained using images from a weakly labeled dataset. The FCN8s model is then used, and the model training function is called to obtain the predicted images, as shown in the following formula:

[0085]

[0086] Where, x w Represents a weakly labeled dataset Chinese image, The image representing the prediction of a fully convolutional network;

[0087] Furthermore, based on the predicted images generated by the network and the weakly labeled masks, the network is trained using the dice loss function and the cross-entropy loss function.

[0088] Furthermore, the model parameters were updated using a stochastic gradient descent optimizer with a learning rate of 0.001, a weight decay rate of 0.0001, a momentum of 0.9, and a batch size of 16. After 50 training epochs, a trained fully convolutional network model was obtained.

[0089] In one embodiment, images from a weakly labeled dataset are input into a fully convolutional network for supervised training. The weights of the fully convolutional network are updated by calculating the supervised loss value and gradient descent, resulting in a trained fully convolutional network model. This trained fully convolutional neural network model can generate weakly labeled images from unlabeled images in an unlabeled dataset.

[0090] S303: Based on the trained fully convolutional network model, predict the weakly labeled semantic segmentation information of images in the pseudo-labeled dataset to obtain pseudo-labeled images.

[0091] In one embodiment, a trained fully convolutional network model is used, the model's eval function is called, and a pseudo-labeled dataset is used. Image data is used for prediction. The predicted labels of images in the pseudo-labeled dataset by the fully convolutional network are saved as pseudo-labeled images with weak labeling information. The prediction formula is as follows:

[0092]

[0093] Where, x p Representative pseudo-labeled dataset Chinese image, The image represents the prediction of a fully convolutional network.

[0094] S304: Use a pseudo-labeling generation algorithm to process the weakly labeled dataset and pseudo-labeled images to generate pseudo-labeled images with labels for all target categories.

[0095] In one embodiment, a pseudo-label generation algorithm was used to generate a weakly labeled dataset and a pseudo-labeled dataset with pseudo-labels. The pseudo-label generation algorithm can produce pseudo-labels with the same effect as manual labeling with only a small amount of manual labeling and sampling.

[0096] S305: Supervised training of the segmentation network is performed based on the generated pseudo-label images. The segmentation network is optimized by calculating the supervised loss value, and the parameters of the first encoder, the first master decoder and the first auxiliary decoder in the segmentation network are updated.

[0097] In one embodiment, the supervised loss value L of the two branches a and b in the segmented network is... s The calculation is as follows:

[0098]

[0099] Where H(·) represents the cross-entropy function, y i Represents pixel x i The corresponding pseudo-label, x i This represents the pixel with index number i in the input image X. and Let W and H represent the pseudo-annotated predicted images generated by branches a and b, respectively. W and H are the height and width of image X in the input network, and X and Y represent the obtained stained slice images of the heart to be predicted. This represents a weakly labeled dataset. This represents a pseudo-labeled dataset.

[0100] S306: Based on unlabeled images in an unlabeled dataset, optimize the segmentation network using the unsupervised loss calculated by the unsupervised consistency loss, and update the parameters of the second encoder, second master decoder, and second auxiliary decoder in the segmentation network.

[0101] In one embodiment, the unsupervised loss value is calculated using an unsupervised consistency loss function to optimize the segmentation network. This updates the parameters of the encoder, main decoder, and auxiliary decoder within the segmentation network, thereby achieving unsupervised consistency training. The unsupervised loss value L for branches a and b in the segmentation network is... c The calculation is as follows:

[0102]

[0103]

[0104]

[0105]

[0106] Where a and b are two branch networks in the semi-supervised segmentation model, x i For image The pixel at index i in the middle, This represents an unlabeled dataset. and These represent the predicted images generated by branches a and b, respectively. h represents the predicted image generated by the auxiliary decoder. a Indicates encoder A, g a Indicates the main decoder A, h b Indicates encoder B, g b Indicates the main decoder B, g k ∈{g1,......g 2K It contains 2K auxiliary decoders, ξ represents the feature perturbation module in the network, and MSE(·) represents the mean square error function.

[0107] In one specific embodiment, the consistency loss function uses the predicted image generated by the main decoder as a pseudo-label to guide the training of the auxiliary encoder. Unsupervised consistency training within the network is jointly completed by the main decoder and the auxiliary decoders. The input image is transformed by the encoder to obtain a feature map, which is then passed through the main decoder and multiple auxiliary decoders containing feature perturbation modules. The feature map is processed by the main decoder to generate the corresponding predicted segmentation image, while the portion processed by the auxiliary decoders generates a predicted image with noisy feature information. The consistency loss function guides the auxiliary decoder's prediction based on the prediction results of the main decoder, strengthening the consistency of predicting the same parts, reducing the differences in predicting different parts, and enhancing the encoder's ability and stability against noise in the features. During training, the main decoder does not participate in training or parameter updates; the calculated gradient is backpropagated and used for parameter updates through the auxiliary decoders and encoder.

[0108] S307: Based on pseudo-labeled and unlabeled images, the unsupervised loss value is optimized and calculated through a cross-pseudo-supervised loss function. The total loss value of the model training is calculated, and the parameters are iteratively updated according to the total loss value to output the trained semi-supervised segmentation model.

[0109] Furthermore, the cross-pseudo-supervised loss function constructs a pseudo-label cross-self-supervised consistency training structure for the model based on all training data. It optimizes and computes the unsupervised loss value L by using one of the predicted segmentation images from two networks with different initialization parameters as pseudo-labels to guide the training of the other branch network. p ;

[0110] Furthermore, the unsupervised loss value L is calculated based on the cross-pseudo-supervised loss function. p ,as follows:

[0111]

[0112] in, Let W represent all images in the training set, and H be the height and width of the image X in the input network. H(·) represents the cross-entropy function. and These are the confidence information generated by the main decoders corresponding to the two branch networks, a and b, respectively. and This is the one-hot encoded information of the corresponding master decoder.

[0113] Furthermore, the total loss value is calculated as follows:

[0114]

[0115] Where λ1 and λ2 are the weights of the unsupervised consistency loss and the cross-spurious supervision loss, respectively, L sL represents the supervised loss value. c L represents the unsupervised loss value calculated using the unsupervised consistency loss. p This represents the unsupervised loss value calculated using the cross-pseudo-supervised loss function.

[0116] Furthermore, in one embodiment, the construction of the semi-supervised segmentation model also includes adding a feature perturbation module between the encoder and the auxiliary decoder, such as... Figure 4 As shown, the feature perturbation module adds different types of noise to the feature information in the encoded feature image generated by the encoder based on the input image and inputs it into the main decoder. The prediction results of the main decoder guide the learning of the prediction results of the auxiliary decoder.

[0117] Figure 4 This is a schematic diagram illustrating the construction structure of a semi-supervised segmentation model provided in an embodiment of the present invention. For example... Figure 4 As shown, the constructed semi-supervised segmentation model mainly includes an encoder, a decoder, and a feature perturbation module.

[0118] In one embodiment, the encoder is constructed based on one or more of the following network models: convolutional neural network, deep neural network, LSTM / GRU.

[0119] A convolutional neural network (CNN or ConvNet) is a type of neural network that uses convolutional layers. Based on the mathematical operations of convolution, it is often used in image recognition.

[0120] Deep Neural Networks (DNNs) have demonstrated state-of-the-art accuracy in many artificial intelligence tasks, but they also suffer from high computational complexity.

[0121] LSTM / GRU is a type of gated recurrent neural network. LSTM stands for Long Short-Term Memory Network and GRU stands for Gated Recurrent Unit. It is an adjustment to the structure of a simple recurrent neural network by adding a gating mechanism to control the transmission of information in the neural network.

[0122] Furthermore, the encoder encodes contextual information, rich spatial information, and detailed information, effectively extracting global information and semantic features at different levels from the input image, thus preventing the loss of spatial information to some extent. For each input image, the encoder structure generates a corresponding feature image.

[0123] In one specific embodiment, the encoding process of the encoder is as follows: a convolutional neural network is used as the backbone network to construct the encoder structure, the input image is used to obtain a feature map, the feature map is passed through four adaptive average pooling layers with different output scales in the pyramid pooling module to obtain local and global contextual feature information after adaptive average pooling, the feature maps at different scales are resampled to obtain the final encoded feature image with the same size as the feature map before the input pyramid pooling module.

[0124] In a more specific embodiment, the encoder's workflow is as follows: the input image is transformed into a feature map by a convolutional neural network using a ResNet50 pre-trained on ImageNet as the backbone. This feature map is then passed through four adaptive average pooling layers at different output scales in the pyramid pooling module, simultaneously obtaining local and global contextual feature information after adaptive average pooling. These feature maps at different scales are resampled to the same size as the feature map before input to the pyramid pooling module. The two feature maps are then merged to obtain the final encoded feature image. The encoder's multi-scale adaptive average pooling layers demonstrate good recognition and segmentation accuracy for stained regions in cardiac chromatograph images, particularly addressing the issues of small stained regions and blurred boundaries. This provides a foundation of abundant local and global contextual feature information for the subsequent decoder operation.

[0125] Furthermore, in one embodiment, the decoder in the semi-supervised segmentation model construction network consists of a 1x1 convolutional layer and four transposed convolutions, and the feature information generated by the upper-level module is processed by the decoder to produce the final semantic segmentation image.

[0126] In one specific embodiment, the detailed workflow of the decoder is as follows: After the upper-layer feature information enters the decoder, it is first processed by a 1x1 convolutional layer to transform the number of channels of the feature tensor, and then passed through 4 layers of transposed convolutions. Each time the feature map passes through a transposed convolution layer, its size is upsampled to twice its original size, until the feature map is restored to the same dimensions as the input image. Each encoder and its corresponding master decoder constitute a semantic segmentation main network. In addition to the master decoder, the semantic segmentation main network also includes several auxiliary decoders. The role of the master decoder is to decode the feature information generated by the encoder into a semantic segmentation image corresponding to the input image, while the role of the auxiliary decoders is to cooperate with the master decoder during network training to complete the unsupervised training part. The structure of the auxiliary decoders can fully utilize unlabeled data, enhance the feature extraction capability of the encoder, and improve the consistency between the decoders.

[0127] Furthermore, in one embodiment, during the construction of the semi-supervised segmentation model, a feature perturbation module is designed between the encoder and the auxiliary decoder. The feature information generated by the encoder passes through the feature perturbation module before entering the auxiliary decoder. The feature perturbation module adds different types of perturbations to the feature information generated by the encoder, adding small amounts of noise to the feature information. This method of adding noise to the feature information enables unsupervised consistency training of the segmentation network.

[0128] Specifically, adding perturbations to feature information is an important part of consistency training. The model's feature perturbation module contains various functions for generating perturbation noise, which are categorized as follows:

[0129] (1) Set the feature information discard probability to 0.3, randomly discard the channel dimensions of the feature tensor, and set the feature values ​​of the discarded dimensions to zero. This method increases the modeling ability of other channels and alleviates the co-adaptation problem between channels.

[0130] (2) Calculate the mean of the feature information channel dimension and normalize it to calculate the attention matrix. A random threshold is uniformly sampled in the range of 0.7 to 0.9. Points in the attention matrix that are greater than the threshold are output as 1, otherwise as 0. The corresponding mask matrix is ​​generated. The feature matrix is ​​multiplied by the mask matrix to generate the final feature information map. In this way, a part of the most active area in the feature map can be masked.

[0131] (3) Use uniform sampling within the range of 0.8 to 1.2 to generate a noise tensor with the same shape as the feature map, and then multiply the noise with the feature map. The injected noise is proportional to the feature value in each generated feature map.

[0132] (4) Randomly erase and zero out some regions on the feature map, and randomly select a rectangular region of a fixed size to erase and zero out. By randomly erasing, the network can make better use of the global information of the image, rather than relying on only a small number of specific visual features.

[0133] In one embodiment, during model training, the batch size is set to 4, images are randomly cropped to 512x512 size, and stochastic gradient descent is used to optimize training parameters. The learning rate is set to 0.01, the learning rate decay is 0.1, the weight decay is 0.0001, the momentum is 0.9, and the parameters in the loss function are λ1 = 30 and λ2 = 2. λ1 and λ2 are used as weights in the unsupervised loss function in the experiment. To avoid the noise generated by inaccurate predictions in the initial unsupervised training from affecting the network training effect as pseudo-labels, the weights λ1 and λ2 are gradually increased from 0 to the maximum value within 10% of the total number of epochs according to the sigmoid function curve at the beginning of the experiment. Each experiment is executed for 200 epochs.

[0134] Furthermore, the training of the model includes the following optimization methods: data preprocessing based on the segmentation task, data augmentation, random weight decay, learning rate, adversarial training, regularization, AdamW optimizer, and self-knowledge distillation.

[0135] In a more specific embodiment, the obtained segmentation and classification results are used as the verification criteria for model training or model validation. To measure the final results of this invention, 100 images from the dataset are selected as the training set and 36 images are selected as the validation set. The images in the dataset are cropped to balance the proportions of each category, and data preprocessing methods such as flipping, rotating, cropping, contrast adjustment, and brightness adjustment are used before training.

[0136] Furthermore, semi-supervised segmentation models can also be implemented based on any one or more of the following frameworks: FA-SSD, FF-SSD, R-CNN, R-FCN, and RefineDet.

[0137] FA-SSD is a network framework built on the SSD (single shot multibox detector) algorithm. It combines feature fusion and attention modules, connects target features and context features by superimposing features, and places a one-stage attention module on the target features to achieve fusion with higher-level feature maps of the target feature layer.

[0138] FF-SSD (feature fusion-based SSD) is an object detection algorithm that can achieve high detection accuracy for various targets in complex backgrounds. It introduces a modulation factor into the original loss function and enhances the semantic information of the low-level feature maps by constructing a feature pyramid and fusing multiple feature maps, thereby improving the accuracy of small target detection and thus improving the overall detection accuracy.

[0139] R-CNN, short for Region-CNN, uses selective search to generate region proposals as its strategy in the region selection stage. Currently, the more mature and widely used R-CNN models are Faster R-CNN and Mask R-CNN.

[0140] R-FCN, short for Region-based Fully Convolutional Net, is a region-based fully convolutional network that uses a location-sensitive score map to handle both location variance and location invariance. All computations in the network are shared.

[0141] RefineDet consists of two interconnected modules: the Anchor RefinementModule and the Object DetectionModule. It achieves better accuracy than two-stage methods while maintaining comparable efficiency to one-stage methods.

[0142] Figure 5 This is a statistical result of the loss value of the semi-supervised segmentation model provided in the embodiment of the present invention. In the experiment, during the first 10% of the training cycle, the weights λ1 and λ2 gradually increase from 0 according to the sigmoid curve until they reach the final fixed values ​​λ1 = 30 and λ2 = 2. The value of parameter λ1 is positively correlated with the number of auxiliary encoders in the model. Figure 5 A represents the loss function curve of the total loss value, from Figure 5 Looking at all the loss function curves, there are supervised loss values ​​(such as...) Figure 5 As shown in B, the overall trend gradually decreases, and the unsupervised consistency loss is adjusted during training (e.g., ...). Figure 5 The weights (as shown in C) continuously increase, gradually converging after the 20th cycle. Before the 20th cycle, the supervised loss value accounts for a relatively larger proportion, and the subnetworks are trained using supervised images with the same labels. Therefore, the consistency between networks increases during this period, which is reflected in the loss function as the supervised loss value and the cross-pseudo-supervised loss (as shown in C). Figure 5 As shown in D, the weight of the unsupervised consistency loss decreases. After the 20th cycle, the weight of the unsupervised consistency loss recovers to its maximum value, but the loss value has not yet fully converged. Therefore, the cross-supervised pseudo-supervised loss function will first increase slightly and then decrease after the 20th cycle.

[0143] In one specific embodiment, random oversampling is performed to account for data type imbalance. The cross-validation process only performs oversampling during the training fold. Features for modeling are selected by F-score sorting, and the optimal number of features is determined by maximizing the macroscopic area under the receiver operating characteristic curve of the cross-validation results.

[0144] In a more specific embodiment, in the clinicopathological analysis, the χ² test was used to compare qualitative variables, and the Kruskal-Wallis test was used to compare continuous data. The model was constructed and evaluated using the Scikit-learn software package and R software. Quantitative data were normally distributed and expressed as mean standard deviation, while qualitative data were described as frequencies (percentages).

[0145] Applying the above method to image analysis and prediction is feasible, demonstrating that a semi-supervised segmentation model is constructed based on a deep learning network model, encoder, decoder, and feature perturbation module. Automatic labeling is achieved by using pseudo-label generation and other techniques for training pseudo-labels. At the same time, cross-pseudo-supervision and consistency training loss are used to help the model converge faster and obtain segmentation and classification prediction results, and provide more accurate semantic segmentation information for evaluation. The performance of the model prediction can better reflect the specific segmentation and classification prediction effectiveness, making this application more advantageous in the analysis of segmentation and classification prediction related to image data.

[0146] An image analysis system provided in this embodiment of the invention includes:

[0147] The acquisition module is used to acquire the image to be processed;

[0148] The prediction module performs image consistency learning based on the constructed and trained semi-supervised segmentation model to obtain the image segmentation result.

[0149] Figure 6 This invention provides an image analysis device based on a semi-supervised segmentation model, comprising: a memory and a processor; the device may also include: an input device and an output device.

[0150] Memory, processor, input devices, and output devices can be connected via a bus or other means. Figure 6 The example shown is a bus connection; the memory is used to store program instructions; the processor is used to call the program instructions. When the program instructions are executed, they are used to execute the above image analysis method or to implement the above semi-supervised segmentation model construction method.

[0151] The present invention provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the above-mentioned image analysis method or semi-supervised segmentation model construction method.

[0152] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for another example, the division of modules is merely a logical functional division, and there may be other division methods in actual implementation; furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0154] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Specifically, some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0155] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The aforementioned integrated modules can be implemented in hardware or as software functional modules.

[0156] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0157] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program that executes instructions for the relevant hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0158] The computer device provided by this invention has been described in detail above. For those skilled in the art, based on the ideas of the embodiments of this invention, there will be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as limiting the invention.

Claims

1. A method for constructing a semi-supervised segmentation model, the construction process of the model comprising: step a: obtaining a weakly-labeled dataset, a pseudo-labeled dataset and an unlabeled dataset; step b: inputting images in the weakly-labeled dataset into a fully convolutional network for supervised training, and training a trained fully convolutional network model based on a supervised loss; step c: predicting weakly-labeled semantic segmentation information of images in the pseudo-labeled dataset based on the trained fully convolutional network model, and obtaining pseudo-labeled images; step d: processing the weakly-labeled dataset and the pseudo-labeled images using a pseudo-label generation algorithm to generate pseudo-labeled images with all target class labels; step e: supervising and training a segmentation network based on the generated pseudo-labeled images, optimizing the segmentation network by calculating a supervised loss value, and updating parameters of a first encoder, a first main decoder and a first auxiliary decoder in the segmentation network; step f: optimizing the segmentation network by calculating an unsupervised loss value based on an unsupervised consistency loss of unlabeled images in the unlabeled dataset, and updating parameters of a second encoder, a second main decoder and a second auxiliary decoder in the segmentation network; step g: optimizing and calculating an unsupervised loss value by a cross-pseudo-supervision loss function based on the pseudo-labeled images and the unlabeled images, then calculating a total loss value of model training, and iteratively updating parameters according to the total loss value, and outputting a trained semi-supervised segmentation model; wherein the total loss value is calculated as follows: wherein, the different types of noise generate perturbation noise through a plurality of different functions, and are specifically divided into: (1) setting a feature information discard probability to 0.3, randomly discarding a channel dimension of a feature tensor, and setting a discarded dimension feature value to zero; (2) calculating a feature information channel dimension mean and normalizing to calculate an attention matrix, uniformly sampling a random threshold value in a range of 0.7-0.9, outputting 1 for a point in the attention matrix greater than the threshold value, otherwise outputting 0, generating a corresponding mask matrix, and multiplying the feature matrix and the mask matrix to generate a final feature information graph; (3) using uniform sampling to generate a noise tensor with the same shape as the encoded feature image in a range of 0.8-1.2, multiplying the noise tensor and the encoded feature image, and keeping the injected noise proportional to the feature value in each generated encoded feature image; (4) randomly erasing and setting to zero in a part of the region based on the encoded feature image, and randomly selecting a fixed size rectangular region for erasing and setting to zero. The segmentation network comprises an encoder, a main decoder and an auxiliary decoder, the encoder generates a corresponding encoded feature image of an input image through an encoder structure, the main decoder decodes feature information in the encoded feature image generated by the encoder into a semantic segmentation image corresponding to the input image, and the auxiliary decoder cooperates with the main decoder to complete unsupervised training in the network training process. The encoder is constructed based on any one or several of the following network models: a convolutional neural network, a deep neural network, an LSTM / GRU. ​ ​ ​ ​ ​ ； wherein, and are the weights of the unsupervised consistency loss and the cross pseudo-supervised loss, respectively, L s represents a supervised loss value, L c represents an unsupervised loss value calculated by the unsupervised consistency loss, L p represents an unsupervised loss value calculated by the cross pseudo-supervised loss function; the cross pseudo-supervised loss function is based on the construction of a pseudo-label cross self-supervised consistency training structure of a semi-supervised segmentation model for all training data, one of the predicted segmentation images of two different initialization parameter networks is used as a pseudo-label to guide the training of the other branch network, and the unsupervised loss value L p is optimized and calculated; the calculation of the unsupervised loss value L p is as follows: ； wherein, represents all images in the training set, W and H are the height and width of the image X in the input network, represents a cross-entropy function, and are the confidence information generated by the main decoder corresponding to the two branch networks a and b respectively, and is the one-hot encoding information of the corresponding main decoder; the construction of the pseudo-label cross self-supervised consistency training structure further comprises supervision of a supervised loss value L s , and calculation of the supervised loss value L s , as follows: ； wherein, denotes a cross-entropy function, denotes a pixel corresponding to the pseudo-label, denotes a pixel with index i in the input image X, ,i and ,i denote the pseudo-label prediction images generated by the two branch networks a and b, W and H are the height and width of the image X in the input network, X and Y represent the obtained cardiac staining slice images to be predicted, denotes a weakly labeled dataset, denotes a pseudo-labeled dataset; The construction of the semi-supervised segmentation model further comprises adding a feature disturbance module between the encoder and the auxiliary decoder, the feature disturbance module adds different types of noise input to the main decoder based on the feature information in the encoded feature image generated by the encoder based on the input image, and guides the auxiliary decoder to learn the prediction result according to the prediction result of the main decoder, and calculates the unsupervised loss value L through the unsupervised consistency loss function c The optimization segmentation network updates the parameters of the second encoder, the second main decoder and the second auxiliary decoder in the segmentation network; and the calculation of the unsupervised loss value L c is as follows: where a, b are two branch networks in the semi-supervised segmentation model, is an image is a pixel at index i in the image, denotes an unlabeled dataset, and denote the predicted images generated by the two branch networks a, b respectively, denotes the predicted image generated by the auxiliary decoder h a、 h b denote the encoders corresponding to the two branch networks a, b respectively, a、 g b denote the main decoders corresponding to the two branch networks a, b respectively, contains 2K auxiliary decoders, represents a feature perturbation module in the network, denotes a mean square error function.

2. The method of claim 1, wherein the semi-supervised segmentation model is constructed by using a semi-supervised learning algorithm. ​ ​ ​ ​ ​ 3. The method of claim 1, wherein the semi-supervised segmentation model is constructed by using a semi-supervised learning algorithm. ​ 4. The method of claim 3, wherein the semi-supervised segmentation model is constructed by using a semi-supervised learning algorithm. ​ 5. The method of claim 3, wherein the semi-supervised segmentation model is constructed by using a semi-supervised learning algorithm. The encoding process of the encoder is as follows: taking an ImageNet pre-trained Resnet50 as a backbone network to construct an encoder structure, inputting an image to obtain a feature map, passing the feature map through four different output scale adaptive average pooling layers in a pyramid pooling module to obtain local and global context feature information after adaptive average pooling, and resampling different scale feature maps to obtain a final encoding feature image with the same size as the feature map before inputting the pyramid pooling module.

6. The method of claim 3, wherein the semi-supervised segmentation model is constructed by using a semi-supervised learning algorithm. The main decoder is composed of a 1x1 convolution layer and a limited number of transpose convolutions.

7. The method of claim 1, wherein the semi-supervised segmentation model is constructed by using a semi-supervised learning algorithm. The weakly labeled dataset, the pseudo-labeled dataset and the unlabeled dataset are also preprocessed, and the preprocessing is realized by any one or several of the following methods: flipping, rotating, scaling, cropping, mirroring, contrast adjustment, brightness adjustment and greying; wherein the images in the weakly labeled dataset contain coarse classification labels of background and foreground, and the images in the pseudo-labeled dataset contain fine classification labels of different target region labels.

8. An image analysis method, comprising: obtaining an image to be processed; performing consistency learning on the image based on the trained semi-supervised segmentation model constructed in any one of claims 1-7 to obtain an image segmentation result; wherein the consistency learning is based on the feature extraction capability of the encoder in the trained semi-supervised segmentation model and the consistency of the prediction results completed by the cooperation of the plurality of auxiliary decoders and the main decoder.

9. An image analysis system, comprising: an acquisition module for acquiring an image to be processed; a prediction module for performing consistency learning on the image based on the trained semi-supervised segmentation model constructed in any one of claims 1-7 to obtain an image segmentation result.

10. An image analysis device, comprising: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, the image analysis method of claim 8 or the semi-supervised segmentation model construction method of any one of claims 1-7 is implemented.

11. A computer-readable storage medium having stored thereon a computer program for performing image analysis, characterized in that The computer program is executed by the processor to implement the image analysis method of claim 8 or the semi-supervised segmentation model construction method of any one of claims 1-7.

Images