An oral potential malignant disease image segmentation method, electronic equipment and medium

Abstract

The application discloses an oral potential malignant disease image segmentation method, an electronic device and a medium, which comprises the following steps: first, an oral potential malignant disease image dataset containing unannotated and annotated images is obtained. The unannotated images are divided into image blocks, and the first mask is randomly added to input a teacher model to obtain a reconstructed image. The loss of the reconstructed image and the original image is calculated to construct a loss map, and the region corresponding to the first K average loss is selected as a second mask. The random mask and the second mask are superimposed to form a third mask, which is added to the unannotated image to input a student model, and the teacher and student models are collaboratively trained. The annotated image is input into the student model encoder to extract features, and then input into a semantic segmentation model for fine-tuning. Finally, the image to be processed is input into the fine-tuned semantic segmentation model to obtain a segmentation result. The method can effectively improve the image segmentation accuracy and provide strong support for the diagnosis of oral potential malignant diseases.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image self-supervised pre-training and semantic segmentation, and in particular to an oral potential malignant disease image segmentation method, an electronic device and a medium. BACKGROUND

[0002] Oral potential malignant disease is a more serious type of oral disease, which has the possibility of evolving into malignant tumor and becoming oral squamous cell carcinoma. Automatic segmentation of its lesion area has strong clinical research significance, which can quickly give the category, size and edge information of the corresponding lesion, assist doctors in clinical rapid diagnosis, and alleviate the problems of difficult training of high-level oral doctors and long diagnosis process. In recent years, methods based on deep learning have become the mainstream of medical image automatic analysis, which can automatically extract features in oral images to realize end-to-end lesion classification, detection and segmentation tasks. However, the research on oral disease images still has the following problems:

[0003] (a) The data of oral potential malignant disease images is scarce, and the pixel-level annotation required for lesion segmentation research is very costly, and the data is very scarce;

[0004] (b) There is a lack of public pre-training data sets in the oral field, and the model parameters directly transferred from natural images may not be well applied in oral image analysis. SUMMARY

[0005] In order to overcome the deficiencies of the prior art, the present application provides an oral potential malignant disease image segmentation method based on mask image modeling, an electronic device and a medium.

[0006] In a first aspect, an oral potential malignant disease image segmentation method is provided by the embodiments of the present application, and the method comprises:

[0007] An oral potential malignant disease image data set is obtained, which includes unannotated oral potential malignant disease images and annotated oral potential malignant disease images;

[0008] The unannotated oral potential malignant disease images are divided into image blocks, and the image blocks are input into a teacher model after a first mask is randomly added to obtain oral potential malignant disease reconstruction images;

[0009] The loss between the oral potential malignant disease reconstruction images and the unannotated oral potential malignant disease images is calculated to construct a loss map. The average loss corresponding to each image block combination in the loss map is calculated, and the region of the image block combination corresponding to the first K average losses is selected as a second mask, wherein the image block combination is composed of an image block and its neighbor image blocks. A random mask is generated, and the second mask and the random mask are superimposed as a third mask.

[0010] The third mask is added to the unlabeled oral potential malignant disease image, and the teacher model and the student model are cooperatively trained based on a loss function;

[0011] The labeled oral potential malignant disease image is input into an encoder of the trained student model for feature extraction, and then input into a semantic segmentation model for fine-tuning.

[0012] The oral potential malignant disease image to be processed is input into the fine-tuned semantic segmentation model to obtain a segmentation result.

[0013] In a second aspect, an electronic device is provided, including a memory and a processor, the memory being coupled to the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the above-mentioned oral potential malignant disease image segmentation method.

[0014] In a third aspect, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the above-mentioned oral potential malignant disease image segmentation method.

[0015] In a fourth aspect, a computer program product is provided, which includes computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the above-mentioned oral potential malignant disease image segmentation method.

[0016] Compared with the prior art, the present application has the following advantages:

[0017] The present application provides an oral potential malignant disease image segmentation method, which aims to solve the problems of lack of labeled data set and difficulty in full supervision training in the oral potential malignant disease image lesion segmentation task. The method effectively extracts features of input images by establishing a multi-scale feature fusion encoder based on separable convolution and attention mechanism. The model training process focuses more on the key areas of the image by using the average teacher-student model and setting the third mask. The consistency loss and contrast learning loss are added to improve the effect of encoder self-supervised pre-training. The pre-trained encoder is migrated to a semantic segmentation model based on the UperNet structure and the FreqFusion feature upsampling method, and the labeled data is trained for semantic segmentation to improve the accuracy of the semantic segmentation model in the pixel-level segmentation of the key lesion area. The method can effectively improve the accuracy of image segmentation and provide strong support for the diagnosis of oral potential malignant diseases. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art based on the drawings without any creative effort should be within the protection scope of the present application.

[0019] Figure 1 The overall flowchart of the oral potential malignant disease image segmentation method provided by the embodiments of the present application is shown in the figure.

[0020] Figure 2 The structure diagram of the teacher model provided by the embodiments of the present application is shown in the figure.

[0021] Figure 3 The schematic diagram of the first mask provided by the embodiments of the present application is shown in the figure.

[0022] Figure 4 The schematic diagram of the second mask provided by the embodiments of the present application is shown in the figure.

[0023] Figure 5 The schematic diagram of the self-guided mask generation and consistency loss and contrast learning loss based on the average teacher-student model in the present application is shown in the figure.

[0024] Figure 6 The mask image reconstruction prediction visualization schematic diagram provided by the embodiments of the present application is shown in the figure.

[0025] Figure 7 The mask image reconstruction effect diagram before and after adding consistency constraint provided by the embodiments of the present application is shown in the figure.

[0026] Figure 8 The semantic segmentation model structure diagram provided by the embodiments of the present application is shown in the figure.

[0027] Figure 9 The schematic diagram of an electronic device is shown in the figure. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort should be within the protection scope of the present application.

[0029] It should be noted that the features in the following embodiments and implementation manners can be combined with each other without conflict.

[0030] As Figure 1As shown in the figure, this embodiment of the invention provides a method for image segmentation of potential oral malignant diseases, the method comprising the following steps:

[0031] Step S1: Obtain an image dataset of potential oral malignant diseases, which includes unlabeled images of potential oral malignant diseases and labeled images of potential oral malignant diseases.

[0032] Furthermore, the oral cavity potential malignant disease image dataset covers multiple parts of the oral cavity, including the cheek, tongue, gums, lips, floor of mouth, hard palate, and soft palate, as well as various diseases such as oral leukoplakia, oral erythema, oral submucosal fibrosis, oral lichen planus, and discoid lupus erythematosus. The oral cavity potential malignant disease image data requires clear images and accurate exposure.

[0033] Furthermore, the labeled images of potential oral malignancies were annotated by experienced dentists using Labelme software, in which the outlines of the lesions were precisely marked as continuous line segments, and the corresponding diagnostic results were recorded.

[0034] Step S2: Divide the unlabeled image of potential oral malignant disease into image blocks, randomly add a first mask to the image blocks and input them into the teacher model to obtain the reconstructed image of potential oral malignant disease.

[0035] Among them, such as Figure 3 As shown, the process of randomly adding a first mask to an image patch includes:

[0036] Preprocessing of unlabeled images of potential oral malignancies includes random cropping, random horizontal flipping, and normalization;

[0037] The preprocessed images of potential oral malignancies are divided into image patches; the default image patch size is 4×4, and the input image size is H×W. The sequence of eigenvectors;

[0038] For unlabeled images of potential oral malignant diseases, a first mask is randomly added according to the image block combination. The image block combination consists of 2×2 image blocks with one image block as the initial reference at the upper left corner and its three neighboring image blocks.

[0039] It should be noted that randomly adding a first mask to an image block is equivalent to performing a random proximity masking operation on the image block. The proximity masking strategy is to randomly mask each image block and the four image blocks within a total of four squares with the top-left initial index of the image block. This is equivalent to increasing the size of the image block of the initial mask by four times.

[0040] Furthermore, such as Figure 2As shown, the encoder of the teacher model includes a first separable convolutional block, a second separable convolutional block, a first attention mechanism block, and a second attention mechanism block connected in sequence; and the SwiGLU activation function is used instead of the original FFN.

[0041] The unlabeled images of potential oral malignancies are H×W in size. These images are input into the teacher model. The encoder of the teacher model extracts features through a first separable convolutional block, a second separable convolutional block, a first attention mechanism block, and a second attention mechanism block, respectively. The output feature vectors have dimensions of H×W. It contains rich multi-scale information.

[0042] The feature vectors output by the first separable convolutional block, the second separable convolutional block, the first attention mechanism block, and the second attention mechanism block are downsampled, then aligned and concatenated to achieve multi-scale feature fusion. The expression for the feature fusion operation is as follows:

[0043] x′ i =DownSampling(x i ), i = 1, 2, 3

[0044] x″=Concat(x i ), i = 1, 2, 3, 4

[0045] x″′=Projection(x″)

[0046] In the formula, x i This represents the features at different scales output from each stage of the encoder's first separable convolutional block, second separable convolutional block, first attention mechanism block, or second attention mechanism block. DownSampling() is a feature downsampling operation, Concat() is a feature vector concatenation operation, and Projection() is a feature dimension projection operation.

[0047] Step S3: Calculate the loss between the reconstructed image of potential oral malignant disease and the unlabeled image of potential oral malignant disease to construct a loss map; calculate the average loss corresponding to each image block combination in the loss map, and select the region of the image block combination corresponding to the first K average losses as the second mask (i.e., loss guidance mask). The image block combination consists of an image block and its neighboring image blocks; generate a random mask, and superimpose the second mask and the random mask as the third mask (i.e., self-guided mask).

[0048] Among them, such as Figure 4 and Figure 6 As shown, the process of setting the second mask includes:

[0049] Calculate the loss of each image block between the decoder output of the teacher model and the unlabeled oral potential malignant disease image, thereby constructing a loss map;

[0050] Calculate the average loss of each image block combination in the loss map, and sort the image block combinations, which are composed of a 2x2 image block with an image block as the initial reference in the upper left corner and its three neighbor image blocks;

[0051] Set the proportion between the second mask and the random mask;

[0052] According to the proportion between the second mask and the random mask, select the region of the image block combination corresponding to the first K average loss as the second mask.

[0053] It should be noted that the present example sets a third mask, so that the teacher model and the student model pay more attention to the key areas of the image and the areas difficult to train during the training process.

[0054] Step S4, input the unlabeled oral potential malignant disease image after adding the third mask to the student model, and cooperatively train the teacher model and the student model based on the loss function.

[0055] Specifically, as shown in Figure 5 The process of cooperatively training the teacher model and the student model based on the loss function includes:

[0056] Set the joint loss function, and the expression is as follows:

[0057]

[0058] In the formula, indicates the reconstruction loss, indicates the consistency loss, indicates the contrast loss, λ con indicates the weight corresponding to the consistency loss, λ cl indicates the weight corresponding to the contrast loss;

[0059] The expression of the reconstruction loss is as follows:

[0060]

[0061] In the formula, M indicates the third mask, x indicates the input image, and f indicates the student model;

[0062] The expression of the consistency loss is as follows:

[0063]

[0064] In the formula, t and s respectively indicate the feature vectors output by the encoder of the teacher model and the student model; and the mask reconstruction effects before and after adding the consistency loss are asFigure 7

[0065] The expression of the contrastive learning loss is as follows:

[0066]

[0067] In the formula, N is the number of input images in a batch, z i and z j represent positive sample pairs, z i and z k represent negative sample pairs, sim(a,b) represents the cosine similarity between two sample pairs, and τ represents the temperature coefficient in contrastive learning.

[0068] The parameters of the teacher model are updated by the exponential moving average of the parameters of the student model, and the expression of the exponential moving average is as follows:

[0069] teacher(t) = (1- a) * teacher(t-1) + a * student(t)

[0070] In the formula, teacher() and student() represent the parameter values of the teacher model and the student model at the training time point t respectively, and a is a smoothing factor, which is in the range of [0, 1].

[0071] Step S5: The labeled oral potential malignant disease image is input into the encoder of the trained student model for feature extraction, and then input into the semantic segmentation model for fine-tuning.

[0072] Specifically, as Figure 8 shown, the labeled oral potential malignant disease image is input into the encoder of the trained student model for feature extraction to obtain a feature vector;

[0073] The feature vector is input into the UperNet model, and the FreqFusion feature upsampling method is used for feature fusion, and then the segmentation head is used to output the segmentation result, so as to complete the fine-tuning of the semantic segmentation model.

[0074] Step S6: The oral potential malignant disease image to be processed is input into the fine-tuned semantic segmentation model to obtain a segmentation result.

[0075] ​In summary, the present application provides an oral potential malignant disease image segmentation method, aiming at the problems of lack of labeled data set and difficulty of full supervision training in the oral potential malignant disease image lesion segmentation task, effective feature extraction of the input image is carried out through the establishment of a multiscale feature fusion encoder based on separable convolution and attention mechanism; By setting the third mask and the average teacher-student model, the model pays more attention to the key areas of the image during the training process; The consistency loss and the contrast learning loss are added to improve the effect of the encoder self-supervised pre-training; The pre-trained encoder is migrated to the semantic segmentation model based on the UperNet structure and the FreqFusion feature upsampling method, the labeled data is trained for semantic segmentation, and the accuracy of the semantic segmentation model for key lesion area pixel-level segmentation is improved. The present application method can effectively improve the image segmentation accuracy and provide strong support for the diagnosis of oral potential malignant diseases.

[0076] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the oral potential malignant disease image segmentation method as described above. As shown in the Figure 9 The oral potential malignant disease image segmentation method provided by the embodiment of the present application is a hardware structure diagram of any data processing capable device. In addition to the Figure 9 As shown in the processor, the memory and the network interface, any data processing capable device in the embodiment of the device usually includes other hardware according to the actual function of the data processing capable device, which will not be described here.

[0077] Correspondingly, the present application also provides a computer readable storage medium having computer instructions stored thereon, which are executed by a processor to implement the oral potential malignant disease image segmentation method as described above. The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any data processing capable device and the external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0078] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be construed as including any patern, process, method, technique, composition of matter, or device that falls within the spirit or scope of the application, including any equivalents thereof. The specification and examples are exemplary only.

[0079] It should be understood that the application is not limited to the precise construction hereinafter described and as shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.

Claims

1. An oral potentially malignant disorder image segmentation method, characterized by, The method comprises: obtaining an oral potential malignant disease image data set, the oral potential malignant disease image data set comprising unlabeled oral potential malignant disease images and labeled oral potential malignant disease images; performing image block division on the unlabeled oral potential malignant disease images, inputting the image blocks after randomly adding a first mask to a teacher model to obtain oral potential malignant disease reconstruction images; calculating the loss between the oral potential malignant disease reconstruction images and the unlabeled oral potential malignant disease images to construct a loss map, calculating the average loss corresponding to each image block combination in the loss map, and selecting the region of the image block combination corresponding to the first K average losses as a second mask, the image block combination consisting of an image block and its neighbor image blocks; generating a random mask, and superimposing the second mask and the random mask as a third mask; inputting the unlabeled oral potential malignant disease images after adding the third mask to a student model to cooperatively train the teacher model and the student model based on a loss function; inputting the labeled oral potential malignant disease images into an encoder of the trained student model to perform feature extraction, and then inputting the labeled oral potential malignant disease images into a semantic segmentation model to fine-tune the semantic segmentation model; wherein the process of fine-tuning the semantic segmentation model comprises: inputting the labeled oral potential malignant disease images into the encoder of the trained student model to perform feature extraction to obtain a feature vector; inputting the feature vector into a UperNet model, combining a FreqFusion feature upsampling method to perform feature fusion, and then outputting a segmentation result through a segmentation head to complete fine-tuning of the semantic segmentation model; inputting the oral potential malignant disease images to be processed into the fine-tuned semantic segmentation model to obtain a segmentation result.

2. The method according to claim 1, wherein, The process of randomly adding a first mask to the image blocks of the unlabeled oral potential malignant disease images comprises: performing image block division on the unlabeled oral potential malignant disease images; randomly adding a first mask to the image blocks according to an image block combination, the image block combination consisting of 2x2 image blocks with an image block as a top-left initial reference and its three neighbor image blocks.

3. The method of claim 1, wherein the method comprises: The encoder of the teacher model comprises a first separable convolution block, a second separable convolution block, a first attention mechanism block, and a second attention mechanism block connected in sequence; The size of the unlabeled oral potential malignant disease image is The unlabeled oral potential malignant disease image is input to the teacher model, the encoder of the teacher model respectively performs feature extraction through the first separable convolution block, the second separable convolution block, the first attention mechanism block and the second attention mechanism block, and the size of the output feature vector is .

4. The method according to claim 1 or 2, wherein, The process of setting the second mask comprises: calculating the loss between each image block of the oral potential malignant disease reconstruction images and the unlabeled oral potential malignant disease images to construct a loss map; calculating the average loss corresponding to each image block combination in the loss map and sorting, the image block combination consisting of 2x2 image blocks with an image block as a top-left initial reference and its three neighbor image blocks; setting the proportion between the second mask and the random mask; According to the proportion between the second mask and the random mask, the region of the image block combination corresponding to the first K average losses is selected as the second mask.

5. The method of claim 1, wherein the method is characterized by: The process of cooperatively training the teacher model and the student model based on the loss function comprises: setting a joint loss function, the expression being as follows: ； In the formula, denotes a reconstruction loss, denotes a consistency loss, denotes a contrast loss, denotes a weight corresponding to the consistency loss, denotes a weight corresponding to the contrast loss; The expression of the reconstruction loss is as follows: ； wherein denotes a third mask, denotes an input image, denotes a student model; The expression of the consistency loss is as follows: ； wherein, and respectively represent the feature vectors of the encoder outputs of the teacher model and the student model. The expression of the contrastive learning loss is as follows: ； wherein is the number of images input within a batch, and denotes a positive sample pair, and denotes a negative sample pair, denotes calculating the cosine similarity between two sample pairs, denotes a temperature coefficient in contrast learning.

6. The method of claim 1 or 5, wherein the method further comprises: The process of cooperatively training the teacher model and the student model further comprises: The parameters of the teacher model are updated by an exponential moving average of the parameters of the student model, and an expression of the exponential moving average is as follows: ； In the formula, and respectively represent the parameter values of the teacher model and the student model at the training time point , is a smoothing factor, and the value range is .

7. An electronic device comprising a memory and a processor, characterized in that The memory is coupled with the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the oral potential malignant disease image segmentation method in any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the oral potential malignant disease image segmentation method in any one of claims 1-6.

9. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the oral potential malignant disease image segmentation method in any one of claims 1-6.

Images