A medical image abnormal region identification method based on large model self-supervised learning

Abstract

The application discloses a medical image abnormal region identification method based on a large model self-supervised learning, and belongs to the technical field of image data processing. The method comprises the following steps: acquiring abnormal sample data set D1 and healthy sample data set D2 of medical images; constructing and training a discrete mask autoencoder; generating a pseudo abnormal graph of the samples in D2 to form a data set D3; automatically labeling D2 and D3; constructing a medical image abnormal region identification network and a total loss; training the network by using D2 and D3 to obtain an abnormal region identification model, which is used for identifying the abnormal region in the samples in D1. The discrete mask autoencoder can enhance the perception ability of the model to structural distortion and semantic mutation, relieve the problem of overfitting reconstruction of the abnormal region, and improve the diversity and effectiveness of the training samples through the pseudo abnormal graph. The abnormal region identification model has a pixel-level abnormal positioning capability, and is suitable for complex medical image features, has a flexible training mechanism, and can realize end-to-end optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and particularly relates to a medical image abnormal region identification method based on large model self-supervised learning. BACKGROUND

[0002] Although a number of studies have made significant progress in the field of unsupervised medical image anomaly detection, there are still several key technical bottlenecks that have not been effectively broken through. There are currently three commonly used methods for medical image anomaly detection, which are (1) CCD method: see the paper Georgescu, M.-I. (2023). Masked autoencoders for unsupervised anomaly detection in medical images. arXiv; (2) single MAE reconstruction method, see the paper Tian, Y., Pang, G., Liu, F., Chen, Y., Shin, S. H., Verjans, J. W., Singh, R., & Carneiro, G. (2021). Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in medical images. arXiv; (3) RealNet method, see the paper: Zhang, X., Xu, M., & Zhou, X. (2024). RealNet: A feature selection network with realistic synthetic anomaly for anomaly detection. arXiv.

[0003] (1) CCD method: relies on complex contrastive learning and data augmentation techniques, increasing the complexity of training and computational resource consumption, while the present method is trained through a simple reconstruction task, with lighter computational burden. Secondly, the CCD method mainly focuses on minor abnormalities, generating pseudo-anomalies through data augmentation, which may lead to excessive sensitivity of the model to abnormal regions and the need for more preprocessing steps. While the present method focuses on global structural information through the reconstruction task, it performs more stably for large-scale anomaly detection. Overall, the CCD method has certain limitations in training complexity and excessive focus on minor abnormalities.

[0004] (2) Single MAE reconstruction method (e.g., MAE-based strategy): Although abnormality detection is achieved by training a generative model using only normal images, the modeling of abnormal regions often lacks structural constraints, and the reconstruction results are more reflected as image differences at the whole image level. MAE usually uses a continuous latent space, which can cause the model to easily over-recover abnormal regions when encountering abnormal regions, thus failing to well distinguish between abnormal and normal parts. At the same time, current evaluation strategies are mostly based on simulated abnormal samples, lack multi-scale feature extraction paths and local region comparison mechanisms, and perform poorly in identifying small lesions or low-contrast abnormalities. This type of method usually does not introduce explicit difference maps, residual maps, or attention mechanisms, which suppresses the positioning accuracy and explainability of the model.

[0005] (3) RealNet method proposed for industrial vision tasks: Although it exhibits good performance in texture defect detection, its design is more suitable for structured industrial scenarios. In medical images, especially in soft tissue or organ images, the structural complexity and abnormality forms are much higher than in industrial backgrounds. Although the selection of abnormal perception features and reconstruction residuals in RealNet enhances feature discrimination, it still lacks attention mechanisms for spatial structure modeling. At the same time, this method generates synthetic abnormal images by texture disturbance, which is difficult to cover structural abnormalities such as tumors and hemorrhages with clear boundaries and complex shapes, thus limiting its generalization ability in medical image scenarios.

[0006] Glossary:

[0007] VQ-VAE: Also known as Vector Quantization Variational Autoencoder, it is a deep learning model that combines Variational Autoencoder (VAE) and Vector Quantization (Vector Quantization) to learn efficient data representation. It solves the problem of too continuous and difficult to control latent variables in traditional VAE by discretizing the continuous latent space.

[0008] ViT-based encoder refers to an encoder based on the ViT (Vision Transformer) model, which includes block embedding layers (image feature embedding), multi-layer Transformer layers, and MLP classification modules. The Transformer layer is based on the multi-head self-attention mechanism (MSA) for feature extraction, and the attention map of the Transformer is the visualization result of the self-attention mechanism (Self-Attention) in the Transformer architecture, which shows the attention weight distribution of different positions when the model processes input data.

[0009] The encoder of the VQ-VAE and the encoder in the multi-scale classifier in the application are both ViT-based encoders, and the two can be set as independent encoders or share the same encoder. SUMMARY

[0010] The application aims to provide a medical image abnormal region identification method based on large model self-supervised learning, which can realize accurate positioning of medical image abnormal regions and solve the above-mentioned problems of certain limitations in training complexity and excessive attention to minor abnormalities, weak positioning accuracy and weak generalization ability.

[0011] To achieve the above-mentioned purpose, the technical scheme adopted by the application is as follows: a medical image abnormal region identification method based on large model self-supervised learning, comprising the following steps:

[0012] S1, obtaining an abnormal sample data set D1 and a healthy sample data set D2 of medical images, the samples in D1 being abnormal samples containing abnormal regions, and the samples in D2 being healthy samples;

[0013] S2, constructing a discrete mask autoencoder M VQ-MAE ;

[0014] Obtaining a VQ-VAE, setting a random mask layer in the block embedding layer and the encoding layer of the encoder of the VQ-VAE, obtaining a VQ-MAE, training the VQ-MAE with D2 to obtain a discrete mask autoencoder M VQ-MAE , the random mask layer is used to generate a mask for the output of the block embedding layer and send it back to the encoding layer, the encoding layer is an L-layer Transformer layer, and the M VQ-MAE is used to input a natural image and output a reconstructed image after reconstruction;

[0015] S3, for each healthy sample in D2, generating a pseudo abnormal image by fusing the disturbance region, and all the pseudo abnormal images constitute a data set D3;

[0016] S4, automatically generating labels for the samples in D2 and D3;

[0017] The samples in D3 are first positive samples, the label y1=1, the samples in D2 are first negative samples, the label y1=0, each sample in D2 and D3 is divided into N blocks based on the block embedding layer, the block containing the disturbance region is the second positive sample, the label y2=1, and the remaining blocks are the second negative sample, the label y2=0;

[0018] S5, constructing a medical image abnormal region identification network, including M VQ-MAE , an attention map positioning unit, a difference map generation unit, a residual map positioning unit, a multi-scale classifier, an abnormal score positioning unit, and a fusion positioning unit;

[0019] The M VQ-MAE For input sample X, output reconstruction sample , the sample is a healthy sample in D2 or a pseudo-abnormal sample in D3;

[0020] The attention map positioning unit is used to obtain X in M VQ-MAE The self-attention matrix of each Transformer layer in is weighted and fused to obtain the attention map ;

[0021] The difference map generating unit is used to generate X and Normalized difference map of ;

[0022] The difference map positioning unit is used to calculate the average residual of each block of the difference map, rearrange it into an image and then upsample it to the sample size X to obtain a residual map ;

[0023] The multi-scale classifier includes an encoder, a first classification head connected to the encoder, a second classification head and a third classification head;

[0024] The encoder is used to input the difference map , after being divided into N blocks, each block is subjected to L layers of Transformer layers to extract L block features. The block features extracted by the lth layer of Transformer layer for the nth block of the difference map are , 1≤n≤N, 1≤l≤L;

[0025] The first classification head is used to output the predicted probability that X is the first positive sample , the second classification head is used to output the predicted probability that each block in X is the second positive sample , the third classification head is used to output is the predicted probability of the abnormal area , and based on the abnormal threshold get Abnormal label , among which, if ,but ,otherwise ;

[0026] The anomaly score locating unit is used to locate the N blocks of the Lth layer. , spliced ​​into a matrix according to the block position, and then upsampled to the sample size X to obtain the abnormal score map ;

[0027] The fusion positioning unit is used to 、 and Weighted fusion to obtain fusion graph , and normalized to a normalized fusion map , and binarization processing to obtain a fusion abnormal region map;

[0028] S6, training the medical image abnormal region identification network with D2 and D3 to obtain an abnormal region identification model, and testing with D1;

[0029] S7, inputting the samples in D1 into the abnormal region identification model to output the corresponding fusion abnormal region map.

[0030] As preferred, in S6, the total loss L total Training the medical image abnormal region identification network;

[0031] ,

[0032] In the formula, L UAD is the normalized difference map loss, L VQ is the reconstruction loss of VQ-MAE, L align is the multi-scale alignment loss calculated based on the third classification head, L region is the region supervision loss calculated based on the second classification head, L cls is the whole map classification loss calculated based on the first classification head, λ vq , λ1, λ2, λ3 are the weights of L VQ , L align , L region , L cls .

[0033] As preferred, L UAD , L align , L region , L cls are obtained as follows, respectively.

[0034] ,

[0035] ,

[0036] ,

[0037] ,

[0038] ,

[0039] L UAD In the formula, X i,j , are the pixel values of the i-th row and j-th column in X and , respectively, and σ i,j is the standard deviation of X and the pixel value of the i-th row and j-th column in the variance map, M is a set of pixel positions in X, |M| is the total number of pixels in X, and ε is a minimum value;

[0040] L align In the formula, λ l is the weight of the l-th layer of the Transformer layer, is the square of the Euclidean distance;

[0041] L region In the formula, L cls (n) is the classification loss of the n-th block of the difference map.

[0042] Preferably, in S2, constructing the VQ-MAE specifically includes steps S21-S24;

[0043] S21, obtaining a VQ-VAE including an encoder and a decoder, the encoder including a block embedding layer, an encoding layer, and a vector quantization layer;

[0044] S22, setting a random mask layer between the block embedding layer and the encoding layer;

[0045] The block embedding layer is configured to divide the abnormal sample into N blocks, embed each block into a high-dimensional space to obtain an embedding vector, and construct a Token sequence with the N embedding vectors, the embedding vector of the n-th block in the Token sequence being zn n ;

[0046] The random mask layer is configured to randomly mask the Token sequence according to a preset masking ratio r to obtain a mask sequence, the mask of the n-th element in the Token sequence being Mask n ;

[0047] The encoding layer is configured to extract features from the mask sequence to obtain a continuous latent representation z;

[0048] The vector quantization layer is configured to quantize the latent representation z into N discrete code words based on a codebook to obtain a discrete Token sequence;

[0049] The decoder is configured to reconstruct the abnormal sample according to the discrete Token sequence to output a reconstructed image;

[0050] S23, constructing a reconstruction loss L VQ of the VQ-MAE,

[0051] ,

[0052] In the formula, e is a codebook vector generated for the latent representation z during the training process, sg[·] is a stop gradient operation, is the square of the L2 norm, and β is the L VQThe second item weight;

[0053] S24, training VQ-MAE with D2 for a preset number of iterations, calculating number L each time VQ , and updating the network parameters of VQ-MAE and the value of codebook vector e, and the codebook vector after the iteration is the discrete Token sequence.

[0054] As a preferred, S3 generates the pseudo anomaly map, which is specifically;

[0055] The healthy sample is input into M VQ-MAE , to obtain the reconstructed image ;

[0056] The is divided into subblocks of different sizes and W subblocks are randomly selected, and the preset disturbance intensity factor of the wth subblock B w is β w , 1≤w≤W, for the ith row and jth column pixel in B , if belongs to B w , then according to , a disturbed pixel is generated, and all disturbed pixels constitute a disturbed region;

[0057] The disturbed region and the healthy sample are fused by Poisson fusion to obtain the pseudo anomaly map .

[0058] As a preferred, the attention map positioning unit obtains an attention map , which is specifically:

[0059] For N block features of the lth layer, attention weights are obtained through a self-attention mechanism, and then the block positions are spliced into a matrix A (l) , and A (l) is upsampled to the size of the sample X to obtain a self-attention matrix of the lth layer ;

[0060] The self-attention matrices of L layers are obtained in sequence, and then the attention map is calculated according to the following formula ;

[0061] ,

[0062] In the formula, is the weight of the lth layer Transformer layer.

[0063] As a preferred, the normalized difference map of the difference map generation unit is obtained according to the following formula;

[0064] ,

[0065] wherein, is a difference map of X and is a variance map obtained based on the difference map, and epsilon is a minimum value.

[0066] As preferred, in S5, the encoder of the multi-scale classifier is a common M VQ-MAE encoder.

[0067] As preferred, the normalized fusion map is obtained according to the following formula: final ;

[0068] ,

[0069] wherein, M final (i,j) and M final (i,j) are respectively the i-th row and j-th column pixel in M , , and tau is a preset fusion positioning threshold.

[0070] As preferred, S6 is a training process method S61-S63.

[0071] S61, preset an iteration number.

[0072] S62, randomly extract a batch of samples from D2 and D3, and input them into the medical image abnormal region recognition network to generate a fusion abnormal region map of each sample, calculate L total , and adjust the network parameters of the multi-scale classifier to minimize L total .

[0073] S63, repeat S62 until the iteration ends to obtain an abnormal region recognition model.

[0074] In the present application, the VQ-MAE is constructed to introduce a vector quantization mechanism to construct a discrete latent space, thereby enhancing the model's perception ability of structural distortion and semantic mutation and alleviating the problem of overfitting reconstruction of abnormal regions.

[0075] The present application proposes a new pseudo abnormality map construction method, which naturally fuses and reconstructs the artifacts in the healthy image to generate a pseudo abnormality map, and the purpose is to guide the model to learn the abnormal perception ability under the condition of no abnormal sample.

[0076] ​The multiscale classifier of the application: introduce random mask to improve the information bottleneck effect of the masked area, and introduce multiscale alignment loss based on multiple layers of Transformer in the encoder, full image classification based on full image classification, and regional supervision loss based on image block classification calculation, so that the multiscale classifier has the ability to jointly guide the positioning and discrimination of abnormal areas of different depth feature layers.

[0077] Compared with the prior art, the application has the advantages that:

[0078] (1) Unsupervised learning, high stability. The method relies on normal images for unsupervised learning, the training process is simple, and the computing resource consumption is less. At the same time, the method introduces a vector quantization (VQ) mechanism to discretize the latent representation of the image, making the model more sensitive to structural distortion and semantic mutation. This can effectively improve the robustness of the model to abnormal areas and reduce the overfitting problem of abnormal areas in the reconstruction process.

[0079] (2) Pseudo abnormal graph construction has more medical structure authenticity. In view of the problem that the MAE method lacks real structure simulation in pseudo abnormal graph generation, the application designs a pseudo abnormal graph generation strategy combining disturbance area control and Poisson fusion, which not only simulates the spatial distribution of abnormalities, but also preserves the structural continuity and boundary features in medical images, improving the diversity and effectiveness of the training samples.

[0080] (3) It has pixel-level abnormal positioning capability. The application cooperatively designs a difference map generation unit, a residual map positioning unit and an attention map positioning unit, completes the weighted integration of multiple source maps in the fusion positioning unit, finally obtains a normalized fusion map and performs binaryzation, which can realize high-resolution explicit identification of abnormal areas, and makes up for the defects of CCD method, RealNet method and the like that can only provide full image level abnormal score and positioning accuracy. A large number of experiments on mainstream medical data sets show that the method of the application is significantly better than existing mainstream self-supervised methods in image-level recognition and pixel-level positioning tasks.

[0081] (4) The structure design is adapted to the characteristics of complex medical images. Unlike RealNet and other industrial image methods, the application focuses on the characteristics of complex structures, diverse textures and fuzzy lesion boundaries in medical images, introduces a ViT encoder and a multiscale feature extraction mechanism, and further aggregates the deep local representation of the Transformer layer in the multiscale classifier, significantly enhancing the model's ability to model soft tissue abnormalities and generalization ability.

[0082] (5) The training mechanism is flexible, and end-to-end optimization is performed. The abnormal region recognition network for medical images is uniformly constructed, a total loss function is constructed on the basis of multi-module output, reconstruction loss, alignment loss, region supervision loss and classification loss are covered, cooperative learning of each module is ensured, and end-to-end training optimization is performed, so that high-precision detection can be completed only by using healthy samples and pseudo-abnormal samples, and data dependency is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0083] Fig. 1 is a VQ-MAE structure diagram;

[0084] Fig. 2 is a flowchart for generating pseudo-abnormal images from healthy samples;

[0085] Fig. 3 is a structure diagram of the abnormal region recognition network for medical images. DETAILED DESCRIPTION

[0086] The application will be further described below in combination with embodiments and drawings.

[0087] Embodiment 1: Referring to Figs. 1-3 A medical image abnormal region recognition method based on large model self-supervised learning comprises the following steps:

[0088] S1, acquiring abnormal sample dataset D1 and healthy sample dataset D2 of medical images, the samples in D1 are abnormal samples containing abnormal regions, and the samples in D2 are healthy samples;

[0089] S2, constructing a discrete mask autoencoder M VQ-MAE ;

[0090] A VQ-VAE is acquired, a random mask layer is arranged at a block embedding layer and an encoding layer of the encoder of the VQ-VAE, a VQ-MAE is obtained, the VQ-MAE is trained by using D2 to obtain a discrete mask autoencoder M VQ-MAE , the random mask layer is used for generating a mask for the output of the block embedding layer and sending the mask into the encoding layer again, the encoding layer is an L-layer Transformer layer, and the M VQ-MAE is used for inputting a natural image and outputting a reconstructed image after reconstruction;

[0091] S3, for each healthy sample in D2, a pseudo-abnormal image is generated by fusing a disturbance region, and all pseudo-abnormal images constitute a dataset D3;

[0092] S4, automatically generating labels for the samples in D2 and D3;

[0093] The sample in D3 is the first positive sample, with label y1=1. The sample in D2 is the first negative sample, with label y1=0. Each sample in D2 and D3 is divided into N blocks based on the block embedding layer. The block containing the perturbation area is the second positive sample, with label y2=1. The remaining blocks are the second negative samples, with label y2=0.

[0094] S5, construct a medical image abnormal region recognition network, including M VQ-MAE , attention map localization unit, difference map generation unit, residual map localization unit, multi-scale classifier, anomaly score localization unit, fusion localization unit;

[0095] The M VQ-MAE For input sample X, output reconstruction sample , the sample is a healthy sample in D2 or a pseudo-abnormal sample in D3;

[0096] The attention map positioning unit is used to obtain X in M VQ-MAE The self-attention matrix of each Transformer layer in is weighted and fused to obtain the attention map ;

[0097] The difference map generating unit is used to generate X and Normalized difference map of ;

[0098] The difference map positioning unit is used to calculate the average residual of each block of the difference map, rearrange it into an image and then upsample it to the sample size X to obtain a residual map ;

[0099] The multi-scale classifier includes an encoder, a first classification head connected to the encoder, a second classification head and a third classification head;

[0100] The encoder is used to input the difference map , after being divided into N blocks, each block is subjected to L layers of Transformer layers to extract L block features. The block features extracted by the lth layer of Transformer layer for the nth block of the difference map are , 1≤n≤N, 1≤l≤L;

[0101] The first classification head is used to output the predicted probability that X is the first positive sample , the second classification head is used to output the predicted probability that each block in X is the second positive sample , the third classification head is used to output is the predicted probability of the abnormal area , and based on the abnormal threshold get Abnormal label , among which, if ,but , otherwise ;

[0102] The abnormal score positioning unit is configured to position the Lth layer N patches of the sample X according to the abnormal score positioning formula , and splice the patch positions into a matrix, and then up-sample to the sample X size to obtain an abnormal score map ;

[0103] The fusion positioning unit is configured to fuse the abnormal score map, the abnormal region positioning map and the abnormal region positioning map to obtain a fusion map , and The fusion positioning unit is configured to fuse the abnormal score map, the abnormal region positioning map and the abnormal region positioning map to obtain a fusion map , normalize the fusion map to obtain a normalized fusion map , and then binarize to obtain a fusion abnormal region map;

[0104] S6, training the medical image abnormal region identification network with D2 and D3 to obtain an abnormal region identification model, and testing with D1;

[0105] S7, inputting the sample in D1 into the abnormal region identification model to output the corresponding fusion abnormal region map.

[0106] In this embodiment, the abnormal sample dataset D1 and the healthy sample dataset D2 can be constructed based on the BraTS2020 dataset or the LUNA16 dataset. Taking the BraTS2020 dataset as an example, the BraTS2020 training dataset contains 369 aligned four-modal MRI data. The tumors presented in the dataset are glioblastoma and low-grade glioma. One of the main tasks of this dataset is tumor segmentation. In this application, a simpler problem is solved: tumor detection. The tumor is regarded as an abnormality, and the sample containing the tumor is stored in D1 as an abnormal sample, and the sample without the tumor is stored in D2 as a healthy sample, and the VQ-MAE is trained only with the healthy samples in D2. Based on D2, D3 is constructed, and the medical image abnormal region identification network is trained with D2 and D3 to obtain an abnormal region identification model, and the abnormal region identification model is tested with D1.

[0107] Embodiment 2: see Figs. 1-3 On the basis of embodiment 1, a loss function design method for training is given, the purpose is to train the medical image abnormal region identification network to convergence, obtain the abnormal region identification model, so that it has the ability to identify the abnormal region in the sample. In S6, the total loss L total training the medical image abnormal region identification network;

[0108] ,

[0109] In the formula, L UAD is the normalized difference value map loss, L VQ is the reconstruction loss of VQ-MAE, and Lalign is the multi-scale alignment loss calculated based on the third classification head, L region is the regional supervision loss calculated based on the second classification head, L cls is the full-image classification loss calculated based on the first classification head, λ vq ,λ1,λ2,λ3 are L VQ , L align , L region , L cls The weight of .

[0110] L UAD , L align , L region , L cls The following formulas are obtained respectively;

[0111] ,

[0112] ,

[0113] ,

[0114] ,

[0115] ,

[0116] X i,j 、 X and The pixel value in row i and column j, σ i,j For X and The pixel value in the i-th row and j-th column of the variance map is M, M is the set of pixel positions in X, |M| is the total number of pixels in X, ϵ is the minimum value, and λ l is the weight of the lth Transformer layer, is the square of the Euclidean distance, L cls (n) is the classification loss of the nth block of the difference map.

[0117] Example 3: See Figs. 1-3 Based on Example 1, the construction of VQ-MAE in S2 specifically includes steps S21 to S24;

[0118] S21, obtaining a VQ-VAE, including an encoder and a decoder, wherein the encoder includes a block embedding layer, a coding layer, and a vector quantization layer;

[0119] S22, setting a random mask layer between the block embedding layer and the coding layer to obtain VQ-MAE;

[0120] The block embedding layer is configured to divide the abnormal sample into N blocks, embed each block into a high-dimensional space to obtain an embedding vector, and construct a Token sequence with the N embedding vectors, wherein the embedding vector of the nth block in the Token sequence is z n ;

[0121] The random mask layer is configured to randomly mask the Token sequence according to a preset masking ratio r to obtain a mask sequence, wherein the mask of the nth element in the Token sequence is Mask n ;

[0122] The encoding layer is configured to perform feature extraction on the mask sequence to obtain a continuous latent representation z.

[0123] The vector quantization layer is configured to quantize the latent representation z into N discrete code words based on a codebook to obtain a discrete Token sequence.

[0124] The decoder is configured to reconstruct the abnormal sample according to the discrete Token sequence to output a reconstructed image.

[0125] S23, construct the reconstruction loss L of VQ-MAE VQ ,

[0126] ,

[0127] wherein e is a codebook vector generated for the latent representation z in the training process, sg[.] is a stop gradient operation, is the square of the L2 norm, and β is the weight of the second term in the L VQ .

[0128] S24, train the VQ-MAE with D2 for a preset number of iterations, calculate the number L VQ of iterations each time, and update the network parameters of the VQ-MAE and the value of the codebook vector e, and the codebook vector after the iteration is completed as the discrete Token sequence.

[0129] Regarding the training of VQ-MAE: the encoder in VQ-MAE adopts a ViT-Base encoder, the number of Transformer blocks is set to 8, the number of attention heads is 16, and the decoder embedding dimension is 512. The abnormal sample dataset D1 can be selected from the BraTS2020 dataset or the LUNA16 dataset. Taking the BraTS2020 dataset as an example, the sample input size is set to 224x224x1, the VQ-MAE is trained with the samples in D1 for 1600 cycles, and the mask ratio is set to 0.75. In order to obtain the reconstruction of the sample, we replace the unmasked labels with the original labels. In order to obtain the final reconstruction of the sample, we pass it through the VQ-MAE framework 4 times and average the result output.

[0130] Regarding training the network for identifying abnormal regions in medical images: The primary focus was on training a multi-scale classifier, whose encoder also used the ViT-Base encoder. This multi-scale classifier and the ViT-Base encoder used in VQ-MAE can be shared. When training the network for identifying abnormal regions in medical images, the training cycle (number of iterations) was 100. Optimization was performed using the AdamW optimizer, with a learning rate of 0.001 and weight decay of 0.05. The simulated experiments were conducted on a GeForce GTX 3090 GPU with 24 GB of VRAM.

[0131] Example 4: See Figs. 1-3 ,S3 generates the pseudo anomaly graph specifically as follows;

[0132] The healthy sample is input as a natural image P into M VQ-MAE , and obtain the reconstructed image ;

[0133] Will Divide into sub-blocks of different sizes and randomly select W sub-blocks, and preset the w-th sub-block B w The perturbation intensity factor is β w , 1≤w≤W, for The pixel in row i and column j ,like Belongs to B w According to , generate perturbation pixels , all disturbed pixels constitute the disturbed area;

[0134] Use Poisson fusion to fuse the disturbance area with the healthy sample to obtain the pseudo anomaly map .

[0135] The attention map positioning unit obtains the attention map Specifically:

[0136] For the N block features of the lth layer, the attention weights are obtained through the self-attention mechanism, and then spliced ​​into a matrix A according to the block position (l) , A (l) Upsample to sample size X to get the self-attention matrix of layer l ;

[0137] Get the self-attention matrix of L layers in turn, and then calculate the attention map according to the following formula ;

[0138] ,

[0139] Where, The weights of the lth Transformer layer.

[0140] The normalized difference map of the difference map generation unit is obtained according to the following formula;

[0141] ,

[0142] wherein, is a difference map of X and , is a variance map obtained based on the difference map, and ε is a minimum value.

[0143] The normalized fusion map The fusion abnormal region map M is obtained according to the following formula final ;

[0144] ,

[0145] wherein, M final (i,j) and M final , are respectively the i-th row and j-th column pixel in M final , , and τ is a preset fusion positioning threshold.

[0146] S6 training process method S61-S63

[0147] S61, presetting the number of iterations;

[0148] S62, randomly extracting a batch of samples from D2 and D3, and inputting the samples into the medical image abnormal region identification network to generate a fusion abnormal region map of each sample, calculating L total , and adjusting the network parameters of the multi-scale classifier to minimize L total .

[0149] S63, repeating S62 until the iteration ends to obtain an abnormal region identification model.

[0150] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying abnormal regions in medical images based on large-scale model self-supervised learning, characterized in that: The following steps are involved: S1, obtain the abnormal sample dataset D1 and the healthy sample dataset D2 of medical images, where the samples in D1 are abnormal samples containing abnormal areas; The samples in D2 are healthy samples; S2, construct discrete mask autoencoder M VQ-MAE ; Get a VQ-VAE, set a random mask layer in the block embedding layer and encoding layer of its encoder to get VQ-MAE, and use D2 to train VQ-MAE to get a discrete mask autoencoder M VQ-MAE The random mask layer is used to generate a mask for the output of the block embedding layer and then send it to the encoding layer. The encoding layer is an L-layer Transformer layer, and the M VQ-MAE Used to input natural images and output reconstructed images after reconstruction; S3, for each healthy sample in D2, a pseudo-abnormal map is generated by fusing the perturbation region. All pseudo-abnormal maps constitute the dataset D3; S4, automatically generates annotations for samples in D2 and D3; The sample in D3 is the first positive sample, with label y1=1. The sample in D2 is the first negative sample, with label y1=0. Each sample in D2 and D3 is divided into N blocks based on the block embedding layer. The block containing the perturbation area is the second positive sample, with label y2=1. The remaining blocks are the second negative samples, with label y2=0. S5, construct a medical image abnormal region recognition network, including M VQ-MAE , attention map localization unit, difference map generation unit, residual map localization unit, multi-scale classifier, anomaly score localization unit, fusion localization unit; The M VQ-MAE For input sample X, output reconstruction sample , the sample is a healthy sample in D2 or a pseudo-abnormal sample in D3; The attention map positioning unit is used to obtain X in M VQ-MAE The self-attention matrix of each Transformer layer in is weighted and fused to obtain the attention map ; The difference map generating unit is used to generate X and Normalized difference map of ; The difference map positioning unit is used to calculate the average residual of each block of the difference map, rearrange it into an image and then upsample it to the sample size X to obtain a residual map ; The multi-scale classifier includes an encoder, a first classification head connected to the encoder, a second classification head and a third classification head; The encoder is used to input the difference map , after being divided into N blocks, each block is subjected to L layers of Transformer layers to extract L block features. The block features extracted by the lth layer of Transformer layer for the nth block of the difference map are , 1≤n≤N, 1≤l≤L; The first classification head is used to output the predicted probability that X is the first positive sample , the second classification head is used to output the predicted probability that each block in X is the second positive sample , the third classification head is used to output is the predicted probability of the abnormal area , and based on the abnormal threshold get Abnormal label , among which, if ,but ,otherwise ; The anomaly score locating unit is used to locate the N blocks of the Lth layer. , spliced ​​into a matrix according to the block position, and then upsampled to the sample size X to obtain the abnormal score map ; The fusion positioning unit is used to 、 and Weighted fusion to obtain fusion graph , normalized to normalized fusion graph , then binarize it to get the fused abnormal area map; S6, use D2 and D3 to train the medical image abnormal region recognition network to obtain the abnormal region recognition model, and test it with D1; S7, input the samples in D1 into the abnormal region recognition model and output the corresponding fused abnormal region map; In S2, constructing VQ-MAE specifically includes steps S21 to S24; S21, obtaining a VQ-VAE, including an encoder and a decoder, wherein the encoder includes a block embedding layer, a coding layer, and a vector quantization layer; S22, setting a random mask layer between the block embedding layer and the coding layer; The block embedding layer is used to divide the abnormal sample into N blocks, embed each block into a high-dimensional space to obtain an embedding vector, and form a Token sequence with the N embedding vectors. The embedding vector of the nth block in the Token sequence is z n ; The random mask layer is used to randomly mask the Token sequence according to the preset occlusion ratio r to obtain a mask sequence. The mask of the nth element in the Token sequence is Mask n ; The encoding layer is used to extract features from the mask sequence to obtain a continuous potential representation z; The vector quantization layer is used to quantize the potential representation z into N discrete codewords based on the codebook to obtain a discrete Token sequence; The decoder is used to reconstruct the abnormal sample according to the discrete Token sequence and output the reconstructed image; S23, construct the reconstruction loss L of VQ-MAE VQ ; ， Where e is the codebook vector generated for the potential representation z during training, sg[∙] is the stop gradient operation, is the square of the L2 norm, β is L VQ The second weight in S24, use D2 to train VQ-MAE according to the preset number of iterations, and calculate the number L in each iteration VQ , and update the network parameters of VQ-MAE and the value of the codebook vector e. The codebook vector after the iteration is used as the discrete Token sequence.

2. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 1, characterized in that: In S6, to minimize the total loss L total Training a network to identify abnormal regions in medical images; ， Where, L UAD is the normalized difference map loss, L VQ is the reconstruction loss of VQ-MAE, L align is the multi-scale alignment loss calculated based on the third classification head, L region is the regional supervision loss calculated based on the second classification head, L cls is the full-image classification loss calculated based on the first classification head, λ vq ,λ1,λ2,λ3 are L VQ 、L align 、L region 、L cls The weight of .

3. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 2, characterized in that: L UAD , L align , L region , L cls The following formulas are obtained respectively; ， ， ， ， ， L UAD In the formula, X i,j 、 X and The pixel value in row i and column j in , σ i,j For X and The pixel value in the i-th row and j-th column of the variance map of , M is the set of pixel positions in X, |M| is the total number of pixels in X, and ϵ is the minimum value; L align In the formula, λ l is the weight of the lth Transformer layer, is the square of the Euclidean distance; L region In the formula, L cls (n) is the classification loss of the nth block of the difference map.

4. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 1, characterized in that: S3 generates a pseudo anomaly graph specifically as follows; The healthy sample is input as a natural image P into M VQ-MAE , and obtain the reconstructed image ; Will Divide into sub-blocks of different sizes and randomly select W sub-blocks, and preset the w-th sub-block B w The perturbation intensity factor is β w , 1≤w≤W, for The pixel in row i and column j ,like Belongs to B w According to , generate perturbation pixels , all disturbed pixels constitute the disturbed area; Use Poisson fusion to fuse the disturbance area with the healthy sample to obtain the pseudo anomaly map .

5. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 1, characterized in that: The attention map positioning unit obtains the attention map Specifically: For the N block features of the lth layer, the attention weights are obtained through the self-attention mechanism, and then spliced ​​into a matrix A according to the block position (l) , A (l) Upsample to sample size X to get the self-attention matrix of layer l ; Get the self-attention matrix of L layers in turn, and then calculate the attention map according to the following formula ; ， Where, The weights of the lth Transformer layer.

6. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 1, characterized in that: Normalized difference map of the difference map generation unit According to the following formula: ， Where, For X and The difference graph of is the variance map obtained based on the difference map, and ϵ is the minimum value.

7. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 1, characterized in that: In S5, the encoder of the multi-scale classifier is a shared M VQ-MAE encoder.

8. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 1, characterized in that: Normalized fusion graph The fusion abnormal area map M is obtained according to the following formula final ; ， Where M final (i,j), M final 、 The pixel in row i and column j in , τ is the preset fusion positioning threshold.

9. The method for identifying abnormal regions in medical images based on large-scale model self-supervised learning according to claim 2, characterized in that: S6 training process method S61~S63; S61, preset number of iterations; S62 randomly extracts a batch of samples from D2 and D3 and feeds them into the medical image abnormal region recognition network to generate a fused abnormal region map for each sample and calculate L total , and to minimize L total Adjust the network parameters of the multi-scale classifier; S63, repeat S62 until the iteration ends, and obtain the abnormal area recognition model.

Images