Medical image anomaly detection and positioning method based on comparative learning

Abstract

The invention relates to the technical field of medical image anomaly detection and positioning, and discloses a medical image anomaly detection and positioning method based on comparative learning, and the method comprises the steps: obtaining a medical image; a medical image anomaly detection and positioning model is constructed and trained; performing feature extraction on the medical image from different angles by using a feature extraction network to obtain fusion features including overall features and lesion edge features; generating a medical image category prediction probability according to the fusion features by using a medical image anomaly detection network to obtain a medical image anomaly detection result; and generating a central thermodynamic diagram according to the fusion features by using a medical image lesion positioning network, predicting a size regression value of the target and an offset of a central point, and obtaining a medical image lesion positioning result. According to the method, the detail and edge information of the image is extracted through the spectral residual error, so that the features in the chest radiograph are described at different angles, and effective medical image anomaly detection and positioning are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image anomaly detection and positioning, and in particular to a medical image anomaly detection and positioning method based on contrastive learning. Background Art

[0002] Medical images are one of the key means to understand the anatomical structure and pathological changes of the human body. With the continuous development of technology, researchers have built an automated medical image analysis system by scanning medical images and loading them into computers. In recent years, with the rapid development of deep learning in computer vision, it has overcome the limitations of traditional machine learning that relies on manual feature extraction and has gradually become the core technical means of computer-aided diagnosis. Anomaly detection in medical images aims to identify possible lesions and further help locate abnormal areas caused by the disease on the human body. Medical image anomaly detection based on deep learning can effectively reduce manpower, time and economic costs, and make important contributions to the advancement of medical services and the improvement of patients' quality of life.

[0003] Chest X-ray images are common diagnostic tools in the medical field. Through X-ray imaging, the condition of the chest area can be observed. For lung lesions such as pneumonia, effusion, and nodules, the images often show increased local density and turbidity, which helps doctors to conduct preliminary analysis and screening of cardiopulmonary diseases. Existing work has studied the detection of medical image anomalies on chest X-rays, but the existing methods for detecting medical image anomalies have the following problems: 1) The feature construction of chest X-ray images is not sufficient. Most existing methods use feature extraction networks trained on large image datasets such as ImageNet to extract image features, and have not fully utilized medical field information such as the edge features of lesions. 2) The correlation between the classification task and the lesion localization task is not fully utilized. Although existing methods have used multi-task learning methods to perform medical image anomaly detection and localization tasks simultaneously, the performance of the localization task is indirectly improved by only introducing the classification task, which affects the feature learning and extraction capabilities of the model, and fails to fully explore the correlation between the classification task and the localization task. 3) The discrimination between normal and abnormal samples is insufficient. Existing methods have used contrastive learning to increase the deep differences between normal and abnormal features, but they only compare the differences between normal and abnormal from a global perspective, and have never considered the local potential differences between normal and abnormal in combination with positioning tasks.

[0004] Prior art 1 uses a single-stage detector (SSD)-based architecture and a multi-task learning method to simultaneously perform medical image disease classification and lesion localization tasks, and uses data enhancement and post-processing techniques to improve the performance of medical image anomaly detection tasks. The limitation of this method is that it does not fully incorporate medical expertise to construct chest X-ray features, which may result in the model being inaccurate in extracting features for complex lesions. In addition, although the classification task and lesion localization task are related, this method fails to effectively utilize this correlation to further improve the overall detection performance.

[0005] Existing technology 2 uses contrast loss and n-shot learning to diagnose pneumonia based on Siamese neural network. By capturing differential features through contrast learning and enhancing the generalization ability of the model, good diagnostic results are achieved. The limitation of this method is that it only compares the difference between normal and abnormal images at the level of global features under the classification task, which is easily affected by irrelevant information and leads to insufficient discrimination. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a medical image anomaly detection and positioning method based on contrastive learning.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0008] A method for detecting and locating abnormalities in medical images based on contrastive learning comprises the following steps:

[0009] Acquiring medical images;

[0010] Constructing and training a medical image anomaly detection and localization model; the medical image anomaly detection and localization model includes a feature extraction network, a medical image anomaly detection network, and a medical image lesion localization network;

[0011] The feature extraction network is used to extract features from medical images from different angles to obtain fusion features including overall features and lesion edge features;

[0012] The medical image anomaly detection network is used to generate the medical image category prediction probability according to the fusion features to obtain the medical image anomaly detection result;

[0013] The medical image lesion localization network is used to generate a central heat map based on the fusion features, predict the size regression value of the target and the offset of the center point, and obtain the medical image lesion localization result.

[0014] Preferably, the feature extraction network comprises:

[0015] Spectral residual calculation module and feature extraction backbone network;

[0016] The spectrum residual calculation module is used to perform spectrum residual calculation on the medical image to generate a saliency map;

[0017] The feature extraction backbone network is used to extract fusion features including overall features and lesion edge features according to the medical image and the saliency map.

[0018] Preferably, the spectral residual calculation module is used to perform spectral residual calculation on the medical image to generate a saliency map, including:

[0019] Perform Fourier transform on medical images to obtain frequency domain representation of the image;

[0020] Taking the logarithm of the amplitude part of the image frequency domain representation to generate a logarithmic amplitude spectrum; then smoothing the logarithmic amplitude spectrum to generate a smoothed logarithmic amplitude spectrum;

[0021] Calculate the difference between the logarithmic amplitude spectrum and the smoothed logarithmic amplitude spectrum to obtain the spectrum residual;

[0022] Obtain the phase angle of the image frequency domain representation to obtain the phase spectrum;

[0023] Reconstruct the spectral residual and phase spectrum to generate a new image frequency domain representation;

[0024] The new image frequency domain representation is inverse Fourier transformed to generate a saliency map.

[0025] Preferably, the feature extraction backbone network is used to extract fusion features including overall features and lesion edge features according to the medical image and the saliency map, including:

[0026] Multiple residual blocks containing multi-layer convolution structures are used to perform residual calculations on medical images and saliency maps to extract fusion features containing overall features and lesion edge features.

[0027] Preferably, the medical image anomaly detection network is used to generate a medical image category prediction probability according to the fusion features to obtain a medical image anomaly detection result, including:

[0028] A global average pooling layer is used to average all values ​​of each channel of the fusion feature to obtain a global feature representation;

[0029] Three fully connected layers are used to generate medical image category prediction probabilities based on global feature representation to obtain medical image anomaly detection results.

[0030] Preferably, a medical image lesion localization network is used to generate a central heat map according to fusion features, predict the size regression value of the target and the offset of the center point, and obtain a medical image lesion localization result, including:

[0031] The fused features are passed through the first convolutional layer and then the activation function is calculated to obtain the central heat map;

[0032] The fused features are passed through the second convolutional layer to calculate the target size corresponding to the local maximum position in the central heat map, and the size regression value of the predicted target is obtained;

[0033] The fused features are passed through the third convolutional layer to calculate the offset of the predicted target center point relative to the nearest grid point to obtain the offset of the center point;

[0034] The predicted center point position is calculated based on the center point offset.

[0035] Preferably, when pre-training the medical image anomaly detection and localization model,

[0036] The training data is input into the medical image anomaly detection network and the medical image lesion localization network to obtain the detection results and localization results respectively;

[0037] Calculate the detection loss based on the detection results and the actual labels;

[0038] Calculate the positioning loss based on the positioning result and the actual positioning label;

[0039] Calculate the pre-training loss based on the detection loss and the localization loss;

[0040] Pre-train the medical image anomaly detection and localization model based on the pre-training loss.

[0041] Preferably, the pre-training loss is:

[0042] loss = λ d ×loss d +λ l ×loss l

[0043] Among them, loss is the pre-training loss, λ d is the detection loss weight coefficient, loss d is the detection loss, λ l is the positioning loss weight coefficient, loss l For positioning loss.

[0044] Preferably, when the medical image anomaly detection and localization model is pre-trained and then end-to-end trained,

[0045] Randomly sample from the training data to obtain positive samples that belong to the same category as the training data and negative samples that belong to different categories from the training data to construct positive sample pairs and negative sample pairs;

[0046] Input the positive sample pairs and the negative sample pairs into the twin network of the feature extraction network shared by the medical image anomaly detection network and the medical image lesion localization network to obtain training feature representation, positive sample feature representation and negative sample feature representation;

[0047] Compute contrastive loss based on training feature representation, positive sample feature representation, and negative sample feature representation;

[0048] Calculate the similarity loss between the center heat map output by the medical image anomaly detection network based on the training data and the target center heat map;

[0049] Calculate the end-to-end loss based on contrast loss, similarity loss, detection loss, and localization loss;

[0050] The medical image anomaly detection and localization model is trained end-to-end based on the end-to-end loss.

[0051] Preferably, the end-to-end loss is:

[0052] loss′=λ cont ×loss cont +λ sim ×loss sim +λ′ d ×loss′ d +λ′ l ×loss′ l

[0053] Among them, loss′ is the end-to-end loss, λ cont is the comparison loss weight coefficient, loss cont is the contrast loss, λ sim is the similarity loss weight coefficient, loss sim is the similarity loss, λ′ d is the detection loss weight coefficient during end-to-end training, loss′ d is the detection loss during end-to-end training, λ′ l is the positioning loss weight coefficient during end-to-end training, loss′ l Localization loss for end-to-end training.

[0054] The present invention has the following beneficial effects:

[0055] 1. The present invention has carried out rich feature construction and representation. In order to comprehensively represent the image features, a multi-angle feature representation is designed for the chest X-ray. In addition to feature extraction through the deep learning model, the image details and edge information are extracted through the spectral residual, thereby describing the features in the chest X-ray at different angles.

[0056] 2. The present invention fully exploits the correlation information of classification and positioning tasks. The heat map of the detection model in the latent space is extracted through the activation mapping algorithm, and gradually approaches the heat map obtained by the positioning model, so that the detection model pays more attention to the location of disease manifestation, so as to correlate and promote the detection and positioning tasks.

[0057] 3. The present invention makes normal and abnormal samples more distinguishable at the feature level. The present invention uses the contrastive learning idea of ​​the twin network in the end-to-end training stage to make normal and abnormal samples more distinguishable in the feature space. With the mutual promotion of detection and positioning tasks, the model pays more attention to the local area where the lesion is located in the abnormal sample, thereby implicitly achieving the effect of making local features more distinguishable. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of a medical image anomaly detection and localization method based on contrastive learning;

[0059] Figure 2 A schematic diagram of the framework of a medical image anomaly detection and localization method based on contrastive learning;

[0060] Figure 3 Schematic diagram of the end-to-end training stage. DETAILED DESCRIPTION

[0061] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0062] The present invention designs a medical image anomaly detection and positioning method based on contrastive learning for the detection and positioning tasks of medical images. The present invention uses a feature extraction network to extract features from images from different angles; uses a twin network to make the same category features close to each other in the feature space, and different category features far away from each other in the feature space, and obtains the feature vector distance of different sample pairs; combines the above-mentioned high-dimensional features, uses the positioning module and the classification module to obtain heat maps for different images respectively, and uses the heat map association constraint to achieve the effect of mutual association and promotion between the detection and positioning tasks, and finally obtains the detection results and the specific lesion location detected.

[0063] like Figure 1 and Figure 2 As shown, a medical image anomaly detection and location method based on contrastive learning provided by an embodiment of the present invention includes the following steps S1 to S5:

[0064] S1. Obtain medical images;

[0065] S2. Constructing and training a medical image anomaly detection and localization model; the medical image anomaly detection and localization model includes a feature extraction network, a medical image anomaly detection network, and a medical image lesion localization network;

[0066] S3, using the feature extraction network to extract features from different angles of the medical image, and obtaining fusion features including overall features and lesion edge features;

[0067] S4, using the medical image anomaly detection network to generate a medical image category prediction probability according to the fusion features, and obtaining a medical image anomaly detection result;

[0068] S5. Use the medical image lesion localization network to generate a central heat map based on the fusion features, predict the size regression value of the target and the offset of the center point, and obtain the medical image lesion localization result.

[0069] First, in order to better represent the features, this embodiment not only extracts the features in the medical image, but also extracts the details and edge information in the image by calculating the spectral residual, so as to describe the features in the medical image from different angles. Secondly, the heat map of the detection network in the latent space is extracted by the activation mapping algorithm, and gradually approaches the heat map obtained by the positioning network, so that the detection network pays more attention to the location of the disease, and achieves the effect of mutual correlation and promotion between the detection and positioning tasks. Finally, the detection network and the positioning network that share the feature extraction network are constructed to perform the detection and positioning tasks simultaneously. Under the twin network framework, by comparing and learning the features of the positive and negative sample pairs, the same categories are close to each other in the feature space, and different categories are far away from each other in the feature space. This scheme uses the feature extraction network to fully combine the professional knowledge in the medical field to construct rich medical image features; because the detection and positioning models share the feature extraction network, the use of the twin network can make a clearer distinction between normal and abnormal features in the feature space; through the mutual constraints of the heat maps of the classification network and the positioning network, the correlation information of the two tasks is fully mined, thereby realizing effective medical image anomaly detection and positioning.

[0070] In an optional embodiment of the present invention, the feature extraction network includes:

[0071] Spectral residual calculation module and feature extraction backbone network;

[0072] The spectrum residual calculation module is used to perform spectrum residual calculation on the medical image to generate a saliency map;

[0073] The feature extraction backbone network is used to extract fusion features including overall features and lesion edge features according to the medical image and the saliency map.

[0074] The spectral residual calculation module is used to perform spectral residual calculation on the medical image to generate a saliency map, including:

[0075] Perform Fourier transform on medical images to obtain frequency domain representation of the image;

[0076] Taking the logarithm of the amplitude part of the image frequency domain representation to generate a logarithmic amplitude spectrum; then smoothing the logarithmic amplitude spectrum to generate a smoothed logarithmic amplitude spectrum;

[0077] Calculate the difference between the logarithmic amplitude spectrum and the smoothed logarithmic amplitude spectrum to obtain the spectrum residual;

[0078] Obtain the phase angle of the image frequency domain representation to obtain the phase spectrum;

[0079] Reconstruct the spectral residual and phase spectrum to generate a new image frequency domain representation;

[0080] The new image frequency domain representation is inverse Fourier transformed to generate a saliency map.

[0081] The feature extraction backbone network is used to extract fusion features including overall features and lesion edge features according to the medical image and the saliency map, including:

[0082] Multiple residual blocks containing multi-layer convolution structures are used to perform residual calculations on medical images and saliency maps to extract fusion features containing overall features and lesion edge features.

[0083] This embodiment takes the detection and location of lung diseases in chest X-rays as an example. The goal of the feature extraction network is to effectively extract the features of the input image and process the input medical image X to obtain a fusion feature h containing the overall features and the lesion edge features. The feature extraction network E consists of two parts, namely the spectral residual calculation module and the feature extraction backbone network.

[0084] Spectral residual is a method for extracting abnormal or significant features from signals, which can effectively capture the changing patterns in the data. Spectral residual is applied to feature extraction of chest X-rays. By analyzing the frequency domain information of the image, potential abnormal information in the lung area can be identified. The feature extraction backbone network is composed of residual blocks of the ResNet50 network. The residual block effectively avoids network performance degradation by directly connecting the input to the output, and reduces the possibility of gradient vanishing and gradient explosion during the network training process.

[0085] In the spectral residual calculation module, the chest X-ray image X is input into the feature extraction network E. After the input image X passes through the spectral residual module S, the saliency map h is obtained. s. Specifically, first perform Fourier transform on the input two-dimensional image X, convert the image from the spatial domain to the frequency domain, and obtain the frequency domain representation of the image. Take the logarithm of the amplitude part of the Fourier transform result to highlight the subtle changes in the spectrum, and then smooth the logarithm to generate a smoothed logarithmic spectrum graph. Calculate the difference between the original logarithmic amplitude spectrum and the smoothed logarithmic amplitude spectrum to obtain the spectral residual r. Combine the spectral residual r with the original phase spectrum P(X) to generate a new frequency domain representation. Perform an inverse Fourier transform on the reconstructed spectrum, convert the frequency domain result back to the spatial domain, and obtain the saliency map h s , expressed as:

[0086] h s =S(X)

[0087]

[0088] in, represents Fourier transform, log represents logarithm, represents smoothing, f arg It means to find the phase angle, represents the inverse Fourier transform.

[0089] In the feature extraction backbone network, the image X and the saliency map h s After inputting the feature extraction backbone network R, the image representation h in the high-dimensional feature space is obtained:

[0090] h=R(X,h s )

[0091] Among them, the feature extraction backbone network R contains multiple residual blocks, each of which contains a multi-layer structure. For a residual block containing an m-layer convolution structure, the feature calculation process in the residual block can be expressed as:

[0092]

[0093] h i =F act (h i，m +h i-1 )

[0094] Among them, Conv is the convolution layer, BN i，j is the jth batch normalization layer in the i-th residual block, f act is the ReLU activation function, h i is the hidden feature output by the i-th residual block. When i is 1, h i-1 is the original image data.

[0095] After the above process, the chest X-ray image X is input to obtain the output h of the feature extraction network E. Where H′×W′ is the spatial dimension and C is the number of channels.

[0096] In an optional embodiment of the present invention, a medical image anomaly detection network is used to generate a medical image category prediction probability according to fusion features to obtain a medical image anomaly detection result, including:

[0097] A global average pooling layer is used to average all values ​​of each channel of the fusion feature to obtain a global feature representation;

[0098] Three fully connected layers are used to generate medical image category prediction probabilities based on global feature representation to obtain medical image anomaly detection results.

[0099] The high-dimensional feature h is passed through the anomaly detection network D to obtain the output P = {p1, p2…p c}, p i Represents the probability that the predicted image belongs to class i.

[0100] P=D(h)

[0101] The medical image anomaly detection network consists of a global average pooling layer and three fully connected layers. After inputting the high-dimensional feature h extracted by the feature extraction network, all values ​​of each channel of h are averaged to obtain a vector h of length C. GAP , which is the representation of global features.

[0102]

[0103] Among them, h(i, j, c) represents the value of the feature vector h at the position (i, j, c), and c is the channel.

[0104] In the next three fully connected layers, a i Represents the output of the i-th fully connected layer:

[0105] a i =f act (W i ·a i-1 +b i )

[0106] Among them, the input of the first fully connected layer is h GAP , W i is the weight of the i-th fully connected layer, b i is the bias of the i-th fully connected layer, f act is the activation function, the first two layers are ReLU, and the last layer is Sigmoid.

[0107] In an optional embodiment of the present invention, a medical image lesion localization network is used to generate a center heat map according to fusion features, predict the size regression value of the target and the offset of the center point, and obtain a medical image lesion localization result, including:

[0108] The fused features are passed through the first convolutional layer and then the activation function is calculated to obtain the central heat map;

[0109] The fused features are passed through the second convolutional layer to calculate the target size corresponding to the local maximum position in the central heat map, and the size regression value of the predicted target is obtained;

[0110] The fused features are passed through the third convolutional layer to calculate the offset of the predicted target center point relative to the nearest grid point to obtain the offset of the center point;

[0111] The predicted center point position is calculated based on the center point offset.

[0112] The medical image lesion localization network is based on CenterNet for detection tasks. Unlike traditional two-stage object detectors, CenterNet is a single-stage object detector that is faster and does not rely on anchor boxes. The model directly regresses the target center, target size, and target center offset on the feature map to achieve target detection. This module takes the output h of the feature extraction network as input and generates the center heat map H, the size regression value S of the predicted target, and the offset O of the center point.

[0113] Center heat map , where H′×W′ is the spatial size of the feature map and C is the number of categories. The heat map value of each category represents the probability that the pixel position is the center of a certain type of target. The feature h passes through a convolution layer with a convolution kernel size of 3×3, and then is calculated by the activation function sigmoid to obtain the center heat map H.

[0114] H=f act (Conv(h))

[0115] Among them, f act is the activation function Sigmoid.

[0116] The size regression value S includes the width and height of the predicted target, and the output size regression value of each target The feature h passes through a convolution layer with a convolution kernel size of 3×3 and an output channel number of 2 to obtain the size regression value S. The target size corresponding to the local maximum position (i, j) of the heat map H is:

[0117] S = Conv(h)

[0118] S i，j =(width,height)

[0119] Among them, width and height are the width and height of the target respectively.

[0120] Since the resolution of the feature map extracted by the feature extraction network is lower than that of the input image, there is an offset between the predicted target center point and the center point of the original image. The feature h passes through a convolution layer with a convolution kernel size of 3×3 and an output channel number of 2 to obtain the center point offset O. Center point offset Express the offset of the center point relative to the nearest grid point:

[0121] O=Conv(h)

[0122]

[0123] Among them, x c and c is the coordinate of the center point of the target in the original image, Indicates rounding the downsampling factor.

[0124] According to the above output, the coordinates x′ of the predicted center point can be calculated c , y′ c :

[0125] x′ c =(i+O i，j [0])×s

[0126] y′ c =(j+O i，j [1])×s

[0127] Among them, (i, j) is the local maximum position of the heat map H, O i，j [0] is the horizontal offset, O i，j [1] is the vertical offset and s is the downsampling factor.

[0128] In an optional embodiment of the present invention, when training a medical image anomaly detection and localization model, the detection and localization module is first pre-trained so that the model has preliminary capabilities for anomaly detection and localization of medical images, and then the entire model is end-to-end trained.

[0129] In this embodiment, when pre-training the medical image anomaly detection and positioning model,

[0130] The training data is input into the medical image anomaly detection network and the medical image lesion localization network to obtain the detection results and localization results respectively;

[0131] Calculate the detection loss based on the detection results and the actual labels;

[0132] Calculate the positioning loss based on the positioning result and the actual positioning label;

[0133] Calculate the pre-training loss based on the detection loss and the localization loss;

[0134] Pre-train the medical image anomaly detection and localization model based on the pre-training loss.

[0135] This embodiment uses training data to pre-train the detection and positioning model, so that the model has preliminary detection and positioning capabilities for lung diseases, which is convenient for end-to-end training, and the accuracy of the heat map generated by the classification module is improved based on the heat map of the positioning module. Among them, the detection model and the positioning model share the feature encoding network E.

[0136] Input training data is X train , whose corresponding label is y train , after inputting the input data into the detection network and the positioning network, the detection results are obtained respectively Positioning results

[0137] Calculate the detection loss based on the detection results and the actual labels:

[0138]

[0139] in, To detect the loss, the cross entropy loss function is used as the specific implementation.

[0140] Calculate the positioning loss based on the positioning result and the actual positioning label:

[0141]

[0142] in, is the positioning loss function.

[0143] Finally, the detection loss and positioning loss are combined to obtain the overall pre-training loss:

[0144] loss = λ d ×loss d +λ l ×loss l

[0145] Among them, λ d and λ l The default value of both is 1.

[0146] The model is pre-trained based on the loss, so that the model has the initial ability to detect and locate lung diseases.

[0147] In this embodiment, when the medical image anomaly detection and localization model is pre-trained and then end-to-end trained,

[0148] Randomly sample from the training data to obtain positive samples that belong to the same category as the training data and negative samples that belong to different categories from the training data to construct positive sample pairs and negative sample pairs;

[0149] Input the positive sample pairs and the negative sample pairs into the twin network of the feature extraction network shared by the medical image anomaly detection network and the medical image lesion localization network to obtain training feature representation, positive sample feature representation and negative sample feature representation;

[0150] Compute contrastive loss based on training feature representation, positive sample feature representation, and negative sample feature representation;

[0151] Calculate the similarity loss between the center heat map output by the medical image anomaly detection network based on the training data and the target center heat map;

[0152] Calculate the end-to-end loss based on contrast loss, similarity loss, detection loss, and localization loss;

[0153] The medical image anomaly detection and localization model is trained end-to-end based on the end-to-end loss.

[0154] like Figure 3 As shown in the figure, in the end-to-end training stage, samples are first randomly selected from the training data of this stage, and positive sample pairs and negative sample pairs are constructed for the selected samples. Through contrastive learning, the model has a stronger ability to distinguish normal data from abnormal data; then, the heat map generated by the positioning network is used as the target, so that the heat map obtained by the detection network through the activation mapping algorithm is close to it; reducing the detection error and positioning error is taken as a component of the optimization target. In this way, the detection and positioning capabilities of the model can be improved while avoiding a large deviation in the feature space.

[0155] Input chest X-ray image is X train , whose corresponding label is y train First, randomly sample from the training data to obtain train Samples belonging to the same category With X train Samples belonging to different categories To construct positive sample pairs and negative sample pairs.

[0156] X train , Input the model separately to get the feature representation h train , And on this basis, the contrast loss is calculated:

[0157]

[0158] loss cont =max(0, d p -d n +α)

[0159] Here, α is a hyperparameter.

[0160] Make the detection model based on X train Exported heatmap heatmap d Close to the target detection model X train Calculated heatmap heatmap l , the corresponding similarity loss is:

[0161] loss sim =f Distance (heatmap d , heatmap l )

[0162] Among them, f Distance The specific implementation is:

[0163] f Distance (heatmap d , heatmap l )=1-SSIM(heatmap d , heatmap l )

[0164] Among them, SSIM is the structural similarity index.

[0165] In addition, according to X train Calculate the detection loss loss′ d and positioning loss loss′ l .

[0166] Finally, the loss of this end-to-end training phase consists of the above 4 parts, namely:

[0167] loss′=λ cont ×loss cont +λ sim ×loss sim +λ′ d ×loss′ d +λ′ l ×loss′ l

[0168] Among them, λ cont , sim ,λ′ d ,λ′ l The default value of both is 1.

[0169] The trained model can be used to detect medical image anomalies on the chest X-ray to be tested, and output the location information of possible lesions. The specific detection process can be divided into the following three sub-steps:

[0170] The chest X-ray image to be tested is input, and the constructed model is used to obtain the probability output of medical image abnormality detection, so as to evaluate whether the patient whose chest X-ray is to be tested has a disease.

[0171] A judgment is made based on the set threshold and the obtained probability output. If the probability score calculated by the detection module is greater than the threshold, it is considered that the chest X-ray contains the pathological characteristics of the disease. Otherwise, it is considered that there are no disease-related symptoms in the chest X-ray.

[0172] If the test results show abnormalities, the output results of the positioning module will be used as the lesion location to mark on the chest X-ray image to help doctors intuitively analyze the lesion area and provide a basis for subsequent treatment decisions.

[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0177] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A medical image anomaly detection and localization method based on contrastive learning, characterized in that: The following steps are involved: Acquiring medical images; Constructing and training a medical image anomaly detection and localization model; the medical image anomaly detection and localization model includes a feature extraction network, a medical image anomaly detection network, and a medical image lesion localization network; The feature extraction network is used to extract features from medical images from different angles to obtain fusion features including overall features and lesion edge features; The medical image anomaly detection network is used to generate the medical image category prediction probability according to the fusion features to obtain the medical image anomaly detection result; The medical image lesion localization network is used to generate a central heat map based on the fusion features, predict the size regression value of the target and the offset of the center point, and obtain the medical image lesion localization result.

2. The method for detecting and locating anomalies in medical images based on contrastive learning according to claim 1, characterized in that: The feature extraction network includes: Spectral residual calculation module and feature extraction backbone network; The spectrum residual calculation module is used to perform spectrum residual calculation on the medical image to generate a saliency map; The feature extraction backbone network is used to extract fusion features including overall features and lesion edge features according to the medical image and the saliency map.

3. The method for detecting and locating medical image anomalies based on contrastive learning according to claim 2, characterized in that: The spectral residual calculation module is used to perform spectral residual calculation on the medical image to generate a saliency map, including: Perform Fourier transform on medical images to obtain frequency domain representation of the image; Taking the logarithm of the amplitude part of the image frequency domain representation to generate a logarithmic amplitude spectrum; then smoothing the logarithmic amplitude spectrum to generate a smoothed logarithmic amplitude spectrum; Calculate the difference between the logarithmic amplitude spectrum and the smoothed logarithmic amplitude spectrum to obtain the spectrum residual; Obtain the phase angle of the image frequency domain representation to obtain the phase spectrum; Reconstruct the spectral residual and phase spectrum to generate a new image frequency domain representation; The new image frequency domain representation is inverse Fourier transformed to generate a saliency map.

4. The method for detecting and locating anomalies in medical images based on contrastive learning according to claim 2, characterized in that: The feature extraction backbone network is used to extract fusion features including overall features and lesion edge features according to the medical image and the saliency map, including: Multiple residual blocks containing multi-layer convolution structures are used to perform residual calculations on medical images and saliency maps to extract fusion features containing overall features and lesion edge features.

5. The method for detecting and locating anomalies in medical images based on contrastive learning according to claim 1, characterized in that: The medical image anomaly detection network is used to generate the medical image category prediction probability based on the fusion features, and the medical image anomaly detection results are obtained, including: A global average pooling layer is used to average all values ​​of each channel of the fusion feature to obtain a global feature representation; Three fully connected layers are used to generate medical image category prediction probabilities based on global feature representation to obtain medical image anomaly detection results.

6. The method for detecting and locating medical image anomalies based on contrastive learning according to claim 1, characterized in that: The medical image lesion localization network is used to generate a central heat map based on the fusion features, predict the size regression value of the target and the offset of the center point, and obtain the medical image lesion localization results, including: The fused features are passed through the first convolutional layer and then the activation function is calculated to obtain the central heat map; The fused features are passed through the second convolutional layer to calculate the target size corresponding to the local maximum position in the central heat map, and the size regression value of the predicted target is obtained; The fused features are passed through the third convolutional layer to calculate the offset of the predicted target center point relative to the nearest grid point to obtain the offset of the center point; The predicted center point position is calculated based on the center point offset.

7. The method for detecting and locating medical image anomalies based on contrastive learning according to claim 1, characterized in that: When pre-training the medical image anomaly detection and localization model, The training data is input into the medical image anomaly detection network and the medical image lesion localization network to obtain the detection results and localization results respectively; Calculate the detection loss based on the detection results and the actual labels; Calculate the positioning loss based on the positioning result and the actual positioning label; Calculate the pre-training loss based on the detection loss and the localization loss; Pre-train the medical image anomaly detection and localization model based on the pre-training loss.

8. The method for detecting and locating medical image anomalies based on contrastive learning according to claim 7, characterized in that: The pre-training loss is: loss=λ d ×loss j +λ l ×loss l Among them, loss is the pre-training loss, λ d is the detection loss weight coefficient, loss d is the detection loss, λ l is the positioning loss weight coefficient, loss l For positioning loss.

9. The method for detecting and locating medical image anomalies based on contrastive learning according to claim 8, characterized in that: When the medical image anomaly detection and localization model is pre-trained and then trained end-to-end, Randomly sample from the training data to obtain positive samples that belong to the same category as the training data and negative samples that belong to different categories from the training data to construct positive sample pairs and negative sample pairs; Input the positive sample pairs and the negative sample pairs into the twin network of the feature extraction network shared by the medical image anomaly detection network and the medical image lesion localization network to obtain training feature representation, positive sample feature representation and negative sample feature representation; Compute contrastive loss based on training feature representation, positive sample feature representation, and negative sample feature representation; Calculate the similarity loss between the center heat map output by the medical image anomaly detection network based on the training data and the target center heat map; Calculate the end-to-end loss based on contrast loss, similarity loss, detection loss, and localization loss; The medical image anomaly detection and localization model is trained end-to-end based on the end-to-end loss.

10. The method for detecting and locating medical image anomalies based on contrastive learning according to claim 9, characterized in that: The end-to-end loss is: loss′=λ cont ×loss cont +λ sim ×loss sim +λ′ d ×loss′ d +λ′ l ×loss′ l Among them, loss′ is the end-to-end loss, λ cont is the comparison loss weight coefficient, loss cont is the contrast loss, λ sim is the similarity loss weight coefficient, loss sim is the similarity loss, λ′ d is the detection loss weight coefficient during end-to-end training, loss′ d is the detection loss during end-to-end training, λ′ l is the positioning loss weight coefficient during end-to-end training, loss′ l Localization loss for end-to-end training.

Images