Medical image artifact classification method based on CN-CLIP model

Abstract

The invention relates to a medical image artifact classification method based on a CN-CLIP model, and the method comprises the steps: obtaining a medical image data set containing a plurality of CT images and MR images, and constructing a text label data set in a multi-segment cue word format based on the image type, examination part and artifact type of each image; performing fine tuning on the CN-CLIP pre-training model by using the medical image data set and the text label data set to obtain an artifact classification model; and inputting a medical image to be classified into the artifact classification model, and outputting an artifact classification result of the medical image. According to the medical image artifact classification method based on the CN-CLIP model, the pre-trained Chinese image-text matching model CN-CLIP is utilized, so that the defect that models need to be trained respectively for different image types and examination parts in the prior art is overcome; efficient and accurate artifact classification of CT and MR medical images of different parts is realized through a unified model.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a medical image artifact classification method based on a CN-CLIP model. Background Art

[0002] In the field of modern medicine, medical imaging technology is undergoing rapid changes. Among them, computed tomography (CT) and magnetic resonance imaging (MR) have rapidly emerged in the field of disease diagnosis and have become the preferred method for diagnosing many diseases due to their non-invasive examination methods and the high-quality, high-resolution images they can provide. The breakthrough progress of these technologies has not only significantly improved the clarity of medical images, allowing doctors to more accurately observe and analyze the microstructure and even functional information of patients, thereby improving the accuracy of diagnosis; moreover, with the maturity and popularization of technology, the cost of CT and MR examinations has also been greatly reduced, making them more affordable, and widely used in various medical institutions, significantly improving the quality and efficiency of medical services.

[0003] However, despite the great progress made in medical imaging technology, the problems that come with it cannot be ignored. That is, with the continuous growth of clinical demand for medical imaging, especially in large medical institutions, the work pressure of imaging technicians has also increased dramatically. They need to complete a large number of scanning and inspection work in an increasingly shortened time. This high-intensity, fast-paced work mode often makes it difficult for them to fully devote their energy during the image acquisition process and to take into account all details and standardized operations. This busy working state increases the risk of human negligence, such as insufficient explanation of precautions for patients and inaccurate breathing training, which may become potential factors for the generation of image artifacts. In addition, increasingly complex scanning sequences and reconstruction algorithms may also introduce new types of artifacts, increasing the difficulty of artifact identification.

[0004] Image artifacts are a common phenomenon in medical images. Their appearance can interfere with or even cover up the true anatomical structure and pathological information. If the artifact happens to block the lesion area in the image, the doctor may miss important early diagnostic information and make wrong diagnosis and treatment decisions, which will not only delay the best treatment time for the patient and increase the financial burden on the patient, but may also lead to worsening of the disease, causing unnecessary pain and loss to the patient, and even endangering his life in severe cases. Therefore, accurate identification and effective removal of artifacts are crucial to ensuring the quality of medical images.

[0005] At present, for the detection of medical image artifacts, the existing technologies are mostly limited to independently detecting artifacts for different commonly used types of images and images of different scanning parts, which means that different artifact detection models need to be trained and deployed for each image type and each scanning part, which is not only inefficient but also has high model management and maintenance costs. In addition, since image artifacts usually present fuzzy and irregular edge features, it is difficult to accurately manually annotate the boundaries of the artifact area, which directly leads to the difficulty in obtaining a large number of artifact classification data sets with high label accuracy and balanced data type distribution, which in turn seriously affects the training effect and generalization ability of the artifact classification model based on deep learning. Especially in the absence of a unified, cross-modal, and cross-site artifact classification model, doctors and technicians often need to rely on experience to make judgments, which is inefficient and highly subjective, and it is difficult to meet the growing demand for medical image quality control. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a medical image artifact classification method based on the CN-CLIP model for realizing efficient, accurate and low-cost medical image artifact classification.

[0007] In order to achieve the above-mentioned purpose, the present invention designs a medical image artifact classification method based on the CN-CLIP model.

[0008] The following steps are involved:

[0009] S100. Acquire a medical image dataset P including a plurality of CT images and MR images, wherein each image has corresponding examination site information;

[0010] S200. Constructing a text label dataset L in a multi-segment prompt word format based on the image type, examination site and artifact type of each image;

[0011] S300. Fine-tune the CN-CLIP pre-trained model using the medical image dataset P and the text label dataset L to obtain an artifact classification model;

[0012] S400. Input the medical image to be classified into the artifact classification model, and output the artifact classification result of the medical image.

[0013] Preferably, the step of acquiring the medical image data set P includes: collecting DICOM data of CT images and MR images; reading tag information in each DICOM data to obtain corresponding window width and window level values; normalizing the DICOM data matrix according to the window width and window level values ​​and converting it into a jpg format image.

[0014] Preferably, the step of labeling the text data set L includes: using a preset multi-segment prompt word label format to perform artifact classification annotation on each image in the medical image data set; the multi-segment prompt word label includes image type, examination site and artifact type.

[0015] Preferably, the step of adjusting the CN-CLIP model comprises:

[0016] S301. Randomly extract images with a batch value of M from the medical image dataset P, crop and scale them to a fixed size, and then input them into the image encoding model of the CN-CLIP model to obtain the image batch feature BP feat =[I1,...I i ,...,I M ], where i∈{1,...,M};

[0017] S302. Extract the text label with batch value M corresponding to the image batch from the text label dataset L and input it into the text encoding model of the CN-CLIP model to obtain the text batch feature BL feat =[T1,...T i ,...,T M ];

[0018] S303. Image feature vector I i and text feature vector T i Perform L2 norm normalization to unit length;

[0019] S304. Image feature vector I normalized by dot product operation i and the normalized text feature vector T i The similarity matrix between

[0020] S305. Calculate the contrast loss based on the similarity matrix, and use the optimizer to minimize the contrast loss to update the parameters of the CN-CLIP model.

[0021] Preferably, the calculation formula of the similarity matrix is:

[0022] S ij =s×I i ×Tj T ;

[0023] Among them, S ij represents the similarity between the i-th image feature vector I and the j-th text feature vector T in the batch, j∈{1,...,M}; s is a learnable scaling factor.

[0024] Preferably, each row of the similarity matrix S is converted into a probability distribution by a softmax function:

[0025]

[0026] Among them, P ij Represents the relative probability value between the i-th image feature vector and the j-th text feature vector.

[0027] Preferably, the calculation formula for calculating the contrast loss is:

[0028]

[0029] Among them, P img→txt Represents the probability distribution matrix of similarity from image to text, P txt→img represents the probability distribution matrix of similarity from text to image, CE represents the cross entropy loss for calculating multiple classifications; img is the target index of the image feature, and y img =[1,...,M]; y txt is the target index of the text feature, and y txt =[1,...,M].

[0030] Preferably, the step of outputting the artifact classification result of the medical image includes: inputting the medical image to be classified into the image encoding model of the trained artifact classification model to obtain an image feature vector; obtaining the deduplicated candidate text labels from the text label data set; inputting the candidate text labels into the text encoding model to obtain candidate text features; calculating the similarity between the image feature vector and the candidate text features; and selecting the text label corresponding to the maximum similarity value as the classification result.

[0031] Preferably, the optimizer is an AdamW optimizer.

[0032] Preferably, the examination sites include the head and neck, chest and abdomen, pelvis and limbs; the artifact types of CT images include normal artifact-free, motion artifact, foreign body artifact and detector artifact; the artifact types of MR images include normal artifact-free, fold artifact, magnetic susceptibility artifact, chemical shift artifact, eccentric artifact, dielectric artifact, radio frequency interference artifact, vascular pulsation artifact, metal foreign body artifact, motion artifact, parallel acquisition related artifact, diffusion imaging deformation artifact in the phase encoding direction, peripheral signal artifact, truncation artifact and cerebrospinal fluid flow artifact.

[0033] The medical image artifact classification method based on the CN-CLIP model designed by the present invention utilizes the pre-trained Chinese image-text matching model CN-CLIP, overcomes the deficiency in the related art that models need to be trained separately for different image types and examination parts, and realizes efficient and accurate artifact classification of CT and MR medical images of different parts with a unified model; its multi-segment prompt word label construction method only needs to label each image with corresponding multi-segment prompt word labels, and does not need to mark specific artifact areas, which greatly improves the data marking speed, can relatively quickly form a large-scale artifact classification data set, significantly improves the efficiency of artifact classification, and provides a more efficient and accurate means for quality control of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a medical image artifact classification method based on the CN-CLIP model provided in an embodiment of the present application.

[0035] Figure 2 It is a schematic diagram of the artifact classification model training process provided in an embodiment of the present application.

[0036] Figure 3 It is a schematic diagram of the artifact classification reasoning process of medical images provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0038] This embodiment describes a medical image artifact classification method based on the CN-CLIP (Contrastive Language-Image Pre-training in Chinese, CN-CLIP) model. The method uses the pre-trained CN-CLIP model and combines multiple prompt word labels to achieve automatic classification of artifacts in medical images.

[0039] like Figure 1 As shown, the method comprises the following steps:

[0040] S100. Acquire a medical image dataset P including a plurality of CT images and MR images, wherein each image has corresponding examination site information.

[0041] In a specific embodiment, the step of acquiring a medical image data set P includes: collecting DICOM data of CT images and MR images; reading tag information in each DICOM data to obtain corresponding window width and window position values; normalizing the DICOM data matrix according to the window width and window position values ​​and converting it into a jpg format image.

[0042] Specifically, DICOM data of different image types (CT, MR) and different parts (head, chest, abdomen, pelvis, etc.) are collected from medical imaging equipment or PACS (Picture Archiving and Communication System) and other systems to obtain a DICOM data set D = {D1, D2, D3, ... D i, ...,D N}, D i It is represented as the DICOM data of the i-th medical image in the dataset, i∈{1,...,N}, N is the total number of DICOM data in the dataset D; then the i-th medical image DICOM data D i Read the tag information to obtain the default window width and window level values ​​displayed in the film reading software, and normalize the data matrix of the original image DICOM according to the window width and window level values ​​to convert it into a jpg format image P i , and so on, the entire data D is converted into an image data set P = {P1, P2, P3, ... P i ,...,P N}.

[0043] S200. Based on the image type, examination part and artifact type of each image, a text label dataset L in a multi-segment prompt word format is constructed.

[0044] In a specific embodiment, the step of labeling the text data set L includes: using a preset multi-segment prompt word label format to classify and annotate each image in the medical image data set; the multi-segment prompt word label includes image type, examination site and artifact type.

[0045] Specifically, it is determined that the image type is a CT image or an MR image according to the image data, and the examination part of the image is determined according to the examination part information obtained in step S100, and then the artifact type of the image is determined according to the manual annotation or predefined artifact type list (such as normal artifact-free, motion artifact, metal foreign body artifact, detector artifact, fold artifact, magnetic susceptibility artifact, chemical shift artifact, eccentric artifact, etc.), and finally, a text label L in a multi-segment prompt word format is constructed according to the image type, examination part and artifact type. i For example, the text label of a CT image can be expressed as "a motion artifact image of a CT chest and abdomen examination", until all images in the medical image dataset P are processed, and finally a text label dataset L is obtained, where L = {L1, L2, ..., L N}, N is the total number of medical images. In a specific embodiment, the multiple prompt word label texts are organized in a fixed order of 'an {image type} {examination part} examination {artifact type} image'.

[0046] In some embodiments, the examination sites include the head and neck, chest and abdomen, pelvis and limbs; the artifact types of CT images include normal artifact-free, motion artifact, foreign body artifact and detector artifact; the artifact types of MR images include normal artifact-free, fold artifact, magnetic susceptibility artifact, chemical shift artifact, eccentric artifact, dielectric artifact, radio frequency interference artifact, vascular pulsation artifact, metal foreign body artifact, motion artifact, parallel acquisition related artifact, diffusion imaging deformation artifact in the phase encoding direction, peripheral signal artifact, truncation artifact and cerebrospinal fluid flow artifact.

[0047] S300. Fine-tune the CN-CLIP pre-trained model using the medical image dataset P and the text label dataset L to obtain an artifact classification model.

[0048] In a specific embodiment, Figure 2 As shown, the steps to adjust the CN-CLIP model include:

[0049] S301. Randomly extract images with a batch value of M from the medical image dataset P, crop and scale them to a fixed size (e.g., 336x336), and then input them into the image encoding model of the CN-CLIP model, i.e., the image encoder, to obtain the image batch feature BP feat =[I1,...I i ,...,I M ], where i∈{1,...,M}. In a specific implementation, the image coding model uses the VIT-L / 14@336px network.

[0050] S302. The text label with the batch value M corresponding to the image batch extracted from the text label dataset L is input into the text encoding model of the CN-CLIP model, that is, the text encoder, to obtain the text batch feature BL feat =[T1,...T i ,...,T M In a specific implementation, the text encoding model uses the RoBERTa-wwm-Base network.

[0051] S303. Image feature vector I i and text feature vector T i The L2 norm (Euclidean norm) is normalized to unit length. That is, ||I i ||2=1 and ||T i ||2=1.

[0052] S304. Image feature vector I normalized by dot product operation i and the normalized text feature vector T i The similarity matrix between .

[0053] In this embodiment, the calculation formula of the similarity matrix is:

[0054] S ij =s×I i ×Tj T ;

[0055] Among them, S ij represents the similarity between the i-th image feature vector I and the j-th text feature vector T in the batch, j∈{1,...,M}; s (i.e., logit_scale) is a learnable scaling factor, so that an M×M similarity matrix S can be constructed.

[0056] Furthermore, each row of the similarity matrix S is converted into a probability distribution through the softmax function:

[0057]

[0058] Among them, P ij Represents the relative probability value between the i-th image feature vector and the j-th text feature vector. Through the above calculation, a new matrix P can be obtained, in which each row is a valid probability distribution, representing the relative probability between the image and all text feature vectors in the batch. In this embodiment, for each image feature (or text feature), its target category index is its own index, that is, for an image, the target index is y img =[1,...,M]; for text, it is also y txt =[1,...,M].

[0059] S305. Calculate the contrast loss (Contrastive Loss), i.e., the symmetric cross entropy loss, based on the similarity matrix, and minimize the contrast loss using an optimizer to update the parameters of the CN-CLIP model. In this embodiment, the optimizer is an AdamW optimizer, i.e., the AdamW optimizer is used to fine-tune the artifact classification model using the contrast loss to obtain an optimized artifact classification model.

[0060] Specifically, the formula for calculating contrast loss is:

[0061]

[0062] Among them, P img→txt Represents the probability distribution matrix of similarity from image to text, P txt→img represents the probability distribution matrix of similarity from text to image, CE represents the cross entropy loss for calculating multiple classifications; img is the target index of the image feature, and y img =[1,...,M]; ytxt is the target index of the text feature, and y txt =[1,...,M].

[0063] S400. Input the medical image to be classified into the artifact classification model, and output the artifact classification result of the medical image.

[0064] In a specific embodiment, the step of outputting the artifact classification result of the medical image includes: inputting the medical image to be classified into the image encoding model of the trained artifact classification model to obtain an image feature vector; obtaining the deduplicated candidate text labels from the text label data set; inputting the candidate text labels into the text encoding model to obtain candidate text features; calculating the similarity between the image feature vector and the candidate text features; and selecting the text label corresponding to the maximum similarity value as the classification result.

[0065] In the actual application of the artifact detection model, such as Figure 2 As shown in the figure, only the image encoder in the model is needed to classify artifacts in medical images, and the candidate text features can be calculated and saved as a matrix using the text encoder before deployment, making the inference speed faster. The specific process is as follows:

[0066] The newly acquired DICOM data of any part of the CT or MR examination is converted into a jpg image according to the window width and window position information in the tag. Then the image is cropped and scaled to a fixed size (such as 336*336) according to the artifact detection model training. Then, the image encoding model (such as VIT-L / 14@336px) in the trained artifact classification model is input to output the feature vector I corresponding to this image. Then, the text label data set L is deduplicated to obtain a new candidate text label list L', and then the text label list is input into the text encoding model (such as RoBERTa-wwm-Base) in the artifact classification model to output the corresponding candidate text label feature list TD=[T1,...,T ND ], ND represents the number of text labels remaining after deduplication from the text label dataset L. Then, the similarity between I and each element in TD is calculated to obtain the similarity list SD, where the text label with the largest similarity value corresponds to the index, which is the inference result of the algorithm model for this image. That is, by splitting the text label, we can know whether this image is CT or MR, as well as the specific examination site and artifact type.

[0067] For example, when a medical image to be classified is an MR pelvic examination and is normal without artifacts, its corresponding text label is "a normal, artifact-free image of an MR pelvic examination". Through the artifact classification of this embodiment, the text label can be accurately output, thereby obtaining information such as the image type, examination site, and artifact type of the medical image. Similarly, multi-segment prompt word text labels for all examination sites and artifact categories of CT and MR images can also be obtained.

[0068] The medical image artifact classification method based on the CN-CLIP model provided in this embodiment uses the pre-trained Chinese image-text matching model CN-CLIP, overcomes the deficiency of the related art that models need to be trained separately for different image types and examination parts, and realizes efficient and accurate artifact classification of CT and MR medical images of different parts with a unified model; its multi-segment prompt word label construction method only needs to label each image with corresponding multi-segment prompt word labels, and does not need to mark specific artifact areas, which greatly improves the data labeling speed, can relatively quickly form a large-scale artifact classification data set, significantly improves the efficiency of artifact classification, and provides a more efficient and accurate means for quality control of medical images.

[0069] In the description of the present invention, it should be noted that the terms "vertical", "up", "down", "horizontal", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0070] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0071] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A medical image artifact classification method based on the CN-CLIP model, characterized in that: The following steps are involved: S100. Acquire a medical image dataset P including a plurality of CT images and MR images, wherein each image has corresponding examination site information; S200. Constructing a text label dataset L in a multi-segment prompt word format based on the image type, examination site and artifact type of each image; S300. Fine-tune the CN-CLIP pre-trained model using the medical image dataset P and the text label dataset L to obtain an artifact classification model; S400. Input the medical image to be classified into the artifact classification model, and output the artifact classification result of the medical image.

2. The method for classifying medical image artifacts based on the CN-CLIP model according to claim 1, characterized in that: The steps of acquiring a medical image data set P include: collecting DICOM data of CT images and MR images; reading tag information in each DICOM data to obtain corresponding window width and window position values; normalizing the DICOM data matrix according to the window width and window position values ​​and converting it into a jpg format image.

3. The medical image artifact classification method based on the CN-CLIP model according to claim 2, characterized in that: The steps of the text label data set L include: using a preset multi-segment prompt word label format to classify and label each image in the medical image data set; the multi-segment prompt word label includes image type, examination site and artifact type.

4. The method for classifying medical image artifacts based on the CN-CLIP model according to claim 1, characterized in that: The steps to adjust the CN-CLIP model include: S301. Randomly extract images with a batch value of M from the medical image dataset P, crop and scale them to a fixed size, and then input them into the image encoding model of the CN-CLIP model to obtain the image batch feature BP feat =[I1,...I i ,...,I M ], where i∈{1,...,M}; S302. Extract the text label with batch value M corresponding to the image batch from the text label dataset L and input it into the text encoding model of the CN-CLIP model to obtain the text batch feature BL feat =[T1,...T i ,...,T M ]; S303. The image feature vector I i and text feature vector T i Perform L2 norm normalization to unit length; S304. Image feature vector I normalized by dot product operation i and the normalized text feature vector T i The similarity matrix between S305. Calculate the contrast loss based on the similarity matrix, and minimize the contrast loss using an optimizer to update the parameters of the CN-CLIP model.

5. The method for classifying medical image artifacts based on the CN-CLIP model according to claim 4, characterized in that: The calculation formula of the similarity matrix is: S ij =s×I i ×Tj T ; Among them, S ij represents the similarity between the i-th image feature vector I and the j-th text feature vector T in the batch, j∈{1,...,M}; s is a learnable scaling factor.

6. The method for classifying medical image artifacts based on the CN-CLIP model according to claim 5, characterized in that: Each row of the similarity matrix S is converted into a probability distribution through the softmax function: Among them, P ij Represents the relative probability value between the i-th image feature vector and the j-th text feature vector.

7. The method for classifying medical image artifacts based on the CN-CLIP model according to claim 6, characterized in that: The formula for calculating contrast loss is: Among them, P img→txt Represents the probability distribution matrix of image-to-text similarity, P txt→img represents the probability distribution matrix of similarity from text to image, CE represents the cross entropy loss for calculating multiple classifications; img is the target index of the image feature, and y img =[1,...,M]; y txt is the target index of the text feature, and y txt =[1,...,M].

8. The method for classifying medical image artifacts based on the CN-CLIP model according to any one of claims 4 to 7, characterized in that: The steps of outputting artifact classification results of medical images include: inputting the medical image to be classified into the image encoding model of the trained artifact classification model to obtain an image feature vector; obtaining the candidate text labels after deduplication from the text label data set; inputting the candidate text labels into the text encoding model to obtain candidate text features; calculating the similarity between the image feature vector and the candidate text features; and selecting the text label corresponding to the maximum similarity value as the classification result.

9. The method for classifying medical image artifacts based on the CN-CLIP model according to claim 4, characterized in that: The optimizer is the AdamW optimizer.

10. The method for classifying medical image artifacts based on the CN-CLIP model according to claim 1, characterized in that: The examination areas include the head and neck, chest and abdomen, pelvis and limbs; the artifact types of CT images include normal artifact-free, motion artifact, foreign body artifact and detector artifact; the artifact types of MR images include normal artifact-free, fold artifact, magnetic susceptibility artifact, chemical shift artifact, eccentric artifact, dielectric artifact, radio frequency interference artifact, vascular pulsation artifact, metal foreign body artifact, motion artifact, parallel acquisition related artifact, diffusion imaging deformation artifact in the phase encoding direction, peripheral signal artifact, truncation artifact and cerebrospinal fluid flow artifact.

Images