Breast X-ray image recognition method based on BEiT V2 model

Abstract

The invention discloses a chest X-ray image recognition method based on a BEiT V2 model, and the method employs a transfer learning principle, carries out the intelligent analysis of a chest X-ray image through a pre-trained BEiT V2 model, and achieves the quick and accurate recognition of a medical image. According to the method, the BEiT V2 model trained on the ImageNet-1k data set in advance is utilized, and fine adjustment is further performed through the specific chest X-ray image data set, so that the model can effectively learn features in the chest X-ray image. The technology not only accelerates the training process, but also significantly improves the accuracy and stability of identification. Experimental results show that the end-to-end chest X-ray image recognition model based on transfer learning provided by the invention achieves 97.78% of recognition accuracy, the excellent performance of the model in the chest X-ray image recognition aspect is proved, a powerful tool is provided for rapidly and accurately completing chest X-ray image recognition clinically, and the method is suitable for popularization and application. The method is helpful for assisting doctors in improving early judgment of related clinical symptoms, so that prognosis and medical quality of patients are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical image analysis, and in particular relates to a chest X-ray image recognition method based on the BEiT V2 model. Background Art

[0002] In the traditional chest X-ray image recognition detection process, doctors rely on experienced radiologists to make judgments by visually inspecting chest X-ray images. Although this method is effective to a large extent, its efficiency and accuracy are often limited when conducting large-scale screening or in remote areas with scarce medical resources. Especially in the face of sudden epidemics, manual judgment methods cannot meet the needs of rapid response, resulting in a slow and error-prone judgment process.

[0003] However, in recent years, with the rapid development of computer vision technology and deep learning, automated chest X-ray image recognition methods have gradually become a reality. These advanced technologies can not only significantly improve the efficiency of diagnosis and judgment, but also guarantee the accuracy of diagnosis and judgment to a certain extent. For example, through deep learning algorithms, computers can automatically identify and analyze abnormal features in X-ray images, so as to quickly determine whether there are abnormalities. These algorithms are trained with a large amount of data and can identify typical imaging manifestations of abnormal symptoms, such as ground-glass opacities and consolidation shadows.

[0004] Nevertheless, automated chest X-ray image abnormality detection methods still face some challenges. First, the imbalance of the dataset is an important obstacle. In practical applications, the number of healthy samples is often much larger than that of diseased samples, which may cause the model to favor healthy samples during training, thereby affecting its ability to identify abnormal cases of chest X-ray images. To solve this problem, researchers are trying to use data augmentation techniques to generate more diseased samples through methods such as rotation, scaling, and cropping to balance the dataset.

[0005] Secondly, the difficulty of feature extraction is also a key issue. Although deep learning technology performs well in image recognition, it is still a challenging task to extract features that can accurately reflect the abnormal characteristics of chest X-ray images in complex medical images. To solve this problem, researchers are exploring multimodal learning methods to provide more comprehensive and judgment information by combining X-ray images and other medical imaging data (such as CT scans) to help the model more accurately identify abnormalities in chest X-ray images. At the same time, using natural language processing technology to analyze patients' clinical records and symptom descriptions can also provide additional and judgment basis for the model. Summary of the invention

[0006] In order to solve the problems existing in the existing chest X-ray image abnormality detection and recognition, such as low accuracy, high misdiagnosis rate and high dependence on doctors' professional knowledge, the present invention provides a chest X-ray image recognition method based on the BEiT V2 model to improve the accuracy and efficiency of chest X-ray image abnormality and judgment.

[0007] The present invention provides a chest X-ray image recognition method based on the BEiT V2 model, which specifically includes the following steps:

[0008] Step 1: Collect images and preprocess them;

[0009] Step 2: Pre-train the model;

[0010] Step 3: Initialize the model;

[0011] Step 4: Perform machine learning and divide the data set;

[0012] Step 5: Extract features;

[0013] Step 6: Use the attention mechanism to enhance the intermediate extracted features;

[0014] Step 7: semantic analysis;

[0015] Step 8: Classification decision;

[0016] Step 9: Result verification;

[0017] Step 10: Model training and optimization;

[0018] Step 11: Output and save;

[0019] Save the trained model, including model weights, configuration files, etc., for subsequent use and deployment.

[0020] Furthermore, step 1 is specifically as follows:

[0021] The chest imaging data is collected by X-ray machine to generate high-quality chest images. Then, the images are preprocessed. First, the generated chest image files are loaded using SimpleITK medical image library. Then, all images are adjusted to a uniform size for easy input. Then, the images are normalized to scale the image pixel values ​​to between 0 and 1. The normalization operation, i.e. scaling the image pixel values ​​to between 0 and 1, can be expressed by the following formula:

[0022] Where I is the pixel value of the original image, I min and I max are the minimum and maximum pixel values, respectively, and I normalized is the normalized pixel value.

[0023] After normalization is completed, further standardization is performed to center the pixel values ​​of the image to zero mean and have unit variance, so that the data set has zero mean and unit variance, which is achieved through the following formula:

[0024]

[0025] Where I is the normalized pixel value, μ is the average value of all pixel values, σ is the standard deviation, and I standardized is the normalized pixel value.

[0026] Finally, a variety of transformations are applied to increase the diversity of the training set, including randomly selecting areas from the original image as new images, randomly flipping the image left and right, randomly flipping the image up and down, and randomly rotating the image by a certain angle. Finally, the processed images are organized into multiple batches, each batch containing multiple images and their labels, so as to obtain the dataset.

[0027] Furthermore, step 2 is specifically as follows:

[0028] The encoder first uses ViT (Vision Transformer) to encode the input image and convert it into a series of feature vectors. Then it finds the nearest neighbor of each feature vector from a codebook. This nearest neighbor is a visual word. This process can be done by calculating the feature h of each image patch. i With all sight words j The minimum cosine distance between them determines the corresponding visual word, and the formula is as follows:

[0029] z i =argmin j ||l2(h i )-l2(e j )||2

[0030] Where l2 represents the L2 regularization operation.

[0031] The decoder receives the L2-regularized visual word sequence as input and outputs a series of features of the reconstructed image patch.

[0032] Furthermore, step 3 is specifically as follows:

[0033] The pre-trained BEiT V2 model is used as the basis. First, import the required libraries and modules, then use BeitForImageClassification and BeitFeatureExtractor in the transformers library to load the pre-trained BEiT V2 model and its feature extractor as the initial model, and then modify the number of classification heads in the last layer of the BEiT V2 model to complete the model initialization.

[0034] Furthermore, step 4 is specifically as follows:

[0035] The entire dataset obtained in step 1 is divided into three parts: training set, validation set, and test set. First, all image file paths and corresponding labels are read into a list or DataFrame, and then the sample order in the dataset is shuffled using a random shuffle method to ensure that the sample order in the dataset is random to avoid potential order bias. Finally, the dataset is divided into training set, validation set, and test set according to a predetermined ratio. Subsequently, 90% of the data in the dataset is randomly used for model training and validation, and the remaining 10% is reserved for testing the final effect of the model.

[0036] Furthermore, step 5 is specifically as follows:

[0037] The preprocessed images are sent to the model in batches. The encoder of the model first divides the input image into multiple overlapping patches and maps these patches to an embedding vector through a linear layer. Then the model adds position encoding to each patch so that the model understands the relative position of the patch in the image. After the position encoding is added, the vector will pass through multiple Transformer encoder layers. The Transformer encoder layer of the model will encode the embedded vector. Each encoder layer includes multi-head self-attention (Multi-Head Self-Attention) and feed-forward neural network (Feed-Forward Network). After passing through the Transformer encoder layer, each patch gets a high-level feature representation, which is the corresponding feature of the final extracted image.

[0038] Furthermore, step 6 is specifically as follows:

[0039] After the previous series of processing, the input image is mapped into an input sequence, and then these input series will be mapped to different subspaces by the model, and then each subspace has its own attention head to focus on different features. The model will generate three linear transformations for the input sequence: query (Query, Q), key (Key, K) and value (Value, V), and then perform self-attention calculation on the input sequence to calculate the attention score of each position and all other positions. The output of the self-attention mechanism used here is the weighted aggregation of values, and the calculation formula is:

[0040]

[0041] After obtaining the attention score of the input series, the input sequence will be further transformed by the feedforward neural network. First, a regularization is performed, which is achieved using layer normalization. Layer normalization can ensure that the output of each layer has zero mean and unit variance, thereby stabilizing network training. The formula is expressed as:

[0042] Xnorm=LayerNorm(x)

[0043] Where x is the input sequence, Xnorm is the sequence after layer normalization, and LayerNorm(x) is the layer normalization function, defined as:

[0044]

[0045] where μ(x) is the mean of x, σ 2 (x) is the variance of x, γ and β are learnable parameters, and ∈ is a small constant for numerical stability.

[0046] Then, through a multi-layer perceptron, the representation ability of the input sequence can be further enhanced after passing through multiple layers of neurons. The multi-layer perceptron is a basic artificial neural network model and can be expressed as

[0047] FFN(A)=max(0,A×W1+b1)×W2+b2

[0048] Where A is the sequence after layer normalization, W1, W2 are weight matrices, b1, b2 are bias terms, max is used as the activation function, and FFN(A) represents the output of MLP.

[0049] Finally, a residual connection is used to help the training of deep networks by adding the input directly to the output of the MLP to prevent feature loss. The residual connection can be expressed as:

[0050] x′=xnorm+z

[0051] where x norm is the input sequence after layer normalization, z is the output of MLP, and x′ is the final output with residual connection.

[0052] Furthermore, step 7 is specifically as follows:

[0053] After waiting for feature extraction to complete, the model is used to perform semantic analysis operations. First, the feature vectors of each patch are fused to obtain a global feature representation of the entire image to identify and distinguish chest X-ray abnormalities and other pathological features. Then, visualization tools (such as Grad-CAM) are used to highlight important areas in the image to observe which areas contribute more to the model's decision. Based on the results of the semantic analysis, the area that best represents the abnormal features of the chest X-ray image is selected to provide a basis for the final classification decision.

[0054] Furthermore, step 8 is specifically as follows:

[0055] Based on the extracted and enhanced feature vectors, a pre-trained classifier is used to determine whether the image contains abnormal signs of chest X-ray images. If there are abnormal signs of chest X-ray images, proceed to the next step; if no abnormal signs of chest X-ray images are found, the image is considered normal and no further processing is required.

[0056] Furthermore, step 9 is specifically as follows:

[0057] The classification results of the model are compared with the judgment results of clinicians to verify the accuracy of the model. In addition, standard machine learning indicators such as accuracy, precision, recall, and F1 score are used to quantify the performance of the model. According to the verification results, the model parameters such as learning rate and network structure are adjusted to further improve the performance of the model.

[0058] Furthermore, step 10 is specifically as follows:

[0059] The model is continuously trained using the training set data, and the validation set is used for regular validation to ensure optimal model performance. The learning rate is 0.001, the batch size is 16, the optimizer type is AdamW optimizer, and the cross entropy loss function is used. The number of training rounds per fold is 10. To avoid overfitting, a series of related strategies such as early-stopping are used to improve the generalization ability of the model.

[0060] Beneficial effects:

[0061] The present invention proposes a chest X-ray image recognition method based on the BEiT V2 model, using the public Chest-X-ray dataset. Since the algorithm design introduces the pre-trained model BEiTV2 based on the Transformer architecture before feature extraction, this innovative strategy can effectively handle complex chest X-ray image problems. It improves the accuracy and robustness of chest X-ray image abnormality detection in practical applications, which is helpful for future practical applications in medical clinics. In addition, data enhancement methods are also used to further improve the stability and reliability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a method flow chart of an embodiment of the present invention.

[0063] Figure 2 Schematic diagram of Vsion Transformer segmentation used in the present invention.

[0064] Figure 3 This is the VQ-KD training flow chart of BEiT V2 in the present invention.

[0065] Figure 4 This is the pre-training flow chart of BEiT V2 in the present invention. DETAILED DESCRIPTION

[0066] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0067] like Figure 1 As shown, this embodiment provides a chest X-ray image recognition method based on the BEiT V2 model, and the method is implemented based on a data processing module, a pre-trained model loading and fine-tuning module, a model training optimization module, a model storage and evaluation module, etc.

[0068] The specific steps include:

[0069] Step 1: Capture and preprocess images

[0070] The chest imaging data is collected by X-ray machine to generate high-quality chest images. Then, the images are preprocessed. First, the generated chest image files are loaded using SimpleITK medical image library. Then, all images are adjusted to a uniform size for easy input. Then, the images are normalized to scale the image pixel values ​​to between 0 and 1. The normalization operation, i.e. scaling the image pixel values ​​to between 0 and 1, can be expressed by the following formula:

[0071] Where I is the pixel value of the original image, I minand I max are the minimum and maximum pixel values, respectively, and I normalized is the normalized pixel value.

[0072] After normalization is completed, further standardization is performed to center the pixel values ​​of the image to zero mean and have unit variance, so that the data set has zero mean and unit variance, which can be achieved by the following formula:

[0073]

[0074] Where I is the normalized pixel value, μ is the average value of all pixel values, σ is the standard deviation, and I standardized is the normalized pixel value.

[0075] Finally, a variety of transformations are applied to increase the diversity of the training set, including randomly selecting areas from the original image as new images, randomly flipping the image left and right, randomly flipping the image up and down, and randomly rotating the image by a certain angle. Finally, the processed images are organized into multiple batches, each batch containing multiple images and their labels, so as to obtain the dataset.

[0076] The above steps are mainly for collecting chest imaging data to form a data set and perform data enhancement, improve the performance of the model and ensure its reliability and accuracy on unknown data. By executing these steps, the effects of feature extraction, image segmentation, matching and recognition can be significantly improved, and ultimately more accurate clinical judgment can be achieved. The relevant data set used in this invention is the public Chest-X-ray data set.

[0077] Step 2: Pre-train the model

[0078] The encoder first uses ViT (Vision Transformer) to encode the input image and convert it into a series of feature vectors. Then it finds the nearest neighbor of each feature vector from a codebook. This nearest neighbor is a visual word. This process can be done by calculating the feature h of each image patch. i With all sight words j The minimum cosine distance between them determines the corresponding visual word, and the formula is as follows:

[0079] z i =argmin j ||l2(h i )-l2(e j )||2

[0080] Where l2 represents the L2 regularization operation.

[0081] The decoder receives the L2-regularized visual word sequence as input and outputs a series of features of the reconstructed image patch.

[0082] In order to train the Tokenizer and Decoder, a method called VQ-KD is used, which is a technology based on model distillation (feature learning strategy). In VQ-KD, the CLIP model is used as the teacher model, and its output features become the target of student model optimization. The goal is to maximize the similarity (i.e., cosine distance) between the output features of the student model and the features generated by the teacher model. At the same time, since some operations in VQ-KD are not differentiable, the direct copy gradient strategy in VQ-VAE is used to solve this problem. The final loss function consists of two parts: one is to maximize the similarity between the model output and the features generated by the teacher model; the other is to minimize the distance between the generated features and the visual words.

[0083] Step 3: Initialize the model

[0084] The pre-trained BEiT V2 model is used as the basis. First, import the required libraries and modules, then use BeitForImageClassification and BeitFeatureExtractor in the transformers library to load the pre-trained BEiT V2 model and its feature extractor as the initial model, and then modify the number of classification heads in the last layer of the BEiT V2 model to complete the model initialization.

[0085] BEiT V2 is a Transformer-based model that was originally pre-trained on the large-scale ImageNet-1k dataset and can learn rich visual features. By loading pre-trained weights, the model can quickly adapt to chest X-ray image detection tasks.

[0086] Step 4: Perform machine learning and divide the dataset

[0087] The entire dataset obtained in step 1 is divided into three parts: training set, validation set, and test set. First, all image file paths and corresponding labels are read into a list or DataFrame, and then the sample order in the dataset is shuffled using a random shuffle method to ensure that the sample order in the dataset is random to avoid potential order bias. Finally, the dataset is divided into training set, validation set, and test set according to a predetermined ratio. Subsequently, 90% of the data in the dataset is randomly used for model training and validation, and the remaining 10% is reserved for testing the final effect of the model.

[0088] This division ratio ensures that there is enough data to train the model and that the performance of the model can be evaluated on an independent test set. The above process follows the standard machine learning process for preprocessing and division, and uses five-fold cross-validation to ensure the fairness and reliability of the experiment; the training set is used for model training, the validation set is used to adjust the model parameters, and the test set is used to evaluate the final performance of the model.

[0089] Step 5: Extract features

[0090] The BEiT V2 model is used to extract features from the preprocessed image. First, the preprocessed image is passed to the model, which will use its powerful image processing capabilities and pre-trained parameters to extract the embedded features of the image. After the model completes the image processing, the extracted features will be stored, and finally the key features of the extracted image will be used to achieve the final classification and recognition task.

[0091] The BEiT V2 model can automatically learn complex features in images, including texture, shape, and potential disease patterns. All input image features are ultimately represented in the form of embedded vectors for subsequent classification tasks.

[0092] The feature extraction in step 5 involves the application of the BEiT V2 model in chest X-ray image detection. The applied BEiT V2 is a pre-trained model based on the Transformer architecture. It learns image representation through self-supervised learning and has powerful feature extraction capabilities.

[0093] The following are the key steps in the feature extraction process:

[0094] The preprocessed images are sent to the model in batches. The encoder of the model first divides the input image into multiple overlapping patches and maps these patches to an embedding vector through a linear layer. Then the model adds position encoding to each patch so that the model understands the relative position of the patch in the image. After the position encoding is added, the vector will pass through multiple Transformer encoder layers. The Transformer encoder layer of the model will encode the embedded vector. Each encoder layer includes multi-head self-attention (Multi-Head Self-Attention) and feed-forward neural network (Feed-ForwardNetwork). After passing through the Transformer encoder layer, each patch gets a high-level feature representation, which is the corresponding feature of the final extracted image.

[0095] Step 6: Use the attention mechanism to enhance the intermediate extracted features

[0096] During feature extraction, the attention mechanism is used to enhance the extracted features. The attention mechanism is used to help the model focus on areas in the image that may contain abnormal lesions in the chest X-ray image, while suppressing interference from irrelevant areas, thereby improving the accuracy and robustness of the model.

[0097] The feature enhancement attention mechanism in step 6 uses the attention mechanism to enhance the extracted features. This mechanism is an important component of the BEiT V2 model based on the Transformer architecture. The model processes the sequence of image blocks through a multi-head attention mechanism and a feedforward neural network to extract and enhance features.

[0098] After the previous series of processing, the input image is mapped into an input sequence, and then these input series will be mapped to different subspaces by the model, and then each subspace has its own attention head to focus on different features. The model will generate three linear transformations for the input sequence: query (Query, Q), key (Key, K) and value (Value, V), and then perform self-attention calculation on the input sequence to calculate the attention score of each position and all other positions. The output of the self-attention mechanism used here is the weighted aggregation of values, and the calculation formula is:

[0099]

[0100] After obtaining the attention score of the input series, the input sequence will be further transformed by the feedforward neural network. First, a regularization is performed, which is achieved using layer normalization. Layer normalization can ensure that the output of each layer has zero mean and unit variance, thereby stabilizing network training. The formula is expressed as:

[0101] Xnorm=LayerNorm(x)

[0102] Where x is the input sequence, x norm is the sequence after layer normalization, LayerNorm(x) is the layer normalization function, defined as:

[0103]

[0104] where μ(x) is the mean of x, σ 2 (x) is the variance of x, γ and β are learnable parameters, and ∈ is a small constant for numerical stability.

[0105] Then, through a multi-layer perceptron, the representation ability of the input sequence can be further enhanced after passing through multiple layers of neurons. The multi-layer perceptron is a basic artificial neural network model and can be expressed as

[0106] FFN(A)=max(0,A×W1+b1)×W2+b2

[0107] Where A is the sequence after layer normalization, W1, W2 are weight matrices, b1, b2 are bias terms, max is used as the activation function, and FFN(A) represents the output of MLP.

[0108] Finally, a residual connection is used to help the training of deep networks by adding the input directly to the output of the MLP to prevent feature loss. The residual connection can be expressed as:

[0109] x′=x norm +z

[0110] where x norm is the input sequence after layer normalization, z is the output of the MLP, and x′ is the final output with residual connection.

[0111] Through these steps, the BEiT V2 model can effectively extract useful features from images and enhance these features using a multi-head attention mechanism and a feedforward neural network. This helps the model better understand the contextual information and spatial relationships in the image, thereby improving the accuracy and stability of chest X-ray image recognition.

[0112] Step 7: Semantic Analysis

[0113] After waiting for feature extraction to complete, the model is used to perform semantic analysis operations. First, the feature vectors of each patch are fused to obtain a global feature representation of the entire image to identify and distinguish chest X-ray image abnormalities and other pathological features. Then, visualization tools (such as Grad-CAM) are used to highlight important areas in the image to observe which areas contribute more to the model's decision. This process helps the model understand the meaning of specific areas in the image and classify them as normal tissue, inflammation, or other diseases. Based on the results of the semantic analysis, the area that best represents the abnormal characteristics of the chest X-ray image is selected to provide a basis for the final classification decision.

[0114] Step 8: Classification Decision

[0115] Based on the extracted and enhanced feature vectors, a pre-trained classifier is used to determine whether the image contains abnormal signs of chest X-ray images. If there are abnormal signs of chest X-ray images, proceed to the next step; if no abnormal signs of chest X-ray images are found, the image is considered normal and no further processing is required.

[0116] Step 9: Result Verification

[0117] The classification results of the model are compared with the judgment results of clinicians to verify the accuracy of the model. In addition, standard machine learning indicators such as accuracy, precision, recall, and F1 score are used to quantify the performance of the model. According to the verification results, the model parameters such as learning rate and network structure are adjusted to further improve the performance of the model.

[0118] Step 10: Model training and optimization

[0119] The model is continuously trained using the training set data, and the validation set is used for regular validation to ensure optimal model performance. The learning rate is 0.001, the batch size is 16, the optimizer type is AdamW optimizer, and the cross entropy loss function is used. The number of training rounds per fold is 10. To avoid overfitting, a series of related strategies such as early-stopping are used to improve the generalization ability of the model.

[0120] Step 11: Save the output

[0121] Save the trained model, including model weights, configuration files, etc., for subsequent use and deployment.

[0122] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A chest X-ray image recognition method based on the BEiT V2 model, characterized in that: The specific steps include: Step 1: Collect images and preprocess them; Step 2: Pre-train the model; Step 3: Initialize the model; Step 4: Perform machine learning and divide the data set; Step 5: Extract features; Step 6: Use the attention mechanism to enhance the intermediate extracted features; Step 7: semantic analysis; Step 8: Classification decision; Step 9: Result verification; Step 10: Model training and optimization; Step 11: Output and save; Save the trained model, including model weights, configuration files, etc., for subsequent use and deployment.

Images