Visual Transform-based chest radiograph classification model construction method and system guided by expert fixation point, terminal and medium
By constructing an expert gaze-guided visual Transformer model, using expert gaze-guided information and hybrid learning strategies, the problem of the lack of interpretability in the deep learning model in chest radiographic image classification is solved, and a more accurate and interpretable chest radio classification is achieved.
Patent Information
- Application Number
- CN202411746529.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-21
AI Technical Summary
Existing deep learning models lack interpretability in chest radiographic image classification, difficult to understand their internal decision-making process, and may be affected by unrelated features, resulting in false diagnostic results.
By constructing a chest radiographic classification model based on visual Transformer guided by expert gaze point, the expert gaze point information is used to introduce it into the model, combining mixed learning strategies and spatial attention mechanisms, the model is guided to focus on disease-related areas in the image.
The decision accuracy and interpretability of the model are improved, the impact of irrelevant features is reduced, and excellent chest radiographic classification performance is achieved.
Smart Images
Figure CN119672422A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of chest X-ray images, and in particular to a method, system, terminal and medium for constructing a chest X-ray classification model guided by an expert's gaze point and based on a visual Transformer. Background Art
[0002] Deep learning models have made significant progress in chest X-ray image classification, but their interpretability remains a major challenge. Deep learning models are often treated as "black boxes," making it difficult to understand their internal decision-making processes. This lack of transparency reduces the credibility of these models in real-world medical applications, especially when life-threatening diagnoses are involved. Studies have shown that deep learning models sometimes make decisions based on fields that are not related to the disease, leading to incorrect diagnostic results. For example, the model may be influenced by irrelevant features in the chest X-ray image, such as text labels or shadows of other organs, leading to incorrect judgments.
[0003] To address this issue, some researchers have tried to incorporate expert gaze data into the model. In the field of medical imaging, radiologists and other healthcare professionals analyze and interpret images by selectively focusing on specific areas based on their extensive experience and knowledge, which is crucial for identifying pathologies. By introducing expert gaze data, the model can learn more meaningful features, thereby improving its decision accuracy and interpretability. Expert gaze data can provide valuable information about the regions of interest in chest X-ray images, guiding the model to focus on the most relevant diagnostic areas. By combining the model's attention with the expert's gaze patterns, this scheme can help reduce the impact of features in chest X-ray images that are irrelevant to disease classification and improve the model's interpretability.
[0004] Previous studies combining expert gaze data mainly adopt the attention consistency method, using gaze data as a supervisory signal. Auxiliary tasks are designed to supervise the model's attention map, using multi-task learning, where the model learns the main task and auxiliary tasks simultaneously. Features learned from the auxiliary tasks improve the performance of the main task. Although multi-task learning can effectively extract domain-specific information from gaze data, thereby improving classification performance, the supervisory signal is not particularly stable and contains certain uncertainties. This dataset samples gaze data from a single doctor, whose attention area may be different, resulting in overfitting of the auxiliary task and inhibiting the training of the main task. Summary of the invention
[0005] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a method, system, terminal and medium for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points.
[0006] (II) Technical solution To achieve the above object, the present invention provides the following technical solution: According to one aspect of the present invention, a method for constructing a chest X-ray classification model is provided, comprising: Construct a chest X-ray classification model based on visual Transformer guided by expert gaze points. By introducing the gaze point information of experts when reading images into the chest X-ray classification model, the medical field information implicit in the expert gaze points and the chest X-ray image features in the visual Transformer can be fully obtained and effectively integrated.
[0007] S1, building a chest X-ray classification model based on visual Transformer guided by expert gaze points; S2, adopts a hybrid learning strategy, by selecting a part of chest X-ray image samples to carry fixation points during training, and the other part only uses chest X-ray images as input, so that the model only needs images as input in the inference stage; S3, using a known chest X-ray classification dataset containing expert gaze points, and generating expert gaze point embedding vectors based on the expert gaze point sequence, to complete the end-to-end training of the chest X-ray classification model based on visual Transformer guided by expert gaze points; S4, using the test images in the chest X-ray classification dataset containing the known expert gaze points to test the trained chest X-ray classification model based on visual Transformer guided by the expert gaze points, and construct the final chest X-ray classification model.
[0008] Preferably, the construction of a chest X-ray classification model based on visual Transformer guided by expert gaze points comprises: S101, constructing an image embedding module, which is used to convert the input image into an embedding vector that can be processed by the visual Transformer network, and add a classification vector and position encoding; S102, constructing an expert gaze point information injection module, extracting the expert gaze points, and converting the expert's gaze data into an embedding vector that can be used by the visual Transformer model, so as to effectively integrate these data into the model; S103, builds a visual Transformer backbone network, uses visual Transformer technology to process embedded image data, can capture long-distance dependencies and extract high-level features; S104, constructing an expert gaze point information injection module, injecting the converted expert gaze point data into the visual Transformer network, and using the expert's gaze information to guide model training; S105, construct a classification head module to convert the output features of the visual Transformer network into the final classification results.
[0009] Preferably, in S101, the image embedding module divides the input image into image blocks of fixed size and flattens each image block into a vector; then, these vectors are mapped to a high-dimensional embedding space through a linear layer to form an image block embedding; then, the position information of each image block is retained by adding a position code, and the classification vector is spliced to achieve the final classification; the expert gaze point information embedding module converts the collected original expert gaze point sequence into a gaze point map, and converts the gaze point map into an expert gaze point embedding vector through normalization and a linear layer; The visual Transformer backbone network consists of multiple Transformer blocks, each of which is mainly composed of multi-head self-attention, layer normalization, and feedforward neural network layers. The multi-head self-attention mechanism captures global features, and the feedforward neural network further processes features. It is used to capture long-distance dependencies of image embedding vectors and extract high-order features; The expert gaze point information injection module consists of a bottleneck layer and a gaze point feature fusion module. The bottleneck structure allows the pre-trained model to adjust its feature representation according to the requirements of the target task, so that it can better adapt to the feature distribution in the chest X-ray image. The gaze point feature fusion module will reduce the redundant information in the expert gaze points through dimensionality reduction, and integrate the domain knowledge contained in the gaze points into the image features through the spatial attention mechanism, guiding the model to focus on the disease-related areas in the image. The classification head module uses the CLS classification vector of the last layer of the visual Transformer as input, and generates the chest X-ray classification results through a fully connected layer and a softmax activation function.
[0010] Preferably, in S2, a hybrid learning strategy is adopted to enable the model to take into account chest X-ray image classification under both conditions of having expert gaze point input and without expert gaze point input during training, including: During training, a portion of chest X-ray image samples are randomly selected to carry expert gaze points to guide the model to focus on the disease-related parts of the expert gaze information. Another portion of samples is input only with chest X-ray images to maintain the classification performance of the model in the absence of expert gaze points, so that the model only needs chest X-ray images as input during the inference stage.
[0011] Preferably, in S3 S301, using a known chest X-ray classification dataset containing expert gaze points, and generating an expert gaze point embedding vector representing expert attention based on the expert gaze point annotations, to complete end-to-end training of a chest X-ray classification model based on a visual Transformer guided by expert gaze points, including: The public chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points are used to perform end-to-end training on the chest X-ray classification model based on visual Transformer guided by expert gaze points. Set the initialization parameters of the expert gaze-guided visual Transformer-based chest X-ray classification model; Determine the loss function of the expert gaze-guided visual Transformer-based chest radiograph classification model; Determine the relevant hyperparameters in an expert gaze-guided visual Transformer-based chest radiograph classification model; The above steps complete the end-to-end training of the chest X-ray classification model based on visual Transformer guided by expert gaze.
[0012] Preferably, known chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points are used, and expert gaze point embedding vectors representing expert attention are generated based on expert gaze point annotations therein to complete end-to-end training of a chest X-ray classification model based on visual Transformer guided by expert gaze points, including: The chest X-ray images and expert gaze point annotations of the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points are obtained, and the expert gaze point embedding vector is generated by using the expert gaze point annotation. The chest X-ray images and the expert gaze point embedding vector are input into the expert gaze point embedding vector model, and the category annotation labels corresponding to the chest X-ray images of the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points are used. The expert gaze point-guided visual Transformer-based chest X-ray classification model is fine-tuned in an end-to-end manner, so that the model automatically learns the mapping relationship between the original chest X-ray images and the classification categories.
[0013] Set the initialization parameters of the expert gaze-guided visual Transformer-based chest X-ray classification model, including: The initial parameters of the expert gaze information injection module and the classification head module are randomly initialized, and the parameters of the remaining modules are pre-trained on the ImageNet dataset as initialization parameters.
[0014] Determine the loss function of the expert gaze-guided visual Transformer-based chest radiograph classification model, including: The loss function uses the cross entropy function.
[0015] Determine the relevant hyperparameters in the expert gaze-guided visual Transformer-based chest radiograph classification model, including: The stochastic gradient descent algorithm is used in the training process. The initial learning rate is 0.005, and it decreases by 10 times every 10 epochs. The batch size is 16, and the training process needs to iterate 40 epochs until the model converges.
[0016] Preferably, the trained chest X-ray classification model guided by expert gaze points and based on visual Transformer is tested using a test image in a chest X-ray classification dataset with known expert gaze points, including: In the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points, the trained expert gaze-guided visual Transformer-based chest X-ray classification model is tested with known test images to evaluate the performance of the model; An expert gaze point embedding acquisition module, which is used to obtain the embedding vector of the expert gaze point; The prediction model building module is used to build a chest X-ray classification model based on visual Transformer guided by expert gaze points; Model training module, which is used to complete the end-to-end training of the chest X-ray classification model based on visual Transformer guided by expert gaze points using the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points; The model testing module is used to test the trained expert gaze-guided visual Transformer-based chest X-ray classification model using known test images to evaluate the classification performance of the constructed model.
[0017] According to another aspect of the present invention, a system for constructing a chest X-ray classification model based on a visual Transformer guided by an expert gaze point is provided, comprising: A prediction model building module for an expert gaze-guided visual Transformer-based chest radiograph classification model; A model training module is used to use the chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points to perform end-to-end training on the visual Transformer-based chest X-ray classification model guided by the expert gaze points.
[0018] The model testing module is used to test the trained expert gaze-guided visual Transformer-based chest X-ray classification model using known test images to evaluate the performance of the constructed model.
[0019] According to the third aspect of the present invention, a chest X-ray classification method based on expert gaze guidance is provided, wherein the classification result is obtained by constructing the chest X-ray classification model using any of the above-mentioned chest X-ray classification model construction methods or construction systems, and any image is used as input of the model to output the classification result.
[0020] According to a fourth aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above methods when executing the program.
[0021] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: According to a fifth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above.
[0022] (III) Beneficial effects Compared with the prior art, the present invention provides a method, system, terminal and medium for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points, which has the following beneficial effects: 1. The expert gaze-guided visual Transformer-based chest X-ray classification model construction method, system, terminal and medium fully consider the characteristics of chest X-ray images and expert gaze points by adopting the expert gaze-guided visual Transformer-based chest X-ray classification model, and achieve excellent performance on the chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points. The gaze point information injection module adopted by the module adjusts the image features in the network through the spatial attention mechanism, guides the model to focus on the disease-related areas in the chest X-ray images, and through the hybrid learning strategy line of sight, the model only needs chest X-ray images as input in the reasoning stage, achieving excellent performance.
[0023] 2. The expert gaze-guided chest X-ray classification model construction method, system, terminal and medium based on visual Transformer extracts high-order features of chest X-ray images by using a backbone network composed of visual Transformer blocks, and guides the model to focus on disease-related features in chest X-ray images through the gaze information injection module, reducing the interference of redundant information and improving the accuracy and interpretability of the chest X-ray classification model. It is highly efficient, low-cost, easy to implement, and very flexible, and can be deployed on a backbone network with fewer parameters or a backbone network with better performance according to actual needs in terms of efficiency and performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1The figure is a workflow diagram of a chest X-ray classification method in a preferred embodiment of the present invention.
[0025] Figure 2 Schematic diagram of the components of a chest X-ray classification system in a preferred embodiment of the present invention.
[0026] Figure 3 These are some subjective experimental results obtained by the chest X-ray classification method and system provided in a preferred embodiment of the present invention on a public dataset in the field of chest X-ray classification combined with expert gaze points. The first row is the chest X-ray images, and the remaining two rows of images are heat maps generated using the class activation mapping method. DETAILED DESCRIPTION
[0027] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] See also Figure 1-3 , Figure 1 This is a workflow diagram of a chest X-ray classification method in a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of the components of a chest X-ray classification system in a preferred embodiment of the present invention; Figure 3 The following are some subjective experimental results obtained by the chest X-ray classification method and system provided in a preferred embodiment of the present invention on a public dataset in the field of chest X-ray classification combined with expert gaze points. The first row is a chest X-ray image, and the area of suspected disease in the image that the expert is concerned about is marked in the middle. The remaining two rows of images are heat maps generated using the class activation mapping method, representing the area of concern of the model. The first row is a heat map generated by the original visual transformer, and the second row is a heat map generated by the chest X-ray classification model based on the visual transformer guided by the expert's gaze point.
[0029] An embodiment of the present invention provides a method for constructing a chest X-ray classification model combined with expert gaze points. The construction method combines the expert gaze point information with the characteristics of the chest X-ray image to realize the construction of a chest X-ray classification model based on a visual Transformer guided by the expert gaze points.
[0030] like Figure 1 As shown, the method for constructing a chest X-ray classification model provided in this embodiment may include: S1, building a chest X-ray classification model based on visual Transformer guided by expert gaze points; S2, adopts a hybrid learning strategy, by selecting a part of chest X-ray image samples to carry fixation points during training, and the other part only uses chest X-ray images as input, so that the model only needs images as input in the inference stage; S2, using a known chest X-ray classification dataset containing expert gaze points, and generating an expert gaze point embedding vector based on the expert gaze point sequence therein, to complete end-to-end training of the chest X-ray classification model based on visual Transformer guided by the expert gaze points; S3, using a test image in a chest X-ray classification data set known to contain expert gaze points to test the trained chest X-ray classification model based on visual Transformer guided by the expert gaze points, and constructing a final chest X-ray classification model.
[0031] In a preferred embodiment of S1, constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points may include: A chest X-ray classification model based on visual Transformer guided by expert gaze points is constructed. The model mainly includes an image embedding module, a gaze point information embedding module, a backbone network feature extraction module, an expert gaze point information injection module, and a classification head to perform end-to-end chest X-ray classification. Among them: S101, construct an image embedding module for converting the input chest X-ray image into an image embedding vector. According to the visual Transformer method, the image embedding module first converts the input chest X-ray image into an image embedding vector. Divide into a non-overlapping grid, the size of each image block in the grid is These image patches are then flattened and linearly projected into a fixed-dimensional space, after which position encoding is added to preserve the spatial information of the image and the classification vector CLS is concatenated. For an image containing n image patches, , the final generated image embedding vector is .
[0032] S102, construct an expert gaze point information embedding module. The gaze point data is collected by the eye tracking device at a certain sampling rate, and the data consists of a series of scattered points. Each scattered point represents the position and duration of the expert's focus on the image. The data can be composed of the following sequence Indicates that is the position of the gaze point, is the fixation duration. We use a map M to represent these fixations, which has the same size as the chest X-ray image. The value of each spatial position in the map corresponds to the duration of the expert's fixation on that point, while the unfixated area is initialized to zero. The map is then normalized to [0,1] and projected to a dimension of The gaze point embedding vector G of .
[0033] S103, construct a visual Transformer backbone network to extract features of the input image and output a multi-order feature map. This network layer uses a pre-trained deep learning-based image classification network as the backbone network to extract features. This layer consists of multiple Transformer blocks, each of which is mainly composed of multi-head self-attention, layer normalization, and feedforward neural network layers. The multi-head self-attention mechanism captures global features, and the feedforward neural network further processes features. This network layer uses a pre-trained deep learning-based image classification network as the backbone network to extract features. This module outputs a feature vector ,in Represented as the visual Transformer block There are 12 layers in total.
[0034] S104, construct an expert gaze information injection module, which is used to integrate the expert's gaze information into the multi-order feature map of the model's visual Transformer backbone network through an additional input module, guide the model to extract the medical domain knowledge contained in the gaze point, and enhance the model's attention to the features related to disease classification in the multi-order feature map. The expert gaze information injection module consists of a bottleneck layer and a gaze feature fusion module (GFF). The expert gaze information injection module extracts image features from the multi-head self-attention output of each Transformer block in the backbone network and fuses the expert gaze features, and then connects to the output of the Transformer block of this layer through a jump connection.
[0035] The bottleneck layer is used to assist transfer learning. This bottleneck structure allows the model to adjust its feature representation according to the requirements of the target task, making it better adapted to the feature distribution in chest X-ray images, while retaining the information obtained in the pre-training stage, thereby improving the effectiveness of transfer learning and the overall performance and robustness of the model.
[0036] Specifically, given the input , which is the output feature vector of the l-th layer of multi-head self-attention, we use a linear layer to reduce the dimension of this feature:
[0037] in The dimension is The compression layer, GELU represents the activation function, Represents the compressed features.
[0038] Afterwards, the gaze feature fusion module is used to fuse the expert domain knowledge contained in the expert gaze points into the image features. When viewing chest X-ray images, experts selectively focus on suspicious abnormal areas instead of trying to Figure 1 Therefore, in the GFF block, we adopt spatial attention to guide the learning of gaze features, which enhances the model's attention to disease-related regions.
[0039] Before the GFF block, we pass the output of the bottleneck layer Divided into n-dimensional image feature vectors and a 1-dimensional classification vector In GFF, the gaze vector is embedded The medical domain knowledge contained in the image is integrated into the image feature vector Before fusion, considering that the expert’s gaze points may contain redundant information and may cause overfitting, we use The gaze information is compressed to obtain a low-rank representation of the gaze point feature. Then, a linear layer is used to To match image features Dimensions:
[0040] Subsequently, we use the spatial attention mechanism to fuse the image features with the gaze features to enhance the model's attention to disease-related areas. Specifically, we concatenate the compressed gaze feature vector and the image feature vector, perform average pooling and maximum pooling respectively, and then use The operations are spliced together and represented by feature F, with the following formula:
[0041] The parameter p is related to our hybrid learning strategy and its value is 0 or 1. It is used to control whether to use the fixation point. When the value is 1, it means that the expert fixation point is used, and when the value is 0, it means that the expert fixation point is not used.
[0042] We reshape the 2D feature vector F into a 2D feature map and use a convolutional layer to integrate local information between adjacent image regions, and then restore it to the original dimension of F. Then apply function to generate spatial attention weights, and then with Multiply to get the adjusted image features .
[0043] Among them, rs means reshaping the vector into a two-dimensional feature map. Represents restoring the two-dimensional feature map to a vector, σ represents operation, ⊙ represents an element-by-element multiplication operation.
[0044]
[0045] Next, we will Output characteristics of the module The separated classification vector Concatenate them and use a linear layer The layer restores the adjusted image features to the same dimension as in the visual Transformer and obtains the output of the GII module. :
[0046] Finally, we use addition to fuse these features with the normalized output features of the middle layer of the Transformer block to obtain the features guided by the expert's gaze.
[0047] S105, the classification head module consists of a fully connected layer and an activation function. The CLS classification vector of the last layer of the visual Transformer is used as input, the vector is converted into a category score through a fully connected layer, and the chest X-ray classification result is generated through a softmax activation function.
[0048] In a preferred embodiment of S2, a hybrid learning strategy is adopted to take into account the classification of chest X-ray images with and without expert gaze points during training, which may include: A hybrid learning strategy is adopted. During training, a portion of chest X-ray image samples are randomly selected to carry expert gaze points to guide the model to focus on the disease-related parts of the expert gaze information. Another portion of samples are only input with chest X-ray images to maintain the classification performance of the model without expert gaze points, so that the model only needs images as input during the inference stage. In a preferred embodiment of S3, a known chest X-ray classification dataset containing expert gaze points is used to perform end-to-end training on a chest X-ray classification model based on visual Transformer guided by expert gaze points: S301, using the public chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points to train the chest X-ray classification model guided by the expert gaze points. The chest X-ray images and expert gaze point information in the chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points are obtained, and the expert gaze point embedding vector is generated using the expert gaze point information. The gaze point embedding vector and the chest X-ray image are taken as input, and the category annotations corresponding to the chest X-ray images in the chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points are taken as labels. The chest X-ray classification model guided by expert gaze points is trained in an end-to-end manner, so that the model automatically learns the mapping relationship between the chest X-ray image and the category annotation; specifically: The EG-CXR dataset includes a total of 1,083 chest X-ray images. Each image is accompanied by aligned data, including transcribed radiology report text, radiologist's dictation audio, and eye gaze coordinates. The dataset contains three class labels: pneumonia (360 images), congestive heart failure (CHF, 363 images), and normal (360 images), with balanced age and gender distribution within each class. The gaze point data in this dataset was generated by a radiologist using the Gazepoint GP3 eye tracker, and the chest X-ray images are from the chest X-ray multi-label classification dataset MIMIC-CXR. The OMSI dataset includes a total of 1,250 chest X-ray images. This dataset is a binary classification dataset, including pneumothorax (975 images) and normal (275 images). The gaze point data in this dataset was generated by a radiologist using the Tobii Pro Nano eye tracker, and the chest X-ray images are from the chest X-ray pneumothorax classification dataset SIIM-ACR.
[0049] S302, setting initialization parameters for the model. The Transformer block and classification head in the backbone network are initialized with parameters pre-trained on a large-scale image classification dataset, and the remaining expert gaze point information embedding module is randomly initialized; S303, determine the loss function. The loss function used in model training is the cross entropy between the probability distribution of the model output category and the classification label in the data set.
[0050] S304, determine the relevant hyperparameters in the model. The initial learning rate is 0.005, and it decreases by 10 times every 10 epochs, and the batch size is 16; In a preferred embodiment of S4, testing the trained chest X-ray classification model based on visual Transformer guided by the expert's gaze point using a test image in a chest X-ray classification data set containing the expert's gaze point to construct a final chest X-ray classification model may include: S401, the performance of the proposed model is tested using two commonly used public datasets in the field of expert gaze-assisted chest X-ray classification, namely EG-CXR and OMSI. During the test, following the previous work on the two datasets, for EG-CXR, a 9:1 training set test set split ratio was used to evaluate the performance of the proposed method, and for OMSI, the official test set was used for evaluation to construct the final chest X-ray classification model.
[0051] An embodiment of the present invention provides a system for constructing a chest X-ray classification model guided by an expert's gaze point, which may include: The classification model building module is used to build a chest X-ray classification model based on visual Transformer guided by expert gaze points; A model training module, which is used to use the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points to perform end-to-end training on the chest X-ray classification model based on visual Transformer guided by the expert gaze points, and select half of the chest X-ray image samples to carry the expert gaze point input and the other half of the chest X-ray image samples not to carry the expert gaze point input during the training process; The model testing module is used to test the trained expert gaze-guided visual Transformer-based chest X-ray classification model using known test images to evaluate the performance of the constructed model.
[0052] In a preferred embodiment, the chest X-ray classification model based on visual Transformer guided by expert gaze may include: The image embedding module is used to convert the input image into an embedding vector that can be processed by the visual Transformer network and add a classification vector and position encoding.
[0053] The expert gaze embedding module extracts the expert gaze points and converts the expert's gaze data into embedding vectors that can be used by the visual Transformer model so that this data can be effectively integrated into the model.
[0054] The visual Transformer backbone network uses visual Transformer technology to process embedded image data, which can capture long-distance dependencies and extract high-level features.
[0055] The expert gaze point information injection module injects the converted expert gaze point data into the visual Transformer network and uses the expert's gaze information to guide model training.
[0056] The classification head module converts the output features of the visual Transformer network into the final classification results.
[0057] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.
[0058] An embodiment of the present invention provides a chest X-ray classification model method based on visual Transformer guided by expert gaze points. The chest X-ray classification model is constructed using the chest X-ray classification model construction method or construction system of any one of the above embodiments. Any image is used as the input of the model, and the chest X-ray classification result is output.
[0059] An embodiment of the present invention provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute any method or system of the above embodiments of the present invention.
[0060] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviated: RAM), such as static random access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.
[0061] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.
[0062] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.
[0063] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.
[0064] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program can be used to execute the method or system of any one of the above embodiments of the present invention.
[0065] The method, system, terminal and medium for constructing a chest X-ray classification model provided by the above-mentioned embodiment of the present invention adopt a chest X-ray classification model based on expert gaze point guidance, which fully considers the characteristics of chest X-ray images and expert gaze points, and obtains excellent performance in chest X-ray classification data sets EG-CXR and OMSI containing expert gaze points. The realization of this performance mainly depends on the exploration of how to use the expert gaze point information. The gaze point information injection module adopted by it adjusts the image features in the network through the spatial attention mechanism, guides the model to focus on the disease-related areas in the chest X-ray image, and through the hybrid learning strategy, the model only needs the chest X-ray image as input in the reasoning stage, and achieves excellent performance.
[0066] The chest X-ray classification model construction method, system, terminal and medium provided in the above-mentioned embodiments of the present invention utilize a backbone network composed of visual Transformer blocks to extract high-order features of the chest X-ray image, and guide the model to focus on disease-related features in the chest X-ray image through the expert gaze point information injection module, thereby reducing the interference of redundant information and improving the accuracy and interpretability of the chest X-ray classification model.
[0067] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points, characterized in that: The following steps are involved: S1, building a chest X-ray classification model based on visual Transformer guided by expert gaze points; S2, adopts a hybrid learning strategy, by selecting a part of chest X-ray image samples to carry fixation points during training, and the other part only uses chest X-ray images as input, so that the model only needs images as input in the inference stage; S3, using a known chest X-ray classification dataset containing expert gaze points, and generating expert gaze point embedding vectors based on the expert gaze point sequence, to complete the end-to-end training of the chest X-ray classification model based on visual Transformer guided by expert gaze points; S4, using the test images in the chest X-ray classification dataset containing the known expert gaze points to test the trained chest X-ray classification model based on visual Transformer guided by the expert gaze points, and construct the final chest X-ray classification model.
2. A method for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points according to claim 1, characterized in that: In S1, a chest X-ray classification model based on visual Transformer guided by expert gaze points is constructed. The model mainly includes: image embedding module, gaze point information embedding module, backbone network feature extraction module, expert gaze point information injection module, and classification head, and performs end-to-end chest X-ray classification, where: S101, constructing an image embedding module, which is used to convert the input image into an embedding vector that can be processed by the visual Transformer network, and add a classification vector and position encoding; S102, constructing an expert gaze point information injection module, extracting the expert gaze points, and converting the expert's gaze data into an embedding vector that can be used by the visual Transformer model, so as to effectively integrate these data into the model; S103, builds a visual Transformer backbone network, uses visual Transformer technology to process embedded image data, can capture long-distance dependencies and extract high-level features; S104, constructing an expert gaze point information injection module, injecting the converted expert gaze point data into the visual Transformer network, and using the expert's gaze information to guide model training; S105, construct a classification head module to convert the output features of the visual Transformer network into the final classification results.
3. A method for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points according to claim 2, characterized in that: In S101, the image embedding module divides the input image into image blocks of fixed size and flattens each image block into a vector; then, these vectors are mapped to a high-dimensional embedding space through a linear layer to form an image block embedding; then, the position information of each image block is retained by adding position encoding, and the classification vector is concatenated to achieve the final classification; the expert gaze point information embedding module converts the collected original expert gaze point sequence into a gaze point map, and converts the gaze point map into an expert gaze point embedding vector through normalization and a linear layer; The visual Transformer backbone network consists of multiple Transformer blocks. Each Transformer block is mainly composed of multi-head self-attention, layer normalization, and feed-forward neural network layers. The multi-head self-attention mechanism captures global features, and the feed-forward neural network further processes features to capture long-distance dependencies of image embedding vectors and extract high-order features. The expert gaze information injection module consists of a bottleneck layer and a gaze feature fusion module. The bottleneck structure allows the pre-trained model to adjust its feature representation according to the requirements of the target task. To make it better adapt to the feature distribution in chest X-ray images, the gaze point feature fusion module will reduce the redundant information in the expert's gaze points through dimensionality reduction, and integrate the domain knowledge contained in the gaze points into the image features through the spatial attention mechanism, guiding the model to focus on the disease-related areas in the image. The classification head module uses the CLS classification vector of the last layer of the visual Transformer as input, and generates chest X-ray classification results through a fully connected layer and a softmax activation function.
4. A method for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points according to claim 1, characterized in that: In S2, a hybrid learning strategy is adopted to make the model take into account the classification of chest X-ray images under both conditions of expert gaze input and without expert gaze input during training, including: During training, a portion of chest X-ray image samples are randomly selected to carry expert gaze points to guide the model to focus on the disease-related parts of the expert gaze information. Another portion of samples is input only with chest X-ray images to maintain the classification performance of the model in the absence of expert gaze points, so that the model only needs chest X-ray images as input during the inference stage.
5. A method for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points according to claim 1, characterized in that: In S3 S301, using a known chest X-ray classification dataset containing expert gaze points, and generating an expert gaze point embedding vector representing expert attention based on the expert gaze point annotations, to complete end-to-end training of a chest X-ray classification model based on a visual Transformer guided by expert gaze points, including: The public chest X-ray classification datasets EG-CXR and OMSI containing expert gaze points are used to perform end-to-end training on the chest X-ray classification model based on visual Transformer guided by expert gaze points. Set the initialization parameters of the expert gaze-guided visual Transformer-based chest X-ray classification model; Determine the loss function of the expert gaze-guided visual Transformer-based chest radiograph classification model; Determine the relevant hyperparameters in an expert gaze-guided visual Transformer-based chest radiograph classification model; The above steps complete the end-to-end training of the chest X-ray classification model based on visual Transformer guided by expert gaze.
6. A method for constructing a chest X-ray classification model based on visual Transformer guided by expert gaze points according to claim 5, characterized in that: We use the known chest X-ray classification datasets EG-CXR and OMSI that contain expert gaze points, and generate expert gaze point embedding vectors representing expert attention based on the expert gaze point annotations to complete the end-to-end training of the chest X-ray classification model based on visual Transformer guided by expert gaze points, including: The chest X-ray images and expert gaze point annotations of the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points are obtained, and the expert gaze point embedding vector is generated by using the expert gaze point annotation. The chest X-ray images and the expert gaze point embedding vector are input into the model, and the category annotation labels corresponding to the chest X-ray images of the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points are used to fine-tune the expert gaze point-guided visual Transformer-based chest X-ray classification model in an end-to-end manner, so that the model automatically learns the mapping relationship between the original chest X-ray images and the classification categories. Set the initialization parameters of the expert gaze-guided visual Transformer-based chest X-ray classification model, including: The initial parameters of the expert gaze information injection module and the classification head module are randomly initialized, and the parameters of the remaining modules are pre-trained on the ImageNet dataset as initialization parameters; Determine the loss function of the expert gaze-guided visual Transformer-based chest radiograph classification model, including: The loss function adopts the cross entropy function; Determine the relevant hyperparameters in the expert gaze-guided visual Transformer-based chest radiograph classification model, including: The stochastic gradient descent algorithm is used in the training process. The initial learning rate is 0.005, and it decreases by 10 times every 10 epochs. The batch size is 16, and the training process needs to iterate 40 epochs until the model converges.
7. The method for constructing a chest X-ray classification model according to claim 1, characterized in that: The trained expert gaze-guided visual Transformer-based chest X-ray classification model is tested using test images from a chest X-ray classification dataset with known expert gaze points, including: In the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points, the trained expert gaze-guided visual Transformer-based chest X-ray classification model is tested with known test images to evaluate the performance of the model; An expert gaze point embedding acquisition module, which is used to obtain the embedding vector of the expert gaze point; The prediction model building module is used to build a chest X-ray classification model based on visual Transformer guided by expert gaze points; Model training module, which is used to complete the end-to-end training of the chest X-ray classification model based on visual Transformer guided by expert gaze points using the chest X-ray classification datasets EG-CXR and OMSI with expert gaze points; The model testing module is used to test the trained expert gaze-guided visual Transformer-based chest X-ray classification model using known test images to evaluate the classification performance of the constructed model.
8. A chest X-ray classification method guided by expert gaze points, characterized in that: A chest X-ray classification model constructed by the chest X-ray classification model construction method of any one of claims 1 to 7 or the chest X-ray classification model construction system of claim 8 is used, and any image is used as the input of the model to output the chest X-ray classification result.
9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method of any one of claims 1-7 or 9.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method of any one of claims 1 to 6 or 8.