A medical image feature extraction model training method and a medical image classification method

Abstract

The present application relates to the technical field of medical image processing, and provides a medical image feature extraction model training method and a medical image classification method, which comprises the following steps: inputting a medical image into a medical image feature extraction model to obtain coarse-grained image features and fine-grained image features; inputting a medical report into a medical report feature extraction model to obtain report coarse-grained features and report fine-grained features; performing cross-attention calculation on the fine-grained image features and the report fine-grained features to obtain cross-modal fusion features; calculating coarse-grained cross-modal alignment loss based on the coarse-grained image features and the report coarse-grained features; calculating fine-grained cross-modal alignment loss based on the fine-grained image features and the cross-modal fusion features; and training the medical image feature extraction model and the medical report feature extraction model with the weighted sum of the losses as the target, so that the image features extracted by the medical image feature extraction model are more accurate.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a method for training a medical image feature extraction model and a method for classifying medical images. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The integration of deep learning technology with the medical field has yielded significant results in recent years. Neural network-based medical imaging diagnostic models have shown good performance in various disease classifications. A common strategy is to train the neural network using labeled medical data to learn on specific tasks. Neural networks, through forward propagation of input data using multiple hidden layers and nonlinear activation functions, automatically extract effective features and learn accurate feature representations. For example, convolutional neural networks have been applied to gastroscopy for polyp detection, ultrasound for breast lesion detection, and radiological image analysis. These medical imaging diagnostic models mostly follow end-to-end label-guided training. As the parameter scale of medical imaging diagnostic models increases, these models increasingly require fitting to large amounts of data to achieve high performance. The parameter scale of these models is in the millions (Mega) or billions (Giga) of data, and the training dataset may contain millions of samples; for example, the ImageNet dataset contains over 14 million labeled images covering more than 20,000 categories.

[0004] However, large-scale parametric models face significant challenges in the medical field. In the past, the end-to-end training paradigm relied on labeled data to guide the model, but the labeling of medical image data often needs to be done manually by professional physicians. This has led to a scarcity of high-quality medical image data, which limits the development of high-performance medical models.

[0005] To address this issue, some studies have proposed using medical reports paired with medical images as information introductory material, similar to medical labels. These reports are then pre-trained in a self-supervised manner to extract features from medical images, allowing for minimal fine-tuning in downstream tasks. Compared to labels created manually by physicians, medical reports paired with images are readily available. These reports are generated simultaneously with the images, written by radiologists, and provide professional analysis and opinions on the images. They support decision-making in medical diagnosis and are often extensively stored in existing hospital databases.

[0006] Medical imaging models that incorporate medical reports involve both image and text modalities. When designing frameworks to handle these two modalities, contrastive learning is a commonly used technique. The idea of ​​contrastive learning can be well applied to cross-modal situations. The CLIP framework performs contrastive learning on image and text modalities, aligning the information of the two modalities in the feature dimension.

[0007] Data augmentation is a crucial technique in implementing image feature learning frameworks. Among data augmentation techniques, leveraging information from model inference is a more sophisticated design; the MAE model blocks parts of the image during model encoding, encouraging the model to generate features from the blocked areas; adversarial training, based on gradients generated during model inference, introduces perturbations into the image, achieving targeted data augmentation.

[0008] Most existing learning frameworks transfer multimodal techniques from general domains to the medical field. However, cross-modal alignment in the medical field requires more granular interactions because medical imaging pathology is often only related to certain local areas in the images, and these diagnostic criteria are expressed in a few words in medical reports. Summary of the Invention

[0009] To address the technical problems existing in the background art, the present invention provides a training method for a medical image feature extraction model and a medical image classification method. Not only does the training process of the dual-modal collaborative model make the image features extracted by the medical image feature extraction model more accurate, but the fine-grained image feature extraction method adopted is based on attention guidance. By ranking the attention of image regions, it is possible to select fine-grained features, making the representation of detailed information more accurate.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] The first aspect of this invention provides a method for training a medical image feature extraction model, comprising:

[0012] Obtain paired medical images and medical reports;

[0013] Based on the medical images, coarse-grained image features and fine-grained image features are obtained through a medical image feature extraction model; based on the medical report, coarse-grained report features and fine-grained report features are obtained through a medical report feature extraction model; cross-attention calculation is performed on the fine-grained image features and the fine-grained report features to obtain cross-modal fusion features;

[0014] Based on coarse-grained image features and report coarse-grained features, coarse-grained cross-modal alignment loss is calculated; based on fine-grained image features and cross-modal fusion features, fine-grained cross-modal alignment loss is calculated; with the goal of minimizing the weighted sum of coarse-grained cross-modal alignment loss and fine-grained cross-modal alignment loss, the medical image feature extraction model and the medical report feature extraction model are trained to obtain a trained medical image feature extraction model;

[0015] The medical image feature extraction model performs aggregation operations on the feature maps obtained by the image encoder at the channel scale, then uses a sliding window to perform average aggregation, selects multiple window regions with the largest average aggregated feature values, and calculates the fine-grained image features.

[0016] Furthermore, the feature maps corresponding to the multiple window regions with the largest average aggregated feature values ​​are local region features, which are then projected into the fine-grained image features through a fully connected layer.

[0017] Furthermore, the medical image feature extraction model uses the image encoder to extract features from the medical image, then performs global average pooling and flattens the image to obtain the coarse-grained image features.

[0018] Furthermore, after the medical report feature extraction model vectorizes the medical report into words, it encodes the report using a report encoder to obtain the coarse-grained features of the report.

[0019] Furthermore, the medical report feature extraction model extracts word features from the output of the report encoder and then performs average aggregation to obtain the fine-grained features of the report.

[0020] A second aspect of the present invention provides a medical image classification method, comprising:

[0021] Acquire medical images to be classified;

[0022] Based on the medical image to be classified, the features of the medical image to be classified are obtained through a trained medical image feature extraction model; the features of the medical image to be classified include fine-grained image features and coarse-grained image features.

[0023] Based on the characteristics of the medical images to be classified, they are classified to obtain the category of the medical images to be classified.

[0024] The trained medical image feature extraction model is obtained using the medical image feature extraction model training method described in the first aspect.

[0025] A third aspect of the present invention provides a medical image feature extraction model training system, comprising:

[0026] The training data acquisition module is configured to acquire pairs of medical images and medical reports.

[0027] The cross-modal feature extraction module is configured to: obtain coarse-grained image features and fine-grained image features based on the medical images using a medical image feature extraction model; obtain report coarse-grained features and report fine-grained features based on the medical reports using a medical report feature extraction model; and perform cross-attention calculation on the fine-grained image features and report fine-grained features to obtain cross-modal fusion features.

[0028] The training module is configured to: calculate the coarse-grained cross-modal alignment loss based on coarse-grained image features and coarse-grained report features; calculate the fine-grained cross-modal alignment loss based on fine-grained image features and cross-modal fusion features; and train the medical image feature extraction model and the medical report feature extraction model with the goal of minimizing the weighted sum of the coarse-grained cross-modal alignment loss and the fine-grained cross-modal alignment loss, thereby obtaining a trained medical image feature extraction model.

[0029] The medical image feature extraction model performs aggregation operations on the feature maps obtained by the image encoder at the channel scale, then uses a sliding window to perform average aggregation, selects multiple window regions with the largest average aggregated feature values, and calculates the fine-grained image features.

[0030] A fourth aspect of the present invention provides a medical image classification system, comprising:

[0031] The image acquisition module is configured to acquire medical images to be classified.

[0032] The image feature extraction module is configured to: obtain the features of the medical image to be classified based on the medical image to be classified through a trained medical image feature extraction model; the features of the medical image to be classified include fine-grained image features and coarse-grained image features.

[0033] The classification module is configured to classify the medical image to be classified based on its features to obtain the category of the medical image to be classified.

[0034] The trained medical image feature extraction model is obtained using a medical image feature extraction model training system as described in the third aspect.

[0035] A fifth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the medical image feature extraction model training method described above.

[0036] A sixth aspect of the present invention provides a computer device including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein the processor executes the program to implement the steps of the medical image feature extraction model training method described above.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] This invention employs cross-modal interaction technology to simultaneously consider cross-modal alignment at both coarse-grained and fine-grained levels for medical images and medical reports. The dual-modal collaborative model training process fully learns the global and detailed semantic information of images and reports, making the image features extracted by the medical image feature extraction model more accurate, and achieving collaboration through shared information.

[0039] The image fine-grained feature extraction method adopted in this invention is based on attention guidance. By ranking the attention of image regions, it can select fine-grained features, obtain more detailed information of the image, and use the information during model inference to achieve data consistency regularization, making the representation of detailed information more accurate.

[0040] This invention employs contrastive learning pre-training technology to construct a pre-trained model framework capable of learning knowledge within the medical field. When targeting specific medical tasks or for field deployment, only minor fine-tuning is required to achieve high performance. By adopting a pre-training-fine-tuning paradigm, it is not necessary to train a specific model for each specific task, thereby reducing the framework training cost, minimizing model deployment and storage requirements, and improving service efficiency.

[0041] This invention is based on two readily available types of data: medical images and medical reports. It has fewer limitations in the practical deployment of clinical decision-making systems and can achieve lightweight deployment. Attached Figure Description

[0042] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0043] Figure 1 This is a flowchart of a medical image feature extraction model training method according to Embodiment 1 of the present invention;

[0044] Figure 2 This is a schematic diagram of an attention-guided dual-granularity cross-modal medical feature learning framework according to Embodiment 1 of the present invention;

[0045] Figure 3 This is a flowchart of medical image encoding according to Embodiment 1 of the present invention;

[0046] Figure 4This is a flowchart of cross-modal alignment according to Embodiment 1 of the present invention.

[0047] Figure 5 This is a schematic diagram of the structure of a computer device according to Embodiment 5 of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0049] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0050] Example 1

[0051] This embodiment provides a method for training a medical image feature extraction model.

[0052] This embodiment provides a training method for a medical image feature extraction model. The goal is to enhance the model's semantic alignment with medical images and medical reports at both coarse and fine granular levels, enabling the model to capture detailed information in the images and understand the relevant descriptions and analyses in the corresponding medical reports.

[0053] This embodiment provides a medical image feature extraction model training method that relies on an attention-guided, dual-granularity, cross-modal medical feature learning framework, such as... Figure 2 As shown.

[0054] The attention-guided dual-granularity cross-modal medical feature learning framework consists of three parts: medical image encoding, which extracts coarse-grained and fine-grained features from images, and uses an attention-guided selection module to denoise and select fine-grained features; medical report encoding, which extracts coarse-grained and fine-grained features from the report modality; and a cross-modal alignment module, which aligns the two modalities and trains the entire model through loss calculation.

[0055] The medical image encoding process is based on an encoder and two feature extraction modules at different granularities. Medical scenarios focus more on detailed information; therefore, local areas of the image may contain important information.

[0056] Medical report coding is also divided into feature extraction at two granularities: coarse-grained and fine-grained.

[0057] This embodiment provides a method for training a medical image feature extraction model, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0058] Step 1: Data preprocessing.

[0059] Step 101: Data cleaning.

[0060] The original paired medical images and medical reports are cleaned. For medical images, severely damaged, unclear, and abnormally angled image data need to be removed; for medical reports, noisy data such as missing values ​​and outliers need to be removed to ensure data quality.

[0061] Step 102: Dataset partitioning.

[0062] The dataset is randomly divided into training, validation, and test sets according to a certain ratio.

[0063] The validation set is used for hyperparameter tuning, and the test set is used to evaluate the generalization performance of the medical image feature extraction model.

[0064] Step 2: Training the medical image feature extraction model. Input the medical image into the medical image feature extraction model to obtain coarse-grained image features v. c and fine-grained image features v f The medical report is input into the medical report feature extraction model to obtain the coarse-grained features t of the report. c and report fine-grained features t f ; to refine image features v f and report fine-grained features t f Cross-attention calculation is performed to obtain cross-modal fusion features z; based on coarse-grained image features v c and report coarse-grained features t c Calculate the coarse-grained cross-modal alignment loss and based on fine-grained image features v f The cross-modal fusion feature z is used to calculate the fine-grained cross-modal alignment loss. The medical image feature extraction model and the medical report feature extraction model are trained with the goal of minimizing the weighted sum of the coarse-grained and fine-grained cross-modal alignment losses, resulting in a trained medical image feature extraction model. The medical image feature extraction model includes an image encoder (ResNet50) and an attention-guided fine-grained feature extraction module. The medical report feature extraction model includes a report encoder and an average aggregation module.

[0065] Step 201: Input paired medical images and medical reports into an attention-guided dual-granularity cross-modal medical feature learning framework. Each image and report is represented mathematically using an embedding method, and then encoded by the image encoder (ResNet50) and report encoder (BERT) respectively, yielding two modal coarse-grained features v. c and t c .

[0066] (1) Extract coarse-grained features from medical images, and select ResNet50 as the image encoder for the input image x. i via image encoder E img Encoded as v c ∈R d The encoding process can be represented as v c =E img (x i ), where v c It is obtained by global average pooling and flattening the feature map calculated by ResNet50 encoding, representing the global information of a medical image, and serving as a coarse-grained image feature v. c .

[0067] (2) Word vectorization was performed on medical reports paired with medical images, converting the report text into several word vectors. The word-piece method was used for word vectorization, which involves decomposing medical terms into sub-words, and vectorizing each sub-word individually. Compared to image segmentation based on pixels and regions, the dependencies between sentences and words in text contain important semantics; therefore, BERT, which can better capture contextual information, was chosen as the report encoder. BioclinicalBERT pre-trained weights were loaded to better extract medical-related features.

[0068] (3) Extract coarse-grained features from the medical report. For the input medical report x t Through the report encoder E rep Encoded as t c ∈R d The encoding process can be represented as t c =E rep (x t ), where t c It takes the CLS word vectors from the last hidden layer of BERT as coarse-grained features that contain global information about the report.

[0069] Step 202: Aggregate the feature maps obtained from the image encoder at the channel scale. Use a sliding window to select a 2×2 feature window, calculate the average feature value, obtain the average feature ranking of the windows, and select the top K windows as the fine-grained features of the image. f .

[0070] Fine-grained features are extracted from medical images. To obtain fine-grained region features of interest for medical tasks, an attention-guided fine-grained feature extraction module is introduced, such as... Figure 3As shown. First, the feature map A before the ResNet50 global average pooling layer is extracted as the initial local region feature, with a shape of C×H×W (2048×7×7), where C is the number of channels in the feature map. This can be viewed as dividing the image into H×W local regions, each represented as a C-dimensional feature. Next, the feature map is aggregated along the channel dimension:

[0071]

[0072] Among them, A i,j Let A represent the feature values ​​of each local region in each channel of feature map A. Through aggregation, an attention activation feature map M containing the attention-focused regions during the forward propagation of an image using a ResNet network is obtained. Then, a sliding window of size h×w is used to slide across feature map M with a step size of 1. At each step, features within the window are selected and averaged to obtain P window features. This operation is used to check the attention of local regions. The P window features are sorted, and the K window regions with the largest average aggregated feature values ​​are selected. These regions are the K sub-regions with the largest attention activation values ​​in a medical image, and their feature representations are the local image features of interest for the medical task and for noise removal. Finally, the obtained local region features are projected onto a fully connected layer to transform into v. f ∈R d As fine-grained image features, by using a sliding window to select important features on the attention activation feature map and removing regions with weak attention, the model can focus more on disease or pathology-related semantic information of medical images during subsequent alignment, while ignoring parts irrelevant to diagnosis. This allows the model to learn more accurate feature representations during pre-training and improve performance in downstream tasks.

[0073] Step 203, for t c Word aggregation is used to obtain the fine-grained feature t of the report. f .

[0074] Extracting fine-grained features from medical reports. Key analyses of images in medical reports are often embodied in several words; therefore, word-level features contain detailed information within the report. Since a word-piece method is used for word vectorization, to obtain the features of each word in the report, sub-words need to be recombined into the original word, and the features of the sub-words are then aggregated to represent the features of the original word. Considering the impact of BERT's self-attention mechanism on the correlation between words, feature extraction is performed on the last three layers of the BERT output hidden layer. After extracting word features, the features from the three layers are averaged and aggregated to obtain the final word-level feature t. fThe specific method for feature extraction from the last three hidden layers of BERT's output is as follows: BERT's attention layer consists of six layers, with the result of each layer serving as the input for the next layer. The output interface provided by BERT is used to obtain the results of the last three layers, which have a shape of 3×B×L×D, where B is the batch size, L is the sequence length, and D is the feature dimension. The average aggregation operation is calculated by adding the Dim dimensions and dividing by 3.

[0075] Step 204: Transfer fine-grained features v f and t f Cross-attention calculation is performed to obtain the cross-modal fusion feature z.

[0076] Cross-attention is used to compute fused features from two modalities. In medical scenarios, information from local regions of images and word-level information in reports have a crucial impact on medical diagnosis. Therefore, finer-grained cross-modal interaction between images and reports is beneficial to feature learning. After acquiring fine-grained features from both image and report modalities, a cross-attention module is introduced to achieve cross-modal interactive computation of these two fine-grained features. The internal structure of this module is as follows: Figure 4 As shown. For a pair of image and report samples, the extracted fine-grained features of the image are... It is a sequence containing k feature vectors, from which the fine-grained features of the report are extracted. Given a sequence containing m feature vectors, where the features in two sequences reside in the same feature space; calculate t. f and v f Weighted correlations are then transformed into cross-modal fusion features z using a linear transformation. i This feature is essentially a fine-grained image feature that takes into account all local text information and has a corresponding relationship with the original fine-grained image feature. Therefore, subsequent fine-grained alignment is to strengthen this correspondence.

[0077]

[0078] Where Q, K, and V are learnable parameter matrices, O is the projection layer of the transformed vector feature dimension, and d is the dimension of the sequence vector. The cross-attention mechanism allows all report fine-grained features to participate in the computation of image fine-grained features, and assigns weights to each feature vector based on attention. Compared to directly calculating the similarity between two fine-grained features to calculate weights, the attention mechanism considers the correlation between two sequences and assigns the degree of participation of each feature in the computation based on the strength of the correlation, ultimately obtaining image fine-grained features z that consider the semantic correspondence of report word-level features. i .

[0079] Step 205: Calculate v at the coarse-grained level using InfoNCE Loss. cand t c The loss is calculated at a fine-grained level using InfoNCE Loss. f And the loss of z.

[0080] (1) Achieving coarse-grained cross-modal alignment. Strengthening the correspondence between the two modalities helps the image encoder understand image semantics, thereby learning accurate feature representations and demonstrating excellent performance in downstream tasks. By designing a proxy task that requires the model to complete the correspondence between medical images and medical reports, the medical image encoder can better understand the semantics of medical images. Therefore, InfoNCE Loss is adopted as the loss function for the coarse-grained cross-modal alignment proxy task. This reduces the distance between positive samples in the feature space and increases the distance between negative samples in the feature space. Paired images and reports are considered positive samples, and unpaired images and reports are considered negative samples. Due to existing hardware limitations, all unpaired images and reports within a training batch are considered negative samples and participate in the loss function calculation.

[0081] Coarse-grained cross-modal alignment loss includes:

[0082]

[0083] τ1 is a temperature hyperparameter used to control the discrimination of negative samples. It is a coarse-grained image feature of the i-th image report pair within a training batch. This refers to the coarse-grained report features paired with the images. The numerator represents the coarse-grained feature similarity between paired images and reports, and the denominator represents the coarse-grained feature similarity between all unpaired images and reports. By optimizing this function, alignment between corresponding samples of the image and report modalities is achieved. To ensure the correspondence between modalities, in addition to using the coarse-grained feature similarity of the specified image and the coarse-grained feature similarity of all reports that are its negative counterparts as the denominator, a further loss is set using the coarse-grained feature similarity of the specified report and the coarse-grained feature similarity of all images that are its negative counterparts as the denominator.

[0084] (2) Achieving fine-grained cross-modal alignment. Coarse-grained cross-modal alignment only strengthens the correspondence of global semantic information without considering the alignment of local semantic information. The fine-grained image features calculated by the cross-attention module contain the local semantic information in the corresponding report, corresponding to the local image region features extracted by the image encoder and attention-guided selection module. Therefore, a proxy task loss function similar to that of coarse-grained cross-modal alignment, namely InfoNCE Loss, is adopted to achieve fine-grained cross-modal alignment. For an input medical image, K local image region features and K corresponding cross-modal image fine-grained features can be generated. Corresponding feature pairs are positive samples to each other, and all other feature pairs are negative samples to each other. Therefore, for a training batch of size N, N×K positive sample feature pairs and (N×K)×(N×K-1) negative sample feature pairs can be generated. The goal of the proxy task is to reduce the distance between positive sample feature pairs in the feature space and increase the distance between negative sample feature pairs in the feature space.

[0085] Fine-grained cross-modal alignment loss includes:

[0086]

[0087] Where S = N × K, and τ² is the temperature hyperparameter. The numerator represents the similarity distance between each pair of positive sample feature pairs, and the denominator represents the similarity distance between all sample pairs. Reducing the loss means that the numerator becomes smaller, i.e., positive sample feature pairs tend to be more similar; the denominator becomes larger, i.e., the distance between negative sample feature pairs gradually increases, ultimately strengthening the correspondence of fine-grained semantic information and achieving fine-grained cross-modal alignment. It is worth noting that, similar to coarse-grained cross-modal alignment, the loss function is designed bidirectionally to ensure the correspondence between modalities.

[0088] Step 206: Construct the total loss function, optimize the loss function using the Adam optimizer, update the model parameters within the framework, and save the updated model parameter weight file.

[0089] A total loss function is constructed to enable framework training. By jointly optimizing the pre-trained models of coarse-grained cross-modal alignment loss and fine-grained cross-modal alignment loss, the image encoder's understanding of coarse-grained and fine-grained semantic information is enhanced, and feature representations with generalization capabilities are learned.

[0090] L total =λ1*(L v c 2 g t +L t c 2 g v )+λ2*(Lv f 2 g z +L z f 2 g v )

[0091] Here, λ1 and λ2 are the weight hyperparameters that balance the two losses. The overall goal of the ADCRL framework is to optimize this loss function.

[0092] Step 207: Load the saved model parameter weight file into an image encoder with the same structure and perform a test.

[0093] Example 2

[0094] This embodiment provides a medical image classification method, characterized by including:

[0095] Step 1) Model deployment.

[0096] (1) Model weight saving: Save the weight file of the trained medical image feature extraction model and provide necessary documentation.

[0097] (2) Model loading: Load the weight file of the medical image feature extraction model into the medical image feature extraction model to ensure that the structure of the new model is consistent with that during training. The model can be integrated into the medical system, and only provides training interface and weight loading interface to ensure a certain degree of privacy.

[0098] (3) Model training: end-to-end training is carried out according to the different medical tasks involved, and the input data is medical images.

[0099] (4) Model testing: The model is tested using data from the database to ensure its accuracy, and public datasets can also be used for evaluation.

[0100] (5) Model Deployment: The model will be officially deployed online to provide doctors with auxiliary diagnostic and treatment decision support. At the same time, the online prediction results of the model will be continuously monitored to ensure its stability and rationality.

[0101] (6) Model application:

[0102] Acquire medical images to be classified;

[0103] Based on the medical image to be classified, the features of the medical image to be classified are obtained through a trained medical image feature extraction model; the features of the medical image to be classified include fine-grained image features and coarse-grained image features.

[0104] Based on the characteristics of the medical images to be classified, they are classified to obtain the category of the medical images to be classified.

[0105] The trained medical image feature extraction model was obtained using the medical image feature extraction model training method described in Example 1.

[0106] Example 3

[0107] This embodiment provides a medical image feature extraction model training system, which specifically includes:

[0108] The training data acquisition module is configured to acquire pairs of medical images and medical reports.

[0109] The cross-modal feature extraction module is configured to: obtain coarse-grained image features and fine-grained image features based on the medical images using a medical image feature extraction model; obtain report coarse-grained features and report fine-grained features based on the medical reports using a medical report feature extraction model; and perform cross-attention calculation on the fine-grained image features and report fine-grained features to obtain cross-modal fusion features.

[0110] The training module is configured to: calculate the coarse-grained cross-modal alignment loss based on coarse-grained image features and coarse-grained report features; calculate the fine-grained cross-modal alignment loss based on fine-grained image features and cross-modal fusion features; and train the medical image feature extraction model and the medical report feature extraction model with the goal of minimizing the weighted sum of the coarse-grained cross-modal alignment loss and the fine-grained cross-modal alignment loss, thereby obtaining a trained medical image feature extraction model.

[0111] The medical image feature extraction model performs aggregation operations on the feature maps obtained by the image encoder at the channel scale, then uses a sliding window to perform average aggregation, selects multiple window regions with the largest average aggregated feature values, and calculates the fine-grained image features.

[0112] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0113] Example 4

[0114] This embodiment provides a medical image classification system, which specifically includes:

[0115] The image acquisition module is configured to acquire medical images to be classified.

[0116] The image feature extraction module is configured to: obtain the features of the medical image to be classified based on the medical image to be classified through a trained medical image feature extraction model; the features of the medical image to be classified include fine-grained image features and coarse-grained image features.

[0117] The classification module is configured to classify the medical image to be classified based on its features to obtain the category of the medical image to be classified.

[0118] The trained medical image feature extraction model was obtained using a medical image feature extraction model training system as described in Example 3.

[0119] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment two, and their specific implementation process is the same, so it will not be repeated here.

[0120] Example 4

[0121] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the medical image feature extraction model training method described in Embodiment 1 above.

[0122] Example 5

[0123] This embodiment provides a computer device, such as... Figure 5 As shown, the system includes a computer-readable storage medium 1003, a processor 1001, a communication interface 1002, and a computer program stored on the computer-readable storage medium 1003 and executable on the processor 1001. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and send data. When the processor 1001 executes the program, it implements the steps in the medical image feature extraction model training method described in Embodiment 1 above.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a medical image feature extraction model, characterized in that, include: Obtain paired medical images and medical reports; Based on the medical images, coarse-grained image features and fine-grained image features are obtained through a medical image feature extraction model; based on the medical report, coarse-grained report features and fine-grained report features are obtained through a medical report feature extraction model; cross-attention calculation is performed on the fine-grained image features and the fine-grained report features to obtain cross-modal fusion features; Based on coarse-grained image features and report coarse-grained features, calculate coarse-grained cross-modal alignment loss; based on fine-grained image features and cross-modal fusion features, calculate fine-grained cross-modal alignment loss. The medical image feature extraction model and the medical report feature extraction model are trained with the goal of minimizing the weighted sum of coarse-grained cross-modal alignment loss and fine-grained cross-modal alignment loss, resulting in a well-trained medical image feature extraction model. The medical image feature extraction model performs aggregation operations on the feature maps obtained by the image encoder at the channel scale, then uses a sliding window to perform average aggregation, selects multiple window regions with the largest average aggregated feature values, and calculates the fine-grained image features.

2. The method for training a medical image feature extraction model as described in claim 1, characterized in that, The feature maps corresponding to the multiple window regions with the largest average aggregated feature values ​​are local region features, which are then projected into the fine-grained image features through a fully connected layer.

3. The method for training a medical image feature extraction model as described in claim 1, characterized in that, The medical image feature extraction model uses the image encoder to extract features from the medical image, then performs global average pooling and flattens the image to obtain the coarse-grained image features.

4. The method for training a medical image feature extraction model as described in claim 1, characterized in that, The medical report feature extraction model vectorizes the medical report into words and then encodes them using a report encoder to obtain the coarse-grained features of the report.

5. The method for training a medical image feature extraction model as described in claim 1, characterized in that, The medical report feature extraction model extracts word features from the output of the report encoder and then performs average aggregation to obtain the fine-grained features of the report.

6. A medical image classification method, characterized in that, include: Acquire medical images to be classified; Based on the medical image to be classified, the features of the medical image to be classified are obtained through a trained medical image feature extraction model; the features of the medical image to be classified include fine-grained image features and coarse-grained image features. Based on the characteristics of the medical images to be classified, they are classified to obtain the category of the medical images to be classified. The trained medical image feature extraction model is obtained using the medical image feature extraction model training method described in any one of claims 1-5.

7. A medical image feature extraction model training system, characterized in that, include: The training data acquisition module is configured to acquire pairs of medical images and medical reports. The cross-modal feature extraction module is configured to: obtain coarse-grained image features and fine-grained image features based on the medical images using a medical image feature extraction model; obtain report coarse-grained features and report fine-grained features based on the medical reports using a medical report feature extraction model; and perform cross-attention calculation on the fine-grained image features and report fine-grained features to obtain cross-modal fusion features. The training module is configured to: compute coarse-grained cross-modal alignment loss based on coarse-grained image features and reported coarse-grained features; and compute fine-grained cross-modal alignment loss based on fine-grained image features and cross-modal fusion features. The medical image feature extraction model and the medical report feature extraction model are trained with the goal of minimizing the weighted sum of coarse-grained cross-modal alignment loss and fine-grained cross-modal alignment loss, resulting in a well-trained medical image feature extraction model. The medical image feature extraction model performs aggregation operations on the feature maps obtained by the image encoder at the channel scale, then uses a sliding window to perform average aggregation, selects multiple window regions with the largest average aggregated feature values, and calculates the fine-grained image features.

8. A medical image classification system, characterized in that, include: The image acquisition module is configured to acquire medical images to be classified. The image feature extraction module is configured to: obtain the features of the medical image to be classified based on the medical image to be classified through a trained medical image feature extraction model; the features of the medical image to be classified include fine-grained image features and coarse-grained image features. The classification module is configured to classify the medical image to be classified based on its features to obtain the category of the medical image to be classified. The trained medical image feature extraction model is obtained using the medical image feature extraction model training system described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the medical image feature extraction model training method as described in any one of claims 1-5.

10. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the medical image feature extraction model training method as described in any one of claims 1-5.

Images