A training method, device and equipment for medical image text alignment model

Abstract

The present invention belongs to the field of medical data processing, and specifically relates to a training method, device and equipment for a medical image text alignment model; the method includes obtaining medical images, enhanced views, medical texts and enhanced texts; using an image encoder to extract image features, and using a text encoder to extract text features; using multi-view supervised image text alignment to obtain more detailed representations of medical images and medical texts; using multi-scale self-attention and cross-attention integrated alignment strategies to understand the contextual information of each modality through the correlation of internal features without the need for external information, and also to learn cross-modal information. Using high-confidence clustering information to guide the positive and negative sample construction algorithm can deeply mine high-order semantic information, so that the pre-trained model can be used for accurate pairing between medical images and their paired medical reports, as well as the ability to accurately locate lesion areas in images, to assist in medical condition diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of medical data processing, and in particular relates to a training method, device and equipment for a medical image text alignment model. Background Art

[0002] In recent decades, when large-scale labeled datasets became available, deep learning techniques have significantly improved medical image understanding. However, assembling such large annotated data is expensive and time-consuming. As an alternative, learning image representations directly from radiology reports with medical images has gradually become the mainstream paradigm for medical image understanding models.

[0003] In the medical field, the application of deep learning technology is gradually shifting from simple image processing to multimodal learning that combines image and text information. For example, Chinese patent CN117809794A discloses a medical report generation method, system and terminal based on cross-view semantic alignment. This patent application realizes fine-grained semantic alignment between multiple images through a cross-view semantic alignment network, thereby improving the feature extraction and information integration capabilities of multiple images, making the generated medical report content more accurate. The auxiliary diagnosis of diseases involving medical images depends on the detailed analysis of image features and the association with semantic information in clinical reports. However, existing methods are still insufficient in fully mining the high-level semantic features of image text, which limits the performance and application scope of the model.

[0004] First, existing methods often lack the effective use of the rich medical knowledge contained in clinical report texts, which contain key diagnostic information and clinical experience, but are often ignored or not fully mined and integrated. Second, existing methods often use separate model structures when processing image and text data, which leads to insufficient cross-modal information fusion. The intrinsic connection between images and texts is not effectively utilized, making it impossible for the model to fully utilize the complementary information between the two modalities, thereby affecting the understanding and expression of high-level semantic features. In addition, fine-grained features in medical images are crucial for the accurate diagnosis of pneumonia in adults. However, existing methods may focus too much on the overall feature extraction of images and ignore the importance of local details. This bias may lead to the model's inability to capture subtle changes and key signals related to the diagnosis of pneumonia. Summary of the invention

[0005] The purpose of the present invention is to provide a training method, device and equipment for a medical image text alignment model, aiming to fully explore the fine-grained features of medical image text while paying more attention to the high-order semantic information within the class, and to provide an efficient and reliable solution for the field of medical image analysis in terms of automated auxiliary diagnosis and precision medicine.

[0006] In a first aspect of the present invention, the present invention provides a training method for a medical image text alignment model, the method comprising:

[0007] Acquire a medical image and an enhanced view of the medical image, a medical text, and an enhanced text corresponding to the medical text;

[0008] Using an image encoder to process the medical image and the enhanced view respectively, to obtain global features and local features of the medical image, and global features and local features of the enhanced view;

[0009] Using a text encoder to process the medical text and the enhanced text respectively, to obtain global features and local features of the medical text, and global features and local features of the enhanced text;

[0010] A nonlinear projection layer is used to convert each global feature and local feature into a low-dimensional embedding to obtain the corresponding global embedding feature and local embedding feature;

[0011] The global embedding features of medical images and the corresponding enhanced views are aligned using similarity metrics, and the global embedding features of medical texts and the corresponding enhanced texts are aligned to construct a multi-view self-supervised image-text alignment loss.

[0012] The local embedding features of the medical image and the local embedding features of the medical text are processed by the self-attention unit of the integrated attention module to obtain the self-attention features of the medical image and the self-attention features of the medical text. The cross-attention unit of the integrated attention module is used to process the self-attention features of the medical text and the self-attention features of the medical image to obtain the cross-modal text embedding features and the cross-modal image embedding features. The cross-modal text embedding features and the cross-modal image embedding features are used to construct the cross-modal tag alignment loss.

[0013] A fusion module is used to fuse the global embedding features of medical images and the global embedding features of medical texts to obtain cross-modal fusion features. The confidence clustering method is used to cluster the cross-modal fusion features, and the positive and negative sample losses guided by high-confidence clustering information are constructed.

[0014] According to the multi-view self-supervised image-text alignment loss, cross-modal tag alignment loss, and positive and negative sample loss guided by high-confidence clustering information, the medical image-text alignment model consisting of image encoder, text encoder, nonlinear projection layer, and integrated attention module is trained until the preset conditions are met.

[0015] In a second aspect of the present invention, the present invention further provides a training device for a medical image text alignment model, the device comprising:

[0016] An image acquisition unit, used for acquiring a medical image and an enhanced view of the medical image;

[0017] A text acquisition unit, used to acquire a medical text and an enhanced text corresponding to the medical text;

[0018] An image input unit, used to input the medical image into an image encoder to obtain global features and local features of the medical image, and input the enhanced view into an image encoder to obtain global features and local features of the enhanced view;

[0019] A text input unit, used for inputting the medical text into a text encoder to obtain global features and local features of the medical text, and inputting the enhanced text into a text encoder to obtain global features and local features of the enhanced text;

[0020] A feature embedding unit, used to convert the corresponding global features and local features into low-dimensional embeddings using a nonlinear projection layer, to obtain embedded features of medical images, embedded features of enhanced views, embedded features of medical texts, and embedded features of enhanced texts; a first computing unit, used to align the embedded features of medical images and the embedded features of corresponding enhanced views using a similarity metric, and to align the embedded features of medical texts and the embedded features of corresponding enhanced texts, to obtain a multi-view self-supervised image-text alignment loss;

[0021] The second computing unit is used to calculate the soft matching between the medical image and the medical text by using the cross-attention mechanism to obtain the cross-modal marker alignment loss;

[0022] The third computing unit is used to fuse the embedded features of the medical image and the embedded features of the medical text by using a fusion module to obtain a cross-modal fusion feature, and cluster the cross-modal fusion feature by using a confidence clustering method to obtain a positive and negative sample loss guided by high-confidence clustering information;

[0023] The execution unit is used to train the medical image-text alignment model composed of the image encoder, the text encoder, the nonlinear projection layer, and the integrated attention module according to the multi-view self-supervised image-text alignment loss, the cross-modal tag alignment loss, and the positive and negative sample loss guided by the high-confidence clustering information until the preset conditions are met.

[0024] In the third aspect of the present invention, the present invention also provides an electronic device, comprising: one or more processors; a storage device, on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the medical image text alignment model described in any one of the first aspects of the present invention.

[0025] The present invention has the following beneficial effects:

[0026] The present invention adopts random cropping technology and random shuffling strategy, which can effectively improve the diversity of enhanced images, effectively simulate the natural variability of medical texts, and increase the adaptability of the model to changes in text structure. The present invention uses multi-view supervised image text alignment to obtain a more detailed representation of medical images and medical texts; the present invention uses multi-scale self-attention and cross-attention integrated alignment strategies to understand the contextual information of each modality through the correlation of internal features without the need for external information, and can also learn cross-modal information. The present invention uses high-confidence clustering information to guide the positive and negative sample construction algorithm to deeply mine high-order semantic information, so that the pre-trained model can be used for accurate pairing between medical images and their paired medical reports, as well as the ability to accurately locate lesion areas in images, to assist in medical disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a training framework structure diagram of a medical image text alignment model provided by an embodiment of the present invention;

[0028] Figure 2 It is a flow chart of a training method of a medical image text alignment model provided by an embodiment of the present invention;

[0029] Figure 3 It is a structural diagram of a training device for a medical image text alignment model provided by an embodiment of the present invention;

[0030] Figure 4 It is a schematic diagram of the performance of semantic segmentation of the model after training in an embodiment of the present invention on SIIM and RSNA datasets;

[0031] Figure 5 It is a schematic diagram of the classification results of the training model on four data sets: RSNA, CheXpert, COVIDx, and Vindr after the training of an embodiment of the present invention is completed. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] In order to more intuitively demonstrate the training method, device and equipment of the present invention, the present invention adopts the following Figure 1The training framework shown is illustrated. In an embodiment of the present invention, the embedded features of the medical image and the embedded features Ev of the enhanced image are extracted by the image encoder; the embedded features of the medical text and the embedded features Et of the enhanced text are extracted by the text encoder; the multi-view self-supervised image-text alignment algorithm (MVSSV), the multi-scale self-attention and cross-attention integrated alignment algorithm (SCIA), and the high-confidence clustering information guided positive and negative sample construction algorithm (HCCSC) are used, and the high-order semantic information is deeply mined at the same time through a multi-step alignment strategy, so that the pre-trained model can be used for the accurate pairing between medical images and their paired reports, as well as the accurate positioning of the lesion area in the image, to assist in the diagnosis of medical conditions.

[0034] like Figure 2 As shown, see Figure 2 , Figure 2 It is a flow chart of a training method for a medical image text alignment model provided by an embodiment of the present invention. The method includes steps 101 to 108.

[0035] 101. Acquire a medical image and an enhanced view of the medical image, a medical text, and an enhanced text corresponding to the medical text;

[0036] In an embodiment of the present invention, the medical image is a medical image with a corresponding medical text. The medical text refers to the medical report text corresponding to the medical image, for example, it can be a test report text. In a possible implementation, the medical image and the medical text can be a chest X-ray image and corresponding report text from RSNA, CheXpert, Covidx, SIIM, etc. (datasets); it can be found that the medical image and the medical text appear in pairs, and the two complement each other, and the medical text can be used to illustrate the medical image.

[0037] In an embodiment of the present invention, the method for acquiring the enhanced view of the medical image includes scaling the medical image to a fixed size; using a random cropping technique to obtain a plurality of block images of different sizes for the medical image of the fixed size; using Gaussian blur within a preset standard deviation range to process each block image, and randomly flipping the processed block image with a certain probability to generate an enhanced view of the medical image.

[0038] In the preferred embodiment of the present invention, the present invention focuses on the use of front view medical images for analysis. In order to ensure consistency and reduce the computational burden, all selected front view images are first scaled to a uniform resolution size, i.e., 256×256 pixels, and then a random cropping technique is used to obtain a 224×224 image. Then we introduce Gaussian blur, use a convolution kernel size of 9, and set the standard deviation to a range of 0.1 to 2 while randomly flipping the image with a certain probability.

[0039] In an embodiment of the present invention, the method for acquiring the enhanced text corresponding to the medical text includes segmenting the medical text, processing each medical text segment using a random shuffling strategy, and generating the enhanced text of the medical text.

[0040] In the preferred embodiment of the present invention, for text, since the data sets in the medical field are often small, effective text enhancement can bring huge benefits. At the same time, in the field of radiology, the sentences in the "findings" and "impressions" parts that contain detailed descriptions and summaries of radiological findings are usually consistent at the sentence level. Therefore, in order to enhance the expressiveness of text data and simulate actual changes in medical reports, the present invention implements a random shuffling strategy for the sentences in each text segment. According to the acquired text and the corresponding language identifier, we first divide the text into independent, semantically complete sentence sequences, and then in order to avoid the inconsistency of semantic logic caused by random shuffling, the present invention evaluates the correlation between sentences by calculating the semantic correlation, and finally selects the combination with the highest correlation from the multiple sentence sequence combinations generated by random shuffling. These combinations meet the semantic coherence and logic while maintaining a certain degree of randomness. This strategy effectively simulates the natural variability of medical text and increases the adaptability of the model to changes in text structure.

[0041] 102. Use an image encoder to process the medical image and the enhanced view respectively to obtain global features and local features of the medical image and global features and local features of the enhanced view;

[0042] In an embodiment of the present invention, the image encoder is used to extract image features specific to medical images and enhanced views, that is, global features and local features belonging to medical images, and global features and local features belonging to enhanced views. In this embodiment, ResNet50 pre-trained on ImageNet is used as the image encoder, and the i-th medical image and the corresponding enhanced view are used as inputs of the image encoder to obtain the global features and local features of the medical image and the global features and local features of the enhanced view.

[0043] 103. Using a text encoder to process the medical text and the enhanced text respectively, to obtain global features and local features of the medical text, and global features and local features of the enhanced text;

[0044] In an embodiment of the present invention, the text encoder is used to respectively extract text features unique to medical text and enhanced text, that is, global features and local features belonging to medical text, as well as global features and local features belonging to enhanced text. In an embodiment of the present invention, CXR-BERT is used as a text encoder, and the i-th text segment and the corresponding text segment are used as inputs of the text encoder to obtain the global features and local features of the medical text as well as the global features and local features of the enhanced text.

[0045] 104. Use a nonlinear projection layer to convert each global feature and local feature into a low-dimensional embedding to obtain corresponding global embedding features and local embedding features;

[0046] In the embodiment of the present invention, two nonlinear projection layers are used to convert the global features and the local features into normalized low-dimensional embeddings, and each embedding feature is represented in turn as follows:

[0047] V g，i , V l，i =ρ(E v (x v，i ))

[0048] t g，i , t l，i =ρ(E t (x t，i ))

[0049]

[0050]

[0051] Among them, ρ represents the nonlinear projection layer, E v represents the image encoder, E t represents the text encoder, V g，i is the global embedding feature of medical images, V l，i is the local embedding feature of the medical image, To enhance the global embedding features of the view, To enhance the local embedding features of the view; g，i is the global embedding feature of medical text, t l，i is the local embedding feature of medical text, To enhance the global embedding features of text, To enhance the local embedding features of text.

[0052] 105. Use similarity measurement to align the global embedding features of medical images with the global embedding features of the corresponding enhanced views, align the global embedding features of medical texts with the global embedding features of the corresponding enhanced texts, and construct a multi-view self-supervised image-text alignment loss;

[0053] In an embodiment of the present invention, in order to obtain more detailed visual and textual representations, visual and textual invariant learning is used as a parallel goal in the medical vision-language pre-training process; in an embodiment of the present invention, enhanced images and enhanced texts are regarded as positive samples, and the corresponding medical images and medical texts are regarded as negative samples, and the cosine similarity of the two is calculated in the embedding space while the scale of the similarity distribution is controlled by adjusting the temperature parameter; specifically, the enhanced view is taken as a positive sample, the medical image is taken as a negative sample, the enhanced text is taken as a positive sample, and the medical text is taken as a negative sample; the similarity between the medical image and the enhanced view in the embedding space, and the similarity between the medical text and the enhanced text in the embedding space are calculated respectively, and the scale of the similarity distribution is controlled by adjusting the temperature parameter to obtain the multi-view self-supervised image-text alignment loss.

[0054] The above process can be expressed as:

[0055]

[0056]

[0057]

[0058] Among them, L VVA represents the image alignment loss, L TTA Represents text alignment loss; N represents the number of samples in a batch; Represents the global embedding feature V of the medical image g，i Global embedding features with enhanced views The cosine similarity of Represents the global embedding feature t of medical text g，i Global embedding features with enhanced text The cosine similarity of σ1 represents the first temperature hyperparameter, and sim represents the similarity function.

[0059] 106. Using the self-attention unit of the integrated attention module to process the local embedding features of the medical image and the local embedding features of the medical text, obtain the medical image self-attention features and the medical text self-attention features, using the cross-attention unit of the integrated attention module to process the medical text self-attention features and the medical image self-attention features, obtain the cross-modal text embedding features and the cross-modal image embedding features; using the cross-modal text embedding features and the cross-modal image embedding features, construct the cross-modal tag alignment loss;

[0060] In an embodiment of the present invention, in order to solve the problem that the traditional technology uses cross-attention-based label alignment, lacks the correlation of internal features, and causes inaccurate labels after alignment, the present invention adopts an effective label alignment strategy. First, for the i-th medical image and medical text pair, the present invention uses a self-attention unit to process the local embedding features of the medical image and the local embedding features of the medical text, respectively, learns the context information of the local embedding features of the medical image and the local embedding features of the medical text, and obtains the self-attention features of the medical image and the self-attention features of the medical text; uses a cross-attention mechanism to process the self-attention features of the medical text and the self-attention features of the medical image, learns the medical image information focused on by the self-attention features of the medical text, and the medical text information focused on by the self-attention features of the medical image, and obtains cross-modal text embedding features and cross-modal image embedding features; uses a contrast loss function to construct a cross-modal text loss and a cross-modal image loss; and superimposes the cross-modal text loss and the cross-modal image loss to obtain a cross-modal label alignment loss.

[0061] In the above embodiment, when the self-attention unit is used to learn the context information of the local embedding features of the medical image and the local embedding features of the medical text, the model can understand the context information of each modality through the correlation of internal features without external information, and fully extract the embedding feature information of each modality; the cross-attention unit is used to learn the medical image information that the medical text self-attention feature pays attention to, so that the local embedding features of the medical image pay attention to all text tag embedding information, and the local embedding features of the medical text pay attention to all image embedding information, so as to calculate the corresponding cross-modal embedding features. The above process can be expressed as:

[0062]

[0063]

[0064]

[0065] Among them, SA represents the self-attention function, Represents the self-attention features of medical images; represents the self-attention feature of medical text; S1 represents the number of text sentences; O represents the cross attention function, represents the cross attention weight, Q, K and V represent the query, key and value respectively, and N represents the number of samples in a batch; represents the self-attention feature of the medical image representing the k-th image patch, Represents the medical text self-attention feature of the k-th text segment.

[0066] For the calculated cross-modal text embedding, the present invention adopts InfoNCE loss to make Close to its corresponding cross-modal text embedding, and away from other embeddings. Therefore, for the cross-modal text loss, its loss function L CIA It can be expressed as:

[0067]

[0068] Among them, σ2 represents the temperature hyperparameter, sim(·) represents the cosine similarity calculation, represents the self-attention feature of medical images, Represents cross-modal text embedding features.

[0069] Similarly, for the jth text token embedding of the i-th text, the cross-modal image embedding is calculated using the above steps: And through InfoNCE loss Close to its corresponding cross-modal text embedding, and away from other embeddings. The above process can be expressed as:

[0070]

[0071] Therefore, for the cross-modal image loss, the loss function L CTA It can be expressed as:

[0072]

[0073] Among them, S2 represents the number of image patches; therefore, the ultimate goal of our multi-scale self-attention and cross-attention integrated alignment module is the combination of the two, expressed as:

[0074]

[0075] 107. A fusion module is used to fuse the global embedding features of medical images and the global embedding features of medical texts to obtain cross-modal fusion features. The confidence clustering method is used to cluster the cross-modal fusion features to construct positive and negative sample losses guided by high-confidence clustering information.

[0076] In an embodiment of the present invention, the present invention adopts a fuzzy C-means clustering algorithm to cluster cross-modal fusion features to obtain the confidence score and clustering information of each cross-modal fusion feature; the high-confidence cross-modal fusion features under the same cluster are used as positive samples, the high-confidence clustering information is used as supervision, and the cross-modal fusion features with high confidence scores are used to construct the positive sample loss; the cross-modal fusion features of the cluster centers with high confidence under different clusters are regarded as negative samples, and the centers with high confidence scores are used to construct the negative sample loss.

[0077] Specifically, existing methods usually directly regard all non-positive samples as negative samples, which often leads to the problem of false negatives. In order to alleviate this problem, the present invention improves the discrimination ability and reliability of positive and negative samples by mining high-confidence clustering information. Specifically, the present invention first projects the features of the two modalities of image and text into a low-dimensional space and fuses the features of the two. This process can be expressed by the formula:

[0078]

[0079] Among them, V g,i is the global embedding feature of medical images, t g,i is the global embedding feature of medical text, E i is the fusion feature; then we perform the Fuzzy c-means clustering algorithm on the fused features to obtain the clustering result. This process can be expressed by the formula:

[0080]

[0081]

[0082] Among them, K represents the number of clusters, U i,j represents the membership matrix, m is the fuzzy factor, which controls the fuzziness of the membership, C p Indicates the center of the cluster. Subsequently, in order to obtain more reliable clustering information, the present invention defines a confidence score, and based on the confidence score and a preset threshold T, determines the index set h of high confidence samples. This process can be expressed by the formula:

[0083]

[0084] h={h1,h2,...,h i ,...}

[0085] Among them, h i Indicates that the i-th sample belongs to the set of T samples with the highest confidence score. T is a threshold used to determine which samples are considered to have high confidence. For example, in this embodiment, it is set to T = 0.7. Therefore, based on these high-confidence sample indexes and their clustering information, a positive and negative sample construction strategy can be executed.

[0086] The process of constructing positive samples mainly includes three steps. First, high-confidence samples of the two modalities of image and text are selected. Then, the samples are divided into K non-overlapping clusters (p=q=(1,2,...,k)) according to the corresponding clustering information. Then, mean square error loss is performed on the positive samples of the two modalities. With high-confidence clustering information as supervision, the model can obtain more discriminative positive samples. This process can be expressed as:

[0087]

[0088]

[0089] in, represents the embedded features of the i-th high confidence sample in the p-th cluster, and np represents the number of high confidence samples in the p-th cluster.

[0090] The construction of negative samples mainly includes two steps. First, we calculate the centers of high-confidence samples of the two modalities, image and text. We regard different high-confidence centers as negative samples. Then we define L neg is the cosine similarity between different centers of high confidence embedding. This process can be expressed as:

[0091]

[0092]

[0093] in, Represented as the center of the i-th high confidence sample in the p-th cluster of the image modality, Denoted as the center of the i-th high confidence sample in the q-th cluster of the text modality.

[0094] Therefore, the positive and negative sample losses guided by high-confidence clustering information can be defined as:

[0095] L HCCSC =L pos +αL neg

[0096] Among them, L pos represents the positive sample loss, α represents the weight factor, L neg represents the negative sample loss.

[0097] 108. According to the multi-view self-supervised image-text alignment loss, cross-modal tag alignment loss, and positive and negative sample losses guided by high-confidence clustering information, the medical image-text alignment model consisting of image encoder, text encoder, nonlinear projection layer, and integrated attention module is trained until the preset conditions are met.

[0098] In an embodiment of the present invention, the multi-view self-supervised image-text alignment loss enables the model to capture richer and more detailed information of the data itself and improve the expressiveness of features. The cross-modal tag alignment loss enables the model to utilize the feature correlation within the image and text, integrate contextual information, and enhance the comprehensive understanding of medical images and text descriptions. The positive and negative sample loss guided by high-confidence clustering information can enhance the model's recognition ability of positive samples and the distinction ability of negative samples, reduce the risk of misclassification as false negative samples, and improve the accuracy and robustness of the model.

[0099] After completing the training of the image encoder, the text encoder, the nonlinear projection layer, and the integrated attention module, steps 101 to 108 are repeated until a preset condition is met, wherein the preset condition is a condition for completing the training. The preset condition may be, for example, the number of trainings or a numerical condition satisfied by the loss function.

[0100] In actual application, the target medical image is input into the trained medical image-text alignment model. After being processed by the image encoder, nonlinear projection layer, integrated attention module and other structures, the medical report text corresponding to the target medical image can be output.

[0101] See also Figure 3 ,like Figure 3 As shown, based on the training method of the medical image text alignment model provided by the above method embodiment, the embodiment of the present invention also provides a training device for the medical image text alignment model, the device comprising:

[0102] An image acquisition unit, used for acquiring a medical image and an enhanced view of the medical image;

[0103] A text acquisition unit, used to acquire a medical text and an enhanced text corresponding to the medical text;

[0104] An image input unit, used to input the medical image into an image encoder to obtain global features and local features of the medical image, and input the enhanced view into an image encoder to obtain global features and local features of the enhanced view;

[0105] A text input unit, used for inputting the medical text into a text encoder to obtain global features and local features of the medical text, and inputting the enhanced text into a text encoder to obtain global features and local features of the enhanced text;

[0106] A feature embedding unit, used to convert the corresponding global features and local features into low-dimensional embeddings using a nonlinear projection layer, to obtain embedded features of medical images, embedded features of enhanced views, embedded features of medical texts, and embedded features of enhanced texts; a first computing unit, used to align the embedded features of medical images and the embedded features of corresponding enhanced views using a similarity metric, and to align the embedded features of medical texts and the embedded features of corresponding enhanced texts, to obtain a multi-view self-supervised image-text alignment loss;

[0107] The second computing unit is used to calculate the soft matching between the medical image and the medical text by using the cross-attention mechanism to obtain the cross-modal marker alignment loss;

[0108] The third computing unit is used to fuse the embedded features of the medical image and the embedded features of the medical text by using a fusion module to obtain a cross-modal fusion feature, and cluster the cross-modal fusion feature by using a confidence clustering method to obtain a positive and negative sample loss guided by high-confidence clustering information;

[0109] The execution unit is used to train the medical image-text alignment model composed of the image encoder, the text encoder, the nonlinear projection layer, and the integrated attention module according to the multi-view self-supervised image-text alignment loss, the cross-modal tag alignment loss, and the positive and negative sample loss guided by the high-confidence clustering information until the preset conditions are met.

[0110] Based on the training method of a medical image text alignment model provided in the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the medical image text alignment model as described in any of the above embodiments.

[0111] Experiments have shown that Figure 4 The performance of semantic segmentation of the pre-trained model of the present invention on the SIIM and RSNA datasets is shown. When fine-tuning on the SIIM dataset using 10% and 100% of the training data, the segmentation performance of the present invention reaches 62.5% and 72.1, respectively, which are 0.3% and 7.7% higher than the most advanced methods, respectively. When fine-tuning on the RSNA dataset using 1% and 10% of the training data, the segmentation performance of the present invention reaches 72.6 and 75.0, respectively, which is higher than the most advanced methods overall. Figure 5The classification results of the pre-trained model of the present invention on four datasets, RSNA, CheXpert, COVIDx, and Vindr, are shown, among which Vindr is a dataset for children. When fine-tuning on the COVIDx dataset using 1% and 100% of the training data, the classification performance of the present invention reaches 78.7% and 93.0%, respectively, which are 2.2% and 0.2% higher than the current most advanced methods, respectively. When fine-tuning on the CheXpert dataset using 100% of the training data, it is also 0.5% higher than the most advanced invention. At the same time, the classification performance of the present invention on the Vindr dataset can reach 84.5%, 90.2%, and 94.9%, respectively, which fully demonstrates the superiority of the performance of the present invention and its good generalization. The present invention is also competitive in other settings, which may be because, relative to the most advanced performance, we only use less data and training resources.

[0112] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0113] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A training method for a medical image text alignment model, characterized in that: The method comprises: Acquire a medical image and an enhanced view of the medical image, a medical text, and an enhanced text corresponding to the medical text; Using an image encoder to process the medical image and the enhanced view respectively, to obtain global features and local features of the medical image, and global features and local features of the enhanced view; Using a text encoder to process the medical text and the enhanced text respectively, to obtain global features and local features of the medical text, and global features and local features of the enhanced text; A nonlinear projection layer is used to convert each global feature and local feature into a low-dimensional embedding to obtain the corresponding global embedding feature and local embedding feature; The global embedding features of the medical image and the global embedding features of the corresponding enhanced view are aligned using similarity measurement, and the global embedding features of the medical text and the global embedding features of the corresponding enhanced text are aligned to construct a multi-view self-supervised image-text alignment loss; the multi-view self-supervised image-text alignment loss is expressed as: Among them, L VVA represents the image alignment loss, L TTA Represents text alignment loss; N represents the number of samples in a batch; Represents the global embedding feature V of the medical image g,i Global embedding features with enhanced views The cosine similarity of Represents the global embedding feature t of medical text g,i Global embedding features with enhanced text The cosine similarity of σ1 represents the first temperature hyperparameter, sim represents the similarity function; The self-attention unit with integrated attention module is used to embed local features of medical images and medical text The local embedding feature processing of the present invention obtains the medical image self-attention feature and the medical text self-attention feature, and the cross attention unit of the integrated attention module is used to process the medical text self-attention feature and the medical image self-attention feature to obtain the cross-modal text embedding feature and the cross-modal image embedding feature; the cross-modal text embedding feature and the cross-modal image embedding feature are used to construct the cross-modal marker alignment loss; the calculation formula of the cross-modal marker alignment loss is expressed as: Among them, L CIA represents the cross-modal text loss, L CTA represents cross-modal image loss; N represents the number of samples in a batch; S1 represents the number of text sentences, and S2 represents the number of image blocks; represents the self-attention feature of medical images, represents the cross-modal text embedding feature; σ2 represents the second temperature hyperparameter, represents the self-attention feature of medical text, Represents cross-modal image embedding features; A fusion module is used to fuse the global embedding features of medical images and the global embedding features of medical texts to obtain cross-modal fusion features. The confidence clustering method is used to cluster the cross-modal fusion features to construct the positive and negative sample losses guided by high-confidence clustering information. The calculation formula of the positive and negative sample losses guided by high-confidence clustering information is expressed as: L HCCSC =L pos +αL neg Among them, L pos represents the positive sample loss, α represents the weight factor, L neg represents the negative sample loss; K represents the number of clusters, np represents the number of high confidence samples in the pth cluster, represents the image embedding feature of the i-th high confidence sample in the p-th cluster, represents the text embedding feature of the i-th high confidence sample in the p-th cluster, Represented as the center of the i-th high confidence sample in the p-th cluster of the image modality, Represented as the center of the i-th high confidence sample in the q-th cluster of the text modality; According to the multi-view self-supervised image-text alignment loss, cross-modal tag alignment loss, and positive and negative sample loss guided by high-confidence clustering information, the medical image-text alignment model consisting of image encoder, text encoder, nonlinear projection layer, and integrated attention module is trained until the preset conditions are met.

2. The training method of the medical image text alignment model according to claim 1, characterized in that: The method for acquiring the enhanced view of the medical image includes scaling the medical image to a fixed size; using a random cropping technique to obtain a plurality of block images of different sizes from the fixed-size medical image; using Gaussian blur within a preset standard deviation range to process each block image, and randomly flipping the processed block image according to a certain probability to generate an enhanced view of the medical image; the method for acquiring the enhanced text corresponding to the medical text includes segmenting the medical text, using a random shuffling strategy to process each medical text segment, and generating an enhanced text of the medical text.

3. The training method of the medical image text alignment model according to claim 1, characterized in that: The process of constructing the multi-view self-supervised image-text alignment loss includes taking the augmented view as positive sample, the medical image as negative sample, the augmented text as positive sample, and the medical text as negative sample; The similarity between medical images and enhanced views in the embedding space, and the similarity between medical texts and enhanced texts in the embedding space are calculated respectively. The scale of the similarity distribution is controlled by adjusting the temperature parameter to obtain the multi-view self-supervised image-text alignment loss.

4. The training method of the medical image text alignment model according to claim 1, characterized in that: The process of constructing the cross-modal tag alignment loss includes respectively using self-attention units to process the local embedding features of medical images and the local embedding features of medical texts, learning the contextual information of the local embedding features of medical images and the local embedding features of medical texts, and obtaining the self-attention features of medical images and the self-attention features of medical texts; using a cross-attention mechanism to process the self-attention features of medical texts and the self-attention features of medical images, learning the medical image information that the self-attention features of medical texts focus on, and the medical text information that the self-attention features of medical images focus on, and obtaining the cross-modal text embedding features and the cross-modal image embedding features; The cross-modal text loss and cross-modal image loss are constructed using the contrastive loss function; the cross-modal text loss and the cross-modal image loss are superimposed to obtain the cross-modal marker alignment loss.

5. The training method of the medical image text alignment model according to claim 1, characterized in that: The process of constructing the positive and negative sample loss guided by high-confidence clustering information includes clustering the cross-modal fusion features using the fuzzy C-means clustering algorithm to obtain the confidence score of each cross-modal fusion feature and its clustering information; taking the high-confidence cross-modal fusion features under the same cluster as positive samples, taking the high-confidence clustering information as supervision, and using the cross-modal fusion features with high confidence scores to construct the positive sample loss; The cross-modal fusion features of high-confidence cluster centers under different clusters are regarded as negative samples, and the centers with high confidence scores are used to construct the negative sample loss.

6. A training device for a medical image text alignment model, characterized in that: The device comprises: An image acquisition unit, used for acquiring a medical image and an enhanced view of the medical image; A text acquisition unit, used to acquire medical text and enhanced text corresponding to the medical text; An image input unit, used to input the medical image into an image encoder to obtain global features and local features of the medical image, and input the enhanced view into an image encoder to obtain global features and local features of the enhanced view; A text input unit, used for inputting the medical text into a text encoder to obtain global features and local features of the medical text, and inputting the enhanced text into a text encoder to obtain global features and local features of the enhanced text; A feature embedding unit, used to convert the corresponding global features and local features into low-dimensional embeddings by using a nonlinear projection layer, so as to obtain embedded features of medical images, embedded features of enhanced views, embedded features of medical texts, and embedded features of enhanced texts; The first computing unit is used to align the embedded features of the medical image and the embedded features of the corresponding enhanced view by using a similarity metric, and to align the embedded features of the medical text and the embedded features of the corresponding enhanced text, so as to obtain a multi-view self-supervised image-text alignment loss; the multi-view self-supervised image-text alignment loss is expressed as: Among them, L VVA represents the image alignment loss, L TTA Represents text alignment loss; N represents the number of samples in a batch; Represents the global embedding feature V of the medical image g,i Global embedding features with enhanced views The cosine similarity of Represents the global embedding feature t of medical text g,i Global embedding features with enhanced text The cosine similarity of σ1 represents the first temperature hyperparameter, sim represents the similarity function; the second calculation unit is used to calculate the soft matching between the medical image and the medical text by using the cross-attention mechanism to obtain the cross-modal marker alignment loss; the calculation formula of the cross-modal marker alignment loss is expressed as: Among them, L CIA represents the cross-modal text loss, L CTA represents cross-modal image loss; N represents the number of samples in a batch; S1 represents the number of text sentences, and S2 represents the number of image blocks; represents the self-attention feature of medical images, represents the cross-modal text embedding feature; σ2 represents the second temperature hyperparameter, represents the self-attention feature of medical text, Represents cross-modal image embedding features; The third computing unit is used to fuse the embedded features of the medical image and the embedded features of the medical text by using a fusion module to obtain a cross-modal fusion feature, cluster the cross-modal fusion feature by using a confidence clustering method, and obtain the positive and negative sample losses guided by the high-confidence clustering information; the calculation formula of the positive and negative sample losses guided by the high-confidence clustering information is expressed as: L HCCSC =L pos +αL neg Among them, L pos represents the positive sample loss, α represents the weight factor, L neg represents the negative sample loss; K represents the number of clusters, np represents the number of high confidence samples in the pth cluster, represents the image embedding feature of the i-th high confidence sample in the p-th cluster, represents the text embedding feature of the i-th high confidence sample in the p-th cluster, Represented as the center of the i-th high confidence sample in the p-th cluster of the image modality, Represented as the center of the i-th high-confidence sample in the q-th cluster of the text modality; an execution unit, used to train the medical image-text alignment model consisting of an image encoder, a text encoder, a nonlinear projection layer, and an integrated attention module according to the multi-view self-supervised image-text alignment loss, the cross-modal tag alignment loss, and the positive and negative sample loss guided by the high-confidence clustering information until the preset conditions are met.

7. An electronic device, characterized in that: include: one or more processors; A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the medical image text alignment model as described in any one of claims 1-5.

Images