Auxiliary diagnosis method and system for chest lesion based on multi-modal contrast pre-training

Abstract

The application provides a chest lesion auxiliary diagnosis method and system based on multi-modal contrast pre-training, which is used for auxiliary identification and classification of lesions in chest medical images of patients. The method comprises the following steps: obtaining chest medical images and corresponding medical report texts, and constructing medical image-report text data pairs; extracting image feature vectors and text feature vectors through image encoders and text encoders respectively, and mapping them to the same feature space to obtain image embedding vectors and text embedding vectors; taking the image embedding vectors and the text embedding vectors in the same data pair as positive sample pairs, and taking the embedding vectors in different data pairs as negative sample pairs, and calculating cross-modal contrast learning loss based on cross-modal feature similarity; screening difficult negative samples according to the cross-modal feature similarity between the negative samples and anchor samples, and calculating difficult negative sample contrast learning loss; pre-training the image encoders and the text encoders based on the above loss to obtain pre-trained image encoders; training a classifier using chest medical images with chest lesion classification labels, and inputting the chest medical images to be diagnosed into the pre-trained image encoders and the classifier to output chest lesion auxiliary diagnosis classification results. The method can improve the visual representation quality of chest medical images and the lesion classification ability.

Description

Technical Field

[0001] This invention belongs to the fields of medical artificial intelligence, computer vision, natural language processing and multimodal learning technology. Specifically, it relates to a method and system for auxiliary diagnosis of chest lesions based on multimodal contrast pre-training, and more particularly to a method and system for auxiliary identification and classification of chest lesions based on chest medical images and their corresponding medical report texts for cross-modal contrast pre-training. Background Technology

[0002] Chest imaging is a crucial tool for the clinical diagnosis of lung and cardiothoracic diseases. Chest X-rays are characterized by their speed, low cost, and wide applicability, and are commonly used for screening and auxiliary diagnosis of chest lesions such as atelectasis, cardiomegaly, pulmonary consolidation, edema, and pleural effusion. These lesions may be associated with various clinical problems, including pneumonia, heart failure, lung infections, and pleural diseases. Timely and accurate identification of these lesions helps improve clinical diagnostic efficiency.

[0003] With the development of deep learning technology, chest medical image-assisted diagnosis methods based on models such as convolutional neural networks and visual Transformers have been extensively studied. These methods typically train classification models using medical images labeled with lesions, enabling the models to output corresponding lesion categories or risk scores for input chest medical images. However, in real-world medical scenarios, obtaining high-quality labeled medical image data is costly, usually requiring manual annotation by experienced radiologists or clinical experts. The annotation process is not only time-consuming, but also prone to inconsistencies in annotation standards among different doctors, limiting the training effectiveness and widespread application of supervised deep learning models in the field of medical imaging.

[0004] To alleviate the problem of insufficient labeled data, existing technologies often use image encoders pre-trained on natural image datasets as initial models, which are then transferred to medical image tasks for fine-tuning. While such methods can improve model training performance to some extent, natural images and medical images differ significantly in imaging mechanisms, visual textures, target structures, and semantic representations. Directly using models pre-trained on natural images often fails to adequately learn the fine-grained lesion features in medical images.

[0005] On the other hand, in hospital information systems, chest medical images are usually accompanied by corresponding imaging report texts. These medical report texts contain descriptions of the imaging findings, lesion location, lesion type, and diagnostic impressions by the physician, thus containing rich medical semantic information. Compared to simply relying on manual classification labels, image-text data pairs consisting of medical images and their corresponding report texts can serve as large-scale weakly supervised or self-supervised learning resources for learning the semantic correspondence between medical images and text.

[0006] Existing multimodal contrastive learning methods can achieve cross-modal semantic alignment through similarity constraints between images and text, enabling the model to learn visual representations related to text semantics. However, in chest medical image scenarios, the visual differences between different lesions are often subtle, and medical report texts also exhibit similar formats, similar terminology, and templated descriptions. This makes the distinction between ordinary negative samples and difficult negative samples particularly important. Difficult negative samples refer to unpaired samples that do not belong to the same image-report text data pair as the anchor sample, but have high similarity to the anchor sample in the feature space. If these difficult negative samples are not effectively handled during training, the model is prone to learning visual features with insufficient discriminative power, affecting subsequent chest lesion classification performance.

[0007] Therefore, it is necessary to provide a method and system for auxiliary diagnosis of chest lesions that can utilize the natural correspondence between chest medical images and medical report texts, learn medical visual representations through cross-modal contrastive pre-training, and further focus on optimization of difficult negative samples. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for auxiliary diagnosis of chest lesions based on multimodal contrastive pre-training, in order to solve the problems of existing chest medical image-assisted diagnosis methods relying on a large amount of manually labeled data, the difference between natural image pre-training models and the medical image domain, and the inadequacy of ordinary contrastive learning methods in distinguishing difficult negative samples.

[0009] To achieve the above objectives, this invention provides a method for auxiliary diagnosis of chest lesions based on multimodal contrast pre-training, used to assist in the identification and classification of lesions in medical images of a patient's chest, comprising the following steps:

[0010] Step S1: Obtain chest medical images and their corresponding medical report texts, and construct medical image-report text data pairs;

[0011] Step S2: Perform image preprocessing on the chest medical image and text preprocessing on the medical report text;

[0012] Step S3: Input the preprocessed chest medical image into the image encoder to obtain the image feature vector, and input the preprocessed medical report text into the text encoder to obtain the text feature vector;

[0013] Step S4: Map the image feature vector and the text feature vector to the same feature space to obtain the image embedding vector and the text embedding vector, respectively.

[0014] Step S5: Using the image embedding vector and text embedding vector in the same medical image-report text data pair as positive sample pairs, and the image embedding vector and text embedding vector in different medical image-report text data pairs as negative sample pairs, calculate the cross-modal feature similarity between the image embedding vector and the text embedding vector.

[0015] Step S6: Calculate the cross-modal contrastive learning loss based on the cross-modal feature similarity to increase the similarity between positive sample pairs and decrease the similarity between negative sample pairs;

[0016] Step S7: Based on the cross-modal feature similarity between negative samples and anchor samples, select difficult negative samples from the negative sample pairs, and calculate the difficult negative sample contrastive learning loss based on the difficult negative samples;

[0017] Step S8: Based on the cross-modal contrastive learning loss and the hard negative sample contrastive learning loss, perform multimodal contrastive pre-training on the image encoder and the text encoder to obtain the pre-trained image encoder.

[0018] Step S9: Input the chest medical image with chest lesion classification labels into the pre-trained image encoder to obtain chest medical image features, and train a classifier based on the chest medical image features;

[0019] Step S10: Input the chest medical image to be diagnosed into the trained image encoder and classifier, and output the chest lesion auxiliary diagnosis classification result.

[0020] In one embodiment, the chest medical image includes a chest X-ray image, and the medical report text includes a findings section, an impression section, or a combination of findings and impression sections in an imaging report corresponding to the chest X-ray image.

[0021] In one embodiment, the image preprocessing includes at least one of image size normalization, pixel value normalization, and image enhancement; the text preprocessing includes at least one of text cleaning, word segmentation, truncation, and padding.

[0022] In one embodiment, the image encoder is a convolutional neural network, a visual Transformer network, or a combination of both, and the text encoder is a medical text encoding model based on a Transformer structure.

[0023] In one embodiment, the image feature vector and the text feature vector are respectively mapped to a feature space of the same dimension through corresponding linear projection layers and then normalized to obtain the image embedding vector and the text embedding vector.

[0024] In one implementation, the cross-modal contrastive learning loss includes image-to-text contrastive learning loss and text-to-image contrastive learning loss.

[0025] In one implementation, the difficult negative sample is a negative sample that does not belong to the same medical image-report text data pair as the anchor sample and whose cross-modal feature similarity with the anchor sample meets a preset screening condition. The preset screening condition includes that the cross-modal feature similarity between the negative sample and the anchor sample is greater than a preset similarity threshold, or that the negative sample is within a preset sorting range after being sorted according to cross-modal feature similarity.

[0026] In one implementation, the hard negative sample contrastive learning loss is obtained by at least one of the following methods: only the selected hard negative samples are included in the negative sample normalization term for calculation; or the selected hard negative samples are assigned a higher loss weight than ordinary negative samples.

[0027] In one implementation, the total loss in the multimodal contrastive pre-training phase is a weighted sum of the cross-modal contrastive learning loss and the hard negative sample contrastive learning loss.

[0028] In one implementation, during classifier training, the parameters of the pre-trained image encoder are frozen, and the parameters of the classifier are updated; the classifier is a linear classifier, including a fully connected layer and a probability output layer, for outputting the predicted probability of each chest lesion category.

[0029] In one implementation, the auxiliary diagnostic classification result for chest lesions includes at least one lesion category among atelectasis, cardiomegaly, pulmonary consolidation, edema, and pleural effusion. The classifier can output single-label classification results, multi-label classification results, or lesion risk scores.

[0030] The present invention also provides a chest lesion auxiliary diagnostic system based on multimodal contrast pre-training, including a data construction module, a preprocessing module, an image encoding module, a text encoding module, a feature projection module, a contrast pre-training module, a difficult negative sample optimization module, a classification fine-tuning module, and a diagnostic output module.

[0031] The present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the above-mentioned auxiliary diagnostic method for chest lesions.

[0032] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described auxiliary diagnostic method for chest lesions.

[0033] Invention Function and Effect

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This invention utilizes the natural correspondence between chest medical images and medical report texts to construct medical image-report text data pairs. Through multimodal contrastive pre-training, the image encoder learns visual representations that are semantically aligned with the medical report, reducing the reliance on large-scale manual classification and annotation data.

[0036] This invention improves the semantic consistency of visual representation of medical images by using bidirectional cross-modal contrastive learning in both the image-to-text and text-to-image directions to make positive sample pairs closer in the same feature space and negative sample pairs relatively farther apart in the same feature space.

[0037] This invention further filters difficult negative samples based on the cross-modal feature similarity between negative samples and anchor samples, and calculates additional contrastive learning loss based on difficult negative samples. This makes the model pay more attention to visually or semantically similar but not truly paired samples during the pre-training stage, which helps to improve the model's ability to distinguish fine-grained chest lesion features.

[0038] In the downstream classification stage, this invention can use a pre-trained image encoder to extract chest medical image features and output chest lesion auxiliary diagnosis classification results through a classifier. It is applicable to the auxiliary identification of chest lesions such as atelectasis, cardiomegaly, pulmonary consolidation, edema, and pleural effusion. Attached Figure Description

[0039] Figure 1 This is a flowchart of a chest lesion auxiliary diagnosis method based on multimodal contrast pre-training in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of the multimodal contrast pre-trained model in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the difficult negative sample screening and optimization process in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the downstream classifier fine-tuning process in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of the chest lesion auxiliary diagnosis system based on multimodal contrast pre-training in an embodiment of the present invention. Detailed Implementation

[0044] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. The described embodiments are only for illustrating the present invention and are not intended to limit the scope of protection of the present invention. Unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0045] Example 1

[0046] This embodiment provides a method for assisted diagnosis of chest lesions based on multimodal contrastive pre-training. This method can be implemented on computing devices equipped with graphics processing units or other AI-accelerated hardware, or deployed on servers, workstations, medical image-assisted diagnostic platforms, or cloud-based medical image analysis systems.

[0047] like Figure 1 As shown, the method in this embodiment includes steps S1 to S10.

[0048] Step S1, construct medical image-report text data pairs

[0049] Obtain chest medical images and their corresponding medical report texts, and construct medical image-report text data pairs.

[0050] In one embodiment, the chest medical image is a chest X-ray image. The medical report text is the imaging report text corresponding to the chest X-ray image. The medical report text may include a findings section, an impression section, or a combination of a findings section and an impression section in the report.

[0051] In practical applications, a medical image-report text data pair can be represented as: ,in, Indicates the first A chest medical image, This refers to the medical report text corresponding to the chest medical image.

[0052] To improve data quality, medical report text can be filtered by fields, such as retaining the findings and impressions sections while removing patient identification information, irrelevant formatting information, empty text, or excessively short text. The above processing does not constitute a limitation on the scope of protection of this invention.

[0053] Step S2, Image Preprocessing and Text Preprocessing

[0054] Image preprocessing is performed on chest medical images, and text preprocessing is performed on medical report text.

[0055] Image preprocessing can include at least one of image size normalization, pixel value normalization, grayscale normalization, and image enhancement. For example, a chest medical image can be resized to a preset size and its pixel values ​​normalized to a preset range.

[0056] Text preprocessing can include at least one of text cleaning, word segmentation, truncation, and padding. For example, irrelevant symbols and formatting fields can be removed from report text, the text can be input into a medical text segmenter for word segmentation, and the segmented text can be truncated or padded to a preset length.

[0057] Step S3: Extract image feature vectors and text feature vectors.

[0058] The preprocessed chest medical image is input into the image encoder to obtain the image feature vector; the preprocessed medical report text is input into the text encoder to obtain the text feature vector.

[0059] In one implementation, the image encoder can be a convolutional neural network, such as ResNet, DenseNet, EfficientNet, etc.; it can also be a visual Transformer network, such as ViT, Swing Transformer, etc.; or it can be a combination of a convolutional neural network and a visual Transformer network.

[0060] In one implementation, the text encoder can be a Transformer-based medical text encoding model, such as BERT, ClinicalBERT, BioClinicalBERT, or other pre-trained language models suitable for medical text.

[0061] For the A medical image-report text data pair, including chest medical images. Input Image Encoder To obtain the image feature vector medical report text Input text encoder To obtain the text feature vector ,in, Indicates the first Image feature vector of a chest medical image, Indicates the first The text feature vector of a medical report text.

[0062] Step S4, mapping to the same feature space

[0063] The image feature vector and the text feature vector are mapped to the same feature space to obtain the image embedding vector and the text embedding vector, respectively.

[0064] Since the feature dimensions output by the image encoder and the text encoder may be different, this embodiment sets up an image projection layer and a text projection layer respectively to map the image feature vector and the text feature vector to a feature space of the same dimension.

[0065] Specifically, the image feature vector The input image projection layer projects the text feature vectors. Input text projection layer, and perform... Normalization yields the image embedding vector. and text embedding vectors :

[0066] ,

[0067] in, Indicates the parameters of the image projection layer. Indicates the text projection layer parameters. ∥⋅∥2\left\|\cdot\right\|_2∥⋅∥2 represents Norm.

[0068] Step S5: Calculate cross-modal feature similarity

[0069] Using the image embedding vector and text embedding vector in the same medical image-report text data pair as positive sample pairs, and the image embedding vector and text embedding vector in different medical image-report text data pairs as negative sample pairs, the cross-modal feature similarity between the image embedding vector and the text embedding vector is calculated.

[0070] In a containing In the training batch of medical image-report text data pairs, the first... Image embedding vectors With the text embedding vectors This constitutes a positive sample pair. Image embedding vectors With the text embedding vectors exist This constitutes a negative sample pair.

[0071] In this embodiment, the cross-modal feature similarity between the image embedding vector and the text embedding vector is defined as:

[0072]

[0073] in, Indicates the first Image embedding vectors With the text embedding vectors The similarity between them. Due to and Normalization has been performed, and dot product similarity is equivalent to cosine similarity.

[0074] In other implementations, cross-modal feature similarity can also be cosine similarity, dot product similarity, or similarity derived from distance metric transformation.

[0075] Step S6: Calculate the cross-modal contrastive learning loss.

[0076] The cross-modal contrastive learning loss is calculated based on cross-modal feature similarity, which increases the similarity between positive sample pairs and decreases the similarity between negative sample pairs.

[0077] In this embodiment, the cross-modal contrastive learning loss includes contrastive learning loss in the image-to-text direction and contrastive learning loss in the text-to-image direction.

[0078] Contrast learning loss in the image-to-text direction

[0079] For the A medical image-report text data pair, with image embedding vectors As anchor points, corresponding to the text embedding vectors As positive samples, use non-corresponding text embedding vectors from the same training batch. These are negative samples. The contrastive learning loss from image to text is defined as:

[0080]

[0081] in, Indicates the first Positive sample similarity between an image embedding vector and its corresponding text embedding vector. Indicates the first The image embedding vector and the first Similarity between text embedding vectors This represents the temperature coefficient.

[0082] The batch-level image-to-text direction loss is:

[0083]

[0084] Contrast learning loss in the text-to-image direction

[0085] For the A medical image-report text data pair, with text embedding vectors As anchor points, corresponding to the image embedding vector For positive samples, embed the vectors of non-corresponding images from the same training batch. These are negative samples. The contrastive learning loss from text to image is defined as:

[0086]

[0087] in, Indicates the first Image embedding vectors With the Similarity between text embedding vectors.

[0088] Batch-level text-to-image orientation loss is:

[0089]

[0090] Cross-modal contrastive learning loss is defined as:

[0091]

[0092] The aforementioned loss can be the InfoNCE loss or a variation thereof. This loss increases the similarity between image embedding vectors and text embedding vectors within the same medical image-report text data pair during training, while simultaneously reducing cross-modal similarity between different medical image-report text data pairs.

[0093] Step S7: Filter out difficult negative samples and calculate the contrastive learning loss for difficult negative samples.

[0094] Based on the cross-modal feature similarity between negative samples and anchor samples, difficult negative samples are selected from the negative sample pairs, and the contrastive learning loss of difficult negative samples is calculated based on the difficult negative samples.

[0095] Difficult negative samples refer to negative samples that do not belong to the same medical image-report text data pair as the anchor sample, but whose cross-modal feature similarity with the anchor sample meets preset screening criteria. In chest medical image scenarios, different images may have similar anatomical structures, similar imaging grayscale, and similar report templates. If such samples are too close to the anchor sample in the feature space, it may reduce the model's discriminative ability. Therefore, this embodiment performs additional optimization for difficult negative samples.

[0096] For the Image embedding vectors The set of difficult negative samples in the text direction is defined as follows:

[0097]

[0098] in, This indicates a preset similarity threshold. Represents image anchor points The corresponding set of negative sample indices for text direction difficulties.

[0099] For the text embedding vectors The set of difficult negative samples in its image orientation is defined as:

[0100]

[0101] in, Indicates text anchor point The corresponding set of image orientation hard negative sample indices.

[0102] In another implementation, a sorting method can be used to filter difficult negative samples. For the image-to-text direction, the set of difficult negative samples can be defined as:

[0103]

[0104] For the text-to-image direction, the set of difficult negative samples can be defined as:

[0105]

[0106] in, This means selecting a preset number of negative samples with the highest similarity from the unpaired samples.

[0107] To enhance the model's ability to distinguish difficult negative samples, this embodiment sets a difficult negative sample contrastive learning loss. The difficult negative sample loss in the image-to-text direction is defined as:

[0108]

[0109] The hard negative sample loss in the text-to-image direction is defined as:

[0110]

[0111] in, The temperature coefficient represents the optimization for hard negative samples. The above loss is used to constrain the decrease in similarity between hard negative samples and positive samples, thereby increasing the feature distance between anchor samples and hard negative samples.

[0112] The hard negative sample contrastive learning loss is defined as:

[0113]

[0114] In other implementations, the hard negative sample contrastive learning loss can also be calculated by including only the selected hard negative samples in the negative sample normalization term, or by assigning a higher loss weight to the selected hard negative samples than to ordinary negative samples.

[0115] Step S8, Multimodal Contrast Pre-training

[0116] Based on cross-modal contrastive learning loss and hard negative sample contrastive learning loss, multimodal contrastive pre-training is performed on the image encoder and text encoder to obtain the pre-trained image encoder.

[0117] The total loss during the multimodal contrastive pre-training phase is defined as:

[0118]

[0119] in, This represents the total loss during the multimodal contrastive pre-training phase. This represents the cross-modal contrastive learning loss. This represents the difficulty of negative sample contrastive learning loss. and These represent the corresponding loss weight coefficients, and .

[0120] In one implementation, .For example, 0.4 can be taken. A value of 0.6 is acceptable. This value is merely an example and does not constitute a limitation on the scope of protection of this invention.

[0121] Through multimodal contrastive pre-training, the image encoder can learn medical visual representations that are semantically aligned with the medical report text. After pre-training, the image encoder can be retained for subsequent chest lesion auxiliary diagnostic classification tasks.

[0122] Step S9, downstream classifier training

[0123] Chest medical images with chest lesion classification labels are input into a pre-trained image encoder to obtain chest medical image features, and a classifier is trained based on the chest medical image features.

[0124] In one implementation, the classifier is a linear classifier, comprising fully connected layers and a probability output layer. During classifier training, the parameters of the pre-trained image encoder can be frozen, and only the classifier parameters are updated. This approach can be used to evaluate the effectiveness of the medical visual representations learned by the pre-trained image encoder.

[0125] In another implementation, the image encoder and classifier can also be jointly fine-tuned.

[0126] For the A sample with a chest lesion classification label is input into a pre-trained image encoder to obtain image features, which are then input into a classifier to obtain the first image feature. The sample belongs to the first Predicted probability of each chest lesion category .

[0127] If the downstream task is a single-label multi-class classification task, then the classification loss can be the cross-entropy loss:

[0128]

[0129] in, This indicates the number of samples in the training batch. This indicates the total number of chest lesion categories. Indicates the first The sample at the th The actual label indicator value on the class, This indicates that the classifier predicts the first... The sample belongs to the first The probability of a class.

[0130] If the downstream task is a multi-label chest lesion classification task, then the classification loss can be a binary cross-entropy loss:

[0131]

[0132] in, , indicating the first Does the sample have the first...? Chest-like lesions; , indicating that the classifier predicts the first The sample has the first The probability of developing chest-like lesions.

[0133] Multi-label classification is more suitable for situations where multiple lesion categories may exist simultaneously in the same chest medical image.

[0134] Step S10: Output the auxiliary diagnostic classification results of chest lesions.

[0135] Input the chest medical image to be diagnosed into the trained image encoder and classifier, and output the chest lesion auxiliary diagnosis classification result.

[0136] In one implementation, the auxiliary diagnostic classification results for chest lesions include at least one lesion category among atelectasis, cardiomegaly, pulmonary consolidation, edema, and pleural effusion.

[0137] In one implementation, the classifier outputs predicted probabilities for multiple chest lesion categories and generates single-label classification results, multi-label classification results, or lesion risk scores based on the predicted probabilities. For example, in a single-label classification task, the category with the highest predicted probability can be selected as the diagnostic classification result; in a multi-label classification task, one or more categories with predicted probabilities exceeding a preset threshold can be used as the output result; in a risk score output task, the probability values ​​corresponding to each lesion category can be directly output as the lesion risk score.

[0138] Example 2

[0139] This embodiment provides an auxiliary diagnostic system for chest lesions based on multimodal contrastive pre-training. For example... Figure 5 As shown, the system includes a data construction module, a preprocessing module, an image encoding module, a text encoding module, a feature projection module, a contrastive pre-training module, a hard negative sample optimization module, a classification fine-tuning module, and a diagnostic output module.

[0140] The data construction module is used to acquire chest medical images and their corresponding medical report texts, and to construct medical image-report text data pairs.

[0141] The preprocessing module is used to perform image preprocessing on chest medical images and text preprocessing on medical report text.

[0142] The image encoding module is used to extract image feature vectors from chest medical images using an image encoder.

[0143] The text encoding module is used to extract text feature vectors from medical report texts using a text encoder.

[0144] The feature projection module is used to map image feature vectors and text feature vectors to the same feature space to obtain image embedding vectors and text embedding vectors.

[0145] The contrastive pre-training module is used to construct positive and negative sample pairs based on image and text embedding vectors, and to calculate the cross-modal contrastive learning loss based on the cross-modal feature similarity between the positive and negative sample pairs.

[0146] The difficult negative sample optimization module is used to filter difficult negative samples based on the cross-modal feature similarity between the negative sample and the anchor sample, and to calculate the difficult negative sample contrastive learning loss based on the difficult negative samples.

[0147] In one implementation, the difficult negative sample optimization module is configured to identify negative samples that do not belong to the same medical image-report text data pair as the anchor sample and whose cross-modal feature similarity meets the preset screening conditions as difficult negative samples.

[0148] The classification fine-tuning module is used to train a classifier based on a pre-trained image encoder and chest medical images with chest lesion classification labels.

[0149] The diagnostic output module is used to input the chest medical image to be diagnosed into the trained image encoder and classifier, and output the auxiliary diagnostic classification results of chest lesions.

[0150] The system modules in this embodiment can be implemented by software programs, hardware circuits, or a combination of both. The specific processing procedures for each module can be found in the corresponding method steps of Embodiment 1, and will not be repeated here.

[0151] Example 3

[0152] This embodiment provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the chest lesion auxiliary diagnosis method based on multimodal contrast pre-training described in Embodiment 1.

[0153] The electronic device may be a server, workstation, personal computer, medical imaging processing equipment, cloud computing device, or other electronic device with data processing capabilities. The processor may include a central processing unit, graphics processing unit, tensor processor, neural network processor, or other processing unit suitable for performing artificial intelligence model calculations. The memory may include read-only memory, random access memory, solid-state drive, hard disk drive, or other storage media.

[0154] Example 4

[0155] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the chest lesion auxiliary diagnosis method based on multimodal contrast pre-training described in Embodiment 1.

[0156] The computer-readable storage medium may include a disk, optical disk, read-only memory, random access memory, flash memory, portable hard disk, or other media capable of storing computer programs.

[0157] Functions and effects of the embodiments

[0158] According to the above embodiments, the present invention constructs cross-modal pre-training data using medical image-report text data, without relying on a large amount of additional manual classification and annotation, and can use the image semantic information contained in the medical report text to guide the image encoder to learn medical visual representations.

[0159] This invention improves the ability of the pre-trained image encoder to express semantic information of chest medical images by using bidirectional comparative learning in the image-to-text direction and the text-to-image direction to achieve semantic alignment between image embedding vectors and text embedding vectors in the same feature space.

[0160] This invention uses a difficult negative sample screening and a difficult negative sample comparison learning loss to enable the model to focus on negative samples that have a high similarity to anchor samples but are not true pairs, thereby enhancing the model's ability to distinguish fine-grained chest lesion features.

[0161] This invention can also transfer the pre-trained image encoder to the downstream chest lesion classification task, and output the chest lesion auxiliary diagnosis classification results through training the classifier. It is applicable to single-label classification, multi-label classification or lesion risk scoring and other application scenarios.

[0162] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications, equivalent substitutions, or variations to the above embodiments without departing from the technical concept of the present invention, and all such modifications, equivalent substitutions, or variations should be included within the protection scope of the present invention.

Claims

1. A method for auxiliary diagnosis of chest lesions based on multimodal contrast pre-training, used to assist in the identification and classification of lesions in medical images of a patient's chest, characterized in that, Includes the following steps: Step S1: Obtain chest medical images and their corresponding medical report texts, and construct medical image-report text data pairs; Step S2: Perform image preprocessing on the chest medical image and text preprocessing on the medical report text; Step S3: Input the preprocessed chest medical image into the image encoder to obtain the image feature vector, and input the preprocessed medical report text into the text encoder to obtain the text feature vector; Step S4: Map the image feature vector and the text feature vector to the same feature space to obtain the image embedding vector and the text embedding vector, respectively. Step S5: Using the image embedding vector and text embedding vector in the same medical image-report text data pair as positive sample pairs, and the image embedding vector and text embedding vector in different medical image-report text data pairs as negative sample pairs, calculate the cross-modal feature similarity between the image embedding vector and the text embedding vector. Step S6: Calculate the cross-modal contrastive learning loss based on the cross-modal feature similarity to increase the similarity between positive sample pairs and decrease the similarity between negative sample pairs; Step S7: Based on the cross-modal feature similarity between negative samples and anchor samples, select difficult negative samples from the negative sample pairs, and calculate the difficult negative sample contrastive learning loss based on the difficult negative samples; Step S8: Based on the cross-modal contrastive learning loss and the hard negative sample contrastive learning loss, perform multimodal contrastive pre-training on the image encoder and the text encoder to obtain the pre-trained image encoder. Step S9: Input the chest medical image with chest lesion classification labels into the pre-trained image encoder to obtain chest medical image features, and train a classifier based on the chest medical image features; Step S10: Input the chest medical image to be diagnosed into the trained image encoder and classifier, and output the chest lesion auxiliary diagnosis classification result.

2. The method for auxiliary diagnosis of chest lesions according to claim 1, characterized in that: In steps S1 and S2, the chest medical image includes a chest X-ray image, and the medical report text includes the findings section, impression section, or a combination of findings and impression sections in the imaging report corresponding to the chest X-ray image; the image preprocessing includes at least one of image size normalization, pixel value normalization, and image enhancement; the text preprocessing includes at least one of text cleaning, word segmentation, truncation, and padding.

3. The method for auxiliary diagnosis of chest lesions according to claim 1, characterized in that: In steps S3 and S4, the image encoder is a convolutional neural network, a visual Transformer network, or a combination of both, and the text encoder is a medical text encoding model based on the Transformer structure; the image feature vector and the text feature vector are respectively mapped to the feature space of the same dimension through corresponding linear projection layers and normalized to obtain the image embedding vector and the text embedding vector.

4. The method for auxiliary diagnosis of chest lesions according to claim 1, characterized in that: In step S5, the cross-modal feature similarity is the cosine similarity, dot product similarity, or similarity obtained by distance metric transformation between the image embedding vector and the text embedding vector.

5. The method for auxiliary diagnosis of chest lesions according to claim 1, characterized in that: In step S6, the cross-modal contrastive learning loss includes contrastive learning loss in the image-to-text direction and contrastive learning loss in the text-to-image direction, and adopts InfoNCE loss or its variant form; in the image-to-text direction, the image embedding vector is used as the anchor point, the corresponding text is a positive sample, and the non-corresponding text is a negative sample, and the text-to-image direction is the opposite.

6. The method for auxiliary diagnosis of chest lesions according to claim 1, characterized in that: In step S7, the difficult negative sample is a negative sample that does not belong to the same medical image-report text data pair as the anchor sample and whose cross-modal feature similarity with the anchor sample meets the preset screening conditions. The preset screening conditions include: the cross-modal feature similarity between the negative sample and the anchor sample is greater than a preset similarity threshold, or the negative sample is within a preset sorting range after being sorted according to cross-modal feature similarity.

7. The method for auxiliary diagnosis of chest lesions according to claim 1, characterized in that: The hard negative sample loss is obtained by including only hard negative samples in the calculation or by giving hard negative samples a higher loss weight; the total loss in the pre-training stage is the weighted sum of the cross-modal contrastive learning loss and the hard negative sample loss.

8. The method for auxiliary diagnosis of chest lesions according to claim 1, characterized in that: When training the classifier, the pre-trained image encoder parameters are frozen, and only the classifier parameters are updated; the classifier is a linear classifier containing a fully connected layer and a probability output layer, and the output results include at least one of atelectasis, cardiomegaly, pulmonary consolidation, edema, and pleural effusion, and can generate single-label, multi-label classification results or lesion risk scores.

9. A chest lesion auxiliary diagnostic system based on multimodal contrast pre-training, characterized in that, include: The data construction module is used to acquire chest medical images and their corresponding medical report texts, and to construct medical image-report text data pairs; The preprocessing module is used to perform image preprocessing on chest medical images and text preprocessing on medical report text. The image encoding module is used to extract image feature vectors from chest medical images through an image encoder; The text encoding module is used to extract text feature vectors from medical report texts using a text encoder. The feature projection module is used to map image feature vectors and text feature vectors to the same feature space to obtain image embedding vectors and text embedding vectors. The contrastive pre-training module is used to construct positive and negative sample pairs based on image and text embedding vectors, and to calculate cross-modal contrastive learning loss based on the cross-modal feature similarity between positive and negative sample pairs. The difficult negative sample optimization module is used to filter difficult negative samples based on the cross-modal feature similarity between the negative sample and the anchor sample, and to calculate the difficult negative sample contrastive learning loss based on the difficult negative sample. A classification fine-tuning module is used to train a classifier based on a pre-trained image encoder and chest medical images with chest lesion classification labels; The diagnostic output module is used to input the chest medical image to be diagnosed into the trained image encoder and classifier, and output the auxiliary diagnostic classification results of chest lesions.

10. An electronic device and a computer-readable storage medium, characterized in that: The electronic device includes a processor and a memory. Both the storage medium and the memory store a computer program. When the processor executes the computer program, it implements the chest lesion auxiliary diagnosis method according to any one of claims 1 to 8.

Images