A method and apparatus for analyzing x-ray films based on vision and language

Abstract

This application belongs to the field of computer vision technology, specifically disclosing a method and apparatus for X-ray film analysis based on vision and language. The method includes extracting image features of the X-ray film to be analyzed using a visual encoder and extracting text features of the X-ray film to be analyzed using a text encoder; obtaining a learnable query vector; inputting the image features, text features, and query vector into a trained large language model; performing cross-modal interaction on the query vector, image features, and text features through the attention layer of the large language model to obtain fused cross-modal features; and performing recognition analysis based on the cross-modal features to obtain the analysis results of the X-ray film image and text to be analyzed. This application can improve the accuracy of X-ray film analysis results.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and more specifically, relates to a method and apparatus for X-ray film analysis based on vision and language. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, cross-modal learning has become a research hotspot. The core objective of cross-modal learning is to improve the performance of analytical tasks by fusing data from different modalities (such as images and text) and uncovering the relationships between them. However, existing cross-modal learning methods still face many challenges in practical applications. First, image and text features are usually extracted independently, lacking an effective cross-modal interaction mechanism, leading to insufficient information fusion. Second, semantic alignment between image features and text descriptions is difficult to achieve, affecting the accuracy and comprehensiveness of the analysis. Furthermore, traditional models struggle to adapt to diverse scenarios and different text description styles when processing complex multimodal data.

[0003] However, existing cross-modal learning methods still suffer from problems such as modal fragmentation, semantic alignment difficulties, and limited model generalization ability when processing medical images such as X-rays, resulting in poor model performance and inaccurate image analysis results. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this application is to provide a vision and language-based X-ray analysis method and apparatus, which aims to solve the problem that poor model performance leads to inaccurate X-ray analysis results.

[0005] To achieve the above objectives, in a first aspect, this application provides a vision- and language-based X-ray analysis method, comprising:

[0006] A visual encoder is used to extract image features from the X-ray images to be analyzed, and a text encoder is used to extract text features from the X-ray images to be analyzed.

[0007] Learnable query vectors are obtained, and the image features, text features, and query vectors are input into a trained large language model. The query vectors, image features, and text features are interacted across modally through the attention layer of the large language model to obtain fused cross-modal features. Based on the cross-modal features, recognition and analysis are performed to obtain the analysis results of the X-ray image and text to be analyzed.

[0008] The large language model is obtained by training and fine-tuning the parameters based on the visual and textual description alignment of the image-text samples in the dataset.

[0009] Optionally, it also includes:

[0010] The image features are mapped to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors;

[0011] The attention layer of the large language model enables cross-modal interaction of query vectors, image features, and text features, including:

[0012] The interaction between query vectors and visual embedding vectors is achieved through the attention layer of a large language model.

[0013] Optionally, the attention layer includes a self-attention layer and a cross-attention layer;

[0014] The method of achieving interaction between query vectors and visual embedding vectors through the attention layer of a large language model includes:

[0015] The interaction between query vectors is achieved through a self-attention layer, and the cross-attention layer enables cross-modal interaction between query vectors and image features, as well as cross-modal interaction between query vectors and text features.

[0016] Optionally, the step of mapping the image features to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors includes:

[0017] The image features are used as embedding vectors. The image embedding vectors are input into a learnable linear projection layer and mapped to the feature space of the language model to generate visual embedding vectors of the target dimension.

[0018] The image features and text features of the visual embedding vector are mapped to the same embedding space, and cross-modal feature alignment is achieved through contrastive learning to obtain the visual embedding vector.

[0019] Optionally, the training method for the large language model includes:

[0020] By utilizing image-text pairs from the MIMIC-CXR dataset, we optimize the cross-modal alignment between image features and text features through contrastive learning to obtain a pre-trained model.

[0021] By using image-text pairs from the OpenI dataset, the parameters of the pre-trained model are fine-tuned to enhance the model's text generation capabilities.

[0022] Optionally, the method for constructing the dataset includes:

[0023] Obtain the first raw data from the MIMIC-CXR dataset, which is used to train and test the model; obtain the second raw data from the OpenI dataset, which is used for model fine-tuning.

[0024] The text report of the raw data is preprocessed, and the preprocessing process includes: removing incomplete reports, excessively short text and special symbols, deleting historical comparison statements and excluding medical image view information from the report summary;

[0025] The preprocessed text reports are merged to obtain an optimized image-text pair dataset, which is used for model training, testing, and fine-tuning.

[0026] Optionally, it also includes:

[0027] The ROUGE score is used as a quantitative indicator to evaluate the similarity between the text generated by the model and the real report, in order to measure the performance of the large language model in real-world situations.

[0028] The quality and coherence of the generated text were analyzed based on the generative pre-trained model GPT, and compared with the baseline model to obtain the GPT evaluation results.

[0029] Receive user input, score the analysis results based on the user input, and obtain user evaluation results;

[0030] The overall performance of the large language model is comprehensively evaluated by combining the ROUGE score, GPT evaluation results, and user evaluation results, resulting in a comprehensive evaluation result for the model.

[0031] Secondly, this application provides a vision- and language-based X-ray film analysis device, comprising:

[0032] The feature extraction module is used to extract image features of the X-ray film to be analyzed using a visual encoder and to extract text features of the X-ray film to be analyzed using a text encoder.

[0033] The feature fusion module is used to obtain a learnable query vector. The image features, text features and query vector are input into the trained large language model. The attention layer of the large language model performs cross-modal interaction on the query vector, visual embedding vector and text features to obtain the fused cross-modal features.

[0034] The analysis module is used to perform recognition and analysis based on the cross-modal features to obtain the analysis results of the X-ray image and text to be analyzed;

[0035] The large language model is obtained by training and fine-tuning the parameters based on the visual and textual description alignment of the image-text samples in the dataset.

[0036] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0037] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0038] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0039] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0040] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0041] (1) This application extracts image features and text features by visual encoder and text encoder respectively, and uses the attention layer of large language model to realize cross-modal interaction, so that the model can better understand the semantic relationship between image and text, improve the model performance, and thus generate more accurate text information related to chest X-ray, thereby improving the accuracy of X-ray analysis results.

[0042] (2) This application uses the MIMIC-CXR dataset for pre-training and the OpenI dataset for fine-tuning to construct a high-quality image-text pair dataset. By preprocessing the original data to remove interference information such as incomplete reports, excessively short texts and special symbols, the high quality and consistency of the dataset are ensured, which further guarantees the model performance and improves the accuracy of model analysis.

[0043] (3) This application uses a visual and textual modal fusion algorithm to deeply fuse image features with textual features, generating aligned cross-modal features. This enables the model to better perform visual-language alignment tasks, ensuring a high degree of semantic consistency between image and textual information. In X-ray analysis, the model can accurately associate target regions in the image with relevant descriptions in the text, thereby achieving more accurate semantic alignment and significantly improving the model's performance in cross-modal tasks, enabling it to better understand the semantic relationship between images and text.

[0044] (4) This application employs a freeze condition design during model training and fine-tuning, which preserves the general features of the pre-trained model and optimizes only the parameters relevant to specific tasks. This design not only reduces training complexity but also lowers the risk of overfitting and improves the robustness of the model. Attached Figure Description

[0045] Figure 1 This is one of the flowcharts of the vision and language-based X-ray analysis method provided in the embodiments of this application;

[0046] Figure 2 This is the second flowchart of the vision and language-based X-ray analysis method provided in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram of the structure of the vision and language-based X-ray analysis device provided in the embodiments of this application;

[0048] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0051] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0052] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0053] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0054] The embodiments of this application are described below with reference to the accompanying drawings.

[0055] Reference Figure 1 This application provides a vision- and language-based X-ray analysis method, including:

[0056] S101. Use a visual encoder to extract image features of the X-ray film to be analyzed, and use a text encoder to extract text features of the X-ray film to be analyzed;

[0057] S102. Obtain a learnable query vector, input the image features, text features and query vector into the trained large language model, and perform cross-modal interaction on the query vector, image features and text features through the attention layer of the large language model to obtain fused cross-modal features;

[0058] S103. Based on the cross-modal features, perform recognition analysis to obtain the analysis results of the X-ray image and text to be analyzed;

[0059] The large language model is obtained by training and fine-tuning the parameters based on the visual and textual description alignment of the image-text samples in the dataset.

[0060] Specifically, in step S101, a visual encoder is used to extract image features from the X-ray image to be analyzed. The visual encoder can capture multi-level feature information from the image. For medical images such as X-rays, the visual encoder focuses on key information such as structure, texture, and abnormal regions in the image. Through multi-layer convolutional operations or self-attention mechanisms, the visual encoder can convert the raw pixel data into high-dimensional feature vectors. These feature vectors can effectively represent the visual information in the image, providing a foundation for subsequent cross-modal interaction.

[0061] During the above process, the visual encoder remains frozen, with its weights unchanged and not updated, thus preserving the general features already learned in the pre-trained model, reducing training complexity and the risk of overfitting, and ensuring its stability and accuracy on medical images.

[0062] Simultaneously, a text encoder is used to extract textual features from the X-ray images to be analyzed. Text encoders are typically based on natural language processing models, capable of extracting semantic information from text. For the textual description of X-ray images, the text encoder captures key terms related to the image, descriptive language, and potential semantic connections. Through word embeddings and self-attention mechanisms, the text encoder transforms the text into high-dimensional feature vectors that characterize the semantic information of the text. The extraction of both image and textual features provides multimodal input for subsequent cross-modal interactions, enabling the model to utilize both visual and textual information for analysis.

[0063] It should be further noted that the visual encoder in this embodiment uses MedClip to extract image features from the X-ray image to be analyzed. MedClip is a visual encoder specifically designed for medical images. Based on the CLIP (Contrastive Language–Image Pretraining) architecture, it can perform comparative learning between images and text, thereby better capturing key information in medical images. For medical images such as X-ray images, MedClip, through multi-layer convolutional operations and a comparative learning mechanism, can extract visual features related to medical diagnosis, such as lesion areas and tissue structures. These features not only possess high semantic information but can also be effectively aligned with text descriptions, providing a foundation for subsequent cross-modal interaction.

[0064] In step S102, the model first acquires a learnable query vector. The query vector is a high-dimensional vector whose purpose is to guide the model to focus on the correlation between image and text features in cross-modal interactions. The query vector is optimized through the training process, enabling the model to better learn the semantic relationships between images and text. Next, the extracted image features, text features, and query vector are input into the trained large language model. The large language model has powerful cross-modal processing capabilities, capable of processing both image and text information simultaneously.

[0065] It should be noted that the preferred large language model in this application embodiment is Vicuna. Vicuna is a large language model based on the LLaMA architecture, which has powerful cross-modal processing capabilities and can process image and text information simultaneously.

[0066] In the Vicuna model, the attention layer is the core mechanism for achieving cross-modal interaction. The attention layer dynamically fuses image and text information by calculating the relevance weights between the query vector and image and text features. This interaction mechanism enables the model to capture the semantic relationships between images and text, such as the correspondence between a region in an image and a specific description in the text. Through multi-layered attention mechanisms, the model progressively fuses image and text features to generate fused cross-modal features. These cross-modal features not only contain independent information from images and text but also capture the deep relationships between them, providing richer input for subsequent recognition and analysis.

[0067] In step S103, the fused cross-modal features are used for the recognition and analysis task. These features not only contain independent information about the image and text but also capture the semantic relationships between them, thus providing a more comprehensive input for the analysis task. Depending on the specific task requirements, the model can perform operations such as classification, generation, or retrieval based on the cross-modal features. For example, in a classification task, the model can identify abnormal regions in an image based on the cross-modal features; in a generation task, the model can generate matching text descriptions based on image features; and in a retrieval task, the model can retrieve relevant images based on the text descriptions.

[0068] Ultimately, the model completes the recognition and analysis task based on cross-modal features and outputs the analysis results of the X-ray image and text. Because cross-modal features integrate information from both images and text, the model's output typically exhibits higher accuracy and robustness. For example, in medical image analysis, the model can combine image features and textual descriptions to generate more accurate diagnostic reports or test results, thereby providing more reliable support for decision-making.

[0069] Optionally, it also includes:

[0070] The image features are mapped to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors;

[0071] The attention layer of the large language model enables cross-modal interaction of query vectors, image features, and text features, including:

[0072] The interaction between query vectors and visual embedding vectors is achieved through the attention layer of a large language model.

[0073] Specifically, to further optimize cross-modal interaction, this application introduces a linear projection layer to map image features to the same feature space as text features. Specifically, the image features extracted by the visual encoder are typically high-dimensional vectors, but their feature space may differ from the text feature space extracted by the text encoder. The linear projection layer maps the image features to a feature space aligned with the text features, generating aligned visual embedding vectors. The purpose of this step is to ensure that image features and text features are semantically comparable, thereby providing more consistent feature input for subsequent cross-modal interaction.

[0074] This embodiment of the application uses a simple linear transformation layer to map the image features extracted by the MedClip visual encoder into the feature space of the language model. Specifically, the visual encoder encodes the input X-ray image as an embedding vector, and then maps it to a 512-dimensional output dimension through a linear projection head. Visual features and textual features are mapped to the same embedding space, enabling the model to align these two features through contrastive learning or other methods, thereby allowing the model to understand and generate textual information corresponding to the chest X-ray.

[0075] The aligned features, along with textual prompts (such as a doctor's diagnostic query), are fed into the frozen Vicuna large language model, and X-ray summaries, diagnostic reports, or answers to doctor's queries related to the input images are generated using the model's pre-trained weights and fine-tuned parameters.

[0076] Optionally, the attention layer includes a self-attention layer and a cross-attention layer;

[0077] The method of achieving interaction between query vectors and visual embedding vectors through the attention layer of a large language model includes:

[0078] The interaction between query vectors is achieved through a self-attention layer, and the cross-attention layer enables cross-modal interaction between query vectors and image features, as well as cross-modal interaction between query vectors and text features.

[0079] Specifically, in cross-modal interaction, the attention layer is the core mechanism for realizing the interaction between image features, text features, and query vectors. The attention layer in this embodiment includes a self-attention layer and a cross-attention layer. The self-attention layer is used to realize the interaction between query vectors, capturing the semantic relationships within the query vectors; the cross-attention layer is used to realize cross-modal interaction between query vectors and image features, and between query vectors and text features. Through this hierarchical attention mechanism, the model can dynamically fuse image and text information to generate more semantically consistent cross-modal features.

[0080] In the self-attention layer, query vectors interact internally through a self-attention mechanism, calculating relevance weights between query vectors to capture semantic associations within them. In the cross-attention layer, query vectors interact with both visual embedding vectors and text features, calculating relevance weights between query vectors and image features, and between query vectors and text features. For example, query vectors can focus on specific regions in an image or key descriptions in text through the cross-attention layer, achieving semantic alignment between images and text. Through this hierarchical attention mechanism, the model can progressively fuse image and text information to generate fused cross-modal features.

[0081] Optionally, the step of mapping the image features to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors includes:

[0082] The image features are used as embedding vectors. The image embedding vectors are input into a learnable linear projection layer and mapped to the feature space of the language model to generate visual embedding vectors of the target dimension.

[0083] The image features and text features of the visual embedding vector are mapped to the same embedding space, and cross-modal feature alignment is achieved through contrastive learning to obtain the visual embedding vector.

[0084] Specifically, to further optimize the alignment between visual embedding vectors and text features, this application introduces a contrastive learning mechanism. Contrastive learning is an unsupervised learning method that achieves feature space alignment by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs.

[0085] In this embodiment, visual embedding vectors and text features are mapped to the same embedding space, and their alignment is optimized through contrastive learning. During contrastive learning, positive sample pairs consist of matching images and text descriptions, while negative sample pairs consist of mismatched images and text descriptions. By calculating similarity scores for positive and negative sample pairs, the model optimizes the alignment of visual embedding vectors and text features using a loss function (such as contrastive loss). For example, for X-ray images and text descriptions, contrastive learning makes image features and text features more consistent in the embedding space, thereby improving the performance of cross-modal tasks. Through contrastive learning, the model can better achieve semantic alignment between images and text, providing higher-quality feature inputs for subsequent cross-modal interactions.

[0086] Optionally, the training method for the large language model includes:

[0087] By utilizing image-text pairs from the MIMIC-CXR dataset, we optimize the cross-modal alignment between image features and text features through contrastive learning to obtain a pre-trained model.

[0088] By using image-text pairs from the OpenI dataset, the parameters of the pre-trained model are fine-tuned to enhance the model's text generation capabilities.

[0089] Specifically, the large language model in this application is trained based on an image-text sample dataset and optimized through a visual and text description alignment task. During training, the model learns how to semantically align image features with text features, such as associating a region in an image with a specific description in the text. This alignment training enables the model to better understand the semantic relationship between images and text, thereby demonstrating stronger capabilities in cross-modal tasks.

[0090] Building upon pre-training, large language models are further optimized through parameter fine-tuning. This fine-tuning process typically employs a partial parameter freezing approach, preserving general features from the pre-trained model and optimizing only task-specific parameters. This design not only reduces training complexity but also mitigates the risk of overfitting while ensuring the model's generalization ability across modalities. Through training and fine-tuning, large language models can better adapt to the needs of specific tasks, such as image-to-text alignment in medical image analysis.

[0091] It's worth noting that the large language model was pre-trained using the MIMIC-CXR dataset, employing a large number of image-text pairs to teach the model how to align visual features with textual descriptions. Subsequently, it was fine-tuned using 3000 carefully selected image-text pairs from the OpenI dataset to further improve the model's performance and ensure it performs better on specific tasks.

[0092] Optionally, the method for constructing the dataset includes:

[0093] Obtain the first raw data from the MIMIC-CXR dataset, which is used to train and test the model; obtain the second raw data from the OpenI dataset, which is used for model fine-tuning.

[0094] The text report of the raw data is preprocessed, and the preprocessing process includes: removing incomplete reports, excessively short text and special symbols, deleting historical comparison statements and excluding medical image view information from the report summary;

[0095] The preprocessed text reports are merged to obtain an optimized image-text pair dataset, which is used for model training, testing, and fine-tuning.

[0096] Specifically, in this embodiment, the first raw data is obtained from the MIMIC-CXR dataset. MIMIC-CXR is a large-scale chest X-ray dataset containing a large number of medical images and their corresponding text reports. This data is used in the training and testing phases of the model, providing the model with rich image-text pair samples. The MIMIC-CXR dataset is characterized by its diversity and realism, covering various clinical scenarios and pathological conditions, which helps the model learn a wide range of medical knowledge and cross-modal associations. By using the MIMIC-CXR dataset, the model can fully capture the semantic relationships between images and text during the training phase and verify its generalization ability during the testing phase.

[0097] The second set of raw data was obtained from the OpenI dataset. OpenI is another publicly available medical imaging dataset containing chest X-rays and their corresponding text reports. Unlike the MIMIC-CXR dataset, the OpenI dataset is typically used for model fine-tuning. The goal of fine-tuning is to improve the model's performance in specific tasks or scenarios. By using the OpenI dataset, the model can further optimize its parameters to adapt to different text description styles and clinical needs. For example, the text reports in the OpenI dataset may contain more detailed descriptions or specific terminology, which can help the model generate more accurate text analysis results after fine-tuning.

[0098] It should be noted that MIMIC-CXR contains 377,110 chest X-ray images and 227,827 related free-text radiology reports, primarily used for training and testing the model. OpenI contains 6,459 chest X-ray images and 3,955 related reports, used for further fine-tuning the model.

[0099] The text report is then preprocessed to remove incomplete reports, excessively short text, special symbols, historical comparison statements, and medical image view information, ensuring the text is clean and consistent. The preprocessed text report is then merged with the medical images to generate an optimized image-text pair dataset, providing high-quality input for the model. This data processing method significantly improves the model's performance in cross-modal tasks, enabling it to better integrate image and text information and generate more accurate analysis results.

[0100] Optionally, it also includes:

[0101] The ROUGE score is used as a quantitative indicator to evaluate the similarity between the text generated by the model and the real report, in order to measure the performance of the large language model in real-world situations.

[0102] The quality and coherence of the generated text were analyzed based on the generative pre-trained model GPT, and compared with the baseline model to obtain the GPT evaluation results.

[0103] Receive user input, score the analysis results based on the user input, and obtain user evaluation results;

[0104] The overall performance of the large language model is comprehensively evaluated by combining the ROUGE score, GPT evaluation results, and user evaluation results, resulting in a comprehensive evaluation result for the model.

[0105] Specifically, this application uses the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score as the main indicator. The ROUGE score is an evaluation metric widely used in text generation tasks, which measures the quality of the generated text by calculating the degree of overlap (such as words, phrases, or sentences) between the generated text and the reference text.

[0106] In this embodiment, the ROUGE score is used to evaluate the similarity between the text generated by the large language model and the real medical report, thereby measuring the model's performance in real-world situations. For example, ROUGE-1 and ROUGE-2 measure the degree of overlap between the generated text and the reference text in terms of word semantics and word semantics, respectively, while ROUGE-L measures the similarity between the generated text and the reference text in terms of sentence structure.

[0107] When calculating the ROUGE score, the model-generated text is first aligned with the real report, and then the ROUGE tool is used to calculate the overlap rate between the generated text and the reference text. For example, if the model-generated text is highly consistent with the real report in key terms and descriptive language, the ROUGE score will be higher, indicating that the generated text is of good quality.

[0108] To further evaluate the quality and coherence of the generated text, this application introduces the generative pre-trained model GPT as an evaluation tool. GPT is used to analyze the semantic, syntactic, and coherent performance of the text generated by the model. By comparing it with text generated by a baseline model, GPT evaluation can reveal the strengths and weaknesses of the generated text.

[0109] To evaluate model performance from a practical application perspective, this application introduces a user evaluation mechanism. User evaluation involves receiving user input and scoring the generated text based on the user's subjective feelings. Users can be radiologists, medical researchers, or experts in other related fields, who score the generated text based on its accuracy, readability, and usability. User evaluation can reflect the model's performance in real-world applications, compensating for the limitations of quantitative indicators.

[0110] In user evaluation, users first review the text generated by the model and then rate it according to preset scoring criteria (such as accuracy, readability, and usability). The ratings can be numerical (e.g., 1-5 points) or qualitative (e.g., "Excellent," "Good," "Average," etc.). By collecting ratings from multiple users, an average or composite score can be calculated to assess the model's performance in real-world applications. User evaluation not only provides real-world feedback on model performance but also offers crucial insights for further model optimization.

[0111] Finally, the overall performance of the large language model is comprehensively evaluated by combining the ROUGE score, GPT evaluation results, and user evaluation results.

[0112] Referring to Table 1 below, the model of this invention has been comprehensively tested and validated in generating high-quality, accurate chest X-ray reports using the evaluation methods described above. The evaluation results show that the model performs excellently in multiple aspects, significantly outperforming the baseline model, and demonstrates reliability in identifying chest X-rays and generating corresponding reports.

[0113] Table 1:

[0114]

[0115] MedCLIP, MedVicuna, and RadVicuna represent progressively more advanced optimization techniques introduced into the baseline methods. The aim is to improve the accuracy and semantic consistency of the generated text by enhancing the model's understanding of medical images and text. As shown in the table, with the introduction of these techniques, the model's scores on the R-1, R-2, and RL metrics gradually increase, indicating significant performance optimization.

[0116] Reference Figure 2 , Figure 2 This is a second flowchart of an embodiment of this application, including the following steps:

[0117] Input a chest X-ray image into the Medical Vision Encoder (VLM);

[0118] Linear transformation;

[0119] Provides a basic description of an X-ray, as well as the main contents and findings of a given X-ray, to a medical large language model (LLM).

[0120] Finally, the X-ray analysis results are output.

[0121] Reference Figure 3 This application provides a vision- and language-based X-ray analysis device, comprising:

[0122] The feature extraction module 310 is used to extract image features of the X-ray film to be analyzed using a visual encoder and to extract text features of the X-ray film to be analyzed using a text encoder.

[0123] The feature fusion module 320 is used to obtain a learnable query vector. The image features, text features and query vector are input into the trained large language model. The attention layer of the large language model performs cross-modal interaction on the query vector, visual embedding vector and text features to obtain the fused cross-modal features.

[0124] Analysis module 330 is used to perform recognition analysis based on the cross-modal features to obtain the analysis results of the X-ray image and text to be analyzed;

[0125] The large language model is obtained by training and fine-tuning the parameters based on the visual and textual description alignment of the image-text samples in the dataset.

[0126] Optionally, it also includes an embedding module for:

[0127] The image features are mapped to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors;

[0128] The feature fusion module is specifically used for:

[0129] The interaction between query vectors and visual embedding vectors is achieved through the attention layer of a large language model.

[0130] Optionally, the attention layer includes a self-attention layer and a cross-attention layer;

[0131] The method of achieving interaction between query vectors and visual embedding vectors through the attention layer of a large language model includes:

[0132] The interaction between query vectors is achieved through a self-attention layer, and the cross-attention layer enables cross-modal interaction between query vectors and image features, as well as cross-modal interaction between query vectors and text features.

[0133] Optionally, the step of mapping the image features to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors includes:

[0134] The image features are used as embedding vectors. The image embedding vectors are input into a learnable linear projection layer and mapped to the feature space of the language model to generate visual embedding vectors of the target dimension.

[0135] The image features and text features of the visual embedding vector are mapped to the same embedding space, and cross-modal feature alignment is achieved through contrastive learning to obtain the visual embedding vector.

[0136] Optionally, the training method for the large language model includes:

[0137] By utilizing image-text pairs from the MIMIC-CXR dataset, we optimize the cross-modal alignment between image features and text features through contrastive learning to obtain a pre-trained model.

[0138] By using image-text pairs from the OpenI dataset, the parameters of the pre-trained model are fine-tuned to enhance the model's text generation capabilities.

[0139] Optionally, the method for constructing the dataset includes:

[0140] Obtain the first raw data from the MIMIC-CXR dataset, which is used to train and test the model; obtain the second raw data from the OpenI dataset, which is used for model fine-tuning.

[0141] The text report of the raw data is preprocessed, and the preprocessing process includes: removing incomplete reports, excessively short text and special symbols, deleting historical comparison statements and excluding medical image view information from the report summary;

[0142] The preprocessed text reports are merged to obtain an optimized image-text pair dataset, which is used for model training, testing, and fine-tuning.

[0143] Optionally, an evaluation module is also included for:

[0144] The ROUGE score is used as a quantitative indicator to evaluate the similarity between the text generated by the model and the real report, in order to measure the performance of the large language model in real-world situations.

[0145] The quality and coherence of the generated text were analyzed based on the generative pre-trained model GPT, and compared with the baseline model to obtain the GPT evaluation results.

[0146] Receive user input, score the analysis results based on the user input, and obtain user evaluation results;

[0147] The overall performance of the large language model is comprehensively evaluated by combining the ROUGE score, GPT evaluation results, and user evaluation results, resulting in a comprehensive evaluation result for the model.

[0148] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0149] Reference Figure 4Based on the methods in the above embodiments, this application provides an electronic device that may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions stored in the memory 430 to execute the methods in the above embodiments.

[0150] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0151] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0152] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0153] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0154] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0155] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0156] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0157] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A visual and linguistic based radiograph analysis method, characterized in that, include: A visual encoder is used to extract image features from the X-ray images to be analyzed, and a text encoder is used to extract text features from the X-ray images to be analyzed. The visual encoder is built on the CLIP architecture. During the process of extracting image features, the visual encoder remains frozen and the weights remain unchanged without being updated. Learnable query vectors are obtained, and the image features, text features, and query vectors are input into a trained large language model. The query vectors, image features, and text features are then interacted across modally through the attention layer of the large language model to obtain fused cross-modal features. Based on the cross-modal features, the analysis results of the X-ray image and text to be analyzed are obtained; The ROUGE score is used as a quantitative indicator to evaluate the similarity between the text generated by the model and the real report, in order to measure the performance of the large language model in real-world situations. The quality and coherence of the generated text were analyzed based on the generative pre-trained model GPT, and compared with the baseline model to obtain the GPT evaluation results. Receive user input, score the analysis results based on the user input, and obtain user evaluation results; The overall performance of the large language model is comprehensively evaluated by combining the ROUGE score, GPT evaluation results, and user evaluation results, resulting in a comprehensive evaluation result of the model. The large language model is obtained by training and fine-tuning the parameters based on the visual and textual description alignment of the image-text samples in the dataset.

2. The X-ray film analysis method based on vision and language according to claim 1, characterized in that, Also includes: The image features are mapped to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors; The attention layer of the large language model enables cross-modal interaction of query vectors, image features, and text features, including: The interaction between query vectors and visual embedding vectors is achieved through the attention layer of a large language model.

3. The X-ray analysis method based on vision and language according to claim 2, characterized in that, The attention layer includes a self-attention layer and a cross-attention layer; The method of achieving interaction between query vectors and visual embedding vectors through the attention layer of a large language model includes: The interaction between query vectors is achieved through a self-attention layer, and the cross-attention layer enables cross-modal interaction between query vectors and image features, as well as cross-modal interaction between query vectors and text features.

4. The X-ray film analysis method based on vision and language according to claim 2, characterized in that, The step of mapping the image features to the same feature space as the text features through a linear projection layer to generate aligned visual embedding vectors includes: The image features are used as embedding vectors. The image embedding vectors are input into a learnable linear projection layer and mapped to the feature space of the language model to generate visual embedding vectors of the target dimension. The image features and text features of the visual embedding vector are mapped to the same embedding space, and cross-modal feature alignment is achieved through contrastive learning to obtain the visual embedding vector.

5. The X-ray film analysis method based on vision and language according to claim 1, characterized in that, The training methods for the large language model include: By utilizing image-text pairs from the MIMIC-CXR dataset, we optimize the cross-modal alignment between image features and text features through contrastive learning to obtain a pre-trained model. By using image-text pairs from the OpenI dataset, the parameters of the pre-trained model are fine-tuned to enhance the model's text generation capabilities.

6. The X-ray film analysis method based on vision and language according to claim 1, characterized in that, The methods for constructing the dataset include: Obtain the first raw data from the MIMIC-CXR dataset, which is used to train and test the model; obtain the second raw data from the OpenI dataset, which is used for model fine-tuning. The text report of the raw data is preprocessed, and the preprocessing process includes: removing incomplete reports, excessively short text and special symbols, deleting historical comparison statements and excluding medical image view information from the report summary; The preprocessed text reports are merged to obtain an optimized image-text pair dataset, which is used for model training, testing, and fine-tuning.

7. An X-ray film analysis device based on vision and language, characterized in that, include: The feature extraction module is used to extract image features of the X-ray film to be analyzed using a visual encoder and to extract text features of the X-ray film to be analyzed using a text encoder. The visual encoder is built based on the CLIP architecture. During the image feature extraction process, the visual encoder remains frozen and its weights remain unchanged without being updated. The feature fusion module is used to obtain a learnable query vector. The image features, text features and query vector are input into the trained large language model. The attention layer of the large language model performs cross-modal interaction on the query vector, visual embedding vector and text features to obtain the fused cross-modal features. The analysis module is used to perform recognition and analysis based on the cross-modal features to obtain the analysis results of the X-ray image and text to be analyzed; The evaluation module is used to evaluate the similarity between the text generated by the model and the real report using ROUGE scores as a quantitative indicator, so as to measure the performance of the large language model in real-world situations; it analyzes the quality and coherence of the generated text based on the generative pre-trained model GPT and compares it with the baseline model to obtain GPT evaluation results; it receives user input and scores the analysis results based on the user input to obtain user evaluation results. The overall performance of the large language model is comprehensively evaluated by combining ROUGE scores, GPT evaluation results, and user evaluation results, resulting in a comprehensive evaluation result. The large language model is obtained by visual and text description alignment training and parameter fine-tuning based on image-text samples from the dataset.

8. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-6.

Images