Medical image diagnosis report traceability method and system based on multi-modal large model

Abstract

The invention discloses a medical image diagnosis report traceability method and system based on a multi-modal large model, relates to the technical field of medical traceability, and solves the problems that cross-modal knowledge migration is blocked and a decision basis is difficult to trace due to lack of natural language explanation ability due to excessive dependence on labeling data of a specific modal in the prior art; the generated content lacks a verifiable physical positioning basis, and direct association between the focus position and the text description is not obviously established. The medical image and the diagnosis report of the same case are acquired; extracting key statements and lesion positions in the diagnosis report by using a medical image multi-modal large model; monitoring the key statement and the focus position; performing visual association between the key statement and the focus position to obtain traceability information; the corresponding lesion area in the image can be quickly positioned, and the specific form, characteristics and the like of the lesion can be checked.

Description

Technical Field

[0001] This application belongs to the field of medical traceability and involves traceability technology for medical image diagnostic reports, specifically a traceability method and system for medical image diagnostic reports based on a multimodal large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, the medical field is actively embracing this transformative technology in order to significantly improve the efficiency and accuracy of diagnosis and treatment. In this development, the application of visual language multimodal large models in medical image analysis has emerged as a leading research area. These advanced models possess powerful data processing capabilities, capable of handling diverse medical imaging data such as X-rays, CT scans, MRI, and ultrasound. They not only automatically generate detailed diagnostic reports but also provide accurate diagnostic opinions and treatment suggestions, offering unprecedented support for medical decision-making.

[0003] Among existing methods for traceability of medical image diagnostic reports, those based on single-modal deep learning models can output disease probabilities or lesion regions and generate report text and lesion regions token by token. However, existing technologies rely excessively on annotated data of specific modalities, which hinders cross-modal knowledge transfer and lacks natural language interpretation capabilities, making it difficult to trace the basis for decision-making. Furthermore, the generated content lacks verifiable physical location evidence and does not clearly establish a direct correlation between lesion location and text description. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a traceable method and system for medical image diagnostic reports based on a multimodal large model, which addresses the technical problems of existing technologies that rely excessively on labeled data of specific modalities, resulting in hindered cross-modal knowledge transfer, lack of natural language interpretation capabilities, making it difficult to trace the basis for decision-making, and lack of verifiable physical location evidence for the generated content, failing to establish a clear direct correlation between lesion location and text description.

[0005] To achieve the above objectives, the first aspect of this application provides a method for traceability of medical image diagnostic reports based on a multimodal large model, including: Obtain medical images and diagnostic reports for the same case; Key sentences and lesion locations in diagnostic reports are extracted using a multimodal medical imaging model; Monitor key statements and lesion locations; Visual associations are established between key statements and lesion locations to obtain source information.

[0006] Based on the above steps, using a multimodal large-scale medical imaging model to extract key sentences and lesion locations from diagnostic reports can quickly filter out core information from a large amount of complex diagnostic report text and images. This avoids the tedious process of manual word-by-word reading and analysis, greatly saving doctors' time and energy and improving diagnostic efficiency. The multimodal large-scale model can accurately identify the location of lesions in images, helping doctors make correct diagnoses more quickly, reducing misdiagnosis and missed diagnosis, and improving diagnostic accuracy. Using cross-entropy loss to supervise the key sentences and lesion locations allows the model to continuously adjust parameters during training to minimize the predicted results. The model learns from and is exposed to a wider range of case data through learning and supervision of medical images and diagnostic reports from a large number of cases. This allows the model to learn more universal features and patterns, improving its generalization ability in different cases and scenarios, enabling it to handle various complex situations more accurately in practical applications. Visual association between key statements and lesion locations and the acquisition of source information allows for a direct link between the textual descriptions in diagnostic reports and the actual lesions in the images. Through source information, the corresponding lesion area in the image can be quickly located, and the specific morphology and characteristics of the lesion can be viewed.

[0007] Preferably, the step of extracting key statements and lesion locations from the diagnostic report using a multimodal medical imaging model includes: Retrieve medical images and diagnostic reports of the same case and integrate them into a sentence analysis sequence; The medical imaging multimodal model is invoked; the sentence analysis sequence is input into the medical imaging multimodal model to obtain key sentences and lesion locations; among them, key sentences include: Findings and Impression sentences; the medical imaging multimodal model is constructed based on an artificial intelligence model.

[0008] Preferably, the medical image multimodal large model is constructed based on an artificial intelligence model, including: Obtain the standard dataset; the standard dataset includes standard input data consistent with the content attributes of the sentence analysis sequence; and standard output data consistent with the content attributes of key sentences and lesion locations; Select a model architecture from the model library and integrate the image encoder and text encoder into the model architecture to obtain a multimodal model; The standard dataset is divided into a training set, a validation set, and a test set; the multimodal model is trained using the training set; the weight parameters of the multimodal model are adjusted using the validation set; and the multimodal model is tested using the test set to obtain test metrics. Obtain the indicator thresholds and compare the test indicators with the indicator thresholds; if all test indicators are greater than the indicator thresholds, then mark the multimodal model as a large multimodal medical imaging model; otherwise, reconstruct and retrain the large multimodal medical imaging model.

[0009] Preferably, the monitoring of key statements and lesion locations includes: Retrieve key statements and lesion locations; perform token-by-token supervision of key statements using cross-entropy loss; the expression for the cross-entropy loss is: ; in, These are the trainable parameters of the model; Indicates the input command; This represents the i-th target token in the labeled data. It is the total number of tokens in the sequence; The model is given input Predict token The probability of; It is the edit distance between the predicted traceable token sequence and the standard answer token sequence; IOU loss is used to supervise the lesion location.

[0010] Preferably, the method of monitoring lesion location using IOU loss includes: The expression for the IOU loss is: ;in, This represents the area of ​​intersection between the predicted bounding box B and the ground truth bounding box A; This represents the area of ​​the union between the predicted bounding box B and the ground truth bounding box.

[0011] Preferably, the methods for obtaining the predicted bounding box and the ground truth bounding box include: Retrieve the location of the lesion; select the corresponding location of the lesion in the medical image to obtain the predicted bounding box; obtain the coordinate box of the area where the disease is located in the standard answer marked by professional doctors, and mark the coordinate box as the true bounding box.

[0012] Preferably, the step of visually associating key statements and lesion locations to obtain source tracing information includes: Enhance semantic consistency between Findings and Impression statements by leveraging semantic matching loss; The expression for the semantic matching loss is: ;in, This indicates the characteristics of the Findings statement; This indicates the characteristics of the Impression statement; Visual localization loss is used to establish a visual association between Impression statements and lesion locations.

[0013] Preferably, the step of using visual localization loss to visually associate the Impression statement with the lesion location includes: Visual localization loss is calculated using cosine similarity to obtain visual localization; the expression for the visual localization loss is: ;in, This indicates the characteristics of the Impression statement; Visual features of the imaging region corresponding to the location of the lesion.

[0014] The second aspect of this application provides a traceable medical image diagnostic report system based on a multimodal large model, including: a data acquisition module, a data analysis module, and a data traceability module; The data acquisition module is used to acquire medical images and diagnostic reports for the same case; The data analysis module is used to extract key sentences and lesion locations from the diagnostic report using a multimodal medical image model; and to supervise the key sentences and lesion locations. The data tracing module is used to visually correlate key statements and lesion locations to obtain tracing information.

[0015] A third aspect of this application provides a computer-readable storage medium storing program instructions for performing the method steps described in any possible implementation of the first aspect.

[0016] Compared with the prior art, the beneficial effects of this application are: 1. This application integrates medical images and diagnostic reports of a case into a sentence analysis sequence, and then uses a multimodal medical image model to extract key sentences and lesion locations. The Findings sentence describes in detail the various features observed in the images, while the Impression sentence provides a preliminary diagnostic conclusion based on these features. By accurately extracting these key sentences, the core content of the image diagnosis can be quickly obtained, avoiding the omission of important information and thus improving the accuracy of the diagnosis. The multimodal model can accurately identify the location of lesions in the images and provide intuitive visual references. Combined with the extraction of key sentences, the characteristics of the lesions and the diagnostic basis can be understood more comprehensively, further reducing the possibility of misdiagnosis and missed diagnosis.

[0017] 2. This application utilizes cross-entropy loss for token-by-token supervision of key statements, enabling the model to predict each token more accurately and improving the accuracy of key statement extraction. It employs IOU loss to supervise lesion location, measuring the overlap between predicted and ground truth boxes, thus promoting more accurate lesion location. The combined use of these two loss functions optimizes for the different characteristics of key statements and lesion locations, further enhancing model performance. Semantic matching loss enhances the semantic consistency between Findings and Impression statements, ensuring consistency and coherence of semantics across different parts of the diagnostic report. This helps to more clearly understand the basis of diagnostic conclusions, avoiding misunderstandings caused by semantic inconsistencies and improving the readability and reliability of the diagnostic report. Visual localization loss establishes a visual association between Impression statements and lesion locations, tightly integrating textual and image information. Through this association, the image region corresponding to the diagnostic conclusion can be visually observed. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the overall method steps of this application; Figure 2 This is a schematic diagram illustrating the specific method steps of this application; Figure 3 This is a schematic diagram of the system module connection structure of this application; Figure 4 This is a schematic diagram illustrating the traceability information of this application; Figure 5 This is a schematic diagram visualizing the location of the lesion in this application. Detailed Implementation

[0020] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] Please see Figure 1 The first aspect of this application provides a method for traceability of medical image diagnostic reports based on a multimodal large model, including: S101. Obtain medical images and diagnostic reports for the same case; S102. Utilize a multimodal medical imaging model to extract key statements and lesion locations from diagnostic reports; S103. Monitor key statements and lesion locations; S104. Visually correlate key statements with lesion locations to obtain source information.

[0022] Based on the above steps, using a multimodal large-scale medical imaging model to extract key sentences and lesion locations from diagnostic reports can quickly filter out core information from a large amount of complex diagnostic report text and images. This avoids the tedious process of manual word-by-word reading and analysis, greatly saving doctors' time and energy and improving diagnostic efficiency. The multimodal large-scale model can accurately identify the location of lesions in images, helping doctors make correct diagnoses more quickly, reducing misdiagnosis and missed diagnosis, and improving diagnostic accuracy. Using cross-entropy loss to supervise the key sentences and lesion locations allows the model to continuously adjust parameters during training to minimize the predicted results. The model learns from and is exposed to a wider range of case data through learning and supervision of medical images and diagnostic reports from a large number of cases. This allows the model to learn more universal features and patterns, improving its generalization ability in different cases and scenarios, enabling it to handle various complex situations more accurately in practical applications. Visual association between key statements and lesion locations and the acquisition of source information allows for a direct link between the textual descriptions in diagnostic reports and the actual lesions in the images. Through source information, the corresponding lesion area in the image can be quickly located, and the specific morphology and characteristics of the lesion can be viewed.

[0023] In one possible implementation of the embodiments of this application, combined with Figure 1 ,like Figure 2 As shown, the above S102-S104 can be specifically implemented through the following S201-S204, which are explained in detail below: S201. Retrieve medical images and diagnostic reports of the same case and integrate them into a sentence analysis sequence.

[0024] S202. Call the medical imaging multimodal large model; input the statement analysis sequence into the medical imaging multimodal large model to obtain the key statements and lesion locations.

[0025] The key statements include the Findings and Impression statements.

[0026] In some possible implementations, the medical imaging multimodal large model is built based on an artificial intelligence model, including: Obtain the standard dataset; the standard dataset includes standard input data consistent with the content attributes of the sentence analysis sequence; and standard output data consistent with the content attributes of key sentences and lesion locations; Select a model architecture from the model library and integrate the image encoder and text encoder into the model architecture to obtain a multimodal model; The standard dataset is divided into a training set, a validation set, and a test set; the multimodal model is trained using the training set; the weight parameters of the multimodal model are adjusted using the validation set; and the multimodal model is tested using the test set to obtain test metrics. Obtain the indicator thresholds and compare the test indicators with the indicator thresholds; if all test indicators are greater than the indicator thresholds, then mark the multimodal model as a large multimodal medical imaging model; otherwise, reconstruct and retrain the large multimodal medical imaging model.

[0027] Example: such as Figure 4 As shown: Using a multimodal medical imaging model, the medical images and diagnostic reports of the current case are analyzed to precisely extract paired Findings and Impression statements corresponding to the disease from the diagnostic report. The Findings statement is: "Clearly defined linear fibrous sclerosis and punctate uneven density increases are seen in the right upper lung field, with clear margins, some of which are calcifications; no enlargement or thickening of the bilateral hilum; a band-like increase in density is seen along the lateral chest wall in the outer zone of the right lower lung field, and the right costophrenic angle is blunted; the size and shape of the heart are normal, and the aorta is normal; the mediastinum is in the center and not enlarged; the two sides of the thoracic cage are symmetrical and without deformity; other: no abnormalities are seen." The Impression statement is: "Old pulmonary tuberculosis in the right upper lobe? Right pleural thickening with calcification, please correlate with clinical findings."

[0028] like Figure 5 The lesion locations (Boxes) determined by the multimodal large model of medical imaging are shown.

[0029] S203. Retrieve key statements and lesion locations; use cross-entropy loss to supervise key statements token by token; use IOU loss to supervise lesion locations.

[0030] The expression for cross-entropy loss is: ; These are the trainable parameters of the model; Indicates the input command; This represents the i-th target token in the labeled data. It is the total number of tokens in the sequence; The model is given input Predict token The probability of; It is the edit distance between the predicted traceable token sequence and the standard answer's token sequence; the expression for IOU loss is: ; This represents the area of ​​intersection between the predicted bounding box B and the ground truth bounding box A; This represents the area of ​​the union between the predicted bounding box B and the ground truth bounding box.

[0031] In one possible implementation, the methods for obtaining the predicted bounding box and the ground truth bounding box include: Retrieve the location of the lesion; select the corresponding location of the lesion in the medical image to obtain the predicted bounding box; obtain the coordinate box of the area where the disease is located in the standard answer marked by professional doctors, and mark the coordinate box as the true bounding box.

[0032] S204. Enhance the semantic consistency between Findings and Impression statements using semantic matching loss; use visual localization loss to visually associate Impression statements with lesion locations.

[0033] The expression for semantic matching loss is: ; This indicates the characteristics of the Findings statement; The features representing the Impression statement; the expression for visual localization loss is: ; This indicates the characteristics of the Impression statement; Visual features of the imaging region corresponding to the location of the lesion.

[0034] Based on the above steps, after integrating the medical images and diagnostic reports of the case into a sentence analysis sequence, a multimodal medical image model is used to extract key sentences and lesion locations. The "Findings" sentence describes in detail the various features observed in the images, while the "Impression" sentence provides a preliminary diagnostic conclusion based on these features. Accurate extraction of these key sentences allows for rapid acquisition of the core content of the image diagnosis, avoiding the omission of important information and thus improving diagnostic accuracy. The multimodal model can accurately identify the location of lesions in the images, providing intuitive visual references. Combined with the extraction of key sentences, a more comprehensive understanding of the characteristics of the lesions and diagnostic criteria can be achieved, further reducing the possibility of misdiagnosis and missed diagnosis. Using cross-entropy loss for token-by-token supervision of key sentences enables the model to predict each token more accurately, improving the extraction of key sentences. Accuracy: The use of IOU loss to supervise lesion location measures the overlap between predicted and ground truth boxes, enabling the model to more accurately locate lesions. The combined use of two loss functions, optimized for the different characteristics of key statements and lesion locations, further improves model performance. Semantic matching loss enhances the semantic consistency between Findings and Impression statements, ensuring consistency and coherence in the semantics of different parts of the diagnostic report. This helps to more clearly understand the basis of diagnostic conclusions, avoids misunderstandings caused by semantic inconsistencies, and improves the readability and reliability of diagnostic reports. Visual localization loss establishes a visual association between Impression statements and lesion locations, tightly integrating textual and image information. Through this association, the image region corresponding to the diagnostic conclusion can be seen intuitively.

[0035] Please see Figure 3 The second aspect of this application provides a method for traceable medical image diagnostic reports based on a multimodal large model, including: a data acquisition module, a data analysis module, and a data traceability module; The data acquisition module is used to acquire medical images and diagnostic reports for the same case; The data analysis module is used to extract key sentences and lesion locations from the diagnostic report using a multimodal medical image model; and to supervise the key sentences and lesion locations. The data tracing module is used to visually correlate key statements and lesion locations to obtain tracing information.

[0036] A third aspect of this application provides a computer-readable storage medium storing program instructions for performing the method steps described in any possible implementation of the first aspect.

[0037] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0038] The working principle of this application is as follows: This application acquires medical images and diagnostic reports of the same case; extracts key sentences and lesion locations from the diagnostic reports using a multimodal large model of medical images; supervises the key sentences and lesion locations; and performs visual correlation between the key sentences and lesion locations to obtain traceability information.

[0039] The above embodiments are only used to illustrate the technical methods of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of this application without departing from the spirit and scope of the technical methods of this application.

Claims

1. A method for traceability of medical image diagnosis report based on multi-modal large model, characterized in that, include: Obtain medical images and diagnostic reports for the same case; Key sentences and lesion locations in diagnostic reports are extracted using a multimodal medical imaging model; Monitor key statements and lesion locations; Visual associations are established between key statements and lesion locations to obtain source information. 2.The multi-modal large model based medical image diagnosis report traceability method of claim 1, wherein, The method of extracting key statements and lesion locations from diagnostic reports using a multimodal large model of medical images includes: Retrieve medical images and diagnostic reports of the same case and integrate them into a sentence analysis sequence; The medical imaging multimodal model is invoked; the sentence analysis sequence is input into the medical imaging multimodal model to obtain key sentences and lesion locations; among them, key sentences include: Findings and Impression sentences; the medical imaging multimodal model is constructed based on an artificial intelligence model.

3. The method for traceable medical image diagnostic reports based on a multimodal large model according to claim 2, characterized in that, The aforementioned medical imaging multimodal large model is constructed based on an artificial intelligence model and includes: Obtain the standard dataset; the standard dataset includes standard input data consistent with the content attributes of the sentence analysis sequence; and standard output data consistent with the content attributes of key sentences and lesion locations; Select a model architecture from the model library and integrate the image encoder and text encoder into the model architecture to obtain a multimodal model; The standard dataset is divided into a training set, a validation set, and a test set; the multimodal model is trained using the training set; the weight parameters of the multimodal model are adjusted using the validation set; and the multimodal model is tested using the test set to obtain test metrics. Obtain the indicator thresholds and compare the test indicators with the indicator thresholds; if all test indicators are greater than the indicator thresholds, then mark the multimodal model as a large multimodal medical imaging model; otherwise, reconstruct and retrain the large multimodal medical imaging model. 4.The multi-modal large model based medical image diagnosis report traceability method of claim 1, wherein, The monitoring of key statements and lesion locations includes: The key sentence and the lesion position are called; the key sentence is supervised by token by using cross-entropy loss; and the expression of the cross-entropy loss is: ; in, These are the trainable parameters of the model; Indicates the input command; This represents the i-th target token in the labeled data. It is the total number of tokens in the sequence; The model is given input Predict token The probability of; It is the edit distance between the predicted traceable token sequence and the standard answer token sequence; IOU loss is used to supervise the lesion location.

5. The method for traceable medical image diagnostic reports based on a multimodal large model according to claim 4, characterized in that, The method of using IOU loss to monitor lesion location includes: The expression for the IOU loss is: ;in, This represents the area of ​​intersection between the predicted bounding box B and the ground truth bounding box A; This represents the area of ​​the union between the predicted bounding box B and the ground truth bounding box. 6.The multi-modal large model based medical image diagnosis report traceability method of claim 5, wherein, The methods for obtaining the predicted bounding box and the ground truth bounding box include: Retrieve the location of the lesion; select the corresponding location of the lesion in the medical image to obtain the predicted bounding box; obtain the coordinate box of the area where the disease is located in the standard answer marked by professional doctors, and mark the coordinate box as the true bounding box.

7. The method for traceable medical image diagnostic reports based on a multimodal large model according to claim 1, characterized in that, The process of visually associating key statements with lesion locations to obtain source tracing information includes: Enhance semantic consistency between Findings and Impression statements by leveraging semantic matching loss; The expression of the semantic matching loss is: ; wherein, represents the features of the Findings statement; represents the features of the Impression statement; Visual localization loss is used to establish a visual association between Impression statements and lesion locations. 8.The multi-modal large model based medical image diagnosis report traceability method of claim 1, wherein, The method of using visual localization loss to establish a visual association between Impression statements and lesion locations includes: The visual positioning is obtained by calculating a visual positioning loss by using cosine similarity; an expression of the visual positioning loss is: ; wherein, Impression indicates a feature of an Impression statement; Visual feature indicates a visual feature of an image region corresponding to a lesion position.

9. A medical image diagnosis report traceable system based on a multi-modal large model, applied to the medical image diagnosis report traceable method based on a multi-modal large model in any one of claims 1-8, characterized in that, include: Data acquisition module, data analysis module, and data traceability module; The data acquisition module is used to acquire medical images and diagnostic reports for the same case; The data analysis module is used to extract key sentences and lesion locations from the diagnostic report using a multimodal medical image model; and to supervise the key sentences and lesion locations. The data tracing module is used to visually correlate key statements and lesion locations to obtain tracing information.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions for performing the steps of the method described in any one of claims 1-8.

Images