Medical image diagnosis method, device and system and storage medium

Abstract

The invention discloses a medical image diagnosis method, device and system and a storage medium, and the method comprises the steps: carrying out the preliminary diagnosis of a medical image through a medical context big language model, and generating a medical report; quantifying the credibility of each sentence in the medical report by using an uncertainty measurement algorithm to obtain a credibility score; dynamically marking report contents by using color gradient mapping, and highlighting high-risk diagnosis entries; generating a structured visual thermodynamic diagram for each sentence of report by utilizing an explainable method based on explicit attribution and an attention mechanism; a React-based front-end interaction framework is utilized to integrate visual interpretation and report editing functions, explainable diagnosis decision basis and risk entry early warning are provided, and doctors are assisted in visual and rapid targeted verification; a continuous learning framework is adopted, feedback of doctors is collected, and sentence-level correction and a flexible training strategy are introduced to continuously optimize diagnosis precision. According to the invention, the interpretability and continuous learning ability of medical image diagnosis are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a medical image diagnosis method, device, system, and storage medium. Background Technology

[0002] The surge in medical imaging data has led to increasingly complex medical imaging diagnostic tasks. Traditional manual imaging diagnostic methods are gradually failing to meet actual clinical needs, prompting the development of AI-assisted clinical diagnostic systems. However, existing systems still primarily operate on a deterministic, one-way output model, lacking interpretable support for the diagnostic reasoning process. This black-box decision-making model conflicts with the high-risk nature of medical scenarios, forcing doctors to fully review AI conclusions and even reverse-engineer their reasoning paths, thus increasing their workload. Furthermore, these systems generally lack dynamic interaction mechanisms, preventing AI from learning from doctors' corrections, leading to repeated errors in similar cases and triggering a crisis of trust among doctors. Even when AI generates correct conclusions, doctors still need to invest equal effort in verification based on past corrections, ultimately creating a vicious cycle of "more corrections - deeper questioning - more frequent verification." To address these issues, there is an urgent need for an intelligent medical imaging diagnostic system that integrates reliable quantification, visual interpretation, and continuous learning mechanisms. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention provides a medical imaging diagnostic method, device, system, and storage medium, enabling two-way empowerment of interpretable dynamic traceability of diagnostic decisions and optimization of clinical feedback.

[0004] To achieve the above objectives, the present invention provides the following solution: A medical imaging diagnostic method, comprising: Using a large language model of medical context, preliminary diagnosis of medical images is performed, and medical reports are generated. The credibility of each sentence in a medical report is quantified using an uncertainty measurement algorithm to obtain a credibility score. Based on the confidence score, the report content is dynamically labeled using color gradient mapping to obtain high-risk diagnostic items; Generate a heatmap of medical image decision-making criteria for each sentence; Using a React-based front-end interaction framework, a heatmap of medical imaging decision-making criteria and high-risk diagnostic entries are displayed.

[0005] As a preferred approach, an interpretable method based on explicit attribution and attention mechanisms is used to generate a heatmap of medical image decision-making criteria for each sentence.

[0006] As an alternative, it also includes: adopting a continuous learning framework, introducing sentence-level correction and flexible training strategies, and continuously optimizing the medical context large language model and diagnostic accuracy.

[0007] The present invention also provides a medical imaging diagnostic device, comprising: The first processing module is used to perform preliminary diagnosis of medical images and generate medical reports by utilizing a large medical context language model. The second processing module is used to quantify the credibility of each sentence in the medical report using an uncertainty measurement algorithm to obtain a credibility score; The third processing module is used to dynamically label the report content based on the confidence score using color gradient mapping to obtain high-risk diagnostic items; The fourth processing module is used to generate a heatmap of medical image decision-making basis for each sentence; The fifth processing module is used to display a heatmap of medical image decision-making criteria and high-risk diagnostic entries using a React-based front-end interaction framework.

[0008] As a preferred option, the third processing module uses an interpretable method based on explicit attribution and attention mechanisms to generate a heatmap of medical image decision-making basis for each sentence.

[0009] As a preferred embodiment, it also includes a sixth processing module, which employs a continuous learning framework to introduce sentence-level correction and flexible training strategies to continuously optimize the medical context large language model and diagnostic accuracy.

[0010] The present invention also provides a medical imaging diagnostic system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program performs a medical imaging diagnostic method when executed by the processor.

[0011] The present invention also provides a storage medium storing a computer program, which executes a medical image diagnosis method when running.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention combines uncertainty analysis to achieve sentence-level credibility assessment and provide risk warnings for false descriptions and omissions of key information in diagnosis; at the same time, it designs an interpretable method based on explicit attribution and attention mechanisms to intuitively demonstrate the reasoning basis of the model and improve the interpretability and clinical credibility of the model's predictions.

[0013] 2. This invention introduces a flexible training strategy based on doctor feedback, which achieves progressive learning by editing the intensity-weighted loss function to avoid model overfitting or catastrophic forgetting; and without changing the backbone model structure, it introduces an Adapter module as a pluggable fine-tuning component, which enables the model to efficiently absorb expert knowledge and support personalized continuous optimization, thereby improving the stability, adaptability and actual deployment efficiency of the automatic report generation model. Attached Figure Description

[0014] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a medical imaging diagnostic method according to an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1 like Figure 1 As shown, the present invention provides a medical imaging diagnostic method, comprising: S100: Utilizes a large language model in medical context to perform preliminary diagnosis of medical images and generate medical reports.

[0019] The large language model takes medical images as input, and its architecture includes, but is not limited to, a Transformer model based on an encoder-decoder structure, a cross-modal fusion model, or a multimodal generation model that integrates visual and linguistic features. The encoder can be a convolutional neural network, a visual Transformer, or other image encoding methods, and the decoder can be a text generation model such as BART or GPT. This embodiment does not limit the specific model structure during implementation; it can be flexibly selected according to actual application needs to ensure the method's versatility and scalability.

[0020] S200: The credibility of each sentence in a medical report is quantified using an uncertainty measurement algorithm to obtain a credibility score.

[0021] As one embodiment of the present invention, the uncertainty measurement algorithm takes the predicted probability distribution as input and calculates the average entropy to reflect the credibility of the generated report. It is assumed that the predicted probability distribution of each token in each sentence of the medical report is as follows: , For the category label of the token, calculate the entropy E according to the following formula: (1) Then calculate the overall credibility score for each sentence: (2) in, This indicates the number of tokens contained in the sentence. This indicates the vocabulary size; the resulting confidence score is used to mark high-risk diagnostic entries.

[0022] S300: Uses color gradient mapping to dynamically annotate report content, highlighting high-risk diagnostic items.

[0023] As one embodiment of the present invention, the color mapping function Using the credibility score as input, it is defined as: (3) in, The credibility score ranges from 0 to 1. Low credibility is marked in red, high credibility in green, and intermediate credibility is marked in a smooth transition, making the risk level of each sentence in the report intuitively visible.

[0024] S400: Utilizes an interpretable approach based on explicit attribution and attention mechanisms to generate structured visual heatmaps for each report.

[0025] Current model-generated reports lack transparent and traceable decision-making basis, reducing the efficiency of doctors' review. Traditional heatmap generation methods only use channel gradient averaging, which may ignore some important regions and is not suitable for language models or multimodal tasks involving text and images. To address these issues, the method provided in this embodiment integrates explicit attribution and attention mechanisms, aiming to provide more accurate, reliable, and fine-grained model decision-making basis.

[0026] Integrated Gradients is a classic attribution method used to measure the contribution of a specific part of the input (such as a region of an image) to a specific output target (such as a word). Its core idea is to gradually transition from a "baseline input" (such as an image with all zeros) to the "true input," accumulating gradients along this path to derive the attribution value. SpatialAttention represents which regions of the image the model focuses on when generating a word / sentence. Combining the two allows us to know not only the regions the model focuses on but also the degree to which these regions contribute to the final prediction result.

[0027] As one embodiment of the present invention, the interpretable method first sets an all-zero image. Based on this, the original image is processed using an encoder based on the Transformer architecture. and benchmark images To obtain the actual input and Several intermediate states on the integration path are constructed through successive interpolation. For each intermediate state Forward generation probability Calculate its gradient with respect to each visual token: (4) Using the Riemann approximation to calculate the integral attribution value IG: (5) in The number of intermediate states; the method obtains the sentence-level attention weight distribution through the cross-attention layer of the model decoder. The specific calculation formula is as follows: (6) in, Indicates the number of layers in the decoder. Indicates the number of heads of attention. For the decoder Layer The query vector for the head attention mechanism This is the key vector corresponding to the visual token; finally, the IG value and attention distribution are fused by multiplication to generate a heatmap. : (7) This is used to visually demonstrate the correspondence between each sentence and the basis for medical imaging decisions.

[0028] S500: Utilizes a React-based front-end interaction framework to integrate visual explanation and report editing functions, providing interpretable diagnostic decision-making basis and risk item warnings to assist doctors in intuitive and rapid targeted verification.

[0029] As one embodiment of the present invention, the interactive interface uses the React framework to realize the interaction between medical images and reports; through component-based development, the image viewing component, report display component, heatmap linkage component, and report editing component are integrated into a unified interactive platform; the report display component dynamically marks high-risk sentences, the heatmap linkage component displays a heatmap of the medical image decision basis for the corresponding sentence in real time when the doctor clicks on the report sentence, and the report editing component allows doctors to modify medical reports in real time and record the modifications for continuous learning.

[0030] S600: Employs a continuous learning framework, collects doctor feedback, and introduces sentence-level correction and flexible training strategies to continuously optimize diagnostic accuracy.

[0031] As one embodiment of the present invention, the continuous learning framework collects doctors' modification behaviors on reports and structures them into high-quality expert feedback samples. It doesn't simply allow the model to relearn the "correct reports" written by doctors, but rather allows the model to learn "how doctors correct erroneous reports." The framework utilizes plug-in fine-tuning technology, inserting a small adapter module into each layer of the original model. Only the parameters of this small module are trained, while the original model parameters are frozen, balancing performance and deployment convenience.

[0032] To prevent the model from forgetting old knowledge after learning new knowledge, the continuous learning framework employs a flexible training strategy to ensure the stability of the generative model. This flexible training strategy first calculates the sentence edit distance (Levenshtein distance) for each sample: (8) in, Indicates the preceding clause of sentence A The character and the first B Edit distance of one character, if If the cost is 0, then the cost is 0; otherwise, it is 1. Further calculation of the edit intensity score (edr) follows: (9) in, This indicates the calculation of sentence edit distance. This represents the initial results generated by the model. This represents the doctor's revised report; the final training process uses a loss function weighted by edit intensity scores for optimization. (10) in, Cross-entropy loss is used; flexible supervision is achieved by applying samples with different modification magnitudes to avoid overfitting and continuously improve diagnostic accuracy.

[0033] Example 2 The present invention also provides a medical imaging diagnostic device, comprising: The first processing module is used to perform preliminary diagnosis of medical images and generate medical reports by utilizing a large medical context language model. The second processing module is used to quantify the credibility of each sentence in the medical report using an uncertainty measurement algorithm to obtain a credibility score; The third processing module is used to dynamically label the report content based on the confidence score using color gradient mapping to obtain high-risk diagnostic items; The fourth processing module is used to generate a heatmap of medical image decision-making basis for each sentence; The fifth processing module is used to display a heatmap of medical image decision-making criteria and high-risk diagnostic entries using a React-based front-end interaction framework.

[0034] As one embodiment of the present invention, the third processing module uses an interpretable method based on explicit attribution and attention mechanisms to generate a heatmap of medical image decision-making basis for each sentence.

[0035] As one embodiment of the present invention, it further includes: a sixth processing module, used to adopt a continuous learning framework, introduce sentence-level correction and flexible training strategies, and continuously optimize the medical context large language model and diagnostic accuracy.

[0036] Example 3 The present invention also provides a medical imaging diagnostic system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program performs a medical imaging diagnostic method when executed by the processor.

[0037] Example 4 The present invention also provides a storage medium storing a computer program, which executes a medical image diagnosis method when running.

[0038] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A medical imaging diagnostic method, characterized in that, include: Using a large language model of medical context, preliminary diagnosis of medical images is performed, and medical reports are generated. The credibility of each sentence in a medical report is quantified using an uncertainty measurement algorithm to obtain a credibility score. Based on the confidence score, the report content is dynamically labeled using color gradient mapping to obtain high-risk diagnostic items; Generate a heatmap of medical image decision-making criteria for each sentence; Using a React-based front-end interaction framework, a heatmap of medical imaging decision-making criteria and high-risk diagnostic entries are displayed.

2. The medical imaging diagnostic method as described in claim 1, characterized in that, Using an interpretable method based on explicit attribution and attention mechanisms, a heatmap of medical image decision-making criteria is generated for each sentence.

3. The medical imaging diagnostic method as described in claim 2, characterized in that, Also includes: We employ a continuous learning framework, introducing sentence-level correction and flexible training strategies to continuously optimize the medical context language model and diagnostic accuracy.

4. A medical imaging diagnostic device, characterized in that, include: The first processing module is used to perform preliminary diagnosis of medical images and generate medical reports by utilizing a large medical context language model. The second processing module is used to quantify the credibility of each sentence in the medical report using an uncertainty measurement algorithm to obtain a credibility score; The third processing module is used to dynamically label the report content based on the confidence score using color gradient mapping to obtain high-risk diagnostic items; The fourth processing module is used to generate a heatmap of medical image decision-making basis for each sentence; The fifth processing module is used to display a heatmap of medical image decision-making criteria and high-risk diagnostic entries using a React-based front-end interaction framework.

5. The medical imaging diagnostic device as described in claim 4, characterized in that, The third processing module uses an interpretable method based on explicit attribution and attention mechanisms to generate a heatmap of medical image decision-making criteria for each sentence.

6. The medical imaging diagnostic device as described in claim 5, characterized in that, Also includes: The sixth processing module is used to adopt a continuous learning framework, introduce sentence-level correction and flexible training strategies, and continuously optimize the medical context large language model and diagnostic accuracy.

7. A medical imaging diagnostic system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the medical image diagnostic method as described in any one of claims 1-3 when executed by the processor.

8. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed, performs the medical image diagnosis method as described in any one of claims 1-3.

Images