Medical image difference perception multi-prompt question and answer method and device

Abstract

The application discloses a medical image difference perception multi-prompt question and answer method and device, and the method comprises the steps of: extracting the feature representation of reference and target medical images by using a pre-trained visual encoder; comparing the features through a change analysis module, predicting whether there is a clinically significant difference and outputting a binary prompt; using a difference perception module to perform spatial alignment and time difference on the features, and generating a difference embedding representing fine-grained changes; and integrating the clinical question text, the binary prompt, the difference embedding and the learnable prompt vector, and inputting them into a pre-trained language decoder to generate an answer text. Through difference decoupling and multi-prompt learning, the application solves the problem that the existing medical VQA technology cannot effectively handle longitudinal comparison and is not sensitive to changes, can generate a concise answer when there is no change, can generate a detailed and standardized clinical description when there is a difference, and significantly improves the accuracy, reliability and practicability of intelligent question and answer in medical image follow-up and efficacy evaluation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence medical technology, and in particular to a method and apparatus for multi-prompt question answering based on medical image difference perception. Background Technology

[0002] Visual Question Answering (VQA) technology for medical images is an important application of artificial intelligence in the medical field. It aims to automatically understand the content of medical images and answer related clinical questions through models, thereby assisting doctors in diagnosis and assessment. Current technologies mostly focus on describing or answering questions about the content of single images, such as identifying lesions or describing anatomical structures.

[0003] However, in real-world clinical follow-up and efficacy evaluation scenarios, doctors need to compare a series of images of the same patient at different time points (i.e., longitudinal comparison) to accurately determine the evolution, remission, or abnormal changes of lesions. Existing medical image visual question answering technologies have significant shortcomings in addressing this core need: First, most methods lack explicit modeling mechanisms for differences between image pairs, typically only performing simple stitching or subtraction of features from two images, making it difficult to capture subtle yet crucial clinical changes. This leads to indiscriminate responses to scenarios of "no significant change" and "difference exists," resulting in low accuracy and practicality of the generated answers. Second, existing methods cannot adaptively adjust their answer strategies based on the presence or absence of change, generating redundant descriptions when there is no change, and potentially omitting key details when there is change, failing to meet clinical reporting standards. Finally, traditional feature fusion methods fail to effectively combine prior clinical knowledge with learnable semantic cues, resulting in limited semantic understanding and reasoning capabilities of the model for complex medical questions.

[0004] Therefore, there is an urgent need in this field for an intelligent question-answering method that can accurately perceive fine-grained differences between medical images and generate answers that conform to clinical norms based on the existence of these differences. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0006] Therefore, to solve the above-mentioned technical problems, the present invention provides the following technical solution: a multi-prompt question-and-answer method for medical image difference perception, comprising: Visual feature representations of the reference medical image and the target medical image are extracted based on a pre-trained visual encoder. The visual feature representation is compared based on the change analysis module to predict whether there are clinically significant differences and generate binary prompt information, thereby distinguishing whether the input image pair belongs to the "no change" scenario or the "difference exists" scenario. Based on the difference perception module, spatial alignment and difference modeling are performed on the visual feature representation to obtain the difference embedding that represents fine-grained changes; The clinical question text, binary prompt information, differential embedding, and learnable prompt vector are jointly input into a pre-trained language decoder to generate the answer text for medical image longitudinal comparison questions; Output the answer text as the medical image difference question and answer result; As a preferred embodiment of the medical image difference perception multi-prompt question answering method described in this invention, the change analysis module generates binary prompts through binary classification prediction to distinguish between two scenarios: "no change" and "difference exists"; the change analysis module is implemented through a Transformer structure, cross-attention mechanism, or contrastive learning method.

[0007] As a preferred embodiment of the medical image difference perception multi-prompt question answering method of the present invention, the difference perception module includes a spatial alignment sub-module and a temporal difference sub-module, which are used for feature spatial alignment (i.e., spatial alignment of features between reference image and target image) and temporal difference calculation (i.e., calculation of temporal series differences between images), respectively.

[0008] As a preferred embodiment of the medical image difference perception multi-prompt question answering method described in this invention, the difference perception module is implemented through a Transformer structure, the spatial alignment sub-module includes a spatial attention layer, and the temporal difference sub-module includes a temporal attention layer, which are used to enhance the cross-dimensional modeling capability of image differences.

[0009] As a preferred embodiment of the medical image difference perception multi-prompt question answering method described in this invention, the learnable prompt vector is a multi-dimensional embedding vector that is adaptively adjusted during training to enhance semantic understanding; the number and dimension of the learnable prompt vector are configurable.

[0010] As a preferred embodiment of the medical image difference perception multi-prompt question answering method of the present invention, the pre-trained language decoder is a general language model with frozen parameters, and its input is composed of the clinical question text, binary prompt information, difference embedding and learnable prompt vector.

[0011] As a preferred embodiment of the medical image difference perception multi-prompt question answering method of the present invention, the pre-trained visual encoder is a pre-trained model based on visual Transformer, and the output includes local features and global features; the parameters of the pre-trained visual encoder are frozen or fine-tuned on small sample medical data.

[0012] As a preferred embodiment of the medical image difference perception multi-prompt question-answering method of the present invention, the binary prompt information is injected as a conditional signal into the language decoder to control the level of detail in the generated answer.

[0013] The above-described medical image difference perception multi-prompt question answering method first acquires the features of the input image pair through a visual encoder and generates difference prompts using a change analysis module. Then, a difference perception module extracts fine-grained difference features and inputs them into a language decoder in combination with question prompts, change prompts, and learnable prompts. Finally, the decoding generation module outputs the medical image difference question answering results. This method can generate concise and consistent descriptions when there are no significant changes, and generate structured and semantically accurate clinical descriptions when there are changes such as lesion progression or remission, thereby improving the accuracy, interpretability, and clinical applicability of intelligent question answering in medical image scenarios with longitudinal comparison.

[0014] A medical image difference perception multi-prompt question-and-answer device, applied to any of the above-described medical image difference perception multi-prompt question-and-answer methods, includes: The feature extraction module is used to extract visual feature representations of the reference image and the target image; The change analysis module is used to generate binary prompt information; The difference-aware module is used to generate difference embeddings; The prompt integration module is used to integrate clinical question text, binary prompt information, differential embeddings, and learnable prompt vectors; The decoding and generation module is used to generate and output the answer text.

[0015] As a preferred embodiment of the medical image difference perception multi-prompt question-and-answer device of the present invention, the prompt integration module can adaptively adjust the weight of different prompts according to the output of the change analysis module to enhance the accuracy of the final generated answer.

[0016] The beneficial effects of this invention are: 1. This invention achieves precise decoupling and adaptive response in longitudinal comparison scenarios. Specifically, this invention performs binary classification on input image pairs through a change analysis module to generate binary prompt information. This decouples the complex question-and-answer task into two distinct processing paths: "no change" and "difference exists," fundamentally solving the problem of the single answer generation mode in existing technologies. This binary prompt, as a conditional signal, directly controls the generation behavior of the language decoder, ensuring that a concise and standard negative description is output when there is "no change," while driving the model to generate a detailed and structured positive finding description when there is "difference exists," greatly improving the clinical applicability and accuracy of the output answer.

[0017] 2. This invention enhances the perception and representation of fine-grained, subtle clinical changes. Specifically, through a difference perception module, it integrates spatial alignment and temporal difference operations, effectively overcoming interference caused by inconsistent imaging conditions (such as angle and displacement) and amplifying the feature differences generated by real biological changes. The difference embedding output by this module can accurately represent subtle evolutions such as lesion size, density, and location, providing rich and reliable visual evidence for the decoder. This enables the invention to detect early or subtle changes that are difficult for the human eye to perceive and generate precise quantitative descriptions (such as volume change rate and displacement coordinates), solving the problem of "difficulty in capturing subtle key changes" in existing technologies and significantly improving the model's perception sensitivity and the amount of information it provides.

[0018] 3. This invention integrates multi-source information to enhance the model's semantic understanding and reasoning capabilities. Specifically, the multi-cue integration mechanism proposed in this invention fuses clinical question text, change cues, difference embeddings, and learnable cue vectors to construct a multi-layered, complementary information guidance system. Among them, the learnable cue vectors adaptively optimize during training, acting as a bridge to embed medical prior knowledge into the model, enhancing the model's deep understanding of professional terminology and clinical intent. Ultimately, under the guidance of this series of strongly relevant, high-information-density cues, the language decoder can perform more reliable medical reasoning, generating answers that are not only accurate but also conform to clinical expression norms, comprehensively solving the pain point of "limited semantic understanding and reasoning capabilities" in the background technology. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the process of the method of the present invention.

[0020] Figure 2 This is an overall architectural diagram of the device of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] Example 1 Reference Figure 1 This is the first embodiment of the present invention, which provides a medical image difference perception multi-prompt question answering method. It aims to solve the problem that in longitudinal medical image comparison, human judgment is easily interfered with due to the subtle and complex changes in lesions, and traditional methods are difficult to effectively capture key differences. The method achieves scene differentiation through change analysis, enhances subtle changes by using difference perception, and provides multi-dimensional guidance for decoding by combining multi-prompt learning, and finally generates accurate answers that are both consistent with medical semantics and have high clinical usability.

[0025] like Figure 1 As shown, this method specifically includes the following steps: S101: Input medical images and extract feature representations; The input consists of a reference medical image (e.g., a patient's baseline CT image) and a target medical image (e.g., a follow-up CT image), typically images of the same patient acquired at different time points (e.g., chest X-rays, CT scans, or MRI scans). A pre-trained visual encoder (e.g., a visual Transformer model) extracts features from both images. The parameters of the visual encoder can be frozen or fine-tuned on small samples of medical data. Its function is to transform the input image (e.g., uniformly adjusted to 512×512 pixels) into a stable high-dimensional semantic representation. The output includes local features (e.g., a feature map of size 768×7×7) and global features (e.g., a 768-dimensional vector), providing support for subsequent differential modeling. It should be noted that the acquisition methods of reference images and target images are not limited to specific image acquisition devices or imaging modes, and this application does not impose any limitations on this.

[0026] S102: Analyze changes and generate binary prompts; The visual feature representation obtained in step S101 is input into the change analysis module 202; this module predicts whether there are clinically relevant changes in the input image pair by interactively modeling the features of the reference image and the target image (which can be achieved by using a Transformer structure, cross-attention mechanism or other contrastive learning methods) in a binary classification manner. The prediction result is converted into a binary prompt: if the prediction result indicates that the difference is lower than the clinical threshold (such as Dice coefficient > 0.9 or nodule volume change rate < 5%), the output "0" indicates "no change", otherwise the output "1" indicates "there is a difference"; this binary prompt will be injected as a conditional signal into the subsequent decoding stage to distinguish between the scenarios of generating a "concise answer" or a "detailed answer".

[0027] S103: Difference-aware modeling and prompting integration; The visual features are input into the difference perception module 203. This module first aligns the features of the reference image and the target image in space using a spatial alignment submodule (such as using a spatial attention layer) to reduce positional deviations caused by imaging angles or noise. Then, it calculates the feature differences through a temporal difference module (such as a temporal attention layer) to obtain temporal change features and extract the subtle evolution of lesions in longitudinal comparison. The difference perception module 203 outputs a difference embedding (e.g., with a size of 768×7×7) that can capture key clinical information such as lesion progression, remission, and positional changes. Based on this, the question prompts obtained after encoding the clinical question text (such as "Have the lung nodules increased in size?"), the binary prompts generated in step S102, the differential embeddings, and the learnable prompt vectors optimized during training (e.g., concatenating five 768-dimensional vectors) are integrated to form a unified prompt vector, which serves as the input to the decoder. The prompt integration module can adaptively adjust the weights of different prompts according to the output of the change analysis module to enhance the accuracy of the final generated answer. It should be noted that the number and dimension of the learnable cue vectors are configurable and can be flexibly adjusted according to different clinical tasks and language models. In one specific implementation, the number of learnable cue vectors can be set to a fixed number of vectors, with the dimension of each vector consistent with the input dimension of the language decoder, so as to facilitate concatenation with question prompts and difference embeddings. Those skilled in the art will understand that the above-described fixed-number implementation is merely an example and not a limitation of the present invention.

[0028] S104: Answer generation and result output; The integrated cue vector is input into a pre-trained language decoder (such as a BERT model with frozen parameters, or a model that supports conditional generation, such as GPT); the decoder generates the answer based on the binary cue conditionalization. If the prompt is "0", generate a concise standard answer, such as "No significant change"; If the prompt is "1", a detailed description will be generated, such as "The nodule in the right lung has increased significantly, accompanied by an expansion of the exudative lesion area" or "The left ventricular end-diastolic diameter increased from 45 mm to 48 mm (change rate 6.7%), which is consistent with a mild enlargement trend". The generated answer text, after post-processing (such as removing redundant words), is output to the clinical system interface as the final medical image longitudinal comparison question and answer result to assist doctors in decision-making.

[0029] Example 2 Reference Figure 2 This is the second embodiment of the present invention, which differs from the first embodiment in that: this embodiment provides a medical image difference perception multi-prompt question-and-answer device, applied to a medical image difference perception multi-prompt question-and-answer method in the above embodiment 1. The device includes the following modules (structure as follows): Figure 2 (as shown) Feature extraction module 201: Based on a pre-trained visual Transformer model, it is used to extract features from the input medical image and output a visual feature representation. Change analysis module 202: used to compare and analyze the features of the reference and target images and generate binary prompt information; Difference perception module 203: includes a spatial alignment submodule (using a spatial attention layer) and a temporal difference molecule module (using a temporal attention layer), used to generate difference embeddings representing fine-grained changes; Hint integration module 204: It is used to integrate and fuse clinical question text, binary hint information, differential embedding and learnable hint vector, and adaptively adjust the weight of different hints according to the output of the change analysis module to enhance the accuracy of the final generated answer; Decoding and generation module 205: Based on a pre-trained language model, it is used to generate answer text that conforms to medical semantics.

[0030] The workflow of a medical image difference perception multi-prompt question-and-answer device is as follows: After the user inputs a clinical question and image pair, the feature extraction module 201 performs the operation described in step S101 of Example 1; the change analysis module 202 performs the operation described in step S102; the difference perception module 203 performs the operation described in step S103; the prompt integration module 204 performs the integration operation described in step S103; and the decoding generation module 205 performs the operation described in step S104, finally outputting a structured medical description.

[0031] Example scenario: A user asked, "Has there been any change in heart size?" The system output a structured medical description: "The left ventricular end-diastolic diameter increased from 45mm to 48mm (a change rate of 6.7%), which is consistent with a mild enlargement trend."

[0032] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-prompt question-and-answer method for perceiving differences in medical images, characterized in that: include: Visual feature representations of the reference medical image and the target medical image are extracted based on a pre-trained visual encoder. The visual feature representations are compared based on the change analysis module to predict whether there are significant clinical differences and generate binary prompt information; Based on the difference perception module, spatial alignment and difference modeling are performed on the visual feature representation to obtain the difference embedding that represents fine-grained changes; The clinical question text, binary prompt information, differential embedding, and learnable prompt vector are jointly input into a pre-trained language decoder to generate the answer text for medical image longitudinal comparison questions; The answer text is output as the medical imaging difference question and answer result.

2. The medical image difference perception multi-prompt question-and-answer method as described in claim 1, characterized in that: The change analysis module generates binary prompts through binary classification prediction to distinguish between two scenarios: "no change" and "difference exists". The change analysis module is implemented through a Transformer structure, cross-attention mechanism or contrastive learning method.

3. The medical image difference perception multi-prompt question-and-answer method as described in claim 2, characterized in that: The difference perception module includes a spatial alignment submodule and a temporal difference submodule, which are used for spatial alignment and temporal difference calculation of features, respectively.

4. The medical image difference perception multi-prompt question-and-answer method as described in claim 3, characterized in that: The difference perception module is implemented through a Transformer structure. The spatial alignment submodule includes a spatial attention layer, and the temporal difference submodule includes a temporal attention layer, which are used to enhance the cross-dimensional modeling capability of image differences.

5. The medical image difference perception multi-prompt question-and-answer method as described in claim 4, characterized in that: The learnable cue vector is a multi-dimensional embedding vector that is adaptively adjusted during training to enhance semantic understanding.

6. The medical image difference perception multi-prompt question-and-answer method as described in claim 5, characterized in that: The pre-trained language decoder is a general language model with frozen parameters, and its input consists of the clinical question text, binary cue information, differential embeddings, and learnable cue vectors concatenated together.

7. The medical image difference perception multi-prompt question-and-answer method as described in claim 6, characterized in that: The pre-trained visual encoder is a pre-trained model based on the visual Transformer, and its output includes local features and global features. The parameters of the pre-trained visual encoder are either frozen or fine-tuned on a small sample of medical data.

8. The medical image difference perception multi-prompt question-and-answer method as described in claim 7, characterized in that: The binary prompt information is injected into the language decoder as a conditional signal to control the level of detail in the generated answer.

9. A medical image difference perception multi-prompt question-and-answer device, applied to the medical image difference perception multi-prompt question-and-answer method according to any one of claims 1 to 8, characterized in that, include: The feature extraction module is used to extract visual feature representations of the reference image and the target image; The change analysis module is used to generate binary prompt information; The difference-aware module is used to generate difference embeddings; The prompt integration module is used to integrate clinical question text, binary prompt information, differential embeddings, and learnable prompt vectors; The decoding and generation module is used to generate and output the answer text.

10. The medical image difference perception multi-prompt question-and-answer device as described in claim 9, characterized in that: The prompt integration module adaptively adjusts the weights of different prompts based on the output of the change analysis module to enhance the accuracy of the final generated answer.

Images