Medical visual question and answer reasoning method based on fine-grained visual prompt

Abstract

The invention discloses a reasoning method for prompting medical visual questions and answers based on fine-grained visual prompts. The method comprises the following steps: 1, generating uniform grid points according to a predefined number of key points, and superposing the uniform grid points on an original image to obtain an augmented image; respectively processing the original image and the augmented image by using an image encoder and a prompt encoder so as to obtain image embedding and prompt embedding, further obtaining a global mask containing related organs, and segmenting the global mask into fine-grained instance-level local masks; 2, converting the instance-level local mask and the original image into multi-modal advanced semantic representation by using a hierarchical extractor; and 3, generating a final answer by using an answer generation module, integrating the image features and the text features by using the answer generation module, and using the generated comprehensive features as the input of the answer generation module through the multi-modal fusion, thereby realizing the generation of the optimal medical answer. According to the method, the accuracy and correlation of answer generation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a deep neural network for medical visual question answering, and in particular to a medical visual question answering reasoning method based on fine-grained visual cues. Background Art

[0002] The integration of breakthroughs in computer vision and natural language processing has promoted great interest in multimodal tasks, such as visual captioning, visual description, and visual question answering. The goal of the visual captioning task is to extract and present text information from videos or images to enhance the accessibility and comprehension of the content; the goal of visual description is to give an effective summary of the content of an image using one or several natural languages; the goal of visual question answering is to use a question described in natural language and a visual data object as input, and the algorithm performs deep reasoning after fully understanding the natural language description and visual content, and finally outputs an answer described in natural language. Among these tasks, visual question answering is relatively more challenging, involving fine-grained understanding of visual content and natural language, and deep knowledge reasoning. Therefore, visual question answering has become a research hotspot in academia in recent years. Among them, medical visual question answering stands out as an important extension of the medical field. It requires in-depth understanding of medical images at various granular levels and the establishment of reliable cross-modal associations, which is crucial to helping doctors diagnose, prevent misdiagnosis, and improve patient care efficiency and experience.

[0003] Medical images, as a visual carrier different from natural images, are usually used for diagnosis and clinical research. They contain details of tissues, organs or lesions and emphasize high resolution and specific imaging modes (such as CT, MRI), while natural images focus on aesthetic appreciation and recording of daily life, and are more colorful. In addition, the processing and analysis of medical images require the use of specialized software for complex image enhancement, segmentation and feature extraction, and follow strict standards and protocols to ensure the accuracy and repeatability of data, while the processing of natural images is relatively simple and relies more on personal creativity. In terms of difficulty, the high complexity of medical images and the precise analysis of subtle features, the influence of noise and artifacts, the large amount of data generated, and the combination of image analysis results with clinical background to ensure the accuracy of diagnosis all make medical image processing more challenging.

[0004] In summary, the difficulty of medical visual question answering lies in how to build an efficient medical visual question answering reasoning framework based on correctly and effectively understanding the content of medical images and fully and accurately understanding the intention of the question, so as to improve the accuracy of answer prediction. Summary of the invention

[0005] The present invention provides a reasoning method based on visual cues for medical visual question answering, which mainly includes: 1. Constructing an adaptive visual cue generator: generating uniform grid points and superimposing them on the original image to form an augmented image. Subsequently, the original image and the augmented image are processed respectively using an image encoder and a cue encoder to obtain image embedding and cue embedding. On this basis, a global mask containing relevant organs is obtained. In order to avoid the coupling of local details, the global mask is further segmented into fine-grained instance-level local masks; 2. Constructing a hierarchical extractor: through feature conversion and cross-modal alignment, the model's ability to understand the relationship between vision and text is enhanced, and a high-level semantic representation is generated; 3. An answer generation module: used to dynamically fuse the high-level semantic representation in the hierarchical extractor, and dynamically generate answers based on context information and input data.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] Step (1), constructing an adaptive visual cue generator;

[0008] First, according to the predefined number of key points, a uniform grid of points is generated and superimposed on the original image to obtain an augmented image. Then, the original image and the augmented image are processed respectively using the image encoder and the hint encoder to obtain image embedding and hint embedding. According to the image embedding and hint embedding, a global mask containing relevant organs is obtained, and the global mask is segmented into fine-grained instance-level local masks.

[0009] Step (2), constructing a hierarchical extractor;

[0010] The hierarchical extractor is used to transform instance-level local masks and original images into multimodal high-level semantic representations, which is the format used by the answer generation module, so that information from different modalities can complement and fuse each other. Through its multi-layer structure, the hierarchical extractor gradually extracts important features from the image and forms a multi-level representation from fine-grained to global.

[0011] Step (3), using the answer generation module to generate the final answer, the answer generation module integrates image features and text features, the image includes instance-level local masks and features of the original image, and the text features are obtained by converting the question through a text encoder; the image features and text features are processed by the hierarchical extractor, and have a deep contextual understanding and semantic hierarchy. The multimodal high-level semantic representation obtained from the hierarchical extractor is used to perform more sophisticated reasoning to improve the quality of question answering.

[0012] Furthermore, step (1) is specifically as follows:

[0013] 1-1. In order to generate visual features, we first generate corresponding grid points according to the predefined number of key points. The design of this grid point takes into account the structure and important areas of the image, ensuring that the key points can effectively cover the key information in the original image. Next, we superimpose the grid points corresponding to this key point on the original image I to obtain the augmented image I g The augmented image not only retains the basic features of the original image, but also introduces additional information and enhances the focus on important areas in the image. This augmentation technique can help the model better learn and understand the details of the image, especially when dealing with complex scenes. g The image encoder and the hint encoder process the original image I and the augmented image I respectively. g Perform deep feature extraction, extract rich visual information through multi-layer convolution and nonlinear transformation, and finally obtain image embedding β i and prompt embedding β p , as shown in formula (1) and formula (2).

[0014] β i =IE(I) (Formula 1)

[0015] β p =PE(I g ) (Formula 2)

[0016] Where IE and PE represent the image encoder and hint encoder respectively. i and β p represent image embedding and hint embedding respectively.

[0017] 1-2. Embedding the image into β i and prompt embedding β p After inputting the mask decoder, a global mask σ containing all relevant organs is obtained, as shown in formula (3). This global mask б accurately identifies the regions of all relevant organs. This global mask not only contains the location information of the organs, but also reflects their relative size and shape in the image. This result provides a reliable basis for subsequent medical image analysis and decision-making, and can help doctors or researchers better understand and evaluate important structures in the image.

[0018] σ=Γ(β i ,β p ,γ) (Formula 3)

[0019] Among them, Γ represents the mask decoder, which is responsible for generating the global mask. γ represents the non-maximum suppression (NMS) threshold, which is a key hyperparameter that affects the final result of the global mask. The setting of the NMS threshold determines which overlapping detection results will be retained, thus affecting the accuracy and effectiveness of the mask. An appropriate NMS threshold can effectively remove redundant detections and ensure that only the most relevant organ regions are retained in the final mask, thereby improving the reasoning ability of the model. However, directly using the global mask as a visual cue may lead to coupling between local details, thereby affecting the model's ability to recognize and analyze fine-grained features. This coupling phenomenon may make it impossible for the model to accurately distinguish the subtle differences between organs when processing complex medical images, thereby affecting the final diagnosis results.

[0020] 1-3. To further enhance the attention to fine-grained local details, we construct an adaptive visual cue generator to split the global mask σ into fine-grained instance-level local masks η i ∈σ={η 1 ,…,η n}, where n is the total number of instances. The adaptive visual cue generator identifies the boundaries and features of each organ or instance by analyzing each region in the global mask σ. The adaptive visual cue generator not only pays attention to the overall information of the global mask, but also dynamically adjusts the degree of attention to each instance according to the context.

[0021] Furthermore, the specific structure of the adaptive visual cue generator is as follows: first, the global mask is input into the adaptive visual cue generator, wherein the network generates a local mask of corresponding form according to the global mask and visual cue form parameters, and the forms of visual cues include mask, box, circle, contour, etc.

[0022] Furthermore, the hierarchical extractor constructed in step (2) is as follows:

[0023] First, the global mask σ is input into the shared ViT network together with the original image I to extract hierarchical visual features and obtain the instance-level local representation f of the organ. l and the global representation f of the original image g , as shown in formula (4):

[0024] f l ,f g =ViT(η 1 ,…,η n ,I g ) (Formula 4)

[0025] in f g ∈R B×P×D; B, P, D represent batch size, number of patches, and dimension, respectively.

[0026] Then, the instance-level local representation f is transformed along the second dimension l and the global representation f g Connect them together to get the final visual feature f v ∈R B×((n+1)×P)×D .

[0027] In order to introduce the minimum parameters while integrating the visual features f v The extraction of is transferred from the natural image space to the medical image space, and the parameters are effectively fine-tuned by integrating LoRA on the shared ViT network; then the proposed hierarchical extractor Φ is extracted from the visual features f v The hierarchical extractor Φ can deeply explore the potential semantic relationships in visual information through multi-level feature analysis and extract features with a higher level of abstraction. v Mapped into the language feature space to obtain a multi-level visual tag v, as shown in formula (5).

[0028] v=Φ(f v ) (Formula 5)

[0029] Furthermore, step (3) generates the final answer using the answer generation module, as follows:

[0030] The answer generation module receives multi-layer processed input data, including text features T and multi-level visual tags v obtained in step (2). These embeddings contain rich semantic information and can provide a solid foundation for generating answers.

[0031] Using deep learning algorithms, comprehensive analysis of text features T and multi-level visual tags v provides a solid foundation for generating answers. In addition, answer generation modules usually contain adaptive mechanisms that can dynamically adjust generation strategies based on different inputs. In this way, the generator can better handle open-ended questions and provide more creative and accurate responses.

[0032] The answer generation module is a language model based on the Transformer architecture, which mainly consists of multiple self-attention layers and feedforward neural networks. Its internal structure adopts an efficient model design, focusing on optimizing the training process and improving the generation ability. The answer generation module processes the text features T and the multi-level visual tags v into an initial token sequence and inputs it into multiple stacked self-attention layers. Each layer captures contextual information through the self-attention mechanism and is further processed in combination with the feedforward neural network. Each self-attention layer also uses residual connections and layer normalization to improve the stability and learning speed of the model. These layers calculate the relationship between each token and other tokens through the self-attention mechanism to capture contextual information. The model first generates a probability distribution based on the existing input tokens to represent the next possible token. The Softmax function is used to convert the output into a probability, and the next token is selected through strategies such as sampling or greedy decoding. Then, the newly generated token is appended to the input sequence, and the process is repeated until the termination condition is met (such as reaching the maximum length or generating a specific end tag).

[0033] Beneficial effects of the present invention:

[0034] This study proposes a new adaptive visual cue generator for answer retrieval and generation problems in the medical field. The adaptive visual cue generator adaptively generates region-level visual cues based on the image features of various organs, providing fine-grained references for the answer generation module, thereby significantly improving the model's ability to accurately locate the original image across modalities. Then, by combining the hierarchical extractor, the answer generation module and the parameter efficient fine-tuning (PEFT) technology, the model's understanding of spatial and contextual information is significantly enhanced with minimal parameter increments, further promoting the consistency between representation learning and medical space. This framework can effectively simulate the medical image analysis process and improve the accuracy and relevance of answer generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention.

[0036] Figure 2 It is an adaptive visual cue generator constructed in the method of the present invention.

[0037] Figure 3 It is a hierarchical extractor and answer generation module constructed in the method of the present invention. DETAILED DESCRIPTION

[0038] The detailed parameters of the present invention are further described in detail below.

[0039] like Figure 1As shown, the present invention provides a prompt-based question answering reasoning method for medical visual question answering.

[0040] Step (1), constructing an adaptive visual cue generator;

[0041] First, according to the predefined number of key points, a uniform grid of points is generated and superimposed on the original image to obtain an augmented image. Then, the original image and the augmented image are processed respectively using the image encoder and the hint encoder to obtain image embedding and hint embedding. According to the image embedding and hint embedding, a global mask containing relevant organs is obtained, and the global mask is segmented into fine-grained instance-level local masks.

[0042] The present invention uses memory components to store the global semantic information of the video. Under the guidance of the global visual features of the problem description, a shallow inference engine is obtained through multiple rounds of memory update iterations to infer the visual information most relevant to the global semantic features of the problem description, such as Figure 2 As shown, the details are as follows:

[0043] 1-1. In order to generate visual features, we first generate corresponding grid points according to the predefined number of key points. The design of this grid point takes into account the structure and important areas of the image, ensuring that the key points can effectively cover the key information in the original image.

[0044] 1-2. Next, we superimpose the grid points corresponding to this key point on the original image I to obtain the augmented image I g The augmented image not only retains the basic features of the original image, but also introduces additional information and enhances the focus on important areas in the image. This augmentation technique can help the model better learn and understand the details of the image, especially when dealing with complex scenes. g The image encoder and the hint encoder process the original image I and the augmented image I respectively. g Deep feature extraction is performed, and rich visual information is extracted through multi-layer convolution and nonlinear transformation, and finally image embedding and prompt embedding are obtained respectively.

[0045] The image encoder is responsible for extracting high-level features of the original image, while the cue encoder focuses on features related to the generated visual cues. This separation ensures that the information obtained from the two perspectives does not interfere with each other, and the resulting image embedding and cue embedding can fully reflect the details and semantics of the image.

[0046] 1-3. After processing the image embedding and the hint embedding, the two embedding data are input into the mask decoder. The main function of the mask decoder is to combine the two embedding information to generate a global mask containing all relevant organs. First, it will standardize the image embedding and the hint embedding to ensure that the data from different sources can be effectively combined. Then, the decoder processes the input layer by layer through a multi-layer neural network structure to extract key features and relationships, thereby generating a preliminary global mask.

[0047] In this process, the non-maximum suppression (NMS) mechanism is introduced to optimize the final generated global mask. NMS is used to remove redundant areas and ensure that only the most representative areas are retained by setting a specific threshold. This key step helps to improve the accuracy of the global mask and ensure that the model can accurately focus on important medical structures.

[0048] 1-4. In order to further enhance the model's attention to fine-grained local details, we use adaptive visual cueing technology to split the generated global mask into multiple instance-level local masks η i , where n represents the total number of instances. In the specific implementation, we first analyze the global mask to identify the different organs and structures in it. By performing detailed region division on the global mask, we can decompose it into multiple local regions, each of which corresponds to a specific organ or structure instance.

[0049] Next, each local region is optimized using adaptive visual cueing technology. This technology dynamically adjusts the shape and size of the local mask according to the characteristics and contextual information of each instance to more accurately capture fine-grained local details. In this way, the model is able to adaptively generate a local mask η associated with each instance based on the specific medical task and image content. i .

[0050] Step (2), constructing a hierarchical extractor;

[0051] The hierarchical extractor is used to transform instance-level local masks and original images into multimodal high-level semantic representations, which is the format used by the answer generation module, so that information from different modalities can complement and fuse each other. The hierarchical extractor gradually extracts important features from the image through its multi-layer structure, forming a multi-level representation from fine-grained to global, as follows:

[0052] First, the global mask is input into the shared ViT network together with the original image I to extract hierarchical visual features and obtain the instance-level local representation f of the organ. l and the global representation f of the original image gSpecifically, ViT encodes the input original image and visual cues to generate multi-level feature representations.

[0053] During the feature extraction process, ViT's underlying network focuses on capturing basic visual elements such as edges, textures, and color distributions, while the middle and high layers are responsible for identifying more complex shapes and semantic features. This hierarchical extraction approach enables the model to obtain instance-level local representations of organs, covering detailed information about each organ. At the same time, ViT also generates a global representation of the original image, providing a comprehensive contextual understanding.

[0054] Finally, after this hierarchical extraction process, the generated visual features are output in a format suitable for processing by the answer generation module. This implementation step significantly improves the model's performance in complex tasks, enabling it to generate high-quality answers more flexibly and fully exploit the complementarity of image and text information.

[0055] Step (3), using the answer generation module to generate the final answer, the answer generation module integrates image features and text features, the image includes instance-level local masks and features of the original image, and the text features are converted from the question through a text encoder; the image features and text features are processed by the hierarchical extractor, and have a deep contextual understanding and semantic hierarchy. The multimodal high-level semantic representation obtained from the hierarchical extractor is used to perform more sophisticated reasoning to improve the quality of question answering, as follows:

[0056] First, the answer generation module receives the output from the hierarchical extractor, which is the hierarchical visual tags of instance-level local representation and global representation. These visual tags have been integrated with the details and context features in the image through a complex feature extraction process, ensuring that the model can capture the key information related to the question.

[0057] At the same time, text embeddings are also input into the answer generation module. These text embeddings are preprocessed and contain rich semantic information, which can effectively reflect the core content and intent of the question. Natural language processing technology is used in the generation of text embeddings to ensure that each embedding can accurately represent its corresponding text information.

[0058] Next, the answer generation module will fuse these two inputs. Through the designed multimodal fusion mechanism, the answer generation module can effectively combine the information of visual tags and text embeddings. This process may include weighted summation, attention mechanism and other techniques to ensure that the information of different modalities can complement each other and enhance the accuracy and relevance of the generated answers.

[0059] The answer generation module will further process the fused information using a deep learning model to extract potential answer features. At this stage, the model will iteratively process the input data through a multi-layer neural network structure to gradually derive the most relevant answer.

[0060] Finally, after this series of processing, the answer generation module will output a high-quality answer.

[0061] Table 1

[0062]

[0063]

[0064] As shown in Table 1 above, the comparative experimental results of FAVP on the test set of the VQA-RAD dataset and the test set of the SLAKE dataset. Generative Methods represents generative methods, and Discriminative Methods represents discriminative methods. Ref and Open represent different evaluation methods for open-set problems. The accuracy of the discriminative method is reported under the Ref column, and the recall of the generative method is reported under the Open column; Closed represents closed-set problems, and the accuracy of this type of problem is reported under the Closed column. All results except the Paramete column are expressed in percentages (%), and the best results are highlighted in bold. Parameter represents the parameter quantity of the method.

Claims

1. A medical visual question answering reasoning method based on fine-grained visual cues, characterized in that: The steps include: Step (1), constructing an adaptive visual cue generator; First, a uniform grid of points is generated according to the predefined number of key points and superimposed on the original image to obtain an augmented image. Then, the original image and the augmented image are processed by the image encoder and the hint encoder respectively to obtain image embedding and hint embedding. A global mask containing relevant organs is obtained based on the image embedding and hint embedding, and the global mask is segmented into fine-grained instance-level local masks. Step (2), constructing a hierarchical extractor; A hierarchical extractor is used to transform instance-level local masks and raw images into multimodal high-level semantic representations, which is the format processed by the answer generation module. Step (3), using the answer generation module to generate the final answer, the answer generation module integrates image features and text features, the image includes instance-level local masks and features of the original image, and the text features are obtained by converting the question through a text encoder.

2. A medical visual question answering reasoning method based on fine-grained visual cues according to claim 1, characterized in that: Step (1) is as follows: 1-1. Generate corresponding grid points according to the predefined number of key points; superimpose the grid points corresponding to the key points on the original image I to obtain the augmented image I g ; The original image I and the augmented image I g They are sent to the image encoder and the prompt encoder for processing respectively; the image encoder and the prompt encoder process the original image I and the augmented image I respectively g Perform deep feature extraction, extract rich visual information through multi-layer convolution and nonlinear transformation, and finally obtain image embedding β i and prompt embedding β p , as shown in formula (1) and formula (2); β i = IE(I) (Formula 1) β p = PE(I g ) (Formula 2) Among them, IE and PE represent the image encoder and prompt encoder respectively; β i and βp represent image embedding and prompt embedding respectively; 1-2. Embedding the image into β i and prompt embedding β p After inputting the mask decoder, a global mask σ containing all relevant organs is obtained, as shown in formula (3); the global mask σ accurately identifies the regions of all relevant organs; σ = Γ(β i , β p , γ) (Equation 3) Among them, Γ represents the mask decoder, which is responsible for generating the global mask; γ represents the non-maximum suppression threshold; 1-3. Construct an adaptive visual cue generator to split the global mask σ into fine-grained instance-level local masks η i ∈σ={η1,…,η n }, where n is the total number of instances; the adaptive visual cue generator identifies the boundaries and features of each organ or instance by analyzing each region in the global mask σ.

3. The medical visual question answering reasoning method based on fine-grained visual cues according to claim 1, characterized in that: Step (2) is as follows: First, the global mask σ is input into the shared ViT network together with the original image I to extract hierarchical visual features and obtain the instance-level local representation f of the organ. l and the global representation f of the original image g , as shown in formula (4): f l ,f g =ViT(η1,…,η n ,I g ) (formula 4) in f g ∈R B×P×D ; B, P, D represent batch size, number of patches, and dimension, respectively; Then, the instance-level local representation f is transformed along the second dimension l and the global representation f g Connect them together to get the final visual feature f v ∈R B×((n+1)×P)×D ; In order to introduce the minimum parameters while integrating the visual features f v The extraction of is transferred from the natural image space to the medical image space, and the parameters are effectively fine-tuned by integrating LoRA on the shared ViT network; then the proposed hierarchical extractor Φ is extracted from the visual features f v Learn higher semantic level visual tags from v Mapped into the language feature space to obtain a multi-level visual tag v, as shown in formula (5); v=Φ(f v ) (Formula 5).

4. The medical visual question answering reasoning method based on fine-grained visual cues according to claim 1, characterized in that: Step (3) is as follows: The answer generation module receives input data that has undergone multiple layers of processing, including text features T and the multi-level visual tags v obtained in step (2). It uses a deep learning algorithm to comprehensively analyze the text features T and the multi-level visual tags v to provide a solid foundation for generating answers. The answer generation module usually includes an adaptive mechanism to dynamically adjust the generation strategy according to different inputs.

Images