Medical visual question and answer method, device and equipment for pixel-level decoding and storage medium

Abstract

The invention discloses a medical visual question and answer method, device and equipment for pixel-level decoding and a storage medium, and relates to the technical field of medical data processing, and the method comprises the steps: carrying out the preprocessing of an input medical image and a query text, and generating multi-modal medical data; performing cross-modal alignment processing on the multi-modal medical data to generate cross-modal fusion features; and based on the cross-modal fusion feature, generating a natural language reply containing the medical entity and a pixel-level segmentation mask corresponding to the medical entity, and forming a multi-modal output result. According to the method, deep fusion of the medical image features and the text features is realized through cross-modal alignment processing, and the accuracy and specialty of question and answer results are improved. The generated multi-modal output result comprises the natural language reply and the pixel-level segmentation mask corresponding to the medical entity, the position of the medical entity can be accurately marked, the readability is improved, and therefore the problems that existing medical visual questions and answers are low in output result precision and low in clinical interpretation efficiency are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, in particular to a pixel-level decoding medical visual question answering method and device, equipment and storage medium. BACKGROUND

[0002] Medical visual question answering (MVQA) is an interdisciplinary task that combines natural language processing and computer vision technology, and requires in-depth understanding of medical images and questions to provide reliable answers.

[0003] However, current medical visual question answering technology tends to rely on text instructions to ask questions in the given entire image case. Although this method can achieve rich problem setting on the text side, unlike the general field, most medical terms are a complex expression of predefinition, and pure text description of the question may cause ambiguity of the question, so the output medical reply has the problem of insufficient accuracy. At the same time, the existing medical reply is relatively complex, and the clinical interpretation efficiency is low.

[0004] Therefore, how to improve the accuracy and readability of medical visual question answering has become a problem to be solved. SUMMARY

[0005] The main purpose of the present application is to provide a pixel-level decoding medical visual question answering method, device, equipment and storage medium, which aims to solve the technical problem of how to improve the accuracy and readability of medical visual question answering.

[0006] To achieve the above-mentioned purpose, the present application provides a pixel-level decoding medical visual question answering method, which comprises: preprocessing the input medical image and query text to generate multi-modal medical data; performing cross-modal alignment processing on the multi-modal medical data to generate cross-modal fusion features; Based on the cross-modal fusion features, generate natural language replies containing medical entities and pixel-level segmentation masks corresponding to the medical entities to form multi-modal output results.

[0007] In an embodiment, the step of generating natural language replies containing medical entities and pixel-level segmentation masks corresponding to the medical entities based on the cross-modal fusion features to form multi-modal output results comprises: Based on the cross-modal fusion features, synchronously generate natural language replies containing medical entities and attention distribution corresponding to the medical entities; Map the attention distribution to the medical image to obtain the pixel-level segmentation mask corresponding to the medical entity; Edge optimization processing is performed on the pixel-level segmentation mask to obtain the final segmentation mask. The natural language response is then associated with the final segmentation mask and output to form a multimodal output result.

[0008] In one embodiment, the step of simultaneously generating a natural language response containing medical entities and a corresponding attention distribution for the medical entities based on the cross-modal fusion features includes: The cross-modal fusion features are input into a lightweight Transformer decoder to obtain text generation features; The text generation features are subjected to linear transformation and Softma pixel-level decoding for medical visual question answering to obtain the probability distribution of the next word; Target words are selected based on the probability distribution, and the target words are concatenated word by word to form a natural language response containing medical entities. The attention distribution corresponding to the medical entities when the target words are generated is recorded.

[0009] In one embodiment, the step of mapping the attention distribution to the medical image to obtain a pixel-level segmentation mask corresponding to the medical entity includes: The attention distribution is upsampled to the image space of the medical image to obtain an attention map corresponding to the medical entity; Generate multi-scale initial segmentation masks based on the attention map; By fusing the initial segmentation masks of the multiple scales, a pixel-level segmentation mask is obtained.

[0010] In one embodiment, the step of performing edge optimization processing on the pixel-level segmentation mask to obtain the final segmentation mask includes: Construct a conditional random field model that includes univariate and bivariate potential functions; The pixel-level segmentation mask and the probability distribution are input into the conditional random field model, and the global energy function of the conditional random field model is optimized by the iterative conditional pattern algorithm to obtain the final segmentation mask.

[0011] In one embodiment, the multimodal medical data includes standardized medical images and word sequence text; The step of performing cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features includes: The standardized medical images are input into an improved visual state space network to obtain medical image features; The word sequence text is input into a lightweight Transformer encoder to obtain text features; Cross-modal fusion features are generated by fusing the medical image features and the text features through a bidirectional attention mechanism.

[0012] In one embodiment, the step of performing cross-modal fusion of the medical image features and the text features using a bidirectional attention mechanism to generate cross-modal fused features includes: A first query vector is constructed based on the text features, a first key vector and a first value vector are constructed based on the medical image features, the attention weight from image to text is determined by the first query vector and the first key vector, and the text enhancement features that fuse image information are obtained by combining the first value vector and the attention weight from image to text. A second query vector is constructed based on the medical image features, a second key vector and a second value vector are constructed based on the text features, the text-to-image attention weight is determined through the second query vector and the second key vector, and the image enhancement features fused with text information are obtained by combining the second value vector and the text-to-image attention weight. The text enhancement features of the fused image information are concatenated with the image enhancement features of the fused text information to generate cross-modal fusion features.

[0013] Furthermore, to achieve the above objectives, this application also proposes a pixel-level decoding medical visual question-answering device, the pixel-level decoding medical visual question-answering device comprising: The image preprocessing module is used to preprocess the input medical images and query text to generate multimodal medical data; A cross-modal fusion module is used to perform cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features; The question-answering output module is used to generate a natural language response containing a medical entity and a pixel-level segmentation mask corresponding to the medical entity based on the cross-modal fusion features, thereby forming a multimodal output result.

[0014] Furthermore, to achieve the above objectives, this application also proposes a pixel-level decoding medical visual question-answering device, the device comprising: a memory, a processor, and a pixel-level decoding medical visual question-answering program stored in the memory and executable on the processor, the pixel-level decoding medical visual question-answering program being configured to implement the steps of the pixel-level decoding medical visual question-answering method mentioned above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium storing a pixel-level decoded medical visual question-and-answer program, which, when executed by a processor, implements the steps of the pixel-level decoded medical visual question-and-answer method mentioned above.

[0016] This application provides a pixel-level decoding method, apparatus, device, and storage medium for medical visual question answering. The method includes: preprocessing input medical images and query text to generate multimodal medical data; performing cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features; and generating a natural language response containing medical entities and a pixel-level segmentation mask corresponding to the medical entities based on the cross-modal fusion features, forming a multimodal output result. This application achieves deep fusion of medical image features and text features through cross-modal alignment processing, enabling the model to more comprehensively and accurately understand user needs and medical image content, thus improving the accuracy and professionalism of the question answering results. In the generated multimodal output result, the natural language response accurately includes medical entities and related knowledge explanations, and the pixel-level segmentation mask precisely marks the location of medical entities. The two are interconnected, greatly improving the readability and practicality of the medical visual question answering results, facilitating users to quickly obtain and understand key information from medical images. Therefore, this application can solve the problems of low accuracy and low clinical interpretation efficiency in existing medical visual question answering methods. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the pixel-level decoding medical visual question-answering method of this application; Figure 2 This is a schematic diagram of the multimodal output results of the first embodiment of the pixel-level decoding medical visual question answering method of this application; Figure 3 This is a schematic diagram of the first process of the second embodiment of the pixel-level decoding medical visual question answering method of this application; Figure 4 This is a second flowchart illustrating a second embodiment of the pixel-level decoding medical visual question-answering method of this application. Figure 5 This is a schematic diagram illustrating the pixel-level decoding process of the medical visual question-answering method of this application. Figure 6 This is a schematic diagram of the module structure of the pixel-level decoding medical visual question-answering device according to an embodiment of this application; Figure 7This is a schematic diagram of the device structure of the hardware operating environment involved in the pixel-level decoding medical visual question answering method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application is as follows: preprocessing the input medical image and query text to generate multimodal medical data; performing cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features; and generating natural language responses containing medical entities and pixel-level segmentation masks corresponding to the medical entities based on the cross-modal fusion features to form multimodal output results.

[0024] Currently, medical visual question-answering technology tends to rely on text commands to formulate questions within a given image context. While this approach allows for rich question settings on the text side, unlike general domains, most medical terms are often predefined and complex expressions. Plain text descriptions can lead to ambiguity, resulting in insufficient accuracy in the output medical responses. Furthermore, existing medical responses are often longer and more structured, incorporating the use and explanation of different medical concepts. Pure text responses can result in overly concise or overly complex text, making them difficult for ordinary users to fully understand and leading to low efficiency in clinical interpretation.

[0025] To address the aforementioned issues, this application proposes a visual question-answering technology solution that integrates natural language processing and computer vision techniques, specifically tailored to medical scenarios, and capable of outputting pixel-level segmentation results. This application achieves deep fusion of medical image features and text features through cross-modal alignment processing, enabling the model to more comprehensively and accurately understand user needs and medical image content, thereby improving the accuracy and professionalism of the question-answering results.

[0026] Meanwhile, the multimodal output generated by this application accurately includes medical entities and related knowledge explanations in its natural language responses, and pixel-level segmentation masks precisely mark the locations of medical entities. These two aspects are interconnected, greatly improving the readability and usability of the medical visual question-answering results, facilitating users' rapid acquisition and understanding of key information from medical images. Therefore, this application can solve the problems of low accuracy and low clinical interpretation efficiency in existing medical visual question-answering systems.

[0027] It should be noted that the execution subject of this embodiment can be a pixel-level decoding medical visual question-answering system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a pixel-level decoding medical visual question-answering device capable of the above functions. This embodiment does not specifically limit it in this way. The following uses a pixel-level decoding medical visual question-answering device (hereinafter referred to as a question-answering device) as the execution subject to describe this embodiment and the following embodiments.

[0028] Based on this, embodiments of this application provide a pixel-level decoding method for medical visual question answering, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the pixel-level decoding medical visual question-answering method of this application.

[0029] In this embodiment, the pixel-level decoding medical visual question-answering method includes steps S10~S30: Step S10: Preprocess the input medical images and query text to generate multimodal medical data; Understandably, the aforementioned medical images can encompass various types of image data used for medical diagnosis, including CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, and pathological slide images. These images must contain medical information that can be used for analysis, such as lesion regions and anatomical structures. The query text can be questions raised by users (such as medical personnel or patients) related to the medical images, such as "segmenting organs located within bounding boxes in the image" or "describing the tissue type and characteristics of the region within the bounding box indicating the nature of the lesion," representing the user's needs for medical image analysis.

[0030] It is easy to understand that, in order to facilitate subsequent model analysis, the question-answering device can perform preprocessing operations on the input medical images and query text, i.e., preprocessing. For example, in this embodiment, the preprocessing operations for medical images may include standardization (if it is a CT image, perform HU value normalization, for example, limit the HU value range to -1000 to 400HU (this range covers the main information of human soft tissue and bones), and remove outliers outside this range; if it is a pathological slide image, perform pixel value normalization, for example, divide the pixel value by 255 and normalize it to the [0,1] interval) and resolution unification (for example, use an image scaling algorithm (such as bilinear interpolation algorithm) to uniformly adjust the resolution of the normalized medical image to 256×256 pixels, which is convenient for subsequent model processing), thereby obtaining a standardized medical image with uniform resolution; For query text, a pre-trained tokenizer tool (such as BERT Tokenizer trained on medical corpus) can be used to segment it into a fixed-length (e.g., 512) sequence of tokens. If the text length is less than the fixed length, it is padded (by adding padding markers, such as [PAD]), and if it is too long, it is truncated.

[0031] For example, “Segment the bladder” can be segmented into [“Segment”, “the”, “bladder”, “[PAD]”]). Simultaneously, a pre-trained medical entity recognition model (such as a BERT-based medical NER model) can be used to scan the word sequence, identify medical entity markers (such as “2nd lumbar vertebrae”, “bladder”), and add special identifiers to these markers (such as adding prefixes or suffixes before or after the entity). <entity>and< / entity> This yields text data in the form of a word sequence, i.e., word sequence text.

[0032] Standardized medical images and word sequence text can constitute the aforementioned multimodal medical data. Since multimodal medical data summarizes data from both visual and textual modalities, it can provide a foundation for subsequent cross-modal processing.

[0033] Step S20: Perform cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features; It is easy to understand that the aforementioned cross-modal alignment processing can be a process that enables data from different modalities (images and text) to be correlated and matched at the feature level. For example, a question-answering device can first extract medical image features and text features separately, and then achieve deep interaction and fusion of the two features through a specific mechanism (such as a bidirectional attention mechanism) to ensure that the image features and text features correspond semantically. After cross-modal alignment processing, the feature data that fuses key information from the medical image (such as the location, shape, and texture features of lesions, and the global semantics and local details of anatomical structures) and key information from the query text (such as user needs and medical entity semantics) possesses both image and text semantic information, and can be used to subsequently generate natural language responses and pixel-level segmentation masks.

[0034] In one feasible implementation, the multimodal medical data includes standardized medical images and word sequence text; in this embodiment, step S20 may include steps A1~A3: Step A1: Input the standardized medical image into an improved visual state space network to obtain medical image features; It should be understood that the aforementioned standardized medical images can be image data after "grayscale normalization" and "resolution unification" processing of original medical images (such as CT, MRI, pathological slides), and the token sequence text can be text data composed of discrete text unit sequences obtained after tokenizing the query text.

[0035] Then, the question-answering device can input standardized medical images into an improved visual state space network (Frequency-Enhanced Lightweight Vision Mamba, FMamba), which sequentially performs initial feature extraction, visual state space processing, frequency feature enhancement, and cross-scale feature fusion to obtain medical image features.

[0036] For example, in the initial feature extraction stage, a 3×3 convolutional layer (stance 1, padding=1) and the ReLU activation function are first used to transform the standardized medical image from the original pixel space (256×256×C, where C is the number of image channels, such as 1 channel for CT images and 3 channels for MRI images) into a low-dimensional feature space (256×256×64, where 64 is the number of feature channels). This filters out noise from the original image, preliminarily compresses redundant information, and obtains the initial feature map.

[0037] Then, in the visual state space processing stage, the core module of FMamba, namely the Visual State Space (VSS) module, dynamically captures long-distance relationships between pixels (such as the spatial relationship between tumors and surrounding blood vessels in medical images) along the width and height dimensions of the initial feature map through the two-dimensional selective scan (SS2D) operation. The output dimension is still a feature map of 256×256×64, which enhances the global semantic and spatial relationship features and obtains the basic feature map.

[0038] In the frequency feature enhancement stage, the basic feature map output by VSS is first fed into three "downsampling-convolution" sub-layers to obtain feature maps with resolutions of 128×128×128, 64×64×256, and 32×32×512, respectively. Then, each feature map with a different resolution is individually fed into the Frequency Feature Enhancement (FFE) module. The features are transformed from the spatial domain to the frequency domain through Fourier transform. The features transformed to the frequency domain are then multiplied element-wise with a learnable frequency mask (learned through training to correspond to the frequency range of effective features in medical images, such as lesion edges corresponding to high-frequency components) to enhance key frequency information. Finally, an inverse Fourier transform is performed to transform the features back to the spatial domain to obtain multi-scale enhanced features. This step can significantly improve the feature representation of lesion edges and texture details.

[0039] Finally, in the cross-scale feature fusion stage, the feature maps after "VSS processing + FFE enhancement" at each resolution, i.e., multi-scale enhanced features, can be fused across scales: high-resolution features (such as 128×128×128) are aligned with low-resolution features (such as 64×64×256) through upsampling (such as bilinear interpolation), and then the number of channels is unified by 1×1 convolution and added element by element to finally output 4 layers of multi-scale medical image features.

[0040] Step A2: Input the word sequence text into a lightweight Transformer encoder to obtain text features; It's important to understand that the question-answering device can input a lexical sequence of text into a lightweight Transformer encoder, which then performs input embedding, multi-head attention computation, and feedforward neural network processing sequentially to obtain text features. The input embedding includes a superposition of word embeddings and positional encodings. Compared to traditional encoders, this lightweight Transformer encoder reduces the number of Transformer blocks (6 layers, compared to 12 layers in the traditional encoder) and the number of multi-head attention heads (8 heads, compared to 12 heads in the traditional encoder). Its input is a lexical sequence of text, and its output is text features, resulting in a computational complexity reduction of approximately 50%. Furthermore, it can enhance the semantic representation of medical entities through "entity tagging augmentation."

[0041] For example, after inputting the word sequence text into the lightweight Transformer encoder, in the input embedding stage, word embedding (mapping each word into a 512-dimensional vector using a pre-trained medical word embedding matrix) and positional encoding (generating a 512-dimensional position vector with the same length as the word sequence using a learnable sinusoidal positional encoding to capture the order information in the text) can be calculated first. The two are then added together to obtain the input embedding vector E, as shown below: E=Embedding(Q)+PositionalEncoding(Q); (1) In the formula, Q is the input word sequence.

[0042] Then, in the multi-head attention computation stage, the input embedding vector can be processed by layer normalization (LayerNorm) to obtain E'; then multi-head attention computation is performed, for example, E' is input into 8 heads respectively, and the attention weights of different subspaces are calculated in parallel to capture the semantic association between words (such as the action-object association between "Segment" and "bladder"), and added to E to obtain E''; then layer normalization is performed on E''' to obtain E'''. Finally, in the feedforward neural network processing stage, E''' can be processed by the feedforward neural network and added to E'' to obtain E''''. The obtained E'''' is the aforementioned text feature. This text feature can fully capture the semantic information of the query text and its association with medical entities.

[0043] In this embodiment, the VSS module and FFE module of the FMamba network are combined to capture long-distance dependencies and high-frequency edge features of medical images (such as the continuous structure of the lumbar vertebral body and the subtle edges of lesions). Cross-scale fusion further integrates global and local information, thereby improving the entity recognition accuracy of medical image features. The entity labeling enhancement of the lightweight Transformer encoder improves the recall rate of medical entities in text features.

[0044] Furthermore, the lightweight design of FMamba and the lightweight Transformer improves the overall inference speed of the model while reducing memory usage, enabling deployment in small medical devices at primary healthcare institutions and expanding the scope of technology applications.

[0045] Step A3: Perform cross-modal fusion of the medical image features and the text features through a bidirectional attention mechanism to generate cross-modal fused features.

[0046] It is easy to understand that the aforementioned bidirectional attention mechanism can be the core mechanism for cross-modal interaction, including "image-to-text attention" and "text-to-image attention", which enables image features and text features to guide each other and become related, rather than being unidirectionally dependent, ensuring that the fused features contain both image details and text semantics.

[0047] In one feasible implementation, step A3 may include steps A31 to A33: Step A31: Construct a first query vector based on the text features, construct a first key vector and a first value vector based on the medical image features, determine the image-to-text attention weight through the first query vector and the first key vector, and obtain the text enhancement feature that fuses image information by combining the first value vector and the image-to-text attention weight. It should be understood that this embodiment can perform cross-modal fusion of medical image features and text features through a bidirectional attention mechanism. First, the question-answering device can construct a first query vector based on text features. (Mapping text features to a specific dimension, such as 512, through a linear projection layer) serves to guide the model to focus on regions in the image features related to the text instructions, such as when the text mentions "2nd lumbar vertebrae". It can guide attention to the lumbar spine area in the image.

[0048] Simultaneously, a first key vector can be constructed based on medical image features. and the first value vector (Similarly, medical image features are mapped to 512 dimensions using a linear projection layer). The first key vector... Used with Calculate similarity to measure the degree of association between image regions and text words; first value vector The specific feature information of the image region is carried and then transferred to the text features after being weighted by attention weights, thereby realizing the fusion of image information into text.

[0049] Then, the question-answering device can base its response on the first query vector. and the first key vector Calculate the normalized weight matrix, which represents the attention weights from image to text. Its dimensions are 256×256 (number of text terms × number of image regions). Each element in the matrix represents the attention level of the i-th text term to the j-th image region. A higher weight value indicates a stronger correlation between the image region and the text term, as shown below: (2) In the formula, The weights are given by the dimension (512) of the key vector. It reflects the degree of attention that text words pay to image regions. For example, the weight value of the bladder region in the image corresponding to the text word "bladder" is significantly higher than that of other regions.

[0050] Finally, the attention weights from the image to text described above are applied. With the first value vector Multiplying them together yields text enhancement features that fuse image information. This text enhancement feature Based on the original text features, image region information related to the text semantics was incorporated (e.g., when the text mentions "segmented bladder"), Image details that can be incorporated into the bladder region (such as the shape or location features of the bladder) make the text features more visually relevant, providing support for the subsequent generation of accurate entity description text.

[0051] Step A32: Construct a second query vector based on the medical image features, construct a second key vector and a second value vector based on the text features, determine the text-to-image attention weights through the second query vector and the second key vectors, and obtain image enhancement features that fuse text information by combining the second value vectors and the text-to-image attention weights. Similarly, question-answering devices can also construct a second query vector based on medical image features. (Mapped to 512 dimensions via a linear projection layer) to guide the model to focus on textual features related to image regions (e.g., when the "liver" region appears in the image). (This can guide attention to segmentation descriptions related to the liver in the text).

[0052] Then, a second key vector is constructed based on text features. Second value vector (Mapped to 512 dimensions via a linear projection layer). Second bond vector Used with Calculate similarity to measure the degree of matching between text instructions and image regions; the second value vector. The specific semantic information carrying the text instructions can be transferred to image features after attention weighting, thereby realizing the fusion of text information into the image.

[0053] At this time, the question-and-answer device can... and The normalized weight matrix is ​​calculated, which represents the attention weights from text to image. Its dimensions can be 256×256×256 (number of image regions × number of text terms). Each element in the matrix represents the response weight of the i-th image region to the j-th text term. The higher the weight value, the stronger the match between the text term and the image region. The calculation formula is as follows: (3) The attention weights from text to image can reflect the response weights of image regions to text commands. For example, the weight value of the text word "liver" corresponding to the liver region in an image is significantly higher than that of other words.

[0054] At this point, the question-answering device can multiply the text-to-image attention weights with the second value vector to obtain image enhancement features that fuse text information. This feature incorporates textual instructions associated with the image region into the original image features (e.g., integrating the liver region features in the image with the semantic instruction "segment liver", such as the anatomical definition or functional description of the liver), making the image features more semantically directional and providing support for subsequent accurate localization of entity regions.

[0055] Step A33: The text enhancement features of the fused image information are concatenated with the image enhancement features of the fused text information to generate cross-modal fusion features.

[0056] It is easy to understand that question-answering devices can enhance text by incorporating image information. Image enhancement features with fused text information The operation involves combining elements based on feature dimensions. Because... The dimensions are 256×512. The dimensions are 256×256×512, and it needs to be expanded first (by repeating the stacking operation). Expanding from 256×512 to 256×256×512, and then concatenating element by element along the feature channel dimension (the last dimension), we finally obtain a cross-modal fusion feature with a dimension of 256×256×1024. This feature simultaneously contains "text-guided image semantics" and "image-supported text semantics", realizing the deep integration of multimodal information and providing a unified and accurate feature foundation for the subsequent generation of "text + mask" multimodal question answering results.

[0057] In this embodiment, a bidirectional attention mechanism is used to calibrate image and text features, enabling mutual guidance between them and avoiding semantic bias caused by unidirectional dependence, thus solving the problem of inaccurate association of multimodal information. Furthermore, for complex medical question-answering tasks involving multiple entities and instructions (such as segmenting the 2nd, 3rd, 4th, and 5th lumbar vertebrae and describing their features), the bidirectional attention weight matrix proposed in this embodiment can simultaneously capture the association between multiple text terms and multiple image regions (e.g., the term "2nd lumbar vertebra" corresponds to the image region of the 2nd lumbar vertebra, and the term "3rd lumbar vertebra" corresponds to the image region of the 3rd lumbar vertebra), solving the matching problem between complex instructions and multi-region images, and effectively supporting complex scenarios such as clinical diagnosis and medical teaching.

[0058] Furthermore, this embodiment can achieve "deep semantic fusion" by integrating text features into precise image region information and image features into precise text instruction information through "attention weight weighted fusion," thereby solving the problem of existing entity localization and description errors.

[0059] Step S30: Based on the cross-modal fusion features, generate a natural language response containing medical entities and a pixel-level segmentation mask corresponding to the medical entities to form a multimodal output result.

[0060] It's important to understand that the aforementioned medical entities can be specific objects within the medical field, including anatomical structures (such as the lumbar spine, femur, bladder, and liver) and lesions (such as skin lesions). These entities are the core focus of medical analysis and question answering. The aforementioned natural language responses can be generated in a human-understandable natural language format to answer the query text. They must include accurate descriptions of the medical entities and explanations of relevant medical knowledge, such as "The image contains 'bladder' and 'liver,' and the segmentation results are visible." They must accurately respond to the query while conforming to medical terminology.

[0061] The aforementioned pixel-level segmentation mask can be a binary image with the same resolution as the medical image, where specific pixel values ​​(such as 1) mark the area corresponding to the medical entity, and other pixel values ​​(such as 0) mark the background area. It can accurately display the position and range of the medical entity in the medical image at the pixel level, such as marking the specific pixel areas of the 2nd lumbar vertebra, the 3rd lumbar vertebra, etc. in the image.

[0062] Therefore, the multimodal output of the question-answering device can simultaneously include natural language responses and pixel-level segmentation masks, and the two outputs are interconnected. The medical entities in the natural language responses correspond one-to-one with the pixel regions in the segmentation masks, facilitating users' intuitive understanding of the medical image analysis results. The output effect can be referenced... Figure 2 , Figure 2 This is a schematic diagram of the multimodal output results of the first embodiment of the pixel-level decoding medical visual question-answering method of this application. Figure 2 As shown, the input is a medical image and a corresponding question, and the output is an image with segmentation masks of different colors, representing the segmented lesion regions, and the corresponding text response.

[0063] In summary, existing medical visual question-answering algorithms lack prior knowledge of specific diseases and fail to incorporate relevant medical concepts, making it difficult to provide professional answers to entities in images. This embodiment addresses this problem by identifying medical entities during the preprocessing stage and integrating medical-related features during feature extraction and fusion, thereby indirectly introducing medical domain knowledge.

[0064] In response to the shortcomings of existing visual question-answering models that rely too heavily on text instructions and whose medical questions described in plain text may lead to ambiguity, this embodiment combines medical images and query text for cross-modal processing. This reduces the reliance on plain text instructions, enabling the model to understand user needs and medical image content more comprehensively and accurately. This improves the accuracy and professionalism of the question-answering results and reduces the probability of ambiguity.

[0065] Finally, addressing the issues of existing medical visual question-answering text responses being difficult to understand, and the potential for overly concise or complex text alone, this embodiment generates a multimodal output. The natural language response accurately includes explanations of medical entities and related knowledge, while pixel-level segmentation masks precisely mark the locations of medical entities. These two elements are interconnected, significantly improving the readability and usability of the medical visual question-answering results, allowing users to quickly obtain and understand key information from medical images.

[0066] This embodiment provides a pixel-level decoding method for medical visual question answering. The method includes: preprocessing the input medical image and query text to generate multimodal medical data; the multimodal medical data includes standardized medical images and word sequence text; inputting the standardized medical images into an improved visual state space network to obtain medical image features; inputting the word sequence text into a lightweight Transformer encoder to obtain text features; constructing a first query vector based on the text features, constructing a first key vector and a first value vector based on the medical image features, determining the image-to-text attention weights through the first query vector and the first key vector, and combining the first value vector and the image-to-text attention weights to obtain text enhancement features that fuse image information; constructing a second query vector based on the medical image features, constructing a second key vector and a second value vector based on the text features, determining the text-to-image attention weights through the second query vector and the second key vector, and combining the second value vector and the text-to-image attention weights to obtain image enhancement features that fuse text information; concatenating the text enhancement features that fuse image information with the image enhancement features that fuse text information to generate cross-modal fusion features. Based on cross-modal fusion features, a natural language response containing medical entities and a pixel-level segmentation mask corresponding to the medical entities are generated, forming a multimodal output result.

[0067] Therefore, existing medical visual question-answering algorithms lack prior knowledge of specific diseases and fail to incorporate relevant medical concepts, making it difficult to provide professional answers to entities in images. This embodiment addresses this problem by identifying medical entities during the preprocessing stage and integrating medical-related features during feature extraction and fusion, thereby indirectly introducing medical domain knowledge.

[0068] In response to the shortcomings of existing visual question-answering models that rely too heavily on text instructions and whose medical questions described in plain text may lead to ambiguity, this embodiment combines medical images and query text for cross-modal processing. This reduces the reliance on plain text instructions, enabling the model to understand user needs and medical image content more comprehensively and accurately. This improves the accuracy and professionalism of the question-answering results and reduces the probability of ambiguity.

[0069] Finally, addressing the issues of existing medical visual question-answering text responses being difficult to understand, and the potential for overly concise or complex text alone, this embodiment generates a multimodal output. The natural language response accurately includes explanations of medical entities and related knowledge, while pixel-level segmentation masks precisely mark the locations of medical entities. These two elements are interconnected, significantly improving the readability and usability of the medical visual question-answering results, allowing users to quickly obtain and understand key information from medical images.

[0070] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description, and will not be repeated hereafter.

[0071] Understandably, traditional object detection methods follow a process from input image to output segmentation mask, and do not support one-to-one alignment of segmentation with text content in the model output. Therefore, this embodiment adopts a completely different architectural design approach, reconstructing the traditional parallel dual-branch structure into a unified framework based on multimodal interaction, and taking text generation as the main focus, treating image segmentation as a natural extension of entity recognition in the text generation process.

[0072] Based on the first embodiment, please refer to Figure 3 , Figure 3 This is a first flowchart illustrating a second embodiment of the pixel-level decoding medical visual question-answering method of this application. In this embodiment, step S30 includes steps B1 to B3: Step B1: Based on the cross-modal fusion features, simultaneously generate a natural language response containing medical entities and the corresponding attention distribution for the medical entities; It should be noted that question-answering devices can fuse features across modalities. The input consists of a text generator and an entity locator built using a Transformer decoder. The text generator generates natural language responses containing medical entities using an autoregressive approach. Simultaneously, during the generation of the natural language response, the attention distribution for each word corresponding to the medical entity is acquired. The two are synchronized in terms of time and data association, ensuring that the attention distribution accurately corresponds to the medical entities in the natural language response and avoiding miscorrelation caused by step-by-step generation.

[0073] Understandably, the attention distribution mentioned above can be the weight distribution of attention to cross-modal fusion features when the text generator generates each word in the natural language response. It reflects the cross-modal information (the related part of image features and text features) that the model depends on when generating the word. For words corresponding to medical entities, the attention distribution can reflect the relevant regional information of the medical entity in the medical image.

[0074] Therefore, in one feasible implementation, refer to Figure 4 , Figure 4 This is a second flowchart illustrating a second embodiment of the pixel-level decoding medical visual question-answering method of this application. In this embodiment, step B1 may include steps B11 to B13: Step B11: Input the cross-modal fusion features into the lightweight Transformer decoder to obtain text generation features; Understandably, question-answering devices can fuse features across modalities. Input a lightweight Transformer decoder, and sequentially perform cross-attention computation, feedforward neural network processing, and layer normalization to obtain text generation features.

[0075] For example, in the cross-attention computation stage, the cross-modal fusion features need to be used to update the attention of the current hidden state of the decoder, so that the decoder can pay attention to the key information related to the current text generation in the cross-modal fusion features, establish the association between text generation and multimodal input, and ensure that the generated text meets the semantic requirements of multimodal input. The Transformer decoder can first perform layer normalization on the initial decoding vector D of the input (initialized as a random word embedding vector (dimension 512) containing the start marker (such as [BOS]) or a specific initial vector) to obtain D'=LayerNorm(D); then use the cross-modal fusion feature F_fusion to update the attention of D', capture the key information in the fusion feature, and add it to D to obtain D''=D+CrossAttention(D',F_fusion); Then, layer normalization is performed on D'' to obtain D''' = LayerNorm(D''). Next, in the feedforward neural network processing stage, D''' undergoes nonlinear transformation and dimensionality adjustment to enhance its expressive power, and is added to D'''' to obtain D'''' = D'' + FeedForward(D'''). D'''' is the aforementioned text generation feature, containing the semantic information required for text generation. This feature integrates key information from cross-modal fusion features and can be directly used to calculate the probability distribution of the next word.

[0076] Step B12: Perform linear transformation and Softmax processing on the text generation features to obtain the probability distribution of the next word; Step B13: Select target words based on the probability distribution, concatenate the target words word by word to form a natural language response containing medical entities, and record the attention distribution corresponding to the medical entities when generating the target words.

[0077] It's easy to understand that question-answering devices can use a linear layer to map the feature dimensions of text generation features to a dimension consistent with the size of the model's vocabulary, matching the feature dimensions with the number of words in the vocabulary. This lays the foundation for subsequently calculating the generation probability of each word. Then, a Softmax linear processing is performed to normalize the linearly transformed features, converting the feature values ​​into probability values ​​within the [0,1] interval, with the sum of all probability values ​​being 1. This yields the distribution data of the probability that the model will generate the next word as each word in the vocabulary, i.e., the probability distribution of the next word P = Softmax(Linear(D'''')), where each element in P represents the probability of generating the corresponding word. This distribution provides an intuitive understanding of the likelihood of each word being generated, serving as an important basis for selecting target words.

[0078] Then, the question-answering device can select target words based on the probability distribution P. It can employ a greedy search strategy, selecting the word with the highest probability as the currently generated target word; or it can use other strategies such as bundle search (selecting the k words with the highest probability as candidates and filtering them step by step), choosing the appropriate word selection method according to actual needs. The target words are then concatenated word by word in the generation order. By continuously repeating the process of selecting target words and concatenating them word by word, a complete natural language response is gradually constructed until an end marker (such as [EOS]) is generated or the preset text length limit is reached.

[0079] Furthermore, during the generation of each target word, the model's processing of cross-modal fusion features can be recorded simultaneously. The attention weight distribution is the attention distribution of the corresponding medical entity (if the current target word is a medical entity-related word).

[0080] In this implementation, a lightweight Transformer decoder is employed, reducing computational complexity and memory consumption. This addresses the poor applicability of traditional text generators in resource-constrained scenarios, enabling the model to be deployed in more medical settings (such as small devices in primary healthcare institutions and mobile medical terminals), thus expanding its applicability. Furthermore, during text generation, cross-attention computation is used to associate medical entity information from cross-modal fusion features, ensuring that the generated text contains accurate medical entities and resolving the issue of insufficient professionalism in responses. The simultaneous recording of attention distribution eliminates the need for additional computational steps to obtain attention information corresponding to medical entities, providing convenient and accurate data support for subsequent generation of pixel-level segmentation masks. This reduces the overall computational steps of the model and improves the generation efficiency of multimodal output results.

[0081] Step B2: Map the attention distribution onto the medical image to obtain a pixel-level segmentation mask corresponding to the medical entity; It is easy to understand that the entity locator can generate a pixel-level segmentation mask corresponding to a medical entity based on the attention distribution and medical image features. For example, the entity locator can transform the attention distribution from the feature space to the original medical image space, and through operations such as upsampling, make the size of the attention distribution consistent with the resolution of the medical image, establish the correspondence between the attention distribution and the pixels of the medical image, thereby determining the approximate location range of the medical entity in the image and obtaining the pixel-level segmentation mask corresponding to the medical entity.

[0082] In one feasible implementation, step B2 may include steps B21 to B23: Step B21: Upsample the attention distribution to the image space of the medical image to obtain an attention map corresponding to the medical entity; Step B22: Generate a multi-scale initial segmentation mask based on the attention map; Step B23: Fuse the initial segmentation masks of the multi-scale to obtain a pixel-level segmentation mask.

[0083] It is easy to understand that in this embodiment, the recorded attention distribution (corresponding to the attention weight distribution of the medical entity) can be mapped to the image space of the original medical image (a spatial range based on the pixel coordinates of the original medical image, with the same resolution as the original medical image (e.g., 256×256 pixels, where each coordinate point corresponds to a specific pixel in the image) through an upsampling operation (such as bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation; this embodiment preferably uses bilinear interpolation, which determines the pixel value of the sampling point by calculating the weighted average of the four pixels surrounding the sampling point, thus better preserving the smoothness and detail of the image and avoiding obvious blockiness) through an upsampling operation (the image space of the original medical image based on the pixel coordinates of the original medical image, with the same resolution as the original medical image (e.g., 256×256 pixels), where each coordinate point corresponds to a specific pixel in the image), so that the size of the attention distribution is consistent with that of the medical image. After the attention distribution is mapped to this space, an attention weight distribution map that directly corresponds to the pixels of the medical image can be established, i.e., the attention map A corresponding to the medical entity is A=Upsample( · ),in The attention distribution for text-to-image processing obtained by the cross-modal fusion module. The value part of the image features is the first value vector mentioned above.

[0084] It's important to understand that the value of each pixel in attention map A represents the degree of association between that location and the corresponding medical entity. A higher value indicates a greater likelihood that the location belongs to a medical entity region, intuitively reflecting the approximate distribution range of medical entities in the medical image. Furthermore, based on attention map A, a transpose convolution operation (ConvTranspose) can be used to adjust the dimensions and restore features of the attention map. Then, after processing with the Sigmoid function, the output values ​​are mapped to the [0,1] interval, generating a binary image M=Sigmoid(ConvTranspose(A)) used to label medical entity regions. During this process, an initial segmentation threshold (e.g., 0.5) can be set. Pixel values ​​in the segmentation mask greater than the initial segmentation threshold are set to 1 (representing medical entity regions), and pixel values ​​less than or equal to the initial segmentation threshold are set to 0 (representing background regions).

[0085] It should be noted that, during the aforementioned medical image feature extraction stage, the FMamba network can output intermediate multi-scale features at three resolutions: 128×128×128, 64×64×256, and 32×32×512. Therefore, this embodiment can extract the corresponding attention distribution (obtained through text-to-image attention calculation in the cross-modal interaction module) for each resolution feature, and use the aforementioned upsampling method to upsample the attention distribution to a dimension of 256×256, obtaining attention maps corresponding to the three resolution features. (128×128 feature correspondence) (64×64 feature correspondence) (32×32 feature correspondence).

[0086] Then, it is possible to perform multi-scale... , , Performing the transposed convolution, sigmoid processing, and thresholding operations described above, respectively, generates the corresponding multi-scale initial segmentation mask, which can be represented as follows: , , All dimensions are 256×256. It places greater emphasis on local details of medical entities (such as edge textures), giving them higher weight. and It places greater emphasis on the global semantic information of medical entities (such as their overall outline), with appropriate weighting.

[0087] Finally, a fusion method that involves adding elements one by one and then normalizing can be used to... , , Learnable weighted fusion is performed, assuming the pre-set weight coefficients are 0.4 ( ), 0.3 ( ), 0.3 ( (The weights can be adjusted experimentally, with the sum of the weights being 1, so that the fused mask takes into account both detailed and global information.) The fusion process can be represented as: =0.4* +0.3* +0.3* .

[0088] Furthermore, the fused result can be further thresholded (with the threshold still set to 0.5), setting pixels with values ​​greater than 0.5 to 1 and pixels with values ​​less than or equal to 0.5 to 0, ultimately obtaining a pixel-level segmentation mask. This mask contains both local details of the medical entity and accurately reflects its overall outline.

[0089] In this embodiment, the attention distribution is mapped to the image space through upsampling, establishing a direct correlation between attention information and image pixels. This solves the problem of mismatched localization dimensions and improves the spatial accuracy of medical entity localization. Simultaneously, by fusing intermediate masks from different levels to the initial segmentation mask, detailed and semantic information is effectively supplemented, ultimately obtaining a high-precision pixel-level segmentation mask. This addresses issues such as incomplete regions and lost details that may exist with single-scale masks, improving the accuracy and completeness of the mask. Furthermore, the entire localization process is based on the attention distribution already present during text generation, eliminating the need for additional annotation of supervision signals. This reduces reliance on manually labeled data, lowers dataset construction costs, simplifies model training, and improves the efficiency of entity localization.

[0090] Step B3: Perform edge optimization processing on the pixel-level segmentation mask to obtain the final segmentation mask, and associate the natural language response with the final segmentation mask to form a multimodal output result.

[0091] Understandably, in this embodiment, the question-answering device can perform boundary adjustment and optimization on the initially generated pixel-level segmentation mask, i.e., the aforementioned edge optimization processing, which aims to solve problems such as blurred or discontinuous boundaries that may exist in the pixel-level segmentation mask. By using image edge information through specific algorithms (such as conditional random field models), the boundaries of the pixel-level segmentation mask are made clearer and more accurate, and the final segmentation mask that conforms to the actual boundary features of entities in medical images is obtained.

[0092] In one feasible implementation, step B3 may include steps B31-B32: Step B31: Construct a conditional random field model that includes univariate and bivariate potential functions; Step B32: Input the pixel-level segmentation mask and the probability distribution into the conditional random field model, and optimize the global energy function of the conditional random field model through an iterative conditional pattern algorithm to obtain the final segmentation mask.

[0093] It is important to understand that the aforementioned Conditional Random Field (CRF) model can be a post-processing tool based on a probabilistic graphical model. By defining "univariate potential functions" and "binary potential functions" to model the probability distribution of pixel labels, it can optimize the segmentation results by utilizing the spatial correlation and intensity differences (edge ​​information) between image pixels. Compared with traditional smoothing algorithms (such as Gaussian filtering), it can more accurately preserve edge details.

[0094] The univariate potential function is a function in the CRF model used to measure the matching degree between a single pixel label in the pixel-level segmentation mask and the predicted probability. It is directly related to the probability distribution output by the entity locator, ensuring the overall consistency between the optimized mask and the initial prediction result. The formula is: (4) In the formula, It is the label of pixel i in the pixel-level segmentation mask (0=background, 1=medical entity); P It is the probability distribution of pixel i output by the entity locator belonging to the medical entity region (the discrete result obtained after taking a threshold for P is the value of pixel i in the pixel-level segmentation mask), which is used to measure the degree of matching between pixel i and the label.

[0095] The binary potential function is used in the CRF model to model the relationship between adjacent pixels, focusing on image edge information. By introducing differences in pixel position and intensity (edge ​​information), it causes adjacent pixels to tend to share the same label in "non-edge regions" to maintain regional continuity, and tend to share different labels in "edge regions" to facilitate distinguishing labels on either side of the edge. Its formula is: (5) In the formula, It is a tag compatibility function (if) = The μ value is relatively small; if ≠ (with a relatively large μ value) and These are the positions of pixels i and j in the pixel-level segmentation mask; It is the intensity of pixel i (the grayscale value or RGB value of pixel i, etc.); This is a hyperparameter.

[0096] Understandably, the aforementioned global energy function can be the objective function in the CRF model that measures the rationality of the overall image label assignment. It is obtained by summing the unary potential function of all pixels and the binary potential function of neighboring pixels, as shown below: (6) In the formula, X is the label assignment for the entire image.

[0097] It is important to understand that the lower the energy corresponding to the global energy function, the more reasonable the label allocation (the more accurate the segmentation result). In this embodiment, the question-answering device can first use the pixel values ​​(0 / 1) of the pixel-level segmentation mask as the initial label set, and calculate the initial global energy according to formula (6). Then, the iterative conditional mode algorithm (ICM) is used to continuously update the initial label set through the iterative process of "fixing other pixel labels → updating a single pixel label → selecting a label that can reduce (or not increase) the initial global energy and minimize the local energy", gradually reducing the global energy until it converges to a local minimum. Finally, the global label allocation X with the lowest corresponding energy is obtained, which is then converted into a binary image (label 1 corresponds to pixel value 1, label 0 corresponds to pixel value 0) to obtain the final segmentation mask. The mask has clear boundaries and continuous regions, corresponding to more accurate segmentation results, and is highly matched with the actual shape of the medical entity.

[0098] It's important to understand that after obtaining the final segmentation mask, the question-answering device can associate the natural language response with the high-precision pixel-level segmentation mask. This establishes a clear correspondence between each medical entity in the natural language response and its corresponding pixel region in the final segmentation mask before outputting the result. For example, medical entities in the natural language response can be marked with special markers (such as labels or specific colors), while the corresponding pixel regions of these entities in the final segmentation mask can be marked with specific colors or pixel values. Integrating the labeled natural language response with the color-marked final segmentation mask forms a multimodal output result. Users can intuitively see the specific location of the medical entities described in the text within the image, thus linking the text description with image regions and improving the efficiency of clinical interpretation.

[0099] Furthermore, since existing model training methods do not take into account the consistency between text generation and entity localization tasks, resulting in poor overall model performance, in one possible implementation, before step S10, a step of training the medical visual question answering model is included. This training step may include: (a) Obtain a medical multimodal training dataset, which includes training multimodal medical data, corresponding text response labels, and real segmentation mask labels input by the user; (b) Based on the training multimodal medical data, generate training text responses and training segmentation masks using the same method described above; (c) Calculate the text generation loss, mask generation loss, and consistency loss, where the text generation loss is calculated based on the cross-entropy between the training text response and the text response label, and is expressed as follows: (7) In the formula, N is the text length. It is the true label of the i-th word. It is the probability distribution predicted by the model.

[0100] The text generator described above ultimately produces It is a global probability distribution (covering all possible words), and the loss function here... It is the predicted probability (i.e., the specific value of P at position i) corresponding to the i-th word position in the global probability distribution, which is essentially a local focus of the same distribution.

[0101] The mask generation loss is calculated based on a combination of the Dice loss and the binary cross-entropy loss of the trained segmentation mask and the real segmentation mask labels, and is expressed as follows: (8) In the formula, It is the actual mask (the image mask determined after data processing of the visual cues input by the user, such as bounding boxes, point coordinates, etc.). It is the predicted training segmentation mask. It is a smoothing term, and BCE is the binary cross-entropy loss. It is a balance parameter.

[0102] The consistency loss is calculated based on the L2 norm of the pooled features resulting from the product of the embedded features of the training text response and the training segmentation mask and medical image features, and is expressed as follows: (9) In the formula, Embed is the text embedding function, and Pool is the pooling operation. These are medical image features.

[0103] (d) Construct the total loss function based on text generation loss, mask generation loss, and consistency loss, and express it as follows: (10) In the formula, and It is a hyperparameter that balances the various loss terms.

[0104] Finally, the parameters of the medical visual question-answering model are optimized by backpropagation through the above formulas (7) to (10) until the model converges.

[0105] Therefore, this embodiment can improve the overall performance of the model in text generation and entity localization tasks through multi-task joint training and consistency constraints.

[0106] In summary, this embodiment ensures a one-to-one correspondence between medical entities in the natural language response and the attention distribution by synchronously generating natural language responses and attention distributions, and generates segmentation masks based on the attention distributions. This solves the problem of text and image regions not being able to correspond, making the association between text descriptions and image regions closer and improving the intuitiveness of medical visual question answering results.

[0107] Meanwhile, this embodiment improves the boundary accuracy of the segmentation mask through edge optimization processing, solving the problem of insufficient accuracy of existing segmentation results. It can more accurately reflect the actual position and shape of medical entities in medical images, meet the needs of medical diagnosis, lesion analysis and other scenarios for high-precision segmentation results, and improve the clinical application value of the model.

[0108] Furthermore, the way natural language responses and final segmentation mask are associated in this embodiment allows users to quickly associate medical entities in the natural language responses with corresponding regions in the image, reducing the difficulty of understanding. This is more convenient for professional medical staff to use for diagnostic reference or for ordinary users to understand medical image information, effectively improving the efficiency of clinical interpretation.

[0109] This embodiment discloses a method for inputting cross-modal fusion features into a lightweight Transformer decoder to obtain text generation features; performing linear transformation and Softmax processing on the text generation features to obtain the probability distribution of the next word; selecting target words based on the probability distribution, concatenating the target words word by word to form a natural language response containing medical entities, and recording the attention distribution of the corresponding medical entities when generating the target words. The attention distribution is upsampled to the image space of the medical image to obtain an attention map corresponding to the medical entities; multi-scale initial segmentation masks are generated based on the attention maps; the multi-scale initial segmentation masks are fused to obtain a pixel-level segmentation mask. A conditional random field (CRF) model containing univariate and binary potential functions is constructed; the pixel-level segmentation mask and probability distribution are input into the CRF model, and the global energy function of the CRF model is optimized through an iterative conditional pattern algorithm to obtain the final segmentation mask.

[0110] Therefore, this embodiment ensures a one-to-one correspondence between medical entities in the natural language response and the attention distribution by synchronously generating natural language responses and attention distributions, and generates segmentation masks based on the attention distributions, thus solving the problem of text and image regions not being able to correspond, making the association between text descriptions and image regions closer, and improving the intuitiveness of medical visual question answering results.

[0111] Meanwhile, this embodiment improves the boundary accuracy of the segmentation mask through edge optimization processing, solving the problem of insufficient accuracy of existing segmentation results. It can more accurately reflect the actual position and shape of medical entities in medical images, meet the needs of medical diagnosis, lesion analysis and other scenarios for high-precision segmentation results, and improve the clinical application value of the model.

[0112] Furthermore, the way natural language responses and final segmentation mask are associated in this embodiment allows users to quickly associate medical entities in the natural language responses with corresponding regions in the image, reducing the difficulty of understanding. This is more convenient for professional medical staff to use for diagnostic reference or for ordinary users to understand medical image information, effectively improving the efficiency of clinical interpretation.

[0113] For example, to help understand the technical concept or principle of the pixel-level decoding medical visual question-answering method combined with Embodiments 1 and 2 above, please refer to Figure 5 , Figure 5 The following is a schematic diagram illustrating the pixel-level decoding process of the medical visual question-answering method of this application: This application proposes a medical visual language model with unified multimodal input and pixel mask output. The model is designed with a multimodal input alignment module, which can make full use of input data from multiple different modalities and effectively fuse them to provide accurate feature representation for large language models, thereby injecting multimodal fusion features into large language models.

[0114] like Figure 5 As shown, the overall architecture of the new model mainly includes the following key components: Image Encoder: Employs an improved Vision Mamba network as the backbone to extract multi-scale features from the input medical image; Text Encoder: Uses a lightweight Transformer encoder to encode the input user request, i.e., the query text; Cross-Modal Interaction Module: Enables deep interaction between medical image features and text features through a bidirectional attention mechanism; Text Generator: Generates natural language responses containing entities based on a lightweight Transformer decoder; Entity Locator: Locates and generates pixel-level segmentation masks of entities mentioned in the text based on the attention distribution during text generation; Mask Refiner: Uses an edge-aware Conditional Random Field (CRF) to refine the initially generated pixel-level segmentation mask, obtaining a final segmentation mask with improved boundary accuracy.

[0115] Compared with traditional methods, the significant feature of the new model is that it integrates the image segmentation task into the text generation process, and automatically guides pixel-level segmentation by focusing on entities during text generation, thereby achieving closer cross-modal association and more efficient resource utilization.

[0116] exist Figure 5 Based on the structure shown, the data processing flow of this application starts from the input layer. In the data preprocessing stage, the medical image I can be normalized and adjusted to a uniform resolution (e.g., 256×256); the user-input query text Text segmented into fixed-length word sequences by a tokenizer; optional visual cues for user input. (Such as bounding boxes and point coordinates) can be normalized to the [0,1] interval and labeled with the type (foreground / background, box corner point category).

[0117] After preprocessing, the multimodal data are fed into their respective single-modal encoders for further processing. In the image encoder, a modified Vision Mamba network is used to extract multi-scale features from the medical image I. During this process, the Visual State Space (VSS) module dynamically captures long-distance relationships between pixels (such as the spatial relationship between tumors and surrounding blood vessels in medical images) along the width and height dimensions of the feature map using a two-dimensional selective scan (SS2D) operation, enhancing global semantic and spatial relational features. Then, the Frequency Feature Enhancement (FFE) module transforms the features from the spatial domain to the frequency domain using Fourier transform, strengthens key frequency information using a learnable frequency mask (learned through training to correspond to the frequency range of effective features in the medical image, such as lesion edges corresponding to high-frequency components), and then transforms it back to the spatial domain using an inverse Fourier transform. Finally, the feature maps after "VSS processing + FFE enhancement" at various resolutions are fused across scales to output multi-scale medical image features. Lightweight Transformer encoder for processing query text It also performs entity tagging enhancement to generate text features that capture semantic relationships. .

[0118] Meanwhile, the visual cue processor uses coordinate mapping and type embedding to... Transformed into visual cue features of a unified dimension .

[0119] Subsequently, medical image features Text features Entering the cross-modal interaction module, cross-modal fusion is performed through a bidirectional attention mechanism (images and text / cues guide and correlate with each other). During this process, text-enhanced features that fuse image information can be generated. Image enhancement features that integrate textual information Then, the two at different scales are fused into a cross-modal fusion feature containing multimodal information. Simultaneously, mapping the cross-modal attention distribution to the original image space provides direct guidance for entity localization, achieving visualization of the attention distribution: This fusion feature First, a lightweight Transformer decoder-based text generator is input. This generator autoregressively generates text responses T containing medical entities, while simultaneously outputting the attention distribution (mapped back to the original image size) corresponding to each entity during the guided generation process. During generation, this text generator pays particular attention to semantic information related to entities, ensuring accurate entity descriptions in the generated text. This is achieved through an entity-aware attention mechanism. The principle is that during generation, entity features output from the text encoder are used as attention weights, causing the model to allocate higher attention to entity-related tokens, ensuring accurate entity descriptions and achieving entity-guided generation. Simultaneously, dynamic dictionary expansion based on medical terminology can improve the accuracy of terminology generation. This is achieved by detecting uncollected medical terms in the input / generation, calling corresponding word embeddings from a pre-defined terminology database, updating the model's dictionary, and improving the accuracy of terminology generation through dynamic dictionary expansion.

[0120] Furthermore, the attention distribution corresponding to each entity can be combined with the low-level detail features output by the image encoder, i.e. the value part of the image features, i.e. the first value vector mentioned above, and fed into the entity locator for threshold filtering and upsampling completion to obtain a preliminary mask (i.e. the pixel-level segmentation mask mentioned above).

[0121] Subsequently, the mask refiner incorporates edge information from the original image and iteratively optimizes the boundary continuity of the initial mask using edge-aware CRF to generate the final segmentation mask; Finally, the generated natural language response and the final segmentation mask are correlated and output as an entity-mask one-to-one mapping, and integrated into a multimodal response containing text and pixel-level segmentation links.

[0122] Furthermore, the image mask corresponding to the aforementioned visual cue features can be used as the true label to further iterate and optimize the model.

[0123] Therefore, this application designs and implements an automated construction strategy for medical multimodal datasets based on medical entities. This strategy starts from four different perspectives, covering basic instruction compliance and complex question-answering scenarios with multimodal inputs / outputs, and is specifically designed to address the challenge of missing medical image entity understanding and text reasoning datasets. Furthermore, a multimodal input alignment strategy is proposed to achieve feature fusion of multimodal input information and extract key information and contextual information from images, text instructions, and visual instructions. Furthermore, this application proposes a medical visual language model construction algorithm based on a pre-trained model that unifies multimodal input and pixel mask output. It injects multimodal fusion features into a large language model and utilizes the powerful language understanding and text generation capabilities of the large language model to achieve unified text and alignment mask output. In summary, this application achieves deep fusion of medical image features and text features, enabling the model to more comprehensively and accurately understand user needs and medical image content. This improves the accuracy and professionalism of question-and-answer results, significantly enhancing the readability and practicality of medical visual question-and-answer results, and facilitating users' rapid acquisition and understanding of key information from medical images. The standardized and unified operations in the preprocessing stage, as well as the use of efficient networks (such as FMamba) in the feature extraction stage, allow the model to adapt to different types and formats of medical data while ensuring processing efficiency, demonstrating strong versatility and practicality.

[0124] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the pixel-level decoding medical visual question answering method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0125] This application also provides a pixel-level decoding medical visual question-answering device; please refer to [reference needed]. Figure 6 , Figure 6 This is a schematic diagram of the module structure of the pixel-level decoding medical visual question-answering device according to an embodiment of this application. In this embodiment, the pixel-level decoding medical visual question-answering device includes: Image preprocessing module 601 is used to preprocess the input medical image and query text to generate multimodal medical data; The cross-modal fusion module 602 is used to perform cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features; The question-answering output module 603 is used to generate a natural language response containing a medical entity and a pixel-level segmentation mask corresponding to the medical entity based on the cross-modal fusion features, thereby forming a multimodal output result.

[0126] As one possible implementation, in this embodiment, the question-answering output module 603 is further configured to simultaneously generate a natural language response containing a medical entity and an attention distribution corresponding to the medical entity based on the cross-modal fusion features; map the attention distribution onto the medical image to obtain a pixel-level segmentation mask corresponding to the medical entity; perform edge optimization processing on the pixel-level segmentation mask to obtain a final segmentation mask; and associate the natural language response with the final segmentation mask for output, forming a multimodal output result.

[0127] As one possible implementation, in this embodiment, the question-answering output module 603 is further configured to input the cross-modal fusion features into a lightweight Transformer decoder to obtain text generation features; perform linear transformation and Softmax processing on the text generation features to obtain the probability distribution of the next word; select a target word based on the probability distribution, concatenate the target word word by word to form a natural language response containing medical entities, and record the attention distribution of the corresponding medical entity when generating the target word.

[0128] As one possible implementation, in this embodiment, the question-and-answer output module 603 is further configured to upsample the attention distribution to the image space of the medical image to obtain an attention map corresponding to the medical entity; generate a multi-scale initial segmentation mask based on the attention map; and fuse the multi-scale initial segmentation mask to obtain a pixel-level segmentation mask.

[0129] As one possible implementation, in this embodiment, the question-and-answer output module 603 is further used to construct a conditional random field model containing a univariate potential function and a binary potential function; input the pixel-level segmentation mask and the probability distribution into the conditional random field model, and optimize the global energy function of the conditional random field model through an iterative conditional pattern algorithm to obtain the final segmentation mask.

[0130] As one possible implementation, in this embodiment, the image preprocessing module 601 is also used to generate multimodal medical data including standardized medical images and word sequence text; The cross-modal fusion module 602 is further configured to input the standardized medical image into an improved visual state space network to obtain medical image features; input the word sequence text into a lightweight Transformer encoder to obtain text features; and perform cross-modal fusion of the medical image features and the text features through a bidirectional attention mechanism to generate cross-modal fusion features.

[0131] As one possible implementation, in this embodiment, the cross-modal fusion module 602 is further configured to: construct a first query vector based on the text features; construct a first key vector and a first value vector based on the medical image features; determine the image-to-text attention weight using the first query vector and the first key vector; and combine the first value vector and the image-to-text attention weight to obtain the text enhancement feature of the fused image information; construct a second query vector based on the medical image features; construct a second key vector and a second value vector based on the text features; determine the text-to-image attention weight using the second query vector and the second key vector; and combine the second value vector and the text-to-image attention weight to obtain the image enhancement feature of the fused text information; and concatenate the text enhancement feature of the fused image information with the image enhancement feature of the fused text information to generate a cross-modal fusion feature.

[0132] The pixel-level decoding medical visual question-answering device provided in this application, employing the pixel-level decoding medical visual question-answering method described in the above embodiments, can solve the diagnostic reliability problems in medical visual question answering caused by a lack of prior disease knowledge, ambiguous text instructions, and difficulties in understanding plain text responses. Compared with the prior art, this device achieves precise localization and structured output of medical entities through a multimodal input alignment and attention-guided mask generation mechanism, significantly improving the efficiency and reliability of clinical diagnosis. Other technical features in the device are the same as those disclosed in the above method embodiments and will not be repeated here.

[0133] This application provides a pixel-level decoding medical visual question-answering device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the pixel-level decoding medical visual question-answering method in Embodiment 1 above.

[0134] The following is for reference. Figure 7 This document illustrates a structural schematic of a medical visual question-and-answer device suitable for implementing pixel-level decoding in the embodiments of this application. The pixel-level decoding medical visual question-and-answer device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7The pixel-level decoding medical vision question-answering device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0135] like Figure 7 As shown, the pixel-level decoding medical vision question-and-answer device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the pixel-level decoding medical vision question-and-answer device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the pixel-level decoding medical vision question-and-answer device to communicate wirelessly or wiredly with other devices to exchange data. Although pixel-level decoding medical vision question-and-answer devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0136] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment disclosed in this application includes a pixel-level decoding medical vision question-and-answer program product, comprising a pixel-level decoding medical vision question-and-answer program carried on a computer-readable medium, the pixel-level decoding medical vision question-and-answer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the pixel-level decoding medical vision question-and-answer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the pixel-level decoding medical vision question-and-answer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0137] The pixel-level decoding medical visual question-answering device provided in this application, employing the pixel-level decoding medical visual question-answering method in the above embodiments, can solve the technical problems of pixel-level decoding medical visual question-answering. Compared with the prior art, the beneficial effects of the pixel-level decoding medical visual question-answering device provided in this application are the same as the beneficial effects of the pixel-level decoding medical visual question-answering method provided in the above embodiments, and other technical features in this pixel-level decoding medical visual question-answering device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0139] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0140] This application provides a storage medium having computer-readable program instructions (i.e., a pixel-level decoded medical visual question-and-answer program) stored thereon, the computer-readable program instructions being used to execute the pixel-level decoded medical visual question-and-answer method in the above embodiments.

[0141] The storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of the storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0142] The aforementioned storage medium may be included in a pixel-level decoding medical vision question-and-answer device; or it may exist independently and not be assembled into a pixel-level decoding medical vision question-and-answer device.

[0143] The aforementioned storage medium carries one or more programs, which, when executed by the pixel-level decoding medical vision question-and-answer device, enable the pixel-level decoding medical vision question-and-answer device to perform pixel-level decoding medical vision question-and-answer.

[0144] The pixel-level decoding medical vision question-answering program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and pixel-level decoding medical visual question-answering program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0147] The readable storage medium provided in this application is a storage medium storing computer-readable program instructions (i.e., a pixel-level decoding medical visual question-answering program) for executing the above-described pixel-level decoding medical visual question-answering method, which can solve the technical problem of pixel-level decoding medical visual question answering. Compared with the prior art, the beneficial effects of the storage medium provided in this application are the same as the beneficial effects of the pixel-level decoding medical visual question-answering method provided in the above embodiments, and will not be repeated here.

[0148] The above are only some embodiments of this application and do not limit the scope of the solution of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A pixel-level decoding method for medical visual question answering, characterized in that, The method includes: The input medical images and query text are preprocessed to generate multimodal medical data; Perform cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features; Based on the cross-modal fusion features, a natural language response containing medical entities and a pixel-level segmentation mask corresponding to the medical entities are generated, forming a multimodal output result.

2. The pixel-level decoding medical visual question-answering method as described in claim 1, characterized in that, The step of generating a natural language response containing medical entities and a pixel-level segmentation mask corresponding to the medical entities based on the cross-modal fusion features, to form a multimodal output result, includes: Based on the cross-modal fusion features, natural language responses containing medical entities and corresponding attention distributions for the medical entities are generated simultaneously. The attention distribution is mapped onto the medical image to obtain a pixel-level segmentation mask corresponding to the medical entity; Edge optimization processing is performed on the pixel-level segmentation mask to obtain the final segmentation mask. The natural language response is then associated with the final segmentation mask and output to form a multimodal output result.

3. The medical visual question-answering method with pixel-level decoding as described in claim 2, characterized in that, The step of simultaneously generating a natural language response containing medical entities and the corresponding attention distribution of the medical entities based on the cross-modal fusion features includes: The cross-modal fusion features are input into a lightweight Transformer decoder to obtain text generation features; The text generation features are subjected to linear transformation and Softmax pixel-level decoding for medical visual question answering to obtain the probability distribution of the next word; Target words are selected based on the probability distribution, and the target words are concatenated word by word to form a natural language response containing medical entities. The attention distribution corresponding to the medical entities when the target words are generated is recorded.

4. The pixel-level decoding medical visual question-answering method as described in claim 2, characterized in that, The step of mapping the attention distribution to the medical image to obtain a pixel-level segmentation mask corresponding to the medical entity includes: The attention distribution is upsampled to the image space of the medical image to obtain an attention map corresponding to the medical entity; Generate multi-scale initial segmentation masks based on the attention map; By fusing the initial segmentation masks of the multiple scales, a pixel-level segmentation mask is obtained.

5. The medical visual question-answering method with pixel-level decoding as described in claim 3, characterized in that, The step of performing edge optimization processing on the pixel-level segmentation mask to obtain the final segmentation mask includes: Construct a conditional random field model that includes univariate and bivariate potential functions; The pixel-level segmentation mask and the probability distribution are input into the conditional random field model, and the global energy function of the conditional random field model is optimized by the iterative conditional pattern algorithm to obtain the final segmentation mask.

6. The medical visual question-answering method with pixel-level decoding as described in claim 1, characterized in that, The multimodal medical data includes standardized medical images and word sequence text; The step of performing cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features includes: The standardized medical images are input into an improved visual state space network to obtain medical image features; The word sequence text is input into a lightweight Transformer encoder to obtain text features; Cross-modal fusion features are generated by fusing the medical image features and the text features through a bidirectional attention mechanism.

7. The medical visual question-answering method with pixel-level decoding as described in claim 6, characterized in that, The step of performing cross-modal fusion of the medical image features and the text features through a bidirectional attention mechanism to generate cross-modal fused features includes: A first query vector is constructed based on the text features, a first key vector and a first value vector are constructed based on the medical image features, the attention weight from image to text is determined by the first query vector and the first key vector, and the text enhancement features that fuse image information are obtained by combining the first value vector and the attention weight from image to text. A second query vector is constructed based on the medical image features, a second key vector and a second value vector are constructed based on the text features, the text-to-image attention weight is determined through the second query vector and the second key vector, and the image enhancement features fused with text information are obtained by combining the second value vector and the text-to-image attention weight. The text enhancement features of the fused image information are concatenated with the image enhancement features of the fused text information to generate cross-modal fusion features.

8. A pixel-level decoding medical visual question-answering device, characterized in that, The pixel-level decoding medical visual question-answering device includes: The image preprocessing module is used to preprocess the input medical images and query text to generate multimodal medical data; A cross-modal fusion module is used to perform cross-modal alignment processing on the multimodal medical data to generate cross-modal fusion features; The question-answering output module is used to generate a natural language response containing a medical entity and a pixel-level segmentation mask corresponding to the medical entity based on the cross-modal fusion features, thereby forming a multimodal output result.

9. A pixel-level decoding medical visual question-answering device, characterized in that, The device includes: a memory, a processor, and a pixel-level decoding medical visual question-and-answer program stored in the memory and executable on the processor, the pixel-level decoding medical visual question-and-answer program being configured to implement the steps of the pixel-level decoding medical visual question-and-answer method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a pixel-level decoded medical visual question-answering program, which, when executed by a processor, implements the steps of the pixel-level decoded medical visual question-answering method as described in any one of claims 1 to 7.

Images