Medical visual question answering method based on multi-scale feature fusion and question enhancement

Abstract

The invention provides a medical visual question answering method based on multi-scale feature fusion and question enhancement. The method comprises the steps of obtaining a to-be-recognized medical image and a to-be-answered text; inputting the to-be-recognized medical image and the to-be-answered text into a pre-trained medical question and answer model to obtain a detection result about the to-be-answered text; the detection result is a text result; wherein the pre-trained medical question and answer model is provided with a visual extraction model, a text extraction model and a classification model; the visual extraction model is a visual model based on a multi-scale visual feature extraction method, a multi-scale visual feature enhancement method and a multi-scale visual feature fusion method; and a multi-head cross attention feature interaction mechanism is arranged between the visual extraction model and the text extraction model. By combining the visual extraction model, the text extraction model and the classification model in the pre-trained medical question and answer model, the understanding ability of the medical image is improved, and finally the recognition precision and the answer quality of the medical visual question and answer task are integrally improved.

Description

Technical Field

[0001] The present invention relates to the field of medical visual question answering, and particularly relates to a medical visual question answering method based on multi-scale feature fusion and question enhancement. Background Art

[0002] Medical Visual Question Answering (Med-VQA) is an extension of the research on Visual Question Answering (VQA) in natural scenes, and has shown important application value in the field of medical image analysis in recent years. The core goal of Med-VQA is to develop an intelligent model that can extract effective information from complex medical images and closely associate these visual information with natural language questions, so as to provide accurate answers to clinical questions. However, medical images usually contain rich multi-scale features, including local details and global context information, which poses extremely high requirements for the feature extraction ability of the model. In addition, medical questions often involve complex semantic understanding, and the model needs to have strong multi-modal fusion ability. Therefore, how to efficiently extract multi-scale information from medical images and deeply fuse it with the semantics of questions has become a key problem to be solved urgently in the field of Med-VQA.

[0003] Currently, for the Med-VQA task, researchers widely adopt deep learning-based methods for visual feature extraction and multi-modal information fusion. For example, convolutional neural networks (CNNs) such as ResNet50 and Transformer models such as Vision Transformer (ViT) are used to extract features from medical images. These methods can capture some key features in the image through a multi-level network structure, and combine natural language processing techniques to achieve the understanding and answering of questions. In addition, some studies attempt to directly jointly model visual features and question features, and improve the performance of the model through cross-modal attention mechanisms. These methods have promoted the development of Med-VQA to a certain extent and shown good performance potential.

[0004] Although the above methods have made some progress, there are still significant deficiencies in the existing technology for Med-VQA task recognition. First, the existing feature extraction methods are difficult to comprehensively capture the multi-scale information in medical images, resulting in inaccurate understanding of the image content during the task recognition process. Second, in the face of complex multi-modal inputs, the existing methods still lack semantic understanding and context reasoning capabilities, which easily lead to incorrect task positioning and thus affect the final answer quality. These problems seriously restrict the application effect of Med-VQA systems in actual medical scenarios. Summary of the Invention

[0005] To solve the above problems existing in the prior art, the present invention provides a medical visual question answering method based on multi-scale feature fusion and question enhancement.

[0006] The technical problems to be solved by the present invention are realized through the following technical solutions:

[0007] In a first aspect, the present invention provides a medical visual question answering method based on multi-scale feature fusion and question enhancement, including:

[0008] Obtain the medical image to be recognized and the text to be answered;

[0009] Input the medical image to be recognized and the text to be answered into a pre-trained medical question answering model to obtain a detection result regarding the text to be answered; the detection result is a text result;

[0010] Among them, the pre-trained medical question answering model is provided with a visual extraction model, a text extraction model, and a classification model; the visual extraction model is a visual model based on multi-scale visual feature extraction, multi-scale visual feature enhancement, and multi-scale visual feature fusion methods; a feature interaction mechanism of multi-head cross-attention is set between the visual extraction model and the text extraction model.

[0011] Optionally, the visual extraction model includes: a first visual model, a second visual model, and a third visual model;

[0012] The first visual model and the second visual model are provided with an equal number of sub-model modules; the sub-model modules of the first visual model and the sub-model modules of the second visual model are connected in series at corresponding scales;

[0013] The first visual model is used to obtain local feature information of the medical image to be recognized; the second visual model is used to obtain global feature information of the medical image to be recognized; the third visual model is used to remove noise information in the medical image to be recognized.

[0014] Optionally, the first visual model adopts a ViT model; the second visual model adopts a Resnet50 model; the third visual model adopts a convolutional denoising autoencoder CDAE.

[0015] Optionally, inputting the medical image to be recognized and the text to be answered into the pre-trained medical question answering model to obtain a detection result regarding the text to be answered includes:

[0016] Input the medical image to be recognized into the first visual model, the second visual model, and the third visual model, and perform feature extraction in corresponding dimensions according to the sub-model modules of the first visual model and the sub-model modules of the second visual model, and correspondingly obtain a plurality of global features, a plurality of local features, and convolutional denoising information;

[0017] Feature enhancement processing is performed on multiple global features and multiple local features respectively, and multiple global enhanced features and multiple local enhanced features are correspondingly obtained;

[0018] Feature fusion processing is performed on multiple global enhanced features and multiple local enhanced features according to the dimensions of the corresponding sub-model modules to obtain initial fusion features;

[0019] Feature splicing processing is performed on the initial fusion features and convolutional denoising information to obtain final visual fusion features;

[0020] The text to be answered is input into the text extraction model to obtain the text of the task to be recognized;

[0021] Feature interaction processing is performed on the final visual fusion features and the text of the task to be recognized by using the multi-head cross-attention mechanism to obtain the final text features to be classified with visual representations;

[0022] The final text features to be classified are input into the classification model to obtain the detection result.

[0023] Optionally, feature enhancement processing is performed on multiple global features and multiple local features respectively, and multiple global enhanced features and multiple local enhanced features are correspondingly obtained, including:

[0024] The atrous convolution processing method is used to perform local feature enhancement processing on multiple global features to obtain multiple global enhanced features;

[0025] The cross-window multi-head attention mechanism is used to perform global feature enhancement processing on multiple local features to obtain multiple local enhanced features.

[0026] Optionally, feature fusion processing is performed on multiple global enhanced features and multiple local enhanced features according to the dimensions of the corresponding sub-model modules to obtain initial fusion features, including:

[0027] Dimensionality reduction processing is performed on multiple global enhanced features and multiple local enhanced features to obtain multiple globally enhanced features after dimensionality reduction and multiple locally enhanced features after dimensionality reduction;

[0028] Feature fusion processing is performed on multiple globally enhanced features after dimensionality reduction and multiple locally enhanced features after dimensionality reduction according to the dimensions of the corresponding sub-model modules to obtain sub-model dimensional fusion features;

[0029] Matrix fusion processing is performed on all sub-model dimensional fusion features to obtain initial fusion features.

[0030] Optionally, inputting the text to be answered into the text extraction model to obtain the text of the task to be recognized, including:

[0031] Map each word in the text to be answered to the corresponding word vector in the word embedding sequence pre-trained by GloVe to obtain the encoded text features;

[0032] Use the encoded text features with the QCR text method to perform task information extraction processing to obtain the text of the task to be recognized.

[0033] Optionally, the training process of the pre-trained medical Q&A model includes:

[0034] Obtain medical image samples and text samples;

[0035] Input the medical image samples and text samples into the initial medical Q&A model, and continuously train the initial medical Q&A model in the direction of the value of the loss function decreasing;

[0036] Take the initial medical Q&A model that meets the preset stop condition as the pre-trained medical Q&A model;

[0037] The preset stop condition includes: the number of iterations meets the preset iteration threshold or the value of the loss function is continuously less than the loss threshold;

[0038] Among them, the initial medical Q&A model and the pre-trained medical Q&A model have the same structure.

[0039] In a second aspect, the present invention provides a medical visual Q&A device based on multi-scale feature fusion and question enhancement. The medical visual Q&A device based on multi-scale feature fusion and question enhancement includes: an acquisition unit and a detection unit;

[0040] The acquisition unit is used to: acquire the medical image to be recognized and the text to be answered;

[0041] The detection unit is used to: input the medical image to be recognized and the text to be answered into the pre-trained medical Q&A model to obtain the detection result regarding the text to be answered; the detection result is a text result;

[0042] Among them, the pre-trained medical Q&A model is provided with a visual extraction model, a text extraction model, and a classification model; the visual extraction model is a visual model based on multi-scale visual feature extraction, multi-scale visual feature enhancement, and multi-scale visual feature fusion methods; a feature interaction mechanism of multi-head cross-attention is set between the visual extraction model and the text extraction model.

[0043] In a third aspect, the present invention provides a medical visual question-answering device based on multi-scale feature fusion and question enhancement, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, when the medical visual question-answering device based on multi-scale feature fusion and question enhancement is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the medical visual question-answering method based on multi-scale feature fusion and question enhancement as described in the first aspect above.

[0044] The present invention provides a medical visual question-answering method based on multi-scale feature fusion and question enhancement, comprising: obtaining a medical image to be identified and a text to be answered; inputting the medical image to be identified and the text to be answered into a pre-trained medical question-answering model to obtain a detection result about the text to be answered; the detection result is a text result; wherein the pre-trained medical question-answering model is provided with a visual extraction model, a text extraction model and a classification model; the visual extraction model is a visual model based on multi-scale visual feature extraction, multi-scale visual feature enhancement and multi-scale visual feature fusion methods; a feature interaction mechanism of multi-head cross attention is provided between the visual extraction model and the text extraction model. In the present invention, firstly, by introducing a visual extraction model based on multi-scale visual feature extraction, multi-scale visual feature enhancement and multi-scale visual feature fusion methods, the problem that the existing feature extraction method is difficult to fully capture the multi-scale information in the medical image is solved. It can more carefully analyze and understand the details of different scales and levels in the medical image, improve the accuracy of the image content understanding, and thus ensure the comprehensive grasp of the image information in the task recognition process. Secondly, the present invention sets a feature interaction mechanism of multi-head cross attention between the visual extraction model and the text extraction model. This mechanism enhances the model's ability in semantic understanding and contextual reasoning, allowing the model to better understand the relationship between images and texts, and avoiding task positioning errors caused by insufficient semantic understanding or contextual reasoning errors. This improvement improves the accuracy of task positioning, thereby improving the quality of the final answer. Finally, by combining the visual extraction model, text extraction model, and classification model in the pre-trained medical question-answering model, the present invention achieves efficient and accurate processing of medical visual question-answering tasks. It not only improves the ability to understand medical images, but also enhances the interaction and reasoning capabilities between text and images, thereby improving the recognition accuracy and answer quality of medical visual question-answering tasks as a whole.

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a medical visual question answering method based on multi-scale feature fusion and question enhancement provided by an embodiment of the present invention;

[0047] Figure 2 Schematic diagrams of the first visual model and the second visual model are exemplarily shown;

[0048] Figure 3 Schematic diagram of a medical visual question answering device based on multi-scale feature fusion and question enhancement provided by an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of a medical visual question answering device based on multi-scale feature fusion and question enhancement provided by an embodiment of the present invention. Detailed implementation manners

[0050] The following further describes the present invention in detail with reference to specific embodiments, but the implementation manners of the present invention are not limited thereto.

[0051] In order to improve the recognition accuracy and answer quality of the medical visual question answering task as a whole, an embodiment of the present invention provides a medical visual question answering method based on multi-scale feature fusion and question enhancement. Figure 1 Flow chart of a medical visual question answering method based on multi-scale feature fusion and question enhancement provided by an embodiment of the present invention. As Figure 1 shown, it includes:

[0052] S101. Obtain a medical image to be recognized and a text to be answered.

[0053] It should be noted that in the embodiment of the present invention, the medical image to be recognized may be medical imaging data that needs to be analyzed, recognized or diagnosed. These images usually come from medical devices such as X-ray machines, CT scanners, MRI scanners, ultrasonic devices, etc., and are used to capture the internal structure, function or abnormalities of the human body. The medical image to be recognized may include but is not limited to the following types: X-ray films, CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images.

[0054] The text to be answered is a question or statement related to the above-mentioned medical image to be recognized, and it requires a pre-trained medical question answering model to provide an answer or explanation based on the provided image information. The text to be answered may be a question raised by a patient, doctor or medical professional regarding visible features in the image, disease diagnosis, treatment plan suggestions, etc. For example: According to the MRI image, is there any abnormality in the patient's brain? Where is the fracture shown in this CT scan?

[0055] S102. Input the medical image to be recognized and the text to be answered into a pre-trained medical question answering model to obtain a detection result regarding the text to be answered.

[0056] Among them, the detection results are text results; the pre-trained medical question-answering model is equipped with a visual extraction model, a text extraction model and a classification model; the visual extraction model is a visual model based on multi-scale visual feature extraction, multi-scale visual feature enhancement and multi-scale visual feature fusion methods; a multi-head cross-attention feature interaction mechanism is set between the visual extraction model and the text extraction model.

[0057] The embodiment of the present invention provides a medical visual question answering method based on multi-scale feature fusion and question enhancement. First, by introducing a visual extraction model based on multi-scale visual feature extraction, multi-scale visual feature enhancement and multi-scale visual feature fusion methods, the problem that the existing feature extraction method is difficult to fully capture the multi-scale information in medical images is solved. It can analyze and understand the details of different scales and levels in medical images more carefully, improve the accuracy of image content understanding, and thus ensure the comprehensive grasp of image information in the task recognition process. Secondly, the present invention sets a multi-head cross-attention feature interaction mechanism between the visual extraction model and the text extraction model. This mechanism enhances the model's ability in semantic understanding and contextual reasoning, so that the model can better understand the relationship between images and texts, and avoid task positioning errors caused by insufficient semantic understanding or contextual reasoning errors. This improvement improves the accuracy of task positioning, thereby improving the quality of the final answer. Finally, by combining the visual extraction model, text extraction model and classification model in the pre-trained medical question answering model, the present invention realizes efficient and accurate processing of medical visual question answering tasks. It not only improves the ability to understand medical images, but also enhances the interaction and reasoning capabilities between text and images, thereby improving the recognition accuracy and answer quality of medical visual question-answering tasks as a whole.

[0058] Optionally, the visual extraction model includes: a first visual model, a second visual model and a third visual model;

[0059] The first visual model and the second visual model are provided with an equal number of sub-model modules; the sub-model modules of the first visual model and the sub-model modules of the second visual model are connected in series at corresponding scales;

[0060] The first visual model is used to obtain local feature information of the medical image to be identified; the second visual model is used to obtain global feature information of the medical image to be identified; and the third visual model is used to remove noise information in the medical image to be identified.

[0061] In order to clearly show the connection relationship between the first visual model and the second visual model provided by the embodiment of the present invention, Figure 2 The schematic diagram of the structure of the first visual model and the second visual model is exemplarily shown. Figure 2As shown, both the first visual model and the second visual model are divided into 4 sub-model modules, and corresponding image encoding modules (Patch Emdeding and conv1) are set before the first visual model and the second visual model to convert the image into a vector for subsequent model feature extraction. Each sub-model module is connected in series at the corresponding scale. The features X1 and Y1 obtained by corresponding Vision Transformer Block1 and Residual Block1 are added to get GLFFM1. The features X2 and Y2 obtained by corresponding Vision Transformer Block2 and Residual Block2 are added to get GLFFM2. The features X3 and Y3 obtained by corresponding Vision Transformer Block3 and Residual Block3 are added to get GLFFM3. The features X4 and Y4 obtained by corresponding Vision Transformer Block4 and Residual Block4 are added to get GLFFM4. Fuse1 corresponds to be equal to GLFFM1, fuse2 corresponds to be equal to the sum of fuse1 and GLFFM2, fuse3 corresponds to be equal to the sum of GLFFM3 and fuse2, and fuse4 corresponds to be equal to the sum of GLFFM4 and fuse3. Fuse4 is the final initial fusion feature.

[0062] Optionally, the first visual model adopts a ViT model; the second visual model adopts a Resnet50 model; the third visual model adopts a convolutional denoising autoencoder CDAE.

[0063] Optionally, the medical image to be recognized and the text to be answered are input into a pre-trained medical question-answering model to obtain the detection result for the text to be answered, including:

[0064] The medical image to be recognized is input into the first visual model, the second visual model, and the third visual model, and feature extraction is performed according to the sub-model modules of the first visual model and the second visual model in the corresponding dimensions, corresponding to obtain a plurality of global features, a plurality of local features, and convolutional denoising information;

[0065] Feature enhancement processing is respectively performed on the plurality of global features and the plurality of local features to correspondingly obtain a plurality of global enhanced features and a plurality of local enhanced features;

[0066] The plurality of global enhanced features and the plurality of local enhanced features are subjected to feature fusion processing according to the dimensions of the corresponding sub-model modules to obtain an initial fusion feature;

[0067] The initial fusion feature and the convolutional denoising information are subjected to feature splicing processing to obtain a final visual fusion feature;

[0068] Input the text to be answered into the text extraction model to obtain the text of the task to be recognized;

[0069] Perform feature interaction processing on the final visual fusion feature and the text of the task to be recognized by using the multi-head cross-attention mechanism to obtain the final text feature to be classified with visual representation;

[0070] Input the final text feature to be classified into the classification model to obtain the detection result.

[0071] Optionally, perform feature enhancement processing on multiple global features and multiple local features respectively to obtain multiple global enhanced features and multiple local enhanced features, including:

[0072] Adopt the dilated convolution processing method to perform local feature enhancement processing on multiple global features to obtain multiple global enhanced features;

[0073] Adopt the cross-window multi-head attention mechanism to perform global feature enhancement processing on multiple local features to obtain multiple local enhanced features.

[0074] Optionally, perform feature fusion processing on multiple global enhanced features and multiple local enhanced features according to the dimensions of the corresponding sub-model modules to obtain the initial fusion feature, including:

[0075] Perform dimensionality reduction processing on multiple global enhanced features and multiple local enhanced features to obtain multiple globally enhanced features after dimensionality reduction and multiple locally enhanced features after dimensionality reduction;

[0076] Perform feature fusion processing on multiple globally enhanced features after dimensionality reduction and multiple locally enhanced features after dimensionality reduction according to the dimensions of the corresponding sub-model modules to obtain the sub-model dimension fusion feature;

[0077] Perform matrix fusion processing on all the sub-model dimension fusion features to obtain the initial fusion feature.

[0078] Optionally, input the text to be answered into the text extraction model to obtain the text of the task to be recognized, including:

[0079] Map each word in the text to be answered to the word vector corresponding to the word embedding sequence pre-trained by GloVe to obtain the encoded text feature;

[0080] Use the encoded text feature with QCR text method to perform task information derivation processing to obtain the text of the task to be recognized.

[0081] It should be noted that QCR textThe execution of the method specifically refers to the relevant content in the document "Medical visual questionanswering via conditional reasoning".

[0082] Optionally, the training process of the pre-trained medical Q&A model includes:

[0083] Obtain medical image samples and text samples;

[0084] Input the medical image samples and text samples into the initial medical Q&A model, and continuously train the initial medical Q&A model in the direction of the value of the loss function decreasing;

[0085] Use the initial medical Q&A model that meets the preset stop condition as the pre-trained medical Q&A model;

[0086] The preset stop condition includes: the number of iterations meets the preset iteration threshold or the value of the loss function is continuously less than the loss threshold;

[0087] Among them, the structures of the initial medical Q&A model and the pre-trained medical Q&A model are the same.

[0088] It should be noted that in the embodiments of the present invention, the loss function is the sum of the cross-entropy loss function and the reconstruction loss function.

[0089] In summary, the embodiments of the present invention propose a medical visual question answering method based on multi-scale feature fusion, which solves the limitations of traditional single-scale visual feature extraction in dealing with complex medical scenarios. Different from traditional methods, the present invention combines local and global visual features, and proposes a new multi-scale visual feature extraction and fusion method, so as to more comprehensively capture the multi-level spatial semantic information in medical images. Especially when dealing with complex medical problems, it can effectively improve the understanding ability of the model. In addition, the present invention also proposes an improved visual-text feature fusion method to fully explore the optimization effect of visual information on text features. In the traditional bilinear attention mechanism, the question features and visual features are often directly combined for modeling, but the potential of visual information in optimizing text features has not been deeply explored. The present invention introduces a new visual-language information interaction mechanism, so that visual information can not only strengthen the expression of question features, but also further optimize the representation of text features, thereby improving the model's ability to answer clinical questions.

[0090] The technical solution of the present invention effectively overcomes the deficiencies of traditional methods, and improves the performance of the medical visual question answering task through multi-scale feature extraction and fusion and the optimization of text features by visual information.

[0091] The method provided by the embodiments of the present invention can be applied to an electronic device. Specifically, the electronic device can be: a desktop computer, a portable computer, a smart mobile terminal, a server, etc., which is not limited in the embodiments of the present invention.

[0092] Based on the same inventive concept, the embodiments of the present invention further provide a medical visual question answering device based on multi-scale feature fusion and question enhancement. Figure 3 As shown in the structural schematic diagram of a medical visual question answering device based on multi-scale feature fusion and question enhancement provided by the embodiments of the present invention, Figure 3 as shown, it includes: an acquisition unit 301 and a detection unit 302;

[0093] The acquisition unit 301 is used for: acquiring a medical image to be recognized and a text to be answered;

[0094] The detection unit 302 is used for: inputting the medical image to be recognized and the text to be answered into a pre-trained medical question answering model to obtain a detection result regarding the text to be answered; the detection result is a text result;

[0095] Among them, the pre-trained medical question answering model is provided with a visual extraction model, a text extraction model, and a classification model; the visual extraction model is a visual model based on multi-scale visual feature extraction, multi-scale visual feature enhancement, and multi-scale visual feature fusion methods; a feature interaction mechanism of multi-head cross-attention is set between the visual extraction model and the text extraction model.

[0096] Figure 4 As shown in the structural schematic diagram of a medical visual question answering device based on multi-scale feature fusion and question enhancement provided by the embodiments of the present invention, it includes: a processor 710, a storage medium 720, and a bus 730. The storage medium 720 stores machine-readable instructions executable by the processor 710. When the medical visual question answering device based on multi-scale feature fusion and question enhancement runs, the processor 710 communicates with the storage medium 720 through the bus 730, and the processor 710 executes the machine-readable instructions to execute the steps of the above method embodiments. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0097] The storage medium may include a random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the storage medium may also be at least one storage device located away from the aforementioned processor.

[0098] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0099] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0100] Although the present invention has been described in connection with various embodiments herein, however, in the process of implementing the claimed present invention, those skilled in the art can understand and achieve other variations of the above-described disclosed embodiments by viewing the drawings and the disclosure. In the description of the present invention, the term "including" does not exclude other components or steps, the term "a" or "one" does not exclude a plurality of cases, and the meaning of "a plurality" is two or more unless otherwise clearly and specifically defined. In addition, certain measures are recited in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0101] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A medical visual question answering method based on multi-scale feature fusion and problem enhancement, characterized in that Including: Obtain the medical image to be recognized and the text to be answered; Input the medical image to be recognized and the text to be answered into a pre-trained medical question-answering model to obtain a detection result regarding the text to be answered; the detection result is a text result; Among them, the pre-trained medical question-answering model is provided with a visual extraction model, a text extraction model, and a classification model; the visual extraction model is a visual model based on multi-scale visual feature extraction, multi-scale visual feature enhancement, and multi-scale visual feature fusion methods; a feature interaction mechanism of multi-head cross-attention is set between the visual extraction model and the text extraction model.

2. The medical visual question answering method based on multi-scale feature fusion and problem enhancement according to claim 1, characterized in that, The visual extraction model includes: a first visual model, a second visual model, and a third visual model; The first visual model and the second visual model are provided with an equal number of sub-model modules; the sub-model modules of the first visual model and the sub-model modules of the second visual model are connected in series at corresponding scales; The first visual model is used to obtain local feature information of the medical image to be recognized; the second visual model is used to obtain global feature information of the medical image to be recognized; the third visual model is used to remove noise information in the medical image to be recognized.

3. The medical visual question answering method based on multi-scale feature fusion and problem enhancement according to claim 2, wherein The first visual model adopts a ViT model; the second visual model adopts a Resnet50 model; the third visual model adopts a convolutional denoising autoencoder CDAE.

4. The medical visual question answering method based on multi-scale feature fusion and problem enhancement according to claim 2, characterized in that The step of inputting the medical image to be recognized and the text to be answered into the pre-trained medical question-answering model to obtain a detection result regarding the text to be answered includes: Input the medical image to be recognized into the first visual model, the second visual model, and the third visual model, and perform feature extraction according to the sub-model modules of the first visual model and the sub-model modules of the second visual model in corresponding dimensions, respectively obtaining a plurality of global features, a plurality of local features, and convolutional denoising information; Perform feature enhancement processing on the plurality of global features and the plurality of local features respectively, obtaining a plurality of global enhanced features and a plurality of local enhanced features correspondingly; Perform feature fusion processing on the plurality of global enhanced features and the plurality of local enhanced features according to the dimensions of the corresponding sub-model modules to obtain an initial fusion feature; Perform feature splicing processing on the initial fusion feature and the convolutional denoising information to obtain a final visual fusion feature; Input the text to be answered into the text extraction model to obtain a text to be recognized task; Perform feature interaction processing on the final visual fusion feature and the text to be recognized task by using a multi-head cross-attention mechanism to obtain a final text feature to be classified with visual representation; Input the final text feature to be classified into the classification model to obtain the detection result.

5. The medical visual question answering method based on multi-scale feature fusion and question enhancement according to claim 4, wherein, The step of performing feature enhancement processing on the plurality of global features and the plurality of local features respectively, obtaining a plurality of global enhanced features and a plurality of local enhanced features correspondingly, includes: Adopt a dilated convolution processing method to perform local feature enhancement processing on the plurality of global features to obtain the plurality of global enhanced features; Perform global feature enhancement processing on the multiple local features by using a cross-window multi-head attention mechanism to obtain the multiple locally enhanced features.

6. The medical visual question answering method based on multi-scale feature fusion and question enhancement according to claim 4, wherein The performing feature fusion processing on the multiple globally enhanced features and the multiple locally enhanced features according to the dimensions of the corresponding sub-model modules to obtain initial fusion features includes: Perform dimensionality reduction processing on the multiple globally enhanced features and the multiple locally enhanced features to obtain multiple dimensionally reduced globally enhanced features and multiple dimensionally reduced locally enhanced features; Perform feature fusion processing on the multiple dimensionally reduced globally enhanced features and the multiple dimensionally reduced locally enhanced features according to the dimensions of the corresponding sub-model modules to obtain sub-model dimension fusion features; Perform matrix fusion processing on all the sub-model dimension fusion features to obtain the initial fusion features.

7. The medical visual question answering method based on multi-scale feature fusion and question enhancement according to claim 4, wherein The inputting the to-be-answered text into the text extraction model to obtain a to-be-identified task text includes: Map each word in the to-be-answered text to a word vector corresponding to the word embedding sequence pre-trained by GloVe to obtain encoded text features; Use the QCR to process the encoded text features text to export task information and obtain the text of the task to be recognized.

8. The medical visual question answering method based on multi-scale feature fusion and problem enhancement according to claim 1, characterized in that, The training process of the pre-trained medical question answering model includes: Obtain medical image samples and text samples; Input the medical image samples and the text samples into an initial medical question answering model, and continuously train the initial medical question answering model in the direction of the value of the loss function decreasing; Use the initial medical question answering model that meets the preset stop condition as the pre-trained medical question answering model; The preset stop condition includes: the number of iterations meets a preset iteration threshold or the value of the loss function is continuously less than a loss threshold; Wherein, the initial medical question answering model has the same structure as the pre-trained medical question answering model.

9. A medical visual question answering device based on multi-scale feature fusion and problem enhancement, characterized in that The medical visual question answering device based on multi-scale feature fusion and question enhancement includes: an acquisition unit and a detection unit; The acquisition unit is used to: acquire a to-be-identified medical image and a to-be-answered text; The detection unit is used to: input the to-be-identified medical image and the to-be-answered text into a pre-trained medical question answering model to obtain a detection result regarding the to-be-answered text; the detection result is a text result; Wherein, the pre-trained medical question answering model is provided with a visual extraction model, a text extraction model, and a classification model; the visual extraction model is a visual model based on multi-scale visual feature extraction, multi-scale visual feature enhancement, and multi-scale visual feature fusion methods; a feature interaction mechanism with multi-head cross-attention is arranged between the visual extraction model and the text extraction model.

10. A medical visual question answering device based on multi-scale feature fusion and question enhancement, characterized in that Includes: A processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the medical visual question answering device based on multi-scale feature fusion and question enhancement runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the medical visual question answering method according to any one of claims 1-8.

Images