A medical question and answer method fusing cross-modal mixed experts

Abstract

This invention provides a medical question-answering method integrating cross-modal hybrid experts. The method includes: acquiring medical images and question text, and preprocessing them to obtain initial visual embedding matrices and initial text embedding matrices; inputting the initial visual embedding matrices and initial text embedding matrices into a trained medical question-answering model to obtain answer text for the medical images and question text; the medical question-answering model includes a visual encoder, a text encoder, a positional encoding module, a feature fusion module, a gated router, a hybrid expert module, and an answer generator. It utilizes a cross-modal Transformer to achieve deep semantic alignment of image and text features, and dynamically schedules multiple expert networks through a gated router to perform specialized processing on different feature vectors, enhancing the model's adaptability to complex lesions and rare cases; it performs multi-angle reasoning to generate accurate and reliable answers, providing an efficient and reliable solution for assisted diagnosis, medical training, and scientific research analysis.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a medical question-and-answer method that integrates cross-modal hybrid experts. Background Technology

[0002] Medical visual question answering aims to combine medical images (such as CT, MRI, and X-rays) with clinical text information (such as case descriptions, laboratory reports, and doctor's questions) to automatically generate accurate and interpretable diagnostic or interpretive answers. It is an important research direction for intelligent assisted diagnosis and clinical decision support.

[0003] Based on medical visual question answering, many applications can be realized. For example: (1) Clinical auxiliary diagnosis platform: In scenarios such as tumor staging, vascular lesions, and lung infections, doctors expect the system to answer difficult questions such as "Is there any abnormality in this area of ​​the image?" and "Location and nature of foreign objects" in real time; (2) Medical education and residency training system: In order to improve the image interpretation ability of resident physicians and radiology interns, the platform needs to support multimodal question answering exercises and provide automatic comparison, difference analysis and feedback with expert standard answers; (3) Multicenter scientific research data analysis: For large-scale clinical images and electronic medical record data, researchers need high-throughput automatic question answering tools to quickly extract statistical features such as lesion distribution and complication association; (4) Telemedicine and multidisciplinary collaboration: In resource-scarce areas, doctors upload images and questions through mobile devices. The system needs to ensure multimodal semantic consistency and realize interactive diagnostic support across regions and professions.

[0004] Currently, medical visual question answering often employs two-stream fusion models, pre-trained multimodal model transfer, or knowledge graph-based or rule-enhanced approaches. However, these methods still face numerous challenges in clinical applications, such as how to reliably fuse features from medical images and question text, and how to more accurately reason and process the fused features. These factors directly impact the accuracy of the response. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides a medical question-answering method that integrates cross-modal hybrid experts to solve the technical problem that the answers in the prior art are not accurate enough.

[0006] To achieve the above and other related objectives, this invention provides a medical question-answering method integrating cross-modal hybrid experts, comprising: acquiring medical images and question text, and preprocessing them to obtain an initial visual embedding matrix and an initial text embedding matrix; inputting the initial visual embedding matrix and the initial text embedding matrix into a trained medical question-answering model to obtain answer text for the medical images and the question text, wherein the medical question-answering model includes: a visual encoder for encoding the initial visual embedding matrix to extract visual feature sequences; a text encoder for encoding the initial text embedding matrix to extract text feature sequences; and a positional encoding module for encoding the visual feature sequences. The system includes: a column and a text feature sequence with added positional encoding information; a feature fusion module, which uses a cross-modal Transformer architecture to perform deep interactive fusion of the visual feature sequence and the text feature sequence after adding positional encoding to obtain a fused feature sequence; a gated router, which calculates the routing probability of the feature vectors at each position in the fused feature sequence being assigned to each expert module; a hybrid expert module, which includes multiple independent expert networks, which performs weighted processing and aggregation on each feature vector in the fused feature sequence according to the routing probability, and outputs an enhanced fused feature sequence; and an answer generator, which generates the answer text autoregressively based on the enhanced fused feature sequence.

[0007] In one embodiment of the present invention, acquiring a medical image and performing preprocessing to obtain an initial visual embedding matrix includes: converting the medical image to a preset size; dividing the resized medical image into N image blocks of size P×P; flattening each image block and linearly projecting it onto a D-dimensional embedding space to obtain the initial visual embedding matrix.

[0008] In one embodiment of the present invention, obtaining the question text and performing preprocessing to obtain an initial text embedding matrix includes: using a word segmenter to segment the question text into M tokens; mapping each token to a D-dimensional embedding space to obtain the initial text embedding matrix.

[0009] In one embodiment of the present invention, both the visual encoder and the text encoder are built based on the Transformer architecture.

[0010] In one embodiment of the present invention, a cross-modal Transformer architecture is employed to perform deep interactive fusion of the visual feature sequence and the text feature sequence after adding positional encoding to obtain a fused feature sequence. This includes: concatenating the visual feature sequence and the text feature sequence after adding positional encoding to obtain a concatenated sequence; inputting the concatenated sequence into a K-layer cross-modal Transformer encoder, with each layer performing the following operations: calculating multi-head self-attention within the sequence to capture cross-modal dependencies; performing nonlinear transformation through a feedforward network; applying residual connections and layer normalization to stabilize training; and after K-layer processing, using the output of the last layer as the fused feature sequence.

[0011] In one embodiment of the present invention, the gated router calculates the routing probability as follows: for the fused feature sequence X∈R L×D Perform a linear transformation: Z = XW g +b g In the formula, W g ∈R D×E It is the weight matrix, b g ∈R E The bias term is L, the sequence length is E, and the total number of expert modules is E. Softmax normalization is applied to Z along the expert dimension to obtain the routing probability matrix S: S = Softmax(Z, dim=1), where the elements S in the routing probability matrix S are... i,e Let be the routing probability that the i-th feature vector in the fused feature sequence is assigned to the e-th expert module, i∈{1,2,…,L}, e∈{1,2,…,E}.

[0012] In one embodiment of the present invention, the expert module consists of a small Transformer or MLP network; the hybrid expert module calculates the enhanced fused feature sequence as follows: for the i-th feature vector x in the fused feature sequence i Calculate the processing results of each expert module: In the formula, E e (x i ) represents the e-th expert module's pair of feature vectors x i The processing results are used to weight and aggregate the processing results using the routing probabilities to obtain the output vector at that location. MSA represents multi-head attention, and FFN represents feedforward network. : Iterate through all positions in the sequence and output a vector. By sequentially concatenating the sequences, the enhanced fusion feature sequence is obtained.

[0013] In one embodiment of the present invention, the answer generator generates the answer text according to the following steps: initializing a lexical sequence containing only a start symbol; for the current decoding time step, performing the following steps: inputting the currently generated lexical sequence into the Transformer decoder, and using the enhanced fused feature sequence as the key and value, calculating the encoder-decoder attention to obtain the decoder hidden state fused with multimodal context information; calculating the probability distribution on the vocabulary according to the decoder hidden state; selecting the next lexical according to the probability distribution and appending it to the generated lexical sequence; stopping the loop when the generated lexical is the end symbol, and converting the generated lexical sequence into the answer text.

[0014] In one embodiment of the present invention, the expression for the loss function of the medical question-answering model during training is as follows: L total =λ1L qa +λ2L align +λ3L consist +βL aux In the formula, λ1, λ2, λ3, and β are the weights of each loss term, and L aux L is an auxiliary loss used to constrain the uniformity of the allocation probabilities among experts. qa For the cross-entropy loss used to measure the accuracy of answer prediction, L align For the contrast loss used to narrow down the distance between positive sample image-text pairs, L consist This is the loss used to supervise the consistency between the attention distribution and the target distribution of the model.

[0015] In one embodiment of the present invention, the formula for calculating the auxiliary loss is as follows: In the formula, the average distribution fraction f e Within a training batch, the routing probability S for all feature vectors being assigned to the e-th expert module. i,e The average value of α is a hyperparameter used to control the load balancing intensity.

[0016] The beneficial effects of this invention are as follows: This invention proposes a medical question-answering method that integrates cross-modal hybrid experts. Firstly, it utilizes a cross-modal Transformer to achieve deep semantic alignment of image and text features, effectively bridging the modal gap. Secondly, it dynamically schedules multiple expert networks through a gated router, performing specialized processing on different feature vectors to achieve fine-grained focusing on lesion regions and medical terminology, enhancing the model's adaptability to complex lesions and rare cases. Finally, the model can perform multi-angle reasoning like a clinical expert, generating accurate and reliable answers, providing an efficient and reliable intelligent solution for assisted diagnosis, medical training, and scientific research analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The accompanying drawings are incorporated in and constitute a part of this specification, illustrating embodiments consistent with this application, and are used together with the description to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a medical question-and-answer method provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of medical image preprocessing provided in an embodiment of the present invention;

[0020] Figure 3 This is a flowchart of a problem text preprocessing method provided in an embodiment of the present invention;

[0021] Figure 4 An architectural diagram of a medical question-answering model provided in an embodiment of the present invention;

[0022] Figure 5 This is a flowchart of the feature fusion module provided in an embodiment of the present invention;

[0023] Figure 6 This is a flowchart of the processing of a gated router provided in an embodiment of the present invention;

[0024] Figure 7 A flowchart illustrating the processing of a hybrid expert module according to an embodiment of the present invention;

[0025] Figure 8 This is a flowchart of the answer generator provided in an embodiment of the present invention. Detailed Implementation

[0026] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other. In addition to the specific methods, equipment, and materials used in the embodiments, based on the knowledge of the prior art and the description of the present invention by those skilled in the art, any prior art methods, equipment, and materials similar to or equivalent to the methods, equipment, and materials in the embodiments of the present invention can be used to implement the present invention.

[0027] It should be understood that the terminology used in the embodiments of this invention is for describing specific implementations and not for limiting the scope of protection of this invention. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art.

[0028] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In some embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0029] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations that may be implemented in the methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0030] Please see Figure 1 , Figure 1 A medical question-answering method integrating cross-modal hybrid experts is provided as an embodiment of the present invention, including steps S101 and S102.

[0031] Step S101: Obtain medical images and problem text, and perform preprocessing to obtain the initial visual embedding matrix and the initial text embedding matrix.

[0032] Please see Figure 2In a specific embodiment of the present invention, medical images are acquired and preprocessed to obtain an initial visual embedding matrix, including: S201, converting the medical images to a preset size, such as 224×224 resolution; S202, dividing the resized medical images into N image blocks of size P×P; S203, flattening each image block and linearly projecting it onto a D-dimensional embedding space to obtain the initial visual embedding matrix. This preprocessing step ensures input consistency by standardizing image sizes and transforms high-resolution images into structured sequence data. Image segmentation and linear projection efficiently encode local visual information (such as texture and edges) into dense vectors, laying a solid foundation for the subsequent visual encoder to capture global dependencies and greatly improving the model's ability to extract key lesion features from medical images.

[0033] Please see Figure 3 In a specific embodiment of the present invention, the question text is acquired and preprocessed to obtain an initial text embedding matrix, including: S301, using a tokenizer to segment the question text into M tokens; S302, mapping each token to a D-dimensional embedding space to obtain the initial text embedding matrix. This preprocessing step deconstructs the medical question into semantic units through tokenization, and then transforms it into a numerical representation rich in semantic information through embedding mapping. This process not only preserves the integrity of professional terminology, but also transforms the abstract medical question into a continuous vector space that the model can process, providing a high-quality text feature foundation for subsequent cross-modal alignment and ensuring the model's accurate understanding of complex medical questions.

[0034] Step S102: Input the initial visual embedding matrix and the initial text embedding matrix into the trained medical question answering model to obtain the answer text for the medical image and the question text.

[0035] In this step, the training of the medical question-answering model is consistent with that of the conventional model, and it can be roughly divided into the following steps: (1) constructing a dataset; (2) constructing a medical question-answering model; (3) using the dataset to train and validate the medical question-answering model. In this way, a well-trained medical question-answering model can be obtained.

[0036] Please see Figure 4In this step, the specific structure of the medical question-answering model is a significant innovation of this invention. It includes a visual encoder, a text encoder, a positional encoding module, a feature fusion module, a gated router, a hybrid expert module, and an answer generator. Specifically, the visual encoder encodes the initial visual embedding matrix to extract visual feature sequences; the text encoder encodes the initial text embedding matrix to extract text feature sequences; the positional encoding module adds positional encoding information to the visual and text feature sequences; the feature fusion module uses a cross-modal Transformer architecture to perform deep interactive fusion of the positionally encoded visual and text feature sequences to obtain a fused feature sequence; the gated router calculates the routing probability of the feature vectors at each position in the fused feature sequence to each expert module; the hybrid expert module includes multiple independent expert networks, which weight and aggregate each feature vector in the fused feature sequence according to the routing probability, outputting an enhanced fused feature sequence; and the answer generator generates answer text autoregressively based on the enhanced fused feature sequence. These modules are described in detail below.

[0037] In one specific embodiment of the present invention, both the visual encoder and the text encoder are built based on the Transformer architecture.

[0038] Taking a visual encoder as an example, it is typically a Vision Transformer or a Swing Transformer. In a multi-layer Transformer architecture, each layer k processes the input X from the layer above. k-1 and output the result X. k Each layer's processing consists of two main parts: multi-head self-attention (MSA) and a feedforward network (FFN). In multi-head self-attention, the input X is processed by... k-1 After performing Layer Normalization (LN) processing, calculate the query Q, key K, and value V:

[0039] , , ,

[0040] In the formula, W Q W K W V This is the learned weight matrix. Next, the attention score is calculated:

[0041] ,

[0042] Finally, the output is:

[0043] ,

[0044] In the formula, Z k The output of the multi-head self-attention method combines a residual connection (skip connection) with the input X. k-1 Add them together. This process helps capture global dependencies in an image, such as the relationship between edges and textures.

[0045] After obtaining the multi-head self-attention output Z k Then, layer normalization is performed, followed by further processing using a feedforward neural network (FFN). The formula for calculating FFN is:

[0046] ,

[0047] In the formula, W1, W2, b1, and b2 are learnable network parameters, and GeLU is the activation function. The final output is:

[0048] ,

[0049] X k For serialized visual features, the vectors represent multi-scale information. After pooling to aggregate the global representation, we use the visual feature sequence output from the last layer as the output of the visual encoder, denoted as V∈R. N×D , where N represents visual length and D represents embedding dimension.

[0050] For text encoders, which are typically based on the BERT model, the processing is similar to that of visual encoders. After multiple layers of Transformer encoding, the output features T∈R of the text encoder are... M×D is a serialized high-dimensional semantic vector, where M represents the text length and D represents the embedding dimension.

[0051] In a specific embodiment of the present invention, in order to preserve the order information of elements in the sequence, positional encoding is added to the input visual feature sequence V and text feature sequence T. Positional encoding enhances the model's order awareness ability by introducing an additional set of vectors to represent the relative or absolute position of each token in the sequence. This can be expressed by the formula:

[0052] , .

[0053] Please see Figure 5 The system employs a cross-modal Transformer architecture to perform deep interactive fusion of the visual feature sequence and the text feature sequence after adding positional encoding, resulting in a fused feature sequence, including steps S501 to S503.

[0054] Step S501: Concatenate the visual feature sequence with added location encoding and the text feature sequence to obtain the concatenated sequence.

[0055] Step S502: Input the spliced ​​sequence into the K-layer cross-modal Transformer encoder. Perform the following operations on each layer: (1) Calculate the multi-head self-attention within the sequence to capture cross-modal dependencies; (2) Perform nonlinear transformation through the feedforward network; (3) Apply residual connections and layer normalization to stabilize training.

[0056] Step S503: After K layers of processing, the output of the last layer is used as the fused feature sequence.

[0057] This fusion step achieves deep bidirectional interaction between visual and textual features through sequence concatenation and multi-layer cross-modal Transformer encoding. The self-attention mechanism at each layer dynamically captures fine-grained semantic relationships between image regions and textual terms, thereby accurately aligning lesion descriptions with imaging manifestations. The final output fused feature sequence fully contains complementary information between the image and text, laying a solid foundation for subsequent refined processing and high-quality answer generation by the expert module.

[0058] Please see Figure 6 In a specific embodiment of the present invention, the gated router calculates the routing probability according to steps S601 and S602.

[0059] Step S601: For the fused feature sequence X∈R L×D Perform a linear transformation: Z = XW g +b g In the formula, W g ∈R D×E It is the weight matrix, b g ∈R E It is the bias term, L is the sequence length, which is equal to M+N, and E is the total number of expert modules;

[0060] Step S602: Perform Softmax normalization on Z along the expert dimension to obtain the routing probability matrix S: S = Softmax(Z, dim=1), where the elements S in the routing probability matrix S are... i,e Let be the routing probability of the i-th feature vector in the fused feature sequence being assigned to the e-th expert module, where i∈{1,2,…,L} and e∈{1,2,…,E}.

[0061] In this embodiment, a learnable linear transformation and Softmax normalization are used to dynamically calculate the matching degree between each feature vector and all experts. This soft routing mechanism can automatically identify the processing modes required for different features, thereby accurately allocating low-level features such as visual edges and textures or high-level semantics such as medical terms to the most proficient expert network, achieving intelligent scheduling of computing resources and efficient utilization of model capacity.

[0062] In a specific embodiment of the present invention, in order to achieve load balancing, an auxiliary loss L is adopted. aux This constrains the uniformity of the allocation probabilities among the experts. The formula for calculating the auxiliary loss is as follows:

[0063] ,

[0064] In the formula, the average distribution score f e Within a training batch, the routing probability S for all feature vectors being assigned to the e-th expert module. i,e The average value of α is a hyperparameter used to control the load balancing intensity.

[0065] Please see Figure 7 In one specific embodiment of the present invention, the expert module consists of a small Transformer or MLP network. The hybrid expert module calculates the enhanced fusion feature sequence according to the following steps S701 and S702.

[0066] Step S701: For the i-th feature vector x in the fused feature sequence i Calculate the processing results of each expert module:

[0067] ,

[0068] In the formula, E e (x i ) represents the e-th expert module's pair of feature vectors x i The processing results are as follows: MSA represents multi-head attention, FFN represents feedforward network, and the subscript e of MSA and FFN represents the e-th expert module. These expert modules do not share parameters and are independent modules.

[0069] Step S702: Use the routing probability to perform weighted aggregation on the processing results to obtain the output vector at that location. :

[0070] ,

[0071] Iterate through all positions in the sequence and output a vector. By concatenating them in sequence, we obtain the enhanced fused feature sequence X'.

[0072] In this embodiment, the hybrid expert module's processing steps involve refining features differentially through parallel, independent expert networks, followed by soft-weighted fusion based on routing probabilities. This mechanism allows the model to leverage the specialized processing capabilities of each expert in parallel; for example, some focus on enhancing local details, while others dedicate themselves to global context modeling. The final fusion result significantly improves the discriminative power and richness of the feature representation, providing a higher-quality information foundation for generating accurate answers.

[0073] Please see Figure 8 In a specific embodiment of the present invention, the answer generator generates the answer text according to the following steps S801~S803.

[0074] Step S801: Initialize a lexical sequence containing only one start symbol. The start symbol can be, for example, [CLS] or... <start>.

[0075] Step S802: For the current decoding time step, perform the following steps: (1) Input the currently generated word sequence into the Transformer decoder, and use the enhanced fusion feature sequence as the key and value to calculate the encoder-decoder attention and obtain the decoder hidden state that incorporates multimodal context information; (2) Calculate the probability distribution on the vocabulary according to the decoder hidden state; (3) Select the next word according to the probability distribution and append it to the generated word sequence.

[0076] Step S803: When the generated word is a terminator, stop the loop and convert the generated word sequence into the answer text. Understandably, during step S802, each generated word is checked to determine if it is a terminator: if not, step S802 is repeated to generate the next word; if it is, step S803 is executed, the loop stops, and the final answer text is output.

[0077] The answer generator in this embodiment employs an autoregressive Transformer decoder to achieve end-to-end natural language answer generation. Its core advantage lies in the fact that each generation step dynamically references the enhanced multimodal sequence (as key-value pairs), ensuring that the answer is always based on the most relevant visual and textual context. This design not only supports generating closed-ended answers such as "yes / no" and categorization, but also outputs complex open-ended answers such as lesion descriptions and diagnostic criteria, significantly enhancing the system's practicality and flexibility in real-world clinical scenarios. Its progressive generation mechanism also provides inherent support for the interpretability of the answers.

[0078] The above details the specific structure of each module of the medical question-answering model. A loss function is used during the training of this model. In a specific embodiment of the present invention, the expression of the loss function for training the medical question-answering model is as follows:

[0079] L total =λ1L qa +λ2L align +λ3L consist +βL aux ,

[0080] In the formula, λ1, λ2, λ3, and β are the weights of each loss term, and L aux L is an auxiliary loss used to constrain the uniformity of the allocation probabilities among experts. qa For the cross-entropy loss used to measure the accuracy of answer prediction, L align For the contrast loss used to narrow down the distance between positive sample image-text pairs, L consist This is the loss used to supervise the consistency between the attention distribution and the target distribution of the model.

[0081] Auxiliary loss L aux The calculation formula has been given in the above embodiments. Other losses will be explained in detail below.

[0082] Cross-entropy loss L, used to measure the accuracy of answer prediction qa The calculation formula is as follows:

[0083] ,

[0084] In the formula, p(y|Y) is the probability distribution of the generated answer obtained by softmax calculation, y is the true answer label, and Y represents the predicted answer to ensure the accuracy of the generated answer.

[0085] Contrast loss L used to narrow down the distance between positive sample image-text pairs align The calculation formula is as follows:

[0086] ,

[0087] In the formula, sim represents the cosine similarity, τ is the temperature parameter, and V pos and T pos T represents the positive visual and text samples in the batch sampling, respectively. neg The negative samples are taken from the batch. This loss facilitates the alignment of visual and textual features, ensuring that the model understands and fuses information from different modalities.

[0088] Loss L used to supervise the consistency between the attention distribution and the target distribution of the model consist The calculation formula is as follows:

[0089] ,

[0090] This loss is primarily designed to ensure consistency in cross-modal attention. KL divergence is used as the loss function to align the model's cross-modal attention distribution with the predefined objective. In the formula, A... cross It is the cross-modal attention map obtained from model calculation, A target It is the target attention map obtained from supervised learning.

[0091] It should be noted that the steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0092] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.< / start>

Claims

1. A medical question-answering method integrating cross-modal hybrid experts, characterized in that, include: Acquire medical images and question text, and perform preprocessing to obtain initial visual embedding matrix and initial text embedding matrix; The initial visual embedding matrix and the initial text embedding matrix are input into the trained medical question-answering model to obtain the answer text for the medical image and the question text, wherein the medical question-answering model includes: A visual encoder is used to encode the initial visual embedding matrix to extract a sequence of visual features; A text encoder is used to encode the initial text embedding matrix to extract a text feature sequence; The position encoding module is used to add position encoding information to the visual feature sequence and the text feature sequence, so as to represent the relative or absolute position of each token in the text feature sequence by introducing an additional set of vectors, so that the medical question answering model can sequentially perceive the corresponding visual features and text features. The feature fusion module is used to perform deep interactive fusion of the visual feature sequence and the text feature sequence after adding position encoding through sequence concatenation and multi-layer cross-modal Transformer encoding to obtain a fused feature sequence. Among them, the self-attention mechanism of each layer can dynamically capture the semantic association between visual features and text features to identify and align the text description of lesions with the visual representation of the image. A gated router is used to calculate the routing probability of the feature vector at each position in the fused feature sequence to each expert module, so as to assign visual features or text features to the expert network that is best at them. A hybrid expert module, comprising multiple independent expert networks, is used to weight and aggregate each feature vector in the fused feature sequence according to the routing probabilities, outputting an enhanced fused feature sequence; and An answer generator is used to autoregressively generate the answer text based on the enhanced fusion feature sequence.

2. The medical question-answering method integrating cross-modal hybrid experts according to claim 1, characterized in that, Acquire medical images and perform preprocessing to obtain an initial visual embedding matrix, including: Convert the medical image to a preset size; The resized medical image is divided into N image blocks of size P×P; Each image patch is flattened and linearly projected into a D-dimensional embedding space to obtain the initial visual embedding matrix.

3. The medical question-answering method integrating cross-modal hybrid experts according to claim 1, characterized in that, Obtain the question text and preprocess it to obtain the initial text embedding matrix, including: The question text was divided into M tokens using a tokenizer. Each token is mapped to a D-dimensional embedding space to obtain the initial text embedding matrix.

4. The medical question-answering method integrating cross-modal hybrid experts according to claim 1, characterized in that, Both the visual encoder and the text encoder are built on the Transformer architecture.

5. The medical question-answering method integrating cross-modal hybrid experts according to claim 1, characterized in that, A cross-modal Transformer architecture is employed to perform deep interactive fusion of the visual feature sequence and the text feature sequence after adding positional encoding, resulting in a fused feature sequence, including: The visual feature sequence with added location encoding and the text feature sequence are concatenated to obtain the concatenated sequence; The concatenated sequence is input into a K-layer cross-modal Transformer encoder, and each layer performs the following operations: Calculate multi-head self-attention within the sequence to capture cross-modal dependencies; Nonlinear transformation is performed using a feedforward network; Residual connections and layer normalization are applied to stabilize training. After K layers of processing, the output of the last layer is used as the fused feature sequence.

6. The medical question-answering method integrating cross-modal hybrid experts according to claim 1, characterized in that, The gated router calculates the routing probability as follows: For the fused feature sequence X∈R L×D Perform a linear transformation: Z=XW g +b g , In the formula, W g ∈R D×E It is the weight matrix, b g ∈R E It is the bias term, L is the sequence length, and E is the total number of expert modules; Softmax normalization is applied to Z along the expert dimension to obtain the routing probability matrix S: S = Softmax(Z, dim = 1), In the formula, the element S in the routing probability matrix S i,e Let be the routing probability that the i-th feature vector in the fused feature sequence is assigned to the e-th expert module, i∈{1,2,…,L}, e∈{1,2,…,E}.

7. The medical question-answering method integrating cross-modal hybrid experts according to claim 6, characterized in that, The expert module consists of a small Transformer or MLP network; The hybrid expert module calculates the enhanced fusion feature sequence as follows: For the i-th feature vector x in the fused feature sequence i Calculate the processing results of each expert module: ， In the formula, E e (x i ) represents the e-th expert module's pair of feature vectors x i The processing results are shown in the diagram. MSA represents multi-head attention, and FFN represents feedforward network. The processing results are weighted and aggregated using the routing probabilities to obtain the output vector at that location. : ， Iterate through all positions in the sequence and output a vector. By sequentially concatenating the sequences, the enhanced fusion feature sequence is obtained.

8. The medical question-answering method integrating cross-modal hybrid experts according to claim 1, characterized in that, The answer generator generates the answer text according to the following steps: Initialize a lexical sequence containing only a start symbol; For the current decoding time step, perform the following steps: The currently generated lexical sequence is input into the Transformer decoder, and the enhanced fused feature sequence is used as the key and value to calculate the encoder-decoder attention, thereby obtaining the decoder hidden state that incorporates multimodal contextual information. Calculate the probability distribution on the vocabulary based on the decoder's hidden state; The next lexical unit is selected according to the probability distribution and appended to the generated lexical unit sequence. When the generated word is a terminator, the loop stops, and the generated word sequence is converted into the answer text.

9. The medical question-answering method integrating cross-modal hybrid experts according to claim 1, characterized in that, The loss function of the medical question-answering model during training is expressed as follows: L total =λ1L qa +λ2L align +λ3L consist +βL aux , In the formula, λ1, λ2, λ3, and β are the weights of each loss term, and L aux L is an auxiliary loss used to constrain the uniformity of the allocation probabilities among experts. qa For the cross-entropy loss used to measure the accuracy of answer prediction, L align For the contrast loss used to narrow down the distance between positive sample image-text pairs, L consist This is the loss used to supervise the consistency between the attention distribution and the target distribution of the model.

10. The medical question-answering method integrating cross-modal hybrid experts according to claim 9, characterized in that, The formula for calculating the auxiliary loss is as follows: ， In the formula, the average distribution score f e Within a training batch, the routing probability S for all feature vectors being assigned to the e-th expert module. i,e The average value of α is a hyperparameter used to control the load balancing intensity.

Images