A question and answer method and system based on multi-modal features

Abstract

The application provides a question and answer method and system based on multi-modal features, which can accurately process multi-modal input data and answer. The method comprises: acquiring multi-modal input data from a client, the multi-modal input data comprising visual input data and text input data; performing feature extraction on the visual input data and the text input data respectively to obtain visual features and language features; performing fine-grained alignment on the visual features and the language features based on an attention mechanism to obtain aligned visual features and aligned language features, the fine-grained alignment indicating that a bidirectional association is established between a fine-grained level of the visual features and a fine-grained level of the language features; fusing the aligned language features and the aligned visual features to obtain cross-modal fusion features; performing knowledge enhancement reasoning on the cross-modal fusion features to obtain reply text corresponding to the multi-modal input data; and sending the reply text to the client.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a question-answering method and system based on multimodal features. Background Technology

[0002] With the continuous development of artificial intelligence technology, intelligent question-answering systems have been widely used in many fields such as customer service, education, medical care and business consulting.

[0003] Currently, common question-answering systems, such as pure text question-answering systems and pure visual question-answering systems, mainly focus on processing single-modal input data. Text question-answering systems are typically based on pre-trained language models. These systems excel at understanding and generating natural language and can deeply utilize textual context and domain knowledge for reasoning, but they cannot process and understand visual information in images or videos. They are unable to understand and respond to queries containing images, exhibiting significant limitations. Visual question-answering systems are typically based on visual models capable of extracting image features. These systems can answer specific questions based on image content, but their text understanding capabilities are usually weak, and they are insufficient in integrating and utilizing complex textual contexts, dialogue history, and external domain knowledge. However, in real-world applications, users' information needs are often not presented in a single modality but rather involve multimodal input data that simultaneously includes images and text.

[0004] Therefore, there is an urgent need for a question-answering solution that can accurately process multimodal input data and provide answers. Summary of the Invention

[0005] This application provides a question-answering method and system based on multimodal features, which can accurately process multimodal input data and provide answers.

[0006] Firstly, a question-answering method based on multimodal features is provided, the method including: Acquire multimodal input data from the client, including visual input data and text input data; Feature extraction was performed on visual input data and text input data respectively to obtain visual features and language features; Fine-grained alignment of visual and linguistic features is performed based on an attention mechanism to obtain aligned visual and linguistic features. Fine-grained alignment refers to establishing a bidirectional association between the fine-grained levels of visual and linguistic features. The aligned linguistic and visual features are then fused to obtain cross-modal fused features. By performing knowledge-enhanced reasoning on cross-modal fusion features, the response text corresponding to the multimodal input data is obtained; Send a reply text to the client.

[0007] In one feasible design, visual and linguistic features are fine-grained aligned based on an attention mechanism to obtain aligned visual and linguistic features. Fine-grained alignment refers to establishing a bidirectional association between the fine-grained levels of visual and linguistic features. The aligned linguistic and visual features are then fused to obtain cross-modal fused features, including: Visual features and linguistic features are mapped using learnable linear mapping matrices to generate corresponding query vectors, key vectors, and value vectors. Based on the query vector obtained from the language feature mapping and the key vector obtained from the visual feature mapping, a first attention matrix from visual features to language features is determined. Each row of the first attention matrix represents the attention of a language word to all visual words. The value vectors of visual features are then weighted and fused based on the first attention matrix to obtain language enhancement features that fuse visual information. Based on the query vector obtained from visual feature mapping and the key vector obtained from linguistic feature mapping, a second attention matrix from linguistic features to visual features is determined. Each row of the second attention matrix represents the attention of a visual word to all linguistic words. The value vectors of linguistic features are then weighted and fused based on the second attention matrix to obtain visually enhanced features that incorporate linguistic information. The linguistic features and linguistic enhancement features are added together using residual connections to obtain aligned linguistic features; visual features and visual enhancement features are added together using residual connections to obtain aligned visual features. The aligned linguistic features and aligned visual features are fused to generate cross-modal fusion features.

[0008] In one feasible design, knowledge-enhanced reasoning is performed on cross-modal fusion features to obtain the response text corresponding to the multimodal input data, including: Extracting global semantic vectors from cross-modal fusion features; The answer is determined based on the global semantic vector and the prior probability distribution based on the knowledge base; Generate response text based on the prior probability distribution of the answers.

[0009] In a feasible design, generating the response text based on the prior probability distribution of the answer includes: Using the prior log odds vector determined by the prior probability distribution based on the answer as prior guidance information, and combining cross-modal fusion features and continuous prompt vectors, a word sequence is generated through autoregressive decoding. The word sequence is a text sequence composed of multiple words. When the stopping condition is met, the final generated word sequence will be used as the response text.

[0010] In a feasible design, the prior log-probability vector determined based on the prior probability distribution of the answer is used as prior guiding information. Combined with cross-modal fusion features and continuous prompt vectors, the generation steps of each word in the word sequence through autoregressive decoding include: Based on cross-modal fusion features, continuous cue vectors, and existing lexical sequences, the decoder hidden state at the current time step is generated. The probability distribution of the current word is obtained by fusing the prior log-odds vector with the decoder's hidden state; The word to be output at the current time step is determined from the vocabulary based on the probability distribution.

[0011] In a feasible design, the steps to obtain continuous cue vectors include: Based on the domain labels and hash values ​​of the corresponding question pattern features of the multimodal input data, a retrieval is performed in a pre-built cue vector cache pool; If a matching consecutive hint vector is found, output the consecutive hint vector.

[0012] In one feasible design, the method is applied to a multimodal large model. The loss function used to train the multimodal large model consists of a weighted sum of cross-entropy loss, contrastive learning loss, and knowledge enhancement loss. The cross-entropy loss is determined based on the generated answer and the true answer, the contrastive learning loss is determined based on the global vector of aligned linguistic features and the global vector of aligned visual features, and the knowledge enhancement loss is determined based on the knowledge enhancement inference result.

[0013] In a feasible design, the method also includes: Generate attention heatmaps to show the correlation between questions based on text input data and image regions based on visual input data; Extract the text content from the attention heatmap; The reply text sent to the client includes: Generate the final response text based on the content text and the response text; Send the final response text to the client.

[0014] In a feasible design, the method also includes: Based on the identifier of each of the K entities, locate the corresponding knowledge nodes from the knowledge base; Based on the location-based association information of each knowledge node, a structured reference fragment corresponding to each knowledge node is generated. The final response text is generated based on the content text and the response text, including: The final response text is generated based on the content text, the quoted segments, and the response text.

[0015] Secondly, a question-answering system based on multimodal features is provided, including: The data acquisition module is used to acquire multimodal input data from the client, including visual input data and text input data; The feature extraction module is used to extract features from visual input data and text input data respectively, to obtain visual features and language features; The feature fusion module is used to perform fine-grained alignment of visual and linguistic features based on an attention mechanism, resulting in aligned visual and linguistic features. Fine-grained alignment refers to establishing a bidirectional association between the fine-grained levels of visual and linguistic features. The aligned linguistic and visual features are then fused to obtain cross-modal fused features. The response text determination module is used to obtain the response text corresponding to the multimodal input data by performing knowledge-enhanced reasoning on cross-modal fusion features; The transmission module is used to send reply text to the client.

[0016] This application extracts features from both visual and text input data to obtain visual and linguistic features. Then, based on an attention mechanism, fine-grained alignment and fusion of these features are performed. This achieves precise bidirectional association between the basic units of visual and linguistic features before fusion, resulting in cross-modal fusion features that contain deep coupling information between visual and semantic elements. This accurately reflects the inherent relationships between multimodal data in user input, overcoming the problems of inaccurate modal alignment and insufficient interaction depth caused by simple concatenation. Furthermore, by introducing a knowledge-enhanced reasoning mechanism, prior information from the knowledge base is combined with the cross-modal fusion features, effectively improving the model's reasoning accuracy in complex scenarios, thereby generating accurate text responses for multimodal queries.

[0017] In summary, this application achieves accurate joint understanding and reliable answer generation of multimodal input data through the synergistic effect of "feature extraction, fine-grained alignment, and knowledge enhancement", thereby improving the accuracy and robustness of large models in responding to complex scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart illustrating an example of a question-answering method based on multimodal features provided in an exemplary embodiment of this application; Figure 2 This is a schematic diagram of an example question-answering system based on multimodal features provided in an exemplary embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Current question-answering systems that support multimodal input typically encode data from each of the single modalities (visual, linguistic, etc.) independently, obtaining their respective global feature vectors. These global feature vectors are then simply concatenated to form a joint feature, which is finally input into subsequent models to complete the task or for early fusion. This approach lacks a fine-grained alignment and collaborative reasoning mechanism for the deep semantic relationships between the two modalities, resulting in inaccurate alignment and insufficient interaction depth between different modalities. It also makes it difficult to fully exploit the complementary information between visual and linguistic data, thus affecting the system's ability to understand multimodal input data and reducing the accuracy of responses.

[0022] To improve the accuracy of question-answering systems in responding to multimodal input data, such as Figure 1 As shown, this application provides a question-answering method based on multimodal features, the method including: S110, acquire multimodal input data from the client.

[0023] The multimodal input data includes visual input data and text input data. Visual input data includes images or video frames, and text input data includes natural language questions posed by the user.

[0024] For example, visual input data I∈R 3×H×W Here, R represents the real number field, indicating that all elements of the visual input data are real numbers. 3 corresponds to the three color channels (RGB) of the image: red, green, and blue. H represents the image height, typically 224. W represents the image width, typically 224.

[0025] Text input data Q = [q1… q n ], where n≤512 tokens, “q1… q n " represents the first to nth tokens after the text has been split, each q iA basic semantic unit in the corresponding text, 1≤i≤n. n is the total number of tokens after the text is split, and the constraint is n≤512, that is, the text contains at most 512 tokens.

[0026] Specifically, after a user uploads multimodal input data through the client, the client can encapsulate the multimodal input data into a request using the OpenAI format and send the request to the gateway. Upon receiving the request, the gateway forwards it to the multimodal large model invocation service. The multimodal large model invocation service parses the request and processes the multimodal input data by invoking the multimodal large model.

[0027] S120 extracts features from visual input data and text input data respectively to obtain visual features and language features.

[0028] For example, visual features can be obtained by extracting features from visual input data using a visual encoder.

[0029] The visual encoder uses a large visual Transformer (ViT-L / 14) structure with a patch size of 14×14.

[0030] Specifically, taking the image data (size 224×224) of a certain channel of visual input data I as an example, the process of feature extraction of visual input data using ViT-L / 14 is as follows: The image is segmented into multiple local patches, each 14×14 pixels in size. Since the image size is 224×224, a total of (224 / 14)×(224 / 14)=256 patches are generated. Each patch is flattened and embedded into a 1024-dimensional vector space via linear projection, forming 256 patch tokens. A learnable [CLS] token is then added before the patch token sequence to aggregate global information, resulting in a token sequence of length 257 (256 patch tokens + 1 [CLS] token), with a dimension of 257×1024. Here, [CLS] is the classification token, which will aggregate the global features of the entire image in subsequent Transformer encoding. The vector corresponding to the [CLS] token serves as the global representation of the entire image, used for subsequent cross-modal alignment and inference.

[0031] A sequence of dimension 257×1024 is input into ViT-L / 14, which consists of 12 Transformer layers, for deep processing. ViT-L / 14 captures global dependencies between patches through a self-attention mechanism, and outputs visual features F of dimension 257×1024. v (Also known as visual feature vector), i.e., F v ∈R 257×1024 This feature contains both local information for each patch and global information for the image via the [CLS] token.

[0032] The above example utilizes ViT-L / 14 to extract features from visual input data, effectively capturing local details and global semantic information of the image, providing a high-quality visual representation foundation for subsequent cross-modal alignment and knowledge reasoning. Each token in the visual feature matrix Fv incorporates global contextual information through a self-attention mechanism. In particular, the vector set corresponding to the [CLS] token encodes the high-level semantics of the entire image, effectively supporting deep interaction with text modalities.

[0033] For example, language features can be obtained by extracting features from text input data using a language encoder.

[0034] The language encoder is a Rotary Position Embedding Transformer (RoPE Transformer).

[0035] Specifically, taking text input data Q as an example, the process of feature extraction using RoPE Transformer is as follows: A text sequence with a length of no more than 512 tokens is transformed into a token embedding matrix QE with a dimension of n×1024 through an embedding layer, where n is the actual number of tokens and 1024 is the feature hiding dimension, which is consistent with the dimension of the visual encoder for subsequent fusion.

[0036] The embedding matrix QE is input into the RoPE Transformer, which consists of 12 layers of RoPE Transformer modules. The RoPE Transformer finally outputs a language feature matrix F with a dimension of n×1024. l Language feature matrix F l It fully preserves the contextual semantic information and sequence position representation of the text, providing a high-quality linguistic feature foundation for cross-modal alignment.

[0037] The above example uses RoPE Transformer to encode text input and injects relative position information into the attention calculation process through rotation position encoding mechanism, thereby effectively capturing the semantic association and sequence order of the text and enhancing the model's ability to model long-distance dependencies.

[0038] S130: Based on the attention mechanism, fine-grained alignment of visual features and language features is performed to obtain aligned visual features and aligned language features. The aligned language features and aligned visual features are then fused to obtain cross-modal fusion features.

[0039] Among them, fine-grained alignment of visual and linguistic features based on attention mechanisms refers to establishing a precise bidirectional association between the basic units of visual features (such as image patches in visual input data at a fine-grained level) and the basic units of linguistic features (such as word units in text input data at a fine-grained level).

[0040] In a feasible design, fine-grained alignment of visual and linguistic features based on an attention mechanism is achieved in the following way, resulting in aligned visual and linguistic features. Fine-grained alignment refers to establishing a bidirectional association between the fine-grained levels of visual and linguistic features. The aligned linguistic and visual features are then fused to obtain cross-modal fused features. Visual features and linguistic features are mapped using learnable linear mapping matrices to generate corresponding query vectors, key vectors, and value vectors. Based on the query vector obtained from the language feature mapping and the key vector obtained from the visual feature mapping, a first attention matrix from visual features to language features is determined. Each row of the first attention matrix represents the attention of a language word to all visual words. The value vectors of visual features are then weighted and fused based on the first attention matrix to obtain language enhancement features that fuse visual information. Based on the query vector obtained from visual feature mapping and the key vector obtained from linguistic feature mapping, a second attention matrix from linguistic features to visual features is determined. Each row of the second attention matrix represents the attention of a visual word to all linguistic words. The value vectors of linguistic features are then weighted and fused based on the second attention matrix to obtain visually enhanced features that incorporate linguistic information. The linguistic features and linguistic enhancement features are added together using residual connections to obtain aligned linguistic features; visual features and visual enhancement features are added together using residual connections to obtain aligned visual features. The aligned linguistic features and aligned visual features are fused to generate cross-modal fusion features.

[0041] For example, the core parameters for defining bidirectional fine-grained cross-attention include: d=1024; h=16 heads; d h =d / h=64, Where d represents the feature hiding dimension, and its value is consistent with the feature dimension output by the visual encoder and the language encoder. h represents the head number, which is the number of parallel heads in a multi-head attention mechanism.

[0042] d h The dimension of each attention head is calculated by dividing the total hidden dimension d by the number of heads h.

[0043] The meaning of this core parameter is: high-dimensional features are split into 16 parallel attention heads through a multi-head attention mechanism, with each head focusing on capturing intermodal correlation information in different dimensions to improve alignment accuracy.

[0044] Based on the core parameters of the aforementioned bidirectional fine-grained cross-attention, the visual feature F v and language features F l The corresponding query vector, key vector, and value vector are generated by mapping through a learnable linear mapping matrix, as shown in the following formulas (1)-(6): Q l =F l W l , formula (1); K l =F l W l , formula (2); V l =F l , formula (3); Q v = F v W v , formula (4); K v =F v W v , formula (5); V v = F v , formula (6).

[0045] Among them, W l W is a linear mapping matrix of language features, whose parameters are learnable and have a dimension of 1024×1024. v Q is a linear mapping matrix of visual features, whose parameters are learnable and have a dimension of 1024×1024. l K is the query vector obtained by linear mapping of language features.l V is the key vector obtained by linear mapping of language features. l For language features that are value vectors. Q v K is the query vector obtained by linear mapping of visual features. v V is the key vector obtained by linear mapping of visual features. v Visual features are used as value vectors.

[0046] W l and W v These linear mapping matrices, which are learnable parameters, are used to adjust specific values ​​during model training or fine-tuning so that the mapped language / visual features better capture the semantic relationships between modalities. The parameters of these two linear mapping matrices are learned from the model training phase and continuously optimized during model fine-tuning.

[0047] The first attention matrix for determining visual features to linguistic features is shown in the following formula (7): , formula (7); Where T represents transpose; express and Matrix multiplication is used to calculate the similarity score between the two. The scaling factor represents the attention score, used to mitigate the problem of excessively large scores caused by matrix multiplication and to avoid gradient vanishing after softmax; softmax is the activation function used to normalize the attention score into a probability distribution. The dimension is n×257, which is the attention matrix from visual features to linguistic features, representing the attention distribution of linguistic features to visual features. Each row represents the attention distribution of a linguistic token to 257 visual tokens, and each column represents the attention distribution of a visual token to n linguistic tokens.

[0048] The value vectors of visual features are weighted and fused based on the first attention matrix to obtain the language enhancement features that fuse visual information, as shown in the following formula (8): , formula (8); in, This represents a language enhancement feature that integrates visual information, with a dimension of n×1024, which realizes the transfer of visual information to the language modality.

[0049] The second attention matrix for determining the transformation from linguistic features to visual features is shown in the following formula (9): , formula (9); in, express and Matrix multiplication is used to calculate the similarity score between the two. The dimension is 257×n, which is the attention matrix from language features to visual features, representing the attention distribution of visual features to language features. Each row represents the attention distribution of a visual token to n language tokens, and each column represents the attention distribution of a language token to 257 visual tokens.

[0050] The value vectors of language features are weighted and fused based on the second attention matrix to obtain the visually enhanced features that fuse language information, as shown in the following formula (10): , formula (10); in, The visual enhancement feature, which integrates linguistic information, has a dimension of 257×1024 and realizes the transfer of linguistic information to the visual modality.

[0051] The linguistic features and linguistic enhancement features are added together by residual connection to obtain the aligned linguistic features, as shown in the following formula (11): Z l = LayerNorm( F l + Z v→l ), formula (11); Here, LayerNorm represents Layer Normalization, used to standardize the feature distribution and accelerate model training convergence. l + Z v→l This represents a residual connection, which is the original language feature F. l Language enhancement features Z that incorporate visual information v→l The two sides are added together to preserve the original feature information and alleviate gradient vanishing. l This represents the final aligned language features, with a dimension of n×1024.

[0052] The visual features and visual enhancement features are added together by residual connection to obtain the aligned visual features, as shown in the following formula (12): Z v = LayerNorm( F v + Z l→v ), formula (12); Among them, F v + Z l→v This represents a residual connection, which is the original visual feature F. vVisual enhancement features Z with fused language information l→v The two sides are added together to preserve the original feature information and alleviate gradient vanishing. v This represents the final aligned visual features, with dimensions of 257×1024.

[0053] The aligned linguistic features and aligned visual features are fused to generate cross-modal fused features, including the following steps: (1) Perform feature splicing, as shown in the following formula (13): F concat = [Z l ;tile a vg(Z v ,n)]∈R n×2048 , formula (13); Among them, [Z l ;tile a vg(Z v [,n)] represents Z l with tile a vg(Z v The feature concatenation operation is performed on (n), where tiles a vg(Z v (n) represents the visual feature Z v After mean pooling, the sample is copied n times to match the language feature Z. l The length of the concatenated feature F (where n is the number of tokens in the text input data). concat The dimension is n×2048. The value of 2048 is formed by concatenating two 1024-dimensional feature vectors, that is, 1024 (language feature dimension) + 1024 (visual feature dimension) = 2048.

[0054] (2) Perform dimensionality reduction operation: The spliced ​​features F are processed through a 1×1 convolutional layer. concat Dimensionality reduced to n×1024, the final cross-modal fusion feature F is obtained. fusion This feature contains fine-grained alignment information for both language and vision, providing a unified multimodal feature for subsequent inference.

[0055] The above embodiment first maps visual and linguistic features to query, key, and value vectors, respectively, and establishes intermodal relationships by calculating bidirectional attention matrices: the first attention matrix encodes the attention of each linguistic token to all visual tokens (such as image patches), thus obtaining linguistic augmented features that fuse visual information; symmetrically, the second attention matrix encodes the attention of each visual token to all linguistic tokens, generating visual augmented features that fuse linguistic information. This bidirectional attention calculation at the token (or patch) level achieves fine-grained alignment of features between different modalities, ensuring that, for example, a specific object region in an image can be accurately correlated with the words describing that object in the text, overcoming the coarse alignment problem of simple concatenation strategies. Furthermore, by adding the original features and augmented features through residual connections, not only is the training process stabilized, but context from another modality is injected while preserving the original information of each modality, achieving deep fusion. Finally, the cross-modal fusion features generated based on these alignment features rich in mutual information provide a unified representation for subsequent inference that simultaneously contains accurate alignment relationships and deep interactive semantics. In summary, the above embodiments achieve deep and precise interaction between visual and linguistic modalities through fine-grained alignment and fusion based on attention mechanisms.

[0056] S140, by performing knowledge-enhanced reasoning on cross-modal fusion features, the response text corresponding to the multimodal input data is obtained.

[0057] In one feasible design, the response text corresponding to the multimodal input data is obtained by performing knowledge-enhanced reasoning on cross-modal fusion features in the following way: Extracting global semantic vectors from cross-modal fusion features; The answer is determined based on the global semantic vector and the prior probability distribution based on the knowledge base; Generate response text based on the prior probability distribution of the answers.

[0058] For example, by performing knowledge-enhanced reasoning on cross-modal fusion features, the response text corresponding to the multimodal input data is further obtained in the following way: Extracting global semantic vectors from cross-modal fusion features; Retrieve the K entities most relevant to the global semantic vector from the knowledge base through semantic matching, as well as at least one relation path corresponding to each entity. The number of hops of each relation path is less than or equal to the first threshold, and K is an integer greater than 0. Graph neural networks are used to encode each entity and its corresponding relation path, generating at least one relation path embedding vector. The first conditional probability of each relation path is determined based on the global semantic vector and the embedding vector of each relation path. Using the first conditional probability of each relation path as a weight, the conditional probabilities of the answers given the corresponding relation path are weighted and summed to obtain the prior probability distribution of the answers. Generate response text based on the prior probability distribution of the answers.

[0059] The values ​​of the first threshold and K are set according to actual needs.

[0060] It should be noted that the relational path corresponding to each entity is formed by the entity itself and other entities that the entity is connected to in the knowledge base through several hops (the number of hops is less than or equal to the first hop), which serve as the starting point and the ending point, respectively.

[0061] For example, from cross-modal fusion features F fusion Extract the vector g∈R corresponding to the [CLS] token. 1024 This refers to the global semantic vector of fused features, which contains global information from the multimodal input and has a dimension of 1×1024.

[0062] A dual-tower dense retrieval method based on semantic matching is employed to retrieve the K entities (including e1...e) most relevant to the global semantic vector from the knowledge base. K (K=5) and at least one relation path associated with each entity, where each relation path r includes relation sub-paths r1...r m For each relational sub-path, m≤3, each sub-path contains only the entities at both ends of the sub-path, excluding entities located between the start and end points. For example, if relational path r includes relational sub-paths r1, r2, and r3, the start point of r1 is the start point of relational path r, the end point of r1 is the start point of r2, the end point of r2 is the start point of r3, and the end point of r3 is the end point of relational path r.

[0063] The entity relation pairs corresponding to each entity are encoded using a graph neural network (GNN) to obtain at least one relation path embedding vector corresponding to each entity. Each entity relation pair consists of an entity and its corresponding relation path. The number of entity relation pairs corresponding to an entity is the number of relation paths corresponding to that entity. The above encoding can be understood through the following formula (14): r'=GNN(e j ,r), formula (14); Among them, e jLet represent the j-th entity, where 1 ≤ j ≤ 5, r represent a relation path corresponding to this entity, and r' be the generated relation path embedding vector. The dimension of r' is 1 × 128. GNN means using a graph neural network to encode entities and relation paths, capturing the association information between entities.

[0064] Based on the global semantic vector g and the embedding vector r' of each relation path, the first conditional probability P(r'|g) of each relation path r' is determined as shown in the following formula (15): P(r'|g)=softmax(g) T W g r'), formula (15); Among them, g T W represents the transpose of the global semantic vector g; g This represents the relational mapping weight matrix, whose parameters are learnable and used to map g... T A linear mapping is performed to fit the dimensional space of r'; P(r'|g) represents the conditional probability of the relation path embedding vector r' occurring given the global semantic vector g. This probability can be used to measure the correlation strength between each relation path and the current multimodal input semantics, thereby selecting the most relevant paths for subsequent inference.

[0065] Using the first conditional probability of each relation path as a weight, the conditional probabilities of the answers given the corresponding relation path are summed in a weighted manner to obtain the prior probability distribution of the answers. The following uses answer A as an example to illustrate this step through formula (16): P(A|E) =Σ r' P(A|r',E)·P(r'|g), formula (16); Where E represents the set of all entities; P(A|r',E) represents the conditional probability of answer A given the entity set E and the relation path embedding vector r', which can be calculated by a lightweight multilayer perceptron (MLP). The parameters of the lightweight MLP are shared when calculating the conditional probability P(A|r',E) for different relation path embedding vectors r' and entity sets E, meaning that the calculation of all P(A|r',E) reuses the same set of MLP parameters; Σ r' This involves iterating through all relation path embedding vectors r' and summing P(A|r',E)·P(r'|g) for each relation path r; P(A|E) represents the prior probability distribution of answer A given entity set E. This method can integrate the contributions of different relation paths to the answer, enhancing the credibility and accuracy of the answer by utilizing entity and relation information in the knowledge base.

[0066] Generate response text based on the prior probability distribution of the answers.

[0067] For example, the method further includes: Output a list of evidence tokens corresponding to K entities. The list includes the identifier (ID) of the token for each entity, facilitating the tracing of each entity's origin in the knowledge base and enhancing the interpretability of the evidence. The number of token IDs included in the evidence token list can be set to be less than or equal to a second threshold to ensure the conciseness and readability of key evidence. The second threshold can be set according to actual needs, for example, a value of 32.

[0068] The above embodiments perform knowledge retrieval based on global semantic vectors extracted through cross-modal fusion features. This accurately recalls entities most relevant to the multimodal input semantics and their multi-hop relationship paths, ensuring the accuracy of the knowledge base. Secondly, graph neural networks are used to encode relationship paths, encoding path sequences in the knowledge base into continuous path embedding vectors, effectively capturing complex semantic relationships between knowledge nodes. Then, by calculating the first conditional probability of the global semantic vector and each relationship path embedding vector, and using each first conditional probability as a weight to weight the answer probabilities derived from different paths, collaborative reasoning of multi-path knowledge is achieved, enhancing the logic and credibility of answer generation. The obtained prior answer probabilities are transformed into log-odds vectors and used to guide text generation, fully integrating prior knowledge from the knowledge base with the contextual semantics of multimodal input data, thereby improving the accuracy and interpretability of the output response text.

[0069] In a feasible design, the response text can be generated based on the prior probability distribution of the answers in the following way: Using the prior log odds vector determined by the prior probability distribution based on the answer as prior guidance information, and combining cross-modal fusion features and continuous prompt vectors, a word sequence is generated through autoregressive decoding. The word sequence is a text sequence composed of multiple words. When the stopping condition is met, the final generated word sequence will be used as the response text.

[0070] For example, the response text can be generated based on the prior probability distribution of the answer in the following way: The prior log odds vector is determined based on the prior probability distribution of the answer. The prior log odds vector is composed of the score of each word in the vocabulary. Obtain continuous suggestion vectors, which are used to guide the model to generate response content that conforms to the domain of the multimodal input data and the corresponding question pattern; Using the prior log-odds vector as prior guiding information, combined with cross-modal fusion features and continuous cue vectors, a word sequence is generated through autoregressive decoding. The word sequence is a text sequence composed of multiple words. When the stopping condition is met, the final generated word sequence will be used as the response text.

[0071] The stopping condition can be set according to actual needs. For example, it can be set to the length of the generated lexical sequence (i.e. the number of tokens in the lexical sequence) reaching a preset value or the end character being decoded.

[0072] A continuous cue vector is a type of cue vector that refers to a continuous, trainable vector of numerical values.

[0073] As a further example, using the prior log-probability vector determined based on the prior probability distribution of the answer as prior guiding information, and combining cross-modal fusion features with continuous prompt vectors, the generation steps for each word in the word sequence through autoregressive decoding include: Based on cross-modal fusion features, continuous cue vectors, and existing lexical sequences, the decoder hidden state at the current time step is generated. The probability distribution of the current word is obtained by fusing the prior log-odds vector with the decoder's hidden state; The word to be output at the current time step is determined from the vocabulary based on the probability distribution.

[0074] Specifically, the generation steps for each word in the word sequence generated by autoregressive decoding can further include: Based on cross-modal fusion features, continuous cue vectors, and existing word sequences, the decoder hidden state at the current time step is generated through the decoder. The prior log odds vector is added to the initial log odds vector obtained based on the decoder hidden state at the current time step to obtain the fused log odds vector at the current time step. The probability distribution of the current word is obtained by performing softmax normalization on the fused log-odds vector. The word with the highest score in the vocabulary is selected as the output of the current time step based on the probability distribution of the current word.

[0075] For example, the prior log odds vector is determined based on the prior probability distribution of the answer as follows: Logarithmic transformation of the prior probability distribution P(A|E) of answer A yields the prior log-odds vector L. A L A =log(P(A|E) / (1 - P(A|E))), is used to map the probability distribution to real-domain scores, i.e., a vector composed of the scores of each token in the vocabulary. A The dimension is 1×V, where V is the vocabulary size.

[0076] Obtain the continuous prompt vector P, where the dimension of P is 10×1024. Based on the cross-modal fusion feature F fusion The process of autoregressive decoding to generate a sequence of tokens using the continuous prompt vector P is as follows: The first step of autoregressive decoding: Input F fusion and P into the decoder that uses the autoregressive decoding method. The decoder generates the hidden state h1, and the decoder uses W o (the output mapping weight matrix with learnable parameters) to map h1, generating the initial logit vector L A '1. The decoder adds the prior logit vector L A to L A ' (or the weighted L A ') to obtain the fused logit vector L A ''1. The decoder performs softmax normalization on L A ''1 to obtain the probability distribution of the first token (including both prior knowledge constraints and the current context). Then, select the token with the highest probability from the vocabulary (e.g., "a") as the output.

[0077] The second step of autoregressive decoding: Input F fusion 、P and the token "a" output in the previous step into the decoder. The decoder generates a new hidden state h2, and the decoder uses W o to map h2, generating the initial logit vector L A '2. The decoder adds the prior logit vector L A to L A '2 (or the weighted L A '2) to obtain the fused logit vector L A ''2. The decoder performs softmax normalization on L A ''2 to obtain the probability distribution of the second token. Then, select the token with the highest probability from the vocabulary (e.g., "si") as the output.

[0078] And so on. Repeat the autoregressive decoding according to the above logic until the end token is generated or the maximum output length of 128 is reached, completing the generation of the token sequence.

[0079] The probability distribution of each token generated by the above autoregressive decoding can be summarized by the following formula (17): p(y t |y <<t},F fusion ,P) = softmax(Wo h t ), formula (17); Where t represents the time step, h t Let y represent the decoder hidden state at step t. t y represents the token generated in step t. <<t p(y) represents the sequence of tokens generated before the t-th token. t |y <<t},F fusion ,P) represents the sequence of tokens (y) that have already been generated. <<t ), Fusion features (F fusion Given P and P, generate the t-th token (y). t The probability of ).

[0080] The above embodiments, by introducing continuous cue vectors, ensure that the generated response text matches the specific domain and question pattern of the multimodal input data in terms of style and content, thus improving the professionalism and accuracy of the response. Furthermore, by introducing a prior log-odds vector as a prior guiding signal into the decoding process, the generation of each lexical unit not only depends on historical context but is also strongly constrained by the probability of the answer derived from knowledge, enhancing the factual correctness and logical consistency of the response. Finally, by iteratively fusing cross-modal fusion features, continuous cue vectors, and prior log-odds vectors through autoregressive decoding to gradually generate a lexical sequence, the fluency and coherence of the generated text are guaranteed. This achieves the generation of high-quality response text that is accurate, fluent, and meets domain requirements in complex multimodal question-answering scenarios.

[0081] For example, the method further includes: Domain labels for multimodal input data are identified through keyword matching or semantic recognition, so that matching sequential cue vectors can be retrieved based on the domain labels in the subsequent process.

[0082] For example, the method further includes: Record the timestamp information of the acquired multimodal input data for use in data processing within the large multimodal model.

[0083] In a feasible design, the continuous cue vector is obtained in the following way: Based on the domain labels and hash values ​​of the corresponding question pattern features of the multimodal input data, a retrieval is performed in a pre-built cue vector cache pool; If a matching consecutive hint vector is found, output the consecutive hint vector.

[0084] For example, the continuous cue vector can be obtained further by the following method: Generate a search key based on the domain labels and corresponding question pattern features of the multimodal input data; The search is performed in a pre-built hint vector cache pool based on the search key. The hint vector cache pool stores multiple key-value pairs. Each key-value pair includes a key consisting of a hash value of a domain label and a question pattern feature, and a value consisting of a corresponding consecutive hint vector. If a continuous suggestion vector matching the search key is found, the continuous suggestion vector is output.

[0085] Domain labels are used to identify the type of domain. The question-answering system of this application can automatically identify domain labels of multimodal input data.

[0086] For example, the question pattern features include semantic features such as grammatical structure and question word type. The syntactic dependency tree and question word type are extracted from the text content of the multimodal input data, and the hash value of the syntactic dependency tree and question word type is calculated to generate the hash value of the question pattern features.

[0087] For example, if no consecutive suggestion vector matching the search key is found (e.g., hit rate less than 0.7), low-rank adaptation (LoRA) is used to determine consecutive suggestion vectors and update the suggestion vector cache pool. Specifically: Using the current cross-modal fusion features, the LoRA method is used to perform G gradient descent iterations on the multimodal large model to obtain the fine-tuned multimodal large model. The value of G can be 3. Extract the vectors corresponding to the key token sequences (such as the [CLS] token and the K most semantically relevant tokens) from the cross-modal fusion features, and adjust the sequence length to the storage format specified by the hint vector cache pool by truncation / padding or mean pooling (e.g., adjust to 10 to get a 10×1024 vector). A new key is constructed using the hash values ​​of the domain labels and problem pattern features corresponding to the current multimodal input data. The adjusted cue vector is then stored as the value in the cue vector cache pool for reuse in subsequent similar requests.

[0088] For example, the hint vector cache pool uses the following eviction policy: When cache space is insufficient, the Least Recently Used (LRU) algorithm is used to identify candidate key-value pairs to be deleted. Key-value pairs accessed less than 3 times per week are selected from the candidate pairs for deletion. This ensures that the data that is ultimately deleted is data that has not been activated recently and has a very low long-term usage rate, thus maintaining the effectiveness of cache space while prioritizing the retention of frequently used key-value pairs.

[0089] The above embodiments set up a hint vector cache pool to allow subsequent matching of consecutive hint vectors that match the question using hash values ​​of domain labels and question pattern features. This facilitates the incorporation of consecutive hint vectors when generating response text, thereby improving the quality and efficiency of the response text.

[0090] S150, send a reply text to the client.

[0091] Specifically, after the multimodal large model calls the service to obtain the response text generated by the multimodal large model, it returns the response text to the gateway, and the gateway then sends the response text to the client.

[0092] In one feasible design, the method is applied to a multimodal large model. The loss function used to train the multimodal large model consists of a weighted sum of cross-entropy loss, contrastive learning loss, and knowledge enhancement loss. The cross-entropy loss is determined based on the generated answer and the true answer, the contrastive learning loss is determined based on the global vector of aligned linguistic features and the global vector of aligned visual features, and the knowledge enhancement loss is determined based on the knowledge enhancement inference result.

[0093] For example, the loss function L used to train a multimodal large model total = L ce +λ1L contrast +λ2L kg , where L ce L represents the cross-entropy loss term, used to optimize the prediction accuracy of the answer token; contrast L represents the contrastive learning loss term, used to optimize inter-modal alignment; kg λ1 and λ2 represent the knowledge enhancement loss term, used to optimize the accuracy of knowledge reasoning. λ1 and λ2 are loss weight coefficients, whose values ​​can be adjusted according to the training effect; typically, λ1 is set to 0.3 and λ2 to 0.1.

[0094] For example, the contrastive learning loss term is shown in the following formula (18): , formula (18); Where log represents the natural logarithm function; exp represents the exponential function; sim() represents the cosine similarity, used to calculate the similarity between two vectors, with a value range of (-1, 1); Represents the global visual vector, which is the aligned visual feature Z. v The vector corresponding to the [CLS] token in the vector, the [CLS] token aggregates the global information of the entire visual feature; The language global vector is the aligned language feature Z. l The global representation; Indicates and The matching positive example language global vector, that is, the language global vector from the same input sample; This represents the negative example language global vector, i.e., the language global vector from other input samples; This represents the temperature parameter, which can be 0.07. It is used to adjust the smoothness of the similarity distribution and control the influence weight of negative examples. The symbol represents the summation operator, and k represents the index of the negative example language global vector, used to traverse all negative example language global vectors.

[0095] For example, = meanpool(Z l )∈R 1024 That is, the language feature Z l Perform mean pooling operation, by calculating Z l The average of all token features yields the global language vector. .

[0096] Optimize the visual global vector by comparing and learning the loss term. Global vectors of positive example languages The similarity, while reducing With all negative example language global vectors The similarity makes cross-modal global vectors from the same input closer in the feature space, and cross-modal global vectors from different inputs more distant, thereby improving the global alignment accuracy between modalities.

[0097] For example, the knowledge enhancement loss term is determined by the following formula (19): L kg =α·L entity +β·L relation , formula (19); Among them, L entity L represents the entity alignment loss, used to constrain the relevance between the entities recalled by the model and the answer; relation The relation path loss is used to constrain the accuracy of the model's encoding of relation paths in the knowledge base, ensuring that the relation probability P(r'|g) matches the actual relation. α and β represent empirical coefficients used to balance the importance of entity alignment and relation alignment, satisfying the condition α+β=1. For example, α can be set to 0.6 and β to 0.4.

[0098] In the above embodiments, the overall training loss is obtained by weighted summation of cross-entropy loss, contrastive learning loss, and knowledge enhancement loss. By jointly optimizing this loss function, the model can simultaneously improve the accuracy of answer generation, cross-modal alignment, and knowledge reasoning, thereby achieving end-to-end multimodal question answering optimization.

[0099] In a feasible design, the method also includes: Generate attention heatmaps to show the correlation between questions based on text input data and image regions based on visual input data; Extract the text content from the attention heatmap; The reply text sent to the client includes: Generate the final response text based on the content text and the response text; Send the final response text to the client.

[0100] For example, the first attention matrix that converts visual features into linguistic features. The maximum number of overheads is used as the attention heatmap matrix M (n×257), where n is the number of language tokens and 257 is the number of visual tokens (including the [CLS] token). The column vectors corresponding to the [CLS] tokens in matrix M are removed, resulting in a heatmap matrix M' of dimension (n×256). Each row in heatmap matrix M' corresponds to the attention intensity of a language token at various spatial locations in the image, with each row having a dimension of 1×256. The 256 visual tokens in each row are rearranged into a 16×16 grid and transformed into 224×224 heatmaps, resulting in n heatmaps. Each heatmap is overlaid with the visual input data from the multimodal input data to generate a visual attention distribution map, intuitively showing the relationship between language tokens and image regions. The attention heatmap includes these n visual attention distribution maps. Text content is extracted from each visual attention distribution map using optical character recognition technology to form the content text. After the content text and response text generated by the multimodal large model are sent to the gateway, the gateway generates prompt words based on the content text and response text. The gateway then calls a service through the large language model and sends these prompt words to the large language model. The large language model's language understanding capabilities are used to semantically fuse the content text and response text based on the prompt words, generating the final response text. The large language model then calls a service to return the final response text to the gateway, which in turn returns the final response text to the client.

[0101] For example, the method further includes: Attention heatmaps are sent to the client, allowing users to intuitively understand the correspondence between image regions and text fragments that the model focuses on when generating responses.

[0102] The above embodiments generate attention heatmaps based on the visual-to-linguistic attention matrix produced during cross-modal alignment. This visually reveals the key image regions the model focuses on during answer reasoning, thus making the internal decision-making process transparent. By extracting textual content from the heatmaps, pixel-level visual attention is transformed into machine-readable semantic summaries. Finally, by fusing this semantic summary with the initial response, the generated final response not only includes the answer itself but also clearly explains the rationale behind its decision, significantly improving the interpretability and user credibility of the question-answering system's output.

[0103] In a feasible design, the method also includes: Based on the identifier of each of the K entities, locate the corresponding knowledge nodes from the knowledge base; Based on the location-based association information of each knowledge node, a structured reference fragment corresponding to each knowledge node is generated. The final response text is generated based on the content text and the response text, including: The final response text is generated based on the content text, the quoted segments, and the response text.

[0104] For example, based on the identifier of each entity in entity set E, a query is performed in the database to obtain the corresponding entity's association information. Based on this association information, a structured reference fragment is generated; for example, the reference fragment for the entity "Aspirin" would be {"Entity": "Aspirin", "Attribute": "Indication", "Value": "Headache"}. After the content text, reference fragments, and response text generated by the multimodal large model are sent to the gateway, the gateway generates prompt words based on these elements. These prompt words are then sent to the large language model via a service call. The large language model's language understanding capabilities are used to semantically fuse the content text, reference fragments, and response text based on the prompt words, generating the final response text. The large language model then returns the final response text to the gateway via the service call, and the gateway returns the final response text to the client.

[0105] For example, multimodal input data includes a photo of a medication uploaded by the user (i.e., visual input data) and the question "How to take it?" (i.e., text input data). The initial response text determined by the multimodal large model is "Take one tablet once a day after meals." The content text determined based on the attention heatmap is "The heatmap shows that the model mainly focused on the area in the image that indicates the medication name 'Aspirin' and the specification '100mg' when answering." The reference fragments include: {"Entity": "Aspirin enteric-coated tablets", "Attribute": "Dosage and administration", "Value": "Take one tablet once a day"}{"Entity": "Aspirin", "Attribute": "Dose of medication", "Value": "Take after meals"}. After sending the above three items of content generated by the multimodal large model to the gateway, the gateway sends the above three items of content to the large language model through the large language model call service. The large language model performs deep semantic fusion on the above three items to generate the final response text: "The image shows aspirin enteric-coated tablets 100mg, once daily, one tablet each time, taken after meals. Detailed source has been provided, and the heatmap has marked the drug name area." The large language model calls the service to return the final response text to the gateway, and the gateway sends the final response text to the client.

[0106] For example, the method further includes: Each reference fragment corresponding to the K entities is sent to the client for display to the user, allowing the user to clearly see the knowledge source on which the model's decisions are based, thereby further enhancing the interpretability of the question-answering system's output and the user's trust in the system.

[0107] The above embodiments achieve traceability of answer sources by locating and generating structured reference fragments from the knowledge base based on identified entities. Then, the content text extracted from the attention heatmap, these structured knowledge references, and the initial response text are fused to generate the final response text. This process ensures that the final response not only includes the core answer but also integrates plaintext descriptions of "visual evidence" and "knowledge evidence," thus presenting a multi-dimensional, multi-source chain of evidence supporting the conclusion while providing a textual response, thereby enhancing the completeness, credibility, and interpretability of the final response text.

[0108] This application extracts features from both visual and text input data to obtain visual and linguistic features. Then, based on an attention mechanism, fine-grained alignment and fusion of these features are performed. This achieves precise bidirectional association between the basic units of visual and linguistic features before fusion, resulting in cross-modal fusion features that contain deep coupling information between visual and semantic elements. This accurately reflects the inherent relationships between multimodal data in user input, overcoming the problems of inaccurate modal alignment and insufficient interaction depth caused by simple concatenation. Furthermore, by introducing a knowledge-enhanced reasoning mechanism, prior information from the knowledge base is combined with the cross-modal fusion features, effectively improving the model's reasoning accuracy in complex scenarios, thereby generating accurate text responses for multimodal queries.

[0109] In summary, this application achieves accurate joint understanding and reliable answer generation of multimodal input data through the synergistic effect of "feature extraction, fine-grained alignment, and knowledge enhancement", thereby improving the accuracy and robustness of large models in responding to complex scenarios.

[0110] like Figure 2 As shown, this application also provides a question-answering system based on multimodal features, including: The data acquisition module is used to acquire multimodal input data from the client, including visual input data and text input data; The feature extraction module is used to extract features from visual input data and text input data respectively, to obtain visual features and language features; The feature fusion module is used to perform fine-grained alignment of visual and linguistic features based on an attention mechanism, resulting in aligned visual and linguistic features. Fine-grained alignment refers to establishing a bidirectional association between the fine-grained levels of visual and linguistic features. The aligned linguistic and visual features are then fused to obtain cross-modal fused features. The response text determination module is used to obtain the response text corresponding to the multimodal input data by performing knowledge-enhanced reasoning on cross-modal fusion features; The transmission module is used to send reply text to the client.

[0111] In a feasible design, the feature fusion module performs fine-grained alignment of visual and linguistic features based on an attention mechanism to obtain aligned visual and linguistic features. Fine-grained alignment refers to establishing a bidirectional association between the fine-grained levels of visual and linguistic features. The aligned linguistic and visual features are then fused to obtain cross-modal fused features. Visual features and linguistic features are mapped using learnable linear mapping matrices to generate corresponding query vectors, key vectors, and value vectors. Based on the query vector obtained from the language feature mapping and the key vector obtained from the visual feature mapping, a first attention matrix from visual features to language features is determined. Each row of the first attention matrix represents the attention of a language word to all visual words. The value vectors of visual features are then weighted and fused based on the first attention matrix to obtain language enhancement features that fuse visual information. Based on the query vector obtained from visual feature mapping and the key vector obtained from linguistic feature mapping, a second attention matrix from linguistic features to visual features is determined. Each row of the second attention matrix represents the attention of a visual word to all linguistic words. The value vectors of linguistic features are then weighted and fused based on the second attention matrix to obtain visually enhanced features that incorporate linguistic information. The linguistic features and linguistic enhancement features are added together using residual connections to obtain aligned linguistic features; visual features and visual enhancement features are added together using residual connections to obtain aligned visual features. The aligned linguistic features and aligned visual features are fused to generate cross-modal fusion features.

[0112] In one feasible design, the response text determination module obtains the response text corresponding to the multimodal input data by performing knowledge-enhanced reasoning on cross-modal fusion features: Extracting global semantic vectors from cross-modal fusion features; The answer is determined based on the global semantic vector and the prior probability distribution based on the knowledge base; Generate response text based on the prior probability distribution of the answers.

[0113] In a feasible design, the response text determination module generates the response text based on the prior probability distribution of the answers in the following way: Using the prior log odds vector determined by the prior probability distribution based on the answer as prior guidance information, and combining cross-modal fusion features and continuous prompt vectors, a word sequence is generated through autoregressive decoding. The word sequence is a text sequence composed of multiple words. When the stopping condition is met, the final generated word sequence will be used as the response text.

[0114] In a feasible design, the response text determination module implements the following steps to generate each word of the word sequence by using a prior log-odds vector determined based on the prior probability distribution of the answer as prior guiding information, combining cross-modal fusion features and continuous prompt vectors: Based on cross-modal fusion features, continuous cue vectors, and existing lexical sequences, the decoder hidden state at the current time step is generated. The probability distribution of the current word is obtained by fusing the prior log-odds vector with the decoder's hidden state; The word to be output at the current time step is determined from the vocabulary based on the probability distribution.

[0115] In a feasible design, the response text determination module obtains the continuous prompt vector in the following way: Based on the domain labels and hash values ​​of the corresponding question pattern features of the multimodal input data, a retrieval is performed in a pre-built cue vector cache pool; If a matching consecutive hint vector is found, output the consecutive hint vector.

[0116] In one feasible design, the data acquisition module, feature extraction module, feature fusion module, and response text determination module are integrated into a multimodal large model. The loss function used to train the multimodal large model consists of a weighted sum of cross-entropy loss, contrastive learning loss, and knowledge enhancement loss. The cross-entropy loss is determined based on the generated answer and the true answer, the contrastive learning loss is determined based on the global vector of aligned linguistic features and the global vector of aligned visual features, and the knowledge enhancement loss is determined based on the knowledge enhancement inference result.

[0117] In one feasible design, the system also includes an attention heatmap generation module, a gateway, and a large language model. The attention heatmap generation module is used to generate an attention heatmap that shows the relationship between the text input data and the image regions of the visual input data; extract the content text of the attention heatmap; the gateway sends the content text and the response text to the large language model to generate the final response text based on the content text and the response text; and the gateway sends the final response text to the client.

[0118] In one feasible design, the system also includes a reference fragment generation module, which is used to locate the corresponding knowledge nodes from the knowledge base based on the identifier of each of the K entities; generate structured reference fragments corresponding to each knowledge node based on the association information of the located knowledge nodes; the gateway sends the content text, each reference fragment, and the response text to the large language model, so that the large language model can generate the final response text based on the content text, each reference fragment, and the response text, and the gateway sends the final response text to the client.

[0119] Other implementation methods and effects of the above system can be found in the description of the question-answering method based on multimodal features, and will not be repeated here.

[0120] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0121] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0122] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0123] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0124] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0125] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A question-answering method based on multimodal features, characterized in that, The method includes: Acquire multimodal input data from the client, the multimodal input data including visual input data and text input data; Feature extraction is performed on the visual input data and the text input data respectively to obtain visual features and language features; The visual features and the language features are finely aligned based on an attention mechanism to obtain aligned visual features and aligned language features. The fine-grained alignment refers to establishing a bidirectional association between the fine-grained level of the visual features and the fine-grained level of the language features. The aligned language features and the aligned visual features are then fused to obtain cross-modal fused features. By performing knowledge-enhanced reasoning on the cross-modal fusion features, the response text corresponding to the multimodal input data is obtained; Send the reply text to the client.

2. The method according to claim 1, characterized in that, The attention mechanism is used to perform fine-grained alignment of the visual features and the language features to obtain aligned visual features and aligned language features. The fine-grained alignment refers to establishing a bidirectional association between the fine-grained level of the visual features and the fine-grained level of the language features. The aligned linguistic features and aligned visual features are fused to obtain cross-modal fused features, including: The visual features and the language features are mapped using a learnable linear mapping matrix to generate corresponding query vectors, key vectors, and value vectors. Based on the query vector obtained by the language feature mapping and the key vector obtained by the visual feature mapping, a first attention matrix from the visual features to the language features is determined, where each row of the first attention matrix represents the attention of a language word to all visual words; the value vector of the visual features is weighted and fused based on the first attention matrix to obtain language enhancement features that fuse visual information. Based on the query vector obtained by the visual feature mapping and the key vector obtained by the language feature mapping, a second attention matrix from the language features to the visual features is determined. Each row of the second attention matrix represents the attention of a visual word to all language words. The value vectors of the language features are weighted and fused based on the second attention matrix to obtain visually enhanced features that fuse language information. The linguistic features and the linguistic enhancement features are added together using a residual connection method to obtain aligned linguistic features; the visual features and the visual enhancement features are added together using a residual connection method to obtain aligned visual features. The aligned linguistic features and the aligned visual features are fused to generate cross-modal fusion features.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the response text corresponding to the multimodal input data by performing knowledge-enhanced reasoning on the cross-modal fusion features includes: Extract global semantic vectors from the cross-modal fusion features; Based on the global semantic vector and the prior probability distribution of the answer determined by the knowledge base; The response text is generated based on the prior probability distribution of the answer.

4. The method according to claim 3, characterized in that, The step of generating the response text based on the prior probability distribution of the answer includes: Using the prior log odds vector determined based on the prior probability distribution of the answer as prior guidance information, and combining the cross-modal fusion features and continuous prompt vector, a word sequence is generated through autoregressive decoding. The word sequence is a text sequence composed of multiple words. When the stopping condition is met, the final generated word sequence will be used as the response text.

5. The method according to claim 4, characterized in that, The step of generating each word in the word sequence by using the prior log-probability vector determined based on the prior probability distribution of the answer as prior guiding information, combined with the cross-modal fusion features and continuous prompt vectors, through autoregressive decoding includes: Based on the cross-modal fusion features, the continuous cue vectors, and the existing lexical sequences, the decoder hidden state at the current time step is generated; The probability distribution of the current word is obtained by fusing the prior log-probability vector with the decoder hidden state; The word to be output at the current time step is determined from the vocabulary based on the probability distribution.

6. The method according to claim 4 or 5, characterized in that, The steps to obtain continuous cue vectors include: Based on the domain labels and corresponding question pattern features hash values ​​of the multimodal input data, a retrieval is performed in a pre-built cue vector cache pool; If a matching consecutive hint vector is found, the consecutive hint vector is output.

7. The method according to claim 2, characterized in that, The method is applied to a multimodal large model. The loss function used to train the multimodal large model is composed of a weighted sum of a cross-entropy loss term, a contrastive learning loss term, and a knowledge enhancement loss term. The cross-entropy loss term is determined based on the generated answer and the true answer. The contrastive learning loss term is determined based on the global vector of the aligned linguistic features and the global vector of the aligned visual features. The knowledge enhancement loss term is determined based on the knowledge enhancement inference result.

8. The method according to claim 3, characterized in that, The method further includes: Generate an attention heatmap to demonstrate the correlation between the text input data and the image regions of the visual input data; Extract the text content from the attention heatmap; Sending the reply text to the client includes: The final response text is generated based on the content text and the response text. Send the final reply text to the client.

9. The method according to claim 8, characterized in that, The method further includes: Based on the identifier of each of the K entities, locate the corresponding knowledge nodes from the knowledge base; Based on the location-based association information of each knowledge node, a structured reference fragment corresponding to each knowledge node is generated. The step of generating the final response text based on the content text and the response text includes: The final response text is generated based on the content text, the quoted segments, and the response text.

10. A question-answering system based on multimodal features, characterized in that, include: The data acquisition module is used to acquire multimodal input data from the client, including visual input data and text input data; The feature extraction module is used to extract features from the visual input data and the text input data respectively to obtain visual features and language features; The feature fusion module is used to perform fine-grained alignment of the visual features and the language features based on an attention mechanism to obtain aligned visual features and aligned language features. The fine-grained alignment refers to establishing a bidirectional association between the fine-grained level of the visual features and the fine-grained level of the language features. The aligned language features and aligned visual features are then fused to obtain cross-modal fused features. The response text determination module is used to obtain the response text corresponding to the multimodal input data by performing knowledge-enhanced reasoning on the cross-modal fusion features; The transmission module is used to send the reply text to the client.

Images