A multi-modal large model-based question and answer data processing method and system

Abstract

The application relates to a multi-modal large model-based question and answer data processing method and system, and belongs to the technical field of data processing. The method comprises the following steps: capturing fine-grained feature relationships in multi-modal through constructing a relationship graph, and obtaining fused multi-modal features; calculating the similarity between the multi-modal fusion features and all knowledge vectors in a knowledge base, setting a threshold to filter the knowledge vectors to construct a knowledge subset; inputting the constructed knowledge subset into a machine learning model to identify the intention and scene of the user's question and answer input; inputting all the obtained features into the decoder of BLIP after splicing to generate a predicted answer; collecting user feedback through multi-round dialogue, and continuously updating the identification features according to the user feedback to generate incentive signals. The application performs cross-modal knowledge retrieval through multi-modal fusion features to construct a knowledge subset, and obtains identification features based on the knowledge subset to generate an extended guiding answer for the user, thereby improving the accuracy and satisfaction of the question and answer.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a question-and-answer data processing method and system based on a multimodal large model. Background Technology

[0002] To further expand the application scope of general understanding capabilities, multimodal large models have emerged. These models can receive data from multiple sources, including not only regular text but also images, audio, video, and even tactile information. This enables the models to understand higher-dimensional and richer information.

[0003] However, existing large-scale model-driven AI question-answering models mostly adopt a "user asks a question—model answers" approach, where the user explicitly expresses their needs, and the model provides an answer after the user asks the question. This model works well for structured or simple questions, but when faced with complex or ambiguous scenarios, especially when the user cannot clearly express their needs or when the user needs to expand their thinking, the model struggles to provide effectively guided dialogue answers. Furthermore, in the process of answer prediction and generation, it often ignores the guidance of user intent and the question-answering context, thus limiting the innovativeness of the generated answers. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention provides a question-and-answer data processing method and system based on a multimodal large model. By using multimodal fusion features to perform cross-modal knowledge retrieval and construct knowledge subsets, and by obtaining recognition features based on these knowledge subsets, extended guided answers are generated for users, thereby improving the accuracy and satisfaction of question-and-answer processing.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] The first aspect of this disclosure provides a question-answering data processing method based on a multimodal large model, comprising the following steps:

[0007] S1. Obtain multimodal fusion features: By constructing a relationship graph, fine-grained feature relationships in multimodal data are captured, and the fusion and matching of cross-modal information are completed to obtain the fused multimodal features.

[0008] S2. Cross-modal knowledge retrieval: A knowledge base is built based on the feature vectors of existing public image and text data. By calculating the similarity between the multimodal fusion features and all knowledge vectors in the knowledge base, a threshold is set to filter knowledge vectors and construct a knowledge subset.

[0009] S3. Intent and Scene Recognition: Input the constructed knowledge subset into the machine learning model to recognize the intent and scene of the user's question and answer input, and output the recognition features of the guiding intent and scene prediction results;

[0010] S4. Answer generation and output: The predicted answer is generated by concatenating all the acquired features and inputting them into the BLIP decoder.

[0011] S5. Scene Update and Expansion: Collect user feedback through multi-round dialogue, generate incentive signals based on user feedback to continuously update recognition features, thereby gradually narrowing the scope of problems and completing iterative optimization of user problems;

[0012] The generation and output of the answer specifically includes the following steps:

[0013] After concatenating image features, text features, knowledge features, and recognition features, the data is input into the BLIP decoder. The decoder generates the answer through autoregression, progressively predicting the probability distribution of each word.

[0014] During the decoding process, the softmax function is used to calculate the probability distribution of each word and select the next word based on the probability distribution until a complete answer is generated.

[0015] Furthermore, the acquisition of multimodal fusion features includes the following steps:

[0016] S11. Multimodal feature extraction: Extract modal features from the user-input image data and text data respectively;

[0017] S12. Relationship Graph Construction: Fine-grained features are used as nodes in the graph, and the relationships between nodes are established through multiple subgraphs to obtain inference features;

[0018] S13, Feature Fusion: Calculate the cross-modal discriminative power of the original features and the inferred features, dynamically adjust the weights of the original features and the inferred features to perform feature fusion, and output multimodal fused features.

[0019] Furthermore, in the multimodal feature extraction, the feature extraction of image data includes the following steps:

[0020] The image data input by the user is preprocessed, and multiple candidate boxes are generated based on RPN using a pre-trained Faster R-CNN model. The output of each candidate box includes the location and confidence score.

[0021] For each generated candidate box, the candidate box region is cropped from the original image, and the cropped image region is input into a convolutional neural network for feature extraction to obtain the feature vector of each candidate box.

[0022] The extracted candidate box features are mapped to the common feature space through a fully connected layer to obtain feature vectors, and a position code is generated for each candidate box. The image is divided into a K×K grid, with the grid covered by the candidate box coded as 1 and the grid not covered coded as 0, thus forming the position information of each candidate box.

[0023] Among them, the feature vector after mapping to the common space O i Represented as:

[0024] O i =Tanh(W v v i +b v );

[0025] In the formula, Tanh() is the activation function. v i It is the first i The original feature vectors of the candidate boxes, W v It is the weight matrix, b v It is the bias vector;

[0026] The feature vectors of all candidate boxes are combined with their corresponding location information to form a complete image feature representation.

[0027] Furthermore, in the multimodal feature extraction, the feature extraction of text data includes the following steps:

[0028] The words in the text are mapped to a dictionary, and a unique code is assigned to each word by building a vocabulary.

[0029] Each word is mapped to a vector space using a pre-trained word embedding model, represented as follows:

[0030] {t1,…,t j , ..., t n};

[0031] Among them, t j is the 300-dimensional feature vector of the j-th word, and n is the length of the sentence;

[0032] The complete text description is input into the bidirectional gated recurrent unit model to obtain the context information of each word and generate a new feature vector that represents the meaning of the word in the context.

[0033] Furthermore, the construction of the relationship graph includes the following steps:

[0034] S121. Node definition: For image features, the extracted candidate box features are used as nodes, and each candidate box represents an object in the image.

[0035] For the text modality, words in the text are used as nodes, and each word is mapped to a vector space through the GloVe model.

[0036] S122. Construction of positional relationship graph: The image is divided into a K×K grid. The grid covered by the candidate box is coded as 1, and the grid not covered is coded as 0, forming the positional information of each candidate box. The relative positional relationship is established by calculating the cosine similarity between the candidate boxes.

[0037] S123. Similarity graph construction: The appearance features of candidate boxes are mapped to the latent space through a fully connected layer, and then the cosine distance between the features of all candidate boxes is calculated as the edge of the graph.

[0038] Based on similarity, edge connections are established between candidate box features to form a similarity graph, which represents the relationship between objects in the image;

[0039] S124. Text Modal Relationship Graph Construction: The words in the text are used for feature inference through a fully connected layer, and then the similarity between words is calculated to build a relationship graph between words;

[0040] Based on similarity, edge connections are established between words to form a relationship graph of text modalities, capturing the semantic associations between words;

[0041] S125. Subgraph reasoning: Using graph convolutional networks to reason about the constructed relational graph. The reasoning process for each subgraph includes updating node features and aggregating information. By integrating all subgraph information, updated image features and text features are obtained.

[0042] Furthermore, the formula for calculating the cross-modal discrimination is as follows:

[0043] ;

[0044] In the formula, Indicates the feature to be measured x i With another modal feature set Y Cross-modal discrimination, It is a feature set Y The number of features, y i It is a feature set Y Elemental characteristics, d ( x i , y j ) is the feature to be measured. x i With sets Y Elemental characteristicsy i The distance between them;

[0045] The feature fusion formula is:

[0046] ;

[0047] In the formula, N X For multimodal fusion features, α The weights are dynamically adjusted based on cross-modal discrimination. O X It is a primitive feature. G X It is a characteristic of reasoning.

[0048] Furthermore, in the cross-modal knowledge retrieval, the similarity calculation formula is:

[0049] ;

[0050] In the formula, Sim ( N X , K i ) represents multimodal fusion features N X With each knowledge vector in the knowledge base K i The similarity between them is used to construct a knowledge subset { K 1, K 2, ..., K t}

[0051] Furthermore, the scene update and expansion includes the following steps:

[0052] Collect user feedback: Collect user feedback information after the model provides answers, and generate reward signals through a human feedback reinforcement learning framework;

[0053] Based on the reward signal, the text data is filtered to generate new text features. Then, based on the text features, cross-modal knowledge retrieval and intent and scene recognition are performed again, thereby generating new dialogue answers based on the updated image features, text features, knowledge features and recognition features.

[0054] The second aspect of this disclosure provides a question-answering data processing system based on a multimodal large model, which applies a question-answering data processing method based on a multimodal large model as described above, including a multimodal feature fusion unit, a cross-modal knowledge retrieval unit, an intent and scene recognition unit, and an answer generation and output unit, wherein the multimodal feature fusion unit, the cross-modal knowledge retrieval unit, the intent and scene recognition unit, and the answer generation and output unit are sequentially and communicatively connected.

[0055] The multimodal feature fusion unit is used to construct a relationship graph by extracting multimodal features, thereby inferring recognition features, and then dynamically adjust the weights by calculating the cross-modal discriminability of the original features and the inferred features to complete the multimodal feature fusion.

[0056] The cross-modal knowledge retrieval unit extracts feature vectors from existing databases to build a knowledge base, and constructs a knowledge subset by obtaining relevant knowledge vectors of multimodal fusion features based on similarity metrics, thereby completing the cross-modal retrieval of user input information.

[0057] As a preferred embodiment of the present invention, the intent and scene recognition unit is used to predict the intent and scene of the user's question through a knowledge subset retrieved based on the user's input data information using a machine learning model, wherein the construction of the machine learning model includes the following steps:

[0058] Dataset construction: A knowledge base is built using feature vectors from existing publicly available image and text data. Each feature vector in the knowledge base is automatically labeled, and information related to intent and scenario is extracted and labels are automatically generated.

[0059] Model training: The model is trained based on the constructed dataset by selecting a multimodal neural network as the machine learning model, and the cross-entropy loss function is selected as the loss function to complete the model optimization;

[0060] The answer generation and output unit is used to predict and generate the answer to each question by continuously updating image features, text features, knowledge features and recognition features during the multi-round question-and-answer process. During the multi-round question-and-answer process, the user's answer feedback is converted into a reward signal based on the human feedback reinforcement learning framework to guide the updating of text features.

[0061] The beneficial effects of this invention are as follows:

[0062] This invention first extracts features from multimodal information input by the user, and then dynamically adjusts the weights of the original and inferred features based on the calculation of cross-modal discriminability to fuse features, thereby providing more discriminative features for cross-modal retrieval. Next, it constructs a knowledge subset through cross-modal knowledge retrieval using multimodal fused features, providing richer input information for user intent and scene recognition, while also providing comprehensive relevant knowledge for initial question answer generation. Furthermore, it acquires extended information beyond existing data features through intent and scene recognition, thus achieving guided answer generation. Finally, through multiple rounds of question answering, it filters the feedback data based on user feedback using a human feedback reinforcement learning framework, iteratively optimizing the answer output, thereby improving the response quality and user satisfaction of the question answering model. Attached Figure Description

[0063] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0064] Figure 1 A flowchart illustrating a question-answering data processing method based on a multimodal large model provided in an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the process for obtaining multimodal fusion features according to an embodiment of the present invention;

[0066] Figure 3 A schematic diagram illustrating the process of constructing a relationship graph provided in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the structure of a question-answering data processing system based on a multimodal large model, provided in an embodiment of the present invention. Detailed Implementation

[0068] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0069] This embodiment provides a question-answering data processing method based on a multimodal large model, such as... Figure 1 As shown, it includes the following steps:

[0070] S1. Obtaining Multimodal Fusion Features: By constructing a relationship graph, fine-grained feature relationships in multimodal data are captured, completing the fusion and matching of cross-modal information, and obtaining the fused multimodal features, such as... Figure 2 As shown, it includes the following steps:

[0071] S11. Multimodal Feature Extraction: Extract modal features from the user-input image data and text data respectively. Image data feature extraction includes the following steps:

[0072] The image data input by the user is preprocessed, and multiple candidate boxes are generated based on RPN using a pre-trained Faster R-CNN model. The output of each candidate box includes its location (coordinates) and confidence score, indicating whether the box contains the target object.

[0073] It should be noted that the Faster R-CNN model is pre-trained on the Visual Genome dataset and exhibits good object detection capabilities. The model consists of two main parts: a Region Proposal Network (RPN) and a Region Classification Network. The RPN generates candidate boxes, while the Region Classification Network classifies and regresses these candidate boxes. The generated candidate boxes (Region Proposals) represent regions in the image that may contain the target object.

[0074] For each generated candidate box, its features are extracted using the feature extraction part of Faster R-CNN (typically a convolutional neural network):

[0075] The candidate bounding box regions are cropped from the original image, and the cropped image regions are input into a convolutional neural network for feature extraction to obtain the feature vector of each candidate bounding box.

[0076] Understandably, these feature vectors are typically high-dimensional and can effectively represent the appearance features of objects within the candidate bounding box.

[0077] The extracted candidate box features are mapped to a common feature space through a fully connected layer to obtain feature vectors, and a position code is generated for each candidate box. The image is divided into a K×K grid (meaning that the width and height of the image are divided into K equal parts respectively). The grid covered by the candidate box is coded as 1, and the grid not covered is coded as 0, thus forming the position information of each candidate box.

[0078] Among them, the feature vector after mapping to the common space O i Represented as:

[0079] O i =Tanh(W v v i +b v );

[0080] In the formula, Tanh() is the activation function. v i It is the first i The original feature vectors of the candidate boxes, W vIt is the weight matrix, b v It is the bias vector.

[0081] The feature vectors of all candidate boxes are combined with their corresponding location information to form a complete image feature representation.

[0082] Understandably, mapping features to a common space is to facilitate subsequent similarity measurement. Position encoding helps the model understand the spatial relationships of candidate boxes in the image, and the final output features are used for subsequent cross-modal retrieval tasks.

[0083] Feature extraction from text data includes the following steps:

[0084] The words in the text are mapped to a dictionary, and each word is assigned a unique code. This is achieved by building a vocabulary that contains all the words that appear in the training data.

[0085] Each word is mapped to a vector space using a pre-trained word embedding model (such as GloVe, Word2Vec, or FastText). This example uses GloVe, where each word is mapped to a 300-dimensional vector, denoted as:

[0086] {t1,…,t j , ..., t n};

[0087] Among them, t j is the 300-dimensional feature vector of the j-th word, and n is the length of the sentence.

[0088] The complete text description is input into the Bidirectional Gated Recurrent Unit (Bi-GRU) model to obtain the context information of each word and generate a new feature vector that represents the meaning of the word in the context.

[0089] GloVe (Global Vectors for Word Representation) is a model for generating word vectors. GloVe combines statistical information based on word co-occurrence with traditional word embedding methods based on local context.

[0090] Understandably, Bi-GRU can capture contextual information from two directions (forward and backward) of text, and the specific steps are as follows:

[0091] The input word embedding vector sequence is fed into Bi-GRU, which processes the input sequence through its gating mechanism to generate a contextual representation for each word.

[0092] S12. Relationship Graph Construction: Fine-grained features are used as nodes in the graph, and relationships between nodes are established through multiple subgraphs to obtain inference features, such as... Figure 3 As shown, it includes the following steps:

[0093] S121. Node definition: For image features, the extracted candidate box features are used as nodes. Each candidate box represents an object in the image, and its feature vector is used to describe the appearance of the object.

[0094] For the text modality, words in the text are used as nodes, and each word is mapped to a vector space through the GloVe model to form its feature representation.

[0095] S122. Construction of positional relationship graph: Divide the image into a K×K grid, with the grid covered by the candidate box coded as 1 and the grid not covered coded as 0, to form the positional information of each candidate box;

[0096] The relative positional relationship is established by calculating the cosine similarity between candidate boxes. The higher the similarity, the closer the relative positional relationship between the objects.

[0097] S123. Similarity graph construction: The appearance features of candidate boxes are mapped to the latent space through a fully connected layer, and then the cosine distance between the features of all candidate boxes is calculated as the edge of the graph.

[0098] Based on similarity, edge connections are established between candidate box features to form a similarity relationship graph, which reflects the relationship between objects in the image.

[0099] Understandably, the latent space is typically obtained by reducing the dimensionality of high-dimensional features. Mapping the appearance features of candidate boxes to the latent space using fully connected layers can reduce the dimensionality of features, extracting more important and abstract features. This helps to remove redundant information and retain the features most useful for the task.

[0100] S124. Text Modal Relationship Graph Construction: The words in the text are used for feature inference through a fully connected layer, and then the similarity between words is calculated to build a relationship graph between words;

[0101] By establishing edge connections between words based on similarity, a relationship graph of text modalities is formed, capturing the semantic associations between words.

[0102] S125. Subgraph reasoning: Graph Convolutional Network (GCN) is used to reason about the constructed relation graph. The reasoning process for each subgraph includes updating node features and aggregating information. All subgraph information is integrated to obtain updated image features and text features.

[0103] S13, Feature Fusion: Calculate the cross-modal discriminative power of the original features and the inferred features, dynamically adjust the weights of the original features and the inferred features to perform feature fusion, and output multimodal fused features;

[0104] The formula for calculating cross-modal discrimination is as follows:

[0105] ;

[0106] In the formula, Indicates the feature to be measured x i With another modal feature set Y Cross-modal discrimination, It is a feature set Y The number of features, y i It is a feature set Y Elemental characteristics, d ( x i , y j ) is the feature to be measured. x i With sets Y Elemental characteristics y i The distance between them.

[0107] The feature fusion formula is:

[0108] ;

[0109] In the formula, N X For multimodal fusion features, α The weights are dynamically adjusted based on cross-modal discrimination. O X It is a primitive feature. G X It is a characteristic of reasoning.

[0110] It should be noted that by dynamically adjusting the feature fusion strategy and weights, for example, when the cross-modal discriminative power of the original features is greater than that of the inference features, then... α The first weight is chosen to preserve more of the original features during feature fusion; when the cross-modal discriminative power of the original features is less than that of the inference features, then... αThe second weight is chosen to retain more of the dissimilar features during feature fusion. Understandably, higher cross-modal discriminative power translates to stronger discriminative ability in cross-modal retrieval; therefore, the first weight is closer to 1, and the second weight is closer to 0. The model can flexibly select more discriminative features based on their performance in cross-modal retrieval, thereby improving overall retrieval performance.

[0111] S2. Cross-modal knowledge retrieval: A knowledge base is built based on the feature vectors of existing publicly available image and text data. The similarity between the multimodal fusion features and all knowledge vectors in the knowledge base is calculated. A threshold is set to filter knowledge vectors and construct a knowledge subset. The similarity calculation formula is as follows:

[0112] ;

[0113] In the formula, Sim ( N X , K i ) represents multimodal fusion features N X With each knowledge vector in the knowledge base K i The similarity between them is used to construct a knowledge subset { K 1, K 2, ..., K t}

[0114] S3. Intent and Scene Recognition: Input the constructed knowledge subset into the machine learning model to recognize the intent and scene of the user's question and answer input, and output the recognition features of the guiding intent and scene prediction results.

[0115] It should be noted that by using a subset of knowledge obtained from multimodal fusion feature retrieval based on user input, the intent and context of the user's question can be predicted. This provides users with guiding intent and context when they lack domain knowledge or have unclear needs, while also avoiding the data sparsity problem in the early stages of question answering.

[0116] S4. Answer Generation and Output: By concatenating all the acquired features and inputting them into the BLIP decoder, a predicted answer is generated, including the following steps:

[0117] Image features, text features, knowledge features, and recognition features are concatenated to generate a fused multimodal feature representation;

[0118] The fused multimodal features are input into the BLIP decoder, which generates the answer through autoregression, progressively predicting the probability distribution of each word.

[0119] During the decoding process, the softmax function is used to calculate the probability distribution of each word and select the next word based on these probabilities until a complete answer is generated.

[0120] It should be noted that knowledge features are extracted through a subset of knowledge obtained from cross-modal knowledge retrieval. BLIP (Bootstrapping Language-Image Pre-training) is a model for visual question answering (VQA) and other multimodal tasks. Concatenation can be a simple vector concatenation or fusion using other methods (such as weighted summation). The decoder is typically a Transformer-based structure responsible for generating the final answer.

[0121] S5. Scene Update and Expansion: Through multiple rounds of dialogue, the identification features are continuously updated to gradually narrow down the scope of the problem, completing the iterative optimization and generation of user questions, including the following steps:

[0122] Collect user feedback: Collect user feedback information based on the answers given by the model, and generate reward signals through the Reinforcement Learning from Human Feedback (RLHF) framework;

[0123] Based on the reward signal, the text data is filtered to generate new text features. Then, based on the text features, cross-modal knowledge retrieval and intent and scene recognition are performed again, thereby generating new dialogue answers based on the updated image features, text features, knowledge features and recognition features.

[0124] Understandably, in RLHF (Real-Time High-Frequency Response), user feedback is typically converted into reward signals to guide model improvement. For example, positive user feedback (such as expressions of satisfaction or confirmation) can be used as a reward signal to reinforce the direction of the answer; negative feedback (such as clarification or correction) prompts the model to change its strategy to avoid similar answers. In this embodiment, the RLHF framework is used to convert user feedback data into reward signals to filter data features for re-entering knowledge retrieval and updating identification features, thereby gradually narrowing down the scope of questions, achieving accurate answers to user questions, and improving the accuracy of the answers.

[0125] This embodiment also provides a question-answering data processing system based on a multimodal large model, such as... Figure 4 As shown, it includes a multimodal feature fusion unit, a cross-modal knowledge retrieval unit, an intent and scene recognition unit, and an answer generation and output unit.

[0126] The multimodal feature fusion unit, cross-modal knowledge retrieval unit, intent and scene recognition unit, and answer generation and output unit are sequentially connected in communication.

[0127] The multimodal feature fusion unit is used to construct a relationship graph by extracting multimodal features, thereby inferring recognition features, and then dynamically adjust the weights to complete feature fusion by calculating the cross-modal discriminability of the original features and the inferred features.

[0128] Understandably, by constructing a relationship graph, fine-grained features can be extracted, which helps to more accurately describe the complex relationships between images and text. By calculating cross-modal discriminative power, the fusion weights of original features and inferred features can be dynamically adjusted, enabling the model to select more discriminative features for fusion according to the needs of specific tasks. Based on the performance of different modal features, the model can flexibly choose to retain original features or inferred features, which can effectively reduce information loss and improve the accuracy of cross-modal knowledge retrieval.

[0129] The cross-modal knowledge retrieval unit establishes a knowledge base by extracting feature vectors from existing databases, and constructs a knowledge subset by obtaining relevant knowledge vectors of multimodal fusion features based on similarity metrics, thereby enabling cross-modal retrieval of user input information.

[0130] It should be noted that cross-modal knowledge retrieval provides rich information input for intent and scene recognition when faced with poorly described questions and unknown knowledge to users, thereby generating more accurate and guiding answers to questions.

[0131] The intent and scenario recognition unit is used to predict the intent and scenario of the user's question using a subset of knowledge retrieved from the user's input data through a machine learning model. The construction of the machine learning model includes the following steps:

[0132] Dataset construction: A knowledge base is built using feature vectors from existing publicly available image and text data. Each feature vector in the knowledge base is automatically labeled, and information related to intent and scenario is extracted and labels are automatically generated.

[0133] Model training: The model is trained based on the constructed dataset by selecting a multimodal neural network as the machine learning model, and the cross-entropy loss function is selected as the loss function to complete the model optimization.

[0134] Understandably, automated annotation can generate descriptive text from image data, and then use natural language processing techniques to extract keywords or identify intent. Since the data contains both image and text features, a multimodal neural network is used for training, and the cross-entropy loss function is chosen as the loss function for the intent and scene classification problem.

[0135] The answer generation and output unit is used to predict and generate the answer to each question by continuously updating image features, text features, knowledge features and recognition features during the multi-round question-and-answer process. During the multi-round question-and-answer process, the user's answer feedback is converted into a reward signal based on the human feedback reinforcement learning framework to guide the updating of text features.

[0136] It should be noted that each answer is generated based on updated image and text features. Knowledge retrieval and intent and scene recognition are combined with the knowledge features and recognition features to predict and generate the answer. Therefore, in this cyclical question-and-answer process, the scope of the user's intent is continuously narrowed and corrected, so as to answer the user's questions in a dynamic and responsive manner, while expanding the direction of answer generation through recognition features.

[0137] This invention first extracts features from multimodal information input by the user, and then dynamically adjusts the weights of the original and inferred features based on the calculation of cross-modal discriminability to fuse features, thereby providing more discriminative features for cross-modal retrieval. Next, it constructs a knowledge subset through cross-modal knowledge retrieval using multimodal fused features, providing richer input information for user intent and scene recognition, while also providing comprehensive relevant knowledge for initial question answer generation. Furthermore, it acquires extended information beyond existing data features through intent and scene recognition, thus achieving guided answer generation. Finally, through multiple rounds of question answering, it filters the feedback data based on user feedback using a human feedback reinforcement learning framework, iteratively optimizing the answer output, thereby improving the response quality and user satisfaction of the question answering model.

[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A question-answering data processing method based on a multimodal large model, characterized in that: Includes the following steps: Obtaining multimodal fusion features: By constructing a relationship graph to capture fine-grained feature relationships in multimodal data, the fusion and matching of cross-modal information is completed, and the fused multimodal features are obtained. This includes the following steps: Multimodal feature extraction: Extracting modal features from user-input image data and text data respectively; Cross-modal knowledge retrieval: A knowledge base is built based on the feature vectors of existing public image and text data. By calculating the similarity between the multimodal fusion features and all knowledge vectors in the knowledge base, a threshold is set to filter knowledge vectors and construct a knowledge subset. Intent and Scene Recognition: Input the constructed knowledge subset into the machine learning model to recognize the intent and scene of the user's question and answer input, and output the recognition features of the guiding intent and scene prediction results; Answer generation and output: Image features, text features, knowledge features, and recognition features are concatenated and then input into the BLIP decoder. The decoder generates the answer through autoregression, progressively predicting the probability distribution of each word. Among them, knowledge features are extracted from a subset of knowledge obtained through cross-modal knowledge retrieval. During the decoding process, the softmax function is used to calculate the probability distribution of each word and the next word is selected based on the probability distribution until a complete answer is generated; Scene updates and expansion: Collect user feedback through multi-round dialogues, generate incentive signals based on user feedback to continuously update recognition features, thereby gradually narrowing down the scope of problems and completing iterative optimization of user problems; The acquisition of multimodal fusion features also includes the following steps: Feature extraction from image data includes the following steps: The image data input by the user is preprocessed, and multiple candidate boxes are generated based on RPN using a pre-trained Faster R-CNN model. The output of each candidate box includes the location and confidence score. For each generated candidate box, the candidate box region is cropped from the original image, and the cropped image region is input into a convolutional neural network for feature extraction to obtain the feature vector of each candidate box. The extracted candidate box features are mapped to the common feature space through a fully connected layer to obtain feature vectors, and a position code is generated for each candidate box. The image is divided into a K×K grid, with the grid covered by the candidate box coded as 1 and the grid not covered coded as 0, thus forming the position information of each candidate box. Among them, the feature vector after mapping to the common space O i Represented as: O i =Tanh(W v v i +b v ); In the formula, Tanh() is the activation function. v i It is the first i The original feature vectors of the candidate boxes, W v It is the weight matrix, b v It is the bias vector; The feature vectors of all candidate boxes are combined with their corresponding location information to form a complete image feature representation; The scenario update and expansion include the following steps: Collect user feedback: Collect user feedback information after the model provides answers, and generate reward signals through a human feedback reinforcement learning framework; Based on the reward signal, the text data is filtered to generate new text features. Then, based on the text features, cross-modal knowledge retrieval and intent and scene recognition are performed again, thereby generating new dialogue answers based on the updated image features, text features, knowledge features and recognition features.

2. The question-answering data processing method based on a multimodal large model according to claim 1, characterized in that: The acquisition of multimodal fusion features also includes the following steps: Relationship graph construction: Fine-grained features are used as nodes in the graph, and the relationships between nodes are established through multiple subgraphs to obtain inference features; Feature fusion: Calculate the cross-modal discriminative power of the original features and the inferred features, dynamically adjust the weights of the original features and the inferred features to perform feature fusion, and output multimodal fused features.

3. The question-answering data processing method based on a multimodal large model according to claim 1, characterized in that: In the multimodal feature extraction, the feature extraction of text data includes the following steps: The words in the text are mapped to a dictionary, and a unique code is assigned to each word by building a vocabulary. Each word is mapped to a vector space using a pre-trained word embedding model, represented as follows: {t1,…,t j ,…,t n }; Among them, t j is the 300-dimensional feature vector of the j-th word, and n is the length of the sentence; The complete text description is input into the bidirectional gated recurrent unit model to obtain the context information of each word and generate a new feature vector that represents the meaning of the word in the context.

4. The question-answering data processing method based on a multimodal large model according to claim 2, characterized in that: The construction of the relationship graph includes the following steps: Node definition: For image features, the extracted candidate box features are used as nodes, and each candidate box represents an object in the image; For the text modality, words in the text are used as nodes, and each word is mapped to a vector space through the GloVe model. Location relationship graph construction: The image is divided into a K×K grid. The grid covered by the candidate box is coded as 1, and the grid not covered is coded as 0, forming the location information of each candidate box. The relative location relationship is established by calculating the cosine similarity between candidate boxes. Similarity graph construction: The appearance features of candidate boxes are mapped to the latent space through a fully connected layer, and then the cosine distance between the features of all candidate boxes is calculated as the edge of the graph; Based on similarity, edge connections are established between candidate box features to form a similarity graph, which represents the relationship between objects in the image; Text modality graph construction: The words in the text are used for feature inference through fully connected layers, and then the similarity between words is calculated to build a relationship graph between words; Based on similarity, edge connections are established between words to form a relationship graph of text modalities, capturing the semantic associations between words; Subgraph reasoning: The constructed relational graph is reasoned using a graph convolutional network. The reasoning process for each subgraph includes updating node features and aggregating information. All subgraph information is integrated to obtain updated image features and text features.

5. The question-answering data processing method based on a multimodal large model according to claim 2, characterized in that: The formula for calculating the cross-modal discrimination is as follows: ； In the formula, Indicates the feature to be measured x i With another modal feature set Y Cross-modal discrimination, It is a feature set Y The number of features, y i It is a feature set Y Elemental characteristics, d ( x i , y j ) is the feature to be measured. x i With sets Y Elemental characteristics y i The distance between them; The feature fusion formula is: N X = αO X +(1- α ) G X ； In the formula, N X For multimodal fusion features, α The weights are dynamically adjusted based on cross-modal discrimination. O X It is a primitive feature. G X It is a characteristic of reasoning.

6. The question-answering data processing method based on a multimodal large model according to claim 1, characterized in that: In the cross-modal knowledge retrieval, the similarity calculation formula is as follows: ； In the formula, Sim ( N X , K i ) represents multimodal fusion features N X With each knowledge vector in the knowledge base K i The similarity between them is used to construct a knowledge subset { K 1, K 2, ..., K t } 7. A question-answering data processing system based on a multimodal large model, employing the question-answering data processing method based on a multimodal large model as described in any one of claims 1-6, characterized in that: It includes a multimodal feature fusion unit, a cross-modal knowledge retrieval unit, an intent and scene recognition unit, and an answer generation and output unit, which are sequentially connected in communication. The multimodal feature fusion unit is used to construct a relationship graph by extracting multimodal features, thereby inferring recognition features, and then dynamically adjust the weights by calculating the cross-modal discriminability of the original features and the inferred features to complete the multimodal feature fusion. The cross-modal knowledge retrieval unit extracts feature vectors from existing databases to build a knowledge base, and constructs a knowledge subset by obtaining relevant knowledge vectors of multimodal fusion features based on similarity metrics, thereby completing the cross-modal retrieval of user input information.

8. The question-answering data processing system based on a multimodal large model according to claim 7, characterized in that: The intent and scenario recognition unit is used to predict the intent and scenario of the user's question using a subset of knowledge retrieved from the user's input data through a machine learning model. The construction of the machine learning model includes the following steps: Dataset construction: A knowledge base is built using feature vectors from existing publicly available image and text data. Each feature vector in the knowledge base is automatically labeled, and information related to intent and scenario is extracted and labels are automatically generated. Model training: The model is trained based on the constructed dataset by selecting a multimodal neural network as the machine learning model, and the cross-entropy loss function is selected as the loss function to complete the model optimization; The answer generation and output unit is used to predict and generate the answer to each question by continuously updating image features, text features, knowledge features and recognition features during the multi-round question-and-answer process. During the multi-round question-and-answer process, the user's answer feedback is converted into a reward signal based on the human feedback reinforcement learning framework to guide the updating of text features.

Images