Question and answer data processing method and system based on multi-modal large model

Abstract

The invention relates to a question and answer data processing method and system based on a multi-modal large model, and belongs to the technical field of data processing. Comprising the following steps: capturing a fine-grained feature relationship in multiple modes by constructing a relational graph, and obtaining fused multi-mode features; by calculating the similarity between the multi-modal fusion features and all knowledge vectors in a knowledge base, setting a threshold value, screening the knowledge vectors, and constructing a knowledge subset; inputting the constructed knowledge subset into a machine learning model to perform intention and scene recognition on question and answer input of the user; splicing all the obtained features and then inputting the spliced features into a decoder of the BLIP to generate a prediction answer; user feedback is collected through multiple rounds of dialogues, and excitation signals are generated according to the user feedback to continuously update recognition features. According to the method, cross-modal knowledge retrieval is carried out through the multi-modal fusion features to construct the knowledge subsets, meanwhile, the recognition features are obtained based on the knowledge subsets, expandable guide answers are generated for users, and the accuracy and satisfaction degree of questions and answers are improved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a question-answering data processing method and system based on a multimodal large model. Background Art

[0002] In order to further expand the application scope of general understanding capabilities, multimodal large models have come into being. They can receive data from multiple sources, including images, audio, video, and even touch in addition to conventional text, which enables the model to understand higher-dimensional and richer information.

[0003] However, the existing AI question-and-answer model driven by large models mostly adopts the "user question-model answer" approach, that is, the user clearly expresses the demand, and the model answers the question after the user asks the question. This model works well when facing structured or simple questions, but when facing complex or ambiguous scenarios, especially when the user cannot clearly express the demand or the user himself needs to expand his thinking, the model is difficult to give an effective dialogue answer, and in the process of answer prediction and generation, the user's intention and the guidance of the question-and-answer scenario are often ignored, thus limiting the innovation of the generated answer. Summary of the invention

[0004] In order to solve the above-mentioned problems existing in the prior art, the present invention provides a question and answer data processing method and system based on a multimodal large model, which constructs a knowledge subset by performing cross-modal knowledge retrieval through multimodal fusion features, and obtains identification features based on the knowledge subset to generate extensible guided answers for users, thereby improving the accuracy and satisfaction of questions and answers.

[0005] The purpose of the present invention can be achieved through the following technical solutions: The first aspect of the present disclosure provides a method for processing question-answering data based on a multimodal large model, comprising the following steps: S1. Obtain multimodal fusion features: Capture the fine-grained feature relationships in multimodality by constructing a relationship graph, complete the fusion and matching of cross-modal information, and obtain the fused multimodal features; S2, cross-modal knowledge retrieval: Establish a knowledge base based on the feature vectors of existing public image data and text data, calculate the similarity between multimodal fusion features and all knowledge vectors in the knowledge base, set thresholds to filter knowledge vectors and construct knowledge subsets; S3, Intent and scenario recognition: Input the constructed knowledge subset into the machine learning model to perform intent and scenario recognition on the user's question and answer input, and output the identification features of the guiding intent and scenario prediction results; S4, answer generation and output: Generate the predicted answer by concatenating all the acquired features and inputting them into the BLIP decoder; S5, scenario update and expansion: Collect user feedback through multiple rounds of dialogue, generate incentive signals based on user feedback and continuously update recognition features, thereby gradually narrowing the scope of the problem and completing the iterative optimization generation of user problems; The answer generation and output specifically includes the following steps: After the image features, text features, knowledge features and recognition features are concatenated, they are input into the BLIP decoder. The decoder generates the answer through autoregression and gradually predicts the probability distribution of each word. During the decoding process, the softmax function is used to calculate the probability distribution of each word, and the next word is selected based on the probability distribution until a complete answer is generated.

[0006] Furthermore, the step of obtaining the multimodal fusion feature comprises the following steps: S11, multimodal feature extraction: extracting modal features of image data and text data input by the user respectively; S12, relationship graph construction: Use fine-grained features as nodes in the graph, and establish associations between nodes through multiple subgraphs to obtain reasoning features; S13, feature fusion: calculate the cross-modal discrimination of original features and inferred features, dynamically adjust the weights of original features and inferred features for feature fusion, and output multimodal fusion features.

[0007] Furthermore, in the multimodal feature extraction, the feature extraction of image data comprises the steps of: Preprocess the image data input by the user, and generate multiple candidate boxes based on RPN through the pre-trained Faster R-CNN model. The output of each candidate box includes the position and confidence score; For each generated candidate box, the candidate box area is cropped from the original image, and the cropped image area is input into the convolutional neural network for feature extraction to obtain the feature vector of each candidate box; The extracted candidate box features are mapped to the public feature space through the fully connected layer to obtain the feature vector, and a position code is generated for each candidate box. The image is divided into K×K grids, and the grids covered by the candidate box are coded as 1, and the grids not covered are coded as 0, forming the position information of each candidate box; Among them, the eigenvector after mapping to the common space O i It is expressed as: O i =Tanh(W v v i +b v ); Where Tanh (﹒) is the activation function, v i It is i The original feature vector of the candidate box, W v is the weight matrix, b v is the bias vector; The feature vectors of all candidate boxes and their corresponding position information are combined to form a complete image feature representation.

[0008] Furthermore, in the multimodal feature extraction, feature extraction of text data includes the steps of: Map the words in the text into a dictionary by building a vocabulary table and assigning a unique code to each word; Use the pre-trained word embedding model to map each word into a vector space, represented as: {t1,…,t j ,…,t n}; Among them, t j is the 300-dimensional feature vector of the jth word, and n is the length of the sentence; The complete text description is input into the bidirectional gated recurrent unit model to obtain the contextual information related to each word and generate a new feature vector representing the meaning of the word in the context.

[0009] Furthermore, the relationship diagram is constructed, comprising the following steps: S121, node definition: For image features, the extracted candidate box features are used as nodes, and each candidate box represents an object in the image; For the text modality, the words in the text are used as nodes, and each word is mapped to a vector space through the GloVe model; S122, constructing a position relationship graph: dividing the image into K×K grids, encoding the grids covered by the candidate boxes as 1, and encoding the uncovered grids as 0, to form the position information of each candidate box, and establishing a relative position relationship by calculating the cosine similarity between the candidate boxes; S123, similarity relationship graph construction: map the appearance features of the candidate boxes to the latent space through the fully connected layer, and then calculate the cosine distance between all candidate box features as the edge of the graph; Based on the similarity, edge connections are established between the candidate box features to form a similarity relationship graph to represent the association relationship between objects in the image; S124, text modality relationship graph construction: perform feature inference on words in the text through a fully connected layer, then calculate the similarity between words and establish a relationship graph between words; Establish edge connections between words based on similarity to form a relational graph of text modalities and capture the semantic associations between words; S125, subgraph reasoning: Use graph convolutional networks to reason about the constructed relationship graph. The reasoning process of each subgraph includes updating node features and aggregating information, integrating all subgraph information to obtain updated image features and text features.

[0010] Furthermore, the calculation formula of the cross-modal discrimination is: ; In the formula, Indicates the feature to be tested x i With another modal feature set Y The cross-modal discrimination of is a feature set Y The number of features, y i is a feature set Y The element characteristics of d ( x i , y j ) is the feature to be measured x i With Collection Y Characteristics of the elements y i The distance between The feature fusion formula is: ; In the formula, N X is the multimodal fusion feature. α is the weight dynamically adjusted according to the cross-modal discrimination, O X is the original feature, G X It is an inference feature.

[0011] Furthermore, in the cross-modal knowledge retrieval, the similarity calculation formula is: ; In the formula, Sim ( N X , K i ) is represented as a multimodal fusion feature N X And each knowledge vector in the knowledge base K i The similarity between them, constructing knowledge subsets { K 1,K 2, ..., K t}.

[0012] Furthermore, the scene updating and expansion includes the steps of: Collect user feedback: Collect feedback information from users based on the answers given by the model, and generate reward signals through the human feedback reinforcement learning framework; Based on the reward signal, feature screening is performed on text data to generate new text features, and then cross-modal knowledge retrieval and intent and scene recognition are performed again based on the text features, so as to generate new dialogue answers based on the updated image features, text features, knowledge features and recognition features.

[0013] A second aspect of the present disclosure provides a question-answering data processing system based on a multimodal large model, which applies a question-answering data processing method based on a multimodal large model as described above, including a multimodal feature fusion unit, a cross-modal knowledge retrieval unit, an intention and scene recognition unit, and an answer generation and output unit, wherein the multimodal feature fusion unit, the cross-modal knowledge retrieval unit, the intention and scene recognition unit, and the answer generation and output unit are sequentially connected in communication; The multimodal feature fusion unit is used to construct a relationship graph by extracting multimodal features, thereby inferring identification features, and then dynamically adjust weights to complete multimodal feature fusion by calculating cross-modal discrimination of original features and inferred features; The cross-modal knowledge retrieval unit extracts feature vectors from data in an existing database to establish a knowledge base, obtains relevant knowledge vectors of multi-modal fusion features based on similarity measurement to construct a knowledge subset, and completes cross-modal retrieval of user input information.

[0014] As a preferred technical solution of the present invention, the intention and scenario recognition unit is used to predict the intention and scenario of the user's question through a machine learning model based on a knowledge subset retrieved from the data information input by the user, wherein the construction of the machine learning model includes the following steps: Build a data set: Build a knowledge base through the feature vectors of existing public image data and text data, automatically annotate each feature vector in the knowledge base, extract information related to intent and scenario, and automatically generate labels; Model training: The model is trained based on the constructed dataset by selecting a multimodal neural network as the machine learning model, and the cross entropy loss function is selected as the loss function to complete the model optimization; The answer generation and output unit is used to predict and generate the answer to each question asked through the image features, text features, knowledge features and recognition features that are continuously updated during multiple rounds of question-answering, and to convert the user's answer feedback into a reward signal to guide the text feature update based on the human feedback reinforcement learning framework during the multiple rounds of question-answering.

[0015] The beneficial effects of the present invention are: The present invention first extracts features based on multimodal information input by users, and dynamically adjusts the weights of original features and inference features based on the calculation of cross-modal discriminability to perform feature fusion, thereby providing more discriminative features for cross-modal retrieval; then, cross-modal knowledge retrieval is performed through multimodal fusion features to construct knowledge subsets, providing richer input information for user intention and scene recognition, while providing comprehensive relevant knowledge for initial question answer generation, and obtaining information extended beyond existing data features through the recognition of intentions and scenes, thereby achieving guided answer generation; finally, through multiple rounds of question and answer, the feedback data is screened according to the user's feedback information through the human feedback reinforcement learning framework to convert the reward signal, iteratively optimize the answer output, and improve the response quality of the question and answer model and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0017] Figure 1 A flowchart of a method for processing question and answer data based on a multimodal large model provided by an embodiment of the present invention; Figure 2 A schematic diagram of a process for obtaining multimodal fusion features provided by an embodiment of the present invention; Figure 3 A schematic diagram of a process for constructing a relationship diagram provided in an embodiment of the present invention; Figure 4 A structural schematic diagram of a question-answering data processing system based on a multimodal large model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0019] This embodiment provides a question-answering data processing method based on a multimodal large model. Figure 1 As shown, the following steps are included: S1. Obtain multimodal fusion features: By constructing a relationship graph to capture the fine-grained feature relationship in multimodality, the fusion and matching of cross-modal information is completed, and the fused multimodal features are obtained, such as Figure 2 As shown, the following steps are included: S11, multimodal feature extraction: extracting modal features of image data and text data input by the user respectively, wherein the feature extraction of image data includes the steps of: The image data input by the user is preprocessed, and multiple candidate boxes are generated based on RPN through the pre-trained Faster R-CNN model. The output of each candidate box includes its position (coordinates) and confidence score, indicating whether the box contains the target object.

[0020] It should be noted that the Faster R-CNN model is pre-trained on the Visual Genome dataset and has good object detection capabilities. The model consists of two main parts: the Region Proposal Network (RPN) and the Region Classification Network; the RPN is used to generate candidate boxes, while the Region Classification Network classifies and regresses these candidate boxes. Among them, the generated multiple candidate boxes (Region Proposals) represent the areas in the image that may contain the target object.

[0021] For each generated candidate box, use the feature extraction part of Faster R-CNN (usually a convolutional neural network) to extract its features: The candidate box area is cropped from the original image, and the cropped image area is input into the convolutional neural network for feature extraction to obtain the feature vector of each candidate box.

[0022] It is understandable that these feature vectors are usually high-dimensional and can effectively represent the appearance characteristics of objects in the candidate boxes.

[0023] The extracted candidate box features are mapped to a common feature space through a fully connected layer to obtain feature vectors, and a position code is generated for each candidate box. The image is divided into K×K grids (indicating that the width and height of the image are divided into K equal parts respectively). The grids covered by the candidate box are coded as 1, and the grids not covered are coded as 0, forming the position information of each candidate box. Among them, the eigenvector after mapping to the common space O i It is expressed as: O i =Tanh(W v v i +b v ); Where Tanh (﹒) is the activation function, v i It is i The original feature vector of the candidate box, W v is the weight matrix, b v is the bias vector.

[0024] The feature vectors of all candidate boxes and their corresponding position information are combined to form a complete image feature representation.

[0025] It is understandable that mapping features to a common space is to facilitate subsequent similarity measurement. Position encoding can help the model understand the spatial relationship of candidate boxes in the image, and the final output features are used for subsequent cross-modal retrieval tasks.

[0026] Feature extraction of text data includes the following steps: Map the words in the text into a dictionary and assign a unique code to each word. This is done by building a vocabulary that contains all the words that appear in the training data.

[0027] Use a pre-trained word embedding model (such as GloVe, Word2Vec, or FastText) to map each word to a vector space. This example uses GloVe, and the word will be mapped to a 300-dimensional vector, denoted as: {t1,…,t j ,…,t n}; Among them, t j is the 300-dimensional feature vector of the j-th word, and n is the length of the sentence.

[0028] The complete text description is input into the Bidirectional Gated Recurrent Unit (Bi-GRU) model to obtain the contextual information related to each word and generate a new feature vector representing the meaning of the word in the context.

[0029] GloVe (Global Vectors for Word Representation) is a model for generating word vectors. GloVe combines statistical information based on word co-occurrence with traditional word embedding methods based on local context.

[0030] It is understandable that Bi-GRU can capture contextual information from two directions of the text (forward and backward). The specific steps are as follows: The input word embedding vector sequence is fed into the Bi-GRU, which processes the input sequence through its gating mechanism to generate a contextual representation of each word.

[0031] S12, Relationship graph construction: Use fine-grained features as nodes in the graph, and establish associations between nodes through multiple subgraphs to obtain inference features, such as Figure 3 As shown, the following steps are included: S121, node definition: For image features, the extracted candidate box features are used as nodes, each candidate box represents an object in the image, and its feature vector is used to describe the appearance of the object; For the text modality, words in the text are used as nodes, and each word is mapped to a vector space through the GloVe model to form its feature representation.

[0032] S122, constructing a position relationship graph: dividing the image into K×K grids, encoding the grids covered by the candidate box as 1, and encoding the uncovered grids as 0, thereby forming the position information of each candidate box; The relative position relationship is established by calculating the cosine similarity between the candidate boxes. The higher the similarity, the closer the relative position relationship between the objects is.

[0033] S123, similarity relationship graph construction: map the appearance features of the candidate boxes to the latent space through the fully connected layer, and then calculate the cosine distance between all candidate box features as the edge of the graph; Based on the similarity, edge connections are established between the candidate box features to form a similarity relationship graph, which reflects the association relationship between objects in the image.

[0034] It is understandable that the latent space is usually obtained by reducing the dimensionality of high-dimensional features. Mapping the appearance features of the candidate box to the latent space through the fully connected layer can reduce the dimension of the features and extract more important and abstract features, which helps to remove redundant information and retain the most useful features for the task.

[0035] S124, text modality relationship graph construction: perform feature inference on words in the text through a fully connected layer, then calculate the similarity between words and establish a relationship graph between words; Edge connections are established between words based on similarity to form a relational graph of text modalities and capture the semantic associations between words.

[0036] S125, Subgraph Reasoning: Use the graph convolutional network (GCN) to reason about the constructed relationship graph. The reasoning process of each subgraph includes updating node features and aggregating information, integrating all subgraph information to obtain updated image features and text features.

[0037] S13, feature fusion: calculate the cross-modal discrimination of original features and inferred features, dynamically adjust the weights of original features and inferred features for feature fusion, and output multi-modal fusion features; Among them, the calculation formula of cross-modal discrimination is: ; In the formula, Indicates the feature to be tested x i With another modal feature set Y The cross-modal discrimination of is a feature set Y The number of features, y i is a feature set Y The element characteristics of d ( x i , y j ) is the feature to be measured x i With Collection Y Characteristics of the elements y i The distance between.

[0038] The feature fusion formula is: ; In the formula, N X is the multimodal fusion feature. α is the weight dynamically adjusted according to the cross-modal discrimination, O X is the original feature, G X It is an inference feature.

[0039] It should be noted that by dynamically adjusting the feature fusion strategy and weight, if the cross-modal discrimination of the original feature is greater than the cross-modal discrimination of the inference feature, α Take the first weight so that more original features are retained during feature fusion; when the cross-modal discrimination of the original feature is less than the cross-modal discrimination of the inference feature, α The second weight is taken so that more push-off features are retained during feature fusion. It is understandable that the greater the cross-modal discrimination, the stronger the discrimination ability in cross-modal retrieval. Therefore, the first weight is closer to 1 and the second weight is closer to 0. The model can flexibly select more discriminative features according to the performance of different features in cross-modal retrieval, thereby improving the overall retrieval effect.

[0040] S2. Cross-modal knowledge retrieval: A knowledge base is established based on the feature vectors of existing public image data and text data. By calculating the similarity between the multimodal fusion features and all knowledge vectors in the knowledge base, a threshold is set to filter the knowledge vectors to construct a knowledge subset. The similarity calculation formula is: ; In the formula, Sim ( N X , K i ) is represented as a multimodal fusion feature N X And each knowledge vector in the knowledge base K i The similarity between them, constructing knowledge subsets { K 1, K 2, ..., K t}.

[0041] S3. Intent and scenario recognition: Input the constructed knowledge subset into the machine learning model to perform intent and scenario recognition on the user's question and answer input, and output the recognition features of the guiding intent and scenario prediction results.

[0042] It should be noted that the intention and scenario of the user's question are predicted by retrieving the knowledge subset obtained through multimodal fusion features based on user input, thereby providing the user with guiding intentions and scenarios when the user has unknown domain knowledge or expresses unclear needs, while also avoiding the data sparsity problem in the early question-and-answer process.

[0043] S4, answer generation and output: All the acquired features are concatenated and input into the decoder of BLIP to generate the predicted answer, including the following steps: Combine image features, text features, knowledge features and recognition features to generate a fused multimodal feature representation; The fused multimodal features are input into the BLIP decoder, which generates answers through autoregression and gradually predicts the probability distribution of each word. During decoding, a softmax function is used to calculate the probability distribution of each word, and the next word is selected based on these probabilities until a complete answer is generated.

[0044] It should be noted that knowledge features are extracted from knowledge subsets obtained through cross-modal knowledge retrieval. BLIP (Bootstrapping Language-Image Pre-training) is a model used for visual question answering (VQA) and other multimodal tasks. Concatenation can be simple vector concatenation or fusion through other methods (such as weighted summation). The decoder is usually a Transformer-based structure responsible for generating the final answer.

[0045] S5, scenario update and expansion: Through multiple rounds of dialogue, the identification features are continuously updated to gradually narrow the scope of the problem, and the iterative optimization generation of user problems is completed, including the following steps: Collect user feedback: Collect feedback information from users based on the answers given by the model, and generate reward signals through the Reinforcement Learning from Human Feedback (RLHF) framework; Based on the reward signal, feature screening is performed on text data to generate new text features, and then cross-modal knowledge retrieval and intent and scene recognition are performed again based on the text features, so as to generate new dialogue answers based on the updated image features, text features, knowledge features and recognition features.

[0046] It is understandable that in RLHF, user feedback is usually converted into reward signals to guide model improvement. For example, positive user feedback (such as expressing satisfaction or confirming information) can be used as a reward signal to strengthen the direction of the answer; negative feedback (such as clarification or correction) will prompt the model to change its strategy to avoid similar answers. In this embodiment, the RLHF framework is used to convert user feedback data into reward signals to filter data features for re-knowledge retrieval and identification feature updates, thereby gradually narrowing the scope of the problem, achieving accurate answers to user questions, and improving the accuracy of the answers.

[0047] This embodiment also provides a question-answering data processing system based on a multimodal large model. Figure 4 As shown, it includes a multimodal feature fusion unit, a cross-modal knowledge retrieval unit, an intention and scene recognition unit, and an answer generation and output unit.

[0048] The multimodal feature fusion unit, the cross-modal knowledge retrieval unit, the intention and scene recognition unit and the answer generation and output unit are communicatively connected in sequence.

[0049] The multimodal feature fusion unit is used to construct a relationship graph by extracting multimodal features, thereby inferring identification features, and then dynamically adjust weights to complete feature fusion by calculating the cross-modal discrimination of original features and inferred features.

[0050] It is understandable that by constructing a relationship graph, fine-grained features can be extracted, which helps to more accurately describe the complex relationship between images and texts; by calculating cross-modal discriminability, the fusion weights of original features and inference features can be dynamically adjusted, so that the model can select more discriminative features for fusion according to the needs of specific tasks, and flexibly choose to retain original features or inference features according to the performance of different modal features, which can effectively reduce information loss and improve the accuracy of cross-modal knowledge retrieval.

[0051] The cross-modal knowledge retrieval unit extracts feature vectors from data in an existing database to establish a knowledge base, obtains relevant knowledge vectors of multi-modal fusion features based on similarity measurement to construct a knowledge subset, and realizes cross-modal retrieval of user input information.

[0052] It should be noted that through cross-modal knowledge retrieval, when faced with unclear questions and knowledge unknown to users, rich information input is provided for intent and scene recognition, thereby generating more accurate and guiding answers to questions.

[0053] The intention and scenario recognition unit is used to predict the intention and scenario of the user's question through a machine learning model based on the knowledge subset retrieved from the data information input by the user, wherein the construction of the machine learning model includes the following steps: Build a data set: Build a knowledge base through the feature vectors of existing public image data and text data, automatically annotate each feature vector in the knowledge base, extract information related to intent and scenario, and automatically generate labels; Model training: The model is trained based on the constructed dataset by selecting a multimodal neural network as the machine learning model, and the cross entropy loss function is selected as the loss function to complete the model optimization.

[0054] It is understandable that automatic annotation can generate descriptive text for image data, and then extract keywords or identify intent based on natural language processing technology. Since the data contains image and text features, a multimodal neural network is used for training, and the cross entropy loss function is selected as the loss function for the classification of intent and scene.

[0055] The answer generation and output unit is used to predict and generate the answer to each question asked through the image features, text features, knowledge features and recognition features that are continuously updated during multiple rounds of question-answering, and to convert the user's answer feedback into a reward signal to guide the text feature update based on the human feedback reinforcement learning framework during the multiple rounds of question-answering.

[0056] It should be noted that each answer is generated based on the updated image features and text features, and the answer prediction is generated after the knowledge features obtained by knowledge retrieval and intent and scene recognition and the identification features are spliced ​​together. Therefore, in this cyclic question-and-answer process, the user's intention scope is continuously narrowed and corrected, so as to answer the questions raised by the user in a dynamic and responsive manner, and at the same time, the direction of answer generation is expanded through identification features.

[0057] The present invention first extracts features based on multimodal information input by users, and dynamically adjusts the weights of original features and inference features based on the calculation of cross-modal discriminability to perform feature fusion, thereby providing more discriminative features for cross-modal retrieval; then, cross-modal knowledge retrieval is performed through multimodal fusion features to construct knowledge subsets, providing richer input information for user intention and scene recognition, while providing comprehensive relevant knowledge for initial question answer generation, and obtaining information extended beyond existing data features through the recognition of intentions and scenes, thereby achieving guided answer generation; finally, through multiple rounds of question and answer, the feedback data is screened according to the user's feedback information through the human feedback reinforcement learning framework to convert the reward signal, iteratively optimize the answer output, and improve the response quality of the question and answer model and user satisfaction.

[0058] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for processing question-answering data based on a multimodal large model, characterized in that: The following steps are involved: Obtain multimodal fusion features: Capture the fine-grained feature relationships in multimodality by constructing a relationship graph, complete the fusion and matching of cross-modal information, and obtain the fused multimodal features; Cross-modal knowledge retrieval: Establish a knowledge base based on the feature vectors of existing public image data and text data. By calculating the similarity between the multimodal fusion features and all knowledge vectors in the knowledge base, set a threshold to filter the knowledge vectors and construct a knowledge subset. Intent and scenario recognition: Input the constructed knowledge subset into the machine learning model to perform intent and scenario recognition on the user's question and answer input, and output the identification features of the guiding intent and scenario prediction results; Answer generation and output: All acquired features are concatenated and input into the BLIP decoder to generate the predicted answer; Scenario update and expansion: Collect user feedback through multiple rounds of dialogue, generate incentive signals based on user feedback, and continuously update recognition features, thereby gradually narrowing the scope of the problem and completing the iterative optimization generation of user problems; The answer generation and output specifically includes the following steps: After the image features, text features, knowledge features and recognition features are concatenated, they are input into the BLIP decoder. The decoder generates the answer through autoregression and gradually predicts the probability distribution of each word. During the decoding process, the softmax function is used to calculate the probability distribution of each word, and the next word is selected based on the probability distribution until a complete answer is generated.

2. The method for processing question and answer data based on a multimodal large model according to claim 1, characterized in that: The multimodal fusion feature is obtained, The following steps are involved: Multimodal feature extraction: extract the modal features of the image data and text data input by the user respectively; Relationship graph construction: Use fine-grained features as nodes in the graph, and establish associations between nodes through multiple subgraphs to obtain reasoning features; feature Fusion: Calculate the cross-modal discrimination of original features and inferred features, dynamically adjust the weights of original features and inferred features for feature fusion, and output multimodal fusion features.

3. The method for processing question and answer data based on a multimodal large model according to claim 2, characterized in that: In the multimodal feature extraction, the feature extraction of image data includes the steps of: Preprocess the image data input by the user, and generate multiple candidate boxes based on RPN through the pre-trained Faster R-CNN model. The output of each candidate box includes the position and confidence score; For each generated candidate box, the candidate box area is cropped from the original image, and the cropped image area is input into the convolutional neural network for feature extraction to obtain the feature vector of each candidate box; The extracted candidate box features are mapped to the public feature space through the fully connected layer to obtain the feature vector, and a position code is generated for each candidate box. The image is divided into K×K grids, and the grids covered by the candidate box are coded as 1, and the grids not covered are coded as 0, forming the position information of each candidate box; Among them, the feature vector after mapping to the common space O i It is expressed as: O i =Tanh(W v v i +b v ); Where Tanh (﹒) is the activation function, v i It is i The original feature vector of the candidate box, W v is the weight matrix, b v is the bias vector; The feature vectors of all candidate boxes and their corresponding position information are combined to form a complete image feature representation.

4. The method for processing question and answer data based on a multimodal large model according to claim 3, characterized in that: In the multimodal feature extraction, feature extraction of text data includes the steps of: Map the words in the text into a dictionary by building a vocabulary table and assigning a unique code to each word; Use the pre-trained word embedding model to map each word into a vector space, represented as: {t1,…,t j ,…,t n }; Among them, t j is the 300-dimensional feature vector of the j-th word, and n is the length of the sentence; The complete text description is input into the bidirectional gated recurrent unit model to obtain the contextual information related to each word and generate a new feature vector representing the meaning of the word in the context.

5. The method for processing question and answer data based on a multimodal large model according to claim 2, characterized in that: The relationship diagram construction includes the following steps: Node definition: For image features, the extracted candidate box features are used as nodes, and each candidate box represents an object in the image; For the text modality, the words in the text are used as nodes, and each word is mapped to a vector space through the GloVe model; Position relationship graph construction: The image is divided into K×K grids. The grids covered by the candidate boxes are coded as 1, and the grids not covered are coded as 0. The position information of each candidate box is formed, and the relative position relationship is established by calculating the cosine similarity between the candidate boxes. Similarity relationship graph construction: Map the appearance features of the candidate boxes to the latent space through the fully connected layer, and then calculate the cosine distance between all candidate box features as the edges of the graph; Based on the similarity, edge connections are established between the candidate box features to form a similarity relationship graph to represent the association relationship between objects in the image; Text modality relationship graph construction: The words in the text are subjected to feature inference through a fully connected layer, and then the similarity between the words is calculated to build a relationship graph between the words; Establish edge connections between words based on similarity to form a relational graph of text modalities and capture the semantic associations between words; Subgraph reasoning: Use a graph convolutional network to reason about the constructed relationship graph. The reasoning process of each subgraph includes updating node features and aggregating information, integrating all subgraph information to obtain updated image features and text features.

6. The method for processing question and answer data based on a multimodal large model according to claim 2, characterized in that: The calculation formula of the cross-modal discrimination is: ； In the formula, Indicates the feature to be tested x i With another modal feature set Y The cross-modal discrimination of is a feature set Y The number of features, y i is a feature set Y The element characteristics of d ( x i , y j ) is the feature to be measured x i With Collection Y Characteristics of the elements y i The distance between The feature fusion formula is: ； In the formula, N X is the multimodal fusion feature. α is the weight dynamically adjusted according to the cross-modal discrimination, O X is the original feature, G X It is an inference feature.

7. The method for processing question and answer data based on a multimodal large model according to claim 1, characterized in that: In the cross-modal knowledge retrieval, the similarity calculation formula is: ； In the formula, Sim ( N X , K i ) is represented as a multimodal fusion feature N X And each knowledge vector in the knowledge base K i The similarity between them, constructing knowledge subsets { K 1, K 2, ..., K t }.

8. The method for processing question and answer data based on a multimodal large model according to claim 1, characterized in that: The scene updating and expansion includes the following steps: Collect user feedback: Collect feedback information from users based on the answers given by the model, and generate reward signals through the human feedback reinforcement learning framework; Based on the reward signal, feature screening is performed on text data to generate new text features, and then cross-modal knowledge retrieval and intent and scene recognition are performed again based on the text features, so as to generate new dialogue answers based on the updated image features, text features, knowledge features and recognition features.

9. A question-answering data processing system based on a multimodal large model, applying a question-answering data processing method based on a multimodal large model as claimed in any one of claims 1 to 8, characterized in that: It includes a multimodal feature fusion unit, a cross-modal knowledge retrieval unit, an intention and scene recognition unit, and an answer generation and output unit, wherein the multimodal feature fusion unit, the cross-modal knowledge retrieval unit, the intention and scene recognition unit, and the answer generation and output unit are sequentially connected in communication; The multimodal feature fusion unit is used to construct a relationship graph by extracting multimodal features, thereby inferring identification features, and then dynamically adjust weights to complete multimodal feature fusion by calculating cross-modal discrimination of original features and inferred features; The cross-modal knowledge retrieval unit extracts feature vectors from data in an existing database to establish a knowledge base, obtains relevant knowledge vectors of multi-modal fusion features based on similarity measurement to construct a knowledge subset, and completes cross-modal retrieval of user input information.

10. A question-answering data processing system based on a multimodal large model according to claim 9, characterized in that: The intention and scenario recognition unit is used to predict the intention and scenario of the user's question through a machine learning model based on the knowledge subset retrieved from the data information input by the user, wherein the construction of the machine learning model includes the following steps: Build a data set: Build a knowledge base through the feature vectors of existing public image data and text data, automatically annotate each feature vector in the knowledge base, extract information related to intent and scenario, and automatically generate labels; Model training: The model is trained based on the constructed dataset by selecting a multimodal neural network as the machine learning model, and the cross entropy loss function is selected as the loss function to complete the model optimization; The answer generation and output unit is used to predict and generate the answer to each question asked through the image features, text features, knowledge features and recognition features that are continuously updated during multiple rounds of question-answering, and to convert the user's answer feedback into a reward signal to guide the text feature update based on the human feedback reinforcement learning framework during the multiple rounds of question-answering.

Images