Cross-modal question and answer processing method and device and storage medium

Abstract

The invention discloses a cross-modal question and answer processing method and device and a storage medium, and belongs to the technical field of natural language processing. The method comprises the steps that in response to multi-modal data input by a user, modal data features corresponding to all data modals in the multi-modal data are extracted; based on an interactive attention mechanism, determining an information weight corresponding to each data mode; according to the information weight, fusing the modal data features corresponding to the data modals, and generating fused data features corresponding to the multi-modal data; and obtaining response data matched with the fused data features, and outputting the response data. According to the method and the device, the information weight between different modalities is dynamically adjusted, so that the most relevant modal information can be more accurately focused in the question and answer process, and the question and answer retrieval accuracy is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a cross-modal question and answer processing method, device and storage medium. Background Art

[0002] At present, with the rapid development of artificial intelligence and natural language processing technology, question-answering systems have become an important tool for information retrieval and user interaction. In related technologies, multimodal question-answering systems use deep learning technology to provide accurate and comprehensive question-answering services by training and understanding multimodal data models. However, when processing multimodal data, it is difficult to effectively integrate information from different modalities because the features of multimodal data are not in the same representation space, and the association between information in different modalities cannot be guaranteed. Therefore, in the process of generating answers by the multimodal question-answering system, it is impossible to fully utilize the complementary advantages of multimodal data and establish the association between different modalities, resulting in a low accuracy rate in answering questions.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a cross-modal question and answer processing method, device and storage medium, aiming to solve the technical problem that the related technology cannot fully utilize the complementary advantages between multimodal data and effectively integrate information of different modalities, resulting in reduced question and answer efficiency.

[0005] To achieve the above-mentioned purpose, the present application provides a cross-modal question and answer processing method, the method comprising the following steps: responding to the multimodal data input by the user, using the corresponding model to extract the modal data features corresponding to each modal data; then placing them in a shared representation space to unify the modal data features, thereby calculating the correlation score of the modal data features, and then performing a fusion operation to obtain the fused data features, matching the relevant response data according to the fused data features and outputting them.

[0006] In one embodiment, the step of determining the information weight corresponding to each of the data modalities based on the interactive attention mechanism includes: Mapping the modal data features to a shared representation space, and determining the information weight of the multimodal data; Calculating the relevance score of each of the modal data features according to the interactive attention mechanism; The information weight is adjusted based on the relevance score.

[0007] In one embodiment, the step of mapping the modal data to a shared representation space and determining the information weight of the multimodal data comprises: Mapping the modal data features to the shared representation space to obtain the same semantic information of the multimodal data features; Based on the same semantic information and the interactive attention mechanism, the information weight of the multimodal data is calculated.

[0008] In one embodiment, the step of fusing the modal data features corresponding to each of the data modalities according to the information weight to generate a fused data feature corresponding to the multimodal data includes: Based on the interactive attention mechanism, adjusting the information weight of the multimodal data to determine relevant modal features; The related modal features are compared and fused to determine the fused data features.

[0009] In one embodiment, after the step of comparing and fusing the relevant modal features to determine the fused data features, the method further includes: The relevant modal features are used as positive samples, and the irrelevant modal features are used as negative samples to input into the similarity model for training; Based on the preset contrast loss function, adjust the similarity of positive samples and negative samples.

[0010] In one embodiment, the step of adjusting the positive sample similarity and the negative sample similarity based on a preset contrast loss function includes: Adding a consistency regularization term to the preset contrast loss function, and calculating the consistency loss of the consistency regularization term; The positive sample similarity and the negative sample similarity are adjusted based on the consistency loss.

[0011] In one embodiment, the step of obtaining response data of the fused data feature matching and outputting the response data includes: Inputting the fused data features into a preset response model to determine the response data; The response data is converted into a target output format and the response data is output.

[0012] In one embodiment, the step of extracting modal data features corresponding to each data modality in the multimodal data in response to the multimodal data input by the user includes: Acquire the multimodal data and determine a modality training model according to the type of the multimodal data; Based on the modal training model, the multimodal data is encoded, and each of the multimodal data is converted into a modal data feature.

[0013] In addition, to achieve the above-mentioned objectives, the present application also provides a cross-modal question and answer processing method, device and apparatus, wherein the apparatus comprises: a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the cross-modal question and answer processing method, device and method as described above.

[0014] In addition, to achieve the above-mentioned objectives, the present application also provides a cross-modal question and answer processing method, device and various storage media, wherein the storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the cross-modal question and answer processing method, device and method as described above are implemented.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application aligns multimodal data features in the same shared representation space, adjusts the information weights corresponding to the multimodal data, performs multimodal data fusion, obtains target modal data, and performs consistency detection to achieve multimodal data fusion analysis of the question, thereby determining the target modal information most relevant to the question, and then inputs the preset answer model to match the reply results, effectively improving the accuracy of answering questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 This is a flow chart of Embodiment 1 of the cross-modal question-and-answer processing method of the present application; Figure 2 This is a flow chart of Embodiment 2 of the cross-modal question-and-answer processing method of the present application; Figure 3 This is a flowchart of Embodiment 3 of the cross-modal question-answering processing method of the present application; Figure 4 This is a flowchart of similarity adjustment in Embodiment 3 of the cross-modal question-answering processing method of this application; Figure 5 This is a flowchart of Embodiment 4 of the cross-modal question-and-answer processing method of the present application; Figure 6This is a flowchart of Embodiment 5 of the cross-modal question-and-answer processing method of the present application; Figure 7 It is a structural diagram of a cross-modal question and answer processing device in the hardware operating environment involved in the embodiment of the present application.

[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Multimodal data contains data of at least two different data modes, where the data modes include but are not limited to text, images, videos, audio, and sensor data. In question-answering systems, question-answering scenarios based on multimodal data are often involved. For example, after a user inputs a picture and a question into a multimodal question-answering system, the question-answering model needs to give corresponding response information based on the picture and the question.

[0023] However, when processing multimodal data, multimodal question-answering systems cannot effectively integrate information from different modalities because the features of multimodal data are not in the same representation space, and the association of information in different modalities cannot be guaranteed. Therefore, in the process of generating answers, the multimodal question-answering system cannot fully utilize the complementary advantages of multimodal data and establish associations between different modalities, resulting in a low accuracy rate in answering questions.

[0024] In order to solve the above problems, the present application proposes a cross-modal question-answering processing method. This method responds to the multimodal data input by the user, first extracts the modal data features corresponding to each data modality in the multimodal data, and then determines the information weight of each data modality, and fuses the modal data features based on the information weight, and then determines the corresponding response data according to the fused data features. In the above scheme, since the final response results of each data modality pair are assigned by weights during feature fusion, after calculating the information weights of each data modality, the fused data features can be further determined to match the corresponding response data with the fused data features.

[0025] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0026] It should be noted that the execution subject of this embodiment can be a cross-modal question-and-answer processing device, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes a cross-modal question-and-answer processing device as an example to illustrate this embodiment and the following embodiments.

[0027] Based on this, the first embodiment of the present application provides a cross-modal question-answering processing method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the cross-modal question and answer processing method of the present application.

[0028] In this embodiment, the cross-modal question-answering processing method includes steps S100 to S400: Step S100 , in response to multimodal data input by a user, extracting modal data features corresponding to each data modality in the multimodal data.

[0029] In this embodiment, multimodal data input by a user is first received, and then the multimodal data is preprocessed. After the preprocessing is completed, the sub-data corresponding to each different data modality are respectively input into the multimodal data to extract modal data features through the multimodal data.

[0030] As an optional solution for receiving multimodal data input by a user, the user may input at least two data belonging to different data modalities to the multimodal question answering system in different or the same manners, such as a real-time picture captured by a camera and an audio clip collected by a microphone, according to the multimodal data input by the user.

[0031] As another optional solution for receiving multimodal data input by the user, a user-friendly interface is first provided. The interface provides a text box or a chat window for inputting text, as well as functions for uploading pictures, audio and video recordings. At least two data belonging to different data modes are input to the multimodal question-answering system. For example, an audio file recorded by the user's mobile phone and a manually input text.

[0032] Optionally, after receiving the multimodal data, noise and redundant information in the multimodal data are first removed, and then the data of different modalities are converted into a unified format, for example, image data is converted into a Base64-encoded string, and audio data is converted into a specific audio format.

[0033] As an optional solution for extracting modal data features corresponding to each data mode in multimodal data, the type of multimodal data input by the user is determined, and the model required for multimodal data processing is determined based on the multimodal data type, so as to call the corresponding model to process the multimodal data and obtain the corresponding modal data features.

[0034] For example, after word segmentation of text data, a pre-trained word embedding language processing model (Bidirectional Encoder Representations from Transformers, BERT) is used to convert words in the text into high-dimensional vector representations; a convolutional neural network (CNN) is used to extract high-level features of image data to capture the complex structure and semantic information of the image; and a recurrent neural network (RNN) and a convolutional neural network are used to extract high-level features of audio.

[0035] Step S200: Based on the interactive attention mechanism, determine the information weight corresponding to each of the data modalities.

[0036] It should be noted that the correlation and importance between different modalities are calculated based on the interactive attention mechanism, and the contribution of each modality in the final decision or representation is dynamically adjusted. The information weight represents the relative importance of different modal data in the model decision process, and the correlation of modal information is optimized by adjusting the information weight.

[0037] Therefore, in this embodiment, after obtaining the modal data features, the correlation between different modal data is calculated, and then based on the correlation score, a normalization function is applied to convert it into an attention weight so that the sum of the weights of all modalities is 1, thereby determining the information weight corresponding to each data modality.

[0038] As an optional solution for calculating the correlation between data of different modalities, the dot product calculation method can be used to represent the data of different modalities in vector form, normalize these vectors, and then calculate the dot product between the normalized vectors to obtain the correlation score.

[0039] As another optional solution for calculating the correlation between data of different modalities, in addition to calculating the correlation between data of different modalities, data of different modalities can be represented as vectors, and cosine similarity can be used to calculate the cosine value of the angle between these vectors to obtain the cosine similarity as the correlation score.

[0040] Before calculating the cosine similarity, the objects are normalized, that is, each component of the vector is divided by the modulus of the vector, which can ensure that the similarity value is between -1 and 1. It should be noted that the closer the cosine similarity value is to 1, the higher the similarity of the two vectors in direction, that is, the stronger the correlation between them, and the closer the cosine similarity value is to -1, the greater the difference between the two vectors in direction, that is, the weaker the correlation between them, and when the cosine similarity value is 0, it means that the two vectors are orthogonal in direction and there is no correlation between them.

[0041] Specifically, after the correlation between different modal data is obtained, the information weight corresponding to the data modality is determined by calculating the correlation score of the modal data.

[0042] Step S300: According to the information weight, the modal data features corresponding to each of the data modalities are fused to generate fused data features corresponding to the multimodal data.

[0043] It should be noted that fusing the features of multimodal data is a complex but important process. After extracting the modal features of multimodal data, the importance of each modal feature to the task is evaluated, and then the features of each modality are fused according to the information weight corresponding to the importance.

[0044] As an optional solution to fuse the features of each modality according to the information weight, each feature and its weight can be multiplied and summed to calculate the fused data feature, which can capture the cross-modal correlation and complementarity in the original data.

[0045] Exemplarily, emotion-related features, such as the frequency of emotional vocabulary and emotional tendency, are extracted from the text content input by the user, and features, such as color saturation, brightness, and facial expression features, are extracted from the input image data. The text data features are evaluated according to the richness of emotional vocabulary and the clarity of emotional tendency in the text during weight evaluation, and the image data features are evaluated according to the clarity of expression and color saturation in the image. The specific value of the weight is determined by a machine learning algorithm. Then, a feature vector is constructed for each modal data, and a fused data feature that combines multiple modal features is obtained by multiplying and summing each feature vector with its corresponding weight.

[0046] As another option to fuse the modalities based on information weights, the data of each modality is first preprocessed, including data cleaning, normalization, and vectorization, and meaningful features of each modality are extracted. Based on the evaluated information weights, a complex deep learning model is used to automatically learn the interactions and weight distribution between the features of different modalities, thereby fusing the features of each modality.

[0047] Step S400: Acquire response data of the fusion data feature matching, and output the response data.

[0048] It should be noted that the target answer refers to matching the corresponding target modal information based on the target question input by the user, and then passing it to the preset answer model to generate the corresponding answer.

[0049] In this embodiment, a trained machine learning model is used to receive fused data features as input and output corresponding response data.

[0050] As an optional implementation, in a smart home system, the user interacts with the system through language, gestures, etc. The system can generate corresponding control instructions according to input data of different modes.

[0051] Exemplarily, the user's voice commands are collected through a microphone, the user's gesture movements are captured through a camera, the voice data and gesture data are recognized, and the voice features and gesture features are extracted respectively. Then, the text features and gesture features are fused using the weight and feature product summation method to determine the fused data features, and the fused data features are input into the trained model to generate corresponding control commands, i.e., response data.

[0052] In this embodiment, a cross-modal question-answering processing method is provided, which extracts modal data features by collecting and processing different modal data including text, images, videos and audio contents, adjusts the information weight of each modal data through an attention mechanism, and fuses these data to determine the most relevant fused data features for a specific query. When responding to a specific question, an accurate answer is generated by matching the most relevant fused data features for the question, which effectively enhances the system's ability to understand and process multimodal data and improves the accuracy of answering questions.

[0053] Based on Example 1, Example 2 of the present application provides a cross-modal question-answering processing method, referring to Figure 2 , Figure 2 This is a flow chart of Embodiment 2 of the cross-modal question-answering processing method of the present application, wherein step S200 includes steps S210 to S230: Step S210: Mapping the modal data features to a shared representation space, and determining the information weight of the multimodal data.

[0054] It should be noted that the shared representation space refers to a unified mathematical space that maps data from different modalities into a unified representation space. In the shared representation space, the semantic information between modalities is aligned, enabling the system to directly compare and fuse between different modalities.

[0055] Optionally, step S210 includes steps S211-S212: Step S211, mapping the modal data features to the shared representation space to obtain the same semantic information of the multimodal data features.

[0056] It should be noted that the same semantic information is the same or similar semantic content extracted from features of data of different modalities.

[0057] As an optional implementation, multimodal data features are mapped to a shared representation space, and identical or similar semantic information between multimodal data features is extracted.

[0058] For example, the text description "cat" and the image of the cat in the picture are obtained, and the corresponding text data and picture data are mapped to the shared representation space. It can be found that the multimodal data have the same semantic information, that is, the concept of "cat".

[0059] Step S212: calculating the information weight of the multimodal data based on the same semantic information and the interactive attention mechanism.

[0060] After extracting the modal data features from the original data, a mapping method is selected based on the component analysis between the modal data features. Then, according to the selected mapping method, the corresponding machine learning model is constructed. Then, the model is trained using multimodal data pairs annotated with the same semantic information. According to the optimization goal of the interactive attention mechanism, the distance between different modal data in the shared representation space is minimized or the correlation between them is maximized.

[0061] As an alternative to selecting a mapping method based on component analysis between modal data features, it is determined whether to use a linear method or a nonlinear method according to the linear relationship between multimodal data.

[0062] Specifically, linear methods include principal component analysis and canonical correlation analysis, and nonlinear methods include autoencoders, neural networks, deep belief networks, etc., which can capture the nonlinear relationship between data.

[0063] As an alternative to training models with multimodal data pairs annotated with the same semantic information, image and text data are aligned in the semantic space, guiding the model to learn to map data of different modalities to the same semantic space and fine-tune it on the dataset according to the specific task.

[0064] Step S220, calculating the correlation score of each of the modal data features according to the interactive attention mechanism.

[0065] The modal data features are input into the attention function as queries in the form of key-value pairs, the attention weights between the query and all key-value pairs are calculated, and the weighted sum is taken for each modal data feature to calculate the correlation scores between different modal data features.

[0066] As an alternative to calculating the attention weights between all key-value pairs, the similarity between the query and the key-value pairs is calculated. The query and the key-value pairs are input as a query matrix, and the score between the query and the key is calculated using the dot product. The score is then converted into a probability distribution and the output result is an attention weight matrix.

[0067] It should be noted that the interactive attention mechanism enables the system to dynamically adjust the information weights of different modalities according to the characteristics of the input data, thereby ensuring that the most relevant modal information can be focused more accurately during the question-answering process.

[0068] Step S230: adjusting the information weight based on the relevance score.

[0069] After obtaining the correlation scores of the modal data features, a weight adjustment strategy is determined to calculate the adjusted weight of each modal data feature, which can better reflect the relative importance of different modal data in the fusion process.

[0070] As a strategy for determining weight adjustment, the adjusted weight options for each modal data feature are calculated, the relevance scores are normalized, and the relevance scores are converted into weights to adjust the information weights.

[0071] This embodiment adopts a cross-modal question-answering processing method, which first maps multimodal data features to a shared representation space to determine their respective information weights, and calculates the correlation scores between modal features through an interactive attention mechanism, and then adjusts the information weights based on these correlation scores, thereby optimizing the fusion and processing of multimodal data.

[0072] Based on Example 1, Example 3 of the present application provides a cross-modal question-answering processing method, referring to Figure 3 , Figure 3 This is a flow chart of Embodiment 3 of the cross-modal question-answering processing method of the present application, where step S300 includes steps S310 to S320: Step S310: adjusting the information weight of the multimodal data based on the interactive attention mechanism to determine relevant modal features.

[0073] Step S320, comparing and fusing the relevant modal features to determine the fused data features.

[0074] Based on the interactive attention mechanism, feature inputs from different modules are received, and the correlation or similarity between different modal features is calculated using clicks, cosine similarity, etc., and then the Softmax function is applied to convert the correlation scores into weights, and the attention weights between them are calculated based on the calculated circles. According to the attention weights, the modal features most relevant to the current task are selected. If feasible, by setting a threshold, only features with weights higher than the threshold are retained. After that, the relevant modal features are compared to identify which features are complementary and which are redundant. Then the relevant modal features are weighted summed and fused to form a unified fusion data feature representation.

[0075] As an optional implementation, after the multimodal data is input into the shared representation space, the attention mechanism is used to capture the interaction information between different modalities, and the weight of each modality is calculated based on this interaction information. Under the same semantic information, the more relevant modality will be given a higher weight, and the multi-data modal information is compared and fused to determine the target modal information.

[0076] Exemplarily, a data set containing three modalities, text, image, and audio, and the question to be answered, "What is the emotional state of the protagonist in the video?", is obtained. Based on the attention mechanism, weights are assigned according to the keywords in the text, the expressions and actions in the image, and the intonation in the audio. For example, when recognizing the emotion "anger", if the keyword "angry" appears in the text, or the image shows an angry expression, or the tone in the audio shows anger, the weight of the corresponding modality will be increased accordingly. At the same time, these modal information are compared and fused to obtain the relevant modal information of "anger", which can be subsequently input into the answer model to quickly obtain the answer to the question at this time, "The emotional state of the protagonist in the video is anger."

[0077] Optionally, in this embodiment, after step S320, steps S330 to S340 are further included, referring to Figure 4 , Figure 4 The flowchart of steps S330 to S340 is as follows: Step S330: input the relevant modal features as positive samples and the irrelevant modal features as negative samples into a similarity model for training.

[0078] Step S340: adjusting the positive sample similarity and the negative sample similarity based on a preset contrast loss function.

[0079] By using contrastive learning, relevant modalities are used as positive samples and irrelevant modal features are used as negative samples. By adjusting the similarity between positive and negative samples, the model can encode the similarities and differences of multimodal data in the feature space. The model can more accurately identify and process the correlation of multimodal data.

[0080] As an optional implementation, the positive and negative sample pairs are processed separately by two branches of the Siamese deep learning model. The model is trained by a preset cosine similarity loss to maximize the similarity of the positive sample pairs and minimize the similarity of the negative sample pairs.

[0081] Exemplarily, a Siamese deep learning model is used to compare the similarity between face images and face text descriptions. The two branches of the network process images and texts respectively, where the positive sample pairs are the facial feature data of the same person, and the negative sample pairs are two unrelated modal data, such as facial feature images and hand text descriptions. By minimizing the Euclidean distance between positive sample pairs and maximizing the Euclidean distance between negative sample pairs, the Siamese deep learning model learns to distinguish similar and dissimilar facial features.

[0082] It should be noted that the Siamese deep learning model is a special type of neural network used to compare the similarity of two sets of input data, and the Euclidean distance is a method for measuring the distance between sample points in the feature space.

[0083] As another optional implementation, the Triplet deep learning model is trained by a triplet loss function, using three samples: an anchor point sample, a positive sample, and a negative sample. The training method is to make the distance between the anchor point and the positive sample smaller than the distance between the anchor point and the negative sample.

[0084] For example, a triple is constructed, where the anchor is the face feature text, the positive sample is the face image related to the face feature, and the negative sample is the hand image unrelated to the face feature. By minimizing the distance between the anchor and the positive sample and maximizing the distance between the anchor and the negative sample, the model learns how to identify relevant and irrelevant target images based on the query text.

[0085] It should be noted that the triplet loss function is a contrastive loss function used to train deep learning models.

[0086] Optionally, step S340 includes steps S341-S342: Step S341, adding a consistency regularization term to the preset contrast loss function, and calculating the consistency loss of the consistency regularization term.

[0087] Step S342: adjusting the positive sample similarity and the negative sample similarity based on the consistency loss.

[0088] As an optional implementation, the contrastive loss function (such as triplet loss) and the consistency regularization term are combined into a joint loss function. Then training is performed by minimizing this joint loss function, where the contrastive loss part is responsible for bringing the positive samples closer and pushing the negative samples further apart, while the consistency regularization term ensures the consistency of cross-modal data.

[0089] For example, when analyzing text and audio data, BERT is used to extract text features, and CNN is used to extract audio features to define a joint loss function, which includes a triplet loss and a consistency regularization term. The triplet loss is responsible for distinguishing between emotionally consistent text-audio pairs (positive samples) and inconsistent pairs (negative samples), while the consistency regularization term ensures that the text and audio features are consistent in the shared representation space. For example, if the text represents "happy", then the corresponding audio (laughter) should be closer in the shared space, and the consistency regularization term penalizes any feature representation that leads to inconsistency.

[0090] It should be noted that in order to further ensure the relevance of cross-modal data, a consistency regularization term is introduced. Through regularization technology, the model can avoid inconsistency problems in cross-modal data, such as the situation where the information in the image does not match the corresponding text description.

[0091] As another optional implementation, a model with a multi-task learning framework is used, where one task is to minimize the contrastive loss function and the other task is to minimize the consistency regularization term. These two tasks share the underlying feature representation but have different loss functions. The model is trained by optimizing these two tasks simultaneously, thereby maintaining consistency across modal data while learning to distinguish between positive and negative samples.

[0092] For example, in a multimodal question-answering system, users can upload images and ask related questions. The model is designed as a multi-task learning framework, where one task is to use a contrastive loss function to distinguish between image and answer pairs related to the question (positive samples) and irrelevant pairs (negative samples), and the other task is to use a consistency regularization term to ensure that the image and answer are consistent in the shared representation space.

[0093] This embodiment provides a cross-modal question-answering processing method, which first maps multimodal data features to a shared representation space and determines information weights to establish a unified comparison benchmark for data of different modalities, thereby identifying and quantifying the contribution of each modality to the overall task. Then, the interactive attention mechanism is used to adjust the information weights according to these features. The model can dynamically focus on the most relevant modal information and fuse multimodal data to determine the target modal information, which can enhance the model's comprehensive understanding of data and improve its performance in complex scenarios, such as providing more accurate answers or predictions in multimodal question-answering, sentiment analysis, or recommendation systems.

[0094] Based on any of the above embodiments, the fourth embodiment of the present application provides a cross-modal question and answer processing method, referring to Figure 5 , Figure 5 This is a flow chart of the fourth embodiment of the cross-modal question-answering processing method of the present application, where step S400 includes steps S410 to S420: Step S410: input the fused data features into a preset response model to determine the response data.

[0095] It should be noted that the preset response model is the result of outputting a response after matching the input fusion data features with a question. If feasible, the response data is the predicted result of the question.

[0096] In this embodiment, the fused data features are input into a preset response model, and the model analyzes and processes these fused features, and finally determines and outputs the response data.

[0097] Step S420: convert the response data into a target output format and output the response data.

[0098] After obtaining the response data, in order to make it meet specific application requirements or display requirements, the response data needs to be converted into the target output format.

[0099] Specifically, if the response data is an image, the target output format may be to convert it into a specific resolution or image format, such as JPEG or PNG, to meet display or storage requirements; if the response data is text, it may be organized in a specific format, such as JSON or XML, to facilitate data parsing and use; if the response data is a numeric value, it may be rounded, truncated, or converted into a percentage according to the application scenario.

[0100] This embodiment provides a cross-modal question and answer processing method, which inputs the fused data features into a preset response model to generate response data, then converts the response data into the required target output format, and finally outputs the data for further use or display, thereby achieving flexible generation of answer content and ensuring the logic and accuracy of the answers.

[0101] Based on any of the above embodiments, Embodiment 5 of the present application provides a cross-modal question-answering processing method, referring to Figure 6 , Figure 6 This is a flowchart of Embodiment 5 of the cross-modal question-answering processing method of the present application, where step S100 includes steps S110 to S120: Step S110, acquiring the multimodal data and determining a modality training model according to the type of the multimodal data.

[0102] Collect multimodal data through appropriate methods, preprocess the multimodal data, and select a suitable modal training model based on the modal type and feature extraction method of the multimodal data according to the type and format of the data.

[0103] It should be noted that based on the type of multimodality, which can be pictures, videos, audios, texts, etc., the corresponding processing model is determined.

[0104] Exemplarily, the text data uses the BERT training model, the video data uses the recursive neural network training model, the audio uses the voiceprint recognition model, and the image data uses the CNN training model.

[0105] Step S120: encoding the multimodal data based on the modal training model, and converting the multimodal data into modal data features.

[0106] Identify and collect multiple multimodal data, then classify them into different modalities based on the type and format of the data, and use the modal training model to encode the multimodal data; the encoding features extract the modal data features.

[0107] It should be noted that data encoding refers to converting raw data into a format or representation so that the machine learning model can process and analyze it more easily. In this embodiment, encoding is to convert data into feature vectors using a specific algorithm or model.

[0108] As an optional implementation, a separate encoder grid is designed for each modality of data, such as using CNN to encode image data, RNN or Transformer to encode text data, and convolutional or recurrent networks to encode audio data. The encoder converts the raw data into a high-dimensional feature vector representation.

[0109] For example, in a multimodal question-answering system, the input images, voice intonations, and comment texts are received, and CNN is used to extract image features, RNN is used to extract voice signal features, and BERT is used to extract text features. Each encoder works independently and converts the input data into feature vectors.

[0110] This embodiment provides a cross-modal question-answering processing method, which encodes native multimodal data through a processing model corresponding to the data type, thereby converting the data into a high-dimensional feature representation, thereby providing an actionable data representation for data analysis, pattern recognition, or machine learning tasks.

[0111] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the cross-modal question-and-answer processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0112] The present application provides a cross-modal question and answer processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the cross-modal question and answer processing method in the above-mentioned embodiment one.

[0113] Reference below Figure 7 , which shows a schematic diagram of the structure of a cross-modal question-answering processing device suitable for implementing the embodiment of the present application. The cross-modal question-answering processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The cross-modal question and answer processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0114] like Figure 7As shown, the cross-modal question-answering processing device may include a processing device 1001 (e.g., a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the cross-modal question-answering processing device are also stored in RAM1004. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the cross-modal question-and-answer processing device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a cross-modal question-and-answer processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0115] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0116] The cross-modal question and answer processing device provided by the present application adopts the cross-modal question and answer processing method in the above-mentioned embodiment, which can solve the technical problems of cross-modal question and answer processing. Compared with the prior art, the beneficial effects of the cross-modal question and answer processing device provided by the present application are the same as the beneficial effects of the cross-modal question and answer processing method provided by the above-mentioned embodiment, and the other technical features in the cross-modal question and answer processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0117] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0118] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0119] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the cross-modal question-and-answer processing method in the above-mentioned embodiment.

[0120] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0121] The computer-readable storage medium may be included in the cross-modal question and answer processing device; or may exist independently without being assembled into the cross-modal question and answer processing device.

[0122] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the cross-modal question-answering processing device, the cross-modal question-answering processing device can write computer program codes for performing the operations of the present application in one or more programming languages ​​or a combination thereof. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0125] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned cross-modal question-answering processing method, and can solve the technical problems of cross-modal question-answering processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the cross-modal question-answering processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0126] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A cross-modal question-answering processing method, characterized in that: The cross-modal question-answering processing method comprises the following steps: In response to multimodal data input by a user, extracting modal data features corresponding to each data modality in the multimodal data; Based on the interactive attention mechanism, determining the information weight corresponding to each of the data modalities; According to the information weight, the modal data features corresponding to each of the data modalities are fused to generate fused data features corresponding to the multimodal data; Acquire response data of the fused data feature matching, and output the response data.

2. The cross-modal question-answering processing method according to claim 1, characterized in that: The step of determining the information weight corresponding to each of the data modalities based on the interactive attention mechanism includes: Mapping the modal data features to a shared representation space, and determining the information weight of the multimodal data; Calculating the relevance score of each of the modal data features according to the interactive attention mechanism; The information weight is adjusted based on the relevance score.

3. The cross-modal question-answering processing method according to claim 2, characterized in that: The step of mapping the modal data to a shared representation space and determining the information weight of the multimodal data comprises: Mapping the modal data features to the shared representation space to obtain the same semantic information of the multimodal data features; Based on the same semantic information and the interactive attention mechanism, the information weight of the multimodal data is calculated.

4. The cross-modal question-answering processing method according to claim 1, characterized in that: The step of fusing the modal data features corresponding to each of the data modalities according to the information weight to generate a fused data feature corresponding to the multimodal data comprises: Based on the interactive attention mechanism, adjusting the information weight of the multimodal data to determine relevant modal features; The related modal features are compared and fused to determine the fused data features.

5. The cross-modal question-answering processing method according to claim 4, characterized in that: After the step of comparing and fusing the relevant modal features to determine the fused data features, the method further includes: The relevant modal features are used as positive samples, and the irrelevant modal features are used as negative samples to input into a similarity model for training; Based on the preset contrast loss function, adjust the similarity of positive samples and negative samples.

6. The cross-modal question-answering processing method according to claim 5, characterized in that: The step of adjusting the positive sample similarity and the negative sample similarity based on the preset contrast loss function includes: Adding a consistency regularization term to the preset contrast loss function, and calculating the consistency loss of the consistency regularization term; The positive sample similarity and the negative sample similarity are adjusted based on the consistency loss.

7. The cross-modal question-answering processing method according to claim 1, characterized in that: The step of obtaining response data of the fusion data feature matching and outputting the response data includes: Inputting the fused data features into a preset response model to determine the response data; The response data is converted into a target output format and the response data is output.

8. The cross-modal question-answering processing method according to claim 1, characterized in that: The step of extracting modal data features corresponding to each data modality in the multimodal data in response to the multimodal data input by the user comprises: Acquire the multimodal data and determine a modality training model according to the type of the multimodal data; Based on the modal training model, the multimodal data is encoded, and each of the multimodal data is converted into a modal data feature.

9. A cross-modal question-answering processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the cross-modal question-answering processing method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the cross-modal question and answer processing method as described in any one of claims 1 to 8 are implemented.

Images