Multi-modal data retrieval, generation and synthesis method and system based on artificial intelligence driving

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal data retrieval, generation and synthesis method and system based on artificial intelligence driving, and the method comprises the steps of multi-modal feature extraction, cross-modal alignment, feature fusion, multi-modal retrieval and sorting and multi-modal generation. Different deep learning models are adopted to carry out feature extraction on multi-modal data and convert the multi-modal data into vectors, through cross-modal alignment, different modal feature vectors are mapped to the same vector space, through feature fusion, correlation weights are calculated through an attention mechanism, weighted summation is carried out on fusion features, and through multi-modal retrieval and sorting, multi-modal data are obtained. According to the method, the similarity between a query vector and candidate data is calculated and sorted, and finally, through multi-modal generation, retrieval knowledge is used as external knowledge to be fused into a generation model, and a multi-modal synthesis answer is generated, so that information in multi-modal data can be effectively integrated and utilized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a multi-modal data retrieval generation synthesis method and system based on artificial intelligence driving. BACKGROUND

[0002] Traditional data processing is usually processed manually, but the manual processing method is low in efficiency and not accurate enough.

[0003] Existing data processing methods are usually designed for specific modal data, such as text processing systems focusing on processing text data and image processing systems focusing on processing image data, lacking the ability to process cross-modal data. Traditional methods often focus on processing single modal data, such as only processing text data, only processing image data, only processing audio data or only processing video data. When a large amount of text data needs to be processed, such as document classification, sentiment analysis, keyword extraction and other tasks, traditional text processing methods can provide efficient and accurate results

[0004] However, existing data processing methods can only process single modal data, and it is difficult to effectively integrate and utilize information in multi-modal data. SUMMARY

[0005] The present application aims to provide a multi-modal data retrieval generation synthesis method and system based on artificial intelligence driving, which aims to solve the problem that existing data processing methods can only process single modal data and are difficult to effectively integrate and utilize information in multi-modal data.

[0006] To achieve the above-mentioned purpose, the present application provides a multi-modal data retrieval generation synthesis method based on artificial intelligence driving, comprising the following steps:

[0007] Different deep learning models are used to extract features of at least two modal data in input text, image, audio and video, and convert them into vectors;

[0008] The extracted feature vectors of different modalities are input into a cross-modal alignment model, and all modal data is mapped to the same vector space by minimizing the distance of different modal feature vectors in the shared vector space;

[0009] In the shared vector space, the correlation weight of different modal features is calculated by attention mechanism and weighted summation to obtain a fusion feature vector to understand the query intention;

[0010] The query vector is extracted from the user query, the similarity between the query vector and the candidate multi-modal data feature vector is calculated in the shared vector space, the similarity is sorted, and the top N results are taken as the retrieval results;

[0011] Receiving the retrieval results from the multi-modal data retrieval step and other multi-modal input data, pre-processing the data of different modalities;

[0012] Inputting the pre-processed multi-modal data into the multi-modal generation model;

[0013] During the processing of the multi-modal generation model, inputting the relevant knowledge information retrieved in the multi-modal data retrieval step into the generation model as external knowledge, and fusing it with the knowledge inside the model;

[0014] The multi-modal generation model generates the final synthesized answer according to the integrated knowledge information and the input multi-modal query data.

[0015] Among them, different deep learning models are used to extract features of at least two modalities of data in the input text, image, audio and video respectively, and convert them into vectors, and the step further includes:

[0016] The text data is extracted by a text encoder based on the Transformer architecture; the image data is extracted by a convolutional neural network; the audio data is extracted by a recurrent neural network or its variants; the video data is first decomposed into image sequences by frame extraction technology, and then the convolutional neural network is used for feature extraction, and the time series analysis method is used to process the time sequence features of the image sequence.

[0017] Among them, the extracted feature vectors of different modalities are input into a cross-modal alignment model, and all modalities of data are mapped to the same vector space by minimizing the distance of different modalities of feature vectors in the shared vector space, and the step further includes:

[0018] The cross-modal alignment model uses a contrastive learning or generative adversarial network architecture.

[0019] Among them, the query vector is extracted by extracting the features of the user query, the similarity between the query vector and the feature vector of the candidate multi-modal data is calculated in the shared vector space, and the top N results are taken as the retrieval results according to the similarity, and the step further includes:

[0020] The similarity calculation method uses cosine similarity or Euclidean distance for calculation.

[0021] Among them, receiving the retrieval results from the multi-modal data retrieval step and other multi-modal input data, pre-processing the data of different modalities, the step further includes:

[0022] The pre-processing includes image normalization, audio noise reduction, text segmentation and part-of-speech tagging.

[0023] In the multi-modal generation model processing process, the relevant knowledge information retrieved in the multi-modal data retrieval step is input as external knowledge into the generation model, and is fused with the internal knowledge of the model.

[0024] The multi-modal generation model adopts a multi-modal large language model based on Transformer.

[0025] An artificial intelligence-driven multi-modal data retrieval and generation synthesis system includes a data input module, a feature extraction module, an alignment module, a fusion module, a sorting module, a processing module, a generation module, and an output module. The feature extraction module is connected to the data input module, the alignment module is connected to the feature extraction module, the fusion module is connected to the alignment module, the sorting module is connected to the fusion module, the processing module is connected to the sorting module, the generation module is connected to the processing module, and the output module is connected to the generation module.

[0026] The data input module is responsible for receiving different modal data input by the user.

[0027] The feature extraction module uses a deep learning model to extract features from the input different modal data and converts it into a vector representation.

[0028] The alignment module maps the feature vectors of different modalities to the same vector space.

[0029] The fusion module fuses the feature vectors of different modalities in the shared vector space to deeply understand the user query intent.

[0030] The sorting module calculates the similarity between the query vector and the candidate multi-modal data feature vector according to the user input query, and sorts the retrieval results.

[0031] The processing module pre-processes the retrieval results of the multi-modal data retrieval module and other possible multi-modal input data.

[0032] The generation module uses a multi-modal large language model based on Transformer to generate output content containing multiple modalities.

[0033] The output module fuses the relevant knowledge information retrieved by the multi-modal data retrieval module with the content generated by the multi-modal data generation module to form the final multi-modal synthesis answer and output.

[0034] The application discloses an artificial intelligence driven multi-modal data retrieval and generation synthesis method and system. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced.

[0036] Figure 1 is a step diagram of the artificial intelligence driven multi-modal data retrieval and generation synthesis method of the first embodiment of the application.

[0037] Figure 2 is a structural schematic diagram of the artificial intelligence driven multi-modal data retrieval and generation synthesis system of the second embodiment of the application.

[0038] In the figure: 201-data input module, 202-feature extraction module, 203-alignment module, 204-fusion module, 205-sorting module, 206-processing module, 207-generation module, 208-output module. DETAILED DESCRIPTION

[0039] The embodiments of the application will be described in detail below, examples of which are shown in the drawings, the embodiments described below by referring to the drawings are exemplary and are intended to explain the application, and cannot be understood as a limitation of the application.

[0040] The first embodiment of the present application is:

[0041] Please refer to Figure 1 , Figure 1 is a step diagram of the artificial intelligence driven multi-modal data retrieval and generation synthesis method of the first embodiment of the application.

[0042] The application provides an artificial intelligence driven multi-modal data retrieval and generation synthesis method, comprising the following steps:

[0043] S101: using different deep learning models to extract features of at least two modal data in input text, image, audio and video respectively, and converting them into vectors;

[0044] S102: input the extracted feature vectors of different modalities into the cross-modal alignment model, map the data of all modalities to the same vector space by minimizing the distance of different modalities feature vectors in the shared vector space;

[0045] S103: in the shared vector space, calculate the correlation weight of different modalities features with attention mechanism and weighted sum to get the fusion feature vector to understand the query intention;

[0046] S104: extract the features of the user query to get the query vector, calculate the similarity with the candidate multi-modal data feature vector in the shared vector space, sort according to the similarity, and take the top N results as the retrieval result;

[0047] S105: receive the retrieval result from the multi-modal data retrieval step and other multi-modal input data, and pre-process the data of different modalities;

[0048] S106: input the pre-processed multi-modal data into the multi-modal generation model;

[0049] S107: in the process of multi-modal generation model processing, input the relevant knowledge information retrieved in the multi-modal data retrieval step into the generation model as external knowledge, and fuse it with the internal knowledge of the model;

[0050] S108: the multi-modal generation model generates the final synthesized answer according to the integrated knowledge information and the input multi-modal query data.

[0051] The preset deep learning model combination is used to respectively extract features of input text, image, audio and video data of at least two modalities, and convert the data of different modalities into vector representations of corresponding dimensions; wherein, a text encoder based on a Transformer architecture is used for feature extraction of text data; a convolutional neural network (CNN) is used for feature extraction of image data; a recurrent neural network (RNN) or its variant is used for feature extraction of audio data; for video data, it is first decomposed into an image sequence through a frame extraction technique, and then the CNN is used for feature extraction, and the time sequence analysis method is used to process the time sequence features of the image sequence. The extracted feature vectors of different modalities are input into a cross-modal alignment model, which adopts a contrast learning or generative adversarial network (GAN) architecture. By minimizing the distance of feature vectors of different modalities in a shared vector space, the data of all modalities are mapped to the same vector space, realizing the comparability of different modalities in the vector space. In the shared vector space, the attention mechanism is used to fuse the feature vectors of different modalities, calculate the correlation weight between the feature vectors of different modalities, and perform weighted summation on the features of each modality according to the weight, to obtain the fused multi-modal feature vector, so as to deeply understand the user query intention. According to the user input query, the same deep learning model combination as in the feature extraction stage is used to extract features of the query, to obtain a query vector. In the shared vector space, the similarity between the query vector and the candidate multi-modal data feature vector is calculated. The similarity calculation method adopts cosine similarity or Euclidean distance. According to the similarity, the retrieval results are sorted, and the top N results with the highest similarity are selected as the retrieval results.

[0052] Further, the retrieval results from the multi-modal data retrieval step and other possible multi-modal input data are received, and the data of different modalities are preprocessed, including image normalization, audio noise reduction, text segmentation and part-of-speech tagging, etc. operations, in order to facilitate subsequent generation processing. The preprocessed multi-modal data is input into a multi-modal generation model, which adopts a multi-modal large language model (MLLM) based on Transformer. The model is trained on large-scale multi-modal data and can understand and process data of multiple modalities. The model generates output content containing at least two modalities of text, image, audio or video according to the input multi-modal data.

[0053] Further, during the multi-modal generation model processing process, the relevant knowledge information retrieved in the multi-modal data retrieval step is input into the generation model as external knowledge, which is fused with the internal knowledge of the model to provide more rich background information for the generation process. The multi-modal generation model generates the final synthesized answer according to the integrated knowledge information and the input multi-modal query data. The synthesized answer is presented in the form of multi-modal, including but not limited to text description combined with related images, video clips and audio commentary, etc. Thus, through multi-modal feature extraction, different deep learning models are used to extract and convert multi-modal data into vectors, different modal feature vectors are mapped to the same vector space through cross-modal alignment, and the features are fused through feature fusion, the relevance weight is calculated through the attention mechanism and weighted summation, and the features are fused. Through multi-modal retrieval and sorting, the similarity between the query vector and the candidate data is calculated and sorted, and finally through multi-modal generation, the retrieved knowledge is integrated into the generation model as external knowledge to generate a multi-modal synthesized answer, so that the information in the multi-modal data can be effectively integrated and utilized.

[0054] When the multi-modal data retrieval and synthesis method based on artificial intelligence driving is used, through multi-modal feature extraction, different deep learning models are used to extract and convert multi-modal data into vectors, different modal feature vectors are mapped to the same vector space through cross-modal alignment, and the features are fused through feature fusion, the relevance weight is calculated through the attention mechanism and weighted summation, and the features are fused. Through multi-modal retrieval and sorting, the similarity between the query vector and the candidate data is calculated and sorted, and finally through multi-modal generation, the retrieved knowledge is integrated into the generation model as external knowledge to generate a multi-modal synthesized answer, so that the information in the multi-modal data can be effectively integrated and utilized.

[0055] The second embodiment of the present application is:

[0056] Based on the first embodiment, please refer to Figure 2 , Figure 2 is a structural schematic diagram of the multi-modal data retrieval and synthesis system based on artificial intelligence driving of the second embodiment of the present application.

[0057] The application provides a multi-modal data retrieval and generation synthesis system based on artificial intelligence driving, which further comprises a data input module 201, a feature extraction module 202, an alignment module 203, a fusion module 204, a sorting module 205, a processing module 206, a generation module 207 and an output module 208, the feature extraction module 202 is connected with the data input module 201, the alignment module 203 is connected with the feature extraction module 202, the fusion module 204 is connected with the alignment module 203, the sorting module 205 is connected with the fusion module 204, the processing module 206 is connected with the sorting module 205, the generation module 207 is connected with the processing module 206, and the output module 208 is connected with the generation module 207.

[0058] The data input module 201 is used for receiving different modal data input by a user.

[0059] The feature extraction module 202 is used for extracting features of the input different modal data by using a deep learning model and converting the features into vector representations.

[0060] The alignment module 203 is used for mapping feature vectors of different modes to the same vector space.

[0061] The fusion module 204 is used for fusing feature vectors of different modes in the shared vector space and deeply understanding the user query intention.

[0062] The sorting module 205 is used for calculating the similarity between a query vector and a candidate multi-modal data feature vector according to the user input query and sorting the retrieval results.

[0063] The processing module 206 is used for preprocessing the retrieval results of the multi-modal data retrieval module and other possible multi-modal input data.

[0064] The generation module 207 is used for generating output content containing multiple modes by using a multi-modal large language model based on the Transformer.

[0065] The output module 208 is used for fusing relevant knowledge information retrieved by the multi-modal data retrieval module and content generated by the multi-modal data generation module 207, forming a final multi-modal synthesis answer and outputting.

[0066] The input data is preliminarily checked and preprocessed by the data input module 201, the image file format is checked for support, the sampling rate, code rate and other parameters of audio and video are checked for compliance, the text is simply segmented and standardized, the different modal data is extracted by the feature extraction module 202 using a deep learning model, and the input data is converted into a vector representation, the alignment module 203 uses a contrast learning architecture to minimize the distance between different modal feature vectors in a shared vector space, and realizes cross-modal alignment, in the training process, a large amount of multi-modal data is used to train the contrast learning model, and the model parameters are optimized, so that similar data in different modalities is closer in the vector space, and dissimilar data is farther apart, the fusion module 204 uses a self-attention mechanism to calculate the correlation weight between different modal feature vectors, and the modal features are weighted and summed according to the weight to obtain a fused multi-modal feature vector. For the fusion of text and image, the proportion of text features and image features in the fused features is dynamically adjusted according to the correlation between the two, the query vector is obtained by the sorting module 205 to extract the features of the user query, and the cosine similarity is used to calculate the similarity between the query vector and the candidate multi-modal data feature vector in the shared vector space. According to the similarity from high to low, the retrieval results are sorted, and the top N results with the highest similarity are selected as the final retrieval results, the image is normalized by the processing module 206 to adjust the image size and color space; the audio is denoised to remove background noise; the text is segmented, tagged and removed, and the preprocessed multi-modal data is input into the MLLM by the generation module 207, and the model generates output content containing text, image, audio or video (at least two modalities) according to the input data. According to the text description and uploaded image input by the user, multi-modal content containing detailed text description, related images and audio explanation is generated, and finally the retrieved related knowledge is input into the generation model as external knowledge by the output module 208, and the model internal knowledge is fused and output. The final output form can be a web page, a mobile application interface or a specific file format, ensuring that users can easily view and use multi-modal information.

[0067] The above only discloses one or more preferred embodiments of the present application, and cannot limit the scope of the present application. Those skilled in the art can understand that all or part of the above-mentioned embodiments can be implemented, and equivalent changes made to the claims of the present application still fall within the scope of the present application.

Claims

1. An artificial intelligence driven multi-modal data retrieval based synthetic generation method, characterized in that, The method comprises the following steps: extracting features of input data of at least two modalities of text, image, audio and video respectively using different deep learning models, and converting the features into vectors; inputting the extracted feature vectors of different modalities into a cross-modal alignment model, and mapping data of all modalities to the same vector space by minimizing the distance of feature vectors of different modalities in the shared vector space; in the shared vector space, calculating the correlation weight of features of different modalities by using an attention mechanism and performing weighted summation to obtain a fusion feature vector to understand the query intent; extracting features of a user query to obtain a query vector, calculating the similarity between the query vector and feature vectors of candidate multi-modal data in the shared vector space, sorting according to the similarity, and taking the first N results as retrieval results; receiving retrieval results from the multi-modal data retrieval step and other multi-modal input data, and pre-processing data of different modalities; inputting the pre-processed multi-modal data into a multi-modal generation model; during processing of the multi-modal generation model, inputting relevant knowledge information retrieved in the multi-modal data retrieval step into the generation model as external knowledge, and fusing the external knowledge with internal knowledge of the model; the multi-modal generation model generates a final synthesized answer according to the integrated knowledge information and the input multi-modal query data.

2. The artificial intelligence driven multi-modal data retrieval generation synthesis method of claim 1, wherein, extracting features of input data of at least two modalities of text, image, audio and video respectively using different deep learning models, and converting the features into vectors, wherein the step further comprises: text data is extracted using a text encoder based on a Transformer architecture; image data is extracted using a convolutional neural network; audio data is extracted using a recurrent neural network or a variant thereof; and video data is first decomposed into an image sequence using a frame extraction technique, and then extracted using the convolutional neural network, and the time sequence characteristics of the image sequence are processed using a time sequence analysis method.

3. The artificial intelligence driven multi-modal data retrieval generation synthesis method of claim 1, wherein, inputting the extracted feature vectors of different modalities into a cross-modal alignment model, and mapping data of all modalities to the same vector space by minimizing the distance of feature vectors of different modalities in the shared vector space, wherein the step further comprises: the cross-modal alignment model uses a contrastive learning or a generative adversarial network architecture.

4. The artificial intelligence driven multi-modal data retrieval generation synthesis method of claim 1, wherein, extracting features of a user query to obtain a query vector, calculating the similarity between the query vector and feature vectors of candidate multi-modal data in the shared vector space, sorting according to the similarity, and taking the first N results as retrieval results, wherein the step further comprises: the similarity calculation method uses cosine similarity or Euclidean distance for calculation.

5. The artificial intelligence driven multi-modal data retrieval generation synthesis method of claim 1, wherein, receiving retrieval results from the multi-modal data retrieval step and other multi-modal input data, and pre-processing data of different modalities, wherein the step further comprises: the pre-processing includes normalization of images, noise reduction of audio, word segmentation and part-of-speech tagging of text.

6. The artificial intelligence driven multi-modal data retrieval generation synthesis method of claim 1, wherein, during processing of the multi-modal generation model, inputting relevant knowledge information retrieved in the multi-modal data retrieval step into the generation model as external knowledge, and fusing the external knowledge with internal knowledge of the model, wherein the step further comprises: the multi-modal generation model uses a multi-modal large language model based on a Transformer.

7. An artificial intelligence driven multi-modal data retrieval generation synthesis system adapted to the artificial intelligence driven multi-modal data retrieval generation synthesis method of claim 1, characterized in that, The system comprises a data input module, a feature extraction module, an alignment module, a fusion module, a sorting module, a processing module, a generation module, and an output module; the feature extraction module is connected with the data input module and the alignment module; the fusion module is connected with the alignment module; the sorting module is connected with the fusion module; the processing module is connected with the sorting module; the generation module is connected with the processing module; and the output module is connected with the generation module. The data input module is configured to receive different modal data input by a user. The feature extraction module is configured to extract features of the input different modal data by using a deep learning model and convert the features into vector representations. The alignment module is configured to map feature vectors of different modal data to the same vector space. The fusion module is configured to fuse the feature vectors of different modal data in the shared vector space and deeply understand the user query intention. The sorting module is configured to calculate the similarity between a query vector and a candidate multi-modal data feature vector according to a user input query, and sort the retrieval results. The processing module is configured to preprocess retrieval results of the multi-modal data retrieval module and other possible multi-modal input data. The generation module is configured to generate output content containing multiple modalities by using a multi-modal large language model based on a Transformer. The output module is configured to fuse relevant knowledge information retrieved by the multi-modal data retrieval module and content generated by the multi-modal data generation module, form a final multi-modal synthesized answer, and output the answer.

Images