A multimodal hybrid database storage and interaction method based on deep learning

Abstract

The present invention discloses a multimodal hybrid database storage and interaction method based on deep learning. The present invention realizes the hybrid storage and real-time retrieval interaction function of different modal data (images, videos, voices and texts) on the basis of the existing separate storage of multimodal data and the use of single-modal hybrid label interaction technology, further increasing the understanding of fused data and improving the interactive performance. On the technical level, the present invention uses a deep learning unified general model to extract and fuse features of images, videos, voices and texts respectively; the present invention adopts the universal storage of multimodal fused data; the present invention adopts the matching calculation of user data and fused feature data; the present invention performs multimodal conversion and returns according to the matching information and the universal unified model.

Description

Technical Field

[0001] The present invention relates to a hybrid data storage and retrieval method of multimodal interaction based on deep learning. Background Art

[0002] There are many systems that perform storage and interaction based on a single modality, but there are few retrieval and interaction methods that can uniformly process multimodal (image, video, voice, text) data and provide universal storage. Summary of the invention

[0003] The purpose of the present invention is to provide a new method for multimodal universal unified hybrid data retrieval and interaction using a universal unified multimodal deep learning model to perform feature extraction, fusion, matching calculation, and multimodal data generation through deep learning technology.

[0004] In order to achieve the above object, the technical solution of the present invention is to disclose a multimodal hybrid database storage and interaction method based on deep learning, which is characterized by comprising the following steps:

[0005] Step 1: input the multimodal data to be used into a trained universal unified multimodal deep learning model, and extract the features after fusion of the multimodal data by the universal unified multimodal deep learning model to obtain fused feature data;

[0006] Step 2: storing the fused feature data obtained in step 1 into a general database;

[0007] Step 3: Using the trained multimodal deep learning model, extract the features of the user information data input by the user, match the features with the fused feature data stored in the general database, calculate the matching score, and determine the multimodal data to be used according to the matching score;

[0008] Step 4: Perform modal conversion on the multimodal data obtained in step 3 using the universal unified multimodal deep learning model, and return the obtained multimodal data to the user using a computer program application.

[0009] Preferably, in step 1, the multimodal data includes image data, video data, language data and natural language data.

[0010] Preferably, in step 1, the universal unified multimodal deep learning model includes an encoding layer and a decoding layer.

[0011] Preferably, in step 2, basic features of the general database include: being able to store corresponding multimodal fusion data; being able to retrieve corresponding multimodal fusion data; and being able to perform vector or other types of mathematical calculations under specific circumstances.

[0012] Preferably, in step 4, the universal unified multimodal deep learning model can input feature data of one or more modalities, and transform one or more modal fusion data into another one or more modal data according to the required modal information.

[0013] The present invention realizes the mixed storage and real-time retrieval interaction of different modal data (images, videos, voices and texts) based on the existing multimodal data separate storage and the use of single-modal mixed label interaction technology, further increasing the understanding of fused data and improving the interactive performance. On the technical level, the present invention uses a deep learning unified general model to extract and fuse features of images, videos, voices and texts respectively; the present invention adopts the universal storage of multimodal fused data; the present invention adopts the matching calculation of user data and fused feature data; the present invention performs multimodal conversion and returns according to the matching information and the universal unified model.

[0014] The method disclosed in the present invention uses a universal unified multimodal deep learning model to obtain data that can be matched to obtain data information that cannot be matched or matched incorrectly by searching other multimodal or single modal databases. The design and use of a unified multimodal database are simpler, and the re-learning of the deep learning model is more convenient. When returning interactive information, user information and user environment information can be used to convert and generate corresponding multimodal (picture, video, voice, text) data for re-return, which is more flexible and intelligent than a single modal interactive system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a core model architecture diagram of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0017] The present invention discloses a multimodal hybrid database storage and interaction method based on deep learning, which involves a deep learning model that can perform multimodal feature extraction and feature matching calculation of images, voices, and natural languages, and a deep learning model generated by multimodal related transformation of images, voices, natural languages, etc., and optimizes the invention for different modules, specifically including the following steps:

[0018]

Step 1

[0019] The multimodal data required, including image data, video data, language data and natural language data, are input into a trained universal unified multimodal deep learning model (the universal unified multimodal deep learning model includes an encoding layer and a decoding layer), and the fused feature data after fusion of the multimodal data is obtained through the universal unified multimodal deep learning model.

[0020]

Step 2

[0021] The fusion feature data obtained in step 1 is stored in a general database. The basic features of the general database include: being able to store corresponding multimodal fusion data, being able to retrieve corresponding multimodal fusion data, and being able to perform vector or other types of mathematical calculations under specific circumstances.

[0022]

Step 3

[0023] The trained multimodal deep learning model is used to extract the features of the multimodal data input by the user, and the features are matched with the multimodal data in the general database described in step 2. In this step, the multimodal data information and the user information data are mapped to a general space, and the matching score between the multimodal data information and the user information data is calculated by an algorithm, and the multimodal data used is determined according to the matching score.

[0024] [Step 4] Transform the determined multimodal data using a universal unified multimodal deep learning model and return it

[0025] The multimodal data obtained in step three is modally transformed using a universal unified multimodal deep learning model.

[0026] In this step, the general unified multimodal deep learning model can input feature data of one or more modalities, and determine what (one or more) modal data the user needs based on the required modal information, for example, user information, and convert one or more modal fusion data into another one or more modal data.

[0027] The obtained multimodal data can be returned to the user using a computer program application, so that the data required for user input in the above steps one to four can be obtained. The data can be input and passed to the deep learning model in the above steps for calculation, and the multimodal data finally calculated can be returned to the user in an appropriate manner according to its corresponding modality.

Claims

1. A multimodal hybrid database storage and interaction method based on deep learning, characterized in that: The following steps are involved: Step 1: input the multimodal data to be used into a trained universal unified multimodal deep learning model, and extract the features after fusion of the multimodal data by the universal unified multimodal deep learning model to obtain fused feature data; Step 2: storing the fused feature data obtained in step 1 into a general database; Step 3: Using the trained multimodal deep learning model, extract the features of the user information data input by the user, match the features with the fused feature data stored in the general database, calculate the matching score, and determine the multimodal data to be used according to the matching score; Step 4: Perform modal conversion on the multimodal data obtained in step 3 using the universal unified multimodal deep learning model, and return the obtained multimodal data to the user using a computer program application.

2. A multimodal hybrid database storage and interaction method based on deep learning as claimed in claim 1, characterized in that: In step 1, the multimodal data includes image data, video data, language data and natural language data.

3. The multimodal hybrid database storage and interaction method based on deep learning as claimed in claim 1, characterized in that: In step 1, the universal unified multimodal deep learning model includes an encoding layer and a decoding layer.

4. The multimodal hybrid database storage and interaction method based on deep learning as claimed in claim 1, characterized in that: In step 2, the basic features of the general database include: being able to store corresponding multimodal fusion data; being able to retrieve corresponding multimodal fusion data; and being able to perform vector or other types of mathematical calculations in specific circumstances.

5. The multimodal hybrid database storage and interaction method based on deep learning as claimed in claim 1, characterized in that: In step 4, the universal unified multimodal deep learning model can input feature data of one or more modalities, and transform one or more modal fusion data into another one or more modal data according to the required modal information.

Images