Multi-modal data storage method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a multi-modal data storage method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining annotation data of multi-modal data, and obtaining a first storage space based on the annotation data; determining first data association information of the first storage space, and obtaining a second storage space based on the first data association information; determining first label associated information of the first storage space, and obtaining a third storage space based on the first label associated information; determining unlabeled data of the multi-modal data, and obtaining second data association information based on the unlabeled data and the labeled data; and labeling the unlabeled data based on the second data association information to obtain fifth label information, and storing the unlabeled data to the first storage space, the second storage space and the third storage space based on the fifth label information. According to the invention, labeling and collection of the multi-modal data do not need to consume excessive resources, and the data can be effectively fused together.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a multimodal data storage method, device, electronic device, storage medium and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Data often exists in multiple forms, such as text, images, audio, etc., which are called multimodal data; the storage of multimodal data is an important research direction in the field of deep learning, which aims to effectively integrate data of different modalities to improve the perception and understanding capabilities of the model.

[0004] However, in related technologies, data of different modalities have different feature representations and semantic structures, so it is impossible to effectively fuse the data together, and the annotation and collection of multimodal data requires a lot of manpower and material resources. Summary of the invention

[0005] In view of this, the purpose of the present disclosure is to propose a multimodal data storage method, device, electronic device, storage medium and program product, which at least to a certain extent solve one of the technical problems in the related art.

[0006] Based on the above purpose, the first aspect of the exemplary embodiment of the present disclosure provides a multimodal data storage method, which is applied to a server, and the method includes:

[0007] Determine the labeling data of the multimodal data, and obtain a first storage space based on the labeling data;

[0008] Determine first data association information of the first storage space, and obtain a second storage space based on the first data association information;

[0009] Determine first tag association information of the first storage space, and obtain a third storage space based on the first tag association information;

[0010] Determine unlabeled data of the multimodal data, and obtain second data association information based on the unlabeled data and the labeled data;

[0011] The unlabeled data is labeled based on the second data association information to obtain fifth label information, and the unlabeled data is stored in the first storage space, the second storage space, and the third storage space based on the fifth label information.

[0012] Based on the same inventive concept, a second aspect of the exemplary embodiment of the present disclosure provides a multimodal data storage device, including:

[0013] A first space determination module is configured to determine the annotation data of the multimodal data, and obtain a first storage space based on the annotation data;

[0014] A second space determination module is configured to determine first data association information of the first storage space, and obtain a second storage space based on the first data association information;

[0015] A third space determination module is configured to determine first tag association information of the first storage space, and obtain a third storage space based on the first tag association information;

[0016] A second association determination module is configured to determine unlabeled data of the multimodal data, and obtain second data association information based on the unlabeled data and the labeled data;

[0017] The data storage determination module is configured to label the unlabeled data based on the second data association information to obtain fifth label information, and store the unlabeled data in the first storage space, the second storage space and the third storage space based on the fifth label information.

[0018] Based on the same inventive concept, a third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0019] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0020] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the method described in the first aspect.

[0021] It can be seen from the above that the embodiments of the present disclosure provide a multimodal data storage method, device, electronic device, storage medium, and program product, the method comprising:

[0022] Determine the labeled data of the multimodal data, obtain a first storage space based on the labeled data, determine first data association information of the first storage space, obtain a second storage space based on the first data association information, determine first label association information of the first storage space, obtain a third storage space based on the first label association information, determine the unlabeled data of the multimodal data, obtain second data association information based on the unlabeled data and the labeled data, label the unlabeled data based on the second data association information to obtain fifth label information, and store the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fifth label information. The labeling and collection of the multimodal data disclosed in the present invention does not need to consume too many resources, and can effectively fuse the data together. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of an application scenario of a multimodal data storage method provided by an exemplary embodiment of the present disclosure;

[0025] Figure 2 A schematic flow chart of a multimodal data storage method provided for an exemplary embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of a structure of a multimodal data storage device provided for an exemplary embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of the hardware structure of a multimodal data storage electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0035] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "a" before an element does not exclude the existence of multiple such elements.

[0036] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0037] As described in the background technology, in the related art, data of different modalities have different feature representations and semantic structures, so it is impossible to effectively merge the data together, and the annotation and collection of multimodal data requires a lot of manpower and material resources. Specifically:

[0038] First, different modal data have unique feature representations. Text data uses natural language as a carrier, and contains rich information such as vocabulary, grammar, and semantics; image data exists in the form of a pixel matrix, and expresses information through visual features such as color, shape, and texture; audio data transmits information through acoustic features such as the frequency, amplitude, and duration of sound waves. The differences between these features make it difficult to directly fuse different modal data, and it is necessary to find appropriate feature conversion and matching methods to effectively integrate them.

[0039] Secondly, the semantic structures of different modal data are also different. Text data usually has a clear semantic structure, such as sentences, paragraphs, and chapters, while image and audio data lack such a clear hierarchical structure and need to be understood and parsed through technologies such as image recognition and speech recognition. This difference in semantic structure makes it difficult to integrate different modal data at the semantic level, and it is necessary to use technologies such as natural language processing and computer vision to achieve semantic integration.

[0040] In addition, the annotation and collection of multimodal data also requires a lot of manpower and material resources. Multimodal data annotation requires special annotation for different modal data, such as text annotation, image annotation, and audio annotation, which requires a large number of professional annotators to work for a long time and is prone to annotation errors. Multimodal data collection requires obtaining data of different modalities from different sources, such as networks, databases, and sensors, which requires the establishment of a complex data collection system and is prone to data quality problems.

[0041] In summary, multimodal data fusion and annotation face huge challenges, and new technologies and methods need to be developed to solve these problems in order to improve the efficiency and accuracy of multimodal data processing.

[0042] In order to solve the above problems, the present disclosure provides a multimodal data storage method, device, electronic device, storage medium and program product solution, which specifically include:

[0043] Determine the labeled data of the multimodal data, obtain a first storage space based on the labeled data, determine the first data association information of the first storage space, obtain a second storage space based on the first data association information, determine the first label association information of the first storage space, obtain a third storage space based on the first label association information, determine the unlabeled data of the multimodal data, obtain second data association information based on the unlabeled data and the labeled data, label the unlabeled data based on the second data association information, obtain fifth label information, and store the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fifth label information. The present invention establishes a data interaction path according to the relationship between different dimensions after knowledge fusion of the labeled multimodal data in different dimensions, stores the multimodal data after knowledge fusion in different storage spaces, and for the unlabeled multimodal data, the present invention labels it according to the relationship between the multimodal data after knowledge fusion and the unlabeled multimodal data, and stores it according to the labeling results.

[0044] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0045] refer to Figure 1 , which is a schematic diagram of an application scenario of the multimodal data storage method provided by an exemplary embodiment of the present disclosure.

[0046] This application scenario includes a terminal device 101 and a server 102. The terminal device 101 and the server 102 may be connected via a wired or wireless communication network to achieve data interaction.

[0047] The terminal device 101 may be an electronic device with data transmission and multimedia input / output functions close to the user side, including but not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of realizing the above functions. The electronic device may include a processor and a display screen with a touch input function, the display screen is used to present a graphical user interface, the graphical user interface can display an application interface, and the processor is used to process application data, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0048] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0049] In some exemplary embodiments, the multimodal data storage method may be run on the terminal device 101 or the server 102 .

[0050] When the multimodal data storage method is run on the server 102 , the server 102 is used to provide a multimodal data storage service to a user of the terminal device 101 .

[0051] The server 102 determines the annotation data of the multimodal data, and the server 102 obtains a first storage space based on the annotation data;

[0052] The server 102 determines the first data association information of the first storage space, and the server 102 obtains the second storage space based on the first data association information;

[0053] The server 102 determines the first tag association information of the first storage space, and the server 102 obtains the third storage space based on the first tag association information;

[0054] The server 102 determines unlabeled data of the multimodal data, and the server 102 obtains second data association information based on the unlabeled data and the labeled data;

[0055] The server 102 labels the unlabeled data based on the second data association information to obtain fifth label information, and the server 102 stores the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fifth label information.

[0056] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0057] refer to Figure 2 , a multimodal data storage method, applied to a server, the method comprising the following steps:

[0058] Step S210: determine the annotation data of the multimodal data, and obtain a first storage space based on the annotation data.

[0059] In specific implementation, the method of determining the labeling data of multimodal data includes but is not limited to at least one of the following:

[0060] Obtain labeled multimodal data such as text, images, videos, audio, etc. through publicly available open source databases, obtain labeled multimodal data such as text, images, videos, audio, etc. through social media platforms, or obtain labeled multimodal data such as text, images, videos, audio, etc. through specific data collection methods.

[0061] In the above exemplary embodiment, a method of obtaining the annotation data of the multimodal data is introduced. Next, a method of obtaining the first storage space will be introduced:

[0062] In a specific implementation, a method of obtaining the first storage space based on the labeled data includes:

[0063] First, the original labeled data collected by the server is cleaned and duplicate records in the data set are deleted to ensure that each data sample is unique;

[0064] Secondly, if the obtained labeled data is text data, irrelevant characters, HTML tags or stop words in the text are removed; if the obtained labeled data is image data, filters (such as Gaussian filtering, median filtering) are used to reduce noise in the image; if the obtained labeled data is audio data, signal processing techniques (such as spectral subtraction, wavelet transform) are applied to remove background noise;

[0065] Then, the data from different sources is converted into a unified format for easy processing and analysis;

[0066] Finally, design a reasonable data storage structure to facilitate data management and quick access.

[0067] Step S220: Determine first data association information of the first storage space, and obtain a second storage space based on the first data association information.

[0068] In a specific implementation, the first data association information refers to:

[0069] In the first storage space, there is a first associative relationship between different modal data; this relationship can be understood as a necessary connection being generated due to the existence of one or more identical labels between different data.

[0070] As a specific example, for example, if there is multimodal data about "beach" and multimodal data about "vacation" in the storage space, the first association relationship may be the connection between "beach" and "vacation" because the beach is usually associated with the vacation experience. In this case, if we see a picture of a beach, we may automatically think of the concept of vacation, even if the word "vacation" is not directly mentioned in the text description or user comments.

[0071] In the above exemplary embodiment, the first data association information is introduced. Now, the method of obtaining the second tag information is introduced:

[0072] In this exemplary embodiment, label fusion is performed on the annotated data in the first storage space based on the first data association information to obtain second label information, including:

[0073] Feature extraction is performed on the annotated data to obtain first label information, and label fusion is performed on the first label information based on the first data association information to obtain the second label information.

[0074] In a specific implementation, the method of extracting features from the labeled data to obtain the first label information includes but is not limited to at least one of the following:

[0075] If the obtained labeled data is text data, the text is converted into a word set and the frequency of word occurrence is used as a feature;

[0076] If the obtained labeled data is image data, the number of pixels of different colors in the image is counted as features;

[0077] If the obtained labeled data is video data, the video is decomposed into frames, and image features are extracted for each frame;

[0078] If the obtained labeled data is audio data, the Mel frequency cepstrum coefficients of the audio signal are extracted as features;

[0079] The data is labeled according to predefined rules, for example, keywords are extracted as the first label information based on the text content; in this scheme, the first label information includes at least two different labels, so that this scheme can capture the diversity and complexity of multimodal data.

[0080] In a specific implementation, a method of performing label fusion on the first label information based on the first data association information to obtain the second label information is as follows:

[0081] The second label information is generated according to the first data association information between the first label information; for example, if two texts have the same first label, they may have the same second label, indicating that they belong to the same topic; if the two texts do not have the same first label, we perform knowledge fusion on the multimodal data according to the first data association information, and generate new second label information by fusing the labels; in this scheme, the second label information contains at least two different labels, so that this scheme can capture the diversity and complexity of multimodal data.

[0082] In the above exemplary embodiment, a method for obtaining the second tag information is introduced. Next, a method for obtaining the second storage space is introduced:

[0083] In this exemplary embodiment, obtaining the second storage space based on the first data association information includes:

[0084] Based on the first data association information, labels are fused on the annotated data in the first storage space to obtain second label information, and based on the second label information, the second storage space is constructed.

[0085] In a specific implementation, the second storage space is constructed based on the second tag information, including but not limited to at least one of the following:

[0086] The second tag information is based on the storage structure of the file system to construct a second storage space, the second tag information is based on the storage structure of the database to construct a second storage space, the second tag information is based on the storage structure of distributed storage to construct a second storage space, or the second tag information is based on the storage structure of cloud storage to construct a second storage space.

[0087] Step S230: Determine first tag association information of the first storage space, and obtain a third storage space based on the first tag association information.

[0088] In a specific implementation, the first tag association information refers to:

[0089] The relationship between different labels in the labeled data can be understood as the relationship between labels generated when setting the labeling rules, or the relationship between labels obtained through big data, such as: the inevitable relationship between a person's name and a company (employment relationship, etc.), the inevitable relationship between names (husband-wife relationship, kinship relationship, etc.), the inevitable relationship between companies (subordinate relationship, etc.), or it can be understood that different labels are included in the data of one dimension, and the inevitable connection between the labels is generated, for example: someone likes to play basketball, someone interacted with someone, and someone also likes basketball, that is, the objective connection between name + sports (having a common hobby), and including a common occupation, etc., or it can be given as they are all cats, but the difference is black cats, white cats, or calico cats.

[0090] In the above exemplary embodiment, the method of associating the first tag with the information is introduced. Now, the method of obtaining the third storage space is introduced:

[0091] The first tag association information of the first tag information is determined, tag fusion is performed on the first tag information based on the first tag association information to obtain third tag information, and the third storage space is constructed based on the third tag information.

[0092] In specific implementation, the advantage of performing label fusion on the first label information based on the first label association information is that according to the inevitable connection between labels of data of the same dimension, data of other dimensions corresponding to the labels are fully fused, so that the fused data exhibits multi-dimensional characteristics.

[0093] In a specific implementation, the third label information includes at least two different labels, so that the scheme can capture the diversity and complexity of multimodal data.

[0094] In a specific implementation, a method of constructing the third storage space based on the third tag information includes but is not limited to at least one of the following:

[0095] The third tag information is based on the storage structure of the file system to construct a third storage space, the third tag information is based on the storage structure of the database to construct a third storage space, the third tag information is based on the storage structure of distributed storage to construct a third storage space or the third tag information is based on the storage structure of cloud storage to construct a third storage space.

[0096] In the above exemplary embodiments, the methods of constructing the first storage space, the second storage space and the third storage space based on the obtained labeled multimodal data are introduced respectively. When different dimensions and different new contents are established, it is also necessary to consider what kind of necessary connection exists between these newly formed fusion knowledge, so as to establish the necessary connection between different storage spaces. It should be noted here that the purpose of storing different labels in different storage spaces is to ensure the security of data when calling data and reduce the storage pressure of storage space. Since there are content-dominated and label-dominated ways of label fusion, as the amount of data continues to increase, the order of magnitude of its fusion must increase exponentially. Then, in the first storage space, that is, the storage of (first label information), the storage pressure will gradually increase, and there is also the risk of data loss when expanding the first storage space. Therefore, it is necessary to release the storage pressure through multiple storage spaces. By obtaining the necessary connection between fusions, a data interaction channel between fusion knowledge and original knowledge is well established, as long as the second storage space and the third storage space are wirelessly expanded according to the channel.

[0097] Step S240: Determine unlabeled data of the multimodal data, and obtain second data association information based on the unlabeled data and the labeled data.

[0098] In a specific implementation, the method of determining the unlabeled data of the multimodal data includes but is not limited to at least one of the following:

[0099] Obtain unlabeled text, images, videos, audio and other multimodal data through publicly available open source databases, obtain unlabeled text, images, videos, audio and other multimodal data through social media platforms, or obtain unlabeled text, images, videos, audio and other multimodal data through specific data collection methods.

[0100] In a specific implementation, the second data association information refers to:

[0101] Refers to the correlation between unlabeled multimodal data and labeled multimodal data. It is used to guide the labeling process of unlabeled data and determine its storage location.

[0102] Step S250: label the unlabeled data based on the second data association information to obtain fifth label information, and store the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fifth label information.

[0103] In this exemplary embodiment, the unlabeled data is labeled based on the second data association information to obtain fifth label information, including:

[0104] Feature extraction is performed on the unlabeled data to obtain feature information of the unlabeled data, the unlabeled data is labeled based on the feature information of the unlabeled data to obtain fourth label information, and the fourth label information is labeled based on the second data association information to obtain fifth label information.

[0105] In a specific implementation, the method of extracting features from the unlabeled data to obtain feature information of the unlabeled data includes but is not limited to at least one of the following:

[0106] If the unlabeled data obtained is image data, the number of pixels of different colors in the image is counted as features;

[0107] If the obtained unlabeled data is video data, the video is decomposed into frames, and image features are extracted for each frame;

[0108] If the unlabeled data obtained is audio data, the Mel frequency cepstrum coefficients of the audio signal are extracted as features;

[0109] Based on the above method, the feature information of unlabeled data is obtained.

[0110] In a specific implementation, the method of labeling the unlabeled data based on the unlabeled data feature information to obtain the fourth label information includes:

[0111] The data is labeled according to predefined rules, for example, keywords are extracted as fourth label information based on the text content; in this solution, the fourth label information includes at least two different labels, so that this solution can capture the diversity and complexity of multimodal data.

[0112] In a specific implementation, the fourth label information is annotated based on the second data association information to obtain the fifth label information:

[0113] For unlabeled multimodal data, when annotating the third label, it is also possible to obtain whether there is "second data association information" between the unlabeled data and the labeled data. The second data association information here can be understood as the unlabeled content corresponding to the fourth signature information may exist: based on the correlation between the labeled content; and then directly obtain the third label of the labeled content according to this correlation to assign it to the unlabeled data. That is to say, when annotating this part of the content with the third label, there is no need to annotate the fourth label information, but directly annotate the fifth label information to this part, thereby eliminating the need to annotate repeated content (which can also be similar content, such as the same picture that may exist between images and videos, the same audio that may exist between sounds and videos, the same text content that may exist between text and videos, etc.).

[0114] In the above exemplary embodiment, a method of obtaining the fifth label information is introduced. Next, a method of storing the unlabeled data in the first storage space, the second storage space, and the third storage space is introduced:

[0115] In this exemplary embodiment, storing the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fifth tag information includes:

[0116] Determine second label association information of the fourth label information, perform label fusion on the fourth label information based on the second label association information to obtain sixth label information, and store the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fourth label information, the fifth label information, and the sixth label information.

[0117] In a specific implementation, the second tag association information refers to:

[0118] The relationship between different labels in unlabeled data may exist. This relationship can be understood as the relationship between labels generated when setting labeling rules, or the relationship between labels obtained through big data.

[0119] In a specific implementation, based on the fourth label information, the fifth label information and the sixth label information, the method of storing the unlabeled data in the first storage space, the second storage space and the third storage space includes:

[0120] The unlabeled data having the fourth label information and / or the fifth label information is stored in the first storage space; the storage of the unlabeled data is based on the update of the data without duplicate content in the storage space,

[0121] The unlabeled data with the fifth label information is stored in the first storage space and the second storage space, wherein a first mapping relationship is established between the second multimodal data with the third label information stored in the first storage space and the unlabeled data with the fourth label information and / or the fifth label information, and based on the first mapping relationship, when the first storage space performs data call according to the fourth label information, the unlabeled data corresponding to the fifth label information is presented; the storage of the unlabeled data here is based on the update operation of the data of the repeated content corresponding to the fifth label in different storage spaces.

[0122] The unlabeled data with the sixth label is stored in the first storage space, the second storage space and the third storage space, wherein a second mapping relationship is established between the unlabeled data with the sixth label stored in the first storage space and the unlabeled data with the fourth label information and / or the fifth label information, and according to the second mapping relationship, when the first storage space performs data call according to the first label or the second label, the unlabeled data with the sixth label is presented;

[0123] A third mapping relationship is established between the second multimodal data with the sixth label stored in the second storage space and the second unlabeled data with the fifth label. According to the third mapping relationship, when data is called according to the fifth label in the second storage space, the unlabeled data corresponding to the sixth label is presented; the storage of the unlabeled data here is based on the update operation of the data with repeated content corresponding to the sixth label in different storage spaces.

[0124] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0125] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a multimodal data storage device.

[0127] refer to Figure 3 , the multimodal data storage device comprises:

[0128] A first space determination module 310 is configured to determine the annotation data of the multimodal data, and obtain a first storage space based on the annotation data;

[0129] The second space determination module 320 is configured to determine first data association information of the first storage space, and obtain a second storage space based on the first data association information;

[0130] A third space determination module 330 is configured to determine first tag association information of the first storage space, and obtain a third storage space based on the first tag association information;

[0131] A second association determination module 340 is configured to determine unlabeled data of the multimodal data, and obtain second data association information based on the unlabeled data and the labeled data;

[0132] The data storage determination module 350 is configured to label the unlabeled data based on the second data association information to obtain fifth label information, and store the unlabeled data in the first storage space, the second storage space and the third storage space based on the fifth label information.

[0133] In this exemplary embodiment, the first space determination module 310 is specifically configured as follows:

[0134] Determine label data of the multimodal data, and obtain a first storage space based on the label data.

[0135] In this exemplary embodiment, the second space determination module 320 is specifically configured as follows:

[0136] Determine first data association information of the first storage space, perform feature extraction on the annotated data to obtain first label information, perform label fusion on the first label information based on the first data association information to obtain second label information, and construct the second storage space based on the second label information.

[0137] In this exemplary embodiment, the third space determination module 330 is specifically configured as follows:

[0138] The first tag association information of the first tag information is determined, tag fusion is performed on the first tag information based on the first tag association information to obtain third tag information, and the third storage space is constructed based on the third tag information.

[0139] In this exemplary embodiment, the second association determination module 340 is specifically configured as follows:

[0140] Unlabeled data of the multimodal data is determined, and second data association information is obtained based on the unlabeled data and the labeled data.

[0141] In this exemplary embodiment, the data storage determination module 350 is specifically configured as follows:

[0142] Perform feature extraction on the unlabeled data to obtain feature information of the unlabeled data, label the unlabeled data based on the feature information of the unlabeled data to obtain fourth label information, label the fourth label information based on the second data association information to obtain fifth label information, determine second label association information of the fourth label information, perform label fusion on the fourth label information based on the second label association information to obtain sixth label information, and store the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fourth label information, the fifth label information, and the sixth label information.

[0143] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0144] The device of the above embodiment is used to implement the corresponding multimodal data storage method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0145] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal data storage method described in any of the above embodiments is implemented.

[0146] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0147] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0148] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0149] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0150] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0151] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0152] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0153] The electronic device of the above embodiment is used to implement the corresponding multimodal data storage method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0154] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multimodal data storage method described in any of the above embodiments.

[0155] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0156] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0157] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multimodal data storage method described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0158] Based on the same inventive concept, corresponding to the multimodal data storage method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor executes the multimodal data storage method. Corresponding to the execution subject corresponding to each step in each embodiment of the multimodal data storage method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0159] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the multimodal data storage method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0160] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0161] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0162] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0163] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0164] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified in the boxes in the flowchart and / or block diagram.

[0166] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0167] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0168] In addition, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0169] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0171] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0172] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0173] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0174] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

[0175] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A multimodal data storage method, characterized in that: include: Determine the labeling data of the multimodal data, and obtain a first storage space based on the labeling data; Determine first data association information of the first storage space, and obtain a second storage space based on the first data association information; Determine first tag association information of the first storage space, and obtain a third storage space based on the first tag association information; Determine unlabeled data of the multimodal data, and obtain second data association information based on the unlabeled data and the labeled data; The unlabeled data is labeled based on the second data association information to obtain fifth label information, and the unlabeled data is stored in the first storage space, the second storage space, and the third storage space based on the fifth label information.

2. The method according to claim 1, characterized in that The determining the first data association information of the first storage space, and obtaining the second storage space based on the first data association information, includes: Based on the first data association information, label fusion is performed on the annotated data in the first storage space to obtain second label information; Based on the second tag information, the second storage space is constructed.

3. The method according to claim 2, characterized in that The step of performing label fusion on the annotated data in the first storage space based on the first data association information to obtain second label information includes: Performing feature extraction on the labeled data to obtain first label information; The first label information is subjected to label fusion based on the first data association information to obtain the second label information.

4. The method according to claim 3, characterized in that The determining the first tag association information of the first storage space, and obtaining the third storage space based on the first tag association information, includes: Determine the first label association information of the first label information, and perform label fusion on the first label information based on the first label association information to obtain third label information; Based on the third tag information, the third storage space is constructed.

5. The method according to claim 1, characterized in that The step of labeling the unlabeled data based on the second data association information to obtain fifth label information includes: Extracting features from the unlabeled data to obtain feature information of the unlabeled data; Labeling the unlabeled data based on the unlabeled data feature information to obtain fourth label information; The fourth label information is labeled based on the second data association information to obtain fifth label information.

6. The method according to claim 5, characterized in that The storing the unlabeled data in the first storage space, the second storage space, and the third storage space based on the fifth label information includes: Determine second label association information of the fourth label information, and perform label fusion on the fourth label information based on the second label association information to obtain sixth label information; Based on the fourth label information, the fifth label information, and the sixth label information, the unlabeled data is stored in the first storage space, the second storage space, and the third storage space.

7. A multimodal data storage device, characterized in that: include: A first space determination module is configured to determine the annotation data of the multimodal data, and obtain a first storage space based on the annotation data; A second space determination module is configured to determine first data association information of the first storage space, and obtain a second storage space based on the first data association information; A third space determination module is configured to determine first tag association information of the first storage space, and obtain a third storage space based on the first tag association information; A second association determination module is configured to determine unlabeled data of the multimodal data, and obtain second data association information based on the unlabeled data and the labeled data; The data storage determination module is configured to label the unlabeled data based on the second data association information to obtain fifth label information, and store the unlabeled data in the first storage space, the second storage space and the third storage space based on the fifth label information.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.

Images