Multi-modal data storage method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal data storage method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring multi-modal data from a plurality of data sources; performing vectorization processing on each piece of modal data in the multi-modal data to obtain vector data corresponding to each piece of modal data; the modal data and the vector data corresponding to the modal data are stored in a pre-constructed database in an associated mode, the technical problem that in the prior art, multi-modal data cannot be fused and stored is solved, and the utilization rate of the multi-modal data is increased.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a storage method, device, electronic device and storage medium for multimodal data. Background Art

[0002] With the rapid development of smart grids, data collection and processing technologies in power systems are facing unprecedented challenges. Sensors, smart meters, video surveillance and other equipment widely deployed in power systems continuously generate a large amount of multimodal data. These multimodal data contain rich information about the operating status of power systems, which is of great significance for improving the operation and maintenance efficiency of power systems, predicting equipment failures, and optimizing resource allocation.

[0003] The existing technical solutions have the following shortcomings: (1) Insufficient data fusion capabilities: Most existing technologies are limited to a single modality or simple parallel processing, and it is difficult to effectively integrate multimodal data, resulting in the value of data not being fully explored. (2) Low storage efficiency: When traditional data warehouses process large-scale, heterogeneous data, the storage efficiency and query performance are limited. (3) Low intelligence level: There is a lack of intelligent means for unified data processing and analysis, which makes it difficult to meet the needs of the intelligent development of power systems. How to effectively integrate these heterogeneous, multi-source multimodal data and achieve efficient and secure storage is an urgent problem to be solved in the current information construction of power systems. Summary of the invention

[0004] In view of this, the purpose of the present application is to propose a storage method, device, electronic device and storage medium for multimodal data to overcome all or part of the deficiencies in the prior art.

[0005] Based on the above purpose, the present application provides a method for storing multimodal data, including: obtaining multimodal data from multiple data sources; vectorizing each modal data in the multimodal data to obtain vector data corresponding to each modal data; and associating the modal data and its corresponding vector data and storing them in a pre-constructed database.

[0006] Optionally, the vectorizing each modal data in the multimodal data to obtain vector data corresponding to each modal data includes: extracting features from the modal data to obtain vector data corresponding to the modal data.

[0007] Optionally, the associating and storing the modal data and the corresponding vector data in a pre-constructed database includes: clustering all the vector data to obtain multiple clusters; and associating and storing the modal data and the corresponding vector data in the database according to the clusters.

[0008] Optionally, clustering all the vector data to obtain a plurality of clusters includes: in response to determining that the similarity between any two vector data in all the vector data is greater than a predetermined similarity, dividing the two vector data into the same cluster.

[0009] Optionally, after the modal data and its corresponding vector data are associated and stored in a pre-constructed database, the method includes: obtaining the data to be retrieved; vectorizing the data to be retrieved to obtain the vector data to be retrieved; and searching the database for the modal data corresponding to the vector data to be retrieved.

[0010] Optionally, after the modal data and its corresponding vector data are associated and stored in a pre-constructed database, the method further includes: searching for the same vector data in the database; performing an association analysis on multiple modal data corresponding to the same vector data to obtain an analysis result, and sending the analysis result to a predetermined user.

[0011] Optionally, before vectorizing each modal data in the multimodal data, the method includes: performing data cleaning on the multiple types of modal data.

[0012] Based on the same inventive concept, the present application also provides a storage device for multimodal data, including: an acquisition module, configured to acquire multimodal data from multiple data sources; a processing module, configured to perform vectorization processing on each modal data in the multimodal data to obtain vector data corresponding to each modal data; a storage module, configured to associate the modal data and its corresponding vector data and store them in a pre-built database.

[0013] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0014] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method as described above.

[0015] From the above, it can be seen that the storage method, device, electronic device and storage medium of multimodal data provided by the present application include obtaining multimodal data from multiple data sources. Each modal data in the multimodal data is vectorized to obtain vector data corresponding to each modal data, and the data representation is unified, so as to achieve the purpose of unifying multi-source heterogeneous multimodal data into homogeneous data. The modal data and its corresponding vector data are associated and stored in a pre-constructed database, so as to achieve the purpose of fusion storage of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of a flow chart of a method for storing multimodal data according to an embodiment of the present application;

[0018] Figure 2 A schematic diagram of the structure of a storage device for multimodal data according to an embodiment of the present application;

[0019] Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0021] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0022] As mentioned in the background technology section, with the rapid development of smart grids, data collection and processing technologies in power systems face unprecedented challenges. Sensors, smart meters, video surveillance and other equipment widely deployed in power systems continuously generate a large amount of multimodal data. These multimodal data contain rich information about the operating status of the power system, which is of great significance for improving the operation and maintenance efficiency of the power system, predicting equipment failures, and optimizing resource allocation.

[0023] This study aims to investigate the difficulties faced by power industry organizations in data retrieval, comparison, analysis, and clustering in the management of multiple modal information such as documents and pictures. The early systems were limited by the technical conditions at the time, and used traditional manual cataloging and AI-assisted methods to manage graphic materials. Faced with a large amount of stock and incremental content of graphics and texts, manual cataloging consumes a lot of manpower and material resources, but the results are not satisfactory. Often, due to insufficient cataloging granularity and changes in cataloging rules, the description of material information is incomplete and inaccurate, which adds a lot of obstacles to future retrieval and use, causing a large amount of material to be "stored in but not retrieved", becoming a dormant data treasure. However, how to effectively integrate these heterogeneous, multi-source, multi-modal data and achieve efficient and secure storage is an urgent problem to be solved in the current information construction of the power system.

[0024] At present, some technical solutions have been proposed and applied for the processing and storage of multimodal power data. Exemplary existing technical solutions mainly include:

[0025] (1) Single-mode data processing: For specific types of data (such as only processing images or text data), a special processing algorithm is used for analysis and storage. However, this method ignores the correlation between data and is difficult to fully reflect the operating status of the power system.

[0026] (2) Parallel processing of multimodal data: Multimodal data can be processed simultaneously through parallel computing technology to improve processing speed. However, this method lacks a unified data fusion framework and is difficult to achieve effective data integration.

[0027] (3) Traditional data warehouse: Establish a unified data warehouse to centrally store multimodal data. However, traditional data warehouses have problems such as poor scalability and low query efficiency when processing heterogeneous and multimodal data.

[0028] In view of this, the present application embodiment proposes a method for storing multimodal data, referring to Figure 1 , including the following steps:

[0029] Step 101: Acquire multimodal data from multiple data sources.

[0030] In this step, sensors, smart meters, video surveillance and other equipment widely deployed in the power system continuously generate a large amount of multimodal data during operation. Exemplarily, the multimodal data generated during the operation of the power system include, but are not limited to, real-time monitoring video data, equipment status data, user electricity consumption data, environmental meteorological data, etc. Therefore, multimodal data includes data from multiple data sources. Multimodal data contains multi-source heterogeneous data and has a high degree of heterogeneity. The heterogeneity of multimodal data is reflected in the fact that it includes numerical data (such as current, voltage), image data (such as equipment appearance images), video data (such as inspection videos) and text data (such as operation logs).

[0031] Step 102: perform vectorization processing on each modal data in the multimodal data to obtain vector data corresponding to each modal data.

[0032] In this step, since the multimodal data comes from different data sources, the types of modal data in the multimodal data are not the same, resulting in the inability to uniformly store the multimodal data, making it difficult to integrate multi-source heterogeneous data. Traditional data management methods are difficult to effectively integrate these multi-source heterogeneous data, resulting in serious data island phenomena, which limits the comprehensive application and value mining of data. Even if multimodal data from multiple data sources can be collected, how to efficiently and accurately fuse these data to form a more comprehensive and accurate description of the state of the power system is a technical problem. The inconsistency of data format, sampling rate, timestamp, etc. increases the complexity of data fusion and reduces the efficiency of data processing. In order to solve the above problems, this application vectorizes each modal data in the multimodal data to obtain vector data corresponding to each modal data. The multimodal data is vectorized and unified in data representation, achieving the purpose of unifying multi-source heterogeneous multimodal data into isomorphic data.

[0033] Step 103: associate the modal data and its corresponding vector data and store them in a pre-built database.

[0034] In this step, the traditional data storage scheme multimodal data can only be stored in the database corresponding to each modal data, and multiple databases form a data warehouse, and multimodal data cannot be stored in the same database. With the development of smart grids, power data has shown an explosive growth trend. Traditional data storage solutions are often difficult to meet the storage needs, query efficiency and cost control requirements of massive data at the same time. How to improve storage efficiency and reduce storage costs on the basis of ensuring safe and reliable data storage has become an urgent problem to be solved. In this application, the vector data corresponding to each modal data belongs to isomorphic data, and the vector data is stored in a pre-built database. At the same time, the vector data also has a corresponding relationship with the modal data, so the modal data and its corresponding vector data can be associated and stored in the same database. The feature vectors after the data is converted by the vector engine will be stored in the database, and the KNN (k-nearestneighbor) algorithm built into the database will be used to recall the massive vectors in seconds. The database is designed to allow users to easily calculate, retrieve, cluster and manage feature vectors, support rich feature vector indexing algorithms and the scheduling of heterogeneous computing resources, and provide a complete user interface, data management components and graphical management tools. By associating and storing modal data and its corresponding vector data in a pre-built database, it has high reusability and does not require a large amount of software adaptive modification. It can significantly improve the overall efficiency of power data management. By efficiently integrating and fusing multi-source heterogeneous data, data islands are eliminated, and the purpose of fusing and storing multi-modal data is achieved.

[0035] It should be noted that when the multimodal data is multimodal power data, it can effectively integrate and fuse the data to achieve efficient storage, processing and intelligent analysis of the data, and provide strong support for the safe and stable operation and intelligent development of the power system. Data full-process management refers to the whole process of power multimodal data samples from sample collection, data cleaning, target labeling, labeling quality assessment, and data intelligent management. At the same time, in view of the defects of the power industry's many types of professional equipment, many operating requirements and specifications, and the difficulty of labeling team building, this patent studies the management of power multimodal data samples with multimodal sample collection and AI-assisted labeling based on the self-developed vector database, connects the existing data source with the sample library, integrates sample collection, automatic labeling, manual labeling, labeling quality inspection and data archiving, and constructs a new data labeling application mode to create an intelligent data labeling system that meets enterprise requirements and is simple to use. Construct a multimodal data intelligent labeling mode to realize the intelligent full-process management of multimodal data from sample collection, data cleaning, target labeling, and labeling quality assessment. It adopts powerful data processing capabilities and uses intelligent AI pre-labeling functions to label the uploaded samples with specified labels. Users can achieve intelligent management of the entire life cycle of multimodal power data samples, intelligently label and manage them according to models, and adapt to standard formats to truly implement data asset management. At the same time, it supports the full-link work of online labeling, from labeling task management to multi-person collaborative labeling, and performs quality inspection and export of labeled data to ensure the quality and accuracy of the data. In response to the needs and scalability requirements of different business scenarios in the power industry, the research on the expansion of pre-labeling targets, as well as the expansion of labeling data types and labeling ranges, greatly improves data labeling efficiency.

[0036] It should also be noted that the physical unit of the platform for accessing and processing multimodal data based on multi-source heterogeneous data is established, and the development and optimization of data processing algorithms that take into account both accuracy and efficiency of multimodal data using the physical unit model are realized. The method of mixed storage and management of multimodal data is adopted, and the data is extracted and quantified through comprehensive research and judgment. At the same time, the data before and after processing are compared through the results of logical algorithm execution, which verifies the accuracy and efficiency of the physical unit program based on the multi-source heterogeneous multimodal data access and processing platform. After data processing, it can be vectorized and stored and managed in a knowledge-based manner, and form the basic data resource library of the overall system. Multimodal data processing refers to the whole process of how to access the system for multi-source heterogeneous multimodal data, including multi-source heterogeneous data decoding vectorization processing, collaborative filtering processing, serialization processing and other intelligent algorithm feature processing.

[0037] Through the above scheme, multimodal data is obtained from multiple data sources. Each modal data in the multimodal data is vectorized to obtain vector data corresponding to each modal data, and the data representation is unified, so as to achieve the purpose of unifying multi-source heterogeneous multimodal data into homogeneous data. The modal data and its corresponding vector data are associated and stored in a pre-built database, so as to achieve the purpose of fusion storage of multimodal data.

[0038] In some embodiments, the vectorizing each modal data in the multimodal data to obtain vector data corresponding to each modal data includes: extracting features from the modal data to obtain vector data corresponding to the modal data.

[0039] In this embodiment, feature extraction is performed on multimodal data to obtain vector data corresponding to the modal data, and multi-source heterogeneous data is unified into isomorphic data to achieve the purpose of accurately determining vector data. It should be noted that each modal data in the multimodal data can be vectorized by a unified model, wherein the unified model is trained by multimodal data from multiple data sources. Feature extraction can also be performed on multimodal data separately. For example, text data is vectorized: before feature extraction, the text content needs to be segmented into words and sentences through natural language understanding, and the context processing and analysis are combined to extract important words from the text data, and annotate them as TAG keywords of the content, add core keywords to the content, and perform knowledge association based on the core keywords, thereby realizing complex semantic retrieval of vectorized data. Vectorization of image data: before feature extraction, the content of the image needs to be identified by AI, including information such as people, objects, scenes, and text, and the key subject of the image is determined. Deep learning technology and algorithm model iteration are used to extract features from the image data and establish the correlation vectorized data between the image and the text. Vectorize audio data: Before feature extraction, voice signals need to be converted into corresponding texts through speech recognition technology, and audio vector data need to be established using text vectorization technology and audio timeline, and features of audio vector data need to be extracted. Vectorize video data: Before feature extraction, video frames need to be extracted and processed using a combination of image vectorization technology and audio vectorization technology to extract features of video data.

[0040] It should be noted that this embodiment needs to effectively construct the correlation between multimodal data of images and texts, use deep learning to conduct in-depth research on multimodal data, and introduce semantic consistency into the correlation learning of multimodal data. Four types of semantic modes are proposed: supervised label extraction, unsupervised hypergraph semantics, local neighbors, and local preservation. The features of these unstructured data are extracted through artificial intelligence technology (deep learning algorithm) and represented by feature vectors. Then, through the optimization of vector retrieval and aggregation algorithms of vector databases and the integration of heterogeneous computing resources, cross-modal retrieval, comparison, clustering and analysis of multimodal data can be realized in the future.

[0041] In some embodiments, the associating and storing the modal data and its corresponding vector data in a pre-constructed database includes: clustering all vector data to obtain multiple clusters; and associating and storing the modal data and its corresponding vector data in the database according to the clusters.

[0042] In this embodiment, if all vector data are directly stored in the database, the utilization rate of the vector data will be relatively low. In order to solve the above problems, the present application clusters all vector data to obtain multiple clusters, and filters out vector data with similar relationships in all vector data. According to the clusters, the modal data and its corresponding vector data are associated and stored in the database. By storing the data in association according to the clusters, the organization and management of the data can be made more convenient. This classification method helps to organize the data clearly, making the search, update and maintenance of the data more efficient. A hierarchical storage strategy is adopted to optimize the allocation of storage resources, reduce storage costs and improve data access efficiency; an intelligent analysis platform is built to deeply explore the value of power data, provide accurate prediction and decision-making support for the safe and stable operation of the power system, and promote the intelligent transformation and development of the power industry.

[0043] In some embodiments, clustering all the vector data to obtain a plurality of clusters includes: in response to determining that the similarity between any two vector data in all the vector data is greater than a predetermined similarity, dividing the two vector data into the same cluster.

[0044] In this embodiment, when the similarity between any two vector data in all vector data is greater than the predetermined similarity, it means that the above two vector data have similarity. Therefore, the above two vector data are divided into the same cluster so that "the points within the cluster are close enough, and the points between clusters are far enough", thereby completing the optimal division of all vector data and reducing the complexity of the data.

[0045] In some embodiments, after the modal data and its corresponding vector data are associated and stored in a pre-constructed database, the method includes: obtaining the data to be retrieved; vectorizing the data to be retrieved to obtain the vector data to be retrieved; and searching the database for the modal data corresponding to the vector data to be retrieved.

[0046] In this embodiment, the data to be retrieved is obtained, wherein the data to be retrieved may be one of text data, image data, audio data or video data. Since the format of the data to be retrieved and the data in the database are not uniform, the data to be retrieved is vectorized to obtain the vector data to be retrieved, and the modal data corresponding to the vector to be retrieved is searched in the database. By finding the modal data corresponding to the data to be retrieved, it is said that the above two types of data are associated. Exemplarily, the above two types of data are derived from the same project. In addition, the above two types of data may belong to data of different modalities. Exemplarily, data retrieval refers to the ability to realize cross-modal data retrieval based on a vectorized database, including searching for text by text, searching for pictures by text, searching for audio by text, searching for videos by text, searching for pictures by pictures, searching for videos by pictures, etc., supporting multi-dimensional retrieval of image information, enabling users to quickly and accurately retrieve and filter sample information, thereby achieving the purpose of cross-modal retrieval.

[0047] It should be noted that, in response to the application needs of the power industry, a demonstration of power multimodal data fusion, collection and hierarchical storage is constructed, and a data model is established for analysis and judgment. Through manual verification, various technical indicators such as data fusion information retrieval efficiency, analysis accuracy, and judgment rationality are evaluated.

[0048] In some embodiments, after the modal data and its corresponding vector data are associated and stored in a pre-constructed database, the method further includes: searching for the same vector data in the database; performing an association analysis on multiple modal data corresponding to the same vector data to obtain an analysis result, and sending the analysis result to a predetermined user.

[0049] In this embodiment, multimodal data contains rich value, but lacks effective intelligent analysis tools and methods, so that the data cannot fully play its role in assisting decision-making. This embodiment searches for the same vector data in the database, performs correlation analysis on multiple modal data corresponding to the same vector data, and obtains analysis results, wherein correlation analysis refers to the existence of information on the same thing in multimodal data, and these multimodal data need to be correlated with each other for analysis, including the establishment of data feature models, information clustering, entity analysis, target tracking, etc. In the case of the same vector data, it means that there is a correlation between the vector data, and the same vector data is subjected to correlation analysis to obtain analysis results, and the analysis results are sent to the predetermined user, so that the predetermined user can perform response analysis based on the analysis combination, so as to achieve deep insight and prediction of the complex operating status of multimodal data in the power system.

[0050] In some embodiments, before vectorizing each modality data in the multimodal data, the method includes: performing data cleaning on the multiple types of modality data.

[0051] In this embodiment, data cleaning is an important step before data analysis. Through data cleaning, the quality of data can be ensured, laying a solid foundation for subsequent data processing and analysis. The role of data cleaning is mainly reflected in improving the quality, efficiency, and accuracy of analysis, ensuring the consistency of data, and data cleaning ensures the authenticity and validity of data by detecting and processing abnormal values, duplicate values, and error values. This process includes auditing and checking the authenticity of data, processing abnormal values, deleting duplicate records, and correcting erroneous data, thereby ensuring the correctness of data.

[0052] Perform data cleaning operations, such as deleting duplicate sample data, so that it can subsequently support online multi-person annotation of sample images and perform quality inspection on model training data, thereby providing intuitive and efficient multimodal data retrieval, annotation and management tools, making the processing and management of visual image data more convenient and efficient.

[0053] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0054] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0055] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a storage device for multimodal data.

[0056] refer to Figure 2 , the storage device for multimodal data comprises:

[0057] The acquisition module 10 is configured to acquire multimodal data from multiple data sources.

[0058] The processing module 20 is configured to perform vectorization processing on each modality data in the multimodal data to obtain vector data corresponding to each modality data.

[0059] The storage module 30 is configured to store the modal data and its corresponding vector data in association with each other in a pre-built database.

[0060] Through the above device, multimodal data is obtained from multiple data sources. Each modal data in the multimodal data is vectorized to obtain vector data corresponding to each modal data, and the data representation is unified, so as to achieve the purpose of unifying multi-source heterogeneous multimodal data into homogeneous data. The modal data and its corresponding vector data are associated and stored in a pre-built database, so as to achieve the purpose of fusion storage of multimodal data.

[0061] In some embodiments, the processing module 20 is further configured to perform feature extraction on the modal data to obtain vector data corresponding to the modal data.

[0062] In some embodiments, the storage module 30 is further configured to cluster all vector data to obtain a plurality of clusters; and according to the clusters, the modal data and its corresponding vector data are associated and stored in the database.

[0063] In some embodiments, the storage module 30 is further configured to, in response to determining that the similarity between any two vector data in the entire vector data is greater than a predetermined similarity, divide the two vector data into the same cluster.

[0064] In some embodiments, a search module is also included, which is configured to obtain the data to be retrieved after the modal data and its corresponding vector data are associated and stored in a pre-built database; vectorize the data to be retrieved to obtain the vector data to be retrieved; and search the database for the modal data corresponding to the vector data to be retrieved.

[0065] In some embodiments, a sending module is also included, and the sending module is configured to search for the same vector data in the database after the modal data and its corresponding vector data are associated and stored in a pre-built database; perform association analysis on multiple modal data corresponding to the same vector data to obtain analysis results, and send the analysis results to a predetermined user.

[0066] In some embodiments, a data cleaning module is further included, and the data cleaning module is configured to perform data cleaning on the multiple types of modal data before vectorizing each modal data in the multimodal data.

[0067] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0068] The device of the above embodiment is used to implement the corresponding multimodal data storage method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0069] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for storing multimodal data as described in any of the above embodiments is implemented.

[0070] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0071] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0072] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0073] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0074] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0075] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0076] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0077] The electronic device of the above embodiment is used to implement the corresponding multimodal data storage method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0078] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multimodal data storage method described in any of the above embodiments.

[0079] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0080] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for storing multimodal data as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0081] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the multimodal data storage method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.

[0082] It should be noted that the embodiments of the present application can be further described in the following manner:

[0083] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0084] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0085] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0086] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0087] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0088] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0089] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0090] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for storing multimodal data, characterized in that: include: Acquire multimodal data from multiple data sources; Performing vectorization processing on each modal data in the multimodal data to obtain vector data corresponding to each modal data; The modal data and the corresponding vector data are associated and stored in a pre-constructed database.

2. The method according to claim 1, characterized in that: The vectorizing process is performed on each modal data in the multimodal data to obtain vector data corresponding to each modal data, including: Feature extraction is performed on the modal data to obtain vector data corresponding to the modal data.

3. The method according to claim 1, characterized in that The step of associating and storing the modal data and the corresponding vector data in a pre-built database comprises: Cluster all vector data to obtain multiple clusters; According to the cluster, the modal data and its corresponding vector data are associated and stored in the database.

4. The method according to claim 1, characterized in that: The clustering of all vector data is performed to obtain multiple clusters, including: In response to determining that the similarity between any two vector data among all the vector data is greater than a predetermined similarity, the two vector data are divided into the same cluster.

5. The method according to claim 1, characterized in that After the modal data and the corresponding vector data are stored in association in a pre-built database, the method includes: Get the data to be retrieved; Performing vectorization processing on the data to be retrieved to obtain vector data to be retrieved; The modal data corresponding to the vector data to be retrieved is searched in the database.

6. The method according to claim 1, characterized in that After the modal data and the corresponding vector data are stored in association in a pre-built database, the method further includes: Searching for the same vector data in the database; The multiple modal data corresponding to the same vector data are subjected to correlation analysis to obtain analysis results, and the analysis results are sent to a predetermined user.

7. The method according to claim 1, characterized in that Before vectorizing each modality data in the multimodality data, the method includes: Data cleaning is performed on the multiple types of modal data.

8. A storage device for multimodal data, characterized in that: include: An acquisition module, configured to acquire multimodal data from multiple data sources; A processing module is configured to perform vectorization processing on each modal data in the multimodal data to obtain vector data corresponding to each modal data; The storage module is configured to associate and store the modal data and its corresponding vector data in a pre-built database.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images