Multimodal data integration management method, apparatus, and medium

Abstract

The application provides a multi-modal data integrated management method, device and medium, the multi-modal data integrated management method comprises the following steps: temporarily storing the first multi-modal data accessed; preprocessing the first multi-modal data to obtain second multi-modal data; extracting the second multi-modal data to obtain metadata comprising data characteristics, data sources and storage locations; and storing the metadata by using a corresponding data storage mode. A customizable algorithm is used to realize the unified integration of text, sound and visual data, the metadata of the text, sound and visual data are uniformly extracted by using the ontology model / meta model, and the unified storage of the multi-modal data is realized by using the object storage and the structured storage mode, the unified management of the high-reliability and high-performance access of the multi-modal data is realized by means of the metadata and the object storage technology, and the multi-modal data analysis efficiency and accuracy are greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a multi-modal data integrated management method and device and medium. BACKGROUND

[0002] With the rapid development of information technology, multi-modal data is becoming the main form of big data, so it is of great value to give computers the ability to process and manage multi-modal data through the access, storage and management methods of multi-modal data.

[0003] Current data storage management is for single-modal data, with the development of multi-modal technology, messages containing images with sound, text and pictures and other multi-modal data need to be processed, in the traditional single-modal processing mode, unified management of multi-modal data cannot be realized, so it is difficult to ensure efficiency when processing multi-modal data. SUMMARY

[0004] The main purpose of the embodiment of the present application is to provide a multi-modal data integrated management method, device and medium, which improves the extraction, storage and management efficiency of multi-modal data.

[0005] One aspect of the present application provides a multi-modal data integrated management method, comprising: in response to a multi-modal data management request, performing temporary storage on accessed first multi-modal data, the first multi-modal data being used to represent a data source, the data source including at least one of text data, sound data and visual data; performing preprocessing on the first multi-modal data to obtain second multi-modal data, the preprocessing including at least one of data cleaning and redundant data removal; performing extraction on the second multi-modal data to obtain metadata including data features, data sources and storage locations; and storing the metadata according to the type of the metadata using a corresponding data storage method.

[0006] According to the multi-modal data integrated management method, wherein the temporary storage of the accessed first multi-modal data comprises:

[0007] The text data, the sound data and the visual data in the data source are extracted to a temporary intermediate layer through an ETL process, and the data source is at least one of a distributed data source and a heterogeneous data source.

[0008] According to the multi-modal data integration management method, the pre-processing of the first multi-modal data is to obtain second multi-modal data, and the pre-processing of the text data includes at least one of deleting special symbols, removing extra spaces in characters, deleting continuously appearing punctuation, deleting data lines without Chinese characters, deleting useless text with short length, and splitting each line of text, and the pre-processed text data is stored in a file line by line through one of a regular expression and a text splitting function; the pre-processing of the sound data includes detecting noise data by using a k-means clustering method, and removing noise data by using an endpoint detection algorithm; and the pre-processing of the visual data includes at least one of removing damaged pictures, removing blurred pictures, and removing similar pictures based on an OPENCV Laplacian operator.

[0009] According to the multi-modal data integration management method, the extraction of the text data includes obtaining at least one of a keyword frequency and a length of the text data; the keyword frequency is filtered by using a term frequency-inverse document frequency algorithm, and a document keyword frequency is counted; and the length of the text data is obtained by using a statistical function to calculate a text length.

[0010] According to the multi-modal data integration management method, the extraction of the sound data includes at least one of sound language class recognition, volume recognition, zero-crossing rate recognition, and pitch recognition; the language class recognition includes endpoint processing of the sound data, feature extraction of the sound by using MFCC, and inputting the features into a classifier to recognize, classify, and output sound features to obtain a language recognition result; the volume recognition includes determining a decibel value of each audio frame of the sound data; the zero-crossing rate recognition includes determining a number of times that a sampling value of each audio frame of the sound data passes through zero; and the pitch recognition is determined by pitch=69+12*log2(F / 440), where pitch is the pitch, and F is a fundamental frequency.

[0011]

[0012]

[0013] According to the multi-modal data integration management method, the extraction of the visual data includes at least one of a storage amount, a resolution, a saturation, a contrast, and a brightness of the sound; the storage amount is determined by a formula c=n*m*p / 8, where c is the storage amount, n represents horizontal pixels, m represents vertical pixels, and p represents a number of bits required for each pixel; and the resolution, the saturation, the contrast, and the brightness are identified by calling an OPENCV interface.

[0014] ​​According to the multi-modal data integration management method, the storing of the metadata is performed in a corresponding data storage mode according to the type of the metadata, and the storing of the metadata into a relational database MySql cluster is performed, the relational database MySql cluster including storage information of multi-modal data and the metadata characteristic information; and a sharding and indexing mode is used to query the first modal data when the relational database MySql cluster is accessed.

[0015] 8. The multi-modal data integration management method of claim 1, further comprising:

[0016] The Hadoop distribution is used for storing and managing the text data, and when the first multi-modal data is stored and calculated, file storage based on double-layer hash coding and HBase is used for optimization; when small files are merged, an extensible hash function is used to construct an index file storage bucket, so that the index file can be dynamically expanded as needed; a hash function is used in each storage bucket to store the position of each file index information in the index file, and when a request for accessing a text file is received, the corresponding file index information in the storage bucket is read to determine the text file; the HBase is used to store the file index information, and an identification column is set to identify data of different modalities, so as to facilitate the storage and management of data of different modalities and improve the reading speed of the file; and a metadata prefetching mechanism based on LRU is established, and an LZ4 compression algorithm is used to compress and store the merged file;

[0017] The Minio object storage method is used to store the sound data and the visual data, wherein the sound data and the visual data include corresponding buckets, users and tenants, the bucket is used to store the second multi-modal data; the tenant is used to isolate storage resources, and the bucket and the object are created through the tenant; the user is an account created under the tenant and used to access different buckets.

[0018] Another aspect of the embodiment of the application provides a multi-modal data integration management device, comprising:

[0019] A first module is configured to perform temporary storage on accessed first multi-modal data according to a multi-modal data management request, the first multi-modal data being used to represent a data source, and the data source including at least one of text data, sound data and visual data.

[0020] A second module is configured to perform preprocessing on the first multi-modal data to obtain second multi-modal data, the preprocessing including at least one of data cleaning and redundant data removal.

[0021] a third module configured to perform extraction on the second multi-modal data to obtain metadata comprising data features, data sources and storage locations;

[0022] a fourth module configured to store the metadata according to a corresponding data storage mode based on the type of the metadata.

[0023] The embodiment of the present application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method described above.

[0024] Additional aspects and advantages of the present application will be given in part in the following description, become apparent from the following description, or be learned by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0026] Figure 1 is a flowchart of multi-modal data integration management of the embodiment of the present application.

[0027] Figure 2 is a flowchart of preprocessing of the embodiment of the present application.

[0028] Figure 3 is a flowchart of metadata extraction of the embodiment of the present application.

[0029] Figure 4 is a schematic diagram of gestures in different regions.

[0030] Figure 5 is a schematic diagram of a device for multi-modal data integration management of the embodiment of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below with reference to the attached drawings, wherein the same or similar components have the same or similar designations throughout the several figures of the drawings and any altered designation implies alteration of the item so designated. In the following description, suffixes "module", "part" or "unit" used for components are merely intended for facilitating description of the present application, and they are not intended to indicate specialized meanings or functions. Therefore, "module", "part" or "unit" can be used interchangeably. "First", "second", etc. are used only to distinguish technical features for the purpose of explanation, and they cannot be understood to indicate or imply relative importance or to implicitly indicate the number of technical features indicated or the chronological relationship between the technical features indicated. In the following description, consecutive numbering of method steps is for the convenience of review and understanding, and adjusting the implementation order between the steps does not affect the technical effects achieved by the technical solution of the present application in combination with the overall technical solution of the present application and the logical relationship between the steps. The embodiments described below with reference to the drawings are exemplary and are used only to explain the present application, and cannot be understood as limiting the present application.

[0032] Terminology:

[0033] Multi-modal: Each source or form of information can be called a modality. For example, a person has touch, hearing, vision, smell; the media of information has sound, video, text, etc.; a variety of sensors such as radar, infrared, accelerometer, etc. Each of the above can be called a modality. At the same time, modalities can also have very broad definitions, such as we can consider two different languages as two modalities, or even the data sets collected in two different situations can be considered as two modalities. Therefore, multi-modal machine learning, the full name in English is MultiModal Machine Learning (MMML), aims to realize the ability to process and understand multi-source modal information through machine learning methods. The current popular research direction is multi-modal learning between images, videos, audio, semantics.

[0034] Big data: Big data is a data set with a particularly large volume and cannot be captured, managed and processed by traditional database tools. Big data mainly contains four dimensional characteristics, which are large data volume, various data types, fast processing speed and low value density. The basic processing flow of big data mainly includes data collection, storage management, processing analysis and result presentation. Therefore, from the perspective of the whole data analysis process, big data technology mainly includes data collection and preprocessing, data storage and management, data processing and analysis, data visualization, data security and privacy protection and other aspects. For different data types and characteristics, big data corresponds to different big data computing modes, which are batch processing computing (for batch processing of large-scale data), stream processing (for real-time computing of stream data), graph computing (for processing of large-scale graph structure data), and query analysis computing (storage management and query analysis of large-scale data).

[0035] Data integration: Data integration is to logically or physically integrate data in several scattered data sources into a unified data set. The core task of data integration is to integrate the distributed heterogeneous data sources associated with each other together, so that users can access these data sources in a transparent way. Integration means maintaining the consistency of the overall data of the data source and improving the efficiency of information sharing and utilization; the transparent way means that users do not need to care about how to realize the access to the data of heterogeneous data sources, but only care about how to access the data. Common data integration methods mainly include federal database, middleware integration method and data warehouse method. Data integration can greatly improve the response speed and efficiency of data analysis, and can also promote intelligent analysis of multiple data sources, greatly reducing the management cost and complexity of data set.

[0036] Metadata: Metadata, also known as intermediate data and relay data, is a kind of data describing data, mainly describing some information of data attributes. The storage location, operation permission and other quantitative information of real data can be reflected in metadata.

[0037] Object storage: Object storage is a general term used to describe the solution and processing method of discrete units. Objects in a hierarchical structure will not have hierarchical structure, and are characterized by extended metadata. Object storage provides object-based data storage services based on distributed systems, and can access data through RESTful API data read-write interface and rich SDK interface. It can be understood as a storage service in a shopping mall. Customers hand over the bag (data) to the waiter (API interface), and the waiter gives the customer a voucher (object address URL). When the customer wants to take the bag, he only needs to provide the voucher, without knowing where and in what form his bag is stored, saving time, effort and energy.

[0038] IF-IDF algorithm: TF-IDF is a term frequency-inverse document frequency statistical method used to evaluate the importance of a word to a file set or a file in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a file, but at the same time, it decreases in inverse proportion to the frequency of its appearance in the corpus.

[0039] Reference Figure 1 , Figure 1 is a flowchart of a multi-modal data integration management model. It includes but is not limited to processes S100-S400:

[0040] S100, in response to a multi-modal data management request, performing temporary storage on the accessed first multi-modal data, the first multi-modal data being used to represent a data source, the data source including at least one of text data, sound data and visual data.

[0041] In some embodiments, the temporary storage represents the aggregation of original multi-modal data, for example, the access of text, sound, visual multi-modal data mainly uses the ETL (Extraction Transformation Loading) process to extract the text, sound, visual data in the distributed, heterogeneous data source to the temporary intermediate layer.

[0042] S200, performing preprocessing on the first multi-modal data to obtain second multi-modal data, the preprocessing including at least one of data cleaning and redundant data removal.

[0043] In some embodiments, referring to the data preprocessing flowchart as shown in Figure 2 , which includes but is not limited to steps S210-S220:

[0044] S210, obtaining the first multi-modal data and identifying the type of the first multi-modal data;

[0045] S220, performing corresponding cleaning processing according to the type of the first multi-modal data.

[0046] In some embodiments, the preprocessing of the first multi-modal data includes preprocessing of text data, sound data and visual data respectively.

[0047] In some embodiments, the preprocessing of the text data includes deleting special symbols, removing redundant spaces in characters, deleting consecutive punctuation marks, deleting data lines without Chinese characters, deleting short useless text, splitting each line of text and storing in a txt file line by line. Regular expressions and text splitting functions are mainly used here.

[0048] In some embodiments, the cleaning of the sound data is processed using an endpoint detection algorithm. Endpoint detection, also known as voice activity detection, aims to distinguish between voice and non-voice regions, that is, to accurately locate the start and end points of the voice from the voice with noise, to remove the silent part and the noise part, and to find the truly effective content of the voice. Then remove the data noise, and use the k-means clustering method to detect noise data. Clustering organizes similar values into groups or "clusters". Values that fall outside the cluster (isolated points) are considered noise.

[0049] In some embodiments, the cleaning of the visual data mainly includes the process of removing damaged pictures, removing blurred pictures, and removing similar pictures. The removal of blurred pictures is determined by using the Laplacian operator provided by OPENCV. The operator calculates the sharpness value. The smaller the value, the lower the sharpness, and the more blurred it is. The removal of similar pictures is obtained by calculating the similarity of the pictures through the histogram.

[0050] S300, performing extraction on the second multi-modal data to obtain metadata including data characteristics, data sources and storage locations.

[0051] In some embodiments, referring to the data preprocessing flowchart as shown in Figure 2 , which includes but is not limited to steps S310-S320:

[0052] S310, obtaining the second multi-modal data and identifying the type of the second multi-modal data;

[0053] S320, according to the data type of the second multi-modal data, using the corresponding extraction method to obtain the metadata of the second multi-modal data.

[0054] Exemplarily, for the second multi-modal data after access preprocessing, different categories of metadata extraction are performed according to the different characteristics of each modal data, as follows:

[0055] (1) Extraction of text data. The text metadata includes the keyword frequency, length, source information and storage location of the text.

[0056] Among them, the keyword frequency is obtained by using the term frequency-inverse document frequency (TF-IDF) algorithm. The use of the TF-IDF algorithm needs to filter the stop words first, and then count the document keyword frequency.

[0057] Length: The text length is calculated using a conventional statistical text length function.

[0058] Source information: The source information is the description of the source of the text data, such as short message data text.

[0059] Storage Location: It is the description of the actual storage location of the access text source file.

[0060] Sound metadata includes sound class identification (such as wind sound, bird sound, human voice, car siren sound, etc.), volume, zero crossing rate, pitch, source information, storage location.

[0061] Class identification: After the endpoint processing of the sound, the sound feature extraction is performed using MFCC (Mel-frequency cepstral coefficients), and then these features are transmitted as parameters into the classifier for sound feature recognition, classification, and output.

[0062] Volume: Volume represents the intensity of sound, also known as loudness, intensity or energy, which can be compared by the signal amplitude size within an audio frame. The method of calculating volume here is to take the sum of the square values of the signal values of each audio frame, and then take the logarithm value with base 10, and multiply by 10. The value obtained by this method is in decibels (Decibels), which is a relative intensity value, which is more consistent with the feeling of human ear to the size of sound.

[0063] Zero crossing rate: Zero crossing rate (Zero Crossing Rate, ZCR for short) is the number of times the sampling value of the sound signal passes through zero in each frame of the sound signal, which is calculated using a calculation function.

[0064] Pitch: Pitch represents the high and low of sound frequency, and this frequency refers to the fundamental frequency (Fundamental Frequency), which is the inverse of the fundamental period (Fundamental Period). The calculation formula of pitch is pitch = 69 + 12*log2(F 440 ), where F is the fundamental frequency.

[0065] Source information: Source information is the description of the source of sound data, such as sound call data.

[0066] Storage Location: It is the description of the actual storage location of the access sound source file.

[0067] Visual data metadata mainly includes image storage, resolution, saturation, contrast, brightness, source information, storage location.

[0068] Storage: The calculation formula of image storage is c = n*m*p / 8, where c is the storage, n represents the horizontal pixel, m represents the vertical pixel, and p represents the number of bits required for each pixel.

[0069] Resolution, saturation, contrast, brightness: calculated by calling the OpenCV (Open Source Computer Vision Library) interface.

[0070] Source information: source information is the source description of visual data, such as website screenshot data.

[0071] Storage location: is to describe the actual storage location of accessing visual source file.

[0072] S400, according to the type of metadata, the corresponding data storage mode is used to store metadata.

[0073] In some embodiments, reference Figure 4 The multi-modal data storage and management process diagram shown in the figure includes metadata storage based on original data, storage and management of multi-modal data:

[0074] Metadata storage, metadata is mainly stored in the relational database MySql cluster, which stores the storage information and various attribute information of each multi-modal data. Here, the cluster method is used for storage to ensure data security. Each access to multi-modal data needs to query and retrieve the original data first. Here, in order to improve the retrieval efficiency of metadata, the technology of sharding and indexing is used to achieve it.

[0075] Storage and management of multi-modal data, using Hadoop distributed file system (HDFS) to store and manage multi-modal original text data, when storing and calculating files, using a mass small file storage optimization method based on double-layer hash coding and HBase. When merging small files, use an extensible hash function to build an index file storage bucket, so that the index file can be dynamically expanded according to needs, and the file appending function is realized. In each storage bucket, use a hash function to store the location of each file index information in the index file. When accessing text files, you don't need to read all the file index information, you just need to read the index information in the corresponding storage bucket, so you can read the file in O(1) time complexity, which improves the file search efficiency; use HBase to store file index information, and set an identification column to identify data of different modalities, which is convenient for storage and management of different modal data, and improves the reading speed of files; in order to further optimize the storage performance, this paper also establishes a metadata prefetching mechanism based on LRU, and uses LZ4 compression algorithm to compress and store the merged files. By comparing file access performance and NameNode memory usage.

[0076] The sound data and the visual data are stored using a Minio object storage technology. The storage of each object corresponds to a corresponding bucket, user and tenant. The bucket is a logical abstraction of a plurality of objects and is a container for storing objects. The tenant is used to isolate storage resources. The bucket and the object can be established under the tenant. The user is an account created under the tenant for accessing different buckets. The purpose of setting the bucket, the user and the tenant for each object is to manage the resources safely and conveniently. Different data is placed under different buckets and different users and tenants perform permissions.

[0077] According to the above embodiment, the technical scheme of the application has at least the following beneficial effects: the unified integration of text, sound and visual data is realized by using a customizable algorithm, the metadata of text, sound and visual data is uniformly extracted by using an ontology model / meta model, and the unified storage of multiple modalities is realized by using object storage assisted by structured storage, the unified management of high-reliability and high-performance access of multi-modal data is realized by means of metadata and object storage technology, and the analysis efficiency and accuracy are greatly improved by extracting and fusing the characteristics of multiple modal data.

[0078] Figure 5 The figure is a multi-modal data integration management and analysis device of an embodiment of the application. The device comprises a first module 501, a second module 502, a third module 503 and a fourth module 504.

[0079] The first module is configured to perform temporary storage on the accessed first multi-modal data according to a multi-modal data management request, and the first multi-modal data is used to represent a data source, and the data source comprises at least one of text data, sound data and visual data; the second module is configured to perform preprocessing on the first multi-modal data to obtain second multi-modal data, and the preprocessing comprises at least one of data cleaning and redundant data removal; the third module is configured to perform extraction on the second multi-modal data to obtain metadata comprising data characteristics, data sources and storage locations; and the fourth module is configured to store the metadata by using a corresponding data storage mode according to the type of the metadata.

[0080] Exemplarily, under cooperation of the collection module, the display module, the gesture recognition module and the article selection module in the device, the embodiment device can implement any one of the foregoing multi-modal data integrated management methods, that is, in response to a multi-modal data management request, performing temporary storage on accessed first multi-modal data, the first multi-modal data being used to represent a data source, the data source including at least one of text data, sound data and visual data; performing preprocessing on the first multi-modal data to obtain second multi-modal data, the preprocessing including at least one of data cleaning and redundant data elimination; performing extraction on the second multi-modal data to obtain metadata including data characteristics, data source and storage location; and performing storage on the metadata according to a type of the metadata by using a corresponding data storage mode. Embodiments of the application realize unified integration of text, sound and visual data by using customizable algorithms, unified extraction of metadata of text, sound and visual data by using an ontology model / meta model, and unified storage of multi-modal data by using object storage supplemented by a structured storage mode, thereby realizing unified management of high-reliability and high-performance access of multi-modal data, and greatly improving analysis efficiency and accuracy by extracting and fusing characteristics of multiple modal data.

[0081] In some alternative embodiments, the functions / operations mentioned in the block diagrams can not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially concurrently or the blocks can sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flow diagrams of the application are provided by way of example only. The disclosed methods are not limited to the operations and logical flows presented in this specification. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of a larger operation are executed independently.

[0082] The embodiment of the application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the foregoing multi-modal data integrated management method.

[0083] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the actual implementation of the modules, in combination with their attributes, functions, and internal relationships, are to be understood within the context of the devices disclosed herein. Thus, those skilled in the art with access to patents, scientific journals, and other public sources known by those skilled in the art will be able to practice the application as set forth in the claims without undue experimentation, using ordinary skill in the art along with the present disclosure. It is also to be understood that the specific concepts disclosed are merely illustrative and that the scope of the present application is to be determined by the entire scope of the claims, along with all equivalents of the claims and their equivalents.

[0084] If the functions are implemented in software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0085] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer readable medium for use by an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from the instruction execution system, device or apparatus, or in conjunction with these instructions. For the purpose of this specification, "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by an instruction execution system, device or apparatus, or in conjunction with these instructions.

[0086] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0087] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques can be used to implement the hardware used in the described embodiments: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0088] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In the specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific feature, structure, material or characteristic described can be combined in any appropriate manner in one or more embodiments or examples.

[0089] Although embodiments of the application have been shown and described, it would be recognized by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application, which is defined by the following claims and their equivalents.

[0090] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A multi-modal data integration management method, characterized by, The method comprises: in response to a multi-modal data management request, performing temporary storage on accessed first multi-modal data, the first multi-modal data being used to represent a data source, the data source comprising at least one of text data, sound data and visual data; performing preprocessing on the first multi-modal data to obtain second multi-modal data, the preprocessing comprising at least one of data cleaning and redundant data removal; performing extraction on the second multi-modal data to obtain metadata comprising data features, data sources and storage locations; storing the metadata according to the type of the metadata using a corresponding data storage method; the extraction of the sound data comprises at least one of prosodic recognition, volume recognition, zero-crossing rate recognition and pitch recognition of the sound; the prosodic recognition comprises endpoint processing of the sound data, feature extraction of the sound using MFCC, and inputting the features as parameters into a classifier for recognition, classification and output of the sound features to obtain a language recognition result; the volume recognition comprises determining the decibel value of each audio frame of the sound data; the zero-crossing rate recognition comprises determining the number of times the sampling value of each audio frame of the sound data passes through zero; the pitch recognition is determined by where pitch is the pitch and F is the fundamental frequency; the extraction of the visual data comprises at least one of the storage amount, resolution, saturation, contrast and brightness of the sound: The storage amount is calculated by the formula determination, where c is the storage amount, n represents the horizontal pixels, m represents the vertical pixels, and p represents the number of bits required for each pixel. the resolution, saturation, contrast and brightness are identified by calling the OPENCV interface; the method further comprises: using Hadoop distribution to store and manage the text data, and when the first multi-modal data stores and calculates files, optimizing the file storage based on double-layer hash coding and HBase; when merging small files, using an extensible hash function to build an index file storage bucket, so that the index file dynamically expands as needed; using a hash function in each storage bucket to store the location of each file index information in the index file, determining the text file by reading the file index information in the corresponding storage bucket when receiving a request to access the text file; using HBase to store file index information and setting an identification column to identify different modal data, facilitating the storage and management of different modal data and improving the reading speed of the file; and establishing an LRU-based metadata prefetching mechanism and using the LZ4 compression algorithm to compress and store the merged files; using the Minio object storage method to store the sound data and the visual data, wherein the sound data and the visual data comprise corresponding buckets, users and tenants, the bucket being used to store the second multi-modal data; wherein the tenant is used to isolate storage resources, and the tenant is used to create a bucket and store objects; wherein the user is an account created under the tenant to access different buckets.

2. The multi-modal data integration management method of claim 1, wherein, the temporary storage of the accessed first multi-modal data comprises: The text data, the sound data and the visual data in the data source are extracted to a temporary intermediate layer through an ETL process, the data source being at least one of a distributed data source and a heterogeneous data source.

3. The multi-modal data integration management method of claim 1, wherein, The pre-processing of the first multi-modal data to obtain second multi-modal data comprises: The pre-processing of the text data comprises at least one of deleting special symbols, removing redundant spaces in characters, deleting continuously appearing punctuation marks, deleting data lines without Chinese characters, deleting short useless texts and splitting each line of text, and the pre-processed text data is stored in a file line by line through one of a regular expression and a text splitting function; The pre-processing of the sound data adopts a k-means clustering method to detect noise data, and an endpoint detection algorithm to remove noise data; The pre-processing of the visual data comprises at least one of removing damaged pictures, removing blurred pictures and removing similar pictures based on an OPENCV Laplacian operator.

4. The multi-modal data integration management method of claim 1, wherein, The extraction of the text data comprises obtaining at least one of a keyword frequency and a length of the text data; The keyword frequency is filtered by stop words using a term frequency-inverse document frequency algorithm, and the keyword frequency of a document is counted; The length of the text data is obtained by calculating the text length using a statistical function.

5. The multi-modal data integration management method of claim 1, wherein, The storage of the metadata according to the type of the metadata comprises: The metadata is stored in a relational database MySql cluster, the relational database MySql cluster comprising storage information of multi-modal data and characteristic information of the metadata; When the relational database MySql cluster is accessed, a first modal data is queried using a sharding and indexing method.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by a processor to realize the multi-modal data integration management method of any one of claims 1-5.

Images