A multi-modal corpus storage method and device, computer equipment and medium

Abstract

The application discloses a multi-modal corpus storage method and device, computer equipment and medium. The compression method respectively performs compression processing on the text modal data and audio-video modal data of the multi-modal corpus, adopts a suitable compression algorithm according to the characteristics of different modal data, and performs similarity calculation on the text compression features, audio-video compression features and corresponding modal features of the newly added multi-modal corpus, so that the semantic redundancy of the newly added multi-modal corpus and the historical compressed corpus can be identified, the fusion compression process is triggered to eliminate cross-modal redundancy and avoid repeated storage of the same semantic information. Furthermore, the corpus features of the corpus not triggering the fusion process are independently stored, and the corpus features triggering the fusion process are fused to generate multi-modal joint features, so that the amount of data stored is further reduced. The corpus features are migrated to the corresponding data storage layer in combination with the access frequency and response time, thereby reducing the storage cost.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a method, apparatus, computer equipment and medium for storing multimodal corpora. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, the application of multimodal corpora (corpora containing data of various modalities, including text, images, audio, and video) is becoming increasingly widespread, involving fields such as government services, social media, and video surveillance. These applications require the storage of large amounts of multimodal corpora. To facilitate the management of these large amounts of stored multimodal corpora, it is necessary to compress the corpora before storing the compressed data.

[0003] For corpus compression, existing technologies mostly employ single-modal static compression methods, such as lossy compression of images using the JPEG standard and audio encoding compression using the MP3 standard. These compression methods only optimize storage for single-modal data and cannot identify and eliminate semantic redundancy between multimodal corpora. While some existing technologies introduce incremental learning frameworks to adapt to newly added multimodal corpora, they lack integrated compression mechanisms, requiring full storage of both new and historical corpora. As data accumulates, storage space usage increases linearly, failing to achieve dynamic lightweighting of corpora and failing to meet the demands for rapid processing and efficient storage of new corpora in specific application scenarios.

[0004] After compressing the multimodal corpus, it is still necessary to store the compressed multimodal corpus. However, existing storage methods usually involve simply splicing or storing data from different modalities such as text, audio, and video independently. Since text, audio, and video data have different data characteristics and redundancy, this storage method does not fully consider the correlation and compressibility between different modalities, resulting in a large amount of wasted storage space. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, and medium for storing multimodal corpora, aiming to solve the problems that existing corpus compression methods are not applicable to multimodal corpus scenarios and that existing storage methods easily lead to wasted storage space.

[0006] In a first aspect, embodiments of this application provide a method for storing multimodal corpora, including:

[0007] Compression processing was performed on the text modal data and audio / video modal data of the multimodal corpus to obtain text compression features and audio / video compression features respectively;

[0008] The similarity between the text compression features, audio and video compression features and the corresponding modal features of the newly added multimodal corpus is calculated, and the fusion compression process is determined based on the calculation results.

[0009] The corpus features that have not triggered the fusion and compression process are stored independently, while the corpus features that have triggered the fusion and compression process are fused into multimodal features to obtain multimodal joint features.

[0010] Based on access frequency and response time, the independently stored corpus features and the multimodal joint features are migrated to the corresponding data storage layer for storage, thereby completing the compression of the multimodal corpus.

[0011] Secondly, embodiments of this application provide a storage device for multimodal corpus, comprising:

[0012] The compression unit is used to perform compression processing on the text modal data and audio / video modal data of the multimodal corpus respectively to obtain text compression features and audio / video compression features;

[0013] The calculation unit is used to calculate the similarity between the text compression features, audio and video compression features and the corresponding modal features of the newly added multimodal corpus, and determine whether to trigger the fusion compression process based on the calculation results.

[0014] The storage or fusion unit is used to independently store the corpus features that have not triggered the fusion and compression process, and to perform multimodal feature fusion on the corpus features that have triggered the fusion and compression process to obtain multimodal joint features.

[0015] The migration unit is used to migrate the independently stored corpus features and the multimodal joint features to the corresponding data storage layer for storage based on access frequency and response time, so as to complete the compression of the multimodal corpus.

[0016] Thirdly, embodiments of this application provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the program to implement the compression method provided in embodiments of this application.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, implements the compression method provided in embodiments of this application.

[0018] In this embodiment, the text modal data and audio / video modal data of the multimodal corpus are compressed separately to adopt more suitable compression algorithms based on the characteristics of different modal data, thereby reducing the storage space occupied. By calculating the similarity between the text compression features, audio / video compression features and the corresponding modal features of the newly added multimodal corpus, the semantic redundancy between the newly added multimodal corpus and the historical compressed corpus can be accurately identified, triggering the fusion compression process to eliminate cross-modal redundancy and avoid the repeated storage of the same semantic information. Furthermore, the corpus features that have not triggered the fusion process are stored independently, while the corpus features that have triggered the fusion process are fused to generate multimodal joint features, further reducing the amount of stored data. By combining access frequency and response time, the corpus features are migrated to the corresponding data storage layer, reducing storage costs. Attached Figure Description

[0019] The technical solution and its beneficial effects will become apparent from the following detailed description of specific embodiments of this application, in conjunction with the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a method for storing multimodal corpora provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the structure of a multimodal corpus storage device provided in an embodiment of this application.

[0022] Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] It should be noted that the terms "first," "second," and "third," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.

[0024] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0025] This application provides a method, apparatus, computer device, and medium for storing multimodal corpora. The execution entity of the multimodal corpus storage method can be the multimodal corpus storage apparatus provided in this application, wherein the compression apparatus can be implemented in hardware or software.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for storing multimodal corpora provided in an embodiment of this application. The process may include:

[0027] In 101, compression processing is performed on the text modal data and audio / video modal data of the multimodal corpus to obtain text compression features and audio / video compression features respectively.

[0028] In this embodiment, when storing multimodal corpora, it is necessary to compress the multimodal corpora first to save storage space. However, since multimodal corpora contain different modal data (including text modal data and audio / video modal data), using the same corpus compression method can easily lead to semantic loss. Therefore, text modal data and audio / video modal data are compressed separately, and appropriate compression methods are selected so that the extracted compressed features can retain key information.

[0029] In practice, the multimodal corpus consists of user input. Upon receiving a request for user input, all user input is collected and then compressed.

[0030] In an optional embodiment, compression processing is performed on the text modal data of the multimodal corpus to obtain text compression features, including:

[0031] A dynamic word co-occurrence matrix is ​​constructed based on text modal data in a multimodal corpus.

[0032] The word vectors in the dynamic word co-occurrence matrix are subjected to dimensionality reduction processing, and a preset proportion of semantic information is retained to obtain low-dimensional text vectors;

[0033] The low-dimensional text vector is encoded using low-bit encoding to obtain text compression features.

[0034] This embodiment compresses text modal data from a multimodal corpus. The specific compression process is as follows: Preprocessing is performed on the text modal data in the multimodal corpus. This preprocessing includes word segmentation, stop word removal, and stemming. The Jieba word segmentation tool combined with a custom dictionary can be used to process the extracted words to form an initial word frequency matrix. Based on a sliding window (e.g., window size 5), the frequency of co-occurrence of each word pair in the initial word frequency matrix is ​​counted, and a dynamic word co-occurrence matrix is ​​constructed. The window size can be adaptively adjusted according to the text length. For example, short texts use a small window (e.g., window size 3) to capture local semantics, while long texts use a large window (e.g., window size 10) to capture global associations.

[0035] Then, all word vectors in the dynamic word co-occurrence matrix are subjected to dimensionality reduction, retaining a preset proportion (e.g., 92%) of semantic information. Dimensionality reduction can be achieved using Singular Value Decomposition (SVD), with a preset proportion of 92%. For example, if the original word vector dimension is 300, SVD can reduce it to 50 dimensions while retaining 92% of the semantic information, resulting in a low-dimensional text vector. The dimensionality-reduced low-dimensional text vector is then encoded using low-bit encoding. For example, the Delta-CoMe algorithm is used for low-bit encoding. Specifically, the Delta-CoMe algorithm allocates different bit widths according to the size of the singular values ​​corresponding to each low-dimensional text vector. If the singular value of the current low-dimensional text vector is greater than the first predetermined singular value (90%), the current low-dimensional text vector is divided into the first singular value vector, and high-precision (such as 8-bit or 16-bit) key semantic information is retained. If the singular value of the current low-dimensional text vector is greater than the second predetermined singular value (60%) and less than the first predetermined singular value (90%), the current low-dimensional text vector is divided into the second singular value vector, and 3-bit quantization is used to reduce storage space. If the singular value of the current low-dimensional text vector is less than the second predetermined singular value (60%), the current low-dimensional text vector is directly discarded.

[0036] This embodiment achieves efficient compression of text modal data through a three-level process of dynamic word co-occurrence matrix construction, dimensionality reduction, and low-bit encoding, while preserving key semantic information. It is suitable for multimodal corpus storage, transmission, and real-time processing scenarios.

[0037] In an optional embodiment, constructing a dynamic word co-occurrence matrix based on text modal data from a multimodal corpus includes:

[0038] Analyze the target semantic information in the text modal data of the multimodal corpus;

[0039] Based on the target semantic information, the co-occurrence frequency of at least two word vectors is counted;

[0040] The word vectors corresponding to words whose co-occurrence frequency exceeds a preset frequency are stored to complete the construction of the dynamic word co-occurrence matrix.

[0041] In this embodiment, BERT-wwm (full-word masking model) or TextRank algorithm can be used for deep analysis of text modal data to extract multiple semantic units and target semantic information in each semantic unit. The extracted semantic units undergo data preprocessing and normalization to ensure the consistency of word vectors for the same concept. For example, "computer," "PC," and "PC" are mapped to the same semantic ID to avoid additional statistical counting of synonymous word vectors due to differences in word usage in the dynamic word co-occurrence matrix. The co-occurrence frequency of word vectors (i.e., the frequency of at least two word vectors appearing together) is counted based on a sliding window, with the window size dynamically adjusted according to the text length and semantic density. If the co-occurrence frequency exceeds a preset frequency (e.g., 3 times), the corresponding word vector is stored, thus forming a dynamic word co-occurrence matrix.

[0042] In practice, the preset frequency can be changed according to specific application scenarios. Therefore, the word vectors in the dynamic word co-occurrence matrix are frequently updated. For example, in the field of government affairs, the preset frequency is associated with the policy revision cycle. If the preset frequency of the existing policy is 5, when a new policy is released and the release time is short, the preset frequency can be reduced from the preset frequency of the existing policy (down to 3 times) to ensure that time-sensitive policy terms are retained first.

[0043] In an optional embodiment, the audio-visual modal data includes video data and audio data. Compression processing is performed on the audio-visual modal data of the multimodal corpus to obtain audio-visual compression features, including:

[0044] The video data is parsed into multiple video frames, and inter-frame difference analysis is performed between adjacent video frames.

[0045] If the inter-frame difference between adjacent video frames is greater than the preset difference, then the corresponding video frame is obtained as the target video frame.

[0046] Feature extraction is performed on the target video frame to obtain a video hash fingerprint;

[0047] The audio data is divided into multiple phoneme units, and at least two phoneme units with the same semantics are merged to obtain phoneme unit segments;

[0048] Extract the MFCC features of the phoneme unit segment to obtain the audio binary fingerprint;

[0049] The video hash fingerprint is combined with the audio binary fingerprint to obtain the audio and video compression features.

[0050] In this embodiment, the audio and video modal data includes video data and audio data. The video data and audio data need to be processed separately. Specifically, the continuous video data is first parsed into discrete video frames according to the time sequence (e.g., 24 frames are extracted per second). Then, the difference is analyzed by means of pixel value comparison, edge feature matching, etc., and the inter-frame difference between adjacent frames is calculated (the higher the inter-frame difference, the more obvious the change in the picture content).

[0051] A preset difference degree is set (which can be adaptively adjusted according to the scene). If the difference degree between two adjacent frames is greater than the preset difference degree, both frames are marked as candidate video frames. All candidate video frames are obtained, and adaptive K-means++ clustering is used to select representative target video frames from these candidate video frames. Specifically, the candidate video frames are grouped by clustering, and the most typical frame in each group (each pair of adjacent frames is a group) is selected as the target video frame to avoid redundancy.

[0052] For the selected target video frames, features are extracted using a convolutional neural network (such as MobileNetV3) to generate corresponding video hash fingerprints. Alternatively, feature extraction can be performed using hash algorithms (such as dHash or pHash). This involves first standardizing each target video frame (uniform size, grayscale), and then generating a fixed-length binary string (i.e., the video hash fingerprint) through pixel comparison. The video hash fingerprint is typically 64 bits or 128 bits long, and the appropriate length can be generated depending on the application scenario. In this implementation, the generated video hash fingerprint is 128 bits long. This completes the processing of the video data.

[0053] The audio data is then segmented into multiple phoneme units using speech recognition technology. Semantic matching is performed on these phoneme units, merging consecutive phoneme units with the same or highly similar meanings to form compact phoneme unit segments. This removes semantic redundancy in the audio, avoids redundant phonemes occupying extra storage, and preserves core speech information. MFCC (Mel-frequency cepstral coefficients) features are extracted from the merged phoneme unit segments, and then these MFCC features are converted into fixed-length binary strings (i.e., audio binary fingerprints) through quantization encoding (e.g., binary quantization). This completes the processing of the audio data.

[0054] The video hash fingerprint (binary string) and the audio binary fingerprint (binary string) are combined and concatenated according to preset rules, such as concatenating the video fingerprint and audio fingerprint in sequence, or concatenating them through concatenation characters, to form a unified format of audio and video compression features.

[0055] In step 102, the similarity between the text compression features, audio and video compression features and the corresponding modal features of the newly added multimodal corpus is calculated, and the fusion compression process is determined based on the calculation results.

[0056] In this embodiment, before similarity calculation, the newly added multimodal corpus needs to undergo feature extraction, transforming it into corresponding modal features of the same dimension and type as the text compression features and audio / video compression features, to avoid similarity calculation deviations due to format mismatch. Then, similarity calculations are performed between the text compression features and text modal features, and between the audio / video compression features and audio / video modal features. The results of the similarity calculations determine whether to trigger the fusion compression process. If the fusion compression process is triggered, it indicates that the newly added multimodal corpus has cross-modal redundancy with the existing corpus, requiring subsequent multimodal feature fusion steps to fuse redundant features and eliminate duplicate storage. If the fusion compression process is not triggered, it indicates that the newly added multimodal corpus is non-redundant data, requiring subsequent independent storage steps to avoid losing new semantic information due to over-compression.

[0057] In an optional embodiment, the step of calculating the similarity between the text compression features, audio / video compression features, and the corresponding modal features of the newly added multimodal corpus, and determining whether to trigger the fusion compression process based on the calculation result, includes:

[0058] Semantic vectors of text data in the newly added multimodal corpus are extracted as text modal features, image vectors of image data in the newly added multimodal corpus are extracted as image modal features, and MFCC feature vectors of audio data in the newly added multimodal corpus are extracted as audio modal features.

[0059] The text compression feature and the text modal feature are subjected to cosine similarity calculation to obtain a text similarity value; the video hash fingerprint and the image modal feature are subjected to similarity calculation to obtain an image similarity value; and the audio binary fingerprint and the audio modal feature are subjected to similarity calculation to obtain an audio similarity value.

[0060] If the text similarity value, image similarity value, and audio similarity value are all greater than the preset similarity threshold, then the fusion compression process is triggered; otherwise, the fusion compression process is not triggered.

[0061] In this embodiment, feature extraction is first performed on the newly added multimodal corpus, which includes text data, image data, and audio data. For the text data in the newly added multimodal corpus, semantic vectors can be extracted using the BERT pre-trained language model to obtain text modal features. For the image data in the newly added multimodal corpus, image vectors are extracted using the ViT (Vision Transformer) model to obtain image modal features. For the audio data in the newly added multimodal corpus, MFCC feature vectors can also be extracted using the ViT (Vision Transformer) model to obtain audio modal features.

[0062] Then, similarity calculations were performed on text compression features and text modal features, video hash fingerprints and image modal features, and audio binary fingerprints and audio modal features, respectively. Specifically, for text compression features and text modal features, the cosine similarity formula can be used for similarity calculation; for video hash fingerprints and image modal features, the structural similarity index can be used to evaluate similarity, combined with a comprehensive score of brightness, contrast, and structure dimensions for similarity calculation. If faster calculation is required, cosine similarity or Euclidean distance can be used; for audio binary fingerprints and audio modal features, the difference between the two can be calculated based on Hamming distance to obtain their similarity. Thus, the text similarity value, image similarity value, and audio similarity value were obtained.

[0063] If the text similarity value, image similarity value, and audio similarity value are all greater than the preset similarity value (e.g., 0.9), it is determined to be high redundant data, and the fusion compression process is triggered at this time; otherwise, the fusion compression process is not triggered.

[0064] In step 103, corpus features that have not triggered the fusion and compression process are stored independently, while corpus features that have triggered the fusion and compression process are fused using multimodal features to obtain multimodal joint features.

[0065] In this embodiment, if any of the text similarity, image similarity, and audio similarity values ​​fails to reach a preset similarity, the fusion compression process is not triggered. The corpus features that do not trigger the fusion compression process (including text modality features, image modality features, and audio modality features) are then acquired and stored independently. In specific implementations, independent storage can employ a hierarchical storage architecture. For example, a base layer stores the original compressed features using columnar storage to optimize query efficiency. A metadata layer records the feature source (e.g., corpus ID, modality type, extraction time) and similarity calculation results (text / image / audio similarity values) to enable rapid retrieval and tracing. Furthermore, an inverted index can be constructed for text modality features, and an LSH index can be constructed for image / audio modality features to achieve rapid retrieval.

[0066] For the corpus features that trigger the fusion compression process, multimodal feature fusion is performed according to the preset fusion strategy to obtain multimodal joint features. The specific preset fusion strategy will be described in subsequent embodiments.

[0067] In an optional embodiment, performing multimodal feature fusion on the corpus features that trigger the fusion compression process to obtain multimodal joint features includes:

[0068] The text compression features and audio / video compression features corresponding to the trigger fusion compression process are selected, and the data formats of the text compression features, audio / video compression features and the corresponding modal features in the newly added multimodal corpus are unified to obtain a standardized feature set;

[0069] The standardized feature set is input into the trained multimodal fusion model to obtain multimodal joint features.

[0070] In this embodiment, based on the similarity calculation results, only corpus features (text compression features and audio / video compression features) whose text similarity, image similarity, and audio similarity all exceed the preset similarity are selected. The corpus features are then unified with the corresponding modal features in the newly added multimodal corpus in terms of data format. Specifically, the text compression features and text modal features do not need to be converted to floating-point values. For binary video hash fingerprints, audio binary fingerprints, and the newly added image and audio modal features, the binary values ​​(0 / 1) are converted to floating-point values ​​(0.0 / 1.0) to avoid subsequent model processing. The gradient vanishing problem in binary data is addressed; then, normalization is performed, compressing the numerical range of all 128-dimensional vectors (all previously obtained corpus features and the corresponding modal features extracted from the new corpus are 128-dimensional vectors) to the [0,1] interval. For example, if the floating-point value corresponding to "payment standard" in the newly added text modal feature is normalized from "1500.0" to "0.85" (based on the value range of government payment standard being 1000-2000), this ensures that the numerical magnitudes of different modal features are consistent, avoiding the model from overemphasizing features with large numerical values ​​(such as payment amount) while ignoring semantic features. The 128-dimensional normalized vectors after format unification are the standardized feature set.

[0071] Then, the standardized feature set is input into the trained multimodal fusion model (such as the VAE model (variational autoencoder)). Then, the vectors with the same semantics are fused (that is, the text compression features, audio compression features, text modal features, image modal features, and audio modal features with the same semantics are fused to obtain the multimodal joint features.

[0072] In the specific implementation process, the multimodal joint features obtained are stored through a joint feature library, and vector databases (such as Faiss and Milvus) are used to support approximate nearest neighbor (ANN) retrieval. An association mapping table is also used to record the mapping relationship between each multimodal joint feature and the original corpus (e.g., "joint feature ID → [text ID, image ID, audio ID]") to support feature backtracking.

[0073] In step 104, based on access frequency and response time, the independently stored corpus features and the multimodal joint features are migrated to the corresponding data storage layer for storage, so as to complete the compression of the multimodal corpus.

[0074] In this embodiment, a two-dimensional evaluation index system is first constructed (i.e., access frequency is a statistical index and response time is a performance index). The independently stored corpus features and multimodal joint features are classified and rated. The two types of features with high access frequency and response time requirements below a preset threshold (e.g., 50ms) are classified as high-frequency and high-sensitivity features. The two types of features with medium access frequency and lenient response time requirements (e.g., 100ms-300ms) are classified as medium-frequency and regular features. The two types of features with low access frequency and no strict response time limit are classified as low-frequency and archived features.

[0075] The data storage layer in this embodiment includes an in-memory database, a distributed file system, and a compatible system. The corresponding data storage layer can be matched for features with different ratings. For example, high-frequency and high-sensitivity features can be migrated to an in-memory database (such as Redis) to ensure millisecond-level access response; medium-frequency and regular features can be stored in a distributed file system (such as HDFS) to balance storage capacity and access efficiency; and low-frequency archive features can be migrated to low-cost object storage (such as S3 compatible storage) to reduce long-term storage costs.

[0076] This embodiment satisfies the access performance requirements of different characteristics and achieves optimized configuration of storage resources.

[0077] In an optional embodiment, the step of migrating the independently stored corpus features and the multimodal joint features to the corresponding data storage layer for storage based on access frequency and response time includes:

[0078] The access frequencies of the independently stored corpus features and the multimodal joint features are statistically analyzed within a preset time period, and the response time of the data storage layer receiving data access requests is obtained.

[0079] If any access frequency is greater than a preset access threshold and the response time is less than a preset time threshold, then the corresponding corpus features and multimodal joint features are migrated to the hot data storage layer for storage; otherwise, they are migrated to the cold data storage layer for storage.

[0080] In this embodiment, the data storage layer includes a hot data storage layer and a cold data storage layer. A distributed tracking system (such as Jaeger or Zipkin) records the query frequency of each corpus feature and multimodal feature, statistically analyzing access frequency by time window (e.g., hour / day), and setting preset access thresholds (e.g., daily access > 100 times for hot data, daily access ≤ 10 times for cold data). If the access frequency of a certain corpus feature or multimodal feature within a preset period (e.g., 1 day) exceeds the preset access threshold (e.g., 100 times), then these features are considered hot data; otherwise, they are considered cold data. At this time, a data access request is initiated to the hot data storage layer, and the response time of the hot data processing layer receiving the data access request is obtained. If the response time is less than a preset time threshold (e.g., 50ms), then the corresponding corpus feature and multimodal joint feature are migrated to the hot data storage layer and stored using HDFS+SSD technology. HDFS is a distributed file system that excels at managing large-scale distributed data. It can distribute hot data across multiple server nodes to avoid single points of failure and supports fast parallel reading. SSD is a solid-state drive with no mechanical read / write components. Its data read speed is 10-100 times faster than traditional hard drives, and it can quickly respond to data access requests. Its storage process involves distributing hot data to multiple SSD storage nodes through HDFS. Each node stores a portion of the feature data, and the HDFS "NameNode" records the storage location of the features.

[0081] Otherwise, the corresponding corpus features and multimodal joint features are migrated to the cold data storage layer and stored using Ceph+HDD technology. Ceph is a distributed storage system that supports multiple storage modes such as object, block, and file, suitable for storing cold data with low access frequency. Storage capacity can be flexibly increased by horizontally scaling server nodes. HDDs are mechanical hard drives, with storage costs only 1 / 5 to 1 / 10 of SSDs, but slower read speeds (suitable for scenarios that do not require fast response). The storage process involves storing cold data as "object storage" in the HDD nodes of the Ceph cluster. Ceph automatically shards the feature data and stores it on different HDD nodes, while using the CRUSH algorithm to ensure data reliability (e.g., automatically generating two backups to prevent data loss due to hard drive failure).

[0082] This embodiment adopts a hot and cold data tiering mechanism (hot data uses HDFS+SSD, cold data uses Ceph+HDD) to automatically migrate highly redundant data, significantly reducing storage costs and optimizing storage resource utilization.

[0083] Please see Figure 2 , Figure 2This is a schematic diagram of a multimodal corpus storage device provided in an embodiment of this application. The multimodal corpus storage device 200 includes: a compression unit 201, a computing unit 202, a storage or fusion unit 203, and a migration unit 204.

[0084] Compression unit 201 is used to perform compression processing on the text modal data and audio / video modal data of the multimodal corpus respectively to obtain text compression features and audio / video compression features;

[0085] The calculation unit 202 is used to calculate the similarity between the text compression features, audio and video compression features and the corresponding modal features of the newly added multimodal corpus, and determine whether to trigger the fusion compression process based on the calculation results.

[0086] The storage or fusion unit 203 is used to independently store the corpus features that have not triggered the fusion compression process, and to perform multimodal feature fusion on the corpus features that have triggered the fusion compression process to obtain multimodal joint features;

[0087] The migration unit 204 is used to migrate the independently stored corpus features and the multimodal joint features to the corresponding data storage layer for storage based on the access frequency and response time, so as to complete the compression of the multimodal corpus.

[0088] In an optional embodiment, the compression unit 201 can be used to: construct a dynamic word co-occurrence matrix based on text modal data in a multimodal corpus; perform dimensionality reduction processing on the word vectors in the dynamic word co-occurrence matrix and retain a preset proportion of semantic information to obtain low-dimensional text vectors; and perform low-bit encoding on the low-dimensional text vectors to obtain text compression features.

[0089] In an optional embodiment, the compression unit 201 can be used to: parse the target semantic information in the text modal data of the multimodal corpus; based on the target semantic information, count the co-occurrence frequency of at least two word vectors; and store the corresponding word vectors whose co-occurrence frequency exceeds a preset frequency to complete the construction of a dynamic word co-occurrence matrix.

[0090] In an optional embodiment, the compression unit 201 can be used to: parse the video data into multiple video frames and perform inter-frame difference analysis between adjacent video frames; if the inter-frame difference between adjacent video frames is greater than a preset difference, then obtain the corresponding video frame as the target video frame; extract features from the target video frame to obtain a video hash fingerprint; segment the audio data into multiple phoneme units and merge at least two semantically identical phoneme units to obtain phoneme unit segments; extract the MFCC features of the phoneme unit segments to obtain an audio binary fingerprint; and combine the video hash fingerprint with the audio binary fingerprint to obtain audio and video compression features.

[0091] In an optional embodiment, the computing unit 202 can be used to: extract semantic vectors of text data from the newly added multimodal corpus as text modal features, extract image vectors of image data from the newly added multimodal corpus as image modal features, and extract MFCC feature vectors of audio data from the newly added multimodal corpus as audio modal features; perform cosine similarity calculation on the text compression features and the text modal features to obtain a text similarity value, perform similarity calculation on the video hash fingerprint and the image modal features to obtain an image similarity value, and perform similarity calculation on the audio binary fingerprint and the audio modal features to obtain an audio similarity value; if the text similarity value, image similarity value, and audio similarity value are all greater than a preset similarity threshold, then the fusion compression process is triggered; otherwise, the fusion compression process is not triggered.

[0092] In an optional embodiment, the storage or fusion unit 203 can be used to: filter the text compression features and audio / video compression features corresponding to the triggering fusion compression process, and unify the data format of the text compression features, audio / video compression features and the corresponding modal features in the newly added multimodal corpus to obtain a standardized feature set; input the standardized feature set into the trained multimodal fusion model to obtain multimodal joint features.

[0093] In an optional embodiment, the migration unit 204 can be used to: count the access frequencies of the independently stored corpus features and the multimodal joint features within a preset time period, and obtain the response time of the data storage layer receiving the data access request; if any access frequency is greater than a preset access threshold and the response time is less than a preset time threshold, then the corresponding corpus features and multimodal joint features are migrated to the hot data storage layer for storage, otherwise they are migrated to the cold data storage layer for storage.

[0094] Please see Figure 3 This application provides a computer device 300, including a processor 320 and a memory, wherein the memory stores a computer program 332, and when the processor executes the program, it calls the computer program 332 stored in the memory to execute a multimodal corpus storage method provided in this embodiment.

[0095] In this embodiment, the computer device 300 includes a processor 320, a memory, and a network interface 350 connected via a system bus 310. The memory may include a non-volatile storage medium 330 and internal memory 340. The non-volatile storage medium 330 stores an operating system 331 and a computer program 332. The computer program 332 includes program instructions that, when executed, cause the processor 320 to execute a method for storing multimodal corpora. The processor 320 provides computational and control capabilities to support the operation of the entire computer device 300. The internal memory 340 provides an environment for the operation of the computer program 332 in the non-volatile storage medium 330. When the processor 320 executes the computer program 332, it causes the processor 320 to execute a method for storing multimodal corpora.

[0096] This network interface 350 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 300 to which the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0097] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program 332 instructing related hardware. The computer program includes program instructions, and the computer program 332 can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0098] Therefore, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform a method for storing multimodal corpora as provided in this embodiment.

[0099] In this embodiment, the processor 320 in the computer device loads the executable code corresponding to the processes of one or more applications into the memory according to the following instructions, and the processor 320 runs the applications stored in the memory, thereby achieving:

[0100] Compression processing was performed on the text modal data and audio / video modal data of the multimodal corpus to obtain text compression features and audio / video compression features respectively;

[0101] The similarity between the text compression features, audio and video compression features and the corresponding modal features of the newly added multimodal corpus is calculated, and the fusion compression process is determined based on the calculation results.

[0102] The corpus features that have not triggered the fusion and compression process are stored independently, while the corpus features that have triggered the fusion and compression process are fused into multimodal features to obtain multimodal joint features.

[0103] Based on access frequency and response time, the independently stored corpus features and the multimodal joint features are migrated to the corresponding data storage layer for storage, thereby completing the compression of the multimodal corpus.

[0104] In an optional embodiment, when the processor 320 performs compression processing on the text modal data of the multimodal corpus to obtain text compression features, it may perform the following: construct a dynamic word co-occurrence matrix based on the text modal data in the multimodal corpus; perform dimensionality reduction processing on the word vectors in the dynamic word co-occurrence matrix and retain a preset proportion of semantic information to obtain low-dimensional text vectors; and perform low-bit encoding on the low-dimensional text vectors to obtain text compression features.

[0105] In an optional embodiment, when the processor 320 executes the text modal data based on the multimodal corpus to construct a dynamic word co-occurrence matrix, it may perform the following: parsing the target semantic information in the text modal data of the multimodal corpus; based on the target semantic information, counting the co-occurrence frequency of at least two word vectors; and storing the corresponding word vectors whose co-occurrence frequency exceeds a preset frequency, so as to complete the construction of the dynamic word co-occurrence matrix.

[0106] In an optional embodiment, when the processor 320 executes the audio-video modal data, which includes video data and audio data, and performs compression processing on the audio-video modal data of the multimodal corpus to obtain audio-video compression features, it may perform the following: parsing the video data into multiple video frames and performing inter-frame difference analysis between adjacent video frames; if the inter-frame difference between adjacent video frames is greater than a preset difference, then the corresponding video frame is obtained as the target video frame; performing feature extraction on the target video frame to obtain a video hash fingerprint; segmenting the audio data into multiple phoneme units and merging at least two semantically identical phoneme units to obtain phoneme unit segments; extracting the MFCC features of the phoneme unit segments to obtain an audio binary fingerprint; and combining the video hash fingerprint with the audio binary fingerprint to obtain audio-video compression features.

[0107] In an optional embodiment, when the processor 320 performs the similarity calculation between the text compression features, audio / video compression features, and the corresponding modal features of the newly added multimodal corpus, and determines whether to trigger the fusion compression process based on the calculation results, it may perform the following: extracting the semantic vector of the text data in the newly added multimodal corpus as the text modal feature, extracting the image vector of the image data in the newly added multimodal corpus as the image modal feature, and extracting the MFCC feature vector of the audio data in the newly added multimodal corpus as the audio modal feature; performing cosine similarity calculation between the text compression features and the text modal features to obtain a text similarity value, performing similarity calculation between the video hash fingerprint and the image modal features to obtain an image similarity value, and performing similarity calculation between the audio binary fingerprint and the audio modal features to obtain an audio similarity value; if the text similarity value, image similarity value, and audio similarity value are all greater than a preset similarity threshold, then it is determined that the fusion compression process is triggered; otherwise, it is determined that the fusion compression process is not triggered.

[0108] In an optional embodiment, when the processor 320 performs multimodal feature fusion on the corpus features that trigger the fusion compression process to obtain multimodal joint features, it may perform the following: filtering the text compression features and audio / video compression features corresponding to the trigger fusion compression process, and unifying the data format of the text compression features, audio / video compression features, and corresponding modal features in the newly added multimodal corpus to obtain a standardized feature set; inputting the standardized feature set into the trained multimodal fusion model to obtain multimodal joint features.

[0109] In an optional embodiment, when the processor 320 performs the step of migrating the independently stored corpus features and the multimodal joint features to the corresponding data storage layer based on access frequency and response time, it may perform the following: separately calculate the access frequency of the independently stored corpus features and the multimodal joint features within a preset time period, and obtain the response time of the data storage layer receiving data access requests; if any access frequency is greater than a preset access threshold and the response time is less than a preset time threshold, then the corresponding corpus features and multimodal joint features are migrated to the hot data storage layer for storage; otherwise, they are migrated to the cold data storage layer for storage.

[0110] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of a multimodal corpus storage method above, which will not be repeated here.

[0111] The multimodal corpus storage device provided in this application embodiment belongs to the same concept as the multimodal corpus storage method in the above embodiment. Any method provided in the multimodal corpus storage method embodiment can be run on the multimodal corpus storage device. For details of its implementation process, please refer to the multimodal corpus storage method embodiment, which will not be repeated here.

[0112] It should be noted that, regarding the method for storing multimodal corpora according to the embodiments of this application, those skilled in the art will understand that all or part of the process of implementing the method for storing multimodal corpora according to the embodiments of this application can be accomplished by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor. During execution, it can include the process of an embodiment of the method for storing multimodal corpora. The computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), etc.

[0113] It is understood that in the specific implementation of this application, user information, such as application usage behavior data, logs and other related data, is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0114] For a multimodal corpus storage device according to an embodiment of this application, its functional modules can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer device, such as a processor or memory, or in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0115] The foregoing has provided a detailed description of a method, apparatus, computer device, and medium for storing multimodal corpora provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for storing multimodal corpora, characterized in that, include: Compression processing was performed on the text modal data and audio / video modal data of the multimodal corpus to obtain text compression features and audio / video compression features respectively; The similarity between the text compression features, audio and video compression features and the corresponding modal features of the newly added multimodal corpus is calculated, and the fusion compression process is determined based on the calculation results. The corpus features that have not triggered the fusion and compression process are stored independently, while the corpus features that have triggered the fusion and compression process are fused into multimodal features to obtain multimodal joint features. Based on access frequency and response time, the independently stored corpus features and the multimodal joint features are migrated to the corresponding data storage layer for storage, so as to complete the compression of the multimodal corpus; Compression processing is performed on the text modal data of the multimodal corpus to obtain text compression features, including: A dynamic word co-occurrence matrix is ​​constructed based on text modal data in a multimodal corpus. The word vectors in the dynamic word co-occurrence matrix are subjected to dimensionality reduction processing, and a preset proportion of semantic information is retained to obtain low-dimensional text vectors; The low-dimensional text vector is encoded using low-bit encoding to obtain text compression features; The audio and video modal data includes video data and audio data. Compression processing is performed on the audio and video modal data of the multimodal corpus to obtain audio and video compression features, including: The video data is parsed into multiple video frames, and inter-frame difference analysis is performed between adjacent video frames. If the inter-frame difference between adjacent video frames is greater than the preset difference, then the corresponding video frame is obtained as the target video frame. Feature extraction is performed on the target video frame to obtain a video hash fingerprint; The audio data is divided into multiple phoneme units, and at least two phoneme units with the same semantics are merged to obtain phoneme unit segments; Extract the MFCC features of the phoneme unit segment to obtain the audio binary fingerprint; The video hash fingerprint is combined with the audio binary fingerprint to obtain the audio and video compression features; The step of calculating the similarity between the text compression features, audio / video compression features, and the corresponding modal features of the newly added multimodal corpus, and determining whether to trigger the fusion compression process based on the calculation results, includes: Semantic vectors of text data in the newly added multimodal corpus are extracted as text modal features, image vectors of image data in the newly added multimodal corpus are extracted as image modal features, and MFCC feature vectors of audio data in the newly added multimodal corpus are extracted as audio modal features. The text compression feature and the text modal feature are subjected to cosine similarity calculation to obtain a text similarity value; the video hash fingerprint and the image modal feature are subjected to similarity calculation to obtain an image similarity value; and the audio binary fingerprint and the audio modal feature are subjected to similarity calculation to obtain an audio similarity value. If the text similarity value, image similarity value, and audio similarity value are all greater than the preset similarity threshold, then the fusion compression process is triggered; otherwise, the fusion compression process is not triggered.

2. The storage method according to claim 1, characterized in that, The text modal data based on multimodal corpus is used to construct a dynamic word co-occurrence matrix, including: Analyze the target semantic information in the text modal data of the multimodal corpus; Based on the target semantic information, the co-occurrence frequency of at least two word vectors is counted; The word vectors corresponding to words whose co-occurrence frequency exceeds a preset frequency are stored to complete the construction of the dynamic word co-occurrence matrix.

3. The storage method according to claim 1, characterized in that, The process of performing multimodal feature fusion on the corpus features that trigger the fusion and compression process to obtain multimodal joint features includes: The text compression features and audio / video compression features corresponding to the trigger fusion compression process are selected, and the data formats of the text compression features, audio / video compression features and the corresponding modal features in the newly added multimodal corpus are unified to obtain a standardized feature set; The standardized feature set is input into the trained multimodal fusion model to obtain multimodal joint features.

4. The storage method according to claim 1, characterized in that, The data storage layer includes a hot data storage layer and a cold data storage layer. The process of migrating the independently stored corpus features and the multimodal joint features to the corresponding data storage layer for storage, based on access frequency and response time, includes: The access frequencies of the independently stored corpus features and the multimodal joint features are statistically analyzed within a preset time period, and the response time of the data storage layer receiving data access requests is obtained. If any access frequency is greater than a preset access threshold and the response time is less than a preset time threshold, then the corresponding corpus features and multimodal joint features are migrated to the hot data storage layer for storage; otherwise, they are migrated to the cold data storage layer for storage.

5. A storage device for a multimodal corpus, characterized in that, include: The compression unit is used to perform compression processing on the text modal data and audio / video modal data of the multimodal corpus respectively to obtain text compression features and audio / video compression features; The calculation unit is used to calculate the similarity between the text compression features, audio and video compression features and the corresponding modal features of the newly added multimodal corpus, and determine whether to trigger the fusion compression process based on the calculation results. The storage or fusion unit is used to independently store the corpus features that have not triggered the fusion and compression process, and to perform multimodal feature fusion on the corpus features that have triggered the fusion and compression process to obtain multimodal joint features. The migration unit is used to migrate the independently stored corpus features and the multimodal joint features to the corresponding data storage layer for storage based on access frequency and response time, so as to complete the compression of the multimodal corpus. The audio and video modal data includes video data and audio data. Compression processing is performed on the audio and video modal data of the multimodal corpus to obtain audio and video compression features, including: The video data is parsed into multiple video frames, and inter-frame difference analysis is performed between adjacent video frames. If the inter-frame difference between adjacent video frames is greater than the preset difference, then the corresponding video frame is obtained as the target video frame. Feature extraction is performed on the target video frame to obtain a video hash fingerprint; The audio data is divided into multiple phoneme units, and at least two phoneme units with the same semantics are merged to obtain phoneme unit segments; Extract the MFCC features of the phoneme unit segment to obtain the audio binary fingerprint; The video hash fingerprint is combined with the audio binary fingerprint to obtain the audio and video compression features; The step of calculating the similarity between the text compression features, audio / video compression features, and the corresponding modal features of the newly added multimodal corpus, and determining whether to trigger the fusion compression process based on the calculation results, includes: Semantic vectors of text data in the newly added multimodal corpus are extracted as text modal features, image vectors of image data in the newly added multimodal corpus are extracted as image modal features, and MFCC feature vectors of audio data in the newly added multimodal corpus are extracted as audio modal features. The text compression feature and the text modal feature are subjected to cosine similarity calculation to obtain a text similarity value; the video hash fingerprint and the image modal feature are subjected to similarity calculation to obtain an image similarity value; and the audio binary fingerprint and the audio modal feature are subjected to similarity calculation to obtain an audio similarity value. If the text similarity value, image similarity value, and audio similarity value are all greater than the preset similarity threshold, then the fusion compression process is triggered; otherwise, the fusion compression process is not triggered.

6. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the storage method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the storage method as described in any one of claims 1 to 4.

Images