Multi-modal data compression method and system based on cross-modal dictionary collaborative generation and dynamic updating

Abstract

The invention discloses a multi-modal data compression method and system based on cross-modal dictionary collaborative generation and dynamic updating, and relates to the field of data transmission and storage. The method comprises the following steps: constructing a double-layer structure of a shared basic dictionary and each modal exclusive sub-dictionary by separating common characteristics and personality characteristics of multi-modal data, and associating the shared dictionary with the sub-dictionaries by constructing an index table; extracting single-mode features corresponding to each mode from the multi-mode data, mapping each single-mode feature to a unified space, obtaining cross-mode features, and training and sharing a basic dictionary and a sub-dictionary by using the features; monitoring reconstruction errors of the sub-dictionaries through a sliding window, triggering increment updating of the corresponding sub-dictionaries, obtaining increment matrixes, and aggregating the increment matrixes of the sub-dictionaries to generate a global dictionary; compression performance for multi-modal data is evaluated. Through the method, efficient, accurate, stable and low-cost multi-modal data compression is realized.

Description

Technical Field

[0001] This application relates to the field of data transmission and storage technology, and in particular to a multimodal data compression method and system based on cross-modal dictionary collaborative generation and dynamic updating. Background Technology

[0002] With the development of 5G and IoT technologies, multimodal data (such as image-text and audio-video) is being used on a large scale in various fields, and the surge in data volume has brought pressure on storage and transmission.

[0003] Existing technologies suffer from four major limitations: First, insufficient elimination of cross-modal redundancy; traditional single-modal compression methods (such as JPEG and MP3) cannot utilize intermodal semantic relationships, resulting in a joint compression rate 40%–60% lower than theoretical values. Second, static dictionaries struggle to adapt to dynamic data distributions; when the data distribution deviates from the training set by more than 30%, the compression rate drops by over 50%. Third, there is an imbalance between cross-modal consistency and compression efficiency; when the compression ratio exceeds 10:1, semantic matching accuracy drops below 60%. Fourth, poor edge-cloud collaboration adaptability; existing centralized training methods cannot meet the demands of low computing power on edge devices and efficient cloud integration, leading to high bandwidth overhead. Therefore, there is an urgent need for a compression method that can efficiently eliminate cross-modal redundancy, dynamically adapt to data changes, balance compression efficiency and modal consistency, and adapt to edge-cloud collaborative architectures. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimodal data compression method and system based on cross-modal dictionary collaborative generation and dynamic updating to address the above-mentioned technical problems, so as to achieve efficient, accurate, stable and low-cost multimodal data compression.

[0005] This application provides a multimodal data compression method and system based on cross-modal dictionary collaborative generation and dynamic updating. The method includes:

[0006] By separating the common and individual features of multimodal data, a two-layer structure of a shared basic dictionary and modality-specific sub-dictionaries is constructed, and an index table is built to link the shared dictionary and the sub-dictionaries;

[0007] Extract the unimodal features corresponding to each modality from the multimodal data, map each unimodal feature to a unified space to obtain cross-modal features, train a shared dictionary using the cross-modal features, and train a sub-dictionary for the corresponding modality using the unimodal features;

[0008] The reconstruction error of each sub-dictionary is monitored by a sliding window. When the reconstruction error meets the preset conditions, the corresponding sub-dictionary is incrementally updated to obtain the incremental matrix. The incremental matrices of each sub-dictionary are aggregated to generate a global dictionary.

[0009] A three-dimensional index system including single-modal reconstruction quality, cross-modal consistency, and dynamic update efficiency is constructed to evaluate the compression performance for multimodal data.

[0010] The aforementioned multimodal data compression method and system based on cross-modal dictionary collaborative generation and dynamic updating achieves efficient, accurate, stable and low-cost multimodal data compression. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a multimodal data compression method based on cross-modal dictionary collaborative generation and dynamic updating in one embodiment.

[0012] Figure 2 This is a schematic diagram of a cross-modal two-layer dictionary structure in one embodiment;

[0013] Figure 3 Here is a flowchart of a cross-modal dictionary collaborative generation mechanism in one embodiment;

[0014] Figure 4 This is a flowchart of a dictionary dynamic update strategy in one embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0016] This application provides a multimodal data compression method based on cross-modal dictionary collaborative generation and dynamic updating, such as... Figure 1 As shown, it includes the following steps:

[0017] S1. By separating the common features and individual features of multimodal data, a two-layer structure of a shared basic dictionary and a sub-dictionary for each modality is constructed, and an index table is built to link the shared dictionary and the sub-dictionaries.

[0018] Multimodal data refers to a data set containing two or more information carriers, such as mixed data composed of images, text, audio, and video, where each modality of data is related semantically or spatiotemporally.

[0019] A dictionary is a set of atoms generated through training. Dictionary learning is a feature learning method that represents raw data as a linear combination of dictionary atoms, expressed by the formula: ,in The original data, It is a dictionary matrix. It is a sparse coefficient vector.

[0020] This step separates the common and unique features of multimodal data, constructing a two-layer structure consisting of a shared basic dictionary and several modality-specific sub-dictionaries. This achieves layered compression by reducing redundancy in shared features and preserving details in modality-specific features. The shared basic dictionary captures cross-modal semantic consistency, while the modality-specific sub-dictionaries retain the unique attributes of each modality. The two are linked through an index table, connecting shared atoms and modality-specific atoms. Finally, the shared basic dictionary and the sub-dictionaries are initialized.

[0021] The specific steps are as follows:

[0022] S101, Define a shared basic dictionary The dimension is ,in: This is the shared feature dimension (e.g., a value of 512), used to store the vector dimension of cross-modal common features; To share the number of atoms (e.g., 1024), each atom corresponds to a cross-modal general feature (e.g., abstract concepts such as "movement" or "rest").

[0023] S102. Taking multimodal data consisting of at least two modalities from images, text, and audio as an example, design a sub-dictionary dimension specific to each modality. Details are as follows:

[0024] S1021. Define image sub-dictionary Dimensions are (e.g., 256×512), where For image feature dimensions, The number of atoms in an image is represented by the number of atoms in the image feature dimension. Each dimension in the image feature dimension is an atom type, representing a specific feature of the image. The atom types of an image include visual morphological features, target local features, and spatial relationship features.

[0025] S1022, Define Text Sub-Dictionary Dimensions are (e.g., 768×512), where For text feature dimensions, The number of text atoms represents the number of text features. Each dimension in the text feature dimension is an atom type, representing a specific feature of the text. The atom types of text include semantic unit features, syntactic structure features, and sentiment features.

[0026] S1023, Define the audio sub-dictionary Dimensions are (e.g., 40×512), where For audio feature dimensions, The number of audio atoms represents the number of atoms in the audio feature dimension. Each dimension in the audio feature dimension is an atom type, representing a specific feature of the audio. The atom types of audio include acoustic features, sound source features, and temporal features.

[0027] S103, For sharing a basic dictionary Each atom in the array is uniquely assigned. It iterates through single-modal data, filters sub-dictionary atoms related to the semantics of shared atoms, and constructs an index table to record the association between them and the modality-specific atoms of the sub-dictionary, forming cross-modal and cross-level associations. This enables rapid association and collaborative invocation between shared atoms and modality-specific atoms. The index table adopts a key-value pair structure with the following format: Where, association weight = semantic similarity (calculated based on contrast loss) × cross-modal co-occurrence probability (co-occurrence frequency / total number of samples), normalized to [0,1].

[0028] S104. Initialize the shared basic dictionary, setting the dimensions of the shared dictionary to be... Randomly generate the initial atomic matrix ,in for Random numbers in an interval are mapped to a linear transformation. The interval ensures that the magnitude of the initial atomic vector is within a controllable range.

[0029] S105. Pre-training optimization: Through supervised learning with cross-modal data (such as image-text pairs), the randomly initialized dictionary atoms are corrected, enabling them to initially capture common features across modalities, laying the foundation for subsequent dictionary collaborative generation mechanisms; cross-modal aligned training data pairs are used, such as... (No. Image-text pairs (No. (Video-audio pairs), each pair containing feature vectors of two modalities. ( (Representing modality type); Each modality feature is mapped to a shared space through a linear layer, i.e. ,in and For modality The mapping parameters, the feature dimension after mapping is Define a loss function that maximizes the cosine similarity of shared atoms and mapped cross-modal features: ,in For the first Cross-modal average characteristics of group data The formula for cosine similarity is... For the number of modes, The number of training samples is [number]; the Stochastic Gradient Descent (SGD) algorithm is used, and the learning rate is set to [value]. Iterative updates The atomic vectors are calculated up to the mean cosine similarity of all atoms with the cross-modal features. The iteration stops when the initial atoms have the ability to characterize common features across modes.

[0030] S106. Initialize each sub-dictionary. Each modality-specific sub-dictionary randomly samples features from the corresponding single-modality data as initial atoms. For different modality samples, the following operations are used to achieve multimodal feature fusion. After completing the following operations, the initial values ​​of the shared basic dictionary and modality-specific sub-dictionary ensure randomness (avoiding local optima) and also conform to the multimodal feature distribution through a data-driven approach, laying the foundation for subsequent collaborative training. Specifically:

[0031] S1061. For image data, image samples from the ImageNet dataset are randomly selected. (e.g., 512) images; for each image, the SIFT (Scale-invariant feature transform) algorithm is used to extract local features. First, a Gaussian pyramid is constructed to detect the extreme points in the scale space, then the orientation of key points is calculated (taking the peak value of the gradient orientation histogram), and finally a 128-dimensional descriptor vector is generated, with a 4×4 grid and 8 orientations; random sampling is performed from the extracted SIFT features. The feature vectors are used to form the initial matrix of the image sub-dictionary. If the SIFT feature dimension is insufficient, it is expanded by zero-padding or linear mapping. (e.g., 256×512).

[0032] S1062. Using text from the Wikipedia corpus, sentences containing high-frequency nouns, verbs, and adjectives are selected; using a GloVe pre-trained model (such as 300-dimensional word vectors), word vectors are extracted from the core vocabulary (such as entity words and attribute words) in the text; random sampling is performed from the word vector set. (e.g., 512) vectors, if the original dimension (300 dimensions) and (e.g., 768) mismatch, resolved through linear transformation Adjust dimensions, composition .

[0033] S1063. Using the ESC-50 audio dataset (containing 50 categories of ambient sounds), randomly select... (e.g., 512) audio segments; extract Mel-frequency cepstral coefficients (MFCC) features for each audio segment. First, divide the audio into frames (frame length 25ms, step size 10ms) and calculate the power spectrum. Then, convert it to a Mel spectrum using a Mel filter bank (40 filters). Finally, take the logarithmic energy and perform a DCT (Discrete Cosine Transform). Retain the first 13 dimensions as the basic features and supplement with first-order and second-order differences to form a 40-dimensional feature vector. The sampled 40-dimensional MFCC feature vectors are directly combined to form... (e.g., 40×512).

[0034] S2. Extract the single-modal features corresponding to each modality from the multimodal data, map each single-modal feature to a unified space, obtain cross-modal features, train a shared dictionary using the cross-modal features, and train a sub-dictionary for the corresponding modality using the single-modal features.

[0035] Cross-modal feature alignment involves mapping each single-modal feature to a unified space. This process transforms different modal features into a unified feature space using a mapping function, minimizing the distance between semantically related cross-modal features. The commonly used loss function is the contrastive loss. ,in For modal pair labels, 1 indicates correlation and 0 indicates no correlation. For feature distance, This is a marginal parameter.

[0036] This step, through joint learning of multimodal features, enables the shared base dictionary and modality-specific sub-dictionaries to complement each other in the feature space. This ensures semantic consistency across modal data while preserving detailed features of single-modal data. The core is optimizing the joint loss function to improve compression performance. Its working principle is as follows: First, a modality adaptive encoder is used to extract and transform features from images, text, and audio respectively. For example, images are dimensionality reduced after feature extraction using ResNet-50, text semantic features are extracted using BERT (Bidirectional Encoder Representation from Transformers), and audio temporal features are extracted using MFCC and LSTM (Long Short-Term Memory). Then, a mapping function maps each modality feature to a unified shared space, and contrastive loss is used for alignment training. Finally, the shared dictionary is trained with the aligned cross-modal features, and the sub-dictionaries are trained with single-modal features. The overall loss function is used for joint optimization, enabling the shared dictionary and sub-dictionaries to work collaboratively. The specific implementation steps are as follows:

[0037] S201. Single-modal feature extraction and transformation: Based on different modal features, the following steps are used respectively:

[0038] S2011, Image Feature Extraction and Transformation: Feature extraction is performed using the ResNet-50 model. The original image is input, and the ResNet-50 model is used for forward propagation. The last fully connected layer is removed, and the output of the second-to-last layer is used as the initial feature (2048 dimensions). The 2048-dimensional feature is then reduced to a smaller dimension using a fully connected layer. The formula is: ,in, This is the weight matrix of the fully connected layer. For bias vectors, ResNet-50 pairs of images The feature output ultimately yields the image features. .

[0039] S2012, Text Feature Extraction and Transformation: The BERT-base model is used for feature extraction. The input text sequence is processed by the BERT-base model, and the output vector (768 dimensions) at the [CLS] position is extracted as the text semantic feature. The formula is as follows: ,in, This indicates that the BERT model is effective for text processing. The output, This means taking the feature vector of the 0th bit (i.e., [CLS] token) to obtain the final text features. .

[0040] S2013. Audio Feature Extraction and Transformation: MFCC combined with LSTM is used for feature extraction. The original audio signal is processed in frames, with a frame length of 25ms and an inter-frame step size of 10ms. A 40-dimensional MFCC is calculated for each audio frame, including acoustic features such as fundamental frequency and spectral envelope. The MFCC features are then input into the LSTM network to extract temporal features, using the following formula: ,in, This represents the feature output of the LSTM network, ultimately yielding the audio features. (Output Dimension) ).

[0041] S202. Cross-modal feature alignment: This involves mapping features from different modalities to a unified shared feature space. Contrastive loss is used to optimize the mapping parameters, making semantically related cross-modal features closer together in the shared space. The specific steps are as follows:

[0042] S2021. Map the features of each modality to a unified shared feature space (dimension). Image feature mapping , in, For mapping weights, As a bias, output mapping features Text feature mapping ,in, For mapping weights, As a bias, output mapping features Audio feature mapping ,in, For mapping weights, As a bias, output mapping features .

[0043] S2022. The contrastive loss formula is used to optimize the mapping parameters, making semantically related cross-modal features closer together in the shared space. ,in, The modality pair is labeled (1 indicates semantic relevance, 0 indicates irrelevance). The Euclidean distance is the value of the mapped features. The marginal parameter is used; the training configuration uses the Adam optimizer, and the learning rate is set to [value missing]. The training runs consist of 50 rounds until the loss converges (the rate of decrease). ).

[0044] S203. Dictionary generation optimization: By training and optimizing a shared dictionary and modality-specific sub-dictionaries, it is ensured that the shared dictionary captures cross-modal commonalities, while the sub-dictionaries retain single-modal characteristics. Simultaneously, both are guaranteed to work collaboratively to balance cross-modal consistency and single-modal accuracy. In implementation, the shared dictionary is first trained using the KSVD (K-Singular Value Decomposition) algorithm with aligned cross-modal features, and the sub-dictionaries are trained separately using single-modal features. Then, the Adam optimizer is used iteratively to optimize the dictionary until convergence using the total loss function (including shared dictionary reconstruction loss, sub-dictionary reconstruction loss, and cross-modal alignment loss). The specific steps are as follows:

[0045] S2031. Shared dictionary training: Using aligned cross-modal features as input, the shared basic dictionary is updated through the KSVD algorithm. The objective function is ,in, For the aligned cross-modal features, It is a sparse coefficient vector. The L1 regularization parameter is optimized through alternating optimization. and The dictionary is updated 10 times.

[0046] S2032, Sub-dictionary Training: Using single-modal features as input, the KSVD algorithm is used to update the sub-dictionary specific to each modality. The specific method is as follows:

[0047] S20321, Image Sub-Dictionary Training, the input is the set of image features extracted by S2011. , For the number of training samples, each Beforehand, normalization is required to scale the feature vectors to a mean of 0 and a variance of 1. The KSVD algorithm implementation steps are as follows:

[0048] (1) Initialization: The initialization in S1061 is used. As an initial dictionary;

[0049] (2) Sparse coding: For each feature The sparse coefficients are solved using the OMP (Orthogonal Matching Pursuit) algorithm. ,satisfy: ,in (Upper limit of non-zero coefficients), iterate 5 times to obtain an approximate solution, and complete the sparse coding;

[0050] (3) Dictionary update: update the dictionary entry for the first entry. Atoms Collect all feature indices that use this atom Construct the residual matrix ,right Perform SVD decomposition ,renew Take the first left singular vector and update the corresponding coefficients. ;

[0051] (4) Iterative optimization: Repeat steps (2) to (3) for a total of 10 iterations, updating 10% of the atoms each time, and selecting the atoms to be updated according to the reconstruction error.

[0052] (5) Stopping condition: When the reconstruction error of two iterations reaches a certain threshold. Decline If necessary, training should be terminated early.

[0053] S20322, Text Sub-Dictionary Training takes the set of text features extracted by S2012 as input. Each Low-energy features need to be filtered out; L2 norm To reduce noise interference, the KSVD algorithm implementation steps are as follows:

[0054] (1) Initialization: The initialization in S1062 is used. As an initial dictionary;

[0055] (2) Sparse coding: FISTA (Fast Iterative Shrinking Threshold Algorithm) is used to solve for the sparse coefficients. Objective function: ,in After 20 iterations, the convergence condition is the gradient magnitude. ;

[0056] (3) Dictionary update: Similar to step (3) of the image sub-dictionary, for the 768-dimensional high-dimensional characteristics of text features, block SVD is used to accelerate the calculation. The residual matrix is ​​divided into blocks by column, and each block is 512×512 in size.

[0057] (4) Iterative optimization: A total of 10 iterations were performed, with 15% of the atoms updated each time. The text features have higher semantic diversity, requiring more atoms to be updated.

[0058] (5) Special processing: Perform semantic consistency verification on the updated atoms, calculate the mean cosine similarity between the atom and 100 random text features in the training set, and if the mean... Then discard the atom and add a new atom from the candidate pool, where the candidate pool consists of unselected text features.

[0059] S20323, Audio Sub-Dictionary Training, with the audio feature set extracted from S2013 as input. Each The KSVD algorithm grouped audio data by category (e.g., human voice, machine voice) and calculated sparsity coefficients for each group separately to enhance intra-class consistency. The implementation steps are as follows:

[0060] (1) Initialization: The initialization in S1063 is used. As an initial dictionary;

[0061] (2) Sparse coding: The CoSaMP (Compressed Sampling Matching Pursuit) algorithm is used to solve for the sparse coefficients. Supports grouping constraints: for the first group... Audio class, only the first one in the dictionary is allowed. Group atoms (pre-divided into 5 groups, corresponding to 5 audio categories), upper limit of non-zero coefficients. Iterate 4 times;

[0062] (3) Dictionary update: Similar to step (3) of the image sub-dictionary, in view of the temporal correlation of audio features, a weighted term of adjacent frame features is added when constructing the residual matrix, with a weight of 0.2, to enhance the adaptability of the dictionary to temporal changes;

[0063] (4) Iterative optimization: A total of 10 iterations were performed, and atoms were updated in groups according to category each time, with the same update ratio for each group;

[0064] (5) Post-processing: Post-processing of the trained data conduct Atomic clustering, Calculate the average similarity of atoms within a class. If the similarity of a certain class is... Then, retrain the atoms of that type to ensure consistency of characteristics among atoms of the same type.

[0065] S2033. Joint optimization: By jointly optimizing the shared dictionary and sub-dictionaries through the total loss function, cross-modal consistency and single-modal accuracy are ensured. The specific method is as follows:

[0066] S20331. Initialize joint optimization parameters:

[0067] (1) Define the components of the loss function: where the shared dictionary reconstruction loss is... Reconstruction error based on alignment features, i.e. , For the sample size, For alignment features, To share dictionary sparsity coefficients; sub-dictionary reconstruction loss Calculate the image sub-dictionary separately by modality. Text sub-dictionary audio sub-dictionary ,in , , These represent the sparsity coefficients for each mode and the cross-modal alignment loss. The contrastive loss trained in section 3.2 is used. ,Right now .

[0068] (2) Set the total loss function: The balance coefficient It can be adjusted according to the experiment, with priority given to ensuring reconstruction accuracy.

[0069] (3) Initialize the optimizer: Use the Adam optimizer and set the learning rate. Weight decay To prevent overfitting, momentum parameter , .

[0070] S20332, Iterative optimization of dictionary sparsity coefficients ( , These coefficients are used to represent the linear combination weights of the feature vectors on dictionary atoms:

[0071] (1) Solve for the sparse coefficients of the shared dictionary for each alignment feature. The solution is obtained using the LARS (Least Angle Regression) algorithm: ,in The iteration is performed 10 times, and the convergence condition is the coefficient update amount. .

[0072] (2) Solving for the sparsity coefficients of the sub-dictionary: Image sub-dictionary coefficients :right Solve using the OMP (Orthogonal Matching Pursuit) algorithm, with an upper limit for non-zero coefficients. Iteration 5 times; text sub-dictionary coefficients :right Solve using the FISTA algorithm. Iterate 20 times; audio sub-dictionary coefficients :right Solve using the CoSaMP algorithm, with an upper limit for non-zero coefficients. Iterate 4 times.

[0073] S20333, Iteratively update dictionary parameters, that is, directly adjust the feature vectors of atoms in the dictionary so that the dictionary can better adapt to the feature distribution reflected by the current sparse coefficients:

[0074] (1) Update the shared dictionary: based on the current and Update atoms using the KSVD algorithm: update the first atom in the dictionary. Atoms Collect sample indexes that use this atom Construct the residual matrix ;right Perform SVD decomposition. ,renew (The first left singular vector), and update the corresponding coefficients. .

[0075] (2) Update sub-dictionary: image sub-dictionary based on and The KSVD process is the same as step 3.2.1, updating 10% of the atoms each time; text sub-dictionary. based on and The KSVD process is the same as in step 3.2.2, but it uses block SVD (512×512 blocks) to speed up the update, updating 15% of the atoms each time; audio sub-dictionary. based on and The KSVD process is the same as in step 3.2.2, but with temporal weights added during updates, and adjacent frame features weighted at 0.2. Each update increments 10% of the atoms.

[0076] S20334. Determine convergence and termination of the iteration. Substitute the coefficients obtained in S20332 and the updated dictionary in S20333 to calculate the total loss value of the current iteration. If in 5 consecutive iterations... The decrease was less than ,Right now If convergence is achieved, the iteration is terminated; otherwise, return to S20332 and repeat the sparse coefficient solution and dictionary update until the maximum number of iterations (preset 200 rounds) is reached.

[0077] S20335. Output the optimization results. After the iteration terminates, save the final shared dictionary. Image sub-dictionary Text sub-dictionary Audio sub-dictionary The parameters for the corresponding sparse coefficient calculation algorithm are provided for use during the compression stage.

[0078] S3. Monitor the reconstruction error of each sub-dictionary through a sliding window. When the reconstruction error meets the preset conditions, trigger the incremental update of the corresponding sub-dictionary, obtain the incremental matrix, and aggregate the incremental matrices of each sub-dictionary to generate a global dictionary.

[0079] This step enables the dictionary to adapt to the dynamic changes in multimodal data distribution, ensuring stable compression ratio and cross-modal consistency while avoiding the high computational and transmission overhead of full retraining. A sliding window monitors data reconstruction errors in real time; when the error exceeds a threshold and persists for a certain period, the edge device is triggered to perform local fine-tuning of the local dictionary. Subsequently, the edge device uploads the updated parameters to the cloud, where a federated averaging algorithm aggregates the global dictionary. Finally, consistency checks ensure the stable performance of the global dictionary across all devices, forming a closed-loop dynamic adjustment mechanism of "monitoring – local update – global fusion." The detailed implementation steps are as follows:

[0080] S301, Sliding window incremental update triggering: By setting a sliding window to monitor the reconstruction error changes of multimodal data, when the error exceeds a preset threshold and continues for a certain period of time, the dictionary update process is triggered to achieve real-time response to changes in data distribution. The implementation method is as follows:

[0081] S3011. Select window type according to application scenario: Time window is used for time-series data such as video / audio. Each 60-second window is used to extract continuous data within that time period as the unit of analysis; non-time-series data such as text / images are analyzed using a data volume window. Each window contains 1000 data entries; the windows slide in a non-overlapping manner, and the next window starts immediately after the previous window ends. For time-series data such as video / audio, each window contains a sequence of video frames and corresponding synchronized audio segments. For non-time-series data such as text / images, the data is sorted by the acquisition time, and there is no overlap between windows.

[0082] S3012. Single-modal reconstruction error calculation: For multimodal data within each window, calculate the average reconstruction error for each modality: image / video reconstruction error. ,in The number of video frames within the window. For the first Image features of a frame For its image sub-dictionary Sparsity coefficients on the surface; text reconstruction error ,in For the first in the window Features of the text For its text sub-dictionary Sparse coefficients on; audio reconstruction error ,in The audio within the window is divided into frames, calculated at 25ms / frame. A 60s window contains 2400 frames. Let be the audio features of the i-th frame. For its audio sub-dictionary The sparsity coefficient on.

[0083] S3013. Update the trigger condition determination; the error threshold for each modality is determined based on the average reconstruction error during the model training phase. , , ,in , , These represent the average reconstruction errors of images, text, and audio on the training set, respectively; the trigger logic is: when the window error of a certain modality exceeds the corresponding threshold for several consecutive windows (e.g., 3 windows),... When three consecutive windows are met, it is determined that the distribution of the modality data has changed significantly, triggering the corresponding sub-dictionary (e.g., ...). The local update process is as follows: Multimodal errors are determined independently, and multiple sub-dictionaries can be updated simultaneously.

[0084] S3014. Post-trigger processing: Record the modal type, window number, and error value of the triggered update, and generate an update task sheet containing a dictionary of sub-dictionaries to be updated. Data feature path within the window and error exceeding the threshold (e.g.) This provides a basis for subsequent local fine-tuning.

[0085] S302. Local dictionary fine-tuning: For the modality that triggers the update, the edge device incrementally adjusts the local sub-dictionary based on the newly added data within the window, updating only atoms related to the new features to avoid the computational overhead caused by full reconstruction. The steps are as follows:

[0086] S3021. Candidate atom screening: Extract the newly added feature set that triggers the update mode within the window. ,in To add new samples within the window, calculate the cosine similarity between each new feature and existing atoms in the sub-dictionary. ,in To add new feature vectors, Atomic vectors in a sub-dictionary; filter by similarity Newly added features are used as candidate atoms, indicating that existing atoms cannot effectively represent the feature. The number of candidates does not exceed 10% of the total number of atoms in the current sub-dictionary. When containing 512 atoms, candidate atoms indivual.

[0087] S3022, Sub-dictionary incremental update, using the online KSVD algorithm to solve the incremental matrix. The objective function is ,in This is the sub-dictionary before the update. represents the sparsity coefficient of the candidate atoms. The L1 regularization parameter is used; for atom-by-atom updates, the optimal increment is obtained by decomposing the residual matrix using SVD for the selected candidate atoms, and the update logic in S203 is used to obtain... Sub-dictionary update: And normalize the unit modulus of the new atoms. This avoids the impact of feature scale deviations on subsequent compression.

[0088] S3023, Update Constraints and Records: Each local fine-tuning only updates up to 10% or less of the atoms in the sub-dictionary, prioritizing the replacement of the atoms with the lowest activity scores to maintain a stable dictionary size; record the incremental matrix of local updates. (k-th edge device) and the amount of data within the corresponding window This is used for subsequent cloud aggregation.

[0089] S303, global dictionary fusion: The cloud aggregates local update parameters from various edge devices, generates a global dictionary, and distributes it to edge nodes to ensure cross-device dictionary consistency and adapt to distributed scenarios. The steps are as follows:

[0090] S3031, Parameter Upload and Verification: Edge devices will perform partial updates. and corresponding data volume Encrypted uploads are sent to the cloud, and the uploaded content only includes parameter increments (such as a 512×51 dimensional matrix) to avoid transmitting the original data; the cloud verifies the parameter format: [Check] Check if the dimension matches the sub-dictionary; invalid parameters will be returned for retransmission.

[0091] S3032, Federated Aggregation Computation: The cloud-based global increment is calculated based on the FedAvg algorithm. ,in The number of edge devices participating in the update. The total data volume is represented by weights that are proportional to the device data volume; update the global dictionary: ,in This is the global dictionary before the update.

[0092] Among them, the federated averaging algorithm is a distributed model training method that updates the global model by aggregating the local model parameters of each node. The formula is as follows: ,in These are global parameters. For the first Node data volume , For the first Local parameters of the node; if the edge device is offline, its parameters are temporarily stored in the cloud and aggregated the next time it comes online. If the data upload fails, it will be retransmitted 3 times. If it still fails, the historical parameters will be used for estimation.

[0093] S3033, Consistency Verification and Distribution, the cloud will... The dictionary is distributed to each edge device, and after loading the new dictionary, the device calculates the reconstruction error of the local data. , For each modal type, calculate the variance of the errors across all devices: ,in The average error; if If the global dictionary consistency is satisfied, the update is completed; otherwise... The aggregation will be retried, with a maximum of 3 retries. If the result is still unsatisfactory, weighted aggregation will be used, giving higher weights to devices with low error rates.

[0094] S3034, Version Management: The version number of the global dictionary is recorded in the cloud. , To ensure consistent dictionary versions across nodes and facilitate problem tracing, edge devices synchronize version information for update frequency and time.

[0095] S4. Enhance the features of the single-modal features; obtain the cross-modal association weights through the attention mechanism and update the cross-modal features; update the atoms in the sub-dictionary and the shared basic dictionary based on the atom activity score.

[0096] This step addresses potential issues in multimodal feature acquisition and transmission, such as noise interference, weak cross-modal semantic associations, and dictionary atomic redundancy. It specifically enhances each modal feature through adaptive image contrast adjustment, text synonym replacement, and audio noise reduction to improve feature robustness. Cross-modal feature association weights are calculated and features are updated using an attention mechanism. Low-contribution atoms are eliminated based on atomic activity scoring to strengthen semantic associations. Atomic lifecycle management controls dictionary size, improving feature representation capabilities and dictionary utilization efficiency, achieving a balance between compression efficiency and storage cost, and ensuring the quality and consistency of the compressed data. The implementation method is as follows:

[0097] S401. Intramodal feature enhancement: Improves feature robustness while preserving key information using modality-specific methods; for image enhancement, adaptive contrast adjustment is employed, with the formula as follows: ,in, These are the original pixel values. The average value of the 3×3 neighborhood pixels. This is the gain coefficient. The function limits pixel values ​​to the range [0, 255] to avoid overexposure or underexposure. For text enhancement, non-core words such as adverbs and prepositions are replaced with a 30% probability based on the WordNet thesaurus. Core words such as noun entities are generated by BERT mask prediction, using the following formula: ,in For the vocabulary list, The text sequence is masked for the core words; for audio enhancement, spectral subtraction is used for noise reduction, with the formula as follows: ,in, The power spectrum includes the noise frequency. The noise power spectrum estimated for the first 100ms of the silent segment. For frequency, For time frames.

[0098] S402, Cross-modal attention alignment: By calculating the association weights between features of different modalities, the interaction of semantically related features is strengthened, achieving deep alignment of cross-modal features. For all pairwise combinations of images (I), text (T), and audio (A), all modal adaptation matrices are used. , , All features are optimized through training, initialized with a random normal distribution, and jointly updated with the mapping parameters from the cross-modal feature alignment stage. After the attention weights are calculated, the updated features need to be L2 normalized. To avoid the impact of feature scale differences on subsequent dictionary learning, the specific processing method is as follows:

[0099] S4021, Image-Text Attention Weight Calculation, applicable to scenarios where the semantic connection between visual content and text description needs to be strengthened:

[0100] S40211, Feature Input: Image Features , For image spatial dimensions, The feature dimension is flattened into a spatial feature sequence. ,common Spatial features; text features , The length of the text sequence. The sequence features are constructed by taking [CLS] and word vectors.

[0101] S40212, Attention Matrix Definition: Constructing the Image-Text Attention Matrix ,in , indicating the image number Spatial features and text Association weights of word features.

[0102] S40213, Weight Calculation Formula: ,in, This is the modality adaptation matrix, used to unify feature dimensions. The inner product result is normalized by softmax to obtain the weights.

[0103] S40214, Feature Weighted Update: Integrating Image Features into Text Semantics ,in (For mapping matrix); text features are integrated with visual information: ,in It is a mapping matrix.

[0104] S4022, Image-Audio Attention Weight Calculation, applicable to scenarios requiring alignment of visual and audio events:

[0105] S40221, Feature Input: Image features are the same as in S40211. , Audio characteristics: , For audio timing frames, The temporal feature sequence is extracted using LSTM.

[0106] S40222, Definition of Attention Matrix, Construction of Image-Audio Attention Matrix ,in Indicates the image number 1 Spatial features and audio Association weights of frame features.

[0107] S40223, Weight Calculation Formula: ,in, This is the modal adaptation matrix.

[0108] S40224, Feature-weighted update: Image features integrated into audio timing. , Audio features integrated into visual space , .

[0109] S4023, Text-Audio Attention Weight Calculation, applicable when text description and audio content need to be associated:

[0110] S40231, Feature Input: Text features are the same as in S4021. , Text length; Audio features: Same as S4022 , This represents the number of audio frames.

[0111] S40232, Attention Matrix Definition: Constructing a Text-Audio Attention Matrix ,in Indicates the text number Word features and audio Association weights of frame features.

[0112] S40233, Weight Calculation Formula: ,in, This is the modal adaptation matrix.

[0113] S40234, Feature-weighted update: Incorporating text features into audio temporal sequences: , Audio features integrated into text semantics: , .

[0114] S403, Dictionary Atomic Management, maintains dictionary efficiency through activity scoring and sparsity constraints, the steps are as follows:

[0115] S4031, Activity Score: Calculate a score for each atom 'a'. ,in, For atomic call frequencies, normalized to [0,1], To average the reconstruction error, normalize to [0,1]. , As weight.

[0116] S4032. Atomic Elimination and Replenishment: Periodic Screening Atom deletion is performed by adding the same number of new atoms from the candidate pool, i.e., the feature vectors that were not selected, to keep the dictionary size stable.

[0117] S4033, Sparse Coding Solution: The LARS algorithm is used to solve for the sparse coefficients. The iterative formula is as follows: ,in, For learning rate, For smoothing terms, The gradient of the loss function is iterated to... convergence.

[0118] S5. Construct a three-dimensional index system including single-modal reconstruction quality, cross-modal consistency, and dynamic update efficiency to evaluate the compression performance for multimodal data.

[0119] This step objectively evaluates the reconstruction accuracy, cross-modal semantic matching degree, and effectiveness of dictionary dynamic updates of the compressed modal data through a three-dimensional indicator system: single-modal reconstruction quality, cross-modal consistency, and dynamic update efficiency. This provides a quantitative basis for parameter optimization and verifies the applicability of the method in different scenarios. The specific implementation steps are as follows:

[0120] S501, Single-Modal Reconstruction Quality Assessment, is the core foundation of multi-dimensional compression performance assessment. It aims to accurately measure the degree of retention of key information in each modality's data during the compression-reconstruction process through modality-specific quantitative indicators and experimental procedures. The assessment process revolves around four steps: data screening, preprocessing, indicator calculation, and result analysis. For the three core modalities of image / video, text, and audio, the specific methods are as follows:

[0121] S5011, Image / Video Modal Reconstruction Quality Assessment: As a carrier of visual information, the reconstruction quality of images / videos needs to focus on pixel-level fidelity and structural similarity to avoid loss of detail, blurring or distortion caused by compression. The core evaluation indicators are PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index).

[0122] S50111. Data screening: stratified sampling from general datasets and special scene datasets to ensure coverage of different resolutions, scene types and image formats; the sample size must meet statistical significance, with ≥1000 single-resolution image samples and ≥100 video samples.

[0123] S50112. Preprocessing: Images are uniformly cropped into squares, and invalid edge areas are removed. Grayscale images are standardized to 8-bit pixel depth with values ​​from 0 to 255. RGB images are processed independently for three channels. Keyframes are extracted from the video, with one frame taken every five frames to ensure continuous motion between frames. The original bitrate of the video is recorded synchronously as a benchmark for compression ratio comparison.

[0124] S50113. Index Calculation: Peak Signal-to-Noise Ratio (PSNR) is used to measure pixel-level error. It reflects the degree of pixel value deviation by calculating the mean square error (MSE) between the original and reconstructed images. A higher PSNR value (in dB) indicates better reconstruction quality. The formula is: ,in, The maximum pixel value is [value]. , Pixel bit depth, mean square error , among which, among which For image height, For width, C is the number of channels, where C=3 for RGB images and C=1 for grayscale images.

[0125] S50114. Result Analysis: Structural similarity index (SSIM) is used to measure structural consistency. Image similarity is evaluated from three dimensions: brightness, contrast, and structure, overcoming the limitation of PSNR which only focuses on pixel errors. The SSIM value ranges from 0 to 1; the closer to 1, the more consistent the reconstructed image is with the original image structure. The SSIM calculation formula is: ,in, The local mean of the original image and the reconstructed image can be calculated using an 11×11 Gaussian window with a standard deviation of 1.5. Represents the local variance between the original image and the reconstructed image; Represents the local covariance between the original image and the reconstructed image; The value is a constant to avoid a denominator of 0, balancing the weights of brightness and contrast; suitable for general scenarios. High-precision scenes At that time, the results met the standards.

[0126] S50115. Video modality can additionally evaluate inter-frame consistency. Video reconstruction requires extra attention to inter-frame motion continuity to avoid problems such as "frame skipping" and "stuttering." The core metric is the Inter-frame Mean Square Error (MSE), and the formula is: ,in The number of keyframes in the video. For the first Frame-reconstructed image; This indicates that the pixel changes between frames are gradual, with no obvious jumps.

[0127] S5012. Text Modal Reconstruction Quality Assessment: As a carrier of semantic information, the core of reconstruction quality is semantic consistency and grammatical integrity. It is necessary to avoid keyword loss, semantic deviation, or grammatical errors caused by compression. The assessment revolves around "word vector matching" and "text generation quality". The core assessment indicators are word vector cosine similarity and bilingual evaluation understudy (BLEU) value.

[0128] S50121. Data screening: Samples are drawn from Wikipedia corpus, news datasets, and professional domain text libraries, covering different text lengths (5-100 characters), types, and language styles, with a sample size of ≥5000 texts. Core semantic keywords are annotated for each text.

[0129] S50122. Preprocessing: unify text encoding to UTF-8 and remove special characters; for professional texts, such as medical and legal texts, retain domain terminology to avoid accidentally deleting key information during preprocessing; record the word count and character count of the original text as a reference benchmark for semantic integrity.

[0130] S50123. Index Calculation: The semantic matching degree is measured using word vector cosine similarity. The original text and reconstructed text are converted into vectors using a pre-trained word vector model (such as Word2Vec or GloVe). The cosine similarity between the vectors is calculated to reflect the degree of semantic association. The formula is: ,in The average word vectors of the original text and the reconstructed text; It is the dot product of vectors; Let L2 be the norm of the vector.

[0131] S50124. Results Analysis, General Text Professional text The results met the standards.

[0132] S50125. The quality of text generation is measured by the BLEU score, and the n-grammar is calculated by comparing the reconstructed text (candidate text) with the original text (reference text). BLEU score, used to assess grammatical completeness and lexical accuracy, ranges from 0 to 1, with values ​​closer to 1 indicating better generation quality. The formula is: ,in for Precision rate, which is the percentage of candidate texts that match the reference text. Quantity / candidate text Total quantity ; Weights, default ; The brevity penalty factor is brevitypenalty, which applies if the candidate text is shorter than the reference text. This avoids misjudging "high overlap of short texts".

[0133] S5013. Audio Modal Reconstruction Quality Assessment: As an acoustic information carrier, audio reconstruction quality needs to focus on signal-to-noise ratio (SNR) and subjective auditory experience, avoiding noise introduction, frequency distortion, or timing discrepancies caused by compression. The assessment revolves around "objective acoustic indicators" and "subjective auditory tests," with the core indicators being signal-to-noise ratio (SNR) and PESQ (Perceptual Evaluation of Speech Quality).

[0134] S50131. Data Filtering: Samples are drawn from the ESC-50 ambient sound, TIMIT speech, and MUSICNET music datasets, covering different audio types: human voices: male / female / children's voices; ambient sounds: rain / car / machinery sounds; music: classical / pop, sampling rates 16kHz and 44.1kHz and durations 1-60s, with a sample size of ≥1000 segments. Each audio segment is labeled with acoustic features, such as fundamental frequency range and spectral peaks.

[0135] S50132. Preprocessing: The unified sampling rate is 16kHz mono, and the quantization accuracy is 16 bits; effective segments are extracted from speech audio, and the complete timing sequence of ambient sound is preserved; the power spectrum and fundamental frequency curve of the original audio are recorded as a benchmark for comparing the frequency distortion after reconstruction.

[0136] S50133. Index Calculation: The signal-to-noise ratio (SNR) (unit: dB) measures the noise suppression effect. It is calculated by dividing the original audio signal power by the reconstruction error (noise) power. The higher the SNR value, the less noise in the reconstructed audio. The formula is: ,in, The original audio signal power is given by the formula: , For the original audio at time The sampled values, This represents the total number of samples. To reconstruct the error power, the formula is: , To reconstruct audio at time The sampled values.

[0137] S50134. Result Analysis: Criteria for Judging Whether the Reconstruction Quality Meets Requirements: Environmental Sound ;voice: ;music: .

[0138] S50135. Subjective speech quality is measured using PESQ, with the following formula: comprehensibility requirements Naturalness requirement .

[0139] S50136. To measure audio event synchronization error, a timing consistency assessment is adopted. For audio synchronized with video, the "audio event timestamp deviation" is calculated, and the times of key events in the original audio are marked. Reconstruct the time of the corresponding event in the audio. Error formula: ,Require .

[0140] S502. Cross-modal consistency assessment, the steps are as follows:

[0141] S5021. Semantic matching accuracy: Semantic matching accuracy is used to quantify the degree of semantic association between different modalities of data after compression and reconstruction. The accuracy calculation formula is as follows: For example, modality For images, modal When it is text, .

[0142] S5022, Timing Synchronization Error: Calculate the mean time difference between events for video-audio or text-audio data. ,in, For audio event time, The duration of the video / text event is in milliseconds (ms).

[0143] S503. Evaluation of the effect of dynamic updates, the steps are as follows:

[0144] S5031, Compression Ratio Change Rate: ,in, Normal fluctuations should .

[0145] S5032, Update Time: Records the total time from triggering the update to the completion of global fusion. ,in, For edge-to-cloud parameter transmission time, To reduce the computation time for local fine-tuning and global aggregation, the following requirements are made. .

[0146] In one embodiment, an application example of the method of the present invention is given for a smart security scenario, which processes multimodal data including images, text, and audio.

[0147] The first step was to collect multimodal data. Image data consisted of keyframes from surveillance videos (1920×1080 resolution, 8-bit RGB images, 1000 samples); text data consisted of event description text (e.g., "2024-05-20 14:30 Someone broke into restricted area A", 10-30 characters long, 1000 samples); and audio data consisted of synchronous environmental audio (16kHz sampling rate, 16-bit mono, 60s / segment, 1000 segments). The core parameters were set as follows: cooperative loss balance coefficient γ=0.1, sliding window T=60s, federated average node count K=5, and sparsity regularization λ=0.01.

[0148] The second step is to construct a cross-modal two-level dictionary (Step 1-Step 6).

[0149] Step 1: Design of Shared Basic Dictionary Dimensions

[0150] Dimension definition: ,in (Shared feature dimension) (Shared number of atoms).

[0151] Step 2: Modality-Specific Sub-Dictionary Dimension Design

[0152] Image sub-dictionary: , , (Atom types: visual morphology, target locality, spatial relationship);

[0153] Text sub-dictionary: , , (Atomic types: semantic units, syntactic structures, sentiment tendencies);

[0154] Audio sub-dictionary: , , (Atom types: acoustics, sound source, timing characteristics).

[0155] Step 3: Index Table Construction

[0156] Association weight formula: (Normalized to [0,1]);

[0157] Example entry: Shared Atom ID:1 → [(Image, Atom ID:32, 0.85), (Text, Atom ID:105,0.78), (Audio, Atom ID:210, 0.62)];

[0158] Calculation process: Semantic similarity (image-share) = 0.9, co-occurrence probability = 0.94 → 0.9×0.94≈0.85.

[0159] Step 4: Initialize the shared dictionary

[0160] formula: ( );

[0161] Calculation example: Take ,but ;

[0162] Initialization result: All atomic vectors are distributed in The interval has a modulus of 1.0 (after normalization).

[0163] Step 5: Pre-training optimization

[0164] Training data: 1000 image-text pairs, 1000 video-audio pairs ( );

[0165] Mapping formula: ( For mapping weights, (for bias);

[0166] Loss function:

[0167] Calculation example: Cross-modal average characteristics of the first set of data (512 dimensions) cosine similarity ;

[0168] Training terminates: After 100 iterations, the mean cosine similarity of all atoms is 0.72 ≥ 0.7, and training stops.

[0169] Step 6: Sub-dictionary initialization

[0170] Image sub-dictionary: 512 images sampled from the ImageNet dataset, 128-dimensional features extracted using SIFT, linearly mapped to 256 dimensions, example atoms. (256 dimensions);

[0171] Text sub-dictionary: 300-dimensional GloVe word vectors from the Wikipedia corpus, linearly transformed.

[0172] ( Example Atoms (768 dimensions);

[0173] Audio sub-dictionary: 512 audio segments were sampled from the ESC-50 dataset, and 40-dimensional features were extracted using MFCC, directly forming the sub-dictionary. .

[0174] The third step is the cross-modal dictionary collaborative generation mechanism module (Step 1-Step 3).

[0175] Step 1: Multimodal Feature Extraction and Transformation

[0176] 1.1 Image Feature Extraction

[0177] formula: ( , );

[0178] Calculation example: Output 2048-dimensional features , , → (This is a single-dimensional example; the actual calculations for all 256 dimensions are similar.)

[0179] result: The mean is 0 and the variance is 1 (after normalization).

[0180] 1.2 Text Feature Extraction

[0181] formula: ;

[0182] Result: BERT outputs a 768-dimensional [CLS] vector. Example , .

[0183] 1.3 Audio Feature Extraction

[0184] formula: ;

[0185] Calculation: Audio frame segmentation (25ms / frame, 10ms step), MFCC extracts 40-dimensional features, LSTM outputs temporal features. ;

[0186] result: .

[0187] Step 2: Cross-modal feature alignment

[0188] 2.1 Feature Mapping

[0189] Image mapping: , ( ), example ;

[0190] Text mapping: , ( ), example ;

[0191] Audio mapping: , ( ), example .

[0192] 2.2 Optimization of Contrast Loss

[0193] formula: , ;

[0194] Calculation example: semantically related pairs ( ), (Euclidean distance); irrelevant pairs ( ), → ;

[0195] Training results: After 50 iterations, the Adam optimizer converged to the desired loss. .

[0196] Step 3: Dictionary generation optimization

[0197] 3.1 Shared Dictionary Training

[0198] Objective function: , ( );

[0199] Calculation example: Alignment feature KSVD iterations 10 times, sparse coefficients There are 10 non-zero terms, and the reconstruction error is... ;

[0200] result: After the update, the average reconstruction error was reduced to 0.025.

[0201] 3.2 Sub-dictionary training (taking image sub-dictionary as an example)

[0202] Sparse coding: Solving with the OMP algorithm , (Upper limit of non-zero coefficients), iterate 5 times → (Only 10 non-zero values);

[0203] Dictionary update: residual matrix SVD decomposition ( ),renew ;

[0204] Termination condition: After 10 iterations, the reconstruction error decreases from 0.05 to... Training has stopped.

[0205] 3.3 Joint Optimization

[0206] Total loss function: , ( );

[0207] Calculation example: , , , , → ;

[0208] Convergence result: After 150 iterations, the loss decreased significantly over 5 consecutive iterations. Output the final dictionary.

[0209] Step 4: Dictionary dynamic update strategy module (Step 1-Step 3).

[0210] Step 1: Triggering incremental update via sliding window

[0211] Window type: Time series data (audio / video) Each window contains 2400 frames of audio;

[0212] Reconstruction error calculation (image): , (60s video keyframes), calculated as follows ;

[0213] Threshold setting: Average error of training set , ;

[0214] Triggering condition: 3 consecutive windows This triggers an update to the image sub-dictionary.

[0215] Step 2: Local dictionary fine-tuning

[0216] Candidate Atom Screening: Adding Image Features and Cosine Similarity to Existing Atoms 51 candidate atoms were screened (512 × 10%).

[0217] Incremental update formula: ;

[0218] Calculation example: , After the update ;

[0219] Constraint: Only 10% of atoms are updated, replacing the 51 atoms with the lowest activity.

[0220] Step 3: Global dictionary fusion

[0221] Parameter upload: 5 edge devices upload and the data volume (total );

[0222] Federated aggregation formula: ,

[0223] Calculation example: , , , , → ;

[0224] Consistency check: The error variance of each device , and the global dictionary is sent down .

[0225] The fifth step, the multi-modal feature processing and management module (Step1-Step3).

[0226] Step1: Intra-modal feature enhancement

[0227] 1.1 Image enhancement

[0228] Formula: , ([[]]ID=49]] , is the 3×3 neighborhood mean);

[0229] Calculation example: , → ,

[0230] clip(0,255) → result 130;

[0231] Effect: The contrast is increased by 15% and there is no overexposure.

[0232] 1.2 Text enhancement

[0233] The synonym replacement rate is 30%: "break into" is replaced by "break into / enter" (probability 0.3), and the core word "Forbidden Area A" is retained;

[0234] Result: The text semantic similarity is maintained at 0.92≥0.85.

[0235] 1.3 Audio enhancement

[0236] Formula: ;

[0237] Calculation example: , , ; ​

[0238] Effect: Noise power reduced by 80%.

[0239] Step 2: Cross-modal attention alignment (image-text example)

[0240] Attention weight formula: ;

[0241] Calculation example: Sum of denominators = 100 → ;

[0242] Feature update: ,

[0243] Original characteristics: ;

[0244] Updated features:

[0245] .

[0246] Normalization formula ;

[0247] Calculation process: ;

[0248] Final result: ,and .

[0249] Step 3: Dictionary Atomic Management

[0250] Activity score formula: ,

[0251] ;

[0252] Calculation example: Atomic call frequency Reconstruction error → ,reserve;

[0253] Sparse coding: LARS algorithm iterates 10 times, coefficient update amount ,convergence.

[0254] Step 6, Multi-dimensional compression performance evaluation module (Step 1-Step 3).

[0255] Step 1: Single-modal reconstruction quality assessment

[0256] 1.1 Image Modality

[0257] PSNR formula: ( );

[0258] calculate: → ;

[0259] SSIM: Calculated to be 0.93 ≥ 0.9 (general scenario standard).

[0260] 1.2 Text Modality

[0261] Word vector cosine similarity: Original text vector Reconstructing text vectors → ;

[0262] BLEU value: ,

[0263] , , , , and → .

[0264] 1.3 Audio Modal

[0265] SNR formula: ;

[0266] calculate: , → ;

[0267] Correction: Adjustment → (Meets environmental noise standards);

[0268] PESQ: Calculated value 3.2 ≥ 3.0 (understandability requirement).

[0269] Step 2: Cross-modal consistency assessment

[0270] Semantic matching accuracy: 92 out of 100 image-text pairs were correctly matched. ;

[0271] Timing synchronization error: audio event timestamps Video incident → .

[0272] Step 3: Dynamically update the effect evaluation

[0273] Compression ratio change rate: , → ;

[0274] Update time: Transmission time Calculation time → .

[0275] The beneficial effects of this invention include: first, by constructing a cross-modal collaborative dictionary structure, redundant information between multimodal data is efficiently eliminated, improving the joint compression rate; second, a dynamic dictionary update mechanism is designed, enabling the dictionary to adapt to changes in data distribution and maintain stable compression performance; third, while improving compression efficiency, semantic consistency and temporal synchronization between multimodal data are ensured (semantic matching accuracy ≥80%); and fourth, distributed dictionary management with edge-cloud collaboration is realized, reducing the computing power consumption of edge devices and the bandwidth overhead of cloud data transmission, meeting the practical application needs of scenarios such as intelligent security and smart healthcare.

[0276] Based on the same inventive concept, this application also provides a multimodal data compression device based on cross-modal dictionary collaborative generation and dynamic updating. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal data compression device based on cross-modal dictionary collaborative generation and dynamic updating provided below can be found in the limitations of the multimodal data compression method based on cross-modal dictionary collaborative generation and dynamic updating described above, and will not be repeated here.

[0277] In one embodiment, a multimodal data compression device based on cross-modal dictionary collaborative generation and dynamic updating is provided, including: a cross-modal two-layer dictionary construction module, used to construct a two-layer structure of a shared basic dictionary and each modality-specific sub-dictionary by separating the common features and individual features of multimodal data, and to associate the shared dictionary and sub-dictionaries by constructing an index table;

[0278] The cross-modal dictionary collaborative generation mechanism module is used to extract the single-modal features corresponding to each modality from multimodal data, map each single-modal feature to a unified space, obtain cross-modal features, train a shared dictionary using cross-modal features, and train a sub-dictionary for the corresponding modality using single-modal features;

[0279] The dynamic update strategy module is used to monitor the reconstruction error of each sub-dictionary through a sliding window. When the reconstruction error meets the preset conditions, the corresponding sub-dictionary is incrementally updated, the incremental matrix is ​​obtained, and the incremental matrices of each sub-dictionary are aggregated to generate a global dictionary.

[0280] The multi-dimensional evaluation module is used to construct a three-dimensional index system including single-modal reconstruction quality, cross-modal consistency, and dynamic update efficiency to evaluate the compression performance for multimodal data.

[0281] The device also includes a feature processing and management module, which enhances feature robustness through intramodal enhancement, strengthens semantic associations through cross-modal attention, manages atoms based on activity scores and sparse constraints, and balances compression efficiency and storage.

[0282] The modules in the aforementioned multimodal data compression device based on cross-modal dictionary collaborative generation and dynamic updating can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0283] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in all of the above method embodiments.

[0284] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in all of the above method embodiments.

[0285] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in all of the above method embodiments.

Claims

1. A multimodal data compression method based on cross-modal dictionary collaborative generation and dynamic updating, characterized in that, The method includes: By separating the common and individual features of multimodal data, a two-layer structure of a shared basic dictionary and modality-specific sub-dictionaries is constructed, and an index table is built to link the shared dictionary and the sub-dictionaries. Extract unimodal features corresponding to each modality from the multimodal data, map each unimodal feature to a unified space to obtain cross-modal features, train the shared dictionary using the cross-modal features, and train the sub-dictionary of the corresponding modality using the unimodal features; The reconstruction error of each sub-dictionary is monitored by a sliding window. When the reconstruction error meets a preset condition, the corresponding sub-dictionary is incrementally updated to obtain an incremental matrix. The incremental matrices of each sub-dictionary are then aggregated to generate a global dictionary. A three-dimensional index system including single-modal reconstruction quality, cross-modal consistency, and dynamic update efficiency is constructed to evaluate the compression performance of the multimodal data.

2. The method according to claim 1, characterized in that, The process involves separating the common and unique features of multimodal data to construct a two-layer structure of a shared basic dictionary and modality-specific sub-dictionaries, and then linking the shared dictionary and the sub-dictionaries by constructing an index table. The common features across modes are abstracted from the multimodal data to form shared atoms, and a shared basic dictionary including several of the shared atoms is constructed. The individual features corresponding to each modality are extracted from the multimodal data to form sub-dictionary atoms, and the sub-dictionary atoms corresponding to the same modality are summarized to construct the sub-dictionary of the corresponding modality; For each shared atom, the sub-dictionary atoms are traversed, and the sub-dictionary atoms that are semantically related to the shared atom are filtered out. An index table is then constructed to record the association between the sub-dictionary atoms and the shared atom.

3. The method according to claim 1, characterized in that, The step of training the shared dictionary using the cross-modal features and training the corresponding modality sub-dictionary using the single-modal features includes: Using the cross-modal features as input, the shared basic dictionary is updated using the KSVD algorithm; Using the single-modal features as input, the sub-dictionary corresponding to each modality is updated using the KSVD algorithm; The updated shared base dictionary and the updated sub-dictionary are jointly optimized by a total loss function to reconstruct the shared dictionary and the sub-dictionary; wherein the total loss function includes the reconstruction loss of the shared base dictionary, the reconstruction loss of each sub-dictionary, and the cross-modal alignment loss.

4. The method according to claim 1, characterized in that, The step of monitoring the reconstruction error of each sub-dictionary through a sliding window and triggering an incremental update of the corresponding sub-dictionary when the reconstruction error meets a preset condition includes: Specify the size of the sliding window for each different modality; Within each of the sliding windows, the reconstruction error for each mode is calculated; For a given modality, if the reconstruction error within multiple consecutive sliding windows exceeds the corresponding threshold, then the edge device is triggered to incrementally update the sub-dictionary corresponding to the modality.

5. The method according to claim 1, characterized in that, The step of obtaining the increment matrix and aggregating the increment matrices of each sub-dictionary to generate a global dictionary includes: Extract the newly added feature set that triggers the update within the window as candidate atoms, use the KSVD algorithm to obtain the incremental matrix based on the candidate atoms, record the data volume within the corresponding window, and upload the incremental matrix and the data volume within the window to the cloud. The cloud receives the incremental matrix and the amount of data within the window uploaded by all edge devices, obtains the global increment based on the federated averaging algorithm, generates the global dictionary based on the global increment, and distributes the global dictionary to each edge device, so that the edge device updates the local sub-dictionary based on the global dictionary.

6. The method according to claim 1, characterized in that, The method further includes: Feature enhancement is performed on the single-modal features; Cross-modal association weights are obtained through an attention mechanism, and the cross-modal features are updated accordingly. The atoms in the sub-dictionary and the shared base dictionary are updated based on the atomic activity score.

7. A multimodal data compression device based on cross-modal dictionary collaborative generation and dynamic updating, characterized in that, The device includes: A cross-modal two-layer dictionary construction module is used to construct a two-layer structure of a shared basic dictionary and modality-specific sub-dictionaries by separating the common and individual features of multimodal data, and to associate the shared dictionary and the sub-dictionaries by constructing an index table; A cross-modal dictionary collaborative generation mechanism module is used to extract single-modal features corresponding to each modality from the multimodal data, map each single-modal feature to a unified space, obtain cross-modal features, train the shared dictionary using the cross-modal features, and train the sub-dictionary of the corresponding modality using the single-modal features; The dynamic update strategy module is used to monitor the reconstruction error of each sub-dictionary through a sliding window, trigger incremental update of the corresponding sub-dictionary when the reconstruction error meets the preset conditions, obtain the incremental matrix, and aggregate the incremental matrices of each sub-dictionary to generate a global dictionary; A multi-dimensional evaluation module is used to construct a three-dimensional index system including single-modal reconstruction quality, cross-modal consistency, and dynamic update efficiency to evaluate the compression performance of the multimodal data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images