Edge cloud collaborative multi-modal data real-time processing method and system

Abstract

The invention discloses an edge cloud collaborative multi-modal data real-time processing method and system. The method mainly comprises the steps that 1) a data sensing system directionally collects multi-modal data; 2) the edge device preprocesses the multi-modal data and uploads the multi-modal data to the cloud computing center; 3) the cloud computing center constructs a multi-modal data model to extract high-order feature vectors of the multi-modal data; 4) the cloud computing center performs data fusion processing on the extracted feature data to obtain fusion feature data with unified dimensions, and transmits the fusion feature data to the edge device; and 5) the edge device performs post-processing and application output on the fusion data. According to the invention, multi-modal data is processed through edge cloud, multi-source information is fused and real-time response is carried out to improve processing efficiency and accuracy, bandwidth pressure and cost are reduced through local preprocessing and edge resource scheduling, and system reliability is enhanced through a distributed architecture and data redundancy.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and multimodal data processing technology, specifically a method and system for real-time processing of multimodal data in an edge-cloud collaborative manner. Background Technology

[0002] With the advancement of information technology, multimodal data processing technology has attracted widespread attention due to its ability to integrate various information formats such as text, images, and audio, thereby providing more comprehensive and accurate analysis results. Meanwhile, edge computing pushes computing power down to the network edge, bringing potential advantages such as low latency and localization to data processing. However, when multimodal data processing tasks are placed in actual edge cloud environments, existing technical solutions reveal a series of shortcomings in efficiency, collaboration, and reliability that urgently need to be addressed.

[0003] Existing solutions for processing multimodal data often employ simple, centralized processing workflows. This typically requires data to be transmitted over long distances to a central cloud for fusion and analysis, introducing significant transmission delays and failing to meet the stringent real-time requirements of scenarios such as intelligent security and industrial quality inspection. Furthermore, existing methods are relatively crude in their multimodal data fusion, failing to effectively leverage the complementary advantages between different modalities and struggling to overcome the inherent ambiguity and incompleteness of single-modal data, thus limiting the accuracy of the final analysis results.

[0004] In edge cloud distributed architectures, existing technologies lack intelligent and adaptive task allocation and collaboration mechanisms. This easily leads to unbalanced computing loads between edge nodes and terminal devices, with some devices overloaded while others remain idle, resulting in low overall system resource utilization. Furthermore, due to the lack of effective data filtering and compression at the edge, large amounts of raw data are transmitted indiscriminately, consuming valuable network bandwidth and exacerbating the computing and storage burden on the central cloud, thus negating the cost control advantages of edge computing.

[0005] Existing solutions rely excessively on a few central nodes or specific data sources, resulting in a fragile system architecture. When an edge node fails or network connectivity becomes unstable, the entire processing flow may be interrupted, leading to poor system robustness. Furthermore, because they fail to fully utilize the inherent redundancy and cross-verification characteristics of multimodal data, the system lacks effective fault tolerance and compensation mechanisms should a single data source fail or be lost, severely threatening the continuity of the analysis process and the reliability of the results.

[0006] In summary, existing technologies have inherent drawbacks when processing multimodal tasks in edge cloud environments, such as poor real-time performance, low fusion accuracy, inefficient resource coordination, and insufficient system reliability. Summary of the Invention

[0007] The purpose of this invention is to provide a real-time processing method for multimodal data in edge cloud collaboration, comprising the following steps:

[0008] Step 1) Use a data sensing system to collect multimodal data in a directional manner, and record the spatiotemporal metadata of the data synchronously during the collection process;

[0009] Step 2) Use edge devices to preprocess the multimodal data, and then upload the preprocessed multimodal data to the cloud computing center according to data priority rules;

[0010] Step 3) The cloud computing center uses a multimodal data model built on deep belief networks to extract features from the high-order feature vectors of the multimodal data;

[0011] The deep belief network is composed of multiple restricted Boltzmann machines stacked together;

[0012] Step 4) The cloud computing center performs weighted fusion on the extracted feature data to obtain fused feature data with a unified dimension, and transmits it to the edge device;

[0013] Step 5) Edge devices perform post-processing and application output on the fused data.

[0014] Furthermore, multimodal data includes video data, audio data, image data, text information, and status values.

[0015] Furthermore, in step 2), the preprocessing of multimodal data using edge devices includes clock unification, linear interpolation, data cleaning, data normalization, data dimensionality reduction, and data format conversion.

[0016] Furthermore, clock unification refers to the process of converting GPS clocks into BeiDou clocks using time format conversion rules.

[0017] The time format conversion rules are as follows:

[0018]

[0019] In the formula, The total number of seconds at the current GPS time is defined as the cumulative number of seconds from the GPS time start point to the data collection time. This represents the total number of seconds in the converted BeiDou time. This refers to the week number of the BeiDou system. The number of seconds within a week when using BeiDou;

[0020] Linear interpolation processing refers to using linear interpolation to process sampling information collected by devices with different sampling frequencies in order to obtain synchronously collected information from different sampling devices at the same time.

[0021] The interpolation result x0 at the target synchronization time t0 is shown below:

[0022]

[0023] In the formula, t1 and t2 are the BeiDou standard times of two adjacent samplings by the device; x1 and x2 are the sampled data values ​​at times t1 and t2 respectively; t0 is the target synchronization time; and x0 is the interpolation result at time t0.

[0024] Data cleaning includes data denoising, missing value handling, and outlier removal;

[0025] Data normalization refers to normalizing data to zero mean or logarithmic mean.

[0026] Zero-mean normalization is shown below:

[0027]

[0028] in It is the raw data. It is the mean of the data. It is the standard deviation of the data. It is normalized data;

[0029] Data format conversion refers to converting image data into a unified format, audio data into a unified format, and text data into digital vectors using the TensorFlow framework. Then, it involves using natural language processing libraries to preprocess and represent the text data, completing word segmentation, part-of-speech tagging, named entity recognition, and word vector training.

[0030] Furthermore, the steps for data dimensionality reduction include:

[0031] S1) Calculate the covariance matrix of the data, i.e.:

[0032]

[0033] In the formula, each row of X represents a sample, and each column represents a feature; μ is a vector composed of the mean values ​​of each feature dimension. ;X i,j C represents the j-th feature of the i-th sample; j,k Let be the element in the j-th row and k-th column of the covariance matrix, representing the covariance between the j-th feature and the k-th feature.

[0034] S2) Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​λ1≥λ2≥...≥λ M and the corresponding unit eigenvectors v1, v2, ..., v M ,Right now:

[0035]

[0036] S3) Based on the cumulative variance contribution rate Select the eigenvectors corresponding to the first K eigenvalues ​​to form the projection matrix. ;

[0037] Among them, the cumulative variance contribution rate As shown below:

[0038]

[0039] S4) Project the data to obtain the dimensionality reduction result Y, i.e.:

[0040]

[0041] In the formula, The first covariance matrix is ​​the first... One eigenvalue; For the first The unit eigenvectors corresponding to each eigenvalue; V is the projection matrix, with each column corresponding to a selected eigenvector; The dataset is after dimensionality reduction; K is the feature dimension after dimensionality reduction; K <M。

[0042] Furthermore, the steps for constructing a multimodal data model based on deep belief networks include:

[0043] S1) Construct a deep belief network consisting of multiple Restricted Boltzmann Machines (RBMs) stacked together;

[0044] S2) Obtain multimodal historical data, including video historical data, audio historical data, historical image data, historical text information, and historical status values;

[0045] S3) Use convolutional neural networks to extract feature data from historical video and image data; use audio feature extraction algorithms to extract feature data from historical audio data; use word vector models to extract feature data from historical text information and historical state values.

[0046] S4) Using historical data of a modality as input, train the first RBM to obtain the feature representation of the historical data of that modality;

[0047] S5) Use the feature representation output by the first RBM as the input of the (i+1)th RBM to train the (i+1)th RBM and obtain the feature representation of the historical data of this modality.

[0048] S6) Let i = i + 1, and return to step S5) until all RBMs have finished training;

[0049] Furthermore, after training all the RBMs, the deep belief network is fine-tuned using the backpropagation algorithm.

[0050] Furthermore, step 5) post-processing of the fused data by the edge device refers to performing real-time calculations on the fused data.

[0051] The system based on the aforementioned edge cloud collaborative multimodal data real-time processing method includes a data perception layer, an edge node layer, a cloud computing center layer, and a network transmission layer.

[0052] The data perception layer uses a data perception system to collect multimodal data in a directional manner, and records the spatiotemporal metadata of the data synchronously during the collection process;

[0053] The edge node layer interfaces with the data perception layer and the cloud computing center layer through the network transmission layer;

[0054] The edge node layer preprocesses the multimodal data and uploads the preprocessed multimodal data to the cloud computing center according to data priority rules;

[0055] The cloud computing center layer uses a multimodal data model built on a deep belief network to extract features from the high-order feature vectors of multimodal data;

[0056] The cloud computing center performs weighted fusion on the extracted feature data to obtain fused feature data with a unified dimension, and then transmits it to the edge device;

[0057] The multimodal data model comprises multiple stacked restricted Boltzmann machines;

[0058] The edge device layer performs post-processing and application output on the fused data.

[0059] The technical effects of this invention are undeniable, and its beneficial effects are as follows:

[0060] 1) Improve data processing efficiency and accuracy by fusing multimodal data such as images, audio, and text in the edge cloud to avoid the defects of single-modal information, and achieve real-time processing by relying on the advantage of the edge's proximity to the data source, so as to quickly respond to the needs of scenarios such as intelligent security and industrial inspection.

[0061] 2) Reduce resource utilization and control costs. At the edge, data is first filtered and compressed, and only key information is uploaded to alleviate network bandwidth pressure. At the same time, edge device resources are fully utilized to reduce dependence on the central cloud and optimize system resource configuration.

[0062] 3) Enhance system adaptability and reliability. With the help of edge cloud distributed architecture and multimodal data redundancy, the system can still operate even if some nodes fail. Moreover, multimodal data can be mutually verified, reducing the risk of failure of a single data source. Attached Figure Description

[0063] Figure 1 This is a flowchart of the data processing process. Detailed Implementation

[0064] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0065] Example 1:

[0066] See Figure 1 A real-time processing method for multimodal data in edge cloud collaboration includes the following steps:

[0067] Step 1) Use a data sensing system to collect multimodal data in a directional manner, and record the spatiotemporal metadata of the data synchronously during the collection process;

[0068] Step 2) Use edge devices to preprocess the multimodal data, and then upload the preprocessed multimodal data to the cloud computing center according to data priority rules;

[0069] Step 3) The cloud computing center uses a multimodal data model built on deep belief networks to extract features from the high-order feature vectors of the multimodal data;

[0070] The deep belief network is composed of multiple restricted Boltzmann machines stacked together;

[0071] Step 4) The cloud computing center performs weighted fusion on the extracted feature data to obtain fused feature data with a unified dimension, and transmits it to the edge device;

[0072] Step 5) Edge devices perform post-processing and application output on the fused data.

[0073] Example 2:

[0074] A real-time processing method for multimodal data in edge cloud collaboration, with the same technical content as in Embodiment 1, further comprising multimodal data including video data, audio data, image data, text information and status values.

[0075] Example 3:

[0076] A real-time processing method for multimodal data in edge cloud collaboration, with the same technical content as any one of embodiments 1-2, further wherein, in step 2), the preprocessing of multimodal data using edge devices includes clock unification, linear interpolation, data cleaning, data normalization, data dimensionality reduction, and data format conversion.

[0077] Example 4:

[0078] A method for real-time processing of multimodal data in edge cloud collaboration, with the same technical content as any one of embodiments 1-3, further wherein clock unification refers to converting GPS clocks into BeiDou clocks using time format conversion rules;

[0079] The time format conversion rules are as follows:

[0080]

[0081] In the formula, The total number of seconds at the current GPS time is defined as the cumulative number of seconds from the GPS time start point to the data collection time. This represents the total number of seconds in the converted BeiDou time. This refers to the week number of the BeiDou system. The number of seconds within a week when using BeiDou;

[0082] Linear interpolation processing refers to using linear interpolation to process sampling information collected by devices with different sampling frequencies in order to obtain synchronously collected information from different sampling devices at the same time.

[0083] The interpolation result x0 at the target synchronization time t0 is shown below:

[0084]

[0085] In the formula, t1 and t2 are the BeiDou standard times of two adjacent samplings by the device; x1 and x2 are the sampled data values ​​at times t1 and t2 respectively; t0 is the target synchronization time; and x0 is the interpolation result at time t0.

[0086] Data cleaning includes data denoising, missing value handling, and outlier removal;

[0087] Data normalization refers to normalizing data to zero mean or logarithmic mean.

[0088] Zero-mean normalization is shown below:

[0089]

[0090] in It is the raw data. It is the mean of the data. It is the standard deviation of the data. It is normalized data;

[0091] Data format conversion refers to converting image data into a unified format, audio data into a unified format, and text data into digital vectors using the TensorFlow framework. Then, it involves using natural language processing libraries to preprocess and represent the text data, completing word segmentation, part-of-speech tagging, named entity recognition, and word vector training.

[0092] Example 5:

[0093] A real-time processing method for multimodal data in edge cloud collaboration, with technical content identical to any one of embodiments 1-4, further comprising the following data dimensionality reduction steps:

[0094] S1) Calculate the covariance matrix of the data, i.e.:

[0095]

[0096] In the formula, each row of X represents a sample, and each column represents a feature; μ is a vector composed of the mean values ​​of each feature dimension. ;X i,j C represents the j-th feature of the i-th sample; j,k Let be the element in the j-th row and k-th column of the covariance matrix, representing the covariance between the j-th feature and the k-th feature.

[0097] S2) Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​λ1≥λ2≥...≥λ M and the corresponding unit eigenvectors v1, v2, ..., v M ,Right now:

[0098]

[0099] S3) Based on the cumulative variance contribution rate Select the eigenvectors corresponding to the first K eigenvalues ​​to form the projection matrix. ;

[0100] Among them, the cumulative variance contribution rate As shown below:

[0101]

[0102] S4) Project the data to obtain the dimensionality reduction result Y, i.e.:

[0103]

[0104] In the formula, The first covariance matrix is ​​the first... One eigenvalue; For the first The unit eigenvectors corresponding to each eigenvalue; V is the projection matrix, with each column corresponding to a selected eigenvector; The dataset is after dimensionality reduction; K is the feature dimension after dimensionality reduction; K <M。

[0105] Example 6:

[0106] A real-time processing method for multimodal data in edge cloud collaboration, with technical content identical to any one of embodiments 1-5, further comprising the following steps for constructing a multimodal data model based on a deep belief network:

[0107] S1) Construct a deep belief network consisting of multiple Restricted Boltzmann Machines (RBMs) stacked together;

[0108] S2) Obtain multimodal historical data, including video historical data, audio historical data, historical image data, historical text information, and historical status values;

[0109] S3) Use convolutional neural networks to extract feature data from historical video and image data; use audio feature extraction algorithms to extract feature data from historical audio data; use word vector models to extract feature data from historical text information and historical state values.

[0110] An unsupervised, layer-by-layer greedy training method is adopted, aiming to minimize the reconstruction error or maximize the likelihood probability of the training data, and each RBM is trained sequentially from the bottom layer to the top layer:

[0111] S4) Using historical data of a modality as input, train the first RBM to obtain the feature representation of the historical data of that modality;

[0112] S5) Use the feature representation output by the first RBM as the input of the (i+1)th RBM to train the (i+1)th RBM and obtain the feature representation of the historical data of this modality.

[0113] S6) Let i = i + 1, and return to step S5) until all RBMs have finished training;

[0114] Example 7:

[0115] A method for real-time processing of multimodal data in edge cloud collaboration, with the same technical content as any one of embodiments 1-6, further comprising: after training all RBMs, using the backpropagation algorithm to perform supervised fine-tuning of the deep belief network.

[0116] Example 8:

[0117] A real-time processing method for multimodal data in edge cloud collaboration, with the same technical content as any one of embodiments 1-7, further comprising: performing feature extraction on data of each modality; for image data, using a convolutional neural network (CNN) to extract feature vectors of the image; for audio data, using an audio feature extraction algorithm to extract audio features; and for text data, using a word vector model to convert the text into a vector representation.

[0118] Example 9:

[0119] A real-time processing method for multimodal data in edge-cloud collaboration, with technical content identical to any one of embodiments 1-8, further comprising the following step: post-processing of fused data by the edge device refers to real-time computational processing of the fused data (e.g., detection and recognition of personnel and vehicle targets, autonomous driving, etc.). In the fields of personnel and vehicle target detection and recognition, and autonomous driving, multimodal data includes traffic data related to vehicle driving. The data perception system includes cameras deployed on roads, positioning devices deployed in vehicles, control devices, etc.

[0120] Example 10:

[0121] The system based on the edge cloud collaborative multimodal data real-time processing method described in any one of Embodiments 1-9 includes a data perception layer, an edge node layer, a cloud computing center layer, and a network transmission layer;

[0122] The data perception layer uses a data perception system to collect multimodal data in a directional manner, and records the spatiotemporal metadata of the data synchronously during the collection process;

[0123] The edge node layer interfaces with the data perception layer and the cloud computing center layer through the network transmission layer;

[0124] The edge node layer preprocesses the multimodal data and uploads the preprocessed multimodal data to the cloud computing center according to data priority rules;

[0125] The cloud computing center layer uses a multimodal data model built on a deep belief network to extract features from the high-order feature vectors of multimodal data;

[0126] The cloud computing center performs weighted fusion on the extracted feature data to obtain fused feature data with a unified dimension, and then transmits it to the edge device;

[0127] The multimodal data model comprises multiple stacked restricted Boltzmann machines;

[0128] The edge device layer performs post-processing and application output on the fused data.

[0129] Example 11:

[0130] A real-time processing method for multi-modal data with edge-cloud collaboration, including the following steps:

[0131] 1) Multi-modal data collection: In the data perception layer, according to the existing data perception system, it includes various types of data such as video data, audio data, image data, text information, and status values. In the edge node layer, it supports the access of front-end field devices, information systems, and databases downward, and docks with the cloud computing center layer upward, including edge gateways and edge servers. Edge devices perform preliminary processing and management on the collected multi-modal data to prepare for subsequent data transmission and processing. Through the network transmission layer, as a bridge for data transmission and scheduling control between the cloud computing center, edge nodes, and even front-end field devices. Ensure that multi-modal data can be stably and efficiently transmitted in different network environments.

[0132] 2) Clock unification for multi-modal data: Convert the GPS clock uniformly to the Beidou time, calculate the total number of seconds at the current GPS time, and then convert it into the total number of seconds and the number of seconds within a week in Beidou time.

[0133] Combined with the time format conversion rules, the quantization formula for converting GPS time to Beidou time is as follows:

[0134]

[0135] : The total number of seconds at the current GPS time (unit: second), defined as "the cumulative number of seconds from the GPS time starting point (0:00:00 on January 6, 1980) to the collection time";

[0136] : The total number of seconds in the converted Beidou time (unit: second);

[0137] : The number of weeks in Beidou time (unit: week), 604800 is the total number of seconds in a week (7×24×3600);

[0138] : The number of seconds within a week in Beidou time (unit: second), with a value range of [0, 604800).

[0139] And interpolation processing is adopted for the sampling information collected by devices with different sampling frequencies to obtain synchronous sampling information of different sampling devices at the same time.

[0140] Using the linear interpolation method, assume that the sampling value of a certain device at time t1 is x1, and the sampling value at time t2 is x2 (t1 < t0 < t2), and find the interpolation result x0 at the target synchronization time t0:

[0141]

[0142] Parameter description:

[0143] t1, t2: BeiDou standard times (unit: seconds, expressed in T) between two consecutive samplings by the device BDT_tow express);

[0144] x1, x2: Sampling data values ​​corresponding to times t1 and t2 (which can be image pixel values, audio amplitude, state values, etc., depending on the data type);

[0145] t0: Target synchronization time (unified number of seconds within the BeiDou week);

[0146] x0: The interpolation result at time t0.

[0147] 3) Data Cleaning: This includes data denoising, handling missing values, and outlier removal. For image data, mean filtering is used to reduce random noise and improve image quality. For audio data, spectral subtraction is used to remove different types of noise and improve audio clarity. For text data, noise is reduced by removing stop words, punctuation marks, and special characters. Missing values ​​are analyzed; if there are few missing values, interpolation is used to fill them. Interpolation infers the values ​​of missing data points based on known data points using a specific algorithm. Common interpolation methods include linear interpolation, polynomial interpolation, and spline interpolation. If there are many missing values, deletion or model prediction methods are used. To ensure data integrity, model prediction methods such as regression models, decision trees, and random forests are used to build predictive models and fill in missing data. Outliers are detected in the collected data using machine learning methods such as isolated forests and local anomaly detection to learn normal patterns in the data and identify outliers. For outliers caused by the collected data, the data can be deleted directly, analyzed as a separate category, or special processing methods such as weighted processing or robust regression can be used.

[0148] 4) Data Normalization: The processed data is formatted and normalized to unify the data format. The data is partitioned into ranges. For data with a small range, zero-mean normalization is used to change the mean to 0 and the standard deviation to 1. For data with a mean of 0 and a standard deviation of 1, zero-mean normalization can be performed using the following formula:

[0149]

[0150] in It is the raw data. It is the mean of the data. It is the standard deviation of the data. It is the normalized data.

[0151] For data with a large range and uneven distribution, logarithmic normalization is used. This is achieved through the formula... The calculated data is then normalized.

[0152] Data dimensionality reduction: Dimensionality reduction is achieved through PCA (Principal Component Analysis). This involves calculating the covariance matrix of the data, solving for the eigenvalues ​​and eigenvectors of the covariance matrix, selecting the eigenvectors corresponding to the largest eigenvalues ​​as new bases, and projecting the original data onto these new bases to obtain the dimensionality-reduced data. This reduces the complexity of the data model and lowers the risk of overfitting.

[0153] Let the normalized original data matrix be X∈R N×M (N is the number of samples, M is the original feature dimension), the data mean vector is μ=[μ 1， μ 2， ...,μ M ] T The covariance matrix C∈R M×M :

[0154]

[0155] Parameter description:

[0156] X: Each row represents a sample, and each column represents a feature;

[0157] μ: A vector composed of the means of each feature dimension. (X) i,j (for the j-th feature of the i-th sample).

[0158] C j,k The element in the j-th row and k-th column of the covariance matrix represents the covariance between the j-th feature and the k-th feature.

[0159] Eigenvalue decomposition and data projection: Eigenvalue decomposition is performed on the covariance matrix C to obtain eigenvalues ​​λ1≥λ2≥...≥λ M and the corresponding unit eigenvectors v1, v2, ..., v M :

[0160]

[0161] Principal components are selected based on the "cumulative variance contribution rate". (usually taken) 0%) Select the eigenvectors corresponding to the first K eigenvalues ​​to form the projection matrix. :

[0162]

[0163] Data projection (dimensionality reduction result):

[0164]

[0165] Parameter description:

[0166] : The first covariance matrix Each eigenvalue represents the "variance contribution" of the corresponding eigenvector;

[0167] : No. The unit eigenvector corresponding to each eigenvalue (with a magnitude of 1);

[0168] V: Projection matrix, each column corresponds to a selected eigenvector;

[0169] The dataset after dimensionality reduction, where K is the feature dimension after dimensionality reduction (Ki). <M)。

[0170] 6) Data Format Conversion: For image data in different formats such as JPEG, PNG, and BMP, the image data is converted to a unified format using the TensorFlow framework to convert it to TFRecord format. OpenCV and PIL libraries are used for image reading, conversion, and saving. For audio data in formats such as WAV, MP3, and AAC, TensorFlow converts it to WAV format. The librosa and pydub libraries provide audio reading, conversion, and processing functions, facilitating convenient audio data format conversion and preprocessing. For text data, it is converted into a numerical vector representation for processing by machine learning algorithms. Natural language processing libraries such as NLTK, spaCy, and gensim are used for text data preprocessing and representation, performing word segmentation, part-of-speech tagging, named entity recognition, and word vector training.

[0171] 7) Constructing a Multimodal Data Model: A deep belief network (DBN) is used to construct the multimodal data model. First, the network structure is determined based on the characteristics of the multimodal data and the task requirements. Typically, a DBN consists of multiple stacked Restricted Boltzmann Machines (RBMs). Each RBM can be viewed as an unsupervised feature learner used to extract feature representations from the input data. RBM Training: Each RBM is trained sequentially. First, one modality of data is used as input to train the first RBM, allowing it to learn the feature representations of that modality. Then, the output of the first RBM is used as the input to the next RBM, and so on. This layer-by-layer training method allows the learning of high-level feature representations of the multimodal data. Network Fine-tuning: After training all RBMs, the entire DBN can be fine-tuned using backpropagation. This step is usually supervised and requires training with labeled multimodal data to optimize network parameters and improve model performance. Once the data model is complete, data fusion can be performed.

[0172] 8) Data Fusion: Feature extraction is performed separately for each modality of data. For example, for image data, convolutional neural networks (CNNs) can be used to extract feature vectors; for audio data, audio feature extraction algorithms, such as Mel-frequency cepstral coefficients (MFCCs), can be used to extract audio features; for text data, word vector models, such as Word2Vec and GloVe, can be used to convert text into vector representations.

[0173] The extracted features should be able to fully reflect the key information of the modality data and have a certain degree of representativeness and distinctiveness.

[0174] Feature vectors are concatenated and fused to form a longer feature vector. For example, suppose the image feature vector is... The audio feature vector is Then the concatenated feature vector is Weighted fusion assigns different weights to features from different modalities and then performs a weighted summation. The weights can be obtained through training or set empirically.

[0175] Weighted fusion formula: Let the set of multimodal feature vectors be {F1, F2, ..., F...} m} (where m is the number of modes), and the corresponding weight set is {w1, w2, ..., w m The fused feature vector F fusion :

[0176]

[0177] The weights must satisfy the normalization constraint:

[0178]

[0179] Parameter description:

[0180] : No. Feature vectors of various modalities (such as image CNN features) Audio MFCC features First, feature dimensions need to be aligned (e.g., zero-padding, linear mapping) to be unified to the same dimension. ;

[0181] : No. The weights of each mode represent the contribution of that mode to the fusion result;

[0182] : Final fused feature vector.

[0183] Based on validation set loss optimization, the weight calculation formula is as follows:

[0184] Taking a classification task as an example, let the label of the validation set sample be y, and the predicted probability output by the classifier after fusing features is... Loss function L:

[0185]

[0186] Update weights using gradient descent:

[0187]

[0188] Parameter description:

[0189] : Number of validation set samples;

[0190] : The true label of the nth validation set sample;

[0191] : The predicted probability of the fused features of the nth sample;

[0192] Learning rate (usually 10) −3 ~10 −1 );

[0193] : The partial derivative of the loss function with respect to the i-th weight.

[0194] Leveraging the computing power of edge devices, the merged data is processed in real time. Kubernetes and containerization technologies are used to manage and schedule edge devices, and the processed data is uploaded to the cloud for distributed computing. Cloud data is then processed in real time using Storm or Flink real-time big data engines.

[0195] The processed data is transmitted from the cloud to the edge cloud for management of multimodal data stored there, including backup, recovery, version control, and security management. This ensures data security and reliability, preventing data loss and leakage. Future improvements will focus on increasing storage efficiency and access speed by compressing image and audio data to reduce storage space usage; and by indexing the data for faster querying and retrieval.

Claims

1. A method for real-time processing of multimodal data in edge cloud collaboration, characterized in that, Includes the following steps: Step 1) Use a data sensing system to collect multimodal data in a directional manner, and record the spatiotemporal metadata of the data synchronously during the collection process; Step 2) Use edge devices to preprocess the multimodal data, and then upload the preprocessed multimodal data to the cloud computing center according to data priority rules; Step 3) The cloud computing center uses a multimodal data model built on deep belief networks to extract features from the high-order feature vectors of the multimodal data; The deep belief network is composed of multiple restricted Boltzmann machines stacked together; Step 4) The cloud computing center performs weighted fusion on the extracted feature data to obtain fused feature data with a unified dimension, and transmits it to the edge device; Step 5) Edge devices perform post-processing and application output on the fused data.

2. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 1, characterized in that, Multimodal data includes video data, audio data, image data, text information, and status values.

3. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 1, characterized in that, In step 2), the preprocessing of multimodal data using edge devices includes clock unification, linear interpolation, data cleaning, data normalization, data dimensionality reduction, and data format conversion.

4. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 3, characterized in that, Clock unification refers to the process of converting GPS clocks into BeiDou clocks using time format conversion rules. The time format conversion rules are as follows: In the formula, The total number of seconds at the current GPS time is defined as the cumulative number of seconds from the GPS time start point to the data collection time. This represents the total number of seconds in the converted BeiDou time. This refers to the week number of the BeiDou system. The number of seconds within a week when using BeiDou; Linear interpolation processing refers to using linear interpolation to process sampling information collected by devices with different sampling frequencies in order to obtain synchronously collected information from different sampling devices at the same time. The interpolation result x0 at the target synchronization time t0 is shown below: In the formula, t1 and t2 are the BeiDou standard times of two consecutive samplings by the device; x1 and x2 are the sampling data values ​​corresponding to times t1 and t2. t0 is the target synchronization time; x0 is the interpolation result at time t0; Data cleaning includes data denoising, missing value handling, and outlier removal; Data normalization refers to normalizing data to zero mean or logarithmic mean. Zero-mean normalization is shown below: in It is the raw data. It is the mean of the data. It is the standard deviation of the data. It is the normalized data.

5. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 3, characterized in that, Data format conversion refers to converting image data into a unified format, audio data into a unified format, and text data into digital vectors using the TensorFlow framework. Then, it involves using natural language processing libraries to preprocess and represent the text data, completing word segmentation, part-of-speech tagging, named entity recognition, and word vector training.

6. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 3, characterized in that, The steps of data dimensionality reduction include: Step S1) Calculate the covariance matrix of the data, i.e.: In the formula, each row of X represents a sample, and each column represents a feature; μ is a vector composed of the mean values ​​of each feature dimension. ;X i,j C represents the j-th feature of the i-th sample; j,k Let be the element in the j-th row and k-th column of the covariance matrix, representing the covariance between the j-th feature and the k-th feature; Step S2) Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​λ1≥λ2≥...≥λ M and the corresponding unit eigenvectors v1, v2, ..., v M ,Right now: Step S3) Based on the cumulative variance contribution rate Select the eigenvectors corresponding to the first K eigenvalues ​​to form the projection matrix. ; Among them, the cumulative variance contribution rate As shown below: Step S4) Project the data to obtain the dimensionality reduction result Y, i.e.: In the formula, The first covariance matrix is ​​the first... One eigenvalue; For the first The unit eigenvectors corresponding to each eigenvalue; V is the projection matrix, with each column corresponding to a selected eigenvector; The dataset is after dimensionality reduction; K is the feature dimension after dimensionality reduction; K <M。 7. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 1, characterized in that, The steps for constructing a multimodal data model based on deep belief networks include: Step S1) Construct a deep belief network consisting of multiple Restricted Boltzmann Machines (RBMs) stacked together; Step S2) Obtain multimodal historical data, including video historical data, audio historical data, historical image data, historical text information, and historical status values; Step S3) Use a convolutional neural network to extract feature data from historical video data and historical image data; use an audio feature extraction algorithm to extract feature data from historical audio data; use a word vector model to extract feature data from historical text information and historical state values. Step S4) Use historical data of a modality as input to train the first RBM and obtain the feature representation of the historical data of that modality; Step S5) Use the feature representation output by the first RBM as the input of the (i+1)th RBM to train the (i+1)th RBM and obtain the feature representation of the historical data of this modality. Step S6) Let i = i + 1, and return to step S5) until all RBM training is completed.

8. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 1, characterized in that, After training all the RBMs, the deep belief network is fine-tuned using the backpropagation algorithm.

9. The method for real-time processing of multimodal data in edge cloud collaboration according to claim 1, characterized in that, Step 5) Post-processing of fused data by edge devices refers to performing real-time calculations on the fused data.

10. A system based on the edge cloud collaborative multimodal data real-time processing method according to any one of claims 1-9, characterized in that: It includes the data perception layer, edge node layer, cloud computing center layer, and network transmission layer; The data perception layer uses a data perception system to collect multimodal data in a directional manner, and records the spatiotemporal metadata of the data synchronously during the collection process; The edge node layer interfaces with the data perception layer and the cloud computing center layer through the network transmission layer; The edge node layer preprocesses the multimodal data and uploads the preprocessed multimodal data to the cloud computing center according to data priority rules; The cloud computing center layer uses a multimodal data model built on a deep belief network to extract features from the high-order feature vectors of multimodal data; The cloud computing center performs weighted fusion on the extracted feature data to obtain fused feature data with a unified dimension, and then transmits it to the edge device; The multimodal data model comprises multiple stacked restricted Boltzmann machines; The edge device layer performs post-processing and application output on the fused data.

Images