A cloud-edge collaboration-based multi-modal data acquisition and fusion method and system

Abstract

The application relates to the field of data collection and specifically discloses a multi-modal data collection and fusion method and system based on cloud-edge cooperation, which comprises the following steps: collecting multi-modal original data by each edge node in an edge layer, and encapsulating the multi-modal original data into a standard data model containing a metadata layer and an initial characteristic value layer after preprocessing; receiving and analyzing the standard data model by a cloud layer to obtain target data and target parameters containing collection accuracy, and matching a fusion algorithm and an initial fusion weight of a target device according to business requirements; the matched fusion algorithm fuses the target data according to the initial fusion weight and the collection accuracy; performing precision evaluation on the fusion result, and generating an adjustment instruction of a data collection strategy and / or a model weight parameter of the edge layer according to the precision evaluation result; and delivering the adjustment instruction to the edge layer. The application realizes unified modeling of multi-modal data and autonomous regulation and control of resources, and improves data processing efficiency, fusion accuracy and system resource utilization.

Description

Technical Field

[0001] This application relates to the field of data acquisition, specifically to a multimodal data acquisition and fusion method and system based on cloud-edge collaboration. Background Technology

[0002] In complex scenarios such as the Industrial Internet and smart cities, data collection sources are becoming increasingly diverse, encompassing multiple modalities including sensors, video surveillance, audio equipment, and system logs. These multimodal data exhibit significant heterogeneity in format, temporal sequence, and semantics, posing a significant challenge to unified data processing and in-depth value mining.

[0003] Currently, common multimodal data processing solutions typically lack unified data modeling standards at the data acquisition and transmission levels. The vastly different formats of raw data across different modalities necessitate cumbersome format conversions before and after transmission, consuming network bandwidth, introducing errors, and causing the loss of underlying data features. At the data fusion level, the cloud often employs fusion algorithms or strategies tied to specific business modalities, such as using CNNs for video data and weighted averages for time-series sensor data. While the algorithms correspond to the modalities, the parameters and weighting strategies within these algorithms lack the ability to dynamically adapt to the data's inherent attributes. This prevents the system from precisely distinguishing the quality differences between different data sources within the same or different modalities, and from assigning appropriate weights to high-precision, high-value data during the fusion process, thus limiting further improvements in fusion accuracy. Furthermore, these technologies generally lack effective closed-loop optimization mechanisms, failing to use the application effects of the fusion results to guide and optimize data acquisition behavior at the edge, resulting in low system resource utilization. Summary of the Invention To address the aforementioned issues, this invention provides a method and system for multimodal data acquisition and fusion based on cloud-edge collaboration, enabling unified modeling and autonomous resource control of multimodal data, thereby improving data processing efficiency, fusion accuracy, and system resource utilization.

[0004] In a first aspect, the technical solution of the present invention provides a multimodal data acquisition and fusion method based on cloud-edge collaboration, comprising the following steps: Each edge node in the edge layer collects multimodal raw data and preprocesses it to obtain structured data. The structured data is then transformed and encapsulated into a standard data model that includes a metadata layer and an initial feature value layer. The metadata layer contains the acquisition accuracy of the device. The standard data model is encrypted and then transmitted to the cloud layer via a communication network; The cloud layer receives and parses the standard data model to obtain target data and target parameters including acquisition accuracy, and matches the initial fusion weights of the fusion algorithm and the target device according to business requirements. The matched fusion algorithm fuses the target data according to the initial fusion weights and acquisition accuracy. The accuracy of the fusion results is evaluated, and data acquisition strategies and / or model weight parameter adjustment instructions for the edge layer are generated based on the accuracy evaluation results. The adjustment instructions are sent to the edge layer via the communication network.

[0005] Secondly, the technical solution of the present invention provides a multimodal data acquisition and fusion system based on cloud-edge collaboration. This system is used to implement the above-mentioned method, including an edge layer, a cloud layer, and a communication network layer between the two. The edge layer includes several edge nodes, each edge node corresponds to at least one set of multimodal terminal devices, and the edge nodes have a built-in standardized model builder to convert the raw multimodal data into a standard data model that includes a metadata layer and an initial feature value layer. The cloud layer includes a dynamic adaptation engine, a fusion strategy library, and a feedback optimization module. The dynamic adaptation engine is used to parse the standard data model to obtain target data and target parameters including acquisition accuracy, and to match the fusion algorithm and the initial fusion weights of the target device according to business requirements. The matched fusion algorithm fuses the target data according to the initial fusion weights and acquisition accuracy. The feedback optimization module is used to evaluate the accuracy of the fusion results and generate adjustment instructions for the data acquisition strategy and / or model weight parameters of the edge layer based on the accuracy evaluation results. The communication network layer is used to implement the standard data model and adjustment command transmission between the edge layer and the meta-fault layer.

[0006] As can be seen from the above technical solutions, this application has the following advantages: (1) Through the standardized model builder of the edge layer, the multimodal raw data is uniformly encapsulated into a standard data model containing a metadata layer and an initial feature value layer. Serialization and compression are adopted to avoid repeated format conversion of data during transmission and interaction, reduce bandwidth usage, and improve the efficiency of cloud-edge collaboration. (2) The dynamic adaptation engine of the cloud layer can dynamically initialize the algorithm parameters based on the data attributes such as the acquisition accuracy and initial fusion weight, according to the fusion algorithm matched according to the business requirements, based on the parsed model features, especially the modality type and acquisition accuracy, to achieve model-driven and refined dynamic adaptation, effectively improving the accuracy of fusion results and the ability to perceive key information. (3) Through feedback optimization, the system can automatically generate adjustment instructions for edge layer data acquisition strategies and model weight parameters based on the accuracy assessment of fusion results. This enables the system to autonomously reduce the amount of invalid data collected, penalize low-quality data sources, and strengthen high-value data, thereby continuously optimizing system performance and improving the overall utilization rate of edge and cloud resources. Attached Figure Description

[0007] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a schematic diagram of a multimodal data acquisition and fusion system based on cloud-edge collaboration, provided as an embodiment of the present invention.

[0009] Figure 2 A schematic diagram of cloud-layer integration and adaptation interaction.

[0010] Figure 3 This is a time-series diagram of the edge layer and the cloud layer.

[0011] Figure 4 This is a schematic diagram of a multimodal data acquisition and fusion method based on cloud-edge collaboration provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0014] Figure 1 A schematic diagram of a cloud-edge collaborative multimodal data acquisition and fusion system structure is provided for an embodiment of the present invention, as shown below. Figure 1 As shown, the system includes an edge layer, a cloud layer, and a communication network layer between the two.

[0015] The edge layer includes several edge nodes, each corresponding to at least one set of multimodal terminal devices. The edge nodes have a built-in standardized model builder to convert the raw multimodal data into a standard data model that includes a metadata layer and an initial feature value layer.

[0016] The cloud layer includes a dynamic adaptation engine, a fusion strategy library, and a feedback optimization module. The dynamic adaptation engine is used to parse the standard data model to obtain target data and target parameters including acquisition accuracy, and to match the fusion algorithm and the initial fusion weights of the target device according to business requirements. The matched fusion algorithm fuses the target data according to the initial fusion weights and acquisition accuracy. The feedback optimization module is used to evaluate the accuracy of the fusion results and generate adjustment instructions for the data acquisition strategy and / or model weight parameters of the edge layer based on the accuracy evaluation results.

[0017] The communication network layer is used to implement the standard data model and adjustment command transmission between the edge layer and the meta-fault layer.

[0018] Specifically, the edge layer is deployed at the data acquisition site and consists of several edge nodes. Each edge node is connected to at least one set of multimodal terminal devices via wired or wireless means. These multimodal terminal devices may include temperature sensors, vibration sensors, high-definition cameras, industrial controllers, audio acquisition devices, etc., and are used to collect multimodal raw data generated in industrial production or urban management.

[0019] Each edge node has a built-in standardized model builder. This builder transforms the collected multimodal raw data into a unified, standardized data interaction unit, i.e., a standard data model. The standard data model is encapsulated using a two-layer structure: Metadata layer: Used to describe background information about the data, such as the device number of the data source, acquisition accuracy, millisecond-level timestamp, device type, etc. Initial feature layer: Used to store structured data after preprocessing and preliminary feature extraction, including normalized numerical values ​​and / or high-dimensional feature vectors.

[0020] By using structured encapsulation and Protocol Buffers serialization technology, the standardized model builder can compress model size, thereby improving transmission efficiency.

[0021] The cloud layer is implemented based on the high-performance server clusters and distributed storage architecture of the cloud computing center. Figure 2This diagram illustrates the cloud-layer fusion adaptation interaction. The cloud-layer's dynamic adaptation engine receives and parses the standard data model uploaded from the edge layer. Based on business needs, it dynamically matches the most suitable fusion algorithm from the fusion strategy library, which pre-stores various fusion algorithms and their applicable conditions. The cloud-layer's feedback optimization module evaluates the accuracy of the fusion results and generates optimization instructions for the edge layer based on the accuracy evaluation results. For example, when it is found that the contribution of a certain type of data in the fusion is consistently low, an instruction is generated requiring the edge layer to reduce the sampling frequency of that type of data; when a decrease in the accuracy of a data source is detected, an instruction is generated to lower its preset weight in the weight identifier layer of that data model.

[0022] The communication network layer adopts a hybrid architecture of "5G private network + edge computing private network". In a specific embodiment, the standard data model is encrypted and transmitted to the cloud layer through the 5G private network of the communication network layer; the adjustment instructions are sent to the edge layer through the edge computing private network of the communication network layer.

[0023] Figure 3 This diagram illustrates the time series between the edge layer and the cloud layer. Each node in the edge layer collects multimodal raw data, which, after preprocessing, is uniformly encapsulated into a standard data model and encrypted by a standardized model builder. This model is transmitted to the cloud layer via a 5G private network in the communication network layer. The cloud layer's dynamic adaptation engine parses the model and matches it with algorithms for fusion calculation. The feedback optimization module then generates optimization instructions based on the fusion effect. These instructions are then sent back to the edge layer via the communication network layer, guiding it to adjust its data acquisition strategy or model parameters.

[0024] Figure 4 This is a schematic diagram of a multimodal data acquisition and fusion method based on cloud-edge collaboration provided in an embodiment of the present invention. The method is based on the system implementation of the above embodiment and specifically includes the following steps.

[0025] SS1 collects multimodal raw data from each edge node in the edge layer and preprocesses it to obtain structured data. The structured data is then transformed and encapsulated into a standard data model containing a metadata layer and an initial feature value layer by a standardized model builder.

[0026] SS11 collects data and preprocesses it to generate structured data.

[0027] SS11a: Edge nodes concurrently collect raw data from multiple connected heterogeneous terminal devices according to a pre-configured adaptive sampling strategy; the adaptive sampling strategy dynamically adjusts the sampling frequency based on the data modality and service priority.

[0028] For high-frequency, high-priority sensor data, the first acquisition frequency is used to ensure that key instantaneous fault characteristics can be captured.

[0029] For mid-frequency streaming data, a second acquisition frequency is used to balance smoothness and processing load.

[0030] For low-frequency, low-priority environmental data, a third acquisition frequency is used to save computing and communication resources for edge nodes.

[0031] The first acquisition frequency > the second acquisition frequency > the third acquisition frequency, to ensure that high-value data is acquired with priority and high precision.

[0032] SS11b preprocesses the acquired multimodal data, including: for visual modal data, it uses a Gaussian filtering algorithm to reduce noise in video frames; for numerical modal data, it uses a 3σ-based outlier detection algorithm to remove outliers and linear interpolation to complete missing data; for audio modal data, it uses spectral subtraction to remove reverberation noise from the original audio signal. Numerical modal data primarily comes from physical sensors, which generate numerical signals, such as temperature sensors, vibration sensors, pressure sensors, and gyroscopes.

[0033] For visual modal data: a Gaussian filtering algorithm is applied to convolve each video frame. A suitable Gaussian kernel is selected to suppress Gaussian noise in the image and improve the image signal-to-noise ratio.

[0034] For numerical modal data: an outlier detection and handling process based on the 3σ principle is adopted. First, the mean μ and standard deviation σ of the data stream are calculated within a time window; then, all data points in the window are compared with the mean, and any data point that meets the 3σ condition is identified as an outlier and removed; finally, for the removed or missing data points, linear interpolation is used to fill in the missing data points with the adjacent valid data points to ensure the continuity of the data sequence.

[0035] For audio modal data: spectral subtraction is used for noise reduction. The algorithm first performs a short-time Fourier transform on the original audio signal to obtain its spectrum; then, it estimates and constructs a spectral model of the environmental steady-state reverberation noise; finally, it subtracts the noise spectrum from the original signal spectrum and performs an inverse transform to reconstruct the clean audio signal after removing the reverberation.

[0036] SS11c maps preprocessed modal data to a predefined feature space of the same dimension, transforming heterogeneous data into homogeneous structured data.

[0037] Specifically, the preprocessed numerical modal data is normalized and labeled to the [0,1] interval to form a one-dimensional or multi-dimensional normalized numerical vector. For the preprocessed visual modal data, a pre-trained convolutional neural network is used to extract its frame image features to form an N1-dimensional visual feature vector. For the preprocessed audio modal data, its Mel-frequency spectral coefficients are extracted to form an N2-dimensional audio feature vector. Through a projection matrix or a fully connected layer, the normalized numerical vector, the N1-dimensional visual feature vector, and the N2-dimensional audio feature vector are converted into target dimension vectors, respectively, to obtain the final normalized numerical vector, the initial visual feature vector, and the initial audio feature vector. The final normalized numerical vector, the initial visual feature vector, and the initial audio feature vector are output as components of the structured data to the standardized model builder.

[0038] For a normalized numerical vector, it is mapped to the target dimension space using a trainable projection matrix. Let Dnum be the original dimension of the normalized numerical vector, and Nm be the target dimension. If Dnum is less than Nm, a projection weight matrix of size [Dnum, Nm] is constructed. The normalized numerical vector is multiplied by this projection weight matrix, and a non-linear activation function is applied to output an Nm-dimensional vector. If Dnum equals Nm, the normalized numerical vector is directly output. If Dnum is greater than Nm, principal component analysis is first used to reduce it to Nm dimensions, or a projection matrix of size [Dnum, Nm] is used for dimensionality reduction.

[0039] For an N1-dimensional visual feature vector, it is refined and mapped to Nm dimensions through a fully connected layer. This fully connected layer has Nm output neurons and its weight matrix has a size of [N1, Nm]. The N1-dimensional visual feature vector is input into this layer, and after calculation, an Nm-dimensional vector is output.

[0040] For the N2-dimensional audio feature vector, the same mapping mechanism as for the visual modality is used, namely, mapping is performed through a fully connected layer.

[0041] SS12 generates standard data models through a standardized model builder.

[0042] This step is performed by the standardized model builder of the edge node, which encapsulates the structured data obtained from upstream preprocessing into a lightweight, self-describing, and value-assessing standard data unit to achieve efficient, semantically unified cloud-edge data interaction.

[0043] SS12a, the standardized model builder, initiates the model building process, receives structured data from the multimodal data preprocessing output, and reads the raw context information associated with the structured data.

[0044] The standardized model builder initiates a model building process. This process first receives raw context information associated with the structured data output from the multimodal data preprocessing module. This information includes at least: the unique identifier of the device that generated the data, the precise acquisition timestamp of the data packet, the device type identifier, and the acquisition accuracy determined after the device was manufactured or calibrated.

[0045] SS12b creates a metadata layer object, which is populated with raw context information in a predefined format and fields.

[0046] The builder creates a structured metadata layer object in memory, which has a predefined format and field specifications. The builder then populates the raw context information read in step SS12a, field by field: the device number is entered into the "Data Source" field; the acquisition precision is entered into the "Acquisition Precision" field; the timestamp is entered into the "Data Acquisition Timestamp" field; and the device type is entered into the "Terminal Device Type" field.

[0047] SS12c creates an initial feature layer object, stores the normalized numerical vector in the "Normalized Value" field, and stores the high-dimensional feature vector generated by the feature extraction algorithm in the "Feature Vector" field.

[0048] The builder synchronously creates an initial feature layer object to store the core content of the data, mainly including the "normalized numerical" field and the "feature vector" field.

[0049] The "Normalized Numeric" field is used to store scalars or low-dimensional vectors in structured data that have been normalized and have a clear physical meaning.

[0050] The “feature vector” field is used to store high-dimensional feature representations extracted from unstructured data.

[0051] SS12d integrates the constructed metadata layer and initial feature value layer into a complete data model.

[0052] SS12e calls the Protocol Buffers serialization protocol interface to convert the complete data model into a binary serialized byte stream.

[0053] To reduce network transmission load, the builder invokes the Protocol Buffers (Protobuf) serialization protocol interface. It transforms and encodes the complete data model integrated in step SS12d into a compact, binary serialized byte stream based on a predefined .proto structure file. During this serialization process, it leverages Protocol Buffers' inherent mechanisms such as replacing field names with field numbers, variable-length integer encoding, and omitting default or null fields, combined with a streamlined design for metadata fields, to achieve lightweight encapsulation of the model. The final standard data model is more than 40% smaller than traditional text-based encapsulation formats such as JSON or XML.

[0054] SS2 encrypts the standard data model and transmits it to the cloud layer via a communication network.

[0055] The edge security and transmission module receives a standard data model from the standardized model builder. The module then initiates a secure transmission protocol: encrypting the byte stream of the standard data model using the AES-256 symmetric encryption algorithm. The encryption process uses a pre-set or dynamically negotiated session key. Before initiating transmission, two-way authentication is performed based on the SM2 national cryptographic algorithm. The edge node uses its SM2 private key to digitally sign the transmission request or specific challenge data, and the cloud layer verifies the signature using the corresponding public key, thereby ensuring the legitimacy of both communicating parties and preventing spoofed nodes from accessing the network.

[0056] The encrypted standard data model is sent to the communication network layer for transmission. Data that needs to be uploaded to the cloud for deep integration and persistent storage is transmitted via a 5G private network.

[0057] SS3, the cloud layer receives and parses the standard data model to obtain target data and target parameters including acquisition accuracy, and matches the initial fusion weights of the fusion algorithm and the target device according to business requirements. The matched fusion algorithm fuses the target data according to the initial fusion weights and acquisition accuracy.

[0058] SS31 filters the initial feature vectors of the target device from the feature vectors in the initial feature layer according to business requirements.

[0059] SS32 matches the fusion algorithm according to business requirements and assigns initial fusion weights to the initial feature vectors of each target device according to business weight rules.

[0060] Based on the business weight rules pre-stored in the cloud layer, an initial fusion weight value is assigned to the data of each target device. The business weight rule is a preset mapping relationship based on device type, that is, different weights are assigned to data from different device types.

[0061] For example: IF Device_Type == 'Critical spindle vibration sensor' THEN Weight = 0.9; IFDevice_Type == 'General ambient temperature and humidity sensor' THEN Weight = 0.3.

[0062] SS33 takes the selected initial feature vector as input and performs data fusion using the matched fusion algorithm based on the initial fusion weights and acquisition accuracy of the corresponding target device. This includes the following matching situations and their processing procedures.

[0063] a) If the data modality is only numerical and there is no temporal dependency between the source data, a weighted average fusion algorithm will be matched. This fusion algorithm normalizes the product of the initial fusion weight and the quality adjustment factor as the effective fusion weight, and sums all the selected initial numerical feature vectors according to their effective fusion weights to obtain the final fused feature vector; where the quality adjustment factor is proportional to the acquisition accuracy.

[0064] Specifically, for the initial numerical feature vectors of multiple selected target devices, this algorithm does not directly use the normalized values ​​after edge preprocessing, but instead fuses them using vectors as units.

[0065] An effective fusion weight is assigned to the feature vector of each target device. This effective weight is calculated from two parts: (1) the initial fusion weight extracted from the weight identifier layer; and (2) the quality adjustment factor based on the device acquisition accuracy in the metadata layer. The effective weight is the product of the initial fusion weight and the quality adjustment factor, and is then normalized.

[0066] In one specific way, the quality adjustment factor = adjustment base + (1 - adjustment base) * (current acquisition accuracy value / standard acquisition accuracy benchmark value).

[0067] The initial numerical feature vectors of all target devices are weighted and summed according to their corresponding effective fusion weights to obtain the final fused feature vector.

[0068] b) If the data modality is only numerical and there is a time-series dependency between the source data, the LSTM temporal modeling and multi-source feature alignment algorithm will be matched. This fusion algorithm calculates the source importance coefficient based on the initial fusion weight and acquisition accuracy. The hidden state output of the LSTM network outputs the features of each data source under the same time series. Then, the attention layer dynamically calculates the attention distribution based on the source importance coefficient of each data source feature at each time point and outputs the fused temporal feature vector.

[0069] Specifically, this algorithm receives initial numerical feature vectors from multiple data sources arranged in a time series.

[0070] Before inputting the temporal feature vectors of each data source into the LSTM network, a source importance coefficient is calculated based on the initial fusion weights and acquisition accuracy. This coefficient is used to initialize the bias terms of the corresponding data source input gates in the LSTM network, so that the network pays more attention to high-weight, high-precision data sources in the early stages of training and inference.

[0071] One specific approach is: Source importance coefficient = Initial fusion weight * [Weight preference coefficient + (1 - Weight preference coefficient) * f (Standard acquisition accuracy baseline value, current acquisition accuracy value)].

[0072] f (standard acquisition accuracy baseline value, current acquisition accuracy value) is the accuracy quality function.

[0073] One specific method is f(standard acquisition accuracy reference value, current acquisition accuracy value) = exp(-sensitivity coefficient * (absolute value of the difference between the standard acquisition accuracy reference value and the current acquisition accuracy value) / standard acquisition accuracy reference value).

[0074] The specific implementation of initializing the bias term of the corresponding data source input gate in the LSTM network is as follows: initial value of bias term = original bias term + log (source importance coefficient).

[0075] The LSTM network outputs the features of each data source in a unified time series from its hidden state. Subsequently, an attention layer dynamically calculates the attention distribution based on the source importance coefficients of the features of each data source at each time step, thereby focusing on key sources and key time steps, and finally outputting the fused temporal feature vector.

[0076] c) If the data modality is only visual, then a CNN feature extraction and attention mechanism fusion algorithm will be matched. This fusion algorithm uses the initial fusion weights as prior knowledge to participate in the calculation of the query vector, and the acquisition accuracy is used to scale the key vector. The attention mechanism calculates a set of attention weights based on the processed query vector and key vector, and performs weighted fusion on the high-level feature vectors extracted by the deep CNN to output the final visual fusion feature.

[0077] Specifically, this algorithm differs from the feature extraction of lightweight CNNs at the edge. The cloud utilizes a deeper CNN network to perform deep feature extraction on the initial visual feature vector uploaded from the edge, thereby obtaining a high-level visual feature vector.

[0078] In the attention mechanism, the initial fusion weights are used as prior knowledge to participate in the calculation of the query vector; while the acquisition precision is used to scale the key vector, and high-precision features will gain higher importance in attention matching.

[0079] One specific implementation is: Query vector = High-level visual feature vector + Query weight matrix * log(initial fusion weight). The log(initial fusion weight) is a vector with the same dimension as the high-level visual feature vector, where each element is the logarithm of the initial fusion weight. Higher initial weights increase the value of the query vector, making the source feature more proactive in subsequent attention matching.

[0080] One specific implementation method is to use a scaling factor to quantize the scaling key vector for acquisition precision. The scaling factor is calculated as: scaling sensitivity coefficient * (1 - (current acquisition precision value / standard acquisition precision baseline value))).

[0081] The attention mechanism calculates a set of attention weights based on the processed query vector and key vector, and performs weighted fusion on the high-level feature vectors extracted by the deep CNN to output the final visual fusion feature.

[0082] d) If the data modality is only audio, an acoustic feature analysis and recognition algorithm will be matched. This algorithm uses the acquisition accuracy to adjust the original probability output by the classifier to obtain the adjusted probability. During fusion, a CNN is used to extract acoustic temporal features related to the business scenario. Then, the adjusted probability is used as the independent recognition result of the audio source in the current fusion cycle and its credibility to participate in decision fusion. During decision fusion, the effective voting weight is calculated based on the maximum value of the initial fusion weight and the adjusted probability, and decision fusion is performed based on the effective voting weight.

[0083] Specifically, a lightweight, pre-trained convolutional neural network or recurrent neural network is used as the acoustic event classifier. This network is distinguished from the feature extractor at the edge: the edge CNN aims to extract general time-frequency features, while the CNN / RNN in the cloud is a classification model specifically trained for a particular business scenario. Its purpose is to further extract high-level abstract features related to specific events such as "bearing failure noise" and "gas leak sound" from the initial audio feature vector and output an event probability vector. Each element in the event probability vector represents the probability that the audio segment belongs to a certain preset event category.

[0084] Data acquisition precision is used to adjust the raw probabilities output by the classifier, serving as prior knowledge of the data source's reliability. This is specifically implemented through a precision-confidence mapping function. Adjusted probability = Original probability * (1 + α * ((Standard acquisition accuracy baseline value / Current acquisition accuracy value - 1))) Here, α is an adjustment coefficient (0 < α ≤ 1), and the baseline acquisition accuracy is the accuracy value of this type of audio device under ideal conditions. The higher the current acquisition accuracy, the closer the adjusted probability will be to or greater than the original probability; the lower the current acquisition accuracy, the more significantly the adjusted probability will be suppressed.

[0085] The adjusted probability will serve as the independent recognition result and its reliability of the audio source within the current fusion cycle, directly participating in subsequent decision fusion to ensure that the recognition result of the high-precision audio source occupies a more important position in the system.

[0086] The initial fusion weights directly affect the voting weights in the decision fusion. For an audio source, the formula for calculating its effective voting weight is: Effective voting weight = Initial fusion weight * Maximum adjusted probability A weighted voting method is used for the final decision. The system iterates through all audio sources, casting a vote for the event category corresponding to the maximum adjusted probability for each source. The weight of this vote is the calculated effective voting weight. Finally, the system calculates the total voting weight for all event categories and determines the event category with the highest total weight as the final fusion recognition result.

[0087] e) If the data modality contains at least two different modalities and there is a temporal correlation between the data of each modality, the LSTM temporal modeling and cross-modal feature alignment algorithm will be matched. The algorithm calculates the preset fusion weight and preset acquisition precision of the data modality according to the initial fusion weight and acquisition precision of the target device with the same data modality. Based on the preset fusion weight and preset acquisition precision, the dynamic alignment weight of the corresponding data modality is calculated. The LSTM network processes the temporal features of each modality respectively. The cross-modal alignment layer uses the dynamic alignment weight to project the LSTM output features of different modalities onto a unified subspace and perform weighted combination to output the multimodal fusion result based on the temporal context.

[0088] Specifically, this algorithm receives temporal initial feature vectors from different modalities (such as numerical, visual, and audio). In the cross-modal feature alignment layer, a dynamic alignment weight is calculated for each modality. The initial value of this weight is determined by the initial fusion weight of that modality and the acquisition accuracy. The LSTM network processes the temporal features of each modality separately, while the cross-modal alignment layer uses the dynamic alignment weight to project the LSTM output features of different modalities onto a unified subspace and perform weighted combination, achieving multimodal deep fusion based on temporal context.

[0089] One specific implementation method is data modality. The preset fusion weight is The initial fusion weights of all N target devices in this mode are calculated by weighted averaging, and the calculation formula is as follows:

[0090] in, It belongs to the modal The Initial fusion weights for each target device For the first The current acquisition accuracy value of each target device. This is a precision weighting function used to give higher-precision devices a greater weight during aggregation. This function can be defined as follows: ,in It is a small positive constant. Because A smaller value indicates higher precision. This function ensures that high-precision devices output larger values.

[0091] Data modal The preset acquisition accuracy is It is obtained by calculating the harmonic mean of the acquisition accuracy of all target devices under this mode, and its calculation formula is as follows:

[0092] Data modal Dynamic alignment weights The preset fusion weights are and preset acquisition accuracy The decision is made jointly, and the calculation formula is as follows:

[0093] Where M is the total number of modalities participating in the fusion, and exp() is the Softmax function, which normalizes the original scores of all modalities into a probability distribution to ensure that the sum of the dynamic alignment weights of all modalities is 1.

[0094] f) If the data modality contains at least two different modalities and there is no temporal correlation between the data of each modality, a cross-modal joint feature fusion algorithm based on weighting or concatenation will be matched. This algorithm calculates the preset fusion weight and preset acquisition precision of the data modality according to the initial fusion weight and acquisition precision of the target device with the same data modality. It calculates the quality coefficient according to the preset acquisition precision, which is positively correlated with the quality coefficient. It calculates the feature scaling factor of the corresponding modality according to the preset fusion weight and quality coefficient. It uses the feature scaling factor to scale the initial feature vector of the corresponding modality. The scaled initial feature vectors are concatenated to form a unified joint feature vector. The joint feature vector is subjected to nonlinear transformation and dimensionality reduction in a fully connected network to output the final cross-modal fusion feature.

[0095] Specifically, this algorithm processes initial feature vectors from different modalities within the same time slice. Before feature concatenation or joining, the feature vector of each modality is first scaled. The scaling factor is determined by the initial fusion weights of that modality and the quality coefficient based on the acquisition accuracy. The scaled feature vectors from each modality are then concatenated to form a unified joint feature vector. Subsequently, this joint feature vector is input into a fully connected network for nonlinear transformation and dimensionality reduction, outputting the final cross-modal fusion feature.

[0096] The calculation of preset fusion weights and preset acquisition accuracy is the same as above, and will not be repeated here.

[0097] One specific implementation method is data modality. The quality coefficient is This is used to quantify the overall reliability of the modal data, and its value is related to the preset acquisition accuracy. Positive correlation. Given that A smaller value indicates higher absolute precision. To achieve "positive correlation," the quality coefficient... The calculation formula is:

[0098] in, This is a quality attenuation coefficient greater than zero, used to control the sensitivity of the acquisition accuracy to the influence of the quality coefficient.

[0099] Data modal Feature scaling factor The preset fusion weights of this mode and quality coefficient This is jointly determined and used to adjust the magnitude of the feature vector of this modality before feature concatenation. The calculation formula is as follows:

[0100] The initial eigenvector of this mode Scaling is represented as The scaled feature vectors of each modality are concatenated to form a unified joint feature vector.

[0101] SS4 performs accuracy evaluation on the fusion results and generates data acquisition strategies for the edge layers and / or adjustment instructions for model weight parameters based on the accuracy evaluation results.

[0102] First, calculate the accuracy evaluation indicators of the fusion results. The accuracy evaluation indicators include the target information recognition accuracy of the business feedback data, the fusion deviation rate based on the benchmark data, and the average contribution of each modality in the fusion process. Then, compare the calculated evaluation indicators with the corresponding preset thresholds to determine whether an adjustment instruction needs to be generated, and generate the corresponding adjustment instruction based on the comparison results.

[0103] In one specific embodiment, calculating the target information recognition accuracy of business feedback data includes: receiving feedback data from a business application system, the feedback data containing the true labels of the target information in the fusion results; within an evaluation period, based on the feedback data, counting the total number of verified fusion results and the number of correctly identified results, and using the percentage of correctly identified results out of the total number of results as the target information recognition accuracy.

[0104] Specifically, an evaluation cycle is calculated based on feedback data. within, share The fusion results were verified, with the number of correctly identified individuals being [number missing]. The target information recognition accuracy rate for,

[0105] The calculation of the fusion deviation rate based on the benchmark data specifically includes: within an evaluation period, calculating the average percentage of the absolute deviation between the fusion result of each data point and the corresponding benchmark data relative to the benchmark data, and using this average percentage as the fusion deviation rate.

[0106] Specifically, within an evaluation cycle within, share For each data point, the fusion bias rate is... for,

[0107] in, For the first The fusion result of individual data points For the first The baseline data for each data point.

[0108] The calculation of the average contribution of each mode during the fusion process includes: recording the effective weight of each mode at each fusion moment during the running of the fusion algorithm; calculating the arithmetic mean of all effective weights of the mode within an evaluation period, and using the arithmetic mean as the average contribution of the mode.

[0109] Specifically, during the execution of the fusion algorithm, each modality is recorded. Effective weights in the final result An evaluation cycle is calculated using the following formula. Average contribution within :

[0110] For the weighted average fusion algorithm, the effective weights are the preset weights, and the modalities... Effective weights of the data source It is directly equal to the preset weight extracted from the weight identifier layer.

[0111] For CNN-attention mechanism fusion algorithms, the effective weights are derived from the attention weights calculated by the attention mechanism. For video modalities, at time step... The algorithm will generate an attention weight vector. The effective weight of this mode in this fusion. It can be taken as the average or maximum value of the attention weights at all time steps in this batch of data, expressed as:

[0112] in, It is the first The average of the attention weights for all spatial locations in the frame. That is the total number of frames.

[0113] For the LSTM and cross-modal alignment fusion algorithm, the effective weights are derived from the dynamic weights calculated during cross-modal alignment. Modality At time step The effective weight is directly equal to the alignment weight at that moment. For the effective weight within an evaluation period, its average value over the time series can be taken, denoted as:

[0114] The calculated evaluation indicators are compared with the corresponding preset thresholds to determine whether an adjustment instruction needs to be generated, and the corresponding adjustment instruction is generated based on the comparison results, including the following situations.

[0115] 1) When the average contribution is lower than the contribution threshold, a data acquisition strategy adjustment instruction is generated to reduce the sampling frequency of the corresponding modality data.

[0116] In one specific embodiment, when the average contribution of a certain mode is lower than the contribution threshold for multiple consecutive evaluation periods, a sampling frequency reduction instruction for that mode is generated. Let the current sampling frequency be... Then the new sampling frequency Adjustments can be made according to the following rules:

[0117] in, This is the minimum guaranteed frequency set for this mode. This is a downward adjustment factor.

[0118] 2) When the target information recognition accuracy is lower than its accuracy threshold and / or the fusion deviation rate is higher than its deviation rate threshold, and the data acquisition accuracy of the data source is detected to be deteriorated, a model weight parameter adjustment instruction is generated to reduce its preset weight in the weight identification layer.

[0119] Here, "deterioration in acquisition accuracy" refers to the deviation of the current acquisition accuracy value from the standard accuracy value exceeding the degradation tolerance threshold. Specifically, it refers to the current acquisition accuracy value of a data source parsed from the metadata layer of the standard data model. Compared to its standard precision value Its error range exceeded the preset degradation tolerance threshold. , is represented as ,

[0120] For example, the standard accuracy of a temperature sensor ±0.1°C. The system is set with a degradation tolerance threshold. The temperature is 0.15°C. If its current precision is parsed from the metadata... If the temperature changes to ±0.3°C, then |0.3 -0.1| = 0.2 > 0.15, which is considered a significant degradation.

[0121] Generate a weight correction instruction for this data source in the weight identification layer. Assume its current preset weight is... The corrected weights for:

[0122] in, For initial or standard acquisition accuracy, Based on the current level of accuracy, It is a decay factor based on the overall contribution.

[0123] 3) When the target information recognition accuracy is lower than its accuracy threshold, generate a model weight parameter adjustment instruction to increase the preset weight of the data source associated with the target information.

[0124] Inaccurate global output indicates that the current data weight allocation strategy is unable to effectively identify key information and thus generate model weight enhancement instructions. For data sources that are defined as "key information" in the business context, their preset weights in the weight identification layer should be increased. To ensure that resources are allocated towards it, as expressed as:

[0125] in, For the enhancement coefficient, As the new weight.

[0126] 4) When the fusion deviation rate is higher than its deviation rate threshold, generate a data acquisition strategy adjustment instruction to increase the sampling frequency of high-weight data sources and / or enhance the intensity of data preprocessing.

[0127] If the fusion deviation rate is higher than its deviation rate threshold, it indicates that the overall quality or reliability of the input data is insufficient. A data quality enhancement instruction is generated. At this time, the sampling frequency of the top K data sources with the highest preset weight can be temporarily increased to obtain more dense effective data, and / or the edge preprocessing module can be instructed to use stronger filtering and denoising parameters, such as reducing the variance of the Gaussian filter kernel and reducing the σ multiple in the 3σ rule.

[0128] SS5 will send adjustment commands to the edge layer via the communication network.

[0129] After generating adjustment instructions based on the accuracy assessment results, the feedback optimization module standardizes and encapsulates these instructions. Once encapsulated, the cloud security module signs the instruction packets using a security protocol agreed upon with the edge endpoint, ensuring the integrity and immutability of the instructions during transmission and preventing unauthorized injection. The encrypted and signed adjustment instruction packets are then distributed through the communication network layer. The edge security and transmission module of the target edge node receives the distributed instruction packets and performs security verification, instruction parsing, and parameter updates.

[0130] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal data acquisition and fusion method based on cloud-edge collaboration, characterized in that, Includes the following steps: Each edge node in the edge layer collects multimodal raw data and preprocesses it to obtain structured data. The structured data is then transformed and encapsulated into a standard data model that includes a metadata layer and an initial feature value layer. The metadata layer contains the acquisition accuracy of the device. The standard data model is encrypted and then transmitted to the cloud layer via a communication network; The cloud layer receives and parses the standard data model to obtain target data and target parameters including acquisition accuracy, and matches the initial fusion weights of the fusion algorithm and the target device according to business requirements. The matched fusion algorithm fuses the target data according to the initial fusion weights and acquisition accuracy. The accuracy of the fusion results is evaluated, and data acquisition strategies and / or model weight parameter adjustment instructions for the edge layer are generated based on the accuracy evaluation results. The adjustment instructions are sent to the edge layer via the communication network.

2. The method according to claim 1, characterized in that, Each edge node in the edge layer collects multimodal raw data and preprocesses it to obtain structured data, specifically including: Edge nodes concurrently collect raw data from multiple connected heterogeneous terminal devices according to a pre-configured adaptive sampling strategy; the adaptive sampling strategy dynamically adjusts the sampling frequency based on the data modality and service priority. The collected multimodal data is preprocessed, including: for visual modal data, Gaussian filtering algorithm is used to denoise video frames; for numerical modal data, outlier detection algorithm based on 3σ principle is used to remove outlier data, and linear interpolation is used to complete missing data; for audio modal data, spectral subtraction is used to subtract reverberation noise from the original audio signal. The preprocessed modal data are uniformly mapped to a feature space of the same preset dimension to transform heterogeneous data into homogeneous structured data.

3. The method according to claim 2, characterized in that, The preprocessed modal data are uniformly mapped to a predefined feature space of the same dimension to transform heterogeneous data into homogeneous structured data. Specifically, this includes: The preprocessed numerical modal data is normalized and calibrated to the [0,1] interval to form a normalized numerical vector; For the preprocessed visual modality data, a pre-trained convolutional neural network is used to extract its frame image features, forming an N1-dimensional visual feature vector. For the preprocessed audio modal data, the time-frequency features of its Mel spectrum coefficients are extracted to form an N2-dimensional audio feature vector; By using a projection matrix or a fully connected layer, the normalized numerical vector, the N1-dimensional visual feature vector, and the N2-dimensional audio feature vector are converted into target dimension vectors, respectively, to obtain the final initial numerical feature vector, initial visual feature vector, and initial audio feature vector. The final initial numerical feature vector, initial visual feature vector, and initial audio feature vector are output as components of the structured data to the standardized model builder.

4. The method according to claim 3, characterized in that, The standardized model builder transforms and encapsulates structured data into a standard data model that includes a metadata layer and an initial feature value layer, specifically including: The standardized model builder initiates the model building process, receives structured data from the preprocessing of multimodal data, and reads the raw context information associated with the structured data. Create a metadata layer object and populate it with the raw context information according to a predefined format and fields; the raw context information includes device number, acquisition precision, acquisition timestamp, and device type; Create an initial feature value layer object, store the initial numerical feature vector in the "normalized value" field, and store the initial feature vector generated by the feature extraction algorithm in the "feature vector" field; The constructed metadata layer and initial feature value layer are integrated into a complete data model; Call the Protocol Buffers serialization protocol interface to convert the complete data model into a binary serialized byte stream.

5. The method according to claim 4, characterized in that, The cloud layer receives and parses the standard data model to obtain target data and target parameters including acquisition accuracy. Based on business requirements, it matches the fusion algorithm with the initial fusion weights of the target device. The matched fusion algorithm fuses the target data according to the initial fusion weights and acquisition accuracy, specifically including: The standard data model is decrypted and deserialized to extract the original context information in the metadata layer and the feature vector in the initial feature layer; Based on business requirements, the initial feature vectors of the target device are selected from the feature vectors in the initial feature layer; The fusion algorithm is matched according to business requirements, and the initial fusion weights are assigned to the initial feature vectors of each target device according to business weight rules. The selected initial feature vectors are used as input, and data fusion is performed using the matched fusion algorithm based on the initial fusion weights and acquisition accuracy of the corresponding target device. This includes... a) If the data modality is only numerical and there is no temporal dependency between the source data, a weighted average fusion algorithm will be matched. This fusion algorithm normalizes the product of the initial fusion weight and the quality adjustment factor as the effective fusion weight, and sums all the selected initial numerical feature vectors according to their effective fusion weights to obtain the final fused feature vector; where the quality adjustment factor is proportional to the acquisition accuracy. b) If the data modality is only numerical and there is a time-series dependency between the source data, the LSTM temporal modeling and multi-source feature alignment algorithm will be matched. This fusion algorithm calculates the source importance coefficient based on the initial fusion weight and acquisition accuracy. The hidden state output of the LSTM network outputs the features of each data source under the same time series. Then the attention layer dynamically calculates the attention distribution based on the source importance coefficient of each data source feature at each time point and outputs the fused temporal feature vector. c) If the data modality is only visual, then the CNN feature extraction and attention mechanism fusion algorithm will be matched. This fusion algorithm uses the initial fusion weights as prior knowledge to participate in the calculation of the query vector, and the acquisition accuracy is used to scale the key vector. The attention mechanism calculates a set of attention weights based on the processed query vector and key vector, and performs weighted fusion on the high-level feature vectors extracted by the deep CNN to output the final visual fusion feature. d) If the data modality is only audio, an acoustic feature analysis and recognition algorithm will be matched. This algorithm uses the acquisition accuracy to adjust the original probability output by the classifier to obtain the adjusted probability. During fusion, a CNN is used to extract acoustic temporal features related to the business scenario. Then, the adjusted probability is used as the independent recognition result of the audio source in the current fusion cycle and its credibility to participate in decision fusion. During decision fusion, the effective voting weight is calculated based on the maximum value of the initial fusion weight and the adjusted probability, and decision fusion is performed based on the effective voting weight. e) If the data modality contains at least two different modalities and there is a temporal correlation between the data of each modality, the LSTM temporal modeling and cross-modal feature alignment algorithm will be matched. The algorithm calculates the preset fusion weight and preset acquisition precision of the data modality according to the initial fusion weight and acquisition precision of the target device with the same data modality. Based on the preset fusion weight and preset acquisition precision, the dynamic alignment weight of the corresponding data modality is calculated. The LSTM network processes the temporal features of each modality respectively. The cross-modal alignment layer uses the dynamic alignment weight to project the LSTM output features of different modalities onto a unified subspace and perform weighted combination to output the multimodal fusion result based on the temporal context. f) If the data modality contains at least two different modalities and there is no temporal correlation between the data of each modality, a cross-modal joint feature fusion algorithm based on weighting or concatenation will be matched. This algorithm calculates the preset fusion weight and preset acquisition precision of the data modality according to the initial fusion weight and acquisition precision of the target device with the same data modality. It calculates the quality coefficient according to the preset acquisition precision, which is positively correlated with the quality coefficient. It calculates the feature scaling factor of the corresponding modality according to the preset fusion weight and quality coefficient. It uses the feature scaling factor to scale the initial feature vector of the corresponding modality. The scaled initial feature vectors are concatenated to form a unified joint feature vector. The joint feature vector is subjected to nonlinear transformation and dimensionality reduction in a fully connected network to output the final cross-modal fusion feature.

6. The method according to claim 5, characterized in that, The accuracy of the fusion results is evaluated, and based on the accuracy evaluation results, data acquisition strategies for the edge layers and / or adjustment instructions for model weight parameters are generated, specifically including: The accuracy evaluation indicators of the fusion results are calculated. These indicators include the target information recognition accuracy of the business feedback data, the fusion deviation rate based on the benchmark data, and the average contribution of each modality in the fusion process. The calculated evaluation indicators are compared with the corresponding preset thresholds to determine whether an adjustment instruction needs to be generated, and the corresponding adjustment instruction is generated based on the comparison results.

7. The method according to claim 6, characterized in that, The calculation of the target information recognition accuracy of the business feedback data specifically includes: receiving feedback data from the business application system, which contains the real labels of the target information in the fusion result; within an evaluation period, based on the feedback data, counting the total number of verified fusion results and the number of times they are correctly identified, and using the percentage of the number of correctly identified results to the total number of results as the target information recognition accuracy; The calculation of the fusion deviation rate based on the benchmark data specifically includes: within an evaluation period, calculating the average percentage of the absolute deviation between the fusion result of each data point and the corresponding benchmark data relative to the benchmark data, and using this average percentage as the fusion deviation rate; The calculation of the average contribution of each mode during the fusion process includes: recording the effective weight of each mode at each fusion moment during the running of the fusion algorithm; calculating the arithmetic mean of all effective weights of the mode within an evaluation period, and using the arithmetic mean as the average contribution of the mode.

8. The method according to claim 7, characterized in that, The calculated evaluation indicators are compared with the corresponding preset thresholds to determine whether an adjustment instruction needs to be generated, and the corresponding adjustment instruction is generated based on the comparison result, specifically including: When the average contribution is lower than the contribution threshold, a data acquisition strategy adjustment instruction is generated to reduce the sampling frequency of the corresponding modality data; When the target information recognition accuracy is lower than its accuracy threshold and / or the fusion deviation rate is higher than its deviation rate threshold, and the data source acquisition accuracy is detected to be degraded, a model weight parameter adjustment instruction is generated to reduce its preset weight in the weight identification layer; whereby, acquisition accuracy degraded means that the deviation of the current acquisition accuracy value from the standard accuracy value exceeds the degradation tolerance threshold. When the target information recognition accuracy is lower than its accuracy threshold, a model weight parameter adjustment instruction is generated to increase the preset weight of the data source associated with the target information. When the fusion bias rate exceeds its bias rate threshold, a data acquisition strategy adjustment instruction is generated to increase the sampling frequency of high-weight data sources and / or enhance the intensity of data preprocessing.

9. The method according to claim 2, characterized in that, The method specifically includes: After encrypting the standard data model, it is transmitted to the cloud layer via the 5G private network of the communication network layer; The adjustment instructions are sent to the edge layer via the edge computing private network of the communication network layer.

10. A multimodal data acquisition and fusion system based on cloud-edge collaboration, characterized in that, The system is used to implement the method of claim 1, including an edge layer, a cloud layer, and a communication network layer between the two; The edge layer includes several edge nodes, each edge node corresponds to at least one set of multimodal terminal devices, and the edge nodes have a built-in standardized model builder to convert the raw multimodal data into a standard data model that includes a metadata layer and an initial feature value layer. The cloud layer includes a dynamic adaptation engine, a fusion strategy library, and a feedback optimization module; The dynamic adaptation engine is used to parse the standard data model to obtain target data and target parameters including acquisition accuracy, and to match the initial fusion weight of the fusion algorithm and the target device according to business requirements. The matched fusion algorithm fuses the target data according to the initial fusion weight and acquisition accuracy. The feedback optimization module is used to evaluate the accuracy of the fusion results and generate adjustment instructions for the data acquisition strategy and / or model weight parameters of the edge layer based on the accuracy evaluation results. The communication network layer is used to implement the standard data model and adjustment command transmission between the edge layer and the meta-fault layer.

Images