A cloud edge-end data fusion processing method and system based on deep learning

Abstract

The application discloses a cloud edge-end data fusion processing method and system based on deep learning, a preset cloud edge-end data fusion model; a plurality of source heterogeneous data are acquired through a terminal device and are processed to obtain a plurality of same-source heterogeneous data; the cloud edge-end data fusion model is trained, then the plurality of same-source heterogeneous data are input into an edge computing platform in the fusion model, after processing, a plurality of high-level fusion feature vectors and a plurality of edge decisions are obtained, the plurality of edge decisions are fed back to the terminal device and are fed forward to a cloud computing center in the fusion model; the plurality of high-level fusion feature vectors are input into the cloud computing center, after processing by a cloud computing center model, a cloud decision is obtained, and the cloud decision is fed back to the terminal device and the edge computing platform. The method can not only effectively solve the communication and calculation problems caused by the plurality of source heterogeneous data, but also improve the overall decision and scheduling capability of the system, and promote the comprehensive development of the Internet of Things.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cloud edge-end data fusion, and in particular to a cloud edge-end data fusion processing method and system based on deep learning. BACKGROUND

[0002] With the advent of the Internet of Everything era, computing demand has experienced explosive growth. Traditional cloud computing architecture has been unable to meet the massive data computing demand brought about by the explosive growth of Internet traffic and geometric growth of data, and the traditional architecture of cloud computing is slowing down. And with the continuous improvement of the computing and storage capabilities of edge computing platforms and terminal devices, sinking cloud computing tasks to the edge side and device side and creating a cloud edge-end collaborative architecture will be an important development trend in the future.

[0003] Traditional cloud computing is built and operated in a whole collection manner, and customers need to use cloud computing resources through the Internet. With the development of cloud computing technology and the popularity of application programs, cloud computing has the following limitations: first, for the case of too much front-end collected data, the traditional data uploading method is high in cost and low in efficiency; second, for real-time interactive scenarios, all data and computing are concentrated in the computing center, which is high in information transmission cost and high in delay; third, for high stability and continuity application requirements, it is strongly dependent on stable cloud services and networks, which reduces the robustness and reliability of application requirements.

[0004] Therefore, edge computing has entered the field of view of major Internet and cloud service providers. Edge computing mainly processes data near the network edge and data source. The edge platform integrates computing, storage, transmission and self-management together. The real-time response characteristics will greatly improve the efficiency of data collection and advanced applications.

[0005] Currently, the cloud edge-end integrated collaborative computing system is in its infancy. Unlike traditional cloud computing centralized multi-source heterogeneous data fusion, cloud edge-end collaborative distributed multi-source heterogeneous data fusion will face many challenges. Among them, how to effectively allocate and utilize computing resources at each level, improve data processing speed and capacity, effectively train distributed deep learning models, and complete the fusion of multi-source heterogeneous data and information are problems that need to be solved urgently. SUMMARY

[0006] The present application designs a cloud edge-end data fusion processing model and method based on deep learning to complete the efficient processing and effective fusion of multi-source heterogeneous data in cloud edge-end collaborative integration.

[0007] A cloud edge-end data fusion processing method based on deep learning, the method comprising:

[0008] S1, build a cloud-edge-terminal data fusion model, the cloud-edge-terminal data fusion model includes a plurality of terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, a cloud computing center model, the edge computing platform model is arranged on the edge computing platform, the cloud computing center model is arranged on the cloud computing center, the plurality of terminal devices and the edge computing platform are connected, the edge computing platform and the cloud computing center are connected, and the cloud computing center and the plurality of terminal devices are connected;

[0009] S2, after obtaining the multi-source heterogeneous data by using the plurality of terminal devices, the multi-source heterogeneous data is preprocessed to obtain a plurality of homogeneous heterogeneous data;

[0010] S3, the plurality of homogeneous heterogeneous data is used as a training set to train the cloud-edge-terminal data fusion model, parameters are updated through back propagation until the trained cloud-edge-terminal data fusion model is obtained, and after the training of the cloud-edge-terminal data fusion model is completed, the plurality of homogeneous heterogeneous data collected and processed by the plurality of terminal devices is input into the trained cloud-edge-terminal data fusion model as a data set;

[0011] S4, after the edge computing platform in the trained cloud-edge-terminal data fusion model receives the plurality of homogeneous heterogeneous data in the data set, the plurality of homogeneous heterogeneous data is processed through the edge computing platform model to obtain a plurality of high-level fusion feature vectors and a plurality of edge decisions, the plurality of high-level fusion feature vectors are transmitted to the cloud computing center, the plurality of edge decisions are fed back to the plurality of terminal devices to control the plurality of terminal devices to collect the multi-source heterogeneous data, and the plurality of edge decisions are fed forward to the cloud computing center;

[0012] S5, the cloud computing center in the trained cloud-edge-terminal data fusion model receives the plurality of high-level fusion feature vectors, processes the plurality of high-level fusion feature vectors through the cloud computing center model to obtain a cloud decision, controls the cloud decision of the cloud computing center according to the plurality of edge decisions fed forward, feeds back the cloud decision to the plurality of terminal devices to control the plurality of terminal devices to collect the multi-source heterogeneous data, and feeds back the cloud decision to the edge computing platform to control the edge decision of the edge computing platform.

[0013] Preferably, after the plurality of terminal devices obtains the multi-source heterogeneous data in S2, the multi-source heterogeneous data is preprocessed to obtain a plurality of homogeneous heterogeneous data, and the specific process includes:

[0014] S21, the plurality of terminal devices obtains the multi-source heterogeneous data;

[0015] S22, the multi-source heterogeneous data is calibrated into multi-source heterogeneous data with a synchronous frequency through time stamp sampling and linear interpolation;

[0016] S23, the multi-source heterogeneous data with the synchronous frequency is standardized to obtain a plurality of homogeneous heterogeneous data.

[0017] Preferably, the plurality of isomeric data is obtained in S23, and the specific formula is:

[0018]

[0019]

[0020]

[0021] wherein x ij represents the jth isomeric data in the ith isomeric data, x ij represents the jth isomeric data before standardization in the ith isomeric data, j = 1, 2, …, N, N represents the total number of the ith isomeric data, μ i , δ i represent the mean and variance of the ith isomeric data as a whole.

[0022] Preferably, the edge computing platform model includes a plurality of parallelly arranged computing node models, and after the edge computing platform in the cloud-edge-end data fusion model trained in S4 receives the plurality of isomeric data in the data set, the edge computing platform model is used for processing to obtain a plurality of high-level fusion feature vectors and a plurality of edge decisions, which specifically includes:

[0023] S41, the plurality of isomeric data in the data set is stored in a plurality of computing node models, and one computing node model stores one isomeric data;

[0024] S42, the computing node model includes a neural network, a first multilayer perceptron, and a first Softmax classifier, the neural network is used for feature extraction on the isomeric data stored in the corresponding computing node model to obtain a primary fusion feature vector, and the first multilayer perceptron is used for feature-level fusion on the primary fusion feature vector to obtain a high-level fusion feature vector;

[0025] S43, the first Softmax classifier is used for classifying the high-level fusion feature vector to obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector;

[0026] S44, the decision probability of the preliminary decision is calculated according to a normalization method, and the preliminary decision with the maximum decision probability value is selected as the edge decision output by the corresponding computing node model;

[0027] S45, the plurality of isomeric data is traversed, and steps S42-S44 are repeated to obtain a plurality of high-level fusion feature vectors and a plurality of edge decisions.

[0028] Preferably, in S42, the first multilayer perceptron is used for feature-level fusion on the primary fusion feature vector to obtain a high-level fusion feature vector, and the specific formula is:

[0029]

[0030] wherein,

[0031] wherein, F i represents the high-level fusion feature vector of the i-th isomorphic and heterogeneous data, F i represents the primary fusion feature vector of the i-th isomorphic and heterogeneous data, W l represents the l-th learnable feature mapping matrix, 1≤l≤L, L represents the total number of feature mapping matrices, f ij t represents the primary fusion feature output after the t-th cycle of the i-th j-th isomorphic and heterogeneous data, N represents the total number of the i-th isomorphic and heterogeneous data.

[0032] Preferably, the cloud computing center in the cloud edge-end data fusion model trained in S5 receives a plurality of high-level fusion feature vectors, processes them through a cloud computing center model to obtain a cloud decision, specifically including:

[0033] S51, the cloud computing center model includes a transformer network, a second multi-layer perceptron and a second Softmax classifier, the transformer network is used for same dimension coding and feature fusion of the plurality of high-level fusion feature vectors, and a plurality of decision-level fusion feature vectors are correspondingly obtained, the second multi-layer perceptron re-fuses the plurality of decision-level fusion feature vectors to obtain a re-fusion feature vector, and the second Softmax classifier classifies the re-fusion feature vector to obtain a final decision corresponding to each feature in the re-fusion feature vector;

[0034] S52, the decision probability of the final decision is calculated according to a normalization method, and the final decision with the maximum decision probability value is selected as the cloud decision.

[0035] Preferably, the transformer network in S51 is used for same dimension coding and feature fusion of the plurality of high-level fusion feature vectors, and a plurality of decision-level fusion feature vectors are correspondingly obtained, and the specific formula is:

[0036]

[0037] wherein, R i represents the decision-level fusion feature vector corresponding to the i-th isomorphic and heterogeneous data, F oi , F pi , F qi represents the high-level fusion feature vector corresponding to the i-th isomorphic and heterogeneous data after same dimension coding, ReLU represents an activation function, d represents a feature dimension, W1 and W2 represent learnable feature mapping matrices, and b1 and b2 represent learnable offset matrices.

[0038] Preferably, the edge computing platform model comprises a plurality of parallelly arranged computing node models, the edge computing platform comprises a plurality of parallelly arranged computing nodes, each computing node comprises a plurality of computing devices, each computing node corresponds to a computing node model, and the cloud-edge-end data fusion model is trained by using a plurality of homogenous heterogeneous data in S3 as a training set, specifically, the edge computing platform model in the cloud-edge-end data fusion model is trained by using a data parallel training method, specifically including:

[0039] S31, store the plurality of homogenous heterogeneous data in the training set in the plurality of computing node models in the edge computing platform model, and each computing node model stores one kind of homogenous heterogeneous data;

[0040] S32, for each computing node model in the edge computing platform model, train by using the plurality of homogenous heterogeneous data in the training set: each computing node model uses the homogenous heterogeneous data stored by itself to train the model, and when the accuracy of each computing node model no longer improves, stop the parameter update of all computing node models;

[0041] S33, for the plurality of computing devices in each computing node corresponding to each computing node model, train by using the homogenous heterogeneous data stored in each computing node model: by using a distributed data parallel technology, evenly distribute the homogenous heterogeneous data to each computing device, load the computing node model corresponding to the current computing node on each computing device, synchronously and in parallel update the computing node model parameters on each computing device by using a parameter sharing method, and ensure the consistency of the computing node model on each computing device, and when the accuracy of the computing node model on each computing device no longer improves, stop the training of the current computing node model.

[0042] Preferably, the cloud-edge-end data fusion model is trained by using a plurality of homogenous heterogeneous data as a training set in S3, specifically, the cloud-edge-end data fusion model is trained by using a model parallel method, specifically including:

[0043] S34, distributed model forward inference of the cloud-edge-end data fusion model: use the plurality of homogenous heterogeneous data in the training set as the input of each computing node model in the edge computing platform model, and perform calculation inference, when the accuracy of the edge computing platform model no longer changes, obtain the calculation inference result of the edge computing platform model, use the calculation inference result of the edge computing platform model as the input of the cloud computing center model, and perform calculation inference, when the accuracy of the cloud computing center model no longer changes, output the calculation inference result of the cloud computing center model, when the calculation inference of the cloud computing center model ends, the forward inference process of the cloud-edge-end data fusion model ends, and the calculation inference result of the cloud computing center model is the calculation inference result of the cloud-edge-end data fusion model;

[0044] S35, distributed model parameter updating of the cloud-edge-end data fusion model: on the cloud computing center model, the decision accuracy is calculated using the calculation and reasoning results of the cloud computing center model, and the accuracy change is calculated to determine whether the accuracy of the cloud computing center model increases. If the accuracy increases, the cloud computing center model loss is calculated, and the parameter gradient is calculated in reverse and the cloud computing center model parameters are updated. When all the parameters of the cloud computing center model are updated and the accuracy of the cloud computing center model no longer increases, the parameter gradient is passed to the edge computing platform model, and the parameter gradient is calculated and the parameters of each computing node model in the edge computing platform model are updated. When the parameter updating of all computing node models in the edge computing platform model is completed, the parameter updating of the cloud-edge-end data fusion model is completed, and the training of the cloud-edge-end data fusion model is completed.

[0045] A cloud-edge-end data fusion processing system based on deep learning adopts a cloud-edge-end data fusion processing method based on deep learning to fuse and process cloud-edge-end data. The system includes a plurality of terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, and a cloud computing center model. The edge computing platform model is arranged on the edge computing platform, and the cloud computing center model is arranged on the cloud computing center. The plurality of terminal devices are connected to the edge computing platform, the edge computing platform is connected to the cloud computing center, and the cloud computing center is connected to the plurality of terminal devices.

[0046] The plurality of terminal devices are used to obtain multi-source heterogeneous data, process the multi-source heterogeneous data, and obtain a plurality of same-source heterogeneous data.

[0047] The edge computing platform is used to receive the plurality of same-source heterogeneous data, process the plurality of same-source heterogeneous data through the edge computing platform model, obtain a plurality of high-level fusion features and a plurality of edge decisions corresponding to the plurality of high-level fusion features, feed back the plurality of edge decisions to the plurality of terminal devices, control the plurality of terminal devices to collect the same-source heterogeneous data, and feed forward the plurality of edge decisions to the cloud computing center.

[0048] The cloud computing center performs feature fusion on the plurality of high-level fusion features through the cloud computing center model to obtain decision-level fusion features and cloud decisions. The cloud computing center receives the fed forward plurality of edge decisions to control the cloud decisions of the cloud computing center. The cloud computing center feeds back the cloud decisions to the plurality of terminal devices to control the plurality of terminal devices to collect the multi-source heterogeneous data, feeds back the cloud decisions to the edge computing platform, and controls the edge decisions of the edge computing platform.

[0049] The cloud edge end data fusion processing method and system based on deep learning can effectively solve the communication and calculation problems caused by multi-source heterogeneous data, improve the overall decision and scheduling capability of the system, break the limitation of traditional centralized cloud computing, establish an efficient distributed data processing system, promote the comprehensive development of Internet of Things, promote the digital transformation of economy and society, and open a new era of Internet of Things. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of the cloud edge end data fusion processing method based on deep learning in an embodiment of the present application;

[0051] Figure 2 A structural schematic diagram of the cloud edge end data fusion processing system based on deep learning in an embodiment of the present application;

[0052] Figure 3 A training flowchart of the cloud edge end data fusion model based on deep learning in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0054] A cloud edge end data fusion processing method based on deep learning, the method comprising:

[0055] S1, a cloud edge end data fusion model is built, the cloud edge end data fusion model comprising a plurality of terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, and a cloud computing center model, the edge computing platform model being arranged on the edge computing platform, the cloud computing center model being arranged on the cloud computing center, the plurality of terminal devices being connected with the edge computing platform, the edge computing platform being connected with the cloud computing center, and the cloud computing center being connected with the plurality of terminal devices;

[0056] S2, after multi-source heterogeneous data is acquired by the plurality of terminal devices, the multi-source heterogeneous data is preprocessed to obtain a plurality of same-source heterogeneous data;

[0057] S3, the cloud edge end data fusion model is trained by using the plurality of same-source heterogeneous data as a training set, parameters are updated through back propagation until a trained cloud edge end data fusion model is obtained, and after the training of the cloud edge end data fusion model is completed, the plurality of same-source heterogeneous data collected and processed by the plurality of terminal devices is input to the trained cloud edge end data fusion model as a data set;

[0058] S4, the edge computing platform in the trained cloud edge end data fusion model receives multiple kinds of homogenous heterogeneous data in the data set, processes through the edge computing platform model, obtains multiple high-level fusion feature vectors and multiple edge decisions, transmits the multiple high-level fusion feature vectors to the cloud computing center, feeds back the multiple edge decisions to the multiple terminal devices, controls the multiple terminal devices to collect the multiple source heterogeneous data, and feeds forward the multiple edge decisions to the cloud computing center;

[0059] S5, the cloud computing center in the trained cloud edge end data fusion model receives the multiple high-level fusion feature vectors, processes through the cloud computing center model, obtains cloud decisions, and controls the cloud decisions of the cloud computing center according to the fed forward multiple edge decisions; the cloud computing center feeds back the cloud decisions to the multiple terminal devices, controls the multiple terminal devices to collect the multiple source heterogeneous data, feeds back the cloud decisions to the edge computing platform, and controls the edge decisions of the edge computing platform.

[0060] Specifically, referring to Figure 1 and Figure 2 , Figure 1 is a flowchart of a cloud edge end data fusion processing method based on deep learning in an embodiment of the application, Figure 2 is a structural schematic diagram of a cloud edge end data fusion processing system based on deep learning in an embodiment of the application.

[0061] The cloud edge end data fusion processing method based on deep learning comprises:

[0062] 1) Building a cloud edge end data fusion model: the cloud edge end data fusion model comprises multiple terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, a cloud computing center model, the edge computing platform model is arranged on the edge computing platform, the cloud computing center model is arranged on the cloud computing center, the multiple terminal devices are connected with the edge computing platform and the cloud computing center, and the edge computing platform is connected with the cloud computing center;

[0063] 2) Data acquisition and preprocessing: acquiring multiple source heterogeneous data such as images, videos, sounds and texts by using various sensors, mobile phones, personal computers and other terminal devices; in order to ensure the synchronism between the multiple source heterogeneous data, a soft synchronization method is used to acquire multiple source heterogeneous data with a synchronous frequency; in order to avoid or reduce the influence of the heterogeneity of each kind of homogenous heterogeneous data in the multiple source heterogeneous data on each computing node model in the edge computing platform model, a standardization method (for example, Z-Score) is usually used for preprocessing, so as to reduce the difference between the homogenous heterogeneous data density, probability distribution and the correlation of internal attributes, and eliminate the non-equivalence of the homogenous heterogeneous data.

[0064] 3) training the cloud-edge-end data fusion model by using multiple homogenous and heterogeneous data as a training set, back-propagating and updating parameters until obtaining the trained cloud-edge-end data fusion model, after the training of the cloud-edge-end data fusion model, inputting the multiple homogenous and heterogeneous data collected and processed by the multiple terminal devices into the trained cloud-edge-end data fusion model as a data set;

[0065] 4) the edge computing platform in the trained cloud-edge-end data fusion model receives the multiple homogenous and heterogeneous data in the data set, processes the multiple homogenous and heterogeneous data through the edge computing platform model to obtain multiple high-level fusion feature vectors and multiple edge decisions, transmits the obtained multiple high-level fusion feature vectors to the cloud computing center, and feeds back the obtained multiple edge decisions to the multiple terminal devices to control the multiple terminal devices to collect the multiple source heterogeneous data and feed forward the multiple edge decisions to the cloud computing center;

[0066] 5) the cloud computing center in the trained cloud-edge-end data fusion model receives the multiple high-level fusion feature vectors, processes the multiple high-level fusion feature vectors through the cloud computing center model to obtain a cloud decision, controls the cloud decision of the cloud computing center according to the fed forward multiple edge decisions, feeds back the cloud decision to the multiple terminal devices to control the multiple terminal devices to collect the multiple source heterogeneous data, and feeds back the cloud decision to the edge computing platform to control the edge decision in the edge computing platform.

[0067] In one embodiment, after the multiple source heterogeneous data is obtained by using the multiple terminal devices in S2, the multiple source heterogeneous data is preprocessed to obtain multiple homogenous and heterogeneous data, and the specific process includes:

[0068] S21, obtaining multiple source heterogeneous data by using multiple terminal devices;

[0069] S22, calibrating the multiple source heterogeneous data into multiple source heterogeneous data with a synchronous frequency by using a time stamp sampling and linear interpolation method;

[0070] S23, performing standardization processing on the multiple source heterogeneous data with a synchronous frequency to obtain multiple homogenous and heterogeneous data.

[0071] In one embodiment, the multiple homogenous and heterogeneous data is obtained in S23, and the specific formula is:

[0072]

[0073]

[0074]

[0075] wherein, x ij represents the jth homogenous and heterogeneous data in the ith homogenous and heterogeneous data, x ijLet μ represent the j-th unnormalized homogeneous data in the i-th homogeneous data set, where j = 1, 2, ..., N, and N represents the total number of homogeneous data in the i-th homogeneous data set. i δ i Let represent the mean and variance of the i-th homogeneous heterogeneous data set.

[0076] Specifically, firstly, multi-source heterogeneous data with the same sampling frequency and timestamps are acquired using soft synchronization. Then, calibration is performed using timestamp sampling and linear interpolation to obtain synchronized frequency data, eliminating the impact of time deviation. Next, for each type of homogeneous heterogeneous data within the multi-source heterogeneous data, Z-Score normalization is applied to reduce differences in density, probability distribution, and correlation of intrinsic attributes among the homogeneous heterogeneous data, thus eliminating the inequivalence of homogeneous heterogeneous data.

[0077]

[0078]

[0079]

[0080] Where, x ij ' represents the j-th homogeneous data in the i-th homogeneous data, x ij Let μ represent the j-th unnormalized homogeneous data in the i-th homogeneous data set, where j = 1, 2, ..., N, and N represents the total number of homogeneous data in the i-th homogeneous data set. i δ i Let represent the mean and variance of the i-th homogeneous heterogeneous data set.

[0081] In one embodiment, the edge computing platform model includes multiple parallel computing node models. In the cloud-edge-device data fusion model trained in S4, the edge computing platform receives various homogeneous and heterogeneous data from the dataset, processes them through the edge computing platform model, and obtains multiple high-level fusion feature vectors and multiple edge decisions, specifically including:

[0082] S41. Store multiple homogeneous data in the dataset in multiple computing node models, with one computing node model storing one type of homogeneous data.

[0083] S42. The computing node model includes a neural network, a first multilayer perceptron, and a first Softmax classifier. The neural network is used to extract features from homogeneous heterogeneous data stored in the corresponding computing node model to obtain a primary fusion feature vector. The first multilayer perceptron performs feature-level fusion on the primary fusion feature vector to obtain a high-level fusion feature vector.

[0084] S43, the first Softmax classifier is used for classifying the high-level fusion feature vector to obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector;

[0085] S44, a decision probability of the preliminary decision is calculated according to a normalization method, and a preliminary decision with the maximum decision probability value is selected as an edge decision output by the computing node model;

[0086] S45, a plurality of high-level fusion feature vectors and a plurality of edge decisions are obtained by repeating steps S42-S44 by traversing a plurality of isomeric data.

[0087] In one embodiment, the first multi-layer perception in S42 performs feature-level fusion on the primary fusion feature vector to obtain a high-level fusion feature vector, and the specific formula is:

[0088]

[0089] wherein,

[0090] In the formula, F i represents the high-level fusion feature vector of the i-th isomeric data, F i represents the primary fusion feature vector of the i-th isomeric data, W l represents the l-th learnable feature mapping matrix, 1≤l≤L, L represents the total number of feature mapping matrices, f ij t represents the primary fusion feature output after the t-th cycle of the i-th j-th isomeric data, and N represents the total number of the i-th isomeric data.

[0091] Specifically, a neural network can be used to extract features from a plurality of isomeric data in a data set: for one-dimensional isomeric data, a recurrent neural network RNN can be used to extract one-dimensional features, that is, a one-dimensional feature vector; for multi-dimensional isomeric data, a convolutional neural network CNN can be used to extract multi-dimensional features.

[0092] Taking one-dimensional feature extraction from one-dimensional isomeric data by using a recurrent neural network RNN as an example, the feature extraction method is as follows:

[0093]

[0094] wherein, is the feature output after the t-th cycle of the i-th j-th isomeric data (x ij is equivalent to isomeric data at t=0), is the feature output after the t-1-th cycle of the i-th j-th isomeric data, Xt represents the input feature after the tth cycle, V is a mapping matrix, Relu represents an activation function, and t represents the tth cycle.

[0095] The global average pooling method GAP is used to convert the multi-dimensional feature into a one-dimensional feature vector, and a cascade method is used for feature splicing and preliminary fusion, which can ensure the fusion of isomeric data features and solve the problem of different lengths of feature vectors caused by information density between isomeric data:

[0096]

[0097] wherein F i represents the preliminary fusion feature vector of the ith isomeric data, F i is a 1*N vector, represents the preliminary fusion feature output after the tth cycle of the jth isomeric data of the ith isomeric data, i.e., the feature output after the tth cycle of the jth isomeric data of the ith isomeric data x ij represents the feature output after the tth cycle, and N represents the total number of isomeric data provided by different terminal devices, 1≤j≤N.

[0098] The multi-layer perception MLP is used to perform feature-level fusion on the preliminary fusion feature vector of each isomeric data to obtain a high-level fusion feature vector:

[0099]

[0100]

[0101] wherein F i represents the high-level fusion feature vector of the ith isomeric data, F i is also a 1*N vector, represents the high-level fusion feature output after the tth cycle of the jth isomeric data of the ith isomeric data, M il represents the lth learnable feature mapping matrix of the ith isomeric data, l=1,2,...,L, L represents the total number of feature mapping matrices, and L represents the use of L layers of MLP.

[0102] The first Softmax classifier is used to classify the high-level fusion feature vector to obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector.

[0103] The decision probability of the preliminary decision is calculated by a normalization method:

[0104]

[0105] wherein S ij is the high-level fusion feature of the jth isomeric data of the ith isomeric data the decision probability value of the corresponding edge decision, 0≤S ij ≤1.

[0106] The preliminary decision with the maximum decision probability value is selected as the edge decision of the i-th computing node model, that is, the edge decision corresponding to the i-th isogenetic and anisogenic data. The edge decision is fed back to multiple different terminal devices, a pop-up window of the human-computer interactive device terminal is prompted, the user is reminded to upload the corresponding data, and an instruction is issued to the active acquisition device terminal to control data acquisition. The i-th high-level fusion feature vector F i ' and the edge decision of the i-th computing node model are uploaded to the cloud computing center.

[0107] In an embodiment, the cloud computing center in the cloud edge data fusion model trained in S5 receives multiple high-level fusion feature vectors, processes the multiple high-level fusion feature vectors through a cloud computing center model, and obtains a cloud decision, specifically including:

[0108] S51, the cloud computing center model includes a transformer network, a second multilayer perception machine and a second Softmax classifier. The transformer network is used for same-dimension coding and feature fusion of the multiple high-level fusion feature vectors, and accordingly obtains multiple decision-level fusion feature vectors. The second multilayer perception machine re-fuses the multiple decision-level fusion feature vectors to obtain a re-fused feature vector. The second Softmax classifier classifies the re-fused feature vector to obtain a final decision corresponding to each feature in the re-fused feature vector.

[0109] S52, the decision probability of the final decision is calculated according to a normalization method, and the final decision with the maximum decision probability value is selected as the cloud decision.

[0110] In an embodiment, the transformer network in S51 is used for same-dimension coding and feature fusion of the multiple high-level fusion feature vectors, and accordingly obtains multiple decision-level fusion feature vectors. The specific formula is:

[0111]

[0112] In the formula, R i represents the decision-level fusion feature vector corresponding to the i-th isogenetic and anisogenic data, F oi ', F pi ', F qi represents the high-level fusion feature vector corresponding to the i-th isogenetic and anisogenic data after same-dimension coding, ReLU represents an activation function, d represents a feature dimension, W1 and W2 represent learnable feature mapping matrices, and b1 and b2 represent learnable offset matrices.

[0113] Specifically, the high-level fusion feature vector provided by each computing node model in the edge computing platform model is first one-hot encoded, which is converted into sparse data with only one bit activated, which can effectively solve the problem of difficult classification of multi-source data decision and effectively expand the features; then the PCA dimension reduction operation is used to reduce the data dimension to alleviate the dimension disaster, and can effectively reduce the data noise and ensure the feature independence, help the subsequent neural network to learn and extract the features, and complete the same dimension coding processing of multiple decision probability features. The self-attention mechanism in the transformer network is used to obtain the correlation between the isomeric data, and the isomeric data is fused at the decision level to obtain the decision-level fusion feature vector corresponding to the isomeric data:

[0114]

[0115] wherein R i represents the decision-level fusion feature corresponding to the i-th isomeric data, ReLU represents an activation function for introducing a nonlinear factor to improve the expression ability of the neural network, F oi , F pi , F qi represents the same dimension coded high-level fusion feature vector corresponding to the i-th isomeric data, d represents the feature dimension, which is used to avoid the problem of gradient explosion caused by large numerical value after multiple multiplication operations, W1 and W2 represent the learnable feature mapping matrix, and b1 and b2 represent the learnable offset matrix.

[0116] A second multi-layer perceptron (MLP) is used to re-fuse multiple decision-level fusion feature vectors to obtain a re-fusion feature vector to reduce the fusion feature dimension, and a second Softmax classifier is used to classify the re-fusion feature vector to obtain the final decision corresponding to each feature in the re-fusion feature vector. According to the normalization method, multiple final decisions are converted into a probability distribution with a range of 0-1 and a sum of 1. The specific calculation method is the same as calculating the decision probability of each preliminary decision in multiple preliminary decisions according to the normalization method, which will not be described here. Then, the final decision with the maximum decision probability value is selected as the cloud decision according to the probability distribution, and the cloud decision is fed back to multiple terminal devices for large-scale terminal device leadership control to control the collection of multi-source heterogeneous data; the cloud decision is fed back to the edge computing platform to control the upload of isomeric data.

[0117] In one embodiment, the edge computing platform model includes a plurality of parallel computing node models, the edge computing platform includes a plurality of parallel computing nodes, each computing node includes a plurality of computing devices, each computing node corresponds to a computing node model, and the cloud edge end data fusion model is trained in S3 using a plurality of homogenous heterogeneous data as a training set. Specifically, the edge computing platform model in the cloud edge end data fusion model is trained using a data parallel training method, which specifically includes:

[0118] S31, store the plurality of homogenous heterogeneous data in the training set in the plurality of computing node models in the edge computing platform model, and each computing node model stores one kind of homogenous heterogeneous data;

[0119] S32, for each computing node model in the edge computing platform model, train using the plurality of homogenous heterogeneous data in the training set: each computing node model uses the homogenous heterogeneous data stored therein to train the model, and when the accuracy of each computing node model no longer improves, stop updating the parameters of all computing node models;

[0120] S33, for the plurality of computing devices in each computing node corresponding to each computing node model, train using the homogenous heterogeneous data stored in each computing node model: use distributed data parallel technology to evenly distribute the homogenous heterogeneous data to each computing device, load the computing node model corresponding to the current computing node on each computing device, and use parameter sharing to synchronously and in parallel update the computing node model parameters on each computing device to ensure the consistency of the computing node model on each computing device. When the accuracy of the computing node model on each computing device no longer improves, stop training the current computing node model.

[0121] Specifically, the plurality of different terminal devices decouple the multi-source heterogeneous data into a plurality of homogenous heterogeneous data, and distribute the plurality of homogenous heterogeneous data to each computing node model in the edge computing platform model. Each computing node model stores one kind of homogenous heterogeneous data, which is beneficial to reduce data access and transmission. The edge computing platform model is trained in parallel using a plurality of homogenous heterogeneous data: each computing node model uses the homogenous heterogeneous data stored in each node model to train, and the nodes do not communicate with each other and do not transmit data. When the accuracy of each node model no longer improves, stop updating the parameters of all node models. In a single computing node model in the edge computing platform model, the corresponding homogenous heterogeneous data is trained in parallel: use distributed data parallel (DDP) technology to synchronously distribute the training data to each computing device in the computing node, and use parameter sharing to synchronously and in parallel update the parameters of the node model. When the accuracy of the computing node model does not improve, stop training.

[0122] The DDP technology is as follows:

[0123] 1) Start multiple processes to be responsible for multiple computing devices in multiple computing nodes, each process loads the computing node model of the corresponding node on each computing device, and distributes training data and training tasks according to the performance of each computing device in the node.

[0124] 2) After each computing node model performs a forward inference process, the computing result of each computing node model is transmitted to each computing device in the node, the result is analyzed and the training loss is calculated, and after completion, the training loss is transmitted to the computing node model, and the gradient calculation and model parameter update are performed on each computing device in the node. When all computing device parameter updates are completed, the current computing node model completes a model parameter update.

[0125] 3) Since the model structure and initialization parameters on multiple computing devices in each computing node are the same, and the same training loss and training strategy are used for gradient calculation and parameter update, after training, the model parameters on multiple computing devices in each computing node model are completely the same.

[0126] In one embodiment, multiple isomorphic heterogeneous data in S3 are used as a training set to train the cloud-edge-end data fusion model, specifically, the cloud-edge-end data fusion model is trained in a model parallel manner, specifically including:

[0127] S34, distributed model forward inference of the cloud-edge-end data fusion model: multiple isomorphic heterogeneous data in the training set are used as the input of each computing node model in the edge computing platform model, and after calculation and inference, when the accuracy of the edge computing platform model no longer changes, the calculation and inference result of the edge computing platform model is obtained. The calculation and inference result of the edge computing platform model is used as the input of the cloud computing center model, and after calculation and inference, when the accuracy of the cloud computing center model no longer changes, the calculation and inference result of the cloud computing center model is output. When the cloud computing center model calculation and inference is completed, the forward inference process of the cloud-edge-end data fusion model is completed, and the calculation and inference result of the cloud computing center model is the calculation and inference result of the cloud-edge-end data fusion model.

[0128] S35, distributed model parameter updating of the cloud-edge-end data fusion model: on the cloud computing center model, the decision accuracy is calculated using the calculation inference result of the cloud computing center model, and the accuracy change is calculated, to determine whether the accuracy of the cloud computing center model increases, if the accuracy increases, the cloud computing center model loss is calculated, and the parameter gradient is calculated in reverse and the cloud computing center model parameters are updated, when all the parameters of the cloud computing center model are updated, and the accuracy of the cloud computing center model no longer increases, the parameter gradient is transmitted to the edge computing platform model, and the parameter gradient is calculated and the parameters of each computing node model in the edge computing platform model are updated, when the parameter updating of all computing node models in the edge computing platform model is completed, the parameter updating of the cloud-edge-end data fusion model is completed, and the training of the cloud-edge-end data fusion model is completed.

[0129] Specifically, referring to Figure 3 , Figure 3 the training flowchart of the cloud-edge-end data fusion model based on deep learning in an embodiment of the application.

[0130] Through the model parallel training method, the edge computing platform model and the cloud computing center model are fine-tuned by using multiple homologous heterogeneous data, and the training of the cloud-edge-end data fusion model is completed:

[0131] 1) Distributed model forward inference of the cloud-edge-end data fusion model: the edge computing platform model uses homologous heterogeneous data as the input of each computing node model, calculates the model result, decision accuracy and accuracy change, and judges according to the accuracy change, when the accuracy change is greater than zero, the model loss and parameter gradient are calculated, and the model parameters are updated, and the homologous heterogeneous data is used as the input of each computing node model again, and the above calculation inference (calculating the model result, decision accuracy and accuracy change, judging according to the accuracy change) is performed, until the accuracy of the computing node model no longer changes, the calculation inference of the computing node model ends. When the calculation inference of all computing node models ends, the calculation inference result of the edge computing platform model is transmitted to the cloud computing center model as input, and the calculation inference is performed (the specific calculation inference method is the same as the calculation inference method of each computing node model, which will not be described here), when the calculation inference of the cloud computing center model ends, the forward inference process of the cloud-edge-end data fusion model ends. The formulaic expression of the forward calculation inference process is as follows:

[0132]

[0133]

[0134] y = (U1(x 11 ')+U1(x 12 ')+···+U1(x1j '))+···+(U i (x i1 ')+U i (x i2 ')+···+U i (x ij '))

[0135] wherein y represents the final calculation result of the cloud-edge-end data fusion model, Y represents the cloud computing center model, U i represents the i-th calculation node model in the edge computing platform model, u i represents the calculation result of the i-th calculation node model in the edge computing platform model, i.e., the calculation result of the i-th homogenous heterogeneous data, x ij ' represents the input of the i-th calculation node model, i.e., the j-th homogenous heterogeneous data in the i-th homogenous heterogeneous data, and it is to be noted that, assuming that N kinds of multi-source heterogeneous data are obtained through N different terminal devices, after processing, no more than N kinds of homogenous heterogeneous data are obtained, at this time, the number of calculation nodes in the edge computing platform model satisfies “each calculation node processes one kind of homogenous heterogeneous data”, and it is not necessarily N.

[0136] 2) Distributed model parameter updating of the cloud-edge-end data fusion model: on the cloud computing center model, the decision accuracy is calculated by using the inference result of the cloud-edge-end data fusion model, and the accuracy change is calculated, to determine whether the cloud computing center model accuracy is increased, if the accuracy is increased, the cloud computing center model loss is calculated, and the parameter gradient is calculated in reverse and the cloud computing center model parameters are updated, when all the parameters of the cloud computing center model are updated, and the accuracy of the cloud computing center model is no longer increased, the parameter gradient is transmitted to the edge computing platform model, and the parameter gradient and the model parameters are calculated and updated on each calculation node model, when the parameter updating of all the calculation node models in the edge computing platform model is completed, i.e., the parameter updating of the cloud-edge-end data fusion model is completed, the training of the cloud-edge-end data fusion model is completed. The formulaic expression of the calculation parameter gradient process is as follows:

[0137]

[0138]

[0139]

[0140] wherein Y' represents the parameter gradient of the cloud computing center model, y represents the final calculation result of the cloud-edge-end data fusion model, U i ' represents the parameter gradient of the i-th calculation node model in the edge computing platform model, u i represents the calculation result of the i-th calculation node model in the edge computing platform model, xij represents the i-th computing node model input, i.e., the j-th homogenous heterogeneous data in the i-th homogenous heterogeneous data.

[0141] In one embodiment, the cloud computing center comprises a plurality of parallel computing devices, and the cloud computing center model is trained using the computing inference results of the edge computing platform model, specifically comprising:

[0142] 1) constructing training data of the cloud computing center model using the inference results of each computing node model in the edge computing platform model;

[0143] 2) synchronously distributing the training data to each computing device in the cloud computing center using the DDP technology, and synchronously and parallelly updating the model parameters using the parameter sharing method, and stopping the training when the accuracy of the cloud computing center model no longer improves.

[0144] In one embodiment, a cloud-edge-end data fusion processing system based on deep learning adopts a deep learning-based cloud-edge-end data fusion processing method to fuse and process cloud-edge-end data. The system comprises a plurality of terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, a cloud computing center model, the edge computing platform model is arranged on the edge computing platform, the cloud computing center model is arranged on the cloud computing center, the plurality of terminal devices are connected with the edge computing platform, the edge computing platform and the cloud computing center are connected, and the cloud computing center and the plurality of terminal devices are connected, wherein:

[0145] The plurality of terminal devices are used to acquire multi-source heterogeneous data, process the multi-source heterogeneous data, and obtain a plurality of homogenous heterogeneous data;

[0146] The edge computing platform is used to receive the plurality of homogenous heterogeneous data, process the plurality of homogenous heterogeneous data through the edge computing platform model, obtain a plurality of high-level fusion features and a plurality of edge decisions corresponding to the plurality of high-level fusion features, feed back the plurality of edge decisions to the plurality of terminal devices to control the plurality of terminal devices to collect the homogenous heterogeneous data, and feed forward the plurality of edge decisions to the cloud computing center;

[0147] The cloud computing center is used to fuse the plurality of high-level fusion features through the cloud computing center model to obtain decision-level fusion features and a cloud decision; the cloud computing center receives the fed forward plurality of edge decisions to control the cloud decision of the cloud computing center; the cloud computing center feeds back the cloud decision to the plurality of terminal devices to control the plurality of terminal devices to collect the multi-source heterogeneous data, feeds back the cloud decision to the edge computing platform to control the edge decision of the edge computing platform.

[0148] The specific limitations of the cloud-edge-end data fusion processing system based on deep learning can be referred to the limitations of the deep learning-based cloud-edge-end data fusion processing method in the foregoing, which will not be repeated here.

[0149] The cloud edge end data fusion processing method and system based on deep learning can not only effectively solve the communication and calculation problems caused by multi-source heterogeneous data, but also improve the overall decision and scheduling capability of the system, break the limitation of traditional centralized cloud computing, establish an efficient distributed data processing system, promote the comprehensive development of Internet of Things, promote the digital transformation of economy and society, and open a new era of Internet of Everything.

[0150] The cloud edge end data fusion processing method and system based on deep learning are described in detail above. In this paper, specific examples are used to explain the principles and implementation methods of the present application. The above examples are only used to help understand the core idea of the present application. It should be pointed out that for ordinary skilled persons in the art, without departing from the principles of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A cloud-edge-device data fusion processing method based on deep learning, characterized in that, The method includes: S1. Build a cloud-edge-device data fusion model, which includes multiple terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, and a cloud computing center model. The edge computing platform model is set on the edge computing platform, and the cloud computing center model is set on the cloud computing center. The multiple terminal devices are connected to the edge computing platform, the edge computing platform is connected to the cloud computing center, and the cloud computing center is connected to the multiple terminal devices. S2. After acquiring multi-source heterogeneous data using multiple terminal devices, the multi-source heterogeneous data is preprocessed to obtain multiple homogeneous heterogeneous data. S3. The cloud-edge-device data fusion model is trained using multiple homogeneous heterogeneous data as training sets, and the parameters are backpropagated and updated until the trained cloud-edge-device data fusion model is obtained. After the training of the cloud-edge-device data fusion model is completed, multiple homogeneous heterogeneous data collected and processed by multiple terminal devices are input as datasets into the trained cloud-edge-device data fusion model. S4. After receiving multiple homogeneous heterogeneous data from the dataset, the edge computing platform in the trained cloud-edge-device data fusion model processes the data through the edge computing platform model to obtain multiple high-level fusion feature vectors and multiple edge decisions. The multiple high-level fusion feature vectors are transmitted to the cloud computing center, and the multiple edge decisions are fed back to the multiple terminal devices to control the collection of multi-source heterogeneous data by the multiple terminal devices. The multiple edge decisions are then fed forward to the cloud computing center. S5. In the trained cloud-edge-device data fusion model, the cloud computing center receives multiple high-level fusion feature vectors, processes them through the cloud computing center model, and obtains cloud decisions. The cloud computing center controls its cloud decisions based on the multiple feedforward edge decisions. The cloud computing center feeds back the cloud decisions to multiple terminal devices, controls the collection of multi-source heterogeneous data by the multiple terminal devices, and feeds back the cloud decisions to the edge computing platform, controls the edge decisions of the edge computing platform. The edge computing platform model includes multiple parallel computing node models. In S4, the edge computing platform in the trained cloud-edge-device data fusion model receives various homogeneous and heterogeneous data from the dataset and processes them through the edge computing platform model to obtain multiple high-level fusion feature vectors and multiple edge decisions, specifically including: S41. Store multiple homogeneous data in the dataset in multiple computing node models, wherein one computing node model stores one homogeneous data. S42. The computing node model includes a neural network, a first multilayer perceptron and a first Softmax classifier. The neural network is used to extract features from homogeneous data stored in the corresponding computing node model to obtain a primary fusion feature vector. The first multilayer perceptron performs feature-level fusion on the primary fusion feature vector to obtain a high-level fusion feature vector. S43. The first Softmax classifier is used to classify the high-level fusion feature vector to obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector; S44. Calculate the decision probability of the preliminary decision according to the normalization method, and select the preliminary decision with the largest decision probability value as the edge decision output by the corresponding computing node model. S45. Traverse multiple homogeneous and heterogeneous data sources and repeat steps S42-S44 to obtain multiple high-level fusion feature vectors and multiple edge decisions. In the cloud-edge-device data fusion model trained in S5, the cloud computing center receives multiple advanced fusion feature vectors, processes them through the cloud computing center model, and obtains cloud decisions, specifically including: S51. The cloud computing center model includes a transformer network, a second multilayer perceptron, and a second Softmax classifier. The transformer network is used to encode and fuse multiple high-level fusion feature vectors in the same dimension to obtain multiple decision-level fusion feature vectors. The second multilayer perceptron re-fused the multiple decision-level fusion feature vectors to obtain a re-fused feature vector. The second Softmax classifier classifies the re-fused feature vector to obtain the final decision corresponding to each feature in the re-fused feature vector. S52. Calculate the decision probability of the final decision according to the normalization method, and select the final decision with the largest decision probability value as the cloud decision.

2. The cloud-edge-device data fusion processing method based on deep learning as described in claim 1, characterized in that, In step S2, after acquiring multi-source heterogeneous data using multiple terminal devices, the multi-source heterogeneous data is preprocessed to obtain various homogeneous heterogeneous data. The specific process includes: S21. Obtain multi-source heterogeneous data using multiple terminal devices; S22. The multi-source heterogeneous data is calibrated into multi-source heterogeneous data with a synchronous frequency by timestamp sampling and linear interpolation. S23. Standardize the multi-source heterogeneous data with synchronous frequency to obtain a variety of homogeneous heterogeneous data.

3. The cloud-edge-device data fusion processing method based on deep learning as described in claim 2, characterized in that, In step S23, multiple homogeneous and heterogeneous data are obtained, and the specific formula is as follows: Where, x ij ' represents the j-th homogeneous data in the i-th homogeneous data, x ij Let μ represent the j-th unnormalized homogeneous data in the i-th homogeneous data set, where j = 1, 2, ..., N, and N represents the total number of homogeneous data in the i-th type. i δ i Let represent the mean and variance of the i-th homogeneous heterogeneous data set.

4. The cloud-edge-device data fusion processing method based on deep learning as described in claim 3, characterized in that, In step S42, the first multilayer perceptron performs feature-level fusion on the primary fusion feature vector to obtain a high-level fusion feature vector, as shown in the following formula: in, In the formula, F i ' represents the high-level fusion feature vector of the i-th homogeneous heterogeneous data, F i W represents the primary fusion feature vector of the i-th homogeneous heterogeneous data. l Let f represent the l-th learnable feature map matrix, 1 ≤ l ≤ L, where L represents the total number of feature map matrices. ij t Let N represent the primary fusion feature output after the t-th iteration of the i-th homogeneous heterogeneous data, and let N represent the total number of the i-th homogeneous heterogeneous data.

5. The cloud-edge-device data fusion processing method based on deep learning as described in claim 4, characterized in that, The transformer network in S51 is used to encode and fuse multiple high-level fusion feature vectors in the same dimension, thereby obtaining multiple decision-level fusion feature vectors. The specific formula is as follows: In the formula, R i F represents the decision-level fusion feature vector corresponding to the i-th type of homogeneous data. oi '、F pi '、F qi ' represents the high-level fusion feature vector encoded in the same dimension for the i-th homogeneous heterogeneous data, ReLU represents the activation function, d represents the feature dimension, W1 and W2 represent the learnable feature mapping matrices, and b1 and b2 represent the learnable offset matrices.

6. The cloud-edge-device data fusion processing method based on deep learning as described in claim 1, characterized in that, The edge computing platform model includes multiple parallel computing node models. Each computing node includes multiple computing devices, and each computing node corresponds to a computing node model. In step S3, various homogeneous heterogeneous data are used as training sets to train the cloud-edge-device data fusion model. Specifically, the edge computing platform model in the cloud-edge-device data fusion model is trained using a data-parallel training method, including: S31. Store multiple homogeneous and heterogeneous data from the training set in multiple computing node models in the edge computing platform model, with each computing node model storing one type of homogeneous and heterogeneous data. S32. For each computing node model in the edge computing platform model, multiple homogeneous and heterogeneous data from the training set are used for training: each computing node model uses its own stored homogeneous and heterogeneous data for model training. When the accuracy of each computing node model no longer improves, the parameter update of all computing node models is stopped. S33. For multiple computing devices in each computing node corresponding to each computing node model, training is performed using homogeneous heterogeneous data stored in each computing node model: using distributed data parallel technology, homogeneous heterogeneous data is evenly distributed to each computing device, and the computing node model corresponding to this computing node is loaded on each computing device. The parameters of the computing node model on each computing device are updated synchronously and in parallel using parameter sharing to ensure the consistency of the computing node model on each computing device. When the accuracy of the computing node model on each computing device no longer improves, the training of the current computing node model is stopped.

7. The cloud-edge-device data fusion processing method based on deep learning as described in claim 6, characterized in that, In step S3, multiple types of homogeneous but heterogeneous data are used as training sets to train the cloud-edge-device data fusion model. Specifically, the cloud-edge-device data fusion model is trained in a model-parallel manner, including: S34. Distributed forward inference of the cloud-edge-device data fusion model: Multiple homogeneous and heterogeneous data from the training set are used as inputs to the computing node models in the edge computing platform model. After computational inference, when the accuracy of the edge computing platform model no longer changes, the computational inference result of the edge computing platform model is obtained. The computational inference result of the edge computing platform model is used as input to the cloud computing center model. After computational inference, when the accuracy of the cloud computing center model no longer changes, the computational inference result of the cloud computing center model is output. When the computational inference of the cloud computing center model ends, the forward inference process of the cloud-edge-device data fusion model ends, and the computational inference result of the cloud computing center model is the computational inference result of the cloud-edge-device data fusion model. S35. Distributed model parameter update of the cloud-edge-device data fusion model: On the cloud computing center model, the decision accuracy is calculated using the computational inference results of the cloud computing center model, and the change in accuracy is calculated. It is determined whether the accuracy of the cloud computing center model has increased. If the accuracy has increased, the loss of the cloud computing center model is calculated, and the parameter gradient is calculated in reverse and the parameters of the cloud computing center model are updated. When all parameters of the cloud computing center model have been updated and the accuracy of the cloud computing center model no longer increases, the parameter gradient is passed to the edge computing platform model, and the parameter gradient is calculated and the parameters of each computing node model are updated on each computing node model in the edge computing platform model. When the parameter updates of all computing node models in the edge computing platform model have been completed, the parameter update of the cloud-edge-device data fusion model is completed, and the training of the cloud-edge-device data fusion model is completed.

8. A cloud-edge-device data fusion processing system based on deep learning, comprising using the cloud-edge-device data fusion processing method based on deep learning as described in any one of claims 1-7 to fuse cloud-edge-device data, characterized in that, The system includes multiple terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, and a cloud computing center model. The edge computing platform model is set on the edge computing platform, and the cloud computing center model is set on the cloud computing center. The multiple terminal devices are all connected to the edge computing platform, the edge computing platform is connected to the cloud computing center, and the cloud computing center is connected to the multiple terminal devices. Multiple terminal devices are used to acquire multi-source heterogeneous data, process the multi-source heterogeneous data, and obtain multiple homogeneous heterogeneous data. The edge computing platform is used to receive multiple homogeneous heterogeneous data, process the multiple homogeneous heterogeneous data through the edge computing platform model to obtain multiple advanced fusion features and corresponding multiple edge decisions, feed back the multiple edge decisions to multiple terminal devices, control the multiple terminal devices to collect homogeneous heterogeneous data, and feed forward the multiple edge decisions to the cloud computing center. The cloud computing center performs feature fusion on multiple advanced fusion features through the cloud computing center model to obtain decision-level fusion features and cloud decisions; the cloud computing center receives multiple feedforward edge decisions to control the cloud decisions of the cloud computing center; the cloud computing center feeds back the cloud decisions to multiple terminal devices to control the collection of multi-source heterogeneous data by the multiple terminal devices, and feeds back the cloud decisions to the edge computing platform to control the edge decisions of the edge computing platform.

Images