Industrial multi-modal data edge cloud collaborative fusion processing method and related equipment

Abstract

The invention provides an industrial multi-modal data edge cloud collaborative fusion processing method and related equipment, and relates to the technical field of industrial internet and edge computing. Acquiring multi-modal data of the industrial equipment at the current stage at the edge node, and performing feature extraction on the multi-modal data to obtain multi-modal features; performing equipment state prediction processing on the multi-modal features through a lightweight fusion model to obtain an equipment state prediction result corresponding to the industrial equipment, model parameters of the lightweight fusion model being obtained by performing model optimization training on the basis of the equipment state prediction result of the industrial equipment in the previous stage and the multi-modal features; and sending an equipment state prediction result and the multi-modal features corresponding to the current stage to the cloud, so as to load model optimization parameters issued by the cloud, and carrying out model parameter updating on the lightweight fusion model, so as to realize a collaborative closed loop of low-delay edge intelligent decision of industrial multi-modal data and high-precision cloud continuous optimization.

Description

Technical Field

[0001] This application relates to the fields of industrial internet and edge computing technology, and in particular to a method and related equipment for industrial multimodal data edge-cloud collaborative fusion processing. Background Technology

[0002] As intelligent manufacturing continues to evolve towards deep perception and intelligent decision-making, industrial sites are generating massive amounts of multimodal data, encompassing both structured and unstructured data. This multi-source, heterogeneous data collectively forms the key information foundation for equipment condition monitoring, fault diagnosis, production quality control, and process optimization.

[0003] Currently, the mainstream multimodal data processing solutions in the industry are mainly centralized cloud processing models or simple edge-cloud separation architectures. In the centralized cloud processing model, the edge devices only handle raw data acquisition and uploading, lacking effective preprocessing and feature extraction capabilities. This results in a large amount of redundant raw data being uploaded to the cloud, which not only increases the network bandwidth burden but also significantly increases system response latency. In the edge-cloud separation architecture adopted to address this issue, the edge devices only perform basic data format conversion or filtering operations. Multimodal fusion still relies on the cloud to simply stitch together data from different modalities, failing to fully model the dynamic semantic relationships between modalities, thus limiting the improvement of fusion accuracy. Furthermore, edge computing resources are not used efficiently, while cloud models are difficult to dynamically adapt and optimize according to the actual operating conditions at the edge.

[0004] It is known that the existing mainstream multimodal data processing solutions in the industry still have significant shortcomings in terms of processing timeliness, fusion accuracy, resource utilization efficiency, model adaptability and system robustness, making it difficult to meet the comprehensive requirements of intelligent manufacturing for real-time performance, accuracy and reliability.

[0005] Therefore, how to achieve a low-latency, high-precision, and highly adaptable edge-cloud collaborative fusion processing solution for industrial multimodal data is an urgent problem to be solved. Summary of the Invention

[0006] This application provides an edge-cloud collaborative fusion processing method and related equipment for industrial multimodal data, which can realize a collaborative closed loop of low-latency edge intelligent decision-making and high-precision cloud continuous optimization of industrial multimodal data. While ensuring real-time performance, the accuracy of equipment status monitoring and overall robustness are improved through dynamic model updates and adaptive fusion.

[0007] In a first aspect, embodiments of this application provide an industrial multimodal data edge-cloud collaborative fusion processing method, applied to edge nodes corresponding to industrial equipment, including:

[0008] Acquire multimodal data of industrial equipment at the current stage;

[0009] Feature extraction is performed on multimodal data to obtain multimodal features;

[0010] The lightweight fusion model is used to predict the equipment status of multimodal features to obtain the equipment status prediction results of industrial equipment corresponding to multimodal data. The model parameters of the lightweight fusion model are obtained by cloud-based model optimization training based on the equipment status prediction results of industrial equipment in the previous stage and multimodal features.

[0011] Send the device status prediction results and multimodal features corresponding to the current stage to the cloud to load the model optimization parameters issued by the cloud, and update the model parameters of the lightweight fusion model based on the model optimization parameters.

[0012] In one possible implementation, the multimodal data includes structured and unstructured data; feature extraction is performed on the multimodal data to obtain multimodal features, including:

[0013] Data cleaning is performed on structured data to obtain cleaned structured data; data cleaning includes anomaly removal, linear interpolation completion, and standardization.

[0014] Time-frequency domain features are extracted from the cleaned structured data to obtain structured features;

[0015] By using feature extraction strategies that match the data types of unstructured data, features are extracted from unstructured data to obtain unstructured features. Data types include video streams, audio signals, and infrared images. The corresponding feature extraction strategies include frame filtering and resolution compression strategies, audio feature parameter extraction strategies, and image enhancement strategies.

[0016] Multimodal features corresponding to multimodal data are obtained based on structured and unstructured features.

[0017] In one possible implementation, a lightweight fusion model is used to predict the equipment status of multimodal features, resulting in equipment status prediction results for industrial equipment corresponding to multimodal data, including:

[0018] Multimodal fault semantic features are obtained by performing parallel feature transformation on multimodal features through a lightweight fusion model.

[0019] The multimodal fault semantic features are weighted and fused based on the fusion weights to obtain the fused features;

[0020] Fault prediction is performed based on fusion features to obtain equipment status prediction results for industrial equipment corresponding to multimodal data. The equipment status prediction results include equipment failure probability values ​​and confidence levels of equipment failure probability values.

[0021] Secondly, embodiments of this application provide an industrial multimodal data edge-cloud collaborative fusion processing method.

[0022] Cloud-based applications for industrial equipment include:

[0023] The edge node receives the multimodal features and equipment status prediction results of the industrial equipment at the current stage. The equipment status prediction results are obtained by the edge node through the multimodal features to predict the equipment status using a lightweight fusion model.

[0024] A joint training set is constructed based on multimodal features and device status prediction results;

[0025] Based on the joint training set, the pre-defined multimodal network model is optimized and trained to obtain the loss function value and model output results;

[0026] Based on the loss function value and the model output results, the model parameters of the preset multimodal network model are dynamically adjusted by backpropagation gradient to obtain the optimized model parameters;

[0027] When the model optimization parameters meet the convergence condition, the model optimization parameters are sent to the edge nodes. The model optimization parameters are used to update the model parameters of the lightweight fusion model. The updated lightweight fusion model is used to predict the equipment status of industrial equipment in the next stage.

[0028] In one possible implementation, the model parameters of the preset multimodal network model are the training weights, and the model optimization parameters are the optimized weights.

[0029] Based on the joint training set, a pre-defined multimodal network model is optimized and trained to obtain the loss function value and model output results, including:

[0030] The multimodal features from the joint training set are input into a pre-defined multimodal network model to obtain multi-path features;

[0031] The loss function is obtained based on the prediction results in the joint training set;

[0032] In each iteration of model optimization training, multi-path features are fused based on training weights to obtain training fused features, which are then used as the model output. The training weights are either the fusion weights of the lightweight fusion model or the optimized weights from the previous iteration.

[0033] In one possible implementation, the multimodal features include structured features and unstructured features, with the unstructured features including visual features and audio features;

[0034] The multimodal features from the joint training set are input into a pre-defined multimodal network model to obtain multi-path features, including:

[0035] The structured features are input into the structured branch of the preset multimodal network model to obtain the first path features;

[0036] The visual features are input into the visual branch of the preset multimodal network model to obtain the second-path features;

[0037] The audio features are input into the audio branch of a pre-defined multimodal network model to obtain the third-path features;

[0038] The first, second, and third features are concatenated to obtain multi-features.

[0039] In one possible implementation, when the model optimization parameters meet the convergence condition, the model optimization parameters are distributed to the edge nodes, including:

[0040] When the number of iterations for model optimization training equals the threshold, obtain the validation set corresponding to the joint training set;

[0041] The preset multimodal network model is optimized based on the model optimization parameters corresponding to the current iteration to obtain the optimized network model.

[0042] The accuracy of the validation set is obtained by validating the optimized network model on the validation set.

[0043] To determine the improvement in accuracy compared to the previous iteration;

[0044] When the improvement is greater than or equal to the magnitude threshold, the model optimization parameters are sent to the edge nodes to update the lightweight fusion model.

[0045] Thirdly, embodiments of this application provide an edge node, including:

[0046] The edge sensing module is used to acquire multimodal data of industrial equipment at the current stage;

[0047] The edge feature processing module is used to extract features from multimodal data to obtain multimodal features;

[0048] The edge inference module is used to predict the equipment status of multimodal features through a lightweight fusion model, and obtain the equipment status prediction results of the multimodal data corresponding to the industrial equipment. The model parameters of the lightweight fusion model are obtained by cloud-based model optimization training based on the equipment status prediction results of the industrial equipment in the previous stage and the multimodal features.

[0049] The edge communication module is used to send the device status prediction results and multimodal features corresponding to the current stage to the cloud, so as to load the model optimization parameters issued by the cloud;

[0050] The edge update module is used to update the model parameters of the lightweight fusion model based on the model optimization parameters.

[0051] Fourthly, embodiments of this application provide a cloud platform, including:

[0052] The cloud communication module is used to receive the multimodal features and equipment status prediction results of the industrial equipment at the current stage transmitted by the edge node of the industrial equipment. The equipment status prediction results are obtained by the edge node through the multimodal features to predict the equipment status using a lightweight fusion model.

[0053] The cloud-based data module is used to construct a joint training set based on multimodal features and device status prediction results;

[0054] The cloud-based training module is used to optimize and train a pre-defined multimodal network model based on a joint training set, and obtain the loss function value and model output results.

[0055] The cloud-based optimization module is used to dynamically adjust the model parameters of a preset multimodal network model through backpropagation gradients based on the loss function value and model output results, thereby obtaining optimized model parameters.

[0056] The cloud communication module is also used to send the model optimization parameters to the edge nodes when the model optimization parameters meet the convergence conditions. The model optimization parameters are used to update the model parameters of the lightweight fusion model. The updated lightweight fusion model is used to predict the equipment status of industrial equipment in the next stage.

[0057] Fifthly, embodiments of this application provide an industrial multimodal data edge-cloud collaborative fusion processing system, including the edge nodes as described in the third aspect and the cloud as described in the fourth aspect.

[0058] The industrial multimodal data edge-cloud collaborative fusion processing method and related equipment provided in this application acquire multimodal data of the industrial equipment at the current stage from the edge node corresponding to the industrial equipment; extract features from the multimodal data to obtain multimodal features; and use a lightweight fusion model to predict the equipment status of the multimodal features to obtain the equipment status prediction result corresponding to the multimodal data of the industrial equipment. The model parameters of the lightweight fusion model are obtained by cloud-based model optimization training based on the equipment status prediction result and multimodal features of the industrial equipment in the previous stage. The current stage's equipment status prediction result and multimodal features are sent to the cloud to load the model optimization parameters issued by the cloud, and the lightweight fusion model is updated based on the model optimization parameters. Thus, this application realizes a collaborative closed loop of low-latency edge intelligent decision-making and high-precision cloud continuous optimization of industrial multimodal data. While ensuring real-time performance, it improves the accuracy and overall robustness of equipment status monitoring through dynamic model updates and adaptive fusion. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0060] Figure 1 A schematic diagram of an implementation environment provided for this application;

[0061] Figure 2 A flowchart illustrating the industrial multimodal data edge-cloud collaborative fusion processing method provided in this application. Figure 1 ;

[0062] Figure 3 This application provides a schematic diagram of the edge node structure.

[0063] Figure 4 A schematic diagram of the internal structure of the edge sensing module provided in this application;

[0064] Figure 5 A functional flowchart of the edge feature processing module provided in this application;

[0065] Figure 6 A flowchart illustrating the industrial multimodal data edge-cloud collaborative fusion processing method provided in this application. Figure 2 ;

[0066] Figure 7 A schematic diagram of the cloud structure provided in this application;

[0067] Figure 8 A schematic diagram of the structure of the industrial multimodal data edge-cloud collaborative fusion processing system provided in this application;

[0068] Figure 9 A schematic diagram illustrating the connection between the resource scheduling module provided in this application and edge nodes and remote devices;

[0069] Figure 10 A schematic diagram of the dynamic scheduling strategy of the resource scheduling module provided in this application.

[0070] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] Figure 1 This is a schematic diagram of an implementation environment provided by this application. The implementation environment includes industrial equipment 10, edge nodes 20 deployed on industrial equipment 10, and a cloud 30 corresponding to industrial equipment.

[0073] Edge node 20 is used to acquire multimodal data of industrial equipment 10 in the current stage; extract features from the multimodal data to obtain multimodal features; perform equipment state prediction processing on the multimodal features through a lightweight fusion model to obtain the equipment state prediction result of the multimodal data corresponding to industrial equipment 10. The model parameters of the lightweight fusion model are obtained by cloud 30 through model optimization training based on the equipment state prediction result and multimodal features of industrial equipment 10 in the previous stage; send the equipment state prediction result and multimodal features corresponding to the current stage to cloud 30 to load the model optimization parameters issued by cloud 30, and update the model parameters of the lightweight fusion model based on the model optimization parameters.

[0074] In existing technologies, mainstream multimodal data processing solutions in industry still have significant shortcomings in terms of processing timeliness, fusion accuracy, resource utilization efficiency, model adaptability, and system robustness, making it difficult to meet the comprehensive requirements of intelligent manufacturing for real-time performance, accuracy, and reliability. The industrial multimodal data edge-cloud collaborative fusion processing method provided in this application, through a collaborative closed loop of low-latency edge intelligent decision-making and high-precision cloud-based continuous optimization of industrial multimodal data, as well as dynamic model updates and adaptive fusion, improves the accuracy and overall robustness of equipment status monitoring while ensuring real-time performance.

[0075] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0076] Figure 2 A flowchart illustrating the industrial multimodal data edge-cloud collaborative fusion processing method provided in this application. Figure 1 ,like Figure 2 As shown, the method includes:

[0077] S201. Acquire multimodal data of industrial equipment at the current stage.

[0078] In this embodiment, edge nodes are deployed on industrial equipment to collect and process multimodal data. In real-world industrial scenarios, such as when monitoring the gearbox status of a large wind farm, edge nodes can be deployed inside the wind turbine tower to collect multimodal data during equipment operation in real time. The industrial multimodal data edge-cloud collaborative fusion processing method provided in this application enables low-latency, high-precision fault prediction.

[0079] S202. Extract features from the multimodal data to obtain multimodal features.

[0080] In this embodiment, for different types of data in multimodal data, a feature extraction method that matches the data type can be adopted to provide an accurate data foundation for the fusion of multimodal features in the subsequent lightweight fusion model.

[0081] S203. The equipment status prediction is performed on the multimodal features through a lightweight fusion model to obtain the equipment status prediction results of the multimodal data corresponding to the industrial equipment. The model parameters of the lightweight fusion model are obtained by cloud-based model optimization training based on the equipment status prediction results and multimodal features of the industrial equipment in the previous stage.

[0082] Among them, the equipment status prediction results are used to characterize the failure probability of industrial equipment, so as to achieve failure prediction for industrial equipment.

[0083] In this embodiment, a lightweight fusion model for fault prediction is constructed and its parameters are optimized in the cloud. Specifically, the cloud performs model optimization training based on the equipment status prediction results and multimodal features of the industrial equipment in the previous stage to obtain the model parameters of the lightweight fusion model. The edge nodes use the lightweight fusion model to perform equipment status prediction processing on the multimodal features to obtain the equipment status prediction results of the multimodal data corresponding to the industrial equipment.

[0084] S204. Send the device status prediction results and multimodal features corresponding to the current stage to the cloud to load the model optimization parameters issued by the cloud, and update the model parameters of the lightweight fusion model based on the model optimization parameters.

[0085] After completing the fault prediction for this stage, the edge node will send the device status prediction results and multimodal features corresponding to the current stage to the cloud, load the model optimization parameters generated by the cloud based on the device status prediction results and multimodal features corresponding to the current stage, and update the model parameters of the lightweight fusion model based on the model optimization parameters for the next stage of fault prediction.

[0086] The industrial multimodal data edge-cloud collaborative fusion processing method provided in this application, on the one hand, achieves latency optimization, accuracy improvement, and efficient resource utilization in multimodal data processing by deploying a lightweight fusion model at edge nodes and combining it with the model optimization training and parameter feedback mechanism in the cloud. This defines a temporal relationship of "previous stage training, current stage prediction and upload, triggering the next stage update," making the evolution of the lightweight fusion model no longer a vague "continuous optimization," but a closed-loop learning process with clear causality and traceability. This enhances logical rigor and feasibility, ensuring absolutely low latency in edge response, unaffected by cloud training cycles. On the other hand, through dynamic optimization of the lightweight fusion model, the fusion accuracy of multimodal data in the model can be ensured, thereby improving the output accuracy of equipment status prediction results.

[0087] As can be seen, this application constructs an edge-cloud collaborative closed loop. At the edge, a lightweight model optimized based on historical data in the cloud is used to achieve real-time, low-latency state prediction of multimodal data of industrial equipment. Simultaneously, the currently generated prediction results and feature data are uploaded to the cloud to drive the next round of model iteration and optimization. This ensures real-time response capabilities at the edge while enabling continuous model evolution based on the latest field data, significantly improving the accuracy, adaptability, and overall engineering reliability of equipment status monitoring, while also optimizing the allocation of computing and communication resources between the cloud and the edge.

[0088] In actual wind farm testing, the industrial multimodal data edge-cloud collaborative fusion processing method provided in this application can control the end-to-end inference latency to within 120ms, achieve a fusion accuracy of 96.7%, which is 11.2% higher than the traditional single-modal method. At the same time, the edge resource utilization rate is increased by 35%, significantly reducing communication overhead and energy consumption. This fully verifies the effectiveness and practicality of the industrial multimodal data edge-cloud collaborative fusion processing method provided in this application in industrial multimodal data processing.

[0089] In an exemplary embodiment of this application, multimodal data includes structured data and unstructured data; therefore, when performing feature extraction on multimodal data, it is necessary to preprocess and extract features from both structured and unstructured data separately. Specific steps may include:

[0090] Data cleaning is performed on structured data to obtain cleaned structured data; data cleaning includes anomaly removal, linear interpolation completion, and standardization.

[0091] Time-frequency domain features are extracted from the cleaned structured data to obtain structured features;

[0092] By using feature extraction strategies that match the data types of unstructured data, features are extracted from unstructured data to obtain unstructured features. Data types include video streams, audio signals, and infrared images. The corresponding feature extraction strategies include frame filtering and resolution compression strategies, audio feature parameter extraction strategies, and image enhancement strategies.

[0093] Multimodal features corresponding to multimodal data are obtained based on structured and unstructured features.

[0094] Structured data can include formatted data such as temperature, vibration and pressure information transmitted by three types of sensor arrays: temperature, vibration and pressure. Unstructured data can include video streams, audio signals and infrared images collected and transmitted by devices such as industrial cameras, microphone arrays and infrared thermal imagers.

[0095] In this embodiment, preprocessing and feature extraction are performed on both structured and unstructured data. Specifically, for structured data, data cleaning is performed to obtain cleaned structured data. Data cleaning includes anomaly removal, linear interpolation completion, and standardization. Time-frequency domain features are extracted from the cleaned structured data to obtain structured features. For unstructured data, feature extraction strategies matching the data type of the unstructured data are used to extract features, resulting in unstructured features. Data types include video streams, audio signals, and infrared images, and the corresponding feature extraction strategies include frame filtering and resolution compression strategies, audio feature parameter extraction strategies, and image enhancement strategies. Finally, multimodal features corresponding to the multimodal data are obtained based on the structured and unstructured features.

[0096] In one embodiment of feature extraction for multimodal data, the mean μ and standard deviation σ are first calculated for the structured data, and outliers satisfying |x_i-μ|>3σ are removed. Then, missing values ​​are filled by linear interpolation of adjacent valid values. Next, Z-Score standardization is performed on the entire sequence to make the mean of each dimension 0 and the variance 1. Finally, frequency domain features are extracted by fast Fourier transform and combined with time domain statistics (mean, variance, peak factor) to generate a 256-dimensional feature vector X_s∈ℝ^{T×256}.

[0097] For unstructured data, the MobileNetV3 backbone network is used for inter-frame motion detection in the video stream. The optical flow amplitude of adjacent frames is calculated, and only frames with amplitudes greater than the threshold T_v=0.15 are retained, while the rest are discarded. At the same time, the resolution is downsampled to 1280×720, and the frame rate is reduced to 15fps. The 13-dimensional MFCC (Mel-Frequency Cepstral Coefficients) are extracted from the audio signal and converted into a 128×128 complex spectrum A∈ℝ^{128×128} through short-time Fourier transform. The contrast-limited adaptive histogram equalization (CLAHE) algorithm is applied to the infrared image. The contrast limit parameter is set to 2.0 and the grid size is 8×8. After enhancement, the image is input into a lightweight ViT for local hotspot encoding, and the final output visual feature V∈ℝ^{3×720×1280} is obtained.

[0098] Thus, through the above embodiments, this application performs robust cleaning and time-frequency domain transformation, including anomaly handling, on structured data to capture critical equipment status signals. For unstructured video, audio, and infrared image data, dedicated strategies adapted to their data characteristics, such as frame filtering, audio parameterization, and image enhancement, are employed for information condensation and quality improvement. This not only improves the reliability and information density of input features from the source, laying a high-quality data foundation for subsequent fusion and decision-making, but also achieves effective alignment of heterogeneous data through a unified feature interface, ensuring the accuracy and stability of multimodal intelligent analysis in complex industrial environments.

[0099] In an exemplary embodiment of this application, the step of performing equipment state prediction processing on multimodal features using a lightweight fusion model to obtain equipment state prediction results for multimodal data corresponding to industrial equipment may specifically include:

[0100] Multimodal fault semantic features are obtained by performing parallel feature transformation on multimodal features through a lightweight fusion model.

[0101] The multimodal fault semantic features are weighted and fused based on the fusion weights to obtain the fused features;

[0102] Fault prediction is performed based on fusion features to obtain equipment status prediction results for industrial equipment corresponding to multimodal data. The equipment status prediction results include equipment failure probability values ​​and confidence levels of equipment failure probability values.

[0103] In this embodiment, the lightweight fusion model has multiple branches for parallel processing of feature transformation of multimodal features to obtain multimodal fault semantic features. Then, the multimodal fault semantic features are weighted and fused based on the fusion weight to obtain fused features. Finally, fault prediction is performed based on the fused features to obtain the equipment status prediction result of the multimodal data corresponding to the industrial equipment. The equipment status prediction result includes the equipment fault probability value and the confidence level of the equipment fault probability value.

[0104] In one embodiment, after inputting multimodal features into a lightweight fusion model, the 1DCNN branch of the lightweight fusion model processes structured features X_s, outputting H_s∈ℝ^{256}; for unstructured features, the MobileViT branch processes video and infrared features V, outputting H_v∈ℝ^{512}; and a 1-layer Transformer branch processes the audio spectrogram A, outputting H_a∈ℝ^{256}. Then, the three features are weighted and fused through a Fusion Bottleneck layer. The initial weights can be set to α=0.4, β=0.4, and γ=0.2, satisfying α+β+γ=1 and each component ∈[0,1], resulting in the fused feature F_fusion=α·H_s+β·H_v+γ·H_a. Finally, the fully connected layer outputs the device failure probability ŷ and the corresponding confidence level, such as the MAE (Mean Absolute Error) value.

[0105] Thus, through the above embodiments, this application achieves deep abstraction from raw features to high-level fault semantics at the edge level via a three-stage processing framework of "parallel feature transformation - adaptive weighted fusion - fault probability prediction." Parallel transformation ensures independent deep encoding of each modality's features, while adaptive fusion weights dynamically evaluate and integrate the contributions of different modalities to the current fault diagnosis. Ultimately, it not only outputs intuitive fault probability values ​​but also provides confidence levels characterizing the reliability of the prediction. This significantly improves the semantic understanding, decision interpretability, and result reliability of edge-side fault diagnosis while ensuring model lightweightness, providing core technical support for achieving accurate and reliable real-time health assessment of industrial equipment.

[0106] Figure 3 The structural diagram of the edge node provided in this application is as follows: Figure 3 As shown, edge node 30 includes:

[0107] Edge sensing module 301 is used to acquire multimodal data of industrial equipment at the current stage;

[0108] The edge feature processing module 302 is used to extract features from multimodal data to obtain multimodal features;

[0109] The edge inference module 303 is used to predict the equipment status of multimodal features through a lightweight fusion model, and obtain the equipment status prediction results of the multimodal data corresponding to the industrial equipment. The model parameters of the lightweight fusion model are obtained by cloud-based model optimization training based on the equipment status prediction results of the industrial equipment in the previous stage and the multimodal features.

[0110] Edge communication module 304 is used to send the device status prediction results and multimodal features corresponding to the current stage to the cloud, so as to load the model optimization parameters issued by the cloud;

[0111] Edge update module 305 is used to update the model parameters of the lightweight fusion model based on the model optimization parameters.

[0112] In an exemplary embodiment of this application, the edge sensing module 301 may include a structured sensing unit and an unstructured acquisition unit. Figure 4 This is a schematic diagram of the internal structure of the edge sensing module provided in this application. Figure 4 As shown, the structured sensing unit connects to three sensor arrays—temperature, vibration, and pressure—via the Modbus TCP protocol, continuously acquiring IEEE 754 floating-point format data at a sampling frequency of 1kHz. The unstructured acquisition unit includes an industrial camera, a microphone array, and an infrared thermal imager. The industrial camera outputs a 1920×1080@30fps video stream using the GigE Vision protocol, the microphone array outputs a 48kHz audio signal via the I²S bus, and the infrared thermal imager outputs 640×480 resolution thermal image data via a USB 3.0 interface. It should be noted that the arrows in the diagram indicate relationships, not data flow directions.

[0113] The edge sensing module 301 acquires structured and unstructured data of industrial equipment through the structured sensing unit and the unstructured acquisition unit. All structured and unstructured data are transmitted to the edge feature processing module 302 via the PCIe 3.0 bus.

[0114] In one possible implementation, the edge feature processing module 302 is further used to perform data cleaning on the structured data to obtain cleaned structured data; the data cleaning process includes anomaly removal, linear interpolation completion, and standardization; time-frequency domain feature extraction is performed on the cleaned structured data to obtain structured features; features are extracted from the unstructured data using a feature extraction strategy that matches the data type of the unstructured data to obtain unstructured features; the data types include video streams, audio signals, and infrared images, and the corresponding feature extraction strategies include frame filtering and resolution compression strategies, audio feature parameter extraction strategies, and image enhancement strategies; multimodal features corresponding to the multimodal data are obtained based on the structured and unstructured features.

[0115] Figure 5 This is a functional flowchart of the edge feature processing module provided in this application. Figure 5 As shown, the edge feature processing module includes a structured cleaning submodule and an unstructured compression submodule. The structured cleaning submodule is used to remove anomalies, fill in missing values, perform Z-Score normalization, and extract FFT features from structured data. The unstructured compression submodule is used to perform video stream processing, audio signal processing, and infrared image processing for unstructured data.

[0116] In other embodiments, the edge node may also include an edge cache unit. Multimodal features are sent to the edge inference module 303 via a shared memory interface and written to the edge cache unit via the NVMe protocol.

[0117] In one possible implementation, the edge inference module 303 is further configured to perform parallel feature transformation on the multimodal features using a lightweight fusion model to obtain multimodal fault semantic features; perform weighted fusion of the multimodal fault semantic features based on fusion weights to obtain fused features; and perform fault prediction based on the fused features to obtain the equipment status prediction result for the multimodal data corresponding to the industrial equipment. The equipment status prediction result includes the equipment fault probability value and the confidence level of the equipment fault probability value. Simultaneously, the equipment status prediction result and the multimodal features are written together into the edge cache unit.

[0118] In one possible implementation, the edge communication module 304 is also used to perform lossless compression of the device status prediction results and multimodal features using the LZ4 compression algorithm, with a compression ratio of not less than 5:1. The compressed data is then uploaded to the cloud via the QoS 1 level of the MQTT-SN protocol to ensure at least one delivery.

[0119] Figure 6 A flowchart illustrating the industrial multimodal data edge-cloud collaborative fusion processing method provided in this application. Figure 2 ,like Figure 6 As shown, this method is applied to the cloud platform corresponding to industrial equipment, including:

[0120] S601. Receive the multimodal features and equipment status prediction results of the industrial equipment at the current stage transmitted by the edge node of the industrial equipment. The equipment status prediction results are obtained by the edge node through the prediction processing of the multimodal features on the equipment status using a lightweight fusion model.

[0121] In this embodiment, after the edge node completes the fault prediction of the industrial equipment in the current stage, it will transmit the multimodal features and equipment status prediction results of the industrial equipment in the current stage to the cloud. The remote end generates model optimization parameters based on the received multimodal features and equipment status prediction results to dynamically optimize the lightweight fusion model, so that the lightweight fusion model has strong adaptability.

[0122] S602. Based on the multimodal features and equipment status prediction results, a joint training set is constructed.

[0123] In this embodiment, after receiving the multimodal features and device status prediction results, the cloud constructs at least a joint training set and a corresponding validation set. The joint training set is used to optimize and train the preset multimodal network model, and the validation set is used to determine whether the training has converged.

[0124] S603. Based on the joint training set, perform model optimization training on the preset multimodal network model to obtain the loss function value and model output results.

[0125] S604. Based on the loss function value and the model output results, the model parameters of the preset multimodal network model are dynamically adjusted by backpropagation gradient to obtain the optimized model parameters.

[0126] In this embodiment, during backpropagation, the gradient pruning threshold can be set to 1.0 to stabilize training, and the exponential moving average (EMA) mechanism is used to smooth weight updates and improve model robustness.

[0127] S605. When the model optimization parameters meet the convergence condition, the model optimization parameters are sent to the edge nodes. The model optimization parameters are used to update the model parameters of the lightweight fusion model. The updated lightweight fusion model is used to predict the equipment status of industrial equipment in the next stage.

[0128] The industrial multimodal data edge-cloud collaborative fusion processing method provided in this application constructs a centralized, data-driven parameter optimization scheme in the cloud. It aggregates multimodal features and prediction results from multiple edge nodes to build a joint training set, which is then used to perform deep training and gradient optimization on a complex pre-defined multimodal network model. This not only overcomes the limitations of single-node data and computing power by utilizing global data and dynamically adjusting model parameters through backpropagation to extract more generalizable fault diagnosis knowledge, but also ensures, through set convergence conditions, that only fully optimized model parameters with significantly improved performance are distributed to the edge. Therefore, this application achieves the collaborative evolution and empowerment of edge models in a safe, reliable, and efficient manner, effectively reducing processing latency, improving fusion accuracy and system robustness, and realizing efficient resource utilization and adaptive model evolution.

[0129] In an exemplary embodiment of this application, the model parameters of the preset multimodal network model are the training weights, and the model optimization parameters are the optimized weights. Correspondingly, the step of performing model optimization training on the preset multimodal network model based on the joint training set to obtain the loss function value and the model output result may specifically include:

[0130] The multimodal features from the joint training set are input into a pre-defined multimodal network model to obtain multi-path features;

[0131] The loss function is obtained based on the prediction results in the joint training set;

[0132] In each iteration of model optimization training, multi-path features are fused based on training weights to obtain training fused features, which are then used as the model output. The training weights are either the fusion weights of the lightweight fusion model or the optimized weights from the previous iteration.

[0133] In this embodiment, a dynamically updatable training weight is introduced in each training iteration in the cloud to fuse multi-path features, and backpropagation is used to continuously optimize the weight, realizing a progressive and adaptive multimodal fusion learning mechanism. This enables the cloud to finely adjust the contribution ratio of different modalities in the final decision in real time according to the distribution characteristics of global data during the training process, thereby significantly improving the model's ability to model the inherent correlation of multi-source data in complex and variable industrial scenarios.

[0134] Thus, through the above embodiments, this application not only ensures that the cloud can extract the optimal fusion strategy from massive edge data, but also makes the model optimization parameters (i.e., the optimized weights) finally sent to the edge highly adaptable to the scene and have diagnostic accuracy, which is the foundation for ensuring the continuous improvement of the fusion performance of the entire edge-cloud collaborative system.

[0135] In another exemplary embodiment, the multimodal features include structured features and unstructured features, with unstructured features including visual features and audio features. The corresponding step of inputting the multimodal features from the joint training set into a preset multimodal network model to obtain multipath features may specifically include:

[0136] The structured features are input into the structured branch of the preset multimodal network model to obtain the first path features;

[0137] The visual features are input into the visual branch of the preset multimodal network model to obtain the second-path features;

[0138] The audio features are input into the audio branch of a pre-defined multimodal network model to obtain the third-path features;

[0139] The first, second, and third features are concatenated to obtain multi-features.

[0140] In this embodiment, a dedicated feature extraction branch (such as LSTM, Visual Transformer, Audio CNN-Transformer, etc.) is designed for each of the three types of features—structured, visual, and audio—within the preset multimodal network model. This includes a structured branch, a visual branch, and an audio branch, thereby forming a heterogeneous and parallel multimodal network architecture. The first, second, and third features obtained through the feature extraction branches are then concatenated to obtain multi-path features for feature fusion.

[0141] In one embodiment of this application, the preset multimodal network model uses a multimodal Transformer backbone network model built with the PyTorch 2.0 framework, employing the Adam optimizer with a learning rate of 0.001. The process of obtaining the loss function value and model output results through the preset multimodal network model can be as follows:

[0142] The structured branch is a bidirectional LSTM with 256-dimensional hidden layers and 2 layers. The input structured feature X_s outputs H_s∈ℝ^{256}.

[0143] The visual branch is MobileViT, with 8 attention heads and 512 embedding dimensions. After inputting video and infrared features V from unstructured features, the output is H_v∈ℝ^{512}.

[0144] The audio branch is a CNN-Transformer hybrid structure. After inputting the audio spectrogram feature A from the unstructured features, it outputs H_a∈ℝ^{256}.

[0145] After concatenating the multi-path features, they are linearly projected into a 1024-dimensional unified space, and then weighted and summed using trainable weight vectors w=[α,β,γ]^T to obtain the trained fusion feature F_fusion=α·H_s+β·H_v+γ·H_a.

[0146] During training, the loss function is jointly optimized using MAE and cross-entropy:

[0147]

[0148] Where y is the true label (such as "normal" or "faulty"), ŷ is the device fault probability value included in the prediction results of the lightweight fusion model, and λ1=0.6 and λ2=0.4 are the balance coefficients.

[0149] Thus, through the above embodiments, this application, by designing dedicated feature extraction branches for structured, visual, and audio features respectively, can fully mine and retain the most effective pattern information unique to each type of data, avoiding mutual interference and information loss between different modal data in the early processing stages. Subsequently, through feature concatenation operations, high-quality, complementary, and dimensionally aligned multi-path feature inputs are provided for the subsequent fusion layer. This ensures, from the model structure level, that deep, high-precision cross-modal information fusion and correlation mining can be performed in the cloud, providing a solid foundation for its superior performance.

[0150] In an exemplary embodiment of this application, the step of distributing the model optimization parameters to the edge nodes when the model optimization parameters meet the convergence condition may specifically include:

[0151] When the number of iterations for model optimization training equals the threshold, obtain the validation set corresponding to the joint training set;

[0152] The preset multimodal network model is optimized based on the model optimization parameters corresponding to the current iteration to obtain the optimized network model.

[0153] The accuracy of the validation set is obtained by validating the optimized network model on the validation set.

[0154] To determine the improvement in accuracy compared to the previous iteration;

[0155] When the improvement is greater than or equal to the magnitude threshold, the model optimization parameters are sent to the edge nodes to update the lightweight fusion model.

[0156] In this embodiment, when the number of iterations for model optimization training equals a threshold, the preset multimodal network model is optimized based on the model optimization parameters corresponding to the current iteration to obtain the optimized network model. The optimized network model is then used to validate the validation set, obtaining the accuracy of the validation set. Additionally, the unoptimized preset multimodal network model is used to validate the validation set, obtaining a reference value. This allows us to determine the improvement in accuracy between the current iteration and the previous iteration. When the improvement is greater than or equal to a threshold, the model optimization parameters are distributed to the edge nodes to update the lightweight fusion model. For example, after every 10 iterations, the validation set accuracy is calculated; when the improvement is ≥1%, the distribution of model optimization parameters is triggered.

[0157] Thus, through the above embodiments, this application evaluates the accuracy of the optimized model based on an independent validation set after training has reached a preset number of iterations, and rigorously calculates its improvement relative to the previous round. The update is only triggered when the improvement reaches or exceeds a preset threshold. This effectively avoids invalid or negative model updates caused by normal fluctuations or overfitting during training, ensuring that each update sent to the edge nodes is a fully validated model with significant performance improvements. This guarantees the continuous evolution of the lightweight fusion model at the edge while greatly improving the reliability, stability, and communication efficiency of the entire edge-cloud collaborative system update process, eliminating the risk of model degradation from a procedural standpoint.

[0158] Figure 7 The cloud structure diagram provided in this application is as follows: Figure 7 As shown, cloud 70 includes:

[0159] The cloud communication module 701 is used to receive the multimodal features and equipment status prediction results of the industrial equipment at the current stage transmitted by the edge node of the industrial equipment. The equipment status prediction results are obtained by the edge node through the multimodal features to predict the equipment status using a lightweight fusion model.

[0160] The cloud data module 702 is used to construct a joint training set based on multimodal features and device status prediction results;

[0161] The cloud training module 703 is used to optimize and train a preset multimodal network model based on a joint training set, and obtain the loss function value and model output results.

[0162] The cloud optimization module 704 is used to dynamically adjust the model parameters of the preset multimodal network model by backpropagation gradient based on the loss function value and the model output results, so as to obtain the optimized model parameters.

[0163] The cloud communication module 701 is also used to send the model optimization parameters to the edge nodes when the model optimization parameters meet the convergence conditions. The model optimization parameters are used to update the model parameters of the lightweight fusion model. The updated lightweight fusion model is used to predict the equipment status of industrial equipment in the next stage.

[0164] In one possible implementation, the model parameters of the preset multimodal network model are the training weights, and the model optimization parameters are the optimized weights; the cloud training module 703 is also used to input the multimodal features from the joint training set into the preset multimodal network model to obtain multi-path features; obtain the loss function based on the prediction results in the joint training set; in each iteration of the model optimization training, the multi-path features are fused based on the training weights to obtain training fused features, and the training fused features are used as the model output; the training weights are the fusion weights of the lightweight fusion model, or the optimized weights corresponding to the previous iteration.

[0165] In one possible implementation, the multimodal features include structured features and unstructured features, with unstructured features including visual features and audio features. The cloud training module 703 is also used to input structured features into the structured branch of the preset multimodal network model to obtain the first feature path; input visual features into the visual branch of the preset multimodal network model to obtain the second feature path; input audio features into the audio branch of the preset multimodal network model to obtain the third feature path; and concatenate the first, second, and third features to obtain the multimodal features.

[0166] In one possible implementation, the cloud optimization module 704 includes a Fusion Bottleneck weight optimization unit, a LayerNorm layer, and a residual block. The Fusion Bottleneck weight optimization unit is used to dynamically adjust the training weights α, β, and γ based on the backpropagation gradient to obtain the model optimization parameters, with strict constraints that α+β+γ=1 and each component ∈[0,1]. The LayerNorm layer normalizes the model optimization parameters and then feeds them into the residual block to avoid gradient vanishing in deep networks.

[0167] The model optimization parameters are transmitted from the cloud optimization module 704 to the cloud communication module 705 via the gRPC protocol, and then distributed through the downlink channel.

[0168] In one possible implementation, the cloud communication module 705 is further configured to: obtain the validation set corresponding to the joint training set when the number of iterations of model optimization training equals a threshold number; optimize the preset multimodal network model based on the model optimization parameters corresponding to the current iteration to obtain the optimized network model; validate the validation set using the optimized network model to obtain the accuracy of the validation set; obtain the improvement between the accuracy and the accuracy corresponding to the previous iteration; and when the improvement is greater than or equal to the magnitude threshold, send the model optimization parameters to the edge nodes to update the lightweight fusion model.

[0169] In one possible implementation, the cloud communication module 705 is also used to quantize the model optimization parameters to be distributed using INT8 to reduce their size, and then transmit them to the cache unit of the edge node via the CoAP protocol for reading, ensuring efficient updates in low-bandwidth environments.

[0170] Figure 8 A schematic diagram of the structure of the industrial multimodal data edge-cloud collaborative fusion processing system provided in this application is shown below. Figure 8 As shown, this includes the edge nodes and cloud corresponding to industrial equipment.

[0171] The industrial multimodal data edge-cloud collaborative fusion processing system provided in this embodiment can execute the methods provided in the above-mentioned method embodiments through the included edge nodes and remote terminals. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0172] In one possible implementation, the industrial multimodal data edge-cloud collaborative fusion processing system also includes a resource scheduling module. Figure 9 This is a schematic diagram showing the connection between the resource scheduling module provided in this application and the edge nodes and remote devices. Figure 9 In this system, the edge communication module and the cloud communication module form the edge-cloud communication unit; the resource scheduling module connects to the CPU and memory monitoring registers of the edge perception module, edge feature processing module, and edge inference module of the edge node through the Sysfs interface, and connects to the GPU load monitoring unit deployed in the cloud through the application layer protocol SNMP (Simple Network Management Protocol) to realize GPU load monitoring.

[0173] Figure 10 This is a schematic diagram illustrating the dynamic scheduling strategy of the resource scheduling module provided in this application. Figure 10 As shown, the resource scheduling module continuously monitors the system load at the edge node and reads the CPU and memory usage of the edge perception module, edge feature processing module, and edge inference module every 100ms through the Sysfs interface. When both are below 30%, additional video stream processing tasks are dynamically started, up to a maximum of 8 concurrent streams, to make full use of idle computing power.

[0174] In the cloud, the resource scheduling module monitors the memory usage of each GPU in the cloud training module via the SNMP protocol. When the memory usage of any GPU exceeds 80%, the current training task is migrated to the idle GPU with the lowest load, and a model parallel strategy is adopted to distribute different layers of Transformer to different devices to achieve load balancing.

[0175] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for collaborative fusion processing of industrial multimodal data edge-cloud, characterized in that, Edge nodes applied to industrial equipment include: Acquire multimodal data of the industrial equipment at the current stage; Feature extraction is performed on the multimodal data to obtain multimodal features; The equipment status prediction process of the multimodal features is performed by a lightweight fusion model to obtain the equipment status prediction result of the industrial equipment corresponding to the multimodal data. The model parameters of the lightweight fusion model are obtained by cloud-based model optimization training based on the equipment status prediction result of the industrial equipment in the previous stage and the multimodal features. The device status prediction result and multimodal features corresponding to the current stage are sent to the cloud to load the model optimization parameters issued by the cloud, and the model parameters of the lightweight fusion model are updated based on the model optimization parameters.

2. The method according to claim 1, characterized in that, The multimodal data includes structured and unstructured data; the feature extraction from the multimodal data to obtain multimodal features includes: The structured data is cleaned to obtain cleaned structured data; the data cleaning process includes anomaly removal, linear interpolation completion, and standardization. Time-frequency domain features are extracted from the cleaned structured data to obtain structured features; By using a feature extraction strategy that matches the data type of the unstructured data, features are extracted from the unstructured data to obtain unstructured features; the data types include video streams, audio signals, and infrared images, and the corresponding feature extraction strategies include frame filtering and resolution compression strategies, audio feature parameter extraction strategies, and image enhancement strategies; The multimodal features corresponding to the multimodal data are obtained based on the structured features and the unstructured features.

3. The method according to claim 1 or 2, characterized in that, The step of using a lightweight fusion model to predict the equipment status of the multimodal features, and obtaining the equipment status prediction result of the industrial equipment corresponding to the multimodal data, includes: The multimodal features are transformed in parallel using a lightweight fusion model to obtain multimodal fault semantic features; The multimodal fault semantic features are weighted and fused based on the fusion weights to obtain fused features; Based on the fusion features, fault prediction is performed to obtain the equipment status prediction result of the industrial equipment corresponding to the multimodal data. The equipment status prediction result includes the equipment fault probability value and the confidence level of the equipment fault probability value.

4. A method for collaborative fusion processing of industrial multimodal data edge-cloud, characterized in that, Cloud-based applications for industrial equipment include: The edge node of the industrial equipment receives the multimodal features and equipment state prediction results of the industrial equipment at the current stage, which are transmitted by the edge node. The equipment state prediction results are obtained by the edge node through the multimodal features to predict the equipment state using a lightweight fusion model. Based on the multimodal features and the device state prediction results, a joint training set is constructed; Based on the joint training set, a preset multimodal network model is optimized and trained to obtain the loss function value and the model output result; Based on the loss function value and the model output, the model parameters of the preset multimodal network model are dynamically adjusted by backpropagation gradient to obtain the optimized model parameters; When the model optimization parameters meet the convergence condition, the model optimization parameters are sent to the edge node. The model optimization parameters are used to update the model parameters of the lightweight fusion model. The updated lightweight fusion model is used to predict the equipment status of the industrial equipment in the next stage.

5. The method according to claim 4, characterized in that, The model parameters of the preset multimodal network model are the training weights, and the model optimization parameters are the optimized weights. The step of optimizing and training a preset multimodal network model based on the joint training set to obtain loss function values ​​and model output results includes: The multimodal features from the joint training set are input into a preset multimodal network model to obtain multi-path features; The loss function is obtained based on the prediction results in the joint training set; In each iteration of model optimization training, the multi-path features are fused based on the training weights to obtain training fused features, and the training fused features are used as the model output; the training weights are the fusion weights of the lightweight fusion model, or the optimized weights corresponding to the previous iteration.

6. The method according to claim 5, characterized in that, The multimodal features include structured features and unstructured features, and the unstructured features include visual features and audio features; The step of inputting the multimodal features from the joint training set into a preset multimodal network model to obtain multi-path features includes: The structured features are input into the structured branch of the preset multimodal network model to obtain the first path features; The visual features are input into the visual branch of the preset multimodal network model to obtain the second feature path; The audio features are input into the audio branch of the preset multimodal network model to obtain the third feature path; The first feature, the second feature, and the third feature are concatenated to obtain a multi-feature.

7. The method according to any one of claims 4 to 6, characterized in that, The step of sending the model optimization parameters to the edge nodes when the model optimization parameters meet the convergence condition includes: When the number of iterations for model optimization training equals the threshold, the validation set corresponding to the joint training set is obtained; The preset multimodal network model is optimized based on the model optimization parameters corresponding to the current iteration to obtain the optimized network model. The accuracy of the validation set is obtained by validating the validation set using the optimized network model. Obtain the improvement in accuracy compared to the accuracy corresponding to the previous iteration; When the improvement is greater than or equal to the magnitude threshold, the model optimization parameters are sent to the edge node to update the lightweight fusion model.

8. An edge node, characterized in that, The edge nodes include: The edge sensing module is used to acquire multimodal data of industrial equipment at the current stage; An edge feature processing module is used to extract features from the multimodal data to obtain multimodal features; The edge inference module is used to predict the equipment status of the multimodal features through a lightweight fusion model, and obtain the equipment status prediction result of the industrial equipment corresponding to the multimodal data. The model parameters of the lightweight fusion model are obtained by cloud-based model optimization training based on the equipment status prediction result of the industrial equipment in the previous stage and the multimodal features. The edge communication module is used to send the device status prediction result and multimodal features corresponding to the current stage to the cloud, so as to load the model optimization parameters issued by the cloud; The edge update module is used to update the model parameters of the lightweight fusion model based on the model optimization parameters.

9. A cloud platform, characterized in that, The cloud includes: The cloud communication module is used to receive the multimodal features and equipment state prediction results of the industrial equipment at the current stage transmitted by the edge node of the industrial equipment. The equipment state prediction results are obtained by the edge node through the multimodal features to predict the equipment state using a lightweight fusion model. The cloud data module is used to construct a joint training set based on the multimodal features and the device state prediction results; The cloud training module is used to perform model optimization training on the preset multimodal network model based on the joint training set, and obtain the loss function value and model output results; The cloud optimization module is used to dynamically adjust the model parameters of the preset multimodal network model by backpropagation gradient based on the loss function value and the model output result, so as to obtain the model optimization parameters; The cloud communication module is further configured to send the model optimization parameters to the edge node when the model optimization parameters meet the convergence condition. The model optimization parameters are used to update the model parameters of the lightweight fusion model. The updated lightweight fusion model is used to predict the equipment status of the industrial equipment in the next stage.

10. An industrial multimodal data edge-cloud collaborative fusion processing system, characterized in that, This includes the edge node as described in claim 8 and the cloud as described in claim 9.

Images