A cloud-edge collaborative computing framework and processing method for multi-modal data stream fusion processing

Abstract

The application relates to a cloud-edge collaborative computing framework and processing method for multi-modal data stream fusion processing, which comprises the following steps: S1: based on the original data stream collected by the edge computing node, an improved Wiener filtering algorithm is used for noise suppression, signal denoising is realized through adaptive threshold wavelet transform, and a time stamp alignment technology is used to solve the time delay difference problem of multi-modal data, and a space-time aligned and purified data stream is generated. The application has the effects that through the combination of the improved Wiener filtering algorithm and the adaptive threshold wavelet transform, the noise suppression efficiency of the original data stream is significantly improved, the time stamp alignment technology effectively solves the time delay difference of multi-modal data, the generation of the space-time aligned and purified data stream ensures that the subsequent processing has a unified time sequence reference, and the space-time attention fusion network adopts the collaborative architecture of a bidirectional gate recurrent unit and a lightweight 3D convolution network.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing, edge computing, and multimodal data processing technology, specifically to a cloud-edge collaborative computing framework and processing method for multimodal data stream fusion processing. Background Technology

[0002] The fields of cloud computing, edge computing, and multimodal data processing technology mainly research distributed computing architectures and heterogeneous data fusion methods. Their core lies in the collaborative work of high-performance computing resources in the cloud and low-latency processing capabilities at the edge to achieve efficient processing of multi-source heterogeneous data such as images, audio, and sensor data. Cloud computing provides elastic and scalable storage and computing resources, suitable for complex model training and large-scale data analysis; edge computing focuses on real-time preprocessing at the data generation end to reduce network transmission load; multimodal data processing focuses on cross-modal feature extraction, spatiotemporal alignment, and fusion algorithms. One such cloud-edge collaborative computing framework and method for multimodal data stream fusion processing refers to constructing a cloud-edge collaborative architecture, combining dynamic task scheduling algorithms with a multimodal fusion model to achieve efficient processing of multi-source data streams. This method employs a hierarchical computing strategy, performing data preprocessing and feature extraction at the edge, while the cloud handles complex model inference and global optimization, automatically allocating computing tasks through a load balancing mechanism.

[0003] Existing cloud computing and edge computing collaborative frameworks have significant limitations in multimodal data processing. Static task allocation strategies cannot adapt to dynamically changing computing loads, leading to task backlogs on edge nodes during high-load scenarios and idle cloud resources during low-load periods, resulting in large fluctuations in resource utilization. Traditional multimodal fusion methods rely on simple feature concatenation and lack cross-modal semantic alignment mechanisms, resulting in high feature matching error rates in video and sensor data fusion scenarios, affecting the accuracy of subsequent analysis. Load prediction models often employ single time-series algorithms, which show significant prediction biases for sudden load fluctuations. Task classification relies on fixed thresholds and cannot be dynamically adjusted according to real-time resource status, causing high-priority tasks to exceed latency limits. Summary of the Invention

[0004] To address the significant limitations of existing cloud computing and edge computing collaborative frameworks in multimodal data processing, where static task allocation strategies cannot adapt to dynamically changing computing loads, leading to task backlogs on edge nodes under high load scenarios and idle cloud resources during low load periods, resulting in large fluctuations in resource utilization, this application provides a cloud-edge collaborative computing framework and processing method for multimodal data stream fusion processing.

[0005] This application provides a cloud-edge collaborative computing framework and processing method for multimodal data stream fusion processing, which adopts the following technical solution:

[0006] A cloud-edge collaborative computing framework method for multimodal data stream fusion processing includes the following steps:

[0007] S1: Based on the raw data stream collected by the edge computing node, the improved Wiener filtering algorithm is used for noise suppression, the signal is denoised by adaptive threshold wavelet transform, and the time stamp alignment technology is used to solve the problem of multimodal data delay difference, generating a spatiotemporally aligned and purified data stream;

[0008] S2: Based on spatiotemporal alignment, the data stream is purified by adopting a spatiotemporal attention fusion network. In the time dimension, a bidirectional gated recurrent unit is used to extract temporal features, and in the spatial dimension, a lightweight 3D convolutional network is used to extract spatial features. Feature alignment is achieved through a cross-attention mechanism. Finally, a gated fusion unit is used to generate a unified representation and generate a multimodal feature tensor.

[0009] S3: Based on multimodal feature tensors, an improved Prophet-LSTM hybrid prediction algorithm is adopted, combined with real-time resource monitoring data, to predict future computing load change trends. The task is divided into real-time tasks and batch computing tasks through a fuzzy logic decision tree, generating a task segmentation decision matrix.

[0010] S4: Based on the task segmentation decision matrix, a dynamically weighted round-robin algorithm optimized by federated learning is used to construct a scheduling strategy with three dimensions: real-time task priority weight, edge node computing power index, and network transmission quality score, generating a resource optimization scheduling graph;

[0011] S5: Based on the operational data of the resource optimization scheduling graph, the elastic weight solidification algorithm is used to perform local incremental learning at the edge. The optimized model structure is generated in the cloud through neural architecture search. The model update package volume is reduced by differential compression technology to generate an adaptive incremental model package.

[0012] S6: Based on the adaptive incremental model package and resource optimization scheduling graph, a digital twin system enhanced by Monte Carlo tree search is used to construct a simulation environment with three levels: resource mirroring layer, task inference layer, and decision optimization layer, and generate resource pre-allocation optimization scheme.

[0013] Preferably, the generation of spatiotemporally aligned and cleaned data streams based on S1 includes the following steps:

[0014] S101: Based on the original multimodal data stream, an improved median filtering algorithm is used to eliminate impulse noise in the video data. The size of the filtering window is dynamically adjusted through an adaptive threshold selection mechanism to generate a video cleaned frame sequence.

[0015] S102: Based on the timestamps of the video cleanup frame sequence, a dynamic time warping algorithm is used to align the audio and sensor data streams. Cubic spline interpolation is used to solve the sampling rate difference problem and generate a time-domain aligned data stream.

[0016] S103: Based on the temporal-domain aligned data stream, the Z-score standardization method is used to unify the data dimensions of each modality, and the sliding window mechanism is combined to realize data segmentation standardization and generate a spatiotemporally aligned and purified data stream.

[0017] Preferably, generating multimodal feature tensors based on S2 includes the following steps:

[0018] S201: Based on spatiotemporal alignment, clean the data stream by using a bidirectional gated recurrent unit network to extract the temporal features of video / audio data, and use an attention mechanism to focus on key time nodes to generate a temporal feature vector.

[0019] S202: Based on spatiotemporal alignment, clean the data stream, use a lightweight 3D convolutional network to extract spatial features, expand the receptive field through dilated convolution, and generate a spatial feature matrix;

[0020] S203: Based on temporal feature vectors and spatial feature matrices, a cross-attention alignment algorithm is used to achieve feature dimension matching. Cross-modal association is established through cosine similarity calculation to generate aligned feature tensors.

[0021] S204: Based on the aligned feature tensor, a gated fusion unit is used to achieve feature weighted fusion, and L2 regularization is used to prevent overfitting, generating a multimodal feature tensor.

[0022] Preferably, the generation task segmentation decision matrix based on S3 includes the following steps:

[0023] S301: Based on multimodal feature tensors, an improved Prophet algorithm is used to predict future computing load trends. Data fluctuations are processed through exponential smoothing to generate a load prediction curve.

[0024] S302: Based on the load prediction curve, a fuzzy logic decision tree is used to divide the task types, and a genetic algorithm is used to optimize the decision threshold to generate a task segmentation decision matrix.

[0025] Preferably, generating a resource optimization scheduling graph based on S4 includes the following steps:

[0026] S401: Based on the task segmentation decision matrix, an evaluation system including computing power, network quality, and task urgency is constructed using the analytic hierarchy process. Consistency checks are used to ensure that the weights are reasonable, and three-dimensional evaluation indicators are generated.

[0027] S402: Based on three-dimensional evaluation metrics, the scheduling strategy is trained using the federated Q-learning algorithm, and the initial scheduling strategy is generated by balancing exploration and utilization through the ε-greedy exploration mechanism.

[0028] S403: Based on the initial scheduling strategy, a dynamic weighted round-robin algorithm is used to allocate computing tasks, and the weights are adjusted in real time through sliding window statistics to generate a dynamic scheduling table;

[0029] S404: Based on a dynamic scheduling table, it uses containerization encapsulation technology to package computing tasks, manages the task execution order through a priority queue, and generates containerized task packages;

[0030] S405: Based on the runtime data of containerized task packages, the scheduling parameters are adjusted using the gradient descent optimization algorithm, and multi-objective optimization is achieved through Pareto front analysis to generate a resource optimization scheduling map.

[0031] Preferably, the S5-based adaptive incremental model generation package includes the following steps:

[0032] S501: Based on the resource optimization scheduling graph, the parameter importance matrix is ​​calculated using the elastic weight solidification algorithm, the parameter sensitivity is evaluated through the Fisher information matrix, and a parameter protection matrix is ​​generated.

[0033] S502: Based on the parameter protection matrix, it uses knowledge distillation technology to compress the cloud model and adjusts the knowledge transfer intensity through temperature coefficient to generate a lightweight model architecture.

[0034] S503: Based on a lightweight model architecture, it uses a differential parameter compression algorithm to generate update packages, and ensures compatibility through a version control mechanism to generate adaptive incremental model packages.

[0035] Preferably, the S6-based resource pre-allocation optimization scheme includes the following steps:

[0036] S601: Based on the adaptive incremental model package, a digital twin is constructed using the state-space modeling method, and state estimation is achieved through Kalman filtering to generate a virtual mirror system;

[0037] S602: Based on a virtual mirror system, it uses a discrete event simulation engine to simulate the task execution process, evaluates different scheduling schemes through Monte Carlo sampling, and generates a scheme evaluation report.

[0038] S603: Based on the scheme evaluation report, a multi-objective genetic algorithm is used to optimize the parameter combination, and the optimal solution set is selected by crowding comparison to generate a parameter optimization scheme;

[0039] S604: Based on the parameter optimization scheme, a dynamic feedback mechanism is used to adjust the resource scheduling strategy, and the system stability is achieved by controlling PID regulation, generating a resource pre-allocation optimization scheme.

[0040] A cloud-edge collaborative computing framework for multimodal data stream fusion processing, comprising any one of the cloud-edge collaborative computing framework processing methods for multimodal data stream fusion processing according to claims 1-7.

[0041] In summary, this application includes at least one of the following beneficial technical effects:

[0042] By combining an improved Wiener filtering algorithm with adaptive threshold wavelet transform, the noise suppression efficiency of the original data stream is significantly improved. Timestamp alignment technology effectively addresses the latency differences in multimodal data. Spatiotemporal alignment purifies the generated data stream, ensuring a unified temporal benchmark for subsequent processing. The spatiotemporal attention fusion network employs a collaborative architecture of bidirectional gated recurrent units and lightweight 3D convolutional networks. It captures long-term and short-term dependencies in the temporal dimension and extracts multi-scale features in the spatial dimension. A cross-attention mechanism achieves cross-modal feature alignment. The gated fusion unit optimizes feature representation through dynamic weight allocation. The construction of multimodal feature tensors enhances the semantic relevance of the data. The improved Prophet-LSTM hybrid prediction algorithm integrates the advantages of temporal prediction and deep learning, dynamically adjusting the prediction based on real-time resource monitoring data. The test model utilizes a fuzzy logic decision tree to achieve intelligent task segmentation based on a multi-dimensional rule set. The generation of the task segmentation decision matrix improves task classification accuracy. The federated learning-optimized dynamic weighted polling algorithm integrates real-time task priority, node computing power, and network quality in three dimensions. Resource scheduling optimization is achieved through a dynamic weight update mechanism. The construction of a resource optimization scheduling graph reduces task allocation bias. The elastic weight solidification algorithm retains key parameter memory at the edge. Neural architecture search automatically generates efficient model structures in the cloud. Differential compression technology reduces the amount of data for model updates. Adaptive incremental model packages support continuous model optimization. The Monte Carlo tree search-enhanced digital twin system uses three-layer simulation—resource mirroring, task inference, and decision optimization—to effectively predict and avoid potential resource conflicts through resource pre-allocation optimization schemes. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the main steps of the present invention;

[0044] Figure 2 This is a detailed schematic diagram of S1 of the present invention;

[0045] Figure 3 This is a detailed schematic diagram of S2 of the present invention;

[0046] Figure 4 This is a detailed schematic diagram of S3 of the present invention;

[0047] Figure 5 This is a detailed schematic diagram of S4 of the present invention;

[0048] Figure 6This is a detailed schematic diagram of S5 of the present invention;

[0049] Figure 7 This is a detailed schematic diagram of S6 of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] The following is in conjunction with the appendix Figures 1-7 This application will be described in further detail.

[0052] See Figure 1 A cloud-edge collaborative computing framework method for multimodal data stream fusion processing includes the following steps:

[0053] S1: Based on the raw data stream collected by the edge computing node, the improved Wiener filtering algorithm is used for noise suppression, the signal is denoised by adaptive threshold wavelet transform, and the time stamp alignment technology is used to solve the problem of multimodal data delay difference, generating a spatiotemporally aligned and purified data stream;

[0054] S2: Based on spatiotemporal alignment, the data stream is purified by adopting a spatiotemporal attention fusion network. In the time dimension, a bidirectional gated recurrent unit is used to extract temporal features, and in the spatial dimension, a lightweight 3D convolutional network is used to extract spatial features. Feature alignment is achieved through a cross-attention mechanism. Finally, a gated fusion unit is used to generate a unified representation and generate a multimodal feature tensor.

[0055] S3: Based on multimodal feature tensors, an improved Prophet-LSTM hybrid prediction algorithm is adopted, combined with real-time resource monitoring data, to predict future computing load change trends. The task is divided into real-time tasks and batch computing tasks through a fuzzy logic decision tree, generating a task segmentation decision matrix.

[0056] S4: Based on the task segmentation decision matrix, a dynamically weighted round-robin algorithm optimized by federated learning is used to construct a scheduling strategy with three dimensions: real-time task priority weight, edge node computing power index, and network transmission quality score, generating a resource optimization scheduling graph;

[0057] S5: Based on the operational data of the resource optimization scheduling graph, the elastic weight solidification algorithm is used to perform local incremental learning at the edge. The optimized model structure is generated in the cloud through neural architecture search. The model update package volume is reduced by differential compression technology to generate an adaptive incremental model package.

[0058] S6: Based on the adaptive incremental model package and resource optimization scheduling graph, a digital twin system enhanced by Monte Carlo tree search is used to construct a simulation environment with three levels: resource mirroring layer, task inference layer, and decision optimization layer, and generate resource pre-allocation optimization scheme.

[0059] By combining an improved Wiener filtering algorithm with adaptive threshold wavelet transform, the noise suppression efficiency of the original data stream is significantly improved. Timestamp alignment technology effectively addresses the latency differences in multimodal data. Spatiotemporal alignment purifies the generated data stream, ensuring a unified temporal benchmark for subsequent processing. The spatiotemporal attention fusion network employs a collaborative architecture of bidirectional gated recurrent units and lightweight 3D convolutional networks. It captures long-term and short-term dependencies in the temporal dimension and extracts multi-scale features in the spatial dimension. A cross-attention mechanism achieves cross-modal feature alignment. The gated fusion unit optimizes feature representation through dynamic weight allocation. The construction of multimodal feature tensors enhances the semantic relevance of the data. The improved Prophet-LSTM hybrid prediction algorithm integrates the advantages of temporal prediction and deep learning, dynamically adjusting the prediction based on real-time resource monitoring data. The test model utilizes a fuzzy logic decision tree to achieve intelligent task segmentation based on a multi-dimensional rule set. The generation of the task segmentation decision matrix improves task classification accuracy. The federated learning-optimized dynamic weighted polling algorithm integrates real-time task priority, node computing power, and network quality in three dimensions. Resource scheduling optimization is achieved through a dynamic weight update mechanism. The construction of a resource optimization scheduling graph reduces task allocation bias. The elastic weight solidification algorithm retains key parameter memory at the edge. Neural architecture search automatically generates efficient model structures in the cloud. Differential compression technology reduces the amount of data for model updates. Adaptive incremental model packages support continuous model optimization. The Monte Carlo tree search-enhanced digital twin system uses three-layer simulation—resource mirroring, task inference, and decision optimization—to effectively predict and avoid potential resource conflicts through resource pre-allocation optimization schemes.

[0060] See Figure 2 The generation of spatiotemporally aligned and cleaned data streams based on S1 includes the following steps:

[0061] S101: Based on the original multimodal data stream, an improved median filtering algorithm is used to eliminate impulse noise in the video data. The size of the filtering window is dynamically adjusted through an adaptive threshold selection mechanism to generate a video cleaned frame sequence.

[0062] Based on the original multimodal data stream, the video data is processed using an improved median filtering algorithm. In intelligent traffic monitoring scenarios, vehicle motion videos captured by cameras are often affected by random impulse noise. A sliding window is used to traverse the pixel matrix. The initial window size is set to 3×3. When the proportion of noise pixels in 5 consecutive frames exceeds a set threshold, the window automatically expands to 5×5. The proportion of noise pixels is calculated by counting the number of pixels in each frame whose difference from the neighborhood mean exceeds a set deviation value. In rainy or foggy weather, for local noise caused by water droplets on the windshield, the algorithm prioritizes using small window filtering for high-motion areas and large window filtering for static background areas. After the audio data is timestamped, it is synchronously transmitted to the edge nodes along with the video cleaned frame sequence. Outliers in the sensor data are removed using the interquartile range method. Discrete data collected by the temperature sensor is filled with missing values ​​through linear interpolation, ultimately generating a continuous and smooth video cleaned frame sequence.

[0063] S102: Based on the timestamps of the video cleanup frame sequence, a dynamic time warping algorithm is used to align the audio and sensor data streams. Cubic spline interpolation is used to solve the sampling rate difference problem and generate a time-domain aligned data stream.

[0064] Based on the timestamps of the video cleanup frame sequence, during the audio data stream alignment process, in industrial equipment monitoring scenarios, there is a sampling rate difference between the equipment vibration audio signal and the infrared thermal imaging video. The dynamic time warping algorithm calculates the optimal matching path between audio frames and video frames by constructing a cumulative cost matrix. During path search, the width of the curved window is limited to 1 / 3 of the video frame rate to prevent excessive distortion of the temporal relationship. The current value sequence in the sensor data is resampled through cubic spline interpolation. The interpolation node spacing is dynamically adjusted according to the data change rate. A larger spacing is used for stable data segments, and dense interpolation is used for abrupt data segments. In the wind turbine monitoring case, the alignment error between the vibration sensor data and the video frame is controlled within milliseconds. The consistency of the time-domain aligned multimodal data stream is verified through a timestamp verification mechanism to generate a time-domain aligned data stream.

[0065] S103: Based on the temporal-domain aligned data stream, the Z-score standardization method is used to unify the data dimensions of each modality, and the sliding window mechanism is combined to realize data segmentation standardization and generate a spatiotemporally aligned and purified data stream.

[0066] Based on time-domain aligned data streams, in the smart agriculture greenhouse monitoring scenario, the dimensions of multimodal sensor data such as temperature, humidity, and light intensity differ significantly. During Z-score standardization, the sliding window length is set according to the data sampling frequency. Data collected 60 times per minute uses a 60-point window. In the standardization calculation, the mean and standard deviation of the data within the window are updated in real time. When the light intensity data changes abruptly, the window is automatically divided into the segment before and after the change and processed separately. Due to the low sampling frequency of soil moisture sensor data, overlapping windows are used to supplement data points, with the window overlap rate set to 50%. After data segmentation and standardization, the distribution range of each modality data converges to the range of [-1,1]. Through the sliding window mechanism, dynamic data normalization is achieved, generating a time-space aligned and purified data stream.

[0067] See Figure 3 The generation of multimodal feature tensors based on S2 includes the following steps:

[0068] S201: Based on spatiotemporal alignment, clean the data stream by using a bidirectional gated recurrent unit network to extract the temporal features of video / audio data, and use an attention mechanism to focus on key time nodes to generate a temporal feature vector.

[0069] Based on spatiotemporal alignment and data stream purification, this system processes vehicle trajectory video and radar signal data in intelligent transportation scenarios. The forward and backward layers of the bidirectional gated recurrent unit network capture past and future features of the time series, respectively. When updating the hidden layer state, the system combines input gate and forget gate mechanisms. The input gate controls the proportion of new information intake, and the forget gate adjusts the degree of historical memory retention. The attention mechanism calculates the similarity score of feature vectors at each time step and assigns higher weights to the time segment before the emergency braking event. In factory equipment monitoring scenarios, for bearing vibration audio signals, the network automatically enhances the feature representation of abnormal frequency ranges. The temporal feature vectors are stacked through multiple layers of network output to achieve multi-scale feature extraction.

[0070] S202: Based on spatiotemporal alignment, clean the data stream, use a lightweight 3D convolutional network to extract spatial features, expand the receptive field through dilated convolution, and generate a spatial feature matrix;

[0071] Based on spatiotemporal alignment for data stream purification, this application processes surveillance videos in retail customer flow analysis scenarios. A lightweight 3D convolutional network employs a channel-separated convolutional structure to reduce the number of parameters. Spatial convolutional kernels focus on single-frame feature extraction, while temporal convolutional kernels capture inter-frame motion changes. Dilated convolutions set dilation coefficients in pedestrian density detection, using smaller dilation rates to preserve detailed features in low-density regions and increasing dilation rates to expand the receptive field in high-density regions. In medical image processing scenarios, for spatial feature extraction from CT video sequences, the network automatically adjusts the convolutional kernel stride, using fine-grained strides for lesion areas and coarse-grained strides for normal tissue areas. The spatial feature matrix is ​​multi-resolution represented by multi-branch feature stitching.

[0072] S203: Based on temporal feature vectors and spatial feature matrices, a cross-attention alignment algorithm is used to achieve feature dimension matching. Cross-modal association is established through cosine similarity calculation to generate aligned feature tensors.

[0073] Based on temporal feature vectors and spatial feature matrices, this algorithm integrates infrared thermal imaging and visible light video data in smart security scenarios. A cross-attention alignment algorithm is used to construct a mapping relationship between video modal query vectors and sensor modal key vectors. By using a projection matrix, features of different dimensions are transformed into the same latent space. During similarity calculation, temperature sensor data and image regions are associated and matched. In smart agriculture scenarios, when soil moisture sensor data is aligned with crop growth video features, the algorithm automatically ignores interference areas caused by drastic changes in light intensity. The alignment feature tensor achieves cross-modal association through weighted splicing of feature channels.

[0074] S204: Based on the aligned feature tensor, a gated fusion unit is used to achieve feature weighted fusion, and L2 regularization is used to prevent overfitting, generating a multimodal feature tensor.

[0075] Based on the aligned feature tensor, X-ray inspection images and ultrasonic sensing data are fused in industrial quality inspection scenarios. The gated fusion unit adopts a dual-channel sigmoid gate structure. The gate value of the video feature channel is dynamically adjusted according to the image clarity, while the gate value of the sensor feature channel depends on the signal stability. During the fusion process, a larger fusion coefficient is assigned to high-confidence features. When the L2 regularization term is applied to the loss function, the regularization strength is dynamically adjusted according to the number of training iterations. In the initial stage, weaker constraints are used to promote feature learning, while stronger constraints are used in the later stage to prevent overfitting. The multimodal feature tensor retains the original feature information through residual connections.

[0076] See Figure 4 The generation of the task segmentation decision matrix based on S3 includes the following steps:

[0077] S301: Based on multimodal feature tensors, an improved Prophet algorithm is used to predict future computing load trends. Data fluctuations are processed through exponential smoothing to generate a load prediction curve.

[0078] Based on multimodal feature tensors, in the scenario of intelligent factory equipment monitoring, the improved Prophet algorithm integrates a temporal convolutional network module to process the fusion features of vibration sensors and thermal imaging videos. When calculating the trend term, prior knowledge of the equipment operation cycle is introduced, the seasonal term decomposition takes into account the alternation of production shifts, the residual term analysis adopts sliding window outlier detection, and the smoothing coefficient is dynamically adjusted according to the feature fluctuation amplitude during exponential smoothing. Sudden load fluctuations are reduced in impact on the prediction results through an adaptive attenuation factor. In the video stream analysis scenario, the algorithm automatically identifies the trend of changes in pedestrian density, and the smoothed load prediction curve is generated as a continuous curve through cubic spline interpolation.

[0079] S302: Based on the load prediction curve, a fuzzy logic decision tree is used to divide the task types, and a genetic algorithm is used to optimize the decision threshold to generate a task segmentation decision matrix.

[0080] Based on load prediction curves, in smart city traffic management scenarios, the fuzzy logic decision tree construction includes three input dimensions (calculation load level, task delay tolerance, and data correlation), defines a fuzzy set (low / medium / high) for task urgency, adopts a trapezoidal distribution for membership function, and contains nine combination rules in the decision rule base. During the initialization of the genetic algorithm, a threshold combination population is randomly generated. The fitness function comprehensively considers task completion rate and resource utilization rate. During the iteration process, high-quality individuals are retained through roulette wheel selection. The crossover operation adopts a single-point crossover strategy, and the mutation operation introduces Gaussian perturbation. In medical image processing scenarios, the optimized decision threshold automatically distinguishes between 3D reconstruction tasks and 2D slice analysis tasks, generating a task type classification matrix.

[0081] See Figure 5 The generation of a resource optimization scheduling graph based on S4 includes the following steps:

[0082] S401: Based on the task segmentation decision matrix, an evaluation system including computing power, network quality, and task urgency is constructed using the analytic hierarchy process. Consistency checks are used to ensure that the weights are reasonable, and three-dimensional evaluation indicators are generated.

[0083] Based on the task segmentation decision matrix, a resource evaluation system is constructed in the smart logistics warehousing scenario. The analytic hierarchy process (AHP) establishes a three-layer structure of computing power, network quality, and task urgency. The computing power index quantifies the floating-point operation capability of nodes through benchmark testing tools. The network quality index is calculated by comprehensively considering latency and packet loss rate. The task urgency is calculated by working backward from the deadline to estimate the remaining time. The judgment matrix is ​​constructed using a nine-level scaling method for pairwise comparisons. Consistency testing is performed by calculating the deviation between the eigenvector and the maximum eigenvalue. When the deviation coefficient exceeds a set threshold, the judgment matrix is ​​readjusted. In the cold chain logistics monitoring scenario, a higher urgency weight is set for temperature alarm tasks, generating a three-dimensional evaluation index that includes node performance score, network status score, and task priority score.

[0084] S402: Based on three-dimensional evaluation metrics, the scheduling strategy is trained using the federated Q-learning algorithm, and the initial scheduling strategy is generated by balancing exploration and utilization through the ε-greedy exploration mechanism.

[0085] Based on three-dimensional evaluation metrics, a scheduling strategy is trained in a cross-regional video analysis scenario. The federated Q-learning algorithm constructs a local Q-table for edge nodes. The state space is defined as a combination of resource utilization and task queue length, and the action space includes two options: local task execution and cloud migration. The ε-greedy mechanism sets a high exploration probability in the early stage of training and gradually decreases it as the number of iterations increases. When updating the Q value, the empirical data of adjacent nodes are integrated. In the smart grid fault detection scenario, a differentiated learning rate is set for nodes in different substations. The initial scheduling strategy generates a cross-regional collaborative decision-making model through multiple rounds of federated aggregation.

[0086] S403: Based on the initial scheduling strategy, a dynamic weighted round-robin algorithm is used to allocate computing tasks, and the weights are adjusted in real time through sliding window statistics to generate a dynamic scheduling table;

[0087] Based on the initial scheduling strategy, task allocation is implemented in the online education live streaming scenario. The dynamic weighted round-robin algorithm initializes the weights to the reciprocal of the node's historical load rate. The sliding window counts the task response time of the most recent time slice. When the response delay of a node exceeds the set threshold for three consecutive window periods, the weight value is reduced according to the exponential decay law. In the e-commerce promotion traffic distribution scenario, a dynamic gain factor is set for the weight of the flash sale business nodes to generate a dynamic scheduling table containing elements such as node number, task batch, and execution time window.

[0088] S404: Based on a dynamic scheduling table, it uses containerization encapsulation technology to package computing tasks, manages the task execution order through a priority queue, and generates containerized task packages;

[0089] Based on a dynamic scheduling table, tasks are encapsulated in the financial risk calculation scenario. Containerization technology packages Monte Carlo simulation tasks and risk models into independent images. When managing priority queues, high-priority real-time risk assessment tasks are inserted at the front of the queue, while batch historical data analysis tasks are placed at the back. Container startup parameters are configured differently according to task type. In the autonomous driving simulation scenario, emergency obstacle avoidance algorithm verification tasks are configured with high-memory containers, while regular path planning tasks use lightweight containers, generating containerized task packages with version tags.

[0090] S405: Based on the runtime data of containerized task packages, the scheduling parameters are adjusted using the gradient descent optimization algorithm, and multi-objective optimization is achieved through Pareto front analysis to generate a resource optimization scheduling map.

[0091] Based on containerized task package runtime data, scheduling parameters are optimized in intelligent manufacturing scenarios. Gradient descent algorithm is used to calculate the joint loss function of resource utilization and task latency. The learning rate is adaptively adjusted according to the rate of change of loss. Pareto front analysis is used to construct a two-dimensional plane of resource consumption and computational accuracy. Parameter combinations that satisfy dual objective constraints are selected. In the smart hospital image diagnosis scenario, GPU memory allocation and network bandwidth ratio parameters are optimized for CT three-dimensional reconstruction tasks. A resource optimization scheduling map containing elements such as node load balancing coefficients and task migration triggering conditions is generated.

[0092] See Figure 6 The S5-based adaptive incremental model generation package includes the following steps:

[0093] S501: Based on the resource optimization scheduling graph, the parameter importance matrix is ​​calculated using the elastic weight solidification algorithm, the parameter sensitivity is evaluated through the Fisher information matrix, and a parameter protection matrix is ​​generated.

[0094] Based on the resource optimization scheduling graph, this paper processes a multi-store sales prediction model in a smart retail scenario. The elastic weight solidification algorithm determines the importance of neurons in each layer by calculating the gradient change magnitude of parameters in historical tasks. When constructing the parameter importance matrix, the gradient variance of the convolutional layer parameters is statistically calculated using a sliding window. The protection priority of the fully connected layer parameters is determined by sorting the absolute values ​​of their weights. When evaluating the Fisher information matrix, differentiated sensitivity thresholds are set for different task types. In the fresh food demand prediction model, higher protection weights are set for feature parameters with high price sensitivity, generating a parameter protection matrix that includes parameter protection level labels.

[0095] S502: Based on the parameter protection matrix, it uses knowledge distillation technology to compress the cloud model and adjusts the knowledge transfer intensity through temperature coefficient to generate a lightweight model architecture.

[0096] Based on the parameter protection matrix, a compressed cloud model is used in the industrial equipment fault diagnosis scenario. Knowledge distillation technology is used to construct a teacher-student model architecture. The teacher model retains all parameters, while the student model uses depthwise separable convolution to replace standard convolution. The temperature coefficient is initially set to a high value to soften the output distribution and is gradually reduced to the standard value as the training rounds increase. In the bearing fault classification task, a knowledge transfer attenuation factor is set for the feature layer corresponding to the protected parameters. The lightweight model architecture removes redundant feature channels through channel pruning to generate a compressed model adapted to edge computing nodes.

[0097] S503: Based on a lightweight model architecture, it uses a differential parameter compression algorithm to generate update packages, and ensures compatibility through a version control mechanism to generate adaptive incremental model packages.

[0098] Based on a lightweight model architecture, an update package is generated in the smart agriculture pest and disease identification scenario. The differential parameter compression algorithm calculates the difference between the parameters of the old and new models, Huffman coding is used to compress the parameters of the fully connected layers, and residual quantization is used to reduce the storage space of the convolution kernel parameters. The version control mechanism uses semantic version numbers to mark model structure changes. When the crop disease identification model is updated, the major version number changes with the network structure adjustment, and the minor version number corresponds to the parameter optimization update, generating an adaptive incremental model package containing version metadata and compressed parameters.

[0099] See Figure 7 The resource pre-allocation optimization scheme based on S6 includes the following steps:

[0100] S601: Based on the adaptive incremental model package, a digital twin is constructed using the state-space modeling method, and state estimation is achieved through Kalman filtering to generate a virtual mirror system;

[0101] Based on an adaptive incremental model package, a digital twin is constructed in the smart port container scheduling scenario. The state space model defines a three-dimensional coordinate system (crane position, ship berthing status, and yard capacity). The system state variables include equipment operating status and cargo throughput. In the Kalman filter prediction stage, the state transition matrix is ​​adjusted in conjunction with the port operation plan. In the update stage, the actual operation data collected by RFID is integrated. When predicting the container loading and unloading progress, the model automatically corrects the operation efficiency deviation caused by weather changes and generates a virtual mirror system that reflects the real-time operation status.

[0102] S602: Based on a virtual mirror system, it uses a discrete event simulation engine to simulate the task execution process, evaluates different scheduling schemes through Monte Carlo sampling, and generates a scheme evaluation report.

[0103] Based on a virtual mirror system, the cargo sorting process is simulated in an air cargo hub scenario. The discrete event simulation engine defines event types (cargo arrival, sorting completion, abnormal delay). The event scheduler generates the arrival time interval distribution based on historical data. When evaluating different sorting schemes using Monte Carlo sampling, a probability analysis is performed on the risk of cargo backlog during peak hours. When evaluating cold chain pharmaceutical transportation schemes, the combination of equipment energy consumption and transportation timeliness under different temperature control strategies is simulated, generating a report that includes a scheme feasibility score and risk indicators.

[0104] S603: Based on the scheme evaluation report, a multi-objective genetic algorithm is used to optimize the parameter combination, and the optimal solution set is selected by crowding comparison to generate a parameter optimization scheme;

[0105] Based on the scheme evaluation report, the generator parameters are optimized in the power dispatch scenario. The multi-objective genetic algorithm initializes the population, including the generator start-up and shutdown thresholds and output allocation coefficients. The fitness function comprehensively considers the power generation cost and carbon emissions. The congestion comparison operator retains diverse solutions. In the case of regional power grid optimization, the algorithm automatically selects parameter combinations that meet the load demand and reserve capacity, and generates parameter optimization schemes that include generator combination schemes and output plans.

[0106] S604: Based on the parameter optimization scheme, a dynamic feedback mechanism is used to adjust the resource scheduling strategy, and the system stability is achieved by controlling PID regulation, generating a resource pre-allocation optimization scheme.

[0107] Based on the parameter optimization scheme, the resource allocation strategy is adjusted in the cloud computing data center scenario. The dynamic feedback mechanism sets dual threshold ranges for resource utilization and response latency. The PID controller calculates the resource allocation correction amount based on real-time monitoring data. The proportional term corresponds to the current deviation, the integral term accumulates the historical deviation, and the derivative term predicts the trend change. When the virtual machine resource usage exceeds the set threshold, the controller dynamically adjusts the CPU core allocation weight and generates a resource pre-allocation optimization scheme that includes elastic scaling rules and migration strategies.

[0108] A cloud-edge collaborative computing framework for multimodal data stream fusion processing, comprising any one of the cloud-edge collaborative computing framework processing methods for multimodal data stream fusion processing according to claims 1-7.

[0109] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A cloud-edge collaborative computing framework processing method for multimodal data stream fusion processing, characterized in that... Includes the following steps: S1: Based on the raw data stream collected by the edge computing node, the improved Wiener filtering algorithm is used for noise suppression, the signal is denoised by adaptive threshold wavelet transform, and the time stamp alignment technology is used to solve the problem of multimodal data delay difference, generating a spatiotemporally aligned and purified data stream; S2: Based on spatiotemporal alignment, the data stream is purified by adopting a spatiotemporal attention fusion network. In the time dimension, a bidirectional gated recurrent unit is used to extract temporal features, and in the spatial dimension, a lightweight 3D convolutional network is used to extract spatial features. Feature alignment is achieved through a cross-attention mechanism. Finally, a gated fusion unit is used to generate a unified representation and generate a multimodal feature tensor. S3: Based on multimodal feature tensors, an improved Prophet-LSTM hybrid prediction algorithm is adopted, combined with real-time resource monitoring data, to predict future computing load change trends. The task is divided into real-time tasks and batch computing tasks through a fuzzy logic decision tree, generating a task segmentation decision matrix. S4: Based on the task segmentation decision matrix, a dynamically weighted round-robin algorithm optimized by federated learning is used to construct a scheduling strategy with three dimensions: real-time task priority weight, edge node computing power index, and network transmission quality score, generating a resource optimization scheduling graph; S5: Based on the operational data of the resource optimization scheduling graph, the elastic weight solidification algorithm is used to perform local incremental learning at the edge. The optimized model structure is generated in the cloud through neural architecture search. The model update package volume is reduced by differential compression technology to generate an adaptive incremental model package. S6: Based on the adaptive incremental model package and resource optimization scheduling graph, a digital twin system enhanced by Monte Carlo tree search is used to construct a simulation environment with three levels: resource mirroring layer, task inference layer, and decision optimization layer, and generate resource pre-allocation optimization scheme.

2. The cloud-edge collaborative computing framework processing method for multimodal data stream fusion processing according to claim 1, characterized in that: The generation of spatiotemporally aligned and cleaned data streams based on S1 includes the following steps: S101: Based on the original multimodal data stream, an improved median filtering algorithm is used to eliminate impulse noise in the video data. The size of the filtering window is dynamically adjusted through an adaptive threshold selection mechanism to generate a video cleaned frame sequence. S102: Based on the timestamps of the video cleanup frame sequence, a dynamic time warping algorithm is used to align the audio and sensor data streams. Cubic spline interpolation is used to solve the sampling rate difference problem and generate a time-domain aligned data stream. S103: Based on the temporal-domain aligned data stream, the Z-score standardization method is used to unify the data dimensions of each modality, and the sliding window mechanism is combined to realize data segmentation standardization and generate a spatiotemporally aligned and purified data stream.

3. The cloud-edge collaborative computing framework processing method for multimodal data stream fusion processing according to claim 1, characterized in that: The generation of multimodal feature tensors based on S2 includes the following steps: S201: Based on spatiotemporal alignment, clean the data stream by using a bidirectional gated recurrent unit network to extract the temporal features of video / audio data, and use an attention mechanism to focus on key time nodes to generate a temporal feature vector. S202: Based on spatiotemporal alignment, clean the data stream, use a lightweight 3D convolutional network to extract spatial features, expand the receptive field through dilated convolution, and generate a spatial feature matrix; S203: Based on temporal feature vectors and spatial feature matrices, a cross-attention alignment algorithm is used to achieve feature dimension matching. Cross-modal association is established through cosine similarity calculation to generate aligned feature tensors. S204: Based on the aligned feature tensor, a gated fusion unit is used to achieve feature weighted fusion, and L2 regularization is used to prevent overfitting, generating a multimodal feature tensor.

4. The cloud-edge collaborative computing framework processing method for multimodal data stream fusion processing according to claim 1, characterized in that: The steps involved in generating the task segmentation decision matrix based on S3 are as follows: S301: Based on multimodal feature tensors, an improved Prophet algorithm is used to predict future computing load trends. Data fluctuations are processed through exponential smoothing to generate a load prediction curve. S302: Based on the load prediction curve, a fuzzy logic decision tree is used to divide the task types, and a genetic algorithm is used to optimize the decision threshold to generate a task segmentation decision matrix.

5. The cloud-edge collaborative computing framework processing method for multimodal data stream fusion processing according to claim 1, characterized in that: The generation of resource optimization scheduling graphs based on S4 includes the following steps: S401: Based on the task segmentation decision matrix, an evaluation system including computing power, network quality, and task urgency is constructed using the analytic hierarchy process. Consistency checks are used to ensure that the weights are reasonable, and three-dimensional evaluation indicators are generated. S402: Based on three-dimensional evaluation metrics, the scheduling strategy is trained using the federated Q-learning algorithm, and the initial scheduling strategy is generated by balancing exploration and utilization through the ε-greedy exploration mechanism. S403: Based on the initial scheduling strategy, a dynamic weighted round-robin algorithm is used to allocate computing tasks, and the weights are adjusted in real time through sliding window statistics to generate a dynamic scheduling table; S404: Based on a dynamic scheduling table, it uses containerization encapsulation technology to package computing tasks, manages the task execution order through a priority queue, and generates containerized task packages; S405: Based on the runtime data of containerized task packages, the scheduling parameters are adjusted using the gradient descent optimization algorithm, and multi-objective optimization is achieved through Pareto front analysis to generate a resource optimization scheduling map.

6. The cloud-edge collaborative computing framework processing method for multimodal data stream fusion processing according to claim 1, characterized in that: The S5-based adaptive incremental model generation package includes the following steps: S501: Based on the resource optimization scheduling graph, the parameter importance matrix is ​​calculated using the elastic weight solidification algorithm, the parameter sensitivity is evaluated through the Fisher information matrix, and a parameter protection matrix is ​​generated. S502: Based on the parameter protection matrix, it uses knowledge distillation technology to compress the cloud model and adjusts the knowledge transfer intensity through temperature coefficient to generate a lightweight model architecture. S503: Based on a lightweight model architecture, it uses a differential parameter compression algorithm to generate update packages, and ensures compatibility through a version control mechanism to generate adaptive incremental model packages.

7. The cloud-edge collaborative computing framework processing method for multimodal data stream fusion processing according to claim 1, characterized in that: The S6-based resource pre-allocation optimization scheme includes the following steps: S601: Based on the adaptive incremental model package, a digital twin is constructed using the state-space modeling method, and state estimation is achieved through Kalman filtering to generate a virtual mirror system; S602: Based on a virtual mirror system, it uses a discrete event simulation engine to simulate the task execution process, evaluates different scheduling schemes through Monte Carlo sampling, and generates a scheme evaluation report. S603: Based on the scheme evaluation report, a multi-objective genetic algorithm is used to optimize the parameter combination, and the optimal solution set is selected by crowding comparison to generate a parameter optimization scheme; S604: Based on the parameter optimization scheme, a dynamic feedback mechanism is used to adjust the resource scheduling strategy, and the system stability is achieved by controlling PID regulation, generating a resource pre-allocation optimization scheme.

8. A cloud-edge collaborative computing framework for multimodal data stream fusion processing, characterized in that: The cloud-edge collaborative computing framework processing method includes any one of the multimodal data stream fusion processing methods according to claims 1-7.

Images