A heterogeneous data packet scheduling method and system based on time-varying characteristics and priorities

Abstract

This invention discloses a heterogeneous data grouping and scheduling method and system based on time-varying features and priorities, relating to the field of data processing. The method includes: collecting multi-source heterogeneous data and converting it into carbon emission data in a unified format using functional data analysis methods; extracting time-varying features and grouping the carbon emission data using a CNN-LSTM model based on an attention mechanism; calculating the demand value for each data group according to a data scheduling demand estimation model; calculating the similarity between each data group and the scheduling link based on the demand value, and scheduling the data groups accordingly. This invention, by grouping and scheduling massive amounts of heterogeneous data, can effectively and systematically manage and utilize such data, improving the intelligence level of the "dual-carbon" digital monitoring center.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a method and system for scheduling heterogeneous data groups based on time-varying characteristics and priorities. Background Technology

[0002] Against the backdrop of increasing global climate change and resource and environmental pressures, "dual-carbon" digital monitoring centers, which use digital means to monitor and calculate carbon emissions in real time, have emerged. These digital monitoring centers integrate data collection, storage, processing, and analysis, providing comprehensive and accurate carbon emission data support for governments, businesses, and the public.

[0003] However, with the continuous development of "dual-carbon" digital intelligent monitoring centers, the types of carbon emission data collected are becoming increasingly diverse, covering fields such as power, industry, transportation, and construction, forming a massive amount of heterogeneous data. This data not only differs in format, structure, and source, but also exhibits different dynamic characteristics over time. Faced with such a complex data environment, how to effectively and systematically manage and utilize this data has become an urgent problem to be solved. The disorderly accumulation of large amounts of heterogeneous data not only increases the difficulty and cost of data processing, but may also affect the accuracy of data analysis and the effectiveness of decision-making.

[0004] Therefore, how to group and schedule massive amounts of heterogeneous data to improve the intelligence level of the "dual-carbon" digital monitoring center is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a heterogeneous data grouping and scheduling method and system based on time-varying characteristics and priorities. By combining the time-varying characteristics of heterogeneous data, the method achieves reasonable data partitioning through dynamic grouping strategies, thereby improving the intelligence level of the "dual-carbon" digital intelligent monitoring center.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention discloses a heterogeneous data grouping and scheduling method based on time-varying characteristics and priorities, the specific steps of which are as follows:

[0008] Collect diverse and heterogeneous data, and use functional data analysis methods to convert the diverse and heterogeneous data into carbon emission data in a unified format;

[0009] The carbon emission data is subjected to time-varying feature extraction and data grouping using an attention-based CNN-LSTM model.

[0010] Based on the data scheduling demand estimation model, the demand value of each data group is calculated; based on the demand value, the similarity between each data group and the scheduling link is calculated, and the data group is scheduled.

[0011] Furthermore, the functional data analysis method specifically includes:

[0012] The multivariate heterogeneous data is smoothly expanded using basis functions to obtain a fitting function;

[0013] The fitted function is smoothed by adding a penalty term;

[0014] The coefficient matrix of the fitted function is fitted using the least squares method, and the penalty factor coefficient of the penalty term is solved using the generalized cross-validation method to obtain the representation function of the discrete data sequence of multivariate heterogeneous data.

[0015] Furthermore, the fitting function is expressed as:

[0016]

[0017] Where F(x) represents the fitting function, x represents the x-th sampling point of the scattered data sequence, and φ i (x) represents the i-th basis function, i = 1, 2, 3, ..., n, where n is the total number of basis functions, and c is the total number of basis functions. i Represents the coefficient vector of the basis functions;

[0018] The specific formula for the penalty term ∈(x) is:

[0019] ∈(x)=λC T RC;

[0020] Where λ represents the penalty factor, and C represents the coefficient vector c i The corresponding coefficient matrix, R, represents the matrix related to the second derivatives of the basis functions;

[0021] The expression function f(x) is given by the following formula:

[0022] f(x)=F(x)+∈(x).

[0023] Furthermore, the basis functions are B-spline basis functions, expressed as:

[0024]

[0025] Among them, B i,k (x) represents the B-spline basis function, t represents the time parameter, t i t i+1 are the time parameters of the i-th and (i+1)-th basis functions, respectively, and k represents the order of the basis function;

[0026] The B-spline basis functions are fitted using a recursive formula, which is:

[0027]

[0028] Furthermore, the least squares method is expressed as:

[0029]

[0030] The solution to the coefficient matrix is ​​expressed as:

[0031] C = (G T G+λR) -1 G T y;

[0032] Among them, y x The original data collected at the x-th sampling point, G is the value of each sampling point of the basis function matrix G=[φ(1),φ(2)…φ(x)], and φ(x) is the superposition vector of all basis functions at the x-th sampling point;

[0033] The formula for the generalized cross-validation method is:

[0034]

[0035] Wherein, GCV represents the value of generalized cross-validation, used to evaluate the goodness of fit of the model, tr() represents the trace of the matrix, defined as the sum of the diagonal elements of the matrix; the value of the penalty factor coefficient is determined by minimizing the GCV value.

[0036] Furthermore, the CNN-LSTM model includes an input layer, a convolutional layer, multiple LSTM layers, an attention layer, and an output layer connected in sequence;

[0037] The input layer samples the carbon emission data according to the set batch size, time step T, and feature dimension D. A data sample is represented as a real number sequence matrix X∈R. T×D Where X represents a data sample, R T×D Let x represent a sequence matrix of real numbers. t "" represents the data vector at time t′ in the real number sequence matrix;

[0038] The convolutional layer performs convolution, pooling, and dimensionality reduction operations on the input data vector sequentially. The feature representation after convolution is: h t′ =f(W·x) t "+b); where h t′ Here, f() represents the output feature, W represents the non-linear activation function, and b represents the bias.

[0039] The multi-layer LSTM uses a gating mechanism to pass the state of time-series data, including a forget gate, a selection memory gate, and an output gate. The forget gate determines the degree of forgetting of the previous hidden layer state, denoted as: f t′ =σ(W f ·[h t′-1 ,x t "]+b f ); where f t′ Let W represent the value of the forget gate at time t′, σ represent the activation function, and W represent the value of the forget gate at time t′. f Let h represent the forget gate weight matrix. t′-1 Let b represent the hidden layer state at time t′-1. f Indicates the bias of the forget gate;

[0040] The attention layer performs a weighted summation of the output of the last hidden layer in the multi-layer LSTM layer using a weighted average, as shown in the formula: Among them, c t′ α t′ h t "" represents the output, weights, and outputs of the attention layer and the multi-layer LSTM layer at time t′, respectively;

[0041] The output layer outputs the final grouping result based on the output of the attention layer.

[0042] Furthermore, the data scheduling demand estimation model is expressed as follows:

[0043] Z = f × I × Q;

[0044] Where Z represents the demand estimate, f represents the frequency of data grouping, I represents the data scheduling amount, and Q represents the data scheduling density;

[0045] The formula for calculating the similarity is:

[0046]

[0047] Where RTT represents the link response time factor; the data packet selection R DPLS Transmission is carried out on the link with the lowest value.

[0048] This invention also discloses a heterogeneous data packet scheduling system based on time-varying characteristics and priorities, comprising:

[0049] Data preprocessing module: Collects diverse and heterogeneous data, and uses functional data analysis methods to convert the diverse and heterogeneous data into carbon emission data in a unified format;

[0050] Feature extraction and grouping module: Using an attention-based CNN-LSTM model, time-varying features are extracted and data is grouped from the carbon emission data;

[0051] Scheduling module: Calculates the demand value of each data group according to the data scheduling demand estimation model; calculates the similarity between each data group and the scheduling link based on the demand value, and schedules the data group accordingly.

[0052] As can be seen from the above technical solution, compared with the prior art, this invention discloses a heterogeneous data grouping and scheduling method and system based on time-varying characteristics and priorities. In the data preprocessing stage, considering the diversity and complexity of heterogeneous data, this invention uses functional data analysis to preprocess the data to ensure data consistency and availability, laying a solid foundation for subsequent data analysis and scheduling. In the data grouping stage, based on the time-varying characteristics of the data and business requirements, massive amounts of data are rationally divided into different groups, providing a clear framework for subsequent scheduling optimization. In the specific scheduling process, data priority, response time factor, load factor, and link performance characteristics are comprehensively considered to calculate the data priority-oriented data scheduling link similarity (DPLS), and data packet scheduling is implemented accordingly. This invention, by grouping and scheduling massive amounts of heterogeneous data, can effectively and orderly manage and utilize massive amounts of heterogeneous data, improving the intelligence level of the "dual-carbon" digital monitoring center. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the CNN-LSTM model structure based on the attention mechanism in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention discloses a heterogeneous data grouping scheduling method based on time-varying characteristics and priorities, such as... Figure 1 As shown, the specific steps are as follows:

[0058] Collect diverse and heterogeneous data, and use functional data analysis methods to convert the diverse and heterogeneous data into carbon emission data in a unified format;

[0059] A CNN-LSTM model based on an attention mechanism is used to extract time-varying features and group data from carbon emission data.

[0060] Based on the data scheduling demand estimation model, the demand value of each data group is calculated; based on the demand value, the similarity between each data group and the scheduling link is calculated, and the data groups are scheduled.

[0061] Specifically, the data collected by the "Dual Carbon" digital monitoring center comes from a wide range of sources, covering fields such as power, industry, transportation, and construction. These data differ significantly in format, structure, and sampling frequency, forming heterogeneous datasets. Furthermore, due to the differences in sampling frequency and period, direct integration is difficult. Functional data analysis methods can unify heterogeneous data into a functional expression, facilitating subsequent comprehensive analysis and processing. By converting discrete data into continuous functions, uniformity and comparability at the function level are achieved for data of different frequencies, thus solving the data fusion problem caused by inconsistent sampling frequencies. Theoretically, methods such as function fitting and interpolation can be used to increase the number of sampling points, achieving expansion. This will help improve the accuracy and robustness of the massive data analysis in the "Dual Carbon" digital monitoring center.

[0062] In a specific embodiment, the functional data analysis method is as follows:

[0063] The fitting function is obtained by smoothly expanding the multivariate heterogeneous data using basis functions;

[0064] The fitted function is smoothed by adding a penalty term;

[0065] The coefficient matrix of the fitted function is fitted using the least squares method, and the penalty factor coefficient of the penalty term is solved using the generalized cross-validation method to obtain the representation function of the discrete data sequence of multivariate heterogeneous data.

[0066] In a specific embodiment, the fitting function is expressed as:

[0067]

[0068] Where F(x) represents the fitting function, x represents the x-th sampling point of the scattered data sequence, and φ i (x) represents the i-th basis function, i = 1, 2, 3, ..., n, where n is the total number of basis functions, and c is the total number of basis functions. i Represents the coefficient vector of the basis functions;

[0069] The specific formula for the penalty term ∈(x) is:

[0070] ∈(x)=λC T RC;

[0071] Where λ represents the penalty factor, and C represents the coefficient vector c i The corresponding coefficient matrix, R, represents the matrix related to the second derivatives of the basis functions;

[0072] The function f(x) is represented by the formula:

[0073] f(x)=F(x)+∈(x).

[0074] In a specific embodiment, B-spline basis functions are used. Assuming the node sequence is non-decreasing within the interval [a, b], the B-spline basis function of order k at a node is expressed as:

[0075]

[0076] Among them, B i,k (x) represents the B-spline basis function, t represents the time parameter, t i t i+1 are the time parameters of the i-th and (i+1)-th basis functions, respectively, and k represents the order of the basis function;

[0077] B-spline basis functions are fitted using a recursive formula, which is:

[0078]

[0079] Specifically, the basis function smoothing method essentially involves linearly combining multiple basis functions to obtain a functional expression of the massive, multi-source, heterogeneous discrete data from the original "Dual-Carbon" digital intelligent monitoring center. The properties of the basis functions are crucial for data fitting. Commonly used basis functions include polynomial, Fourier, and B-spline basis functions. Since the time series data from the "Dual-Carbon" digital monitoring center is aperiodic, B-spline basis functions are typically used, and their fitting function is expressed through a recursive formula.

[0080] When fitting time-series data of the "Dual-Carbon" digital intelligent monitoring center using the basis function smoothing method, the fitting function should express the main characteristics of the original discrete data as much as possible. To avoid overfitting the curve and causing excessive fluctuations or local variations, a roughness penalty factor is introduced to control the degree of curve fitting, making the function expression of the discrete data more reasonable. The roughness penalty method is based on the basis function method, which controls the curvature of the fitted curve by adding a penalty term. Typically, the square integral of the second derivative is used as the penalty term to measure the roughness of the curve. This yields a matrix related to the second derivative of the basis functions.

[0081] In a specific embodiment, the least squares method is expressed as:

[0082]

[0083] The solution to the coefficient matrix is ​​expressed as:

[0084] C = (G T G+λR) -1 G T y;

[0085] Among them, y x The original data collected for the x-th sampling point is G, which is the basis function matrix representing the value of each sampling point, G = [φ(1), φ(2), ... φ(x)], and φ(x) is the superposition vector of all basis functions of the x-th sampling point;

[0086] The formula for generalized cross-validation is:

[0087]

[0088] Wherein, GCV represents the value of generalized cross-validation, used to evaluate the goodness of fit of the model, tr() represents the trace of the matrix, defined as the sum of the diagonal elements of the matrix; the value of the penalty factor coefficient is determined by minimizing the GCV value.

[0089] The Functional Data Analysis (FDA) method transforms the massive, multi-source, heterogeneous data from the "Dual Carbon" digital intelligent monitoring center into continuous functions through basis function smoothing, thereby resolving the issue of frequency inconsistencies. Furthermore, it improves data accuracy by adding penalty terms to achieve data smoothing. The least squares method is used to calculate coefficients and optimize the fitting effect. This process unifies the data format, reduces data dimensionality, and provides a high-quality data foundation for subsequent carbon emission monitoring and energy efficiency assessment, thus promoting the development of digital intelligent applications.

[0090] In one specific embodiment, to achieve grouping and scheduling of massive heterogeneous data in a "dual-carbon" digital intelligent monitoring center, a CNN-LSTM model based on an attention mechanism is used for data grouping after preprocessing. This model combines the advantages of convolutional neural networks (CNNs) in feature extraction with the powerful capabilities of long short-term memory networks (LSTMs) in processing time-series data. By introducing an attention mechanism, the model can focus more on key information in the data, thereby improving the accuracy and efficiency of time-varying feature extraction. This approach can better accomplish the task of grouping massive heterogeneous data, providing strong support for subsequent data scheduling. Figure 2 As shown, the CNN-LSTM model consists of an input layer, a convolutional layer, multiple LSTM layers, an attention layer, and an output layer connected in sequence.

[0091] The input layer samples carbon emission data according to the set batch size, time step T, and feature dimension D. A data sample is represented as a real number sequence matrix X∈R.T×D Where X represents a data sample, R T×D Let x represent a sequence matrix of real numbers. t "" represents the data vector at time t′ in the real number sequence matrix;

[0092] Convolutional layers perform convolution, pooling, and dimensionality reduction operations sequentially on the input data vector, extracting spatial connections between features at different times, thus compensating for the inability of LSTM to capture the spatial components of data. When the data enters the CNN layer, convolution, pooling, and node unrolling (dimensionality reduction) operations are performed sequentially. For sequence data, the model uses one-dimensional convolution, with the convolution kernel performing convolution only in a single time domain direction. The convolutional feature representation is: h t′ =f(W·x) t "+b); where h t′ Here, f() represents the output feature, W represents the non-linear activation function, and b represents the bias.

[0093] A multi-layer LSTM system consists of multiple LSTM layers. Each LSTM layer has a memory function and uses a gating mechanism to pass the state of time-series data. This gating mechanism includes a forget gate, a selection gate, and an output gate. The forget gate determines the degree of forgetting of the previous hidden layer's state, denoted as f. t′ =σ(W f ·[h t′-1 ,x t "]+b f ); where f t′ Let W represent the value of the forget gate at time t′, σ represent the activation function, and W represent the value of the forget gate at time t′. f Let h represent the forget gate weight matrix. t′-1 Let b represent the hidden layer state at time t′-1. f This indicates the bias of the forget gate. The selection of the memory gate controls the degree to which new information is added. The cell state is updated jointly through the outputs of the forget gate and the memory gate, and the output gate controls the output level of the current cell. The final hidden layer output of the multi-layer LSTM serves as the grouping result of heterogeneous data and enters the attention layer for further processing.

[0094] The attention layer enhances the role of important time steps in the LSTM, thereby further reducing the heterogeneous data grouping error of the model. This layer performs a weighted summation of the outputs of the last hidden layer in the multi-layer LSTM, using the following formula: Among them, c t′ α t′ h t "" represents the output, weights, and outputs of the attention layer and the multi-layer LSTM layer at time t′, respectively;

[0095] The output layer outputs the final grouping results based on the output of the attention layer.

[0096] In one specific embodiment, after the data grouping of the "dual-carbon" digital intelligent monitoring center is completed, differentiated priorities are set for different data groups, and the data scheduling link similarity (DPLS) is calculated by combining the data scheduling demand estimation model, RTT factor and congestion factor to realize data group scheduling.

[0097] The data scheduling demand estimation model is expressed as follows:

[0098] Z = f × I × Q;

[0099] Where Z represents the demand estimate, f represents the frequency of data grouping, i.e. the frequency at which the original data is divided into the current group, I represents the data scheduling amount, i.e. the total amount of data in the current group, and Q represents the data scheduling density, i.e. the proportion of the amount of data in the current group that needs to be scheduled to the total amount of data.

[0100] The formula for calculating similarity is:

[0101]

[0102] Where RTT represents the link response time factor; data packet selection R DPLS The link with the lowest value is used for transmission to ensure data transmission priority, latency, and load balancing.

[0103] This invention also discloses a heterogeneous data packet scheduling system based on time-varying characteristics and priorities, comprising:

[0104] Data preprocessing module: Collects diverse and heterogeneous data and uses functional data analysis methods to convert the diverse and heterogeneous data into carbon emission data in a unified format;

[0105] Feature extraction and grouping module: Utilizes an attention-based CNN-LSTM model to perform time-varying feature extraction and data grouping on carbon emission data;

[0106] Scheduling module: Based on the data scheduling demand estimation model, calculate the demand value of each data group; calculate the similarity between each data group and the scheduling link based on the demand value, and schedule the data groups accordingly.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0108] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A time-varying feature and priority based heterogeneous data packet scheduling method, characterized by, The specific steps are as follows: Collecting multi-element heterogeneous data, and converting the multi-element heterogeneous data into carbon emission data in a unified format by using a functional data analysis method; Using a CNN-LSTM model based on an attention mechanism to perform time-varying feature extraction and data grouping on the carbon emission data; According to a data scheduling demand estimation model, calculating the demand value of each data group, and calculating the similarity between each data group and a scheduling link according to the demand value to schedule the data groups; The data scheduling demand estimation model is represented as: ； wherein, denotes a demand estimate value, denotes a frequency of data packets, i.e. a frequency at which raw data is divided into current packets, denotes a data scheduling amount, denotes a data scheduling density, i.e. a proportion of a data amount that needs to be scheduled by a current packet; The calculation formula of the similarity is: ； wherein, represents a response time factor of the link; the data packet is selected for transmission by the link having the smallest value.

2. The method of claim 1, wherein, The functional data analysis method is specifically as follows: Expanding the multi-element heterogeneous data by using a base function to obtain a fitting function; Performing smoothing processing on the fitting function by adding a penalty term; Using a least square method to fit the coefficient matrix of the fitting function, and using a generalized cross-validation method to solve the penalty factor coefficient of the penalty term to obtain a representation function of the discrete data sequence of the multi-element heterogeneous data.

3. The method of claim 2, wherein, The fitting function is represented as: ； in, Represents the fitted function. Represents the first discrete data sequence One sampling point, Indicates the first basis functions , Total number of basis functions Represents the coefficient vector of the basis functions; The penalty term The specific formula is: ； wherein denotes a penalty factor, denotes a coefficient vector corresponding coefficient matrix, denotes a matrix related to the second derivative of the basis functions; The representation function is , and the formula is: 。 4. The method of claim 3, wherein, The base function uses a B-spline base function, which is represented as: ； wherein denotes a B-spline basis function, denotes a time parameter, , are the time parameters of the th and th basis function, respectively, denotes the order of the basis function; The B-spline base function is fitted by using a recursive formula, and the recursive formula is: 。 5. The method of claim 3, wherein, The least square method is represented as: ； The solution of the coefficient matrix is represented as: ； wherein, the first original data collected at the first value of each sampling point of the basis function matrix , the first superposition vector of all basis functions of the first The formula of the generalized cross-validation method is: ； wherein, represents a value of the generalized cross-validation for assessing the goodness of fit of the model, represents the trace of a matrix, defined as the sum of the diagonal elements of the matrix; the value of the trace is minimized by determines the value of the penalty factor coefficient.

6. The method of claim 1, wherein, The CNN-LSTM model includes an input layer, a convolution layer, a multi-layer LSTM layer, an attention layer, and an output layer connected in sequence; The input layer samples the carbon emission data according to a set batch size, a time step , and a feature dimension , and one data sample is represented as a real sequence matrix , wherein represents a data sample, represents a real sequence matrix, represents a data vector at a time point in the real sequence matrix . The convolution layer sequentially performs convolution, pooling and dimension reduction operations on the input data vector, and the feature after convolution is represented as: ; wherein, represents the output feature, represents a nonlinear activation function, represents a weight matrix, represents a bias; The multi-layer LSTM layer realizes state transmission of time series data through a gating mechanism, including a forgetting gate, a selection memory gate and an output gate. ; wherein, represents the value of the forgetting gate at the moment, represents an activation function, represents a forgetting gate weight matrix, represents the hidden layer state at the moment, represents a forgetting gate bias; The attention layer performs a weighted summation of the output of the last hidden layer in the multi-layer LSTM layer using a weighted average, as shown in the formula: ;in, , , They represent The output and weights of the attention layer at each moment, and the output of the multi-layer LSTM layer; The output layer outputs a final grouping result according to the output of the attention layer.

7. A time-varying feature and priority based heterogeneous data packet scheduling system, applying the time-varying feature and priority based heterogeneous data packet scheduling method of any one of claims 1-6, characterized in that, It includes: A data preprocessing module: collecting multi-element heterogeneous data, and converting the multi-element heterogeneous data into carbon emission data in a unified format by using a functional data analysis method; A feature extraction and grouping module: using a CNN-LSTM model based on an attention mechanism to perform time-varying feature extraction and data grouping on the carbon emission data; A scheduling module: according to a data scheduling demand estimation model, calculating the demand value of each data group, and calculating the similarity between each data group and a scheduling link according to the demand value to schedule the data groups.

Images