Resource allocation method and system based on cloud edge collaboration

Abstract

The invention relates to a resource allocation method and system based on cloud edge collaboration. Historical to-be-processed data is acquired and input into a data prediction model for training; the data prediction model predicts the to-be-processed data of each terminal device, pre-splits the predicted to-be-processed data, matches the corresponding edge nodes according to the data type and data volume involved in the pre-split information, collects the to-be-processed data of each terminal device in real time, compares the real-time to-be-processed data with the predicted to-be-processed data, and determines the to-be-processed data of each terminal device. When the real-time to-be-processed data exceeds the prediction, the exceeding part is split again, otherwise, different data units are obtained through direct splitting, and the same type of data units obtained through splitting are packaged and transmitted to matched edge nodes; and each edge node generates a result after processing, returns the result to the relay node, splits each processing result data packet, integrates the processing result data packet according to the processing result of the same data source, and returns the integrated processing result data packet to the equipment terminal. And collaboration between cloud side resource allocation and data generated by large-scale terminal equipment is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of resource allocation in cloud-edge computing, and in particular, relates to a resource allocation method and system based on cloud-edge collaboration. Background Art

[0002] With the rapid development of 5G, artificial intelligence and Internet of Things technologies, the explosive growth of data has put forward higher requirements on computing power and response speed. Edge computing, as a distributed computing architecture, can decompose and distribute computing tasks to the edge of the network, thereby reducing latency and bandwidth consumption. In order to solve the bottleneck problem of cloud computing in low-latency applications, cloud-edge collaboration came into being. By reasonably allocating tasks of cloud computing and edge computing, real-time processing and analysis of data can be achieved to better respond to user needs. In the fields of manufacturing, smart cities, smart transportation, etc., the application scenarios of cloud-edge collaboration are increasing.

[0003] With the development of cloud-edge collaborative computing models, the scale of cloud-edge collaboration has gradually increased, and the number of edge nodes and terminal devices has continued to increase, which has doubled the pressure on central management. The existing cloud-edge collaborative architecture sends the tasks generated by terminal devices to the matching edge nodes for calculation. However, with the sharp increase in the number of edge nodes and terminal devices, on the one hand, it will increase the difficulty of matching tasks with edge nodes, and on the other hand, it may cause the accumulation of tasks on a certain edge node, reducing the efficiency of task processing.

[0004] Therefore, how to allocate resources between edge nodes and the cloud, and achieve coordination between resource allocation between edge nodes and the cloud and data generated by large-scale terminal devices, is a technical problem that urgently needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a resource allocation method and system based on cloud-edge collaboration, which is used to perform resource allocation on edge nodes and the cloud, and to achieve collaboration between resource allocation on edge nodes and the cloud and data generated by large-scale terminal devices.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, a resource allocation method based on cloud-edge collaboration is provided, comprising the following steps: S1: deploy several relay nodes, establish a connection channel between each relay node and a designated edge node, and a connection channel between each relay node and a designated terminal device, obtain the historical data to be processed of each terminal device, extract features and annotate them, and construct training samples and verification samples in a preset ratio; S2: Create a data prediction model on the relay node, input the training sample into the data prediction model for model training, and input the verification sample into the data prediction model to verify the model performance. When the data prediction model performance meets the preset requirements, execute step S3, otherwise re-train the model; S3: The data prediction model predicts the data to be processed of each terminal device, and the relay node pre-splits the data to be processed of each terminal device according to the predicted data to be processed, and adds a virtual tag to each pre-split data unit, wherein the virtual tag includes data source information, data type, data volume and latency requirement, and matches the corresponding edge node according to the data type and data volume involved in the pre-split information of the data to be processed of each terminal device. S4: collecting the data to be processed generated by each terminal device during operation in real time and transmitting it to the corresponding relay node. The relay node compares the real-time data to be processed with the predicted data to be processed to determine whether the real-time data to be processed exceeds the predicted data to be processed. If so, the part of the data that exceeds the predicted data to be processed is re-split, and the part of the data that does not exceed the step S5 is executed. If not, step S5 is directly executed; S5: directly split the real-time to-be-processed data of each terminal device according to the pre-splitting scheme to obtain different data units, pack the split data units of the same type, construct to-be-processed data packets of each data type, and transmit the to-be-processed data packets of each data type to the matching edge node or cloud; S6: The cloud or each edge node performs corresponding processing according to the received data packets to be processed, generates corresponding data results, and packages them after retaining the marks, and returns the processing result data packets to the relay node. The relay node splits the processing result data packets received from each edge node, integrates the processing results of the same data source according to the marks, and returns the integrated results of each data source to the corresponding terminal device; S7: Each of the terminal devices performs a corresponding action according to the received integration result.

[0007] Preferably, the specific process of feature extraction in step S1 is as follows: S11: constructing a feature selection model, inputting a labeled data set into the feature selection model, fitting the feature selection model through the labeled data set, and in the process of fitting the model, controlling the sparsity of the feature by adjusting a regularization parameter; S12: acquiring feature coefficients in the historical data to be processed based on the trained feature selection model, and sorting the data according to the feature coefficients according to a specified rule; S13: setting a feature coefficient threshold for the sorted feature coefficients by cross-validation, and extracting all features whose coefficients are greater than the feature coefficient threshold.

[0008] Preferably, the data prediction model in step S2 comprises an input layer, a hidden layer and an output layer, the input layer is connected to the hidden layer, and the hidden layer is connected to the output layer; The number of neurons in the input layer is set to the number of features extracted in step S13, which is used to input feature information of the data to be processed; The hidden layer is provided with multiple layers, each hidden layer is provided with a number of neurons, and the neurons are provided with a set of weights and a bias. The next hidden layer is connected to the previous hidden layer through a set of weights, and the bias is used to adjust the activation threshold of the neurons of the corresponding hidden layer. The feature information input to each hidden layer is output after being processed by weighting and a preset activation function; The output layer outputs the results processed by all hidden layers.

[0009] Preferably, the training process of the data prediction model is as follows: S21: input the feature information extracted from the historical data to be processed into the model through the input layer; S22: The extracted feature information is transmitted from the output layer to the first layer of the hidden layer, the first hidden layer performs weighting and preset activation function processing on the feature information and then outputs it to the second hidden layer, the second hidden layer performs weighting and preset activation function processing on the feature information and then outputs it to the third hidden layer, until all hidden layers are processed and arrive at the output layer, and the output layer outputs the processing result; S23: calculating the error signal between the processing result and the expected value by a preset mean square error function, and calculating the gradient of the error signal with respect to the weight of each hidden layer by a preset rule, wherein the gradient is the degree of influence of the weight change on the error reduction; S24: Iteratively update and reversely update the weights of each hidden layer through a gradient descent algorithm until the error signal between the processing result and the expected value is less than a preset error threshold.

[0010] Preferably, the specific process of iteratively updating and reversely updating the weights of each hidden layer by the gradient descent algorithm in step S24 is as follows: S241: Obtain the gradient of the error signal with respect to the weight of each hidden layer through a preset mean square error function to obtain the slope of the loss function in the weight space; S242: Move along the direction with the largest slope by a certain specified step length, and update the weight of each hidden layer until the error signal between the processing result and the expected value is less than a preset error threshold or reaches a preset number of iterations.

[0011] Preferably, the specific process of splitting the data to be processed in step S5 is as follows: S51: create a ring buffer and set multiple partitions in the ring buffer; S52: Input the data to be processed into the ring buffer, split it according to the type logic, and distribute it to the partitions respectively.

[0012] Preferably, the specific process of data integration in step S6 is as follows: S61: extracting features from each processing result of the same data source according to the process of step S11 to step S13; S62: constructing feature vectors corresponding to the respective processing results from the features of the respective processing results, and assigning corresponding weights to the respective processing results; S63: Perform weighted summation on the feature vectors corresponding to the processing results to obtain a final feature vector, and convert the final feature vector into a final processing result.

[0013] In a second aspect, a resource allocation system based on cloud-edge collaboration is used to implement the resource allocation method based on cloud-edge collaboration, including a relay node deployment module, a data acquisition module, a data prediction model, a data splitting module, a data packaging module and a data integration module, wherein the relay node deployment module is connected to the relay node, the data acquisition module is connected to the terminal device, the data splitting module is connected to the relay node, the data prediction model is connected to the relay node and the data splitting module, the data splitting module is connected to the data packaging module, and the data integration module is connected to the edge node and the terminal device; The relay node deployment module is used to deploy a number of relay nodes, establish a connection channel between each relay node and a designated edge node, and a connection channel between each relay node and a designated terminal device; The data acquisition module is used to obtain the historical data to be processed of each terminal device and to collect the data to be processed generated by each terminal device during operation in real time, and transmit it to the corresponding relay node; The data prediction model is used to predict the data to be processed of each terminal device; The data splitting module is used to split and process the to-be-processed data generated by each terminal device during operation; The data packaging module is used to split the part of the data that exceeds the predicted data to be processed and directly split the real-time data to be processed of each terminal device according to the pre-splitting scheme to obtain different data units, pack the split data units of the same type, construct data packets to be processed of each data type, and transmit the data packets to be processed of each data type to the matching edge node; The data integration module is used to integrate the processing results of the same data source according to the mark, and return the integration results of each data source to the corresponding terminal device.

[0014] The beneficial effects of the present invention include: The resource allocation method and system based on cloud-edge collaboration provided by the present invention deploys several relay nodes, obtains historical data to be processed and inputs the data prediction model for training; the data prediction model predicts the data to be processed of each terminal device, pre-splits the predicted data to be processed, matches the corresponding edge node according to the data type and data volume involved in the pre-split information, collects the data to be processed of each terminal device in real time, compares the real-time data to be processed with the predicted data to be processed, and when the real-time data to be processed exceeds the prediction, the excess part is split again, otherwise it is directly split to obtain different data units, and the split data units of the same type are packaged and transmitted to the matching edge node; each edge node generates a result after processing and returns it to the relay node, splits each processing result data packet, integrates it according to the processing results of the same data source, and returns it to the terminal device. The collaboration between the resource allocation of edge nodes and the cloud and the data generated by large-scale terminal devices is realized.

[0015] First, by constructing a feature selection model, inputting the labeled data set into the feature selection model, fitting the feature selection model with the labeled data set, and in the process of fitting the model, controlling the sparsity of the features by adjusting the regularization parameters; obtaining the feature coefficients in the historical data to be processed based on the trained feature selection model, and sorting them according to the specified rules based on the feature coefficients; setting the feature coefficient threshold for the sorted feature coefficients by cross-validation, and extracting all the features whose coefficients are greater than the feature coefficient threshold, the process realizes the accurate extraction of features in the historical data to be processed, provides a data basis for the subsequent training of the data prediction model, improves the training efficiency, and makes the model have good robustness and generalization.

[0016] Second, the data prediction model is set up to include an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is set to match the number of extracted features; the hidden layer has multiple layers, and each hidden layer has several neurons. Neurons can help the model learn complex patterns and features by extracting and transforming input features. Several neurons are each provided with a set of weights and a bias. The number of neurons and the weight bias improve the model's fitting and generalization capabilities. The next hidden layer is connected to the previous hidden layer through a set of weights. The feature information input to each hidden layer is output after being weighted and processed by a preset activation function. The activation function realizes nonlinear transformation, and the model can process complex nonlinear relationships. The output layer outputs the results of all hidden layer processing. The model structure can accurately capture the relationship between the features of the data to be processed and time, and improve the accuracy of the model's prediction results. By setting up a prediction model, the data to be processed can be known in advance, pre-splitting and pre-allocation of cloud-edge resources can be achieved, which is convenient for subsequent direct processing of real-time data, improving data processing efficiency and reducing latency.

[0017] Third, the feature information extracted from the historical data to be processed is input into the model through the input layer; the extracted feature information is passed from the output layer to the first hidden layer, the first hidden layer weights the feature information and processes it with a preset activation function and then outputs it to the second hidden layer, the second hidden layer weights the feature information and processes it with a preset activation function and then outputs it to the third hidden layer, until all hidden layers are processed and output; the preset mean square error function calculates the error signal between the processing result and the expected value, and the preset rule calculates the gradient of the error signal with respect to the weight of each hidden layer; the training process of iteratively updating and reversely updating the weight of each hidden layer through the gradient descent algorithm so that the error signal is less than the preset error threshold, so that the model has the ability to accurately capture the relationship between data and time, and achieve accurate prediction of the data to be processed.

[0018] Fourth, the gradient of the error signal with respect to the weight of each hidden layer is obtained through a preset mean square error function to obtain the slope of the loss function in the weight space; a certain specified step size is moved along the direction of the largest slope to update the weight of each hidden layer until the error signal between the processing result and the expected value is less than a preset error threshold or reaches a preset number of iterations, so that the hidden layer of the model reaches a relatively optimal weight in a very short time, thereby optimizing the model.

[0019] Fifth, by creating a circular buffer and setting multiple partitions in the circular buffer, the data to be processed is input into the circular buffer, split according to type logic and distributed to different partitions, thus realizing data splitting and facilitating the subsequent packaging of the same type of data to be processed and sending it to the matching edge server or cloud server for processing, thereby improving data processing efficiency and reducing latency.

[0020] Sixth, by extracting features from each processing result of the same data source, constructing feature vectors corresponding to each processing result based on the features of each processing result, and assigning corresponding weights to each of them; performing weighted summation on the feature vectors corresponding to each processing result to obtain a final feature vector, and converting the final feature vector into a final processing result, data integration is achieved after efficient processing of data of the same terminal device on different servers, and the terminal device executes corresponding actions based on the final result of the integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the resource allocation method based on cloud-edge collaboration of the present invention.

[0022] Figure 2 A schematic diagram of the training process of the data prediction model of the present invention.

[0023] Figure 3This is a schematic diagram of the architecture of the resource allocation system based on cloud-edge collaboration of the present invention. DETAILED DESCRIPTION

[0024] The following is combined with Figure 1~Figure 3 The present invention is further described in detail: Example 1 See attached Figure 1 As shown, a resource allocation method based on cloud-edge collaboration includes the following steps: S1: deploy several relay nodes, establish a connection channel between each relay node and a designated edge node, and a connection channel between each relay node and a designated terminal device, obtain the historical data to be processed of each terminal device, extract features and annotate them, and construct training samples and verification samples in a preset ratio. The ratio of training samples to verification samples is seven to three, so that the training effect that meets the specified requirements is achieved, and then the verification is performed through the verification samples.

[0025] Before feature extraction, data preprocessing is required to remove duplicate data and invalid data and fill in missing values. The data annotation process includes annotating the data with the extracted features, adding corresponding tags and metadata, and improving the quality of data annotation through multiple rounds of iterative processing.

[0026] S2: Create a data prediction model on the relay node, input the training sample into the data prediction model for model training, and input the verification sample into the data prediction model to verify the model performance. When the data prediction model performance meets the preset requirements, execute step S3, otherwise re-train the model; S3: The data prediction model predicts the data to be processed of each terminal device, and the relay node pre-splits the data to be processed of each terminal device according to the predicted data to be processed, and adds a virtual tag to each pre-split data unit, wherein the virtual tag includes data source information, data type, data volume and delay requirements, and matches the corresponding edge node according to the data type and data volume involved in the pre-split information of the data to be processed of each terminal device. The purpose of creating a data prediction model is to predict the data to be processed, and pre-split it based on the predicted data, and then match the data to be processed generated by different terminals to the corresponding edge server and cloud server based on the pre-split results. After real-time collection of the data to be processed, it is only necessary to verify whether the prediction result is accurate. If it is accurate, it is directly split according to the pre-split scheme, and the same type of data is packaged and sent to the matching edge server or cloud server for processing, which can reduce the delay. Because the performance of different servers is different, the data types processed are also different, and some data require special servers to process, and some data can only be processed by cloud servers. Matching servers according to data types can effectively improve data processing efficiency and the quality of processing results.

[0027] S4: Collect the data to be processed generated by each terminal device during operation in real time and transmit it to the corresponding relay node. The relay node compares the real-time data to be processed with the predicted data to be processed to determine whether the real-time data to be processed exceeds the predicted data to be processed. If so, re-split the part of the data that exceeds the predicted data to be processed, and execute step S5 for the part of the data that does not exceed. If not, directly execute step S5. Due to the high prediction accuracy of the model, only in very rare cases may the prediction result be inaccurate. Therefore, through model prediction, the efficiency of data splitting and server matching is greatly improved.

[0028] S5: directly split the real-time to-be-processed data of each terminal device according to the pre-splitting scheme to obtain different data units, pack the split data units of the same type, construct to-be-processed data packets of each data type, and transmit the to-be-processed data packets of each data type to the matching edge node or cloud; S6: The cloud or each edge node performs corresponding processing according to the received data packets to be processed, generates corresponding data results, and packages them after retaining the marks, and returns the processing result data packets to the relay node. The relay node splits the processing result data packets received from each edge node, integrates the processing results of the same data source according to the marks, and returns the integrated results of each data source to the corresponding terminal device; S7: Each of the terminal devices performs a corresponding action according to the received integration result.

[0029] As the cloud-edge collaborative computing model develops, the scale of cloud-edge collaboration grows, and the number of edge nodes and terminal devices continues to increase, the pressure on central management and control has doubled. The existing cloud-edge collaborative architecture sends the tasks generated by the terminal devices to the matching edge nodes for calculation. However, with the sharp increase in the number of edge nodes and terminal devices, the matching difficulty between tasks and edge nodes increases, which can easily lead to the accumulation of tasks on a certain edge node, reducing the efficiency of task processing.

[0030] Therefore, the present invention deploys several relay nodes to obtain historical data to be processed and inputs it into a data prediction model for training; the data prediction model predicts the data to be processed of each terminal device, pre-splits the predicted data to be processed, matches the corresponding edge node according to the data type and data volume involved in the pre-split information, collects the data to be processed of each terminal device in real time, compares the real-time data to be processed with the predicted data to be processed, and when the real-time data to be processed exceeds the prediction, the excess part is split again, otherwise it is directly split to obtain different data units, and the split data units of the same type are packaged and transmitted to the matching edge node; each edge node generates a result after processing and returns it to the relay node, splits each processing result data packet, integrates it according to the processing results of the same data source, and returns it to the terminal device, thereby realizing the coordination between the resource allocation of the edge node and the cloud and the data generated by large-scale terminal devices. Since the same type of data to be processed can be sent to the matching edge server or cloud server for processing, the processing efficiency is higher.

[0031] Example 2 Based on Example 1, the specific process of feature extraction in step S1 is as follows: S11: constructing a feature selection model, inputting a labeled data set into the feature selection model, fitting the feature selection model through the labeled data set, and in the process of fitting the model, controlling the sparsity of the feature by adjusting a regularization parameter; S12: acquiring feature coefficients in the historical data to be processed based on the trained feature selection model, and sorting the data according to the feature coefficients according to a specified rule; S13: setting a feature coefficient threshold for the sorted feature coefficients by cross-validation, and extracting all features whose coefficients are greater than the feature coefficient threshold.

[0032] In the prior art, the feature extraction process is generally to extract features whose frequency is greater than a specified threshold by counting the keyword frequency in the statistical data set. This method of feature extraction has low efficiency and accuracy. The present invention constructs a feature selection model, inputs a labeled data set into the feature selection model, fits the feature selection model through the labeled data set, and in the process of fitting the model, controls the sparsity of the features by adjusting the regularization parameter; obtains the feature coefficients in the historical data to be processed based on the trained feature selection model, and sorts them according to the specified rules based on the feature coefficients; sets a feature coefficient threshold for the sorted feature coefficients by cross-validation, and extracts all features whose coefficients are greater than the feature coefficient threshold, thereby realizing the accurate extraction of features in the historical data to be processed, providing a data basis for the training of subsequent data prediction models, improving training efficiency, and making the model have good robustness and generalization.

[0033] Example 3 On the basis of Embodiment 2, the data prediction model in step S2 includes an input layer, a hidden layer and an output layer, the input layer is connected to the hidden layer, and the hidden layer is connected to the output layer; The number of neurons in the input layer is set to the number of features extracted in step S13, which is used to input feature information of the data to be processed; The hidden layer is provided with multiple layers, each hidden layer is provided with a number of neurons, and the neurons are provided with a set of weights and a bias. The next hidden layer is connected to the previous hidden layer through a set of weights, and the bias is used to adjust the activation threshold of the neurons of the corresponding hidden layer. The feature information input to each hidden layer is output after being processed by weighting and a preset activation function; The output layer outputs the results processed by all hidden layers.

[0034] The data prediction model sets up input layer, hidden layer and output layer. The number of neurons in the input layer is set to match the number of extracted features. The hidden layer has multiple layers, and each hidden layer has several neurons. Neurons can help the model learn complex patterns and features by extracting and converting input features. Several neurons are set up with a set of weights and a bias. The number of neurons and weight bias improve the model's fitting ability and generalization ability. The next hidden layer is connected to the previous hidden layer through a set of weights. The feature information input to each hidden layer is output after weighting and processing with a preset activation function. The activation function realizes nonlinear transformation and enables the model to process complex nonlinear relationships. The output layer outputs the results of all hidden layer processing. The model structure can accurately capture the relationship between the features of the data to be processed and time, and improve the accuracy of the model's prediction results. By setting up a prediction model, the data to be processed can be known in advance, pre-splitting and pre-allocation of cloud-edge resources can be realized, which is convenient for subsequent direct processing of real-time data and can improve data processing efficiency.

[0035] In this embodiment, the training process of the data prediction model is as follows: S21: input the feature information extracted from the historical data to be processed into the model through the input layer; S22: The extracted feature information is transmitted from the output layer to the first layer of the hidden layer, the first hidden layer performs weighting and preset activation function processing on the feature information and then outputs it to the second hidden layer, the second hidden layer performs weighting and preset activation function processing on the feature information and then outputs it to the third hidden layer, until all hidden layers are processed and arrive at the output layer, and the output layer outputs the processing result; S23: calculating the error signal between the processing result and the expected value by a preset mean square error function, and calculating the gradient of the error signal with respect to the weight of each hidden layer by a preset rule, wherein the gradient is the degree of influence of the weight change on the error reduction; S24: Iteratively update and reversely update the weights of each hidden layer through a gradient descent algorithm until the error signal between the processing result and the expected value is less than a preset error threshold.

[0036] The specific process of iteratively updating and reversely updating the weights of each hidden layer by the gradient descent algorithm in step S24 is as follows: S241: Obtain the gradient of the error signal with respect to the weight of each hidden layer through a preset mean square error function to obtain the slope of the loss function in the weight space; S242: Move along the direction with the largest slope by a certain specified step length, and update the weight of each hidden layer until the error signal between the processing result and the expected value is less than a preset error threshold or reaches a preset number of iterations.

[0037] The feature information extracted from the historical data to be processed is input into the model through the input layer; the extracted feature information is passed from the output layer to the first hidden layer, the first hidden layer weights the feature information and processes it with a preset activation function and then outputs it to the second hidden layer, the second hidden layer weights the feature information and processes it with a preset activation function and then outputs it to the third hidden layer, until all hidden layers are processed and output; the preset mean square error function calculates the error signal between the processing result and the expected value, and the preset rule calculates the gradient of the error signal with respect to the weight of each hidden layer; the training process of iteratively updating and reversely updating the weight of each hidden layer through the gradient descent algorithm so that the error signal is less than the preset error threshold, so that the model has the ability to accurately capture the relationship between data and time, and achieve accurate prediction of the data to be processed.

[0038] The gradient of the error signal with respect to the weight of each hidden layer is obtained through a preset mean square error function to obtain the slope of the loss function in the weight space; a certain specified step size is moved along the direction of the maximum slope to update the weight of each hidden layer until the error signal between the processing result and the expected value is less than the preset error threshold or reaches the preset number of iterations, so that the hidden layer of the model reaches the relatively optimal weight in a very short time, and the model is optimized.

[0039] Example 4 Based on Example 3, the specific process of splitting the data to be processed in step S5 is as follows: S51: create a ring buffer and set multiple partitions in the ring buffer; S52: Input the data to be processed into the ring buffer, split it according to the type logic, and distribute it to the partitions respectively.

[0040] By creating a circular buffer and setting multiple partitions in the circular buffer, the data to be processed is input into the circular buffer, split according to type logic and distributed to different partitions. This realizes the splitting of data, facilitates the subsequent packaging of the same type of processed data and sends it to the matching edge server or cloud server for processing, thereby improving data processing efficiency and reducing latency.

[0041] In this embodiment, the specific process of data integration in step S6 is as follows: S61: extracting features from each processing result of the same data source according to the process of step S11 to step S13; S62: constructing feature vectors corresponding to the respective processing results from the features of the respective processing results, and assigning corresponding weights to the respective processing results; S63: Perform weighted summation on the feature vectors corresponding to the processing results to obtain a final feature vector, and convert the final feature vector into a final processing result.

[0042] By extracting features from each processing result of the same data source, a feature vector corresponding to each processing result is constructed from the features of each processing result, and corresponding weights are assigned to each of them; the feature vectors corresponding to each processing result are weightedly summed to obtain a final feature vector, and the final feature vector is converted into a final processing result, thereby achieving data integration of the same terminal device after efficient processing on different servers, and the terminal device performs corresponding actions based on the final result of the integration.

[0043] A resource allocation system based on cloud-edge collaboration is used to implement a resource allocation method based on cloud-edge collaboration, including a relay node deployment module, a data acquisition module, a data prediction model, a data splitting module, a data packaging module and a data integration module, wherein the relay node deployment module is connected to the relay node, the data acquisition module is connected to the terminal device, the data splitting module is connected to the relay node, the data prediction model is connected to the relay node and the data splitting module, the data splitting module is connected to the data packaging module, and the data integration module is connected to the edge node and the terminal device. The relay node deployment module is used to deploy a number of relay nodes, establish a connection channel between each relay node and a designated edge node, and a connection channel between each relay node and a designated terminal device.

[0044] The data acquisition module is used to obtain the historical data to be processed of each terminal device and to collect the data to be processed generated by each terminal device during operation in real time, and transmit it to the corresponding relay node. The data prediction model is used to predict the data to be processed of each terminal device; the data splitting module is used to split and process the data to be processed generated by each terminal device during operation; the data packaging module is used to split the part of the data that exceeds the predicted data to be processed and directly split the real-time data to be processed of each terminal device according to the pre-splitting scheme to obtain different data units, package the split data units of the same type, construct data packets to be processed of each data type, and transmit the data packets to be processed of each data type to the matching edge node; the data integration module is used to integrate the processing results of the same data source according to the tag, and return the integration results of each data source to the corresponding terminal device.

[0045] In summary, the resource allocation method and system based on cloud-edge collaboration provided by the present invention deploys several relay nodes, obtains historical data to be processed and inputs the data prediction model for training; the data prediction model predicts the data to be processed of each terminal device, pre-splits the predicted data to be processed, matches the corresponding edge node according to the data type and data volume involved in the pre-split information, collects the data to be processed of each terminal device in real time, compares the real-time data to be processed with the predicted data to be processed, and when the real-time data to be processed exceeds the prediction, the excess part is split again, otherwise it is directly split to obtain different data units, and the split data units of the same type are packaged and transmitted to the matching edge node; each edge node generates a result after processing and returns it to the relay node, splits each processing result data packet, integrates it according to the processing results of the same data source, and returns it to the terminal device. The collaboration between the resource allocation of edge nodes and the cloud and the data generated by large-scale terminal devices is realized.

[0046] By constructing a feature selection model, the feature coefficients in the historical data to be processed are obtained, and the data are sorted according to the specified rules based on the feature coefficients; the feature coefficient threshold is set for the sorted feature coefficients by cross-validation, and all features whose coefficients are greater than the feature coefficient threshold are extracted, so as to realize the accurate extraction of features in the historical data to be processed, provide a data basis for the subsequent training of the data prediction model, improve the training efficiency, and make the model have good robustness and generalization. The number of neurons in the input layer is set to match the number of extracted features; the hidden layer has multiple layers, and each hidden layer has several neurons. Neurons can help the model learn complex patterns and features by extracting and converting the input features. Several neurons are each provided with a set of weights and a bias. The number of neurons and the weight bias improve the fitting ability and generalization ability of the model. The next hidden layer is connected to the previous hidden layer through a set of weights. The feature information input to each hidden layer is output after being weighted and processed by a preset activation function. The nonlinear transformation is realized through the activation function, so that the model can process complex nonlinear relationships. The output layer outputs the results of all hidden layer processing. The model structure can accurately capture the relationship between the features and time of the data to be processed, and improve the accuracy of the model's prediction results. By setting up a prediction model, the data to be processed can be known in advance, and pre-splitting and pre-allocation of cloud-edge resources can be achieved, which facilitates the subsequent direct processing of real-time data, improves data processing efficiency, and reduces latency.

[0047] The feature information extracted from the historical data to be processed is input into the model through the input layer; the extracted feature information is transmitted from the output layer to the first layer of the hidden layer, the first hidden layer weights the feature information and processes it with a preset activation function, and then outputs it to the second hidden layer; the second hidden layer weights the feature information and processes it with a preset activation function, and then outputs it to the third hidden layer, until all hidden layers are processed and output; the preset mean square error function calculates the error signal between the processing result and the expected value, and the preset rule calculates the gradient of the error signal with respect to the weight of each hidden layer; the training process of iteratively updating and reversely updating the weight of each hidden layer through the gradient descent algorithm so that the error signal is less than the preset error threshold, so that the model has the ability to accurately capture the relationship between data and time, and realizes accurate prediction of the data to be processed. The gradient of the error signal with respect to the weight of each hidden layer is obtained through the preset mean square error function to obtain the slope of the loss function in the weight space; a certain specified step length is moved along the direction of the maximum slope, and the weight of each hidden layer is updated until the error signal between the processing result and the expected value is less than the preset error threshold or reaches the preset number of iterations, so that the model reaches the relatively optimal weight of its hidden layer in a very short time, so that the model is optimized.

[0048] By creating a circular buffer and setting multiple partitions in the circular buffer, the data to be processed is input into the circular buffer, split according to the type logic and distributed to different partitions, the data is split, which is convenient for the subsequent packaging of the same type of data to be processed and sent to the matching edge server or cloud server for processing, improving data processing efficiency and reducing latency. By extracting features from each processing result of the same data source, the feature vectors corresponding to each processing result are constructed from the features of each processing result, and the corresponding weights are assigned respectively; the feature vectors corresponding to each processing result are weighted summed to obtain the final feature vector, and the final feature vector is converted into the final processing result, which realizes the data integration of the same terminal device after efficient processing on different servers, and the terminal device performs corresponding actions based on the final result of the integration.

Claims

1. A resource allocation method based on cloud-edge collaboration, characterized in that: The following steps are involved: S1: deploy several relay nodes, establish a connection channel between each relay node and a designated edge node, and a connection channel between each relay node and a designated terminal device, obtain the historical data to be processed of each terminal device, extract features and annotate them, and construct training samples and verification samples in a preset ratio; S2: Create a data prediction model on the relay node, input the training sample into the data prediction model for model training, and input the verification sample into the data prediction model to verify the model performance. When the data prediction model performance meets the preset requirements, execute step S3, otherwise re-train the model; S3: The data prediction model predicts the data to be processed of each terminal device, and the relay node pre-splits the data to be processed of each terminal device according to the predicted data to be processed, and adds a virtual tag to each pre-split data unit, wherein the virtual tag includes data source information, data type, data volume and latency requirement, and matches the corresponding edge node according to the data type and data volume involved in the pre-split information of the data to be processed of each terminal device. S4: collecting the data to be processed generated by each terminal device during operation in real time and transmitting it to the corresponding relay node. The relay node compares the real-time data to be processed with the predicted data to be processed to determine whether the real-time data to be processed exceeds the predicted data to be processed. If so, the part of the data that exceeds the predicted data to be processed is re-split, and the part of the data that does not exceed the step S5 is executed. If not, step S5 is directly executed; S5: directly split the real-time to-be-processed data of each terminal device according to the pre-splitting scheme to obtain different data units, pack the split data units of the same type, construct to-be-processed data packets of each data type, and transmit the to-be-processed data packets of each data type to the matching edge node or cloud; S6: The cloud or each edge node performs corresponding processing according to the received data packets to be processed, generates corresponding data results, and packages them after retaining the marks, and returns the processing result data packets to the relay node. The relay node splits the processing result data packets received from each edge node, integrates the processing results of the same data source according to the marks, and returns the integrated results of each data source to the corresponding terminal device; S7: Each of the terminal devices performs a corresponding action according to the received integration result.

2. According to claim 1, a resource allocation method based on cloud-edge collaboration is characterized in that: The specific process of feature extraction in step S1 is as follows: S11: constructing a feature selection model, inputting a labeled data set into the feature selection model, fitting the feature selection model through the labeled data set, and in the process of fitting the model, controlling the sparsity of the feature by adjusting a regularization parameter; S12: acquiring feature coefficients in the historical data to be processed based on the trained feature selection model, and sorting the data according to the feature coefficients according to a specified rule; S13: setting a feature coefficient threshold for the sorted feature coefficients by cross-validation, and extracting all features whose coefficients are greater than the feature coefficient threshold.

3. The resource allocation method based on cloud-edge collaboration according to claim 2 is characterized in that: The data prediction model in step S2 includes an input layer, a hidden layer and an output layer, the input layer is connected to the hidden layer, and the hidden layer is connected to the output layer; The number of neurons in the input layer is set to the number of features extracted in step S13, which is used to input feature information of the data to be processed; The hidden layer is provided with multiple layers, each hidden layer is provided with a number of neurons, and the neurons are provided with a set of weights and a bias. The next hidden layer is connected to the previous hidden layer through a set of weights, and the bias is used to adjust the activation threshold of the neurons of the corresponding hidden layer. The feature information input to each hidden layer is output after being processed by weighting and a preset activation function; The output layer outputs the results processed by all hidden layers.

4. The resource allocation method based on cloud-edge collaboration according to claim 3 is characterized in that: The training process of the data prediction model is as follows: S21: input the feature information extracted from the historical data to be processed into the model through the input layer; S22: The extracted feature information is transmitted from the output layer to the first layer of the hidden layer, the first hidden layer performs weighting and preset activation function processing on the feature information and then outputs it to the second hidden layer, the second hidden layer performs weighting and preset activation function processing on the feature information and then outputs it to the third hidden layer, until all hidden layers are processed and arrive at the output layer, and the output layer outputs the processing result; S23: calculating the error signal between the processing result and the expected value by a preset mean square error function, and calculating the gradient of the error signal with respect to the weight of each hidden layer by a preset rule, wherein the gradient is the degree of influence of the weight change on the error reduction; S24: Iteratively update and reversely update the weights of each hidden layer through a gradient descent algorithm until the error signal between the processing result and the expected value is less than a preset error threshold.

5. The resource allocation method based on cloud-edge collaboration according to claim 4 is characterized in that: The specific process of iteratively updating and reversely updating the weights of each hidden layer by the gradient descent algorithm in step S24 is as follows: S241: Obtain the gradient of the error signal with respect to the weight of each hidden layer through a preset mean square error function to obtain the slope of the loss function in the weight space; S242: Move along the direction with the largest slope by a certain specified step length, and update the weight of each hidden layer until the error signal between the processing result and the expected value is less than a preset error threshold or reaches a preset number of iterations.

6. The resource allocation method based on cloud-edge collaboration according to claim 1, characterized in that: The specific process of splitting the data to be processed in step S5 is as follows: S51: create a ring buffer and set multiple partitions in the ring buffer; S52: Input the data to be processed into the ring buffer, split it according to the type logic, and distribute it to the partitions respectively.

7. The resource allocation method based on cloud-edge collaboration according to claim 2, characterized in that: The specific process of data integration in step S6 is as follows: S61: extracting features from each processing result of the same data source according to the process of step S11 to step S13; S62: constructing feature vectors corresponding to the respective processing results from the features of the respective processing results, and assigning corresponding weights to the respective processing results; S63: Perform weighted summation on the feature vectors corresponding to the processing results to obtain a final feature vector, and convert the final feature vector into a final processing result.

8. A resource allocation system based on cloud-edge collaboration, used to implement a resource allocation method based on cloud-edge collaboration as described in any one of claims 1 to 7, characterized in that: It includes a relay node deployment module, a data acquisition module, a data prediction model, a data splitting module, a data packaging module and a data integration module, wherein the relay node deployment module is connected to the relay node, the data acquisition module is connected to the terminal device, the data splitting module is connected to the relay node, the data prediction model is connected to the relay node and the data splitting module, the data splitting module is connected to the data packaging module, and the data integration module is connected to the edge node and the terminal device; The relay node deployment module is used to deploy a number of relay nodes, establish a connection channel between each relay node and a designated edge node, and a connection channel between each relay node and a designated terminal device; The data acquisition module is used to obtain the historical data to be processed of each terminal device and to collect the data to be processed generated by each terminal device during operation in real time, and transmit it to the corresponding relay node; The data prediction model is used to predict the data to be processed of each terminal device; The data splitting module is used to split and process the to-be-processed data generated by each terminal device during operation; The data packaging module is used to split the part of the data that exceeds the predicted data to be processed and directly split the real-time data to be processed of each terminal device according to the pre-splitting scheme to obtain different data units, pack the split data units of the same type, construct data packets to be processed of each data type, and transmit the data packets to be processed of each data type to the matching edge node; The data integration module is used to integrate the processing results of the same data source according to the mark, and return the integration results of each data source to the corresponding terminal device.

Images