A computing power processing system and method based on artificial intelligence
Through the computing power processing method based on artificial intelligence, task resource data is collected and optimized in real time, and the delay, energy consumption, accuracy and security problems of computing power management systems in the existing technology are solved, efficient resource integration and scheduling are achieved, and the improvement of the Computational performance and real-time.
Patent Information
- Application Number
- CN202410622591.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-05-20
AI Technical Summary
The existing computing power management system has shortcomings in terms of latency, energy consumption, accuracy and safety, and cannot effectively integrate and schedule computing resources.
Using a computing power processing method based on artificial intelligence, we collect task resource data in real time, filter, extract, and classify, and sort and allocate CPU resources according to data priorities and correlations, and optimize resource scheduling using distributed computing and prediction algorithms.
It improves the rationality and efficiency of task resource data processing, reduces redundant data and non-standard data, reduces processing delay, and improves calculation accuracy and real-time.
Smart Images

Figure CN118445083B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computing resource data processing, and specifically to a computing power processing system and method based on artificial intelligence. Background Art
[0002] As the demand for computing power continues to grow, the proposal of computing power network has become an important solution to support the development of intelligent society. The core idea of computing power network is to efficiently dispatch computing power, storage and other resources between users and computing power nodes through network links to meet the service needs of intelligent business. However, due to the need for efficient integration and real-time scheduling of resources, the latency, energy consumption, accuracy, and security of the current computing power management system cannot meet the requirements.
[0003] Chinese patent CN114168313A configures the charging strategy generation model and the charging network simulation environment as the first target task and the second target task respectively, and determines the target number of physical machines required by the computing unit at the same time, allocates the physical machines required by the computing unit to the computing unit according to the target number of physical machines required by the computing unit, and finally adjusts the task goals of the physical machines in real time according to the operating status of the computing unit. However, the computing unit cannot complete the efficient integration of resources and has great limitations. Summary of the invention
[0004] 1. Technical issues to be resolved
[0005] In view of the shortcomings of the prior art, the present invention provides an artificial intelligence-based computing power processing system and method, which has the advantages of real-time and high efficiency, and solves the problems that the current computing power management system cannot meet the requirements in terms of latency, energy consumption, accuracy, and security.
[0006] (II) Technical solution
[0007] In order to solve the technical problems that the latency, energy consumption, accuracy, and security of the current computing power management system cannot meet the requirements, the present invention provides the following technical solutions:
[0008] The present invention discloses a computing power processing method based on artificial intelligence, comprising the following steps:
[0009] S1, collect the task resource data that needs to be calculated in real time and save it to the task resource pool;
[0010] S2. Setting a task resource data standard in the task resource pool, and preprocessing the task resource data in the task resource pool to obtain processed task resource data;
[0011] S3, setting a task resource data priority based on the processed task resource data, sorting the processed task resource data according to the set task resource data priority to obtain sorted task resource data, and constructing a waiting queue according to the sorted task resource data to obtain task resource data of allocated system resources and task resource data of unallocated system resources;
[0012] S4, calculating the correlation between the task resource data of the allocated system resources, obtaining independent tasks and partially reused tasks, and allocating CPUs to the independent tasks and partially reused tasks based on an allocation algorithm;
[0013] S5. Calculate the independent tasks and some reused tasks based on a distributed computing method to obtain the processing time of any independent task or some reused task;
[0014] S6. Predicting the time cost required to complete processing of all independent tasks and some reused tasks based on the prediction allocation algorithm, and processing task resource data of unallocated system resources based on the predicted time cost;
[0015] The present invention processes task resource data by filtering, extracting and classifying the task resource data collected in real time, and sets the task resource data priority based on the processed task resource data, and sorts the processed task resource data according to the set task resource data priority. After the sorting is completed, the processing order of the task resource data is determined by calculating the correlation between the sorted task resource data. At the same time, in the process of processing the task resource data, the waiting time of other processing task resource data is recorded in real time, and the task resource data is scheduled based on the waiting time, thereby improving the rationality and efficiency of the task resource data calculation.
[0016] Preferably, the setting of task resource data standards in the task resource pool includes:
[0017] Setting task resource data includes: task resource parameters, required resource parameters, and task resource identifiers;
[0018] The required resource parameters are set to include the resource type and quantity required for the current task resources;
[0019] Set the task resource ID to 0 or 1. When the task resource ID is 0, the current task resource data is normal data. When the task resource ID is 1, the current task resource data is urgent data.
[0020] The task resource parameters are set including: the task resource arrival time, the source address of the task resource, the task resource content and the task resource data length.
[0021] Preferably, the preprocessing of the task resource data in the task resource pool comprises the following steps:
[0022] S21, task resource data filtering;
[0023] S211. Establish data standards and filter abnormal task resource data;
[0024] Set the length standard of task resource data, delete the task resource data that exceeds the length standard, and add 0 to the end of the task resource data that is less than the length standard;
[0025] S212, filtering and removing duplicate task resource data;
[0026] Filter task resource data based on the Bloom filter algorithm:
[0027] Create an array of length m and select A hash function traverses each data in the task resource data and saves the traversal results in an array;
[0028] When two data have the same traversal results during the hash function traversal, each bit in the two data is compared.
[0029] When the comparison results are consistent, the two current data are set to be the same data, and based on the arrival time of the task resources, the task resource data that arrives later is deleted;
[0030] After the traversal is completed, the task resource data stored in the array during the traversal process is summarized to obtain the filtered task resource data;
[0031] S22, extracting features of the filtered task resource data;
[0032] The filtered task resource data is input into the convolutional neural network for feature extraction;
[0033] The convolutional neural network calculation formula is as follows:
[0034] f = I × z + b;
[0035] Where I represents the input filtered task resource data, z represents the weight of the corresponding convolution kernel, b represents the bias value, and f represents the output feature;
[0036] For the features of the extracted task resource data, the feature dimension reduction is performed through the pooling layer;
[0037] The data is processed by the activation function during feature extraction;
[0038] Set the data set Q of each task resource data = (x1, x2, ..., x s ) as input array, each array x i The corresponding input weight is z i , b is the offset, and the output results obtained after inputting these input values into the convolutional neural network are:
[0039]
[0040] Among them, g is the corresponding activation function, x i represents the i-th array in the task resource data, y represents the feature array after being processed by the activation function, and s represents the number of input arrays in the data set;
[0041] Convolve the features of the task resource data in a continuously stacked convolution and pooling process;
[0042] Pass the convolved task resource data features to the fully connected layer;
[0043] The features are expanded and combined through the fully connected layer to obtain the feature array and save it;
[0044] S23, classifying the extracted feature data, thereby classifying the filtered task resource data, wherein the classification result includes the classified task resource data;
[0045] The classified task resource data are aggregated to obtain processed task resource data.
[0046] Preferably, the classification of the task resource data after feature extraction comprises the following steps:
[0047] S232, setting the number of divided clusters to K;
[0048] When the number of clusters is less than K, the binary division method is used;
[0049] Steps of binary division:
[0050] Calculate the average value of the feature data and use the calculated average value as the basis for cluster division, dividing the original cluster into two clusters on average;
[0051] Calculating the mean of feature data is as follows:
[0052]
[0053] in, is the average value of the feature data calculated in the current cluster, y max is the maximum value of the feature data in the current cluster, y min is the minimum value of the characteristic data in the current cluster;
[0054] S233, repeatedly performing binary division on the divided clusters until the number of divided clusters is equal to K;
[0055] S234, after the division is completed, the variance of the feature data in each cluster is calculated, and the variance threshold is set;
[0056]
[0057] Among them, ε 2 is the variance of the feature data in the current cluster, y j represents the jth feature data in the current cluster, and n represents the number of feature data in the current cluster;
[0058] When the variance of the feature data in each cluster after division is less than or equal to the set variance threshold, it means that the current clustering result is the optimal result;
[0059] When the variance of the feature data in each cluster after division is greater than the set variance threshold, it means that the current clustering result is not the optimal result;
[0060] When the clustering result is not the optimal result, set K=K+1, repeat S232 and S233 until the value of K no longer increases, the iteration stops, and the optimal result after clustering is output, and the optimal result after clustering is set as the classification result.
[0061] The present invention completes the processing of task resource data by setting task resource data standards in a task resource pool and filtering, extracting and classifying the task resource data, thereby reducing redundant data and non-standard data in the task resource data and improving the efficiency of task resource data processing.
[0062] Preferably, setting the task resource data priority based on the processed task resource data, sorting the processed task resource data according to the set task resource data priority, and constructing a waiting queue according to the sorted task resource data includes:
[0063] For each type of task resource data in the processed task resource data, a sorting queue of each type of task resource data is constructed based on the classified task resource data;
[0064] Sorting based on the task resource identifier in the task resource data and the arrival time of each task resource data;
[0065] Put the task resource identified as 1 and the first arriving task resource data at the top of the sorting queue;
[0066] Select the task resource data at the top of each sorting queue to build a waiting queue;
[0067] The task resource data in the waiting queue is set as the task resource data of the allocated system resources, and the task resource data not in the waiting queue is set as the task resource data of the unallocated system resources.
[0068] Preferably, the step of calculating the correlation between the task resource data of the allocated system resources, obtaining the independent tasks and the partially reused tasks, and allocating CPUs to the independent tasks and the partially reused tasks based on the allocation algorithm comprises the following steps:
[0069] The Pearson correlation coefficient is used to calculate the correlation between the task resource data in the waiting queue:
[0070] Pearson correlation coefficient:
[0071]
[0072] Where γ is the Pearson correlation coefficient, COV(α,β) is the covariance between task resource data α and task resource data β, D(α) is the variance of task resource data α, and D(β) is the variance of task resource data β;
[0073] Thresholds were set based on the calculated Pearson correlation coefficient;
[0074] When the Pearson correlation coefficient between two sets of task resource data exceeds a set threshold, the two sets of task resource data are set to be correlated;
[0075] Since the calculated Pearson correlation coefficient ranges from [-1, 1], when the calculated Pearson correlation coefficient between two task resource data is [-1, 0], the two task resource data are set to be independent tasks;
[0076] When the Pearson correlation coefficient calculated between two task resource data is (0,1], the two task resource data are assumed to be partially reused tasks;
[0077] Based on the correlation between the task resource data in the waiting queue, the task resource data is divided into independent tasks and partially reused tasks;
[0078] Set the resource data of the first a tasks with the highest Pearson correlation coefficient in some reused tasks to prioritize CPU allocation;
[0079] When the first a task resource data with the highest Pearson correlation coefficient in the partially reused task are executed, CPU is allocated to other task resource data in the partially reused task first;
[0080] When all tasks in the partially reused task are completed, the CPU is allocated according to the arrival time of the task resource data in the independent task.
[0081] The present invention determines the correlation between two groups of task resource data by calculating the Pearson correlation coefficient, and reduces the search delay in the task resource data processing process by giving priority to the task resource data with high correlation, thereby improving the data processing performance.
[0082] Preferably, the step of calculating the independent tasks and some reused tasks in a distributed computing manner to obtain the processing time of any independent task or some reused task comprises the following steps:
[0083] Distributed computing is as follows:
[0084]
[0085] Among them, H(t) represents the processing time, t represents time, H represents the function related to the processing time, f represents the proportion of reusable computing to the total computing, τ represents the search delay, Calculate duration.
[0086] The present invention calculates the computing time of independent tasks and partially reused tasks by using a distributed computing method, quantifies the task resource data processing process, and improves the accuracy of task resource data processing.
[0087] Preferably, the predicting of the time cost required to complete the processing of all independent tasks and some reused tasks, and processing the task resource data of the unallocated system resources based on the predicted time cost comprises the following steps:
[0088] The estimated time cost required to complete all independent tasks and some reused tasks is as follows:
[0089]
[0090] Where P(t) is the time cost required to complete the processing of all independent tasks and some reused tasks, and H a (t) represents the processing time of the ath task resource data, and q is the number of concurrent input task resource data;
[0091] Calculate the waiting weight of task resource data of unallocated system resources;
[0092]
[0093] Wherein, w is the waiting weight of the task resource data of the system resource that has not been allocated, and Twait is the waiting time of the task resource data of the system resource that has not been allocated;
[0094] Real-time sorting is performed based on the calculated waiting weights, and task resource data with the largest waiting weight among the task resource data of the unallocated system resources is processed first.
[0095] The present invention comprehensively considers the sorting of task resource data of unallocated system resources by combining distribution calculation and calculation of waiting weights, and makes real-time adjustments to the sorting of task resource data of unallocated system resources, thereby improving the real-time and rationality of the data processing process.
[0096] The present invention also discloses a computing power processing system based on artificial intelligence, comprising: a data collection module, a task resource pool, a data filtering module, a feature extraction module, a data classification module and a data calculation module;
[0097] The data collection module is used to collect task resource data that needs to be calculated in real time and store the data in a task resource pool;
[0098] The task resource pool is used to store task resource data from the data collection module;
[0099] The data filtering module is used to extract task resource data from the task resource pool for filtering, and transmit the filtered task resource data to the feature extraction module;
[0100] The feature extraction module is used to receive the filtered task resource data, perform feature extraction through a convolutional neural network, and transmit the extracted feature data to the data classification module;
[0101] The data classification module is used to classify the received extracted feature data and transmit the classified data to the data calculation module;
[0102] The data calculation module is used to perform distributed calculation and scheduling control on the received data.
[0103] (III) Beneficial effects
[0104] Compared with the prior art, the present invention provides a computing power processing system and method based on artificial intelligence, which has the following features:
[0105] Beneficial effects:
[0106] 1. The invention processes task resource data by filtering, extracting and classifying the task resource data collected in real time, and sets the task resource data priority based on the processed task resource data, and sorts the processed task resource data according to the set task resource data priority. After the sorting is completed, the processing order of the task resource data is determined by calculating the correlation between the sorted task resource data. At the same time, in the process of processing the task resource data, the waiting time of other processing task resource data is recorded in real time, and the task resource data is scheduled based on the waiting time, thereby improving the rationality and efficiency of the task resource data calculation.
[0107] 2. The invention completes the processing of task resource data by setting task resource data standards in the task resource pool and filtering, extracting and classifying the task resource data, thereby reducing redundant data and non-standard data in the task resource data and improving the efficiency of task resource data processing.
[0108] 3. The invention determines the correlation between two groups of task resource data by calculating the Pearson correlation coefficient, and reduces the search delay in the task resource data processing process by giving priority to the task resource data with high correlation, thereby improving the data processing performance.
[0109] 4. The invention calculates the computing time of independent tasks and partially reused tasks by using distributed computing, quantifies the task resource data processing process, and improves the accuracy of task resource data processing.
[0110] 5. The present invention comprehensively considers the sorting of task resource data of unallocated system resources by combining distribution calculation and calculation of waiting weights, and makes real-time adjustments to the sorting of task resource data of unallocated system resources, thereby improving the real-time and rationality of the data processing process. BRIEF DESCRIPTION OF THE DRAWINGS
[0111] Figure 1 It is a schematic diagram of the flow structure of the computing power processing method based on artificial intelligence of the present invention. DETAILED DESCRIPTION
[0112] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0113] The present invention discloses a computing power processing method based on artificial intelligence, which specifically comprises the following steps:
[0114] S1, collect the task resource data that needs to be calculated in real time and save it to the task resource pool;
[0115] S2. Setting a task resource data standard in the task resource pool, and preprocessing the task resource data in the task resource pool to obtain processed task resource data;
[0116] The task resource data standards in the task resource pool are as follows:
[0117] Setting task resource data includes: task resource parameters, required resource parameters, and task resource identifiers;
[0118] The required resource parameters are set to include the resource type and quantity required for the current task resources;
[0119] Set the task resource ID to 0 or 1. When the task resource ID is 0, the current task resource data is normal data. When the task resource ID is 1, the current task resource data is urgent data.
[0120] Furthermore, the task resource parameters include: task resource arrival time, task resource source address, task resource content, and task resource data length;
[0121] The preprocessing of the task resource data in the task resource pool comprises the following steps:
[0122] S21, task resource data filtering;
[0123] S211. Establish data standards and filter abnormal task resource data;
[0124] Set the length standard of task resource data, delete the task resource data that exceeds the length standard, and add 0 to the end of the task resource data that is less than the length standard;
[0125] S212, filtering and removing duplicate task resource data;
[0126] Filter task resource data based on the Bloom filter algorithm:
[0127] Create an array of length m and select A hash function traverses each data in the task resource data and saves the traversal results in an array;
[0128] When two data have the same traversal results during the hash function traversal, each bit in the two data is compared.
[0129] When the comparison results are consistent, the two current data are set to be the same data, and based on the arrival time of the task resources, the task resource data that arrives later is deleted;
[0130] After the traversal is completed, the task resource data stored in the array during the traversal process is summarized to obtain the filtered task resource data;
[0131] S22, extracting features of the filtered task resource data;
[0132] The filtered task resource data is input into the convolutional neural network for feature extraction;
[0133] The convolutional neural network calculation formula is as follows:
[0134] f = I × z + b;
[0135] Where I represents the input filtered task resource data, z represents the weight of the corresponding convolution kernel, b represents the bias value, and f represents the output feature;
[0136] For the features of the extracted task resource data, the feature dimension reduction is performed through the pooling layer;
[0137] Furthermore, the data is processed by the activation function during feature extraction;
[0138] Set the data set Q of each task resource data = (x1, x2, ..., x s ) as input array, each array x i The corresponding input weight is z i , b is the offset, and the output results obtained after inputting these input values into the convolutional neural network are:
[0139]
[0140] Among them, g is the corresponding activation function, x i represents the i-th array in the task resource data, y represents the feature array after being processed by the activation function, and s represents the number of input arrays in the data set;
[0141] Furthermore, the features of the task resource data are convolved in a continuously stacked convolution and pooling process;
[0142] Pass the convolved task resource data features to the fully connected layer;
[0143] The features are expanded and combined through the fully connected layer to obtain the feature array and save it;
[0144] S23, classifying the extracted feature data, thereby classifying the filtered task resource data, wherein the classification result includes the classified task resource data;
[0145] Aggregating the classified task resource data to obtain processed task resource data;
[0146] Classify the feature data of the extracted task resource data based on the bi-kmeans clustering algorithm;
[0147] S231, summarizing all feature data extracted by the convolutional neural network to form a cluster;
[0148] S232, setting the number of divided clusters to K;
[0149] Furthermore, when the number of clusters is less than K, the binary division method is used;
[0150] Steps of binary division:
[0151] Calculate the average value of the feature data and use the calculated average value as the basis for cluster division, dividing the original cluster into two clusters on average;
[0152] Calculating the mean of feature data is as follows:
[0153]
[0154] in, is the average value of the feature data calculated in the current cluster, y max is the maximum value of the feature data in the current cluster, y min is the minimum value of the characteristic data in the current cluster;
[0155] S233, repeatedly performing binary division on the divided clusters until the number of divided clusters is equal to K;
[0156] S234, after the division is completed, the variance of the feature data in each cluster is calculated, and the variance threshold is set;
[0157]
[0158] Among them, ε 2 is the variance of the feature data in the current cluster, y j represents the jth feature data in the current cluster, and n represents the number of feature data in the current cluster;
[0159] When the variance of the feature data in each cluster after division is less than or equal to the set variance threshold, it means that the current clustering result is the optimal result;
[0160] When the variance of the feature data in each cluster after division is greater than the set variance threshold, it means that the current clustering result is not the optimal result;
[0161] When the clustering result is not the optimal result, set K=K+1, repeat S232 and S233 until the value of K no longer increases, the iteration stops, and the optimal result after clustering is output, and the optimal result after clustering is set as the classification result;
[0162] S3, setting a task resource data priority based on the processed task resource data, sorting the processed task resource data according to the set task resource data priority to obtain sorted task resource data, and constructing a waiting queue according to the sorted task resource data to obtain task resource data of allocated system resources and task resource data of unallocated system resources;
[0163] For each type of task resource data in the processed task resource data, a sorting queue of each type of task resource data is constructed based on the classified task resource data;
[0164] Sorting based on the task resource identifier in the task resource data and the arrival time of each task resource data;
[0165] Put the task resource identified as 1 and the first arriving task resource data at the top of the sorting queue;
[0166] Select the task resource data at the top of each sorting queue to build a waiting queue;
[0167] The task resource data in the waiting queue is set as the task resource data of the allocated system resources, and the task resource data not in the waiting queue is set as the task resource data of the unallocated system resources;
[0168] S4, calculating the correlation between the task resource data of the allocated system resources, obtaining independent tasks and partially reused tasks, and allocating CPUs to the independent tasks and partially reused tasks based on an allocation algorithm;
[0169] The Pearson correlation coefficient is used to calculate the correlation between the task resource data in the waiting queue:
[0170] Pearson correlation coefficient:
[0171]
[0172] Where γ is the Pearson correlation coefficient, COV(α,β) is the covariance between task resource data α and task resource data β, D(α) is the variance of task resource data α, and D(β) is the variance of task resource data β;
[0173] Thresholds were set based on the calculated Pearson correlation coefficient;
[0174] When the Pearson correlation coefficient between two sets of task resource data exceeds a set threshold, the two sets of task resource data are set to be correlated;
[0175] Since the calculated Pearson correlation coefficient ranges from [-1, 1], when the calculated Pearson correlation coefficient between two task resource data is [-1, 0], the two task resource data are set to be independent tasks;
[0176] When the Pearson correlation coefficient calculated between two task resource data is (0,1], the two task resource data are assumed to be partially reused tasks;
[0177] Based on the correlation between the task resource data in the waiting queue, the task resource data is divided into independent tasks and partially reused tasks;
[0178] Set the resource data of the first a tasks with the highest Pearson correlation coefficient in some reused tasks to prioritize CPU allocation;
[0179] When the first a task resource data with the highest Pearson correlation coefficient in the partially reused task are executed, CPU is allocated to other task resource data in the partially reused task first;
[0180] When all tasks in the partially reused task are completed, the CPU is allocated according to the arrival time of the task resource data in the independent task;
[0181] S5. Calculate the independent tasks and some reused tasks based on a distributed computing method to obtain the processing time of any independent task or some reused task;
[0182] Distributed computing is as follows:
[0183]
[0184] Among them, H(t) represents the processing time, t represents time, H represents the function related to the processing time, f represents the proportion of reusable computing to the total computing, τ represents the search delay, Calculate duration;
[0185] S6. Predicting the time cost required to complete processing of all independent tasks and some reused tasks based on the prediction allocation algorithm, and processing task resource data of unallocated system resources based on the predicted time cost;
[0186] The estimated time cost required to complete all independent tasks and some reused tasks is as follows:
[0187]
[0188] Where P(t) is the time cost required to complete the processing of all independent tasks and some reused tasks, and H a (t) represents the processing time of the ath task resource data, and q is the number of concurrent input task resource data;
[0189] Calculate the waiting weight of task resource data of unallocated system resources;
[0190]
[0191] Wherein, w is the waiting weight of the task resource data of the system resource that has not been allocated, and Twait is the waiting time of the task resource data of the system resource that has not been allocated;
[0192] Real-time sorting is performed based on the calculated waiting weights, and task resource data with the largest waiting weight among the task resource data of the unallocated system resources is processed first;
[0193] The present invention also discloses a computing power processing system based on artificial intelligence, comprising: a data collection module, a task resource pool, a data filtering module, a feature extraction module, a data classification module and a data calculation module;
[0194] The data collection module is used to collect task resource data that needs to be calculated in real time and store the data in a task resource pool;
[0195] The task resource pool is used to store task resource data from the data collection module;
[0196] The data filtering module is used to extract task resource data from the task resource pool for filtering, and transmit the filtered task resource data to the feature extraction module;
[0197] The feature extraction module is used to receive the filtered task resource data, perform feature extraction through a convolutional neural network, and transmit the extracted feature data to the data classification module;
[0198] The data classification module is used to classify the received extracted feature data and transmit the classified data to the data calculation module;
[0199] The data calculation module is used to perform distributed calculation and scheduling control on the received data.
[0200] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A computing power processing method based on artificial intelligence, It is characterized in that The following steps are involved: S1, collect the task resource data that needs to be calculated in real time and save it to the task resource pool; S2. Setting a task resource data standard in the task resource pool, and preprocessing the task resource data in the task resource pool to obtain processed task resource data; S3, setting a task resource data priority based on the processed task resource data, sorting the processed task resource data according to the set task resource data priority to obtain sorted task resource data, and constructing a waiting queue according to the sorted task resource data to obtain task resource data of allocated system resources and task resource data of unallocated system resources; S4, calculating the correlation between the task resource data of the allocated system resources, obtaining independent tasks and partially reused tasks, and allocating CPUs to the independent tasks and partially reused tasks based on an allocation algorithm; S5. Calculate the independent tasks and some reused tasks based on a distributed computing method to obtain the processing time of any independent task or some reused task; S6. Predicting the time cost required to complete processing of all independent tasks and some reused tasks based on the prediction allocation algorithm, and processing task resource data of unallocated system resources based on the predicted time cost; The calculation of the correlation between the task resource data of the allocated system resources, obtaining the independent tasks and the partially reused tasks, and allocating CPUs to the independent tasks and the partially reused tasks based on the allocation algorithm comprises the following steps: The Pearson correlation coefficient is used to calculate the correlation between the task resource data in the waiting queue: Pearson correlation coefficient: Where γ is the Pearson correlation coefficient, COV(α,β) is the covariance between task resource data α and task resource data β, D(α) is the variance of task resource data α, and D(β) is the variance of task resource data β; Thresholds were set based on the calculated Pearson correlation coefficient; When the Pearson correlation coefficient between two sets of task resource data exceeds a set threshold, the two sets of task resource data are set to be correlated; Since the calculated Pearson correlation coefficient ranges from [-1, 1], when the calculated Pearson correlation coefficient between two task resource data is [-1, 0], the two task resource data are set to be independent tasks; When the Pearson correlation coefficient calculated between two task resource data is (0,1], the two task resource data are assumed to be partially reused tasks; Based on the correlation between the task resource data in the waiting queue, the task resource data is divided into independent tasks and partially reused tasks; Set the resource data of the first a tasks with the highest Pearson correlation coefficient in some reused tasks to prioritize CPU allocation; When the first a task resource data with the highest Pearson correlation coefficient in the partially reused task are executed, CPU is allocated to other task resource data in the partially reused task first; When all tasks in the partially reused task are completed, the CPU is allocated according to the arrival time of the task resource data in the independent task; The method of calculating the independent tasks and some reused tasks based on the distributed computing method to obtain the processing time of any independent task or some reused task includes the following steps: Distributed computing is as follows: Among them, H(t) represents the processing time, t represents time, H represents the function related to the processing time, f represents the proportion of reusable computing to the total computing, τ represents the search delay, Calculate duration.
2. According to the computing power processing method based on artificial intelligence as described in claim 1, It is characterized in that The task resource data standard for setting the task resource pool includes: Setting task resource data includes: task resource parameters, required resource parameters, and task resource identifiers; The required resource parameters are set to include the resource type and quantity required for the current task resources; Set the task resource ID to 0 or 1. When the task resource ID is 0, the current task resource data is normal data. When the task resource ID is 1, the current task resource data is urgent data. The task resource parameters are set including: the task resource arrival time, the source address of the task resource, the task resource content and the task resource data length.
3. According to the computing power processing method based on artificial intelligence as described in claim 1, It is characterized in that The preprocessing of the task resource data in the task resource pool comprises the following steps: S21, task resource data filtering; S211. Establish data standards and filter abnormal task resource data; Set the length standard of task resource data, delete the task resource data that exceeds the length standard, and add 0 to the end of the task resource data that is less than the length standard; S212, filtering and removing duplicate task resource data; Filter task resource data based on the Bloom filter algorithm: Create an array of length m and select A hash function traverses each data in the task resource data and saves the traversal results in an array; When two data have the same traversal results during the hash function traversal, each bit in the two data is compared. When the comparison results are consistent, the two current data are set to be the same data, and based on the arrival time of the task resources, the task resource data that arrives later is deleted; After the traversal is completed, the task resource data stored in the array during the traversal process is summarized to obtain the filtered task resource data; S22, extracting features of the filtered task resource data; The filtered task resource data is input into the convolutional neural network for feature extraction; The convolutional neural network calculation formula is as follows: f = I × z + b; Where I represents the input filtered task resource data, z represents the weight of the corresponding convolution kernel, b represents the bias value, and f represents the output feature; For the features of the extracted task resource data, the feature dimension reduction is performed through the pooling layer; The data is processed by the activation function during feature extraction; Set the data set Q of each task resource data = (x 1 ,x 2 ,...,x s ) as input array, each array x i The corresponding input weight is z i , b is the offset, and the output results obtained after inputting these input values into the convolutional neural network are: Among them, g is the corresponding activation function, x i represents the i-th array in the task resource data, y represents the feature array after being processed by the activation function, and s represents the number of input arrays in the data set; Convolve the features of the task resource data in a continuously stacked convolution and pooling process; Pass the convolved task resource data features to the fully connected layer; The features are expanded and combined through the fully connected layer to obtain the feature array and save it; S23, classifying the extracted feature data, thereby classifying the filtered task resource data, wherein the classification result includes the classified task resource data; The classified task resource data are aggregated to obtain processed task resource data.
4. According to the computing power processing method based on artificial intelligence as claimed in claim 3, It is characterized in that The classification of the task resource data after feature extraction comprises the following steps: S232, setting the number of divided clusters to K; When the number of clusters is less than K, the binary division method is used; Steps of binary division: Calculate the average value of the feature data and use the calculated average value as the basis for cluster division, dividing the original cluster into two clusters on average; Calculating the mean of feature data is as follows: in, is the average value of the feature data calculated in the current cluster, y max is the maximum value of the feature data in the current cluster, y min is the minimum value of the characteristic data in the current cluster; S233, repeatedly performing binary division on the divided clusters until the number of divided clusters is equal to K; S234, after the division is completed, the variance of the feature data in each cluster is calculated, and the variance threshold is set; Among them, ε 2 is the variance of the feature data in the current cluster, y j represents the jth feature data in the current cluster, and n represents the number of feature data in the current cluster; When the variance of the feature data in each cluster after division is less than or equal to the set variance threshold, it means that the current clustering result is the optimal result; When the variance of the feature data in each cluster after division is greater than the set variance threshold, it means that the current clustering result is not the optimal result; When the clustering result is not the optimal result, set K=K+1, repeat S232 and S233 until the value of K no longer increases, the iteration stops, and the optimal result after clustering is output, and the optimal result after clustering is set as the classification result.
5. The computing power processing method based on artificial intelligence according to claim 1, It is characterized in that The step of setting the task resource data priority based on the processed task resource data, sorting the processed task resource data according to the set task resource data priority, and constructing a waiting queue according to the sorted task resource data includes: For each type of task resource data in the processed task resource data, a sorting queue of each type of task resource data is constructed based on the classified task resource data; Sorting based on the task resource identifier in the task resource data and the arrival time of each task resource data; Put the task resource identified as 1 and the first arriving task resource data at the top of the sorting queue; Select the task resource data at the top of each sorting queue to build a waiting queue; The task resource data in the waiting queue is set as the task resource data of the allocated system resources, and the task resource data not in the waiting queue is set as the task resource data of the unallocated system resources.
6. The method for processing computing power based on artificial intelligence according to claim 1, It is characterized in that The method of predicting the time cost required to complete the processing of all independent tasks and some reused tasks, and processing the task resource data of the unallocated system resources based on the predicted time cost includes the following steps: The estimated time cost required to complete all independent tasks and some reused tasks is as follows: Where P(t) is the time cost required to complete the processing of all independent tasks and some reused tasks, and H a (t) represents the processing time of the ath task, and q is the number of concurrent input tasks; Calculate the waiting weight of task resource data of unallocated system resources; Wherein, w is the waiting weight of the task resource data of the system resource that has not been allocated, and Twait is the waiting time of the task resource data of the system resource that has not been allocated; Real-time sorting is performed based on the calculated waiting weights, and task resource data with the largest waiting weight among the task resource data of the unallocated system resources is processed first.
7. An artificial intelligence-based computing power processing system for implementing the artificial intelligence-based computing power processing method according to any one of claims 1 to 6, It is characterized in that include: Data collection module, task resource pool, data filtering module, feature extraction module, data classification module and data calculation module; The data collection module is used to collect task resource data that needs to be calculated in real time and store the data in a task resource pool; The task resource pool is used to store task resource data from the data collection module; The data filtering module is used to extract task resource data from the task resource pool for filtering, and transmit the filtered task resource data to the feature extraction module; The feature extraction module is used to receive the filtered task resource data, perform feature extraction through a convolutional neural network, and transmit the extracted feature data to the data classification module; The data classification module is used to classify the received extracted feature data and transmit the classified data to the data calculation module; The data calculation module is used to perform distributed calculation and scheduling control on the received data.
Citation Information
Patent Citations
Computing power scheduling system
CN114168313A
Method and system for forecasting task resource waiting time
CN104516784A
A cloud platform resource utilization prediction method and terminal equipment
CN109714395A
Medium-and-long-term power load combined prediction system and method based on cloud calculation technology
CN111340291A
Predictive apparatus and method for predicting workload group metrics of a workload management system of a database system
US20210004675A1
