Computer-aided data analysis method and system based on dynamic resource scheduling

Abstract

The present invention discloses a computer-assisted data analysis method and system based on dynamic resource scheduling, and relates to the field of computer technology. The method comprises: obtaining a computer-assisted data analysis task to be executed, and analyzing the task characteristics of each computer-assisted data analysis task; wherein the task characteristics include data scale, computational complexity and expected output, so as to determine the resource requirements of each computer-assisted data analysis task; constructing a dependency graph between computer-assisted data analysis tasks by using a graph theory algorithm, so as to identify the data flow dependencies and control flow dependencies between computer-assisted data analysis tasks; dynamically scheduling the computing resources of a computer according to the data flow dependencies, control flow dependencies and the resource requirements of each computer-assisted data analysis task; executing the computer-assisted data analysis task by using the scheduled computing resources; and outputting the data analysis results of the computer-assisted data analysis task.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a computer-aided data analysis method and system based on dynamic resource scheduling. Background Art

[0002] Computer-aided analysis is a research work that uses relevant computer systems to conduct statistics, analysis, simulation, and reasoning on information for a specific object and give corresponding statistical analysis results.

[0003] In traditional computer-aided data analysis methods, resource allocation is often static and cannot be dynamically adjusted according to the data volume and real-time needs of the analysis task. This results in resources not being fully utilized or resource bottlenecks when the data volume surges or the complexity of the analysis task changes.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a computer-aided data analysis method and system based on dynamic resource scheduling to solve the above technical problems.

[0006] The present application provides a computer-assisted data analysis method based on dynamic resource scheduling, including: obtaining computer-assisted data analysis tasks to be executed, and analyzing the task characteristics of each computer-assisted data analysis task; wherein the task characteristics include data scale, computational complexity and expected output, so as to determine the resource requirements of each computer-assisted data analysis task; using a graph theory algorithm to construct a dependency graph between the computer-assisted data analysis tasks, so as to identify the data flow dependencies and control flow dependencies between the computer-assisted data analysis tasks; dynamically scheduling the computing resources of a computer according to the data flow dependencies, the control flow dependencies and the resource requirements of each computer-assisted data analysis task; using the scheduled computing resources to execute the computer-assisted data analysis task; and outputting the data analysis results of the computer-assisted data analysis task.

[0007] The present application provides a computer-assisted data analysis system based on dynamic resource scheduling, including: a computer-assisted data analysis task acquisition module, used to acquire computer-assisted data analysis tasks to be executed, and analyze the task characteristics of each computer-assisted data analysis task; wherein the task characteristics include data scale, computational complexity and expected output, so as to determine the resource requirements of each computer-assisted data analysis task; a task dependency identification module, used to construct a dependency graph between the computer-assisted data analysis tasks using a graph theory algorithm, so as to identify the data flow dependencies and control flow dependencies between the computer-assisted data analysis tasks; a resource dynamic scheduling module, used to dynamically schedule the computing resources of a computer according to the data flow dependencies, the control flow dependencies and the resource requirements of each computer-assisted data analysis task; an auxiliary data analysis module, used to execute the computer-assisted data analysis task using the scheduled computing resources; and an analysis result output module, used to output the data analysis results of the computer-assisted data analysis task.

[0008] Based on the embodiments provided in the present application, computer-assisted data analysis tasks to be executed are obtained, and the task characteristics of each computer-assisted data analysis task are analyzed; wherein the task characteristics include data scale, computational complexity and expected output, so as to determine the resource requirements of each computer-assisted data analysis task; a dependency graph between computer-assisted data analysis tasks is constructed using a graph theory algorithm to identify the data flow dependencies and control flow dependencies between computer-assisted data analysis tasks; the computing resources of the computer are dynamically scheduled according to the data flow dependencies, control flow dependencies and the resource requirements of each computer-assisted data analysis task; the computer-assisted data analysis tasks are executed using the scheduled computing resources; and the data analysis results of the computer-assisted data analysis tasks are output. Therefore, through the dynamic resource scheduling mechanism, resource allocation can be adjusted in real time according to the actual needs of data analysis tasks, thereby improving resource utilization and reducing resource waste; through dynamic resource scheduling, unnecessary resource occupation can be reduced, especially in cloud environments, which helps to reduce computing costs and alleviate economic burdens; it can adapt to the growth of data volume and the increase in the complexity of analysis tasks, so that computer-aided data analysis methods can be flexibly expanded to meet the needs of large-scale data analysis; by dynamically scheduling the computer's computing resources, while ensuring the performance of data analysis tasks, resource usage is optimized and cost control is achieved, especially in cloud service environments, costs can be predicted and controlled more accurately; dynamic resource scheduling can ensure that key tasks obtain the necessary resources, thereby speeding up task execution and improving overall data analysis efficiency; it can quickly respond to changes in data analysis tasks, dynamically adjust resource allocation, and enhance the adaptability and flexibility of the system to cope with changing data analysis needs; it can effectively handle the dependencies between data analysis tasks, optimize the task scheduling process, and improve the execution efficiency of complex data analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0010] Figure 1 is a flowchart of an optional computer-aided data analysis method based on dynamic resource scheduling according to an embodiment of the present application;

[0011] Figure 2 This is a structural diagram of an optional computer-aided data analysis system based on dynamic resource scheduling according to an embodiment of the present application.

[0012] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0013] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0014] Alternatively, if Figure 1 As shown, the present application provides a computer-assisted data analysis method based on dynamic resource scheduling, comprising:

[0015] S101, obtaining a computer-assisted data analysis task to be performed, and analyzing the task characteristics of each computer-assisted data analysis task; wherein the task characteristics include data size, computational complexity, and expected output, so as to determine the resource requirements of each computer-assisted data analysis task;

[0016] S102, constructing a dependency graph between computer-assisted data analysis tasks using a graph theory algorithm to identify data flow dependencies and control flow dependencies between the computer-assisted data analysis tasks;

[0017] S103, dynamically scheduling the computing resources of the computer according to the data flow dependency, the control flow dependency and the resource requirements of each computer-assisted data analysis task;

[0018] S104, using the scheduled computing resources to perform computer-assisted data analysis tasks;

[0019] S105, outputting the data analysis results of the computer-assisted data analysis task.

[0020] Furthermore, when dynamically scheduling the computing resources of a computer, resource scheduling can be performed across multiple computing environments; wherein the multiple computing environments include a cloud computing environment and a local computing environment.

[0021] Based on the embodiments provided in the present application, computer-assisted data analysis tasks to be executed are obtained, and the task characteristics of each computer-assisted data analysis task are analyzed; wherein the task characteristics include data scale, computational complexity and expected output, so as to determine the resource requirements of each computer-assisted data analysis task; a dependency graph between computer-assisted data analysis tasks is constructed using a graph theory algorithm to identify the data flow dependencies and control flow dependencies between computer-assisted data analysis tasks; the computing resources of the computer are dynamically scheduled according to the data flow dependencies, control flow dependencies and the resource requirements of each computer-assisted data analysis task; the computer-assisted data analysis tasks are executed using the scheduled computing resources; and the data analysis results of the computer-assisted data analysis tasks are output. Therefore, through the dynamic resource scheduling mechanism, resource allocation can be adjusted in real time according to the actual needs of data analysis tasks, thereby improving resource utilization and reducing resource waste; through dynamic resource scheduling, unnecessary resource occupation can be reduced, especially in cloud environments, which helps to reduce computing costs and alleviate economic burdens; it can adapt to the growth of data volume and the increase in the complexity of analysis tasks, so that computer-aided data analysis methods can be flexibly expanded to meet the needs of large-scale data analysis; by dynamically scheduling the computer's computing resources, while ensuring the performance of data analysis tasks, resource usage is optimized and cost control is achieved, especially in cloud service environments, costs can be predicted and controlled more accurately; dynamic resource scheduling can ensure that key tasks obtain the necessary resources, thereby speeding up task execution and improving overall data analysis efficiency; it can quickly respond to changes in data analysis tasks, dynamically adjust resource allocation, and enhance the adaptability and flexibility of the system to cope with changing data analysis needs; it can effectively handle the dependencies between data analysis tasks, optimize the task scheduling process, and improve the execution efficiency of complex data analysis tasks.

[0022] Alternatively, if Figure 2 As shown, the present application provides a computer-aided data analysis system based on dynamic resource scheduling, and the system implements the method as claimed in claim 1, including:

[0023] A computer-assisted data analysis task acquisition module 201 is used to acquire computer-assisted data analysis tasks to be performed, and analyze the task characteristics of each computer-assisted data analysis task; wherein the task characteristics include data size, computational complexity and expected output, so as to determine the resource requirements of each computer-assisted data analysis task;

[0024] A task dependency identification module 202 is used to construct a dependency graph between computer-assisted data analysis tasks using a graph theory algorithm to identify data flow dependencies and control flow dependencies between computer-assisted data analysis tasks;

[0025] A resource dynamic scheduling module 203, used to dynamically schedule the computing resources of the computer according to the data flow dependency, the control flow dependency and the resource requirements of each computer-assisted data analysis task;

[0026] The auxiliary data analysis module 204 is used to perform computer-assisted data analysis tasks using the scheduled computing resources;

[0027] The analysis result output module 205 is used to output the data analysis results of the computer-assisted data analysis task.

[0028] Furthermore, the graph theory algorithm is a spectrum-based graph neural network; the task dependency identification module uses the graph theory algorithm to construct a dependency graph between computer-assisted data analysis tasks to identify data flow dependencies and control flow dependencies between computer-assisted data analysis tasks, and is configured as follows:

[0029] Each computer-assisted data analysis task is represented as a task node; wherein the feature vector of the task node contains the task characteristics of the computer-assisted data analysis task;

[0030] A task graph is constructed using task nodes and edges between task nodes; wherein the edges between task nodes represent the dependency relationship between computer-aided data analysis tasks;

[0031] Using the graph Lasso algorithm, the Laplacian matrix of the task graph is constructed by estimating the correlation between computer-aided data analysis tasks. The Laplacian matrix is ​​used to represent the eigenvectors of the task nodes in the task graph and the connection strength between the task nodes.

[0032] Apply Chebyshev spectral graph convolution operation to perform approximate graph convolution operation based on the Laplacian matrix of the constructed task graph to capture the static dependencies between computer-aided data analysis tasks;

[0033] Among them, the Chebyshev spectral graph convolution operation is the ChebConv operation. The approximate graph convolution operation here refers to the use of a spectrum-based method to achieve the effect of graph convolution. It is a method for performing feature transformation on a graph structure that can capture the local connection pattern of nodes.

[0034] Integrate the time series data of the computer-assisted data analysis task into the time fusion transformation model; wherein the time series data includes the start time, end time and resource usage of the computer-assisted data analysis task; the time fusion transformation model processes the time series data through a gated residual network and a variable selection network;

[0035] Among them, the temporal fusion transformation model is TemporalFusionTransformer, the gated residual network is GatedResidualNetwork, and the variable selection network is VariableSelectionNetwork.

[0036] Use a temporal fusion transformation model to learn long-term dependencies between computer-aided data analysis tasks and select dynamically changing features through a gating mechanism and a variable selection network;

[0037] Combining the static dependencies and dynamic change characteristics between computer-assisted data analysis tasks, a dependency graph between computer-assisted data analysis tasks is constructed to identify the data flow dependencies and control flow dependencies between computer-assisted data analysis tasks.

[0038] Furthermore, the graph Lasso algorithm is used to estimate the correlation between computer-aided data analysis tasks and construct the Laplacian matrix of the task graph, which is configured as:

[0039] Construct the adjacency matrix A of the task graph; where the adjacency matrix A is a square matrix, A ij represents the correlation strength between computer-assisted data analysis task i and computer-assisted data analysis task j; if there is a dependency relationship between computer-assisted data analysis task i and computer-assisted data analysis task j, then A ij is non-zero; if there is no direct dependency, then A ij is 0;

[0040] Construct the degree matrix D of the task graph; where the degree matrix D is a diagonal matrix, D kk represents the degree of task node k, that is, the number of tasks directly connected to the computer-assisted data analysis task corresponding to task node k;

[0041] Using the graph Lasso algorithm, the correlation between computer-aided data analysis tasks is estimated by minimizing the regularized objective function and the Laplacian matrix L of the task graph is constructed.

[0042] in,

[0043] Further,

[0044] The approximate graph convolution operation g×X is configured as:

[0045]

[0046] Among them, g is the graph convolution kernel; θ l is the learning parameter; T kis the kth order Chebyshev polynomial; L is the order of the Chebyshev polynomial; X is the feature vector of the task node in the task graph.

[0047] Furthermore, a temporal fusion transformation model is used to learn long-term dependencies between computer-aided data analysis tasks, and dynamically changing features are selected through a gating mechanism and a variable selection network, which is configured as follows:

[0048] Use adaptive weight matrix to combine features of different time scales and construct time fusion feature vector;

[0049]

[0050] in, represents the time fusion feature vector of the task node e1 at time point t. The fusion feature vector fuses the features of the task node e1 at different time scales and the features of other task nodes related to the node e1. S is a preset time scale set. is a learnable weight matrix corresponding to the task node e1 and the time scale s. The learnable weight matrix is ​​used to adjust the contribution of different time scale features to the fusion feature vector; represents the feature vector of task node e1 at time point ts; N(e1) represents the set of other task nodes directly connected to task node e1; A e1e2 It is the adjacency matrix element between task node e1 and task node e2, indicating the dependency relationship between the two task nodes; FFT stands for Fast Fourier Transform, which is used to extract frequency domain features and identify periodic patterns and trends;

[0051] Define the long-term dependency coefficient L and learn the long-term dependency relationship in time series data through the recurrent neural network RNN; Among them, α represents the network parameters of RNN;

[0052] Applying a gating mechanism to the temporally fused feature vector;

[0053] Based on the fused feature vector and the long-term dependency coefficient L after applying the gating mechanism, a variable selection network is used to select features whose relevance to the computer-aided data analysis task is higher than a preset threshold;

[0054] Based on the output of the gating mechanism and the output of the variable selection network, the dynamic change characteristics are determined.

[0055] Furthermore, the resource dynamic scheduling module dynamically schedules the computing resources of the computer according to the data flow dependency, the control flow dependency and the resource requirements of each computer-assisted data analysis task, and is configured as follows:

[0056] Based on the output of the temporal fusion transformation model, the resource requirements of each computer-aided data analysis task are corrected;

[0057] Calculate the priority reference information of each computer-assisted data analysis task according to the constructed dependency graph between the computer-assisted data analysis tasks and the corrected resource requirement of each computer-assisted data analysis task;

[0058] Determine a real-time resource scheduling strategy; wherein the real-time resource scheduling strategy is used to dynamically schedule the computer's computing resources based on the priority reference information of the computer-assisted data analysis task and the revised resource requirements of each computer-assisted data analysis task.

[0059] Furthermore, the real-time resource scheduling strategy includes:

[0060] Monitor resource usage in computer systems in real time;

[0061] An improved greedy algorithm combined with the nearest deadline condition is adopted to calculate the remaining resources of the slices in real time and compare the slice service weights according to the revised resource requirements of each computer-assisted data analysis task. At the same time, real-time priority queuing adjustment of computer-assisted data analysis tasks is performed at the user scheduling layer.

[0062] Resource usage in a computer system provides a macro perspective that shows the allocation and usage status of resources in the entire system. This includes the overall usage of resources such as CPU, memory, storage, and network bandwidth.

[0063] The remaining resources of a slice are calculated based on the resource usage in the computer system. Specifically, for each specific task or service (referred to as a "slice"), its remaining resources refer to the portion allocated to the task but not yet used.

[0064] Understanding resource usage in a computer system helps scheduling algorithms determine which resources are idle, or which are being used less frequently and can be reallocated to tasks with fewer remaining resources in the slice.

[0065] Furthermore, the resource dynamic scheduling module can perform resource scheduling across multiple computing environments; wherein the multiple computing environments include cloud computing environments and local computing environments.

[0066] It should be noted that in the present application, the embodiments implemented on the computer-assisted data analysis system side based on dynamic resource scheduling can be cross-referenced with the embodiments implemented on the computer-assisted data analysis method side based on dynamic resource scheduling, and the present application will not describe them one by one.

[0067] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A computer-aided data analysis method based on dynamic resource scheduling, characterized in that: include: Acquire computer-assisted data analysis tasks to be performed, and analyze the task characteristics of each computer-assisted data analysis task; wherein the task characteristics include data size, computational complexity, and expected output, so as to determine the resource requirements of each computer-assisted data analysis task; Constructing a dependency graph between the computer-assisted data analysis tasks using a graph theory algorithm to identify data flow dependencies and control flow dependencies between the computer-assisted data analysis tasks; Dynamically schedule the computing resources of the computer according to the data flow dependency, the control flow dependency and the resource requirements of each computer-assisted data analysis task; Utilizing the scheduled computing resources to perform the computer-assisted data analysis task; Output the data analysis results of the computer-assisted data analysis task.

2. The computer-aided data analysis method based on dynamic resource scheduling according to claim 1, characterized in that: When dynamically scheduling the computing resources of a computer, the resources can be scheduled across multiple computing environments; wherein the multiple computing environments include cloud computing environments and local computing environments.

3. A computer-aided data analysis system based on dynamic resource scheduling, wherein the system implements the method according to claim 1, characterized in that: include: A computer-assisted data analysis task acquisition module, used to acquire computer-assisted data analysis tasks to be performed, and analyze the task characteristics of each computer-assisted data analysis task; wherein the task characteristics include data scale, computational complexity and expected output, so as to determine the resource requirements of each computer-assisted data analysis task; The task dependency identification module is used to construct a dependency graph between the computer-assisted data analysis tasks using a graph theory algorithm to identify the data flow dependencies and control flow dependencies between the computer-assisted data analysis tasks.

4. A computer-aided data analysis system based on dynamic resource scheduling, wherein the system implements the method according to claim 3, characterized in that: include: A resource dynamic scheduling module, used for dynamically scheduling the computing resources of the computer according to the data flow dependency, the control flow dependency and the resource requirements of each computer-assisted data analysis task; An auxiliary data analysis module, used to perform the computer-assisted data analysis task using the scheduled computing resources; The analysis result output module is used to output the data analysis result of the computer-assisted data analysis task.

5. The computer-aided data analysis system based on dynamic resource scheduling according to claim 4, characterized in that: The graph theory algorithm is a spectrum-based graph neural network; the task dependency identification module uses the graph theory algorithm to construct a dependency graph between the computer-assisted data analysis tasks to identify the data flow dependency and control flow dependency between the computer-assisted data analysis tasks, and is configured as follows: Representing each computer-assisted data analysis task as a task node; wherein the feature vector of the task node contains the task characteristics of the computer-assisted data analysis task; A task graph is constructed using the task nodes and the edges between the task nodes; wherein the edges between the task nodes represent the dependency relationships between the computer-assisted data analysis tasks.

6. The computer-aided data analysis system based on dynamic resource scheduling according to claim 5, characterized in that: Using a graph Lasso algorithm, by estimating the correlation between the computer-aided data analysis tasks, a Laplace matrix of the task graph is constructed; wherein the Laplace matrix is ​​used to represent the eigenvectors of the task nodes in the task graph and the connection strength between the task nodes; A Chebyshev spectral graph convolution operation is applied to perform an approximate graph convolution operation based on the Laplacian matrix of the constructed task graph to capture the static dependencies between the computer-aided data analysis tasks.

7. The computer-aided data analysis system based on dynamic resource scheduling according to claim 6, characterized in that: Integrating the time series data of the computer-assisted data analysis task into a time fusion transformation model; wherein the time series data includes the start time, end time and resource usage of the computer-assisted data analysis task; the time fusion transformation model processes the time series data through a gated residual network and a variable selection network; Using the temporal fusion transformation model to learn the long-term dependencies between the computer-aided data analysis tasks, and selecting dynamically changing features through a gating mechanism and a variable selection network; In combination with the static dependencies and dynamic change characteristics between the computer-assisted data analysis tasks, a dependency graph between the computer-assisted data analysis tasks is constructed to identify the data flow dependencies and control flow dependencies between the computer-assisted data analysis tasks.

8. The computer-aided data analysis system based on dynamic resource scheduling according to claim 4, characterized in that: The resource dynamic scheduling module dynamically schedules the computing resources of the computer according to the data flow dependency, the control flow dependency and the resource requirements of each computer-assisted data analysis task, and is configured as follows: modifying resource requirements of each computer-assisted data analysis task based on outputs of the temporal fusion transformation model; Calculating priority reference information of each computer-assisted data analysis task based on the constructed dependency graph between the computer-assisted data analysis tasks and the corrected resource requirements of each computer-assisted data analysis task; Determine a real-time resource scheduling strategy; wherein the real-time resource scheduling strategy is used to dynamically schedule the computer's computing resources based on the priority reference information of the computer-assisted data analysis task and the revised resource requirements of each computer-assisted data analysis task.

9. The computer-aided data analysis system based on dynamic resource scheduling according to claim 8, characterized in that: The real-time resource scheduling strategy includes: Monitor resource usage in computer systems in real time; An improved greedy algorithm combined with the nearest deadline condition is adopted to calculate the remaining resources of the slices in real time and compare the slice service weights according to the revised resource requirements of each computer-assisted data analysis task. At the same time, the real-time priority queuing of the computer-assisted data analysis tasks is adjusted at the user scheduling layer.

10. The computer-aided data analysis system based on dynamic resource scheduling according to claim 4, characterized in that: The resource dynamic scheduling module can perform resource scheduling across multiple computing environments; wherein the multiple computing environments include cloud computing environments and local computing environments.

Images