Data analysis method and system based on visual modeling and job scheduling

Abstract

The present invention discloses a data analysis method and system based on visual modeling and job scheduling, which relates to the field of data analysis technology, including: designing data analysis processes and service nodes according to data analysis requirements, building data analysis models; loading data analysis models and initializing them; preprocessing and converting data to form a unified data set; building a comprehensive target optimization model, outputting a resource allocation plan; dynamically adjusting the task execution order according to the resource allocation plan and the unified data set; the scheduling engine adopts a step-by-step resource scheduling strategy for resource allocation and task execution. The present invention combines the results of the comprehensive optimization target model with real-time monitoring and prediction through an intelligent resource scheduling strategy, dynamically adjusts resource allocation and task execution order, thereby achieving efficient execution of data analysis tasks and maximizing resource utilization, effectively solving the problems of low resource utilization and low task execution efficiency in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a data analysis method and system based on visual modeling and job scheduling. Background Art

[0002] With the advent of the big data era, data analysis has become an important basis for decision-making in all walks of life. However, traditional data analysis methods usually rely on complex programming and cumbersome data processing processes, which not only increases the difficulty of data analysis, but also reduces the efficiency of data analysis. Therefore, the development of an intuitive and efficient data analysis system has important practical significance and application value. In addition, the open source job scheduling framework has certain limitations in terms of enterprise-level multi-tenant scheduling strategies, scheduling performance, system resource utilization, and support for large cluster scalability. The present invention uses a self-developed job scheduling and scheduling engine as the base of the data analysis system to greatly improve the production efficiency of data visualization analysis work. Summary of the invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by the present invention is: how to achieve efficient execution of data analysis tasks and maximize resource utilization through intelligent resource scheduling strategies, thereby solving the problems of low resource utilization and low task execution efficiency in the prior art.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a data analysis method based on visual modeling and job scheduling, which includes the following steps:

[0006] Design data analysis processes and service nodes, and build data analysis models.

[0007] The data analysis model is loaded and initialized in real time through the service orchestration engine.

[0008] The data processing nodes in the data analysis model preprocess and transform the data to form a unified data set.

[0009] Build a comprehensive target optimization model and output resource allocation plan.

[0010] Dynamically adjust the task execution order based on the resource allocation plan and unified data set.

[0011] The orchestration and scheduling engine adopts a step-by-step resource scheduling strategy for resource allocation and task execution.

[0012] As a preferred solution of the data analysis method based on visual modeling and job scheduling described in the present invention, wherein: the construction of the data analysis model is to design and manage server-side components, save the business configuration and extended components, the user sends the data preprocessing logic and task scheduling strategy to the REST API through an HTTP request, and after the REST API receives the request, the task scheduling strategy is stored in the database of the server. The user updates the stored task scheduling strategy in real time through the REST API, and the REST API overwrites the old task scheduling strategy in the database with the new task scheduling strategy. The Aviator engine extracts the latest task scheduling strategy from the database for parsing, and converts the parsed task scheduling strategy into a task scheduling strategy instruction set for the data processing node.

[0013] When the service node is initialized, the parsed task scheduling strategy instruction set is bound to the service node; and a periodic executor is created in the service node. The periodic executor periodically monitors the process configuration table to detect changes in the task scheduling strategy.

[0014] A process configuration table is automatically generated during initialization to record the operating status and dependencies of the service node. When the periodic executor detects an update to the configuration table, the service node automatically performs a hot update of the internal process. The service orchestration engine adjusts the execution order and resource allocation of the current task according to the updated task scheduling policy instructions, generates an update notification, and notifies the client about the change in process configuration. The client receives the update notification through the process configuration bus, uses the scanner to scan the computing element components of the Component interface, and sends the component information to the bus. The reporting controller takes over the component information from the bus and packages it to the server. The pull controller obtains the latest process configuration from the server and stores it in the local cache.

[0015] As a preferred solution of the data analysis method based on visual modeling and job orchestration scheduling described in the present invention, wherein: the real-time loading of the data analysis model and initialization through the service orchestration engine is after the data analysis model is built, the service orchestration engine starts to load and initialize the data analysis model in real time, the service orchestration engine extracts the task scheduling strategy stored by the user through the REST API from the server database and calls the Aviator engine to parse the extracted task scheduling strategy, and converts the parsing results into specific operation instructions of the data processing node.

[0016] The service node initializes the operating environment by obtaining the parsed instructions and uses the process configuration table to determine the dependencies and execution order between nodes. The service orchestration engine uses an asynchronous communication mechanism to coordinate the startup order of service nodes and allocate the required computing resources. The periodic executor monitors the state changes of the data processing nodes in real time and updates and reconfigures the nodes according to the process configuration table. After all nodes are ready, the service orchestration engine starts the data analysis model, so that the data processing nodes perform data processing and task scheduling according to the parsed task scheduling strategy.

[0017] The process configuration table is automatically generated when the model is constructed, and records the current status and dependency chain of each service node, so that the node can be started at the right time.

[0018] As a preferred solution of the data analysis method based on visual modeling and job scheduling described in the present invention, the preprocessing and conversion are performed after the service node is initialized, and the data processing node performs data preprocessing operations according to the rule instruction set parsed by the Aviator engine, including collecting original data from the data source node, removing null values, removing duplicate values, detecting and processing abnormal values, and unifying the data format.

[0019] After completing preprocessing, the data processing node performs data conversion to convert the preprocessed data into a data type that meets the analysis requirements, including data type conversion, aggregation, and feature engineering operations, integrating data from different sources into a unified data set.

[0020] As a preferred solution of the data analysis method based on visual modeling and job scheduling described in the present invention, wherein: the construction of the comprehensive target optimization model is to complete the dynamic adjustment of resource allocation, the sum of all CPU resources allocated to the task nodes must be greater than or equal to the CPU resources required by the task nodes, and the sum of all memory resources allocated to the task nodes must be greater than or equal to the memory resources required by the task nodes.

[0021] The total resources allocated to each computing resource node cannot exceed the available resources. The sum of all CPU resources allocated to task nodes must be less than or equal to the available CPU resources of the resource node. The sum of all memory resources allocated to task nodes must be less than or equal to the available memory resources of the resource node.

[0022] Based on the current load of the resource node plus the sum of all the resources allocated to the current task, an expected target value is output. The expression is:

[0023]

[0024] Among them, l j Resource node C jThe current load, Resource node C j Assigned to task node T i The amount of CPU resources, Resource node C j Assigned to task node T i Memory resources, CPU j MEM j They are task nodes T i The amount of CPU resources and memory resources required.

[0025] Compress the task execution time to the shortest possible time, use the resource utilization efficiency in historical data, and optimize resource allocation. The expression is:

[0026]

[0027] Among them, t i is the task node T i The execution time, h j Resource node C j Health status, e ij Resource C in historical data j In the execution of task T i The execution efficiency, p i is the task node T i The priority weight of .

[0028] Construct a comprehensive optimization objective function, the expression is:

[0029]

[0030] Among them, λ1 and λ2 are the weight of task execution time and load balancing weight respectively, B is the expected load balancing value, n and m are the total number of task nodes and the total number of resource types respectively, and i is the index of the task node, from 1 to n.

[0031] As a preferred solution of the data analysis method based on visual modeling and job scheduling described in the present invention, the dynamic adjustment of the task execution order is to load a unified data set, obtain the data required by the task node, and use it as the input of the comprehensive target optimization model to output a resource allocation plan.

[0032] Based on the resource allocation plan output by the comprehensive target optimization model, specific computing resources are allocated to each task node.

[0033] A visual directed acyclic graph is constructed through the resource allocation plan. The directed acyclic graph is used to represent the dependencies between tasks, where each node represents a task and the edges represent the dependencies between tasks. The topological structure is automatically generated based on the prerequisites and dependencies of the task nodes. All task nodes are sorted according to their positions in the directed acyclic graph, and the subsequent tasks are started only after the predecessor tasks are completed.

[0034] The computing resources are CPU resources and memory resources.

[0035] As a preferred solution of the data analysis method based on visual modeling and job scheduling and scheduling described in the present invention, the resource allocation and task execution are based on the step-by-step allocation of computing resources and the execution of tasks in the order of task priority, the initial computing resources are allocated to each task according to the resource allocation scheme, and the resource usage of the current task at time t is monitored in real time. Each task node T i There is a corresponding initial resource allocation A ik , record resource usage dynamics and compare them with the resource allocation plan. If the task is completed ahead of schedule or delayed, the released or required resources will be based on the resource allocation plan. The expression is:

[0036] r ik =A ik

[0037]

[0038] r ik (t+1)=r ik (t)+Δr ik

[0039] Among them, r ik Assigned to task T i The amount of resources of resource type k, A ik is the resource allocation scheme assigned to the task node T i The initial resource quantity of resource type k, U k (t) The total usage of resource type k by all tasks at time t, u ik (t) is the task node T i The usage of resource type k at time t, Δr ik To monitor the task node T according to the real-time monitoring feedback i The resource adjustment amount of resource type k;

[0040] According to the task dependencies and resource allocation scheme in the DAG structure, the task nodes T are gradually scheduled. i Execute and build the task scheduling sequence. The expression is:

[0041]

[0042] Among them, p i is the task node T i The priority of t i is the task node T i Execution time.

[0043] The task execution process is divided into multiple stages. At the end of each stage, the task execution plan and resource allocation strategy for the next stage are adjusted based on the results of task execution and resource usage.

[0044] Collect task nodes T i The resource usage and task execution time during the execution process are used to construct the feedback optimization objective function, which is expressed as:

[0045]

[0046] Among them, t i is the task node T i The execution time of k is the usage weight of resource type k.

[0047] Another object of the present invention is to provide a data analysis system based on visual modeling and job scheduling, which can achieve efficient execution of data analysis tasks and maximize resource utilization through intelligent resource scheduling strategies, solving the problems of low resource utilization and low task execution efficiency in the prior art.

[0048] To solve the above technical problems, the present invention provides the following technical solutions: a data analysis system based on visual modeling and job orchestration scheduling, including: a data analysis modeling module, a service orchestration engine module, an intelligent resource scheduling module, a real-time monitoring and feedback module, and a feedback optimization module.

[0049] The data analysis modeling module designs data analysis processes and service nodes and constructs data analysis models according to data analysis requirements.

[0050] The service orchestration engine module loads the data analysis model in real time and performs initialization, and performs resource allocation according to the resource allocation scheme output by the comprehensive optimization target model.

[0051] The intelligent resource scheduling module adopts intelligent resource scheduling strategies to perform resource allocation and task execution, and dynamically adjusts resources and scheduling strategies through prediction and real-time monitoring.

[0052] The real-time monitoring and feedback module monitors the task execution and resource usage in real time, and provides feedback data to optimize resource scheduling strategies.

[0053] The feedback optimization module forms a closed-loop optimization by continuously collecting task execution data, updating the machine learning model and the comprehensive optimization target model.

[0054] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the data analysis method based on visual modeling and job scheduling are implemented.

[0055] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data analysis method based on visual modeling and job scheduling as described above.

[0056] Beneficial effects of the present invention: The present invention combines the results of the comprehensive optimization target model with real-time monitoring and prediction through an intelligent resource scheduling strategy, dynamically adjusts resource allocation and task execution order, thereby achieving efficient execution of data analysis tasks and maximizing resource utilization, effectively solving the problems of low resource utilization and low task execution efficiency in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0058] Figure 1 An overall flow chart of a data analysis method based on visual modeling and job scheduling provided for the first embodiment of the present invention.

[0059] Figure 2 A structural diagram of a data analysis system based on visual modeling and job scheduling provided for the second embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0061] Example 1, reference Figure 1 As an embodiment of the present invention, a data analysis method based on visual modeling and job scheduling is provided, including:

[0062] Design data analysis processes and service nodes, and build data analysis models.

[0063] The data analysis model is loaded and initialized in real time through the service orchestration engine.

[0064] The data processing nodes in the data analysis model preprocess and transform the data to form a unified data set.

[0065] Build a comprehensive target optimization model and output resource allocation plan.

[0066] Dynamically adjust the task execution order based on the resource allocation plan and unified data set.

[0067] The orchestration and scheduling engine adopts a step-by-step resource scheduling strategy for resource allocation and task execution.

[0068] Design and manage server-side components, save business configurations and extended components, users send data preprocessing logic and task scheduling strategies to REST API through HTTP requests, after receiving the request, REST API stores the task scheduling strategies in the server's database, users update the stored task scheduling strategies in real time through REST API, REST API overwrites the old task scheduling strategies in the database with the new task scheduling strategies, Aviator engine extracts the latest task scheduling strategies from the database for parsing, and converts the parsed task scheduling strategies into task scheduling strategy instruction sets for data processing nodes.

[0069] When the service node is initialized, the parsed task scheduling strategy instruction set is bound to the service node; and a periodic executor is created in the service node. The periodic executor periodically monitors the process configuration table to detect changes in the task scheduling strategy.

[0070] A process configuration table is automatically generated during initialization to record the operating status and dependencies of the service node. When the periodic executor detects an update to the configuration table, the service node automatically performs a hot update of the internal process. The service orchestration engine adjusts the execution order and resource allocation of the current task according to the updated task scheduling policy instructions, generates an update notification, and notifies the client about the change in process configuration. The client receives the update notification through the process configuration bus, uses the scanner to scan the computing element components of the Component interface, and sends the component information to the bus. The reporting controller takes over the component information from the bus and packages it to the server. The pull controller obtains the latest process configuration from the server and stores it in the local cache.

[0071] After the data analysis model is built, the service orchestration engine starts to load and initialize the data analysis model in real time. The service orchestration engine extracts the task scheduling strategy stored by the user through the REST API from the server database and calls the Aviator engine to parse the extracted task scheduling strategy, and converts the parsing results into specific operation instructions for the data processing node.

[0072] The service node initializes the operating environment by obtaining the parsed instructions and uses the process configuration table to determine the dependencies and execution order between nodes. The service orchestration engine uses an asynchronous communication mechanism to coordinate the startup order of service nodes and allocate the required computing resources. The periodic executor monitors the state changes of the data processing nodes in real time and updates and reconfigures the nodes according to the process configuration table. After all nodes are ready, the service orchestration engine starts the data analysis model, so that the data processing nodes perform data processing and task scheduling according to the parsed task scheduling strategy.

[0073] The process configuration table is automatically generated when the model is built, recording the current status and dependency chain of each service node, so that the node can be started at the right time.

[0074] After the service node is initialized, the data processing node performs data preprocessing operations according to the rule instruction set parsed by the Aviator engine, including collecting raw data from the data source node, removing null values, removing duplicate values, detecting and processing outliers, and unifying the data format.

[0075] After completing preprocessing, the data processing node performs data conversion, converting the preprocessed data into a data type that meets the analysis requirements, including data type conversion, aggregation, and feature engineering operations, integrating data from different sources into a unified data set.

[0076] The construction of a comprehensive target optimization model is to complete the dynamic adjustment of resource allocation. The sum of all CPU resources allocated to the task node must be greater than or equal to the CPU resources required by the task node, and the sum of all memory resources allocated to the task node must be greater than or equal to the memory resources required by the task node.

[0077] The total resources allocated to each computing resource node cannot exceed the available resources. The sum of all CPU resources allocated to task nodes must be less than or equal to the available CPU resources of the resource node. The sum of all memory resources allocated to task nodes must be less than or equal to the available memory resources of the resource node.

[0078] For each task node T i , whose resource requirements must be met:

[0079]

[0080] For each computing resource C j , the total resources allocated to it cannot exceed its available resources:

[0081]

[0082] In order to ensure the load balance of the system, the tasks assigned to each resource node should be as balanced as possible. According to the current load of the resource node plus the sum of all the resources assigned to the current task, an expected target value is output. The expression is:

[0083]

[0084] Among them, l j Resource node C j The current load, Resource node C j Assigned to task node T i The amount of CPU resources, Resource node C j Assigned to task node T i Memory resources, CPU j MEM j They are task nodes T i The amount of CPU resources and memory resources required.

[0085] Compress the task execution time to the shortest possible time, use the resource utilization efficiency in historical data, and optimize resource allocation. The expression is:

[0086]

[0087] Among them, t i is the task node T i The execution time, h j Resource node C j Health status, e ij Resource C in historical data j In the execution of task T i The execution efficiency, p i is the task node T i The priority weight of .

[0088] Construct a comprehensive optimization objective function, the expression is:

[0089]

[0090] Among them, λ1 and λ2 are the weight of task execution time and load balancing weight respectively, B is the expected load balancing value, n and m are the total number of task nodes and the total number of resource types respectively, and i is the index of the task node, from 1 to n.

[0091] It should be further explained that by optimizing the solution process of the objective function, a result including a specific resource allocation scheme is finally obtained, rather than a single accurate value. These allocation schemes specifically specify the amount of resources that should be allocated to each task node to achieve the optimization of the overall system performance. In the present invention, these allocation schemes are actually applied in the data processing and task scheduling process to ensure the efficient use of system resources and the optimization of task execution.

[0092] Advanced optimization methods such as Mixed-Integer Linear Programming (MILP) can be used to solve the above comprehensive optimization problem.

[0093] By introducing influencing parameters such as task execution time, node load balancing, and historical data, the problem of dynamically adjusting computing resource allocation is transformed into a more complex and precise mathematical optimization problem. By solving this optimization problem, the service orchestration engine can achieve more efficient and precise resource allocation, thereby optimizing system resource utilization and task execution efficiency.

[0094] It is further explained that:

[0095]

[0096] The purpose of this item is to make the load of each resource node close to a certain balance value. j Current resource node C j Indicates the current load of the node.

[0097] The additional load after resource allocation is allocated to node C j The load caused by the task resource requirements.

[0098] The balance value is the expected load balance value. It can be the average load of the overall system resources, or other defined balance criteria.

[0099] During the optimization process, we hope that the load of each resource node is close to the balanced value. This means that the closer the load is to the balanced value, the more balanced the load of the entire system is, thus avoiding overloading or idleness of a single node. To achieve this goal, the degree to which the load deviates from the balanced value is expressed in square form, with the aim of allowing the optimization process to more strictly reduce the imbalance of the load.

[0100] Represents node Cj The square term is used to emphasize the impact of large deviations, so that the optimization process pays more attention to nodes with severely unbalanced loads.

[0101] The comprehensive optimization objective function combines the requirements of resource utilization, task execution time and load balancing, making resource allocation more intelligent and efficient.

[0102] The introduction of load balancing items is to ensure that the load of each resource node is as close to the balanced value as possible. The use of square items can make the optimization process more strictly reduce the impact of load imbalance. In this way, the optimized resource allocation can not only guarantee the resource requirements of high-priority tasks, but also effectively utilize system resources, avoid resource overload and idleness, and improve the overall performance and efficiency of the system.

[0103] Construct different optimization objective functions according to different special cases:

[0104] Optimization objective function under special requirements.

[0105] In the case of sudden load increase, the system needs to prioritize load balancing and response speed. The reciprocal of the task execution time is used as the weight to prioritize tasks with shorter execution time:

[0106] Prioritize load balancing and response speed by i The reciprocal of is used as the weight to optimize resource response speed.

[0107] The dynamic balance value takes into account the current system load and historical data.

[0108]

[0109] Where C is the dynamically expected load balancing value.

[0110] By using the inverse of the task execution time as the weight, tasks with shorter execution time can be processed first, responding quickly to sudden loads and improving the processing efficiency of the system.

[0111] In the case of sudden load increases, prioritizing the completion of short tasks can free up resources more quickly to process more tasks.

[0112] In the event of a resource node failure, the system needs to prioritize task reallocation to avoid task interruption and adjust the health status of resource nodes and task execution time.

[0113] Prioritize reallocation of tasks to avoid task interruption by adjusting health status and adjust task execution time To optimize resource allocation.

[0114]

[0115] Adjusted health status Reflects the impact of the fault on the health status of the node.

[0116] By reducing the health weight of the faulty node, the resource allocation algorithm will prioritize allocating resources to nodes with better health status. When the health status of the faulty node decreases, the system will reduce the resource allocation to it, thereby reducing the risk of task interruption.

[0117] Adjusted task execution time Reflects task delays due to node failures.

[0118] Recalculate execution time to take into account the impact of failures, thereby optimizing the timeliness of task reallocation. Accurately estimating the impact of failures on task execution time helps to properly arrange task priorities and resource allocation.

[0119] By setting different optimization objective functions for different special needs, we can better deal with various special situations that occur in the system. The adjusted characters help optimize resource allocation, improve task execution efficiency, and ensure system stability in the current corresponding state. These adjustments are based on actual needs and system status, providing reasonable supporting reasons and expected effects.

[0120] Dynamically adjusting the task execution order is to load a unified data set, obtain the data required by the task nodes, and use it as the input of the comprehensive target optimization model to output the resource allocation plan.

[0121] Based on the resource allocation plan output by the comprehensive target optimization model, computing resources are allocated to each task node.

[0122] A visual directed acyclic graph is constructed through the resource allocation plan. The directed acyclic graph is used to represent the dependencies between tasks, where each node represents a task and the edges represent the dependencies between tasks. The topological structure is automatically generated based on the prerequisites and dependencies of the task nodes. All task nodes are sorted according to their positions in the directed acyclic graph, and the subsequent tasks are started only after the predecessor tasks are completed.

[0123] Computing resources refer to the amount of CPU resources and memory resources.

[0124] According to the step-by-step allocation of computing resources and the execution of tasks in order of task priority, initial computing resources are allocated to each task according to the resource allocation plan, and the resource usage of the current task at time t is monitored in real time. Each task node T i There is a corresponding initial resource allocation A ik, record resource usage dynamics and compare them with the resource allocation plan. If the task is completed ahead of schedule or delayed, the released or required resources will be based on the resource allocation plan. The expression is:

[0125] r ik =A ik

[0126]

[0127] r ik (t+1)=r ik (t)+Δr ik

[0128] Among them, r ik Assigned to task T i The amount of resources of resource type k, A ik is the resource allocation scheme assigned to the task node T i The initial resource quantity of resource type k, U k (t) The total usage of resource type k by all tasks at time t, u ik (t) is the task node T i The usage of resource type k at time t, Δr ik To monitor the task node T according to the real-time monitoring feedback i The resource adjustment amount for resource type k.

[0129] It is further explained that:

[0130] r ik =A ik This means that the amount of resources actually allocated to the task is equal to the initial allocation calculated in the optimization model, which is usually the static resource allocated when the system starts executing. Subsequent steps include dynamic adjustment, which means that the amount of resources r ij Adjustments will be made based on monitoring feedback.

[0131] As the task execution progresses, the resource usage of the task nodes is monitored in real time. ik (t), and the planned allocation A ik For comparison, dynamic resource adjustments are required.

[0132] If a task node is completed ahead of schedule or resource requirements change, the system will calculate the resource adjustment amount Δr based on real-time monitoring feedback ik , and update the resource allocation of the task node. The adjusted resource allocation formula is:

[0133] r ik (t+1)=r ik (t)+Δr ik.

[0134] According to the task dependencies and resource allocation scheme in the DAG structure, the task nodes T are gradually scheduled. i Execute and build the task scheduling sequence. The expression is:

[0135]

[0136] Among them, p i is the task node T i The priority of t i is the task node T i Execution time;

[0137] The task execution process is divided into multiple stages. At the end of each stage, the task execution plan and resource allocation strategy for the next stage are adjusted according to the results of task execution and resource usage.

[0138] Collect task nodes T i The resource usage and task execution time during the execution process are used to construct the feedback optimization objective function, which is expressed as:

[0139]

[0140] Among them, t i is the task node T i The execution time of k is the usage weight of resource type k.

[0141] Example 2, reference Figure 2 , which is an embodiment of the present invention, provides a system for a data analysis method based on visual modeling and job scheduling and scheduling, including: a data analysis and modeling module, a service scheduling engine module, an intelligent resource scheduling module, a real-time monitoring and feedback module, and a feedback optimization module;

[0142] The data analysis modeling module designs data analysis processes and service nodes and builds data analysis models according to data analysis requirements;

[0143] The service orchestration engine module loads the data analysis model in real time and initializes it, and allocates resources according to the resource allocation plan output by the comprehensive optimization target model;

[0144] The intelligent resource scheduling module uses intelligent resource scheduling strategies to allocate resources and execute tasks, and dynamically adjusts resources and scheduling strategies through prediction and real-time monitoring;

[0145] The real-time monitoring and feedback module monitors task execution and resource usage in real time, and provides feedback data to optimize resource scheduling strategies;

[0146] The feedback optimization module forms a closed-loop optimization by continuously collecting task execution data, updating the machine learning model and the comprehensive optimization target model.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0148] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0149] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0150] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0151] Embodiment 3: In this embodiment, in order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments. This embodiment experiments the existing traditional method and the method of this embodiment respectively.

[0152] The simulation experiment of the present invention is designed to be carried out on a data analysis platform, which needs to process multiple data analysis tasks, each of which requires different computing resources (such as CPU resources and memory resources). The experiment compares the intelligent resource scheduling strategy of the present invention with the traditional static resource allocation method to evaluate its performance in terms of resource utilization, task execution time and system load balancing.

[0153] Traditional methods use a static resource allocation strategy, that is, before the task is executed, a fixed amount of resources is allocated to each task based on the estimated resource requirements. No dynamic adjustment is performed during the task execution, and even if the resources are released after the task is completed, they will not be immediately reallocated to other tasks.

[0154] Table 1 Data comparison table

[0155]

[0156]

[0157] It can be seen from the experimental data that the intelligent resource scheduling strategy of the present invention is significantly superior to the traditional static resource allocation strategy in terms of resource utilization, task execution time and system load balance.

[0158] The resource utilization rate of the method of the present invention under different task numbers is higher than that of the traditional method. When the number of tasks is 10, the resource utilization rate is increased by about 20 percentage points, from 65% to 85%; when the number of tasks is 30, the resource utilization rate is also increased by about 20 percentage points, from 55% to 75%.

[0159] Using the method of the present invention, the average task execution time is significantly reduced. When the number of tasks is 20, the average task execution time is reduced from 200 seconds of the traditional method to 160 seconds, which is shortened by about 20%; when the number of tasks is 30, the average task execution time is reduced from 300 seconds to 240 seconds, which is shortened by about 20%.

[0160] The system load balance is measured by the load standard deviation of each resource node. The smaller the value, the more balanced the load. The system load balance of the method of the present invention is significantly better than the traditional method under different task numbers. For example, when the number of tasks is 10, the load standard deviation is reduced from 15 in the traditional method to 5; when the number of tasks is 30, the load standard deviation is reduced from 35 to 10.

[0161] In summary, the intelligent resource scheduling strategy of the present invention achieves maximum resource utilization, minimization of task execution time and balanced distribution of system load by dynamically adjusting resource allocation and task execution order, effectively solving the problems of low resource utilization and low task execution efficiency in the prior art, and demonstrating the significant beneficial effects of the present invention.

[0162] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data analysis method based on visual modeling and job scheduling, characterized in that: include: Design data analysis processes and service nodes, and build data analysis models; Load the data analysis model in real time and initialize it through the service orchestration engine; The data processing nodes in the data analysis model preprocess and transform the data to form a unified data set; Build a comprehensive target optimization model and output resource allocation plan; Dynamically adjust the order of task execution based on resource allocation plans and unified data sets; The scheduling engine uses a step-by-step resource scheduling strategy to allocate resources and execute tasks; The construction of the data analysis model is to design and manage the server-side components, save the business configuration and extended components, the user sends the data preprocessing logic and task scheduling strategy to the REST API through HTTP request, after the REST API receives the request, the task scheduling strategy is stored in the database of the server, the user updates the stored task scheduling strategy in real time through the REST API, the REST API overwrites the old task scheduling strategy in the database with the new task scheduling strategy, the Aviator engine extracts the latest task scheduling strategy from the database for parsing, and converts the parsed task scheduling strategy into the task scheduling strategy instruction set of the data processing node; When initializing the service node, the parsed task scheduling strategy instruction set is bound to the service node; and a periodic executor is created in the service node. The periodic executor periodically monitors the process configuration table to detect changes in the task scheduling strategy; A process configuration table is automatically generated during initialization to record the operating status and dependencies of the service node. When the periodic executor detects an update to the configuration table, the service node automatically performs a hot update of the internal process. The service orchestration engine adjusts the execution order and resource allocation of the current task according to the updated task scheduling policy instructions, generates an update notification, and notifies the client about the change in process configuration. The client receives the update notification through the process configuration bus, uses the scanner to scan the computing element components of the Component interface, and sends the component information to the bus. The reporting controller takes over the component information from the bus and packages it to the server. The pull controller obtains the latest process configuration from the server and stores it in the local cache.

2. The data analysis method based on visual modeling and job scheduling as claimed in claim 1, characterized in that: The real-time loading and initialization of the data analysis model by the service orchestration engine is that after the data analysis model is built, the service orchestration engine starts to load and initialize the data analysis model in real time, extracts the task scheduling strategy stored by the user through the REST API from the server database and calls the Aviator engine to parse the extracted task scheduling strategy, and converts the parsing result into specific operation instructions of the data processing node; The service node initializes the operating environment by obtaining the parsed instructions and uses the process configuration table to determine the dependencies and execution order between the nodes. The service orchestration engine uses an asynchronous communication mechanism to coordinate the startup order of the service nodes and allocate the required computing resources. The periodic executor monitors the state changes of the data processing nodes in real time and updates and reconfigures the nodes according to the process configuration table. After all nodes are ready, the service orchestration engine starts the data analysis model, so that the data processing nodes perform data processing and task scheduling according to the parsed task scheduling strategy. The process configuration table is automatically generated when the model is constructed, and records the current status and dependency chain of each service node, so that the node can be started at the right time.

3. The data analysis method based on visual modeling and job scheduling as claimed in claim 2, characterized in that: The preprocessing and conversion is that after the service node is initialized, the data processing node performs data preprocessing operations according to the rule instruction set parsed by the Aviator engine, including collecting raw data from the data source node, removing null values, removing duplicate values, detecting and processing abnormal values, and unifying the data format; After completing preprocessing, the data processing node performs data conversion to convert the preprocessed data into a data type that meets the analysis requirements, including data type conversion, aggregation, and feature engineering operations, integrating data from different sources to form a unified data set.

4. The data analysis method based on visual modeling and job scheduling as claimed in claim 3, characterized in that: The construction of the comprehensive target optimization model is to determine the task scheduling strategy and output the dynamic adjustment of resource allocation. The sum of all CPU resources allocated to the task node must be greater than or equal to the CPU resources required by the task node, and the sum of all memory resources allocated to the task node must be greater than or equal to the memory resources required by the task node; The total resources allocated to each computing resource node cannot exceed the available resources. The sum of all CPU resources allocated to task nodes must be less than or equal to the available CPU resources of the resource node. The sum of all memory resources allocated to task nodes must be less than or equal to the available memory resources of the resource node. Based on the current load of the resource node plus the sum of all the resources allocated to the current task, an expected target value is output. The expression is: Among them, l j Resource node C j The current load, Resource node C j Assigned to task node T i The amount of CPU resources, Resource node C j Assigned to task node T i The amount of memory resources, CPU j MEM j They are task nodes T i The amount of CPU and memory resources required; Compress the task execution time to the shortest possible time, use the resource utilization efficiency in historical data, and optimize resource allocation. The expression is: Among them, t i is the task node T i The execution time, h j Resource node C j Health status, e ij Resource C in historical data j In the execution of task T i The execution efficiency, p i is the task node T i The priority weight of Construct a comprehensive optimization objective function, the expression is: Among them, λ1 and λ2 are the weight of task execution time and load balancing weight respectively, B is the expected load balancing value, n and m are the total number of task nodes and the total number of resource types respectively, and i is the index of the task node, from 1 to n.

5. The data analysis method based on visual modeling and job scheduling as claimed in claim 4, characterized in that: The dynamic adjustment of the task execution order is to load a unified data set, obtain the data required by the task node, and use it as the input of the comprehensive target optimization model to output a resource allocation plan; Allocate computing resources to each task node based on the resource allocation plan output by the comprehensive target optimization model; A visual directed acyclic graph is constructed through the resource allocation scheme. The directed acyclic graph is used to represent the dependency relationship between tasks. Each node represents a task, and the edge represents the dependency between tasks. The topology structure is automatically generated based on the preconditions and dependency relationships of the task nodes. All task nodes are sorted according to their positions in the directed acyclic graph. Subsequent tasks are started only after the predecessor tasks are completed. The computing resources are CPU resources and memory resources.

6. The data analysis method based on visual modeling and job scheduling as claimed in claim 5, characterized in that: The resource allocation and task execution are based on the step-by-step allocation of computing resources and the execution of tasks in order of task priority. Initial computing resources are allocated to each task according to the resource allocation plan, and the resource usage of the current task at time t is monitored in real time. Each task node T i There is a corresponding initial resource allocation A ik , record resource usage dynamics and compare them with the resource allocation plan. If the task is completed ahead of schedule or delayed, the released or required resources will be based on the resource allocation plan. The expression is: r ik ＝A ik r ik (t+1)=r ik (t)+Δr ik Among them, r ik Assigned to task T i The amount of resources of resource type k, A ik is the resource allocation scheme assigned to the task node T i The initial resource quantity of resource type k, U k (t) The total usage of resource type k by all tasks at time t, u ik (t) is the task node T i The usage of resource type k at time t, Δr ik To monitor the task node T according to the real-time monitoring feedback i The resource adjustment amount of resource type k; According to the task dependency and resource allocation scheme in the directed acyclic graph, the task nodes T are gradually scheduled. i Execute and build the task scheduling sequence. The expression is: Among them, p i is the task node T i The priority weight, t i is the task node T i Execution time; The task execution process is divided into multiple stages. At the end of each stage, the task execution plan and resource allocation strategy for the next stage are adjusted according to the results of task execution and resource usage. Collect task nodes T i The resource usage and task execution time during the execution process are used to construct the feedback optimization objective function, which is expressed as: Among them, t i is the task node T i The execution time of k is the usage weight of resource type k.

7. A system using the data analysis method based on visual modeling and job scheduling as claimed in any one of claims 1 to 6, characterized in that: It includes data analysis and modeling module, service orchestration engine module, intelligent resource scheduling module, real-time monitoring and feedback module, and feedback optimization module; The data analysis modeling module designs data analysis processes and service nodes and builds data analysis models according to data analysis requirements; The service orchestration engine module loads the data analysis model in real time and performs initialization, and performs resource allocation according to the resource allocation scheme output by the comprehensive optimization target model; The intelligent resource scheduling module uses intelligent resource scheduling strategies to perform resource allocation and task execution, and dynamically adjusts resources and scheduling strategies through prediction and real-time monitoring; The real-time monitoring and feedback module monitors the task execution and resource usage in real time, and provides feedback data to optimize resource scheduling strategies; The feedback optimization module forms a closed-loop optimization by continuously collecting task execution data, updating the machine learning model and the comprehensive optimization target model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the data analysis method based on visual modeling and job scheduling and scheduling are implemented as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data analysis method based on visual modeling and job scheduling and scheduling as described in any one of claims 1 to 6 are implemented.

Images