Log analysis task scheduling system based on neural network

Abstract

The invention relates to the technical field of task scheduling, in particular to a log analysis task scheduling system based on a neural network, which can realize accurate matching of tasks and resources and optimize task scheduling efficiency by extracting task resource requirements and task execution duration from a database and mapping the task resource requirements and the task execution duration to a task resource dependency graph. According to the method, the task execution sequence is optimized, the dependency relationship between the tasks is established, optimization of the task execution sequence is more flexible and real-time, the priority and the time window of the tasks are accurately analyzed through the long-short-term memory neural network, the task execution time can be efficiently estimated, the task execution sequence is optimized, and the task execution efficiency is improved. The task concurrent execution sequence is reasonably arranged under the condition of limited resources, generation of a resource allocation scheme is more intelligent through application of the graph neural network technology in resource allocation, whether current system resources are enough to support task execution or not can be judged in real time, the task execution sequence is adjusted in real time, and the resource allocation efficiency is improved. And the task is prevented from being blocked or delayed due to resource bottleneck.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and particularly to a log analysis task scheduling system based on a neural network. Background Art

[0002] The goal of the technical field of task scheduling is to optimize resource utilization, reduce task execution time, and improve the overall system performance, effectively allocate multiple tasks and workloads to different processing units, and enable tasks to be executed efficiently under limited resources.

[0003] A log analysis task scheduling system based on a neural network aims to automatically analyze log data through deep learning technology, identify key patterns and potential problems, and automatically schedule relevant tasks according to the analysis results to achieve more efficient log management and analysis. It aims to improve the processing efficiency of log analysis tasks and optimize resource allocation through intelligent scheduling, and enhance the stability and reliability of system operation.

[0004] The prior art lacks in-depth analysis of task dependencies. The execution order of tasks is often fixed and cannot be flexibly adjusted according to the actual dependencies and priorities between tasks, resulting in task delays and resource conflicts easily occurring under high load conditions. It cannot effectively handle complex dependencies between tasks and cannot make real-time feedback according to the actual requirements and resource status during task execution, leading to low resource allocation efficiency and long task execution time, affecting the overall system performance and stability. Summary of the Invention

[0005] The purpose of the present invention is to solve the deficiencies existing in the prior art and propose a log analysis task scheduling system based on a neural network.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions: A log analysis task scheduling system based on a neural network includes: Graph construction module: Extract task metadata through the database, obtain the execution duration and required resources of each task, map task and resource nodes, establish the dependencies between tasks as edges, determine whether additional edges need to be added according to the dependency strength, and use task priority and resource quota information as additional attributes of the nodes to construct a task-resource dependency graph; Dependency resolution module: Based on the task-resource dependency graph, obtain the mutual dependencies of all task nodes, calculate the dependency matrix between nodes, judge the dependency relationship between each pair of nodes, sort the nodes according to the dependency strength, adjust the node priority when the dependency strength exceeds the threshold, and judge the executable status of the tasks to generate a task execution dependency list; Sequential prediction module: Based on the task execution dependency list, using a long short-term memory neural network, analyze the priority and time window sequence of tasks, estimate the execution time of tasks, determine the appropriate task execution order, sort the task execution order according to the priority, perform timing scheduling based on the execution cycle and priority, and combine the delay determination and resource requirements between tasks to generate a task execution order table; Resource allocation module: Based on the task execution order table, using a graph neural network, sequentially extract the resource requirements of each task, compare the current resource status, determine whether there are sufficient resources to meet the concurrent execution of tasks, and adjust the resource allocation according to the task priority. If the current resources are insufficient, suspend the execution of some tasks according to the priority, divide the resources into multiple priorities, and generate an optimized resource allocation plan; Execution control module: Based on the optimized resource allocation plan, determine whether each resource meets the execution requirements of the task. If the resource is available, start the task execution and record the execution status. If the resource is insufficient, suspend the task and reschedule it, continuously monitor the status changes of each task, update the task execution mark, and generate a task execution status report.

[0007] As a further solution of the present invention, the graph construction module includes: Task metadata extraction sub-module: Extract task metadata through the database, query the database at the same time, obtain the relevant information of all tasks, and store it in the data table to generate a task metadata table; Node mapping sub-module: Based on the task metadata table, map tasks and required resources to nodes in the graph. Each task corresponds to a task node, and each resource corresponds to a resource node. By connecting the task node and the resource node, generate a task-resource node mapping table; Dependency relationship construction sub-module: Based on the task-resource node mapping table, obtain the dependency relationship between tasks. Through the dependency relationship identifier between tasks, establish a directed edge between tasks. Determine whether a new edge needs to be added according to the dependency strength. If the dependency strength is higher than the predetermined threshold, add an edge connection to generate a task-resource dependency graph.

[0008] As a further solution of the present invention, the dependency parsing module includes: Relationship extraction sub-module: Based on the task-resource dependency graph, perform a node traversal operation. By reading the start node and end node identifiers of each edge in the graph and comparing the task numbers, obtain the corresponding relationship between task nodes, and extract the edge connection weight as the dependency strength value to construct a dependency relationship weight matrix and generate a node dependency weight set; Priority sorting sub-module: Based on the node dependency weight set, perform numerical comparison operations on the dependency strength. By summing the weight values in the out-edge set of each node and sorting them by size, extract strongly dependent nodes and establish corresponding priority order tags, adjust the priority arrangement in task scheduling, and generate a task priority order mapping table; Execution verification sub-module: Based on the task priority order mapping table, judge the executable status of each task node. By comparing the hierarchical position of the task in the topological structure and verifying whether all prerequisite nodes have uncompleted dependencies, mark the executable and pending execution statuses, summarize the sorted nodes and status information, and generate a task execution dependency list.

[0009] As a further solution of the present invention, the sequence prediction module includes: Time structure analysis sub-module: Based on the task execution dependency list, use a long short-term memory neural network to extract the priority, dependent node number, and time window boundary corresponding to each task node, perform time interval comparison operations, filter overlapping interval tasks and calculate the priority difference, establish the time interaction structure of each task, and generate a time interaction matrix set; Execution duration evaluation sub-module: Based on the time interaction matrix set, solve the difference between the start time and end time of each group of tasks, and combine the task resource usage density value to judge the actual duration of the task. Update and redistribute the task execution duration in intervals, and generate a task interval duration schedule; Scheduling order generation sub-module: Based on the task interval duration schedule, jointly sort the priorities, time spans, and required resources of all tasks. Construct a scheduling start point identification table through time continuity and resource distribution sparsity, determine the execution order of tasks, establish an executable process chain, and generate a task execution order table.

[0010] As a further solution of the present invention, the long short-term memory neural network, according to the formula:

[0011] Where: represents task and task the priority gap between them, represents the end time of task represents the end time of task represents the end time of task represents the end time of task represents the priority of task represents task​​​​​ The priority of represents the weight coefficient of the overlapping part of the task time for calculating the priority difference, represents the weight coefficient of the impact of the task priority difference on the calculation result, represents the task and the task The weight coefficient of the task dependency between them on the calculation result, represents the weight coefficient of the difference in task start time for calculating the priority difference, represents the task and the task The task dependency between them.

[0012] As a further solution of the present invention, the resource allocation module includes: Resource requirement extraction sub-module: Based on the task execution order list, using a graph neural network, perform task index extraction and locate the task number, and by reading the resource type identifier and resource quantity value of the corresponding task fields, construct a combination of numbers and values, match the relationship between the task and the required resources, and register the matching result in a unified record table to generate a task resource requirement list; Allocation condition judgment sub-module: Based on the task resource requirement list, read the system resource status snapshot and establish a resource remaining quantity mapping table, calculate the numerical difference between the resources required for the task and the remaining resources in sequence, mark whether the current resources meet the task execution conditions, and append a queuing identifier to the insufficient resource items to generate a resource allocation feasibility mark set; Resource priority division sub-module: Based on the resource allocation feasibility mark set, extract all schedulable task numbers and corresponding priority values, perform priority interval division and construct a resource level queue, map the remaining system resources to each task group in proportion and register the mapping relationship between the task and the resources to generate a resource allocation optimization plan.

[0013] As a further solution of the present invention, the graph neural network, according to the formula:

[0014] Among them: represents the total resource requirement of the task calculated by the graph neural network, is the unit resource consumption value of the resource type is the task for the resource type demand quantity, is the quantity of all resource types, is the resource type for the task weight coefficient of resource requirement,​ For the task and the resource type the dependency between them is the adjustment weight coefficient of the time resource consumption for the total resource demand is the adjustment weight coefficient of the computing resource consumption for the total resource demand For the task the time resource demand For the task the computing resource demand

[0015] As a further solution of the present invention, the execution control module includes: Task trigger judgment sub-module: Based on the resource allocation optimization plan, conduct a joint check on the task and resource locking status, perform an availability judgment operation on the status field of the resources bound to the task, mark the tasks with the resources in the non-occupied state as the to-be-started state and establish a numbered list, and generate a task activation identification set; Status record execution sub-module: Based on the task activation identification set, traverse all the task numbers to be executed, obtain the execution start time and resource usage status, record the task execution time period and write it into the task status field, set the resources as the occupied state, construct the mapping relationship between the number and time, and generate a task execution status record table; Execution result update sub-module: Based on the task execution status record table, poll to obtain the current execution status value of the task, release the occupied resources and update the execution status code and task completion time by determining whether the task completion flag and the execution duration upper limit meet the termination conditions, complete the status write operation, and generate a task execution status report.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: 1. In the present invention, by extracting the task resource requirements and task execution duration from the database and mapping them to the task resource dependency graph, it is possible to achieve the precise matching of tasks and resources, optimize the task scheduling efficiency, establish the dependency relationship between tasks, and dynamically adjust the task priority based on the dependency strength, making the optimization of the task execution order more flexible and real-time; 2. In the present invention, by accurately analyzing the task priority and time window through the long short-term memory neural network, it is possible to efficiently estimate the task execution time, optimize the task execution order, and reasonably arrange the order of concurrent execution of tasks under limited resources; 3. In the present invention, through the application of graph neural network technology in resource allocation, the generation of the resource allocation plan becomes more intelligent, it is possible to judge in real time whether the current system resources are sufficient to support task execution, and adjust the task execution order in real time to avoid the blockage or delay of tasks caused by resource bottlenecks. Brief description of the drawings

[0017] Figure 1 is the system flow chart of the present invention; Figure 2 is the schematic diagram of the system framework of the present invention. Specific Embodiments

[0018] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, in the description of the present invention, the meaning of "a plurality" is two or more unless otherwise specifically defined.

[0020] Please refer to Figure 1 , the present invention provides a technical solution: A log analysis task scheduling system based on a neural network includes: Graph construction module: Extract task metadata through a database, obtain the execution duration and required resources of each task, map tasks and resource nodes, establish the dependency relationship between tasks as edges, determine whether additional edges need to be added according to the dependency strength, and use task priorities and resource quota information as additional attributes of nodes to construct a task resource dependency graph; Dependency parsing module: Based on the task resource dependency graph, obtain the mutual dependency relationships of all task nodes, calculate the dependency matrix between nodes, determine the dependency relationship between each pair of nodes, sort the nodes according to the dependency strength, adjust the node priorities when the dependency strength exceeds the threshold, and determine whether the tasks can be in an executable state to generate a task execution dependency list; Sequential prediction module: Based on the task execution dependency list, use a long short-term memory neural network to analyze the task priorities and time window sequences, estimate the execution time of tasks, determine the appropriate task execution order, sort the task execution order according to the priorities, perform timing scheduling according to the execution period and priorities, and generate a task execution order list in combination with the delay determination and resource requirements between tasks; Resource Allocation Module: Based on the task execution order list, using a graph neural network, extract the resource requirements of each task in sequence, compare with the current resource status, judge whether there are sufficient resources to meet the concurrent execution of tasks, and adjust resource allocation according to task priorities. If the current resources are insufficient, suspend the execution of some tasks according to priorities, divide the resources into multiple priorities, and generate an optimized resource allocation plan; Execution Control Module: Based on the optimized resource allocation plan, judge whether each resource meets the execution requirements of the task. If the resource is available, start the task execution and record the execution status. If the resource is insufficient, pause the task and reschedule it, continuously monitor the status changes of each task, update the task execution mark, and generate a task execution status report.

[0021] Please refer to Figure 2 , the graph construction module includes: Task Metadata Extraction Sub-module: Extract task metadata from the database, query the database at the same time, obtain the relevant information of all tasks, and store it in a data table to generate a task metadata table; Node Mapping Sub-module: Based on the task metadata table, map tasks and required resources to nodes in the graph. Each task corresponds to a task node, and each resource corresponds to a resource node. By connecting the task nodes and resource nodes, generate a task-resource node mapping table; Dependency Relationship Construction Sub-module: Based on the task-resource node mapping table, obtain the dependency relationships between tasks. Through the dependency relationship identifiers between tasks, establish directed edges between tasks, and judge whether additional edges need to be added according to the dependency strength. If the dependency strength is higher than the predetermined threshold, add edge connections to generate a task-resource dependency graph; Task Metadata Extraction Sub-module: Based on the database, use SQL query statements to extract task metadata from the database. Select task-related fields through the SELECT statement, including task ID, execution duration, and resource requirements. Execute the query operation to obtain the relevant information of all tasks, store it in a temporary table, and store the extracted data in the task metadata table, including task ID, execution duration, and resource requirement information, to generate tabular structured data; Node Mapping Sub-module: Based on the task metadata table, use the node mapping method in graph theory to map tasks and required resources to nodes in the graph. Through the NetworkX library in Python, use the add_node() function to create nodes for each task and resource respectively. The task nodes include task ID, execution duration, and resource requirement information, and the resource nodes include resource type, available quantity, etc. information. Then, use the add_edge() function to connect the task nodes and resource nodes to generate a task-resource node mapping table; Dependency relationship construction sub-module: Based on the task resource node mapping table, use the Dijkstra algorithm to calculate the dependency relationships between task nodes. Through the threshold judgment method of dependency strength, according to the dependency strength between tasks, if the dependency strength is higher than the preset threshold, new dependency relationships are added, directed edges between tasks are constructed, and a task resource dependency graph is formed.

[0022] Please refer to Figure 2 , the dependency parsing module includes: Relationship extraction sub-module: Based on the task resource dependency graph, perform node traversal operations. By reading the start node and end node identifiers of each edge in the graph and comparing the task numbers, obtain the corresponding relationships between task nodes, and extract the edge connection weight as the dependency strength value. Construct a dependency relationship weight matrix and generate a node dependency weight set; Priority sorting sub-module: Based on the node dependency weight set, perform numerical comparison operations on the dependency strength. By summing the weight values in the out-edge set of each node and sorting them by size, extract strongly dependent nodes and establish corresponding priority order labels, adjust the priority arrangement in task scheduling, and generate a task priority order mapping table; Execution verification sub-module: Based on the task priority order mapping table, judge the executable status of each task node. By comparing the hierarchical position of the task in the topological structure and verifying whether all prerequisite nodes have uncompleted dependencies, mark the executable and pending execution statuses, summarize the sorted nodes and status information, and generate a task execution dependency list; Relationship extraction sub-module: Based on the task resource dependency graph, use the graph traversal algorithm. Through depth-first search for node traversal, read the start node and end node identifiers of each edge in the graph, and compare the task numbers to obtain the corresponding relationships between task nodes, obtain the connection weight of each edge, extract the edge connection weight as the dependency strength value, initialize the dependency relationship weight matrix, fill the corresponding values in the matrix according to the node dependency relationship, and generate a node dependency weight set; Priority sorting sub-module: Based on the node dependency weight set, use the sorting algorithm. By summing the weight values in the out-edge set of each node, performing an accumulation summation operation on each weight in the out-edge set, obtaining the total dependency strength value of each node, sorting the dependency strength values of each node in ascending or descending order, extracting the nodes with higher dependency strength values, establishing a priority order label for the sorted nodes, storing the mapping relationship between the nodes and their corresponding priorities, adjusting the priority arrangement in task scheduling, and generating a task priority order mapping table; Execution parity check sub-module: Based on the task priority order mapping table, using the topological sorting algorithm, perform topological sorting on task nodes, judge the executable status of each task node, judge the in-degree of each node, verify whether all prerequisite nodes have uncompleted dependencies. If the in-degree is 0 and all prerequisite nodes have completed dependencies, mark it as executable; if there are uncompleted dependencies, mark it as pending execution. Summarize the sorted nodes and status information to generate a task execution dependency list.

[0023] Please refer to Figure 2 , and the sequential prediction module includes: Time structure analysis sub-module: Based on the task execution dependency list, using the long short-term memory neural network, extract the priority, dependent node number, and time window boundary corresponding to each task node, perform a time interval comparison operation, screen tasks in overlapping intervals and calculate the priority difference, establish the time interaction structure of each task, and generate a set of time interaction matrices; Execution duration evaluation sub-module: Based on the set of time interaction matrices, solve the difference between the start time and end time of each group of tasks, and combine the task resource usage density value to judge the actual duration of the task, perform interval update and reallocation on the task execution duration, and generate a task interval duration schedule; Scheduling order generation sub-module: Based on the task interval duration schedule, jointly sort the priorities, time spans, and required resources of all tasks, construct a scheduling start point identification table through time continuity and resource distribution sparsity, determine the execution order of tasks, establish an executable process chain, and generate a task execution order table; Time structure analysis sub-module: Based on the task execution dependency list, using the long short-term memory neural network, take the priority, dependent node number, and time window boundary corresponding to each task node as input features, construct a neural network model through the Keras library, use the Adam optimizer, set the learning rate to 0.001, perform model training, use the time-distributed function to perform a time interval comparison operation on each task node, screen tasks in overlapping intervals and calculate the priority difference, perform a time interval overlap judgment, establish the time interaction structure of each task, and generate a set of time interaction matrices; Execution duration evaluation sub-module: Based on the set of time interaction matrices, use the time difference calculation method to calculate the difference between the start time and end time of each group of tasks to obtain the duration of each task. Combine the task resource usage density value, and through the resource usage density formula, calculate the actual duration of each task, perform interval update and reallocation on the task execution duration, limit the task execution time within the specified time range, and generate a task interval duration schedule after update; Scheduling order generation sub-module: Based on the task interval duration schedule, a sorting algorithm is adopted. By jointly sorting the priorities, time spans, and required resources of all tasks, a custom sorting function is used to calculate the weights of each task node during sorting, calculate the time continuity of the tasks and the sparsity of resource distribution, evaluate the sparsity of resource distribution, construct a scheduling start point identification table, determine the order of task execution, construct a directed graph, establish an executable process chain between tasks, and generate a task execution order table.

[0024] Long short-term memory neural network, according to the formula:

[0025] Where: represents the task and the task the priority gap between them, represents the end time of the task represents the end time of the task represents the end time of the task represents the end time of the task represents the priority of the task represents the priority of the task represents the weight coefficient of the task time overlap part for the priority difference calculation, represents the weight coefficient of the task priority difference for the calculation result, represents the task and the task the weight coefficient of the task dependency between them for the calculation result, represents the weight coefficient of the task start time difference for the priority difference calculation, represents the task and the task the task dependency between them; Execution process: Calculate the priority difference between the task and the task First, consider the time overlap situation of the tasks. The time overlap degree of the task and the task is measured by calculating and The former represents the gap between the start time of the task and the end time of the task The latter represents the end time of the task and the task​​​​​​ The interval between start times, and then by introducing a weight coefficient , the impact of the time overlap part is adjusted, and its role in the total priority difference calculation is more flexible and adjustable. Then the priority difference of the task is weighted and adjusted by the coefficient Considering the impact of the priority difference on task scheduling, the interdependence between tasks is further introduced , and by the coefficient The mutual dependence between tasks is quantified, reflecting the dependence relationship that must be followed between tasks during the scheduling process. Finally, the formula calculates to consider tasks and tasks The start time difference, and the coefficient is used to adjust the weight of the difference in the total priority difference. By comprehensively considering the time, priority of tasks, and the dependence relationship between tasks, a more accurate scheduling priority difference is generated.

[0026] Please refer to Figure 2 , the resource allocation module includes: Resource requirement extraction sub-module: Based on the task execution order list, using a graph neural network, task index extraction is performed to locate the task number. By reading the resource type identifier and resource quantity value of the corresponding fields of the task, number and value combination construction is carried out to match the relationship between the task and the required resources, and the matching results are registered in the unified record table to generate a task resource requirement list; Allocation condition judgment sub-module: Based on the task resource requirement list, a snapshot of the system resource status is read and a resource remaining quantity mapping table is established. The numerical difference calculation between the resources required by the task and the remaining resources is carried out in sequence, and it is marked whether the current resources meet the task execution conditions. A queuing identifier is appended to the insufficient resource items to generate a resource allocation feasibility mark set; Resource priority division sub-module: Based on the resource allocation feasibility mark set, all schedulable task numbers and corresponding priority values are extracted, priority interval division is carried out, and a resource level queue is constructed. The remaining system resources are mapped to each task group in proportion and the mapping relationship between tasks and resources is registered to generate a resource allocation optimization plan; Resource requirement extraction sub-module: Based on the task execution order list, using a graph neural network, task index extraction is performed to locate the task number. The graph data is loaded, the task nodes and resource nodes are mapped, the resource type identifier and resource quantity value of the corresponding fields of the task nodes are read, the resource requirement information of the task nodes is extracted, number and value combination construction is carried out to match the relationship between the task and the required resources, and the matching results are registered in the unified record table to generate a task resource requirement list; Allocation condition judgment sub-module: Based on the task resource requirement list, adopt the resource status snapshot reading method, read the current system resource status through the SQL query statement, use the SELECT statement to obtain the remaining resource quantity from the resource status table, construct a mapping table of the remaining resource quantity, and calculate the numerical difference between the resources required for the task and the remaining resources in sequence. Calculate the difference between the task resource requirements and the remaining resources, mark whether the current resources meet the task execution conditions, use the if-else statement to judge whether the resources are sufficient, and if not, append a queuing identifier to the corresponding resource item to generate a resource allocation feasibility mark set; Resource priority division sub-module: Based on the resource allocation feasibility mark set, extract all schedulable task numbers and corresponding priority values, extract the task numbers and priorities, perform priority interval division and construct a resource level queue, divide the remaining resources proportionally into each task group, round the resource allocation for each task to ensure the accuracy of resource allocation, register the mapping relationship between tasks and resources, and generate an optimized resource allocation plan.

[0027] Graph neural network, according to the formula:

[0028] Where: Represents the task The total resource requirement calculated by the graph neural network, Is the resource type The unit resource consumption value of, Is the task For the resource type The required quantity of, Is the quantity of all resource types, Is the resource type For the task The weight coefficient of the resource requirement, Is the task And the resource type The dependency between, Is the adjustment weight coefficient of the time resource consumption for the total resource requirement, Is the adjustment weight coefficient of the computing resource consumption for the total resource requirement, Is the task The time resource requirement of, Is the task The computing resource requirement of; Execution process: First, calculate the consumption value of each resource according to the resource type And the task The required quantity Represents the resource type The unit resource consumption value of, and then introduce the weight coefficient , aiming to adjust the influence of different resource types on the resource requirements of the task, reflecting the importance of each resource type. Then, the task and the resource dependency is further adjusted for calculation. The higher the dependency, the stronger the task's demand for resources, which affects the consumption of resources. Next, consider the time resource requirement and the computing resource requirement , which respectively represent the time and computing power resources required for task execution. Through the weight coefficients and , further adjust the proportion of resources in the total demand to ensure that the time and computing resource requirements of the task are appropriately considered. Finally, through comprehensive calculation, the total resource requirement of the task is obtained, providing an optimized resource allocation basis for the task scheduling system.

[0029] Please refer to Figure 2 , the execution control module includes: Task trigger judgment sub-module: Based on the resource allocation optimization scheme, conduct a joint check on the task and resource locked status. By performing an availability judgment operation on the status field of the resources bound to the task, mark the tasks with the resources in the non-occupied state as the to-be-started state and establish a numbered list, generating a task activation identification set; Status record execution sub-module: Based on the task activation identification set, traverse all the numbers of the to-be-executed tasks, and obtain the execution start time and resource usage status. Record the task execution time period and write it into the task status field, set the resource as the occupied state, construct the mapping relationship between the number and time, generating a task execution status record table; Execution result update sub-module: Based on the task execution status record table, poll to obtain the current execution status value of the task. By determining whether the task completion flag and the execution duration upper limit meet the termination conditions, release the occupied resources and update the execution status code and the task completion time, complete the status write operation, generating a task execution status report; Task trigger judgment sub-module: Based on the resource allocation optimization scheme, use an SQL query statement to conduct a joint check on the task and resource locked status. Extract the resource occupancy status field from the resource status table through the SELECT statement, perform an availability judgment operation, and use the if-else statement to judge the status of the resources bound to the task. If the resource is in the non-occupied state, mark the task as the to-be-started state, add the numbers of the to-be-started tasks to the numbered list, generating a task activation identification set; Status record execution sub-module: Based on the task activation identification set, use a for loop to traverse all task numbers to be executed, use an SQL query statement to read the execution start time and resource usage status of the task, construct a data table for the execution status of the task, record the task execution time period and update the task status field, set the resource to the occupied state, construct the mapping relationship between the number and time, and generate a task execution status record table; Execution result update sub-module: Based on the task execution status record table, use a polling algorithm to regularly obtain the current execution status value of the task, obtain the completion flag and execution duration information of the task, use an if-else statement to determine whether the task completion flag and the execution duration upper limit meet the termination condition. If the condition is met, release the occupied resources and update the execution status code and task completion time through the SQL UPDATE statement, perform the UPDATE operation, update the status field of the task, complete the status write operation, and generate a task execution status report.

[0030] The above is only a preferred embodiment of the present invention, and it does not limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A log analysis task scheduling system based on a neural network, characterized in that The system includes: Graph construction module: Extract task metadata through the database, obtain the task execution duration and resource requirements, map tasks and resource nodes, establish the dependencies between tasks as edges, determine whether to add new edges based on the dependency strength, and use the priority and resource quota as node attributes to construct a task-resource dependency graph; Dependency resolution module: Based on the task-resource dependency graph, obtain the dependencies between task nodes, calculate the dependency matrix, sort the nodes according to the dependency strength, adjust the priority if the dependency strength exceeds the threshold, determine whether the task can be executed, and generate a task execution dependency list; Sequence prediction module: Based on the task execution dependency list, use a long short-term memory neural network to analyze the task priority and time window, estimate the execution time, sort the task execution order according to the priority, and combine the task delay and resource requirements to generate a task execution order table; Resource allocation module: Based on the task execution order table, use a graph neural network to extract the resource requirements of each task, compare the resource status, determine whether there are sufficient resources to support concurrent execution, adjust the resource allocation according to the priority, suspend the execution of some tasks if the resources are insufficient, and generate an optimized resource allocation plan; Execution control module: Based on the optimized resource allocation plan, determine whether the resources meet the task execution requirements. If available, start the task execution and register the status. If insufficient, suspend the task and reschedule, monitor the change of task status, update the execution mark, and generate a task execution status report.

2. The log analysis task scheduling system based on a neural network according to claim 1, wherein The graph construction module includes: Task metadata extraction sub-module: Extract task metadata through the database, query the database at the same time to obtain the relevant information of all tasks, and store it in a data table to generate a task metadata table; Node mapping sub-module: Based on the task metadata table, map tasks and required resources to nodes in the graph. Each task corresponds to a task node, and each resource corresponds to a resource node. By connecting the task nodes and resource nodes, generate a task-resource node mapping table; Dependency relationship construction sub-module: Based on the task-resource node mapping table, obtain the dependencies between tasks, establish directed edges between tasks through the dependency relationship identifiers between tasks, determine whether new edges need to be added according to the dependency strength. If the dependency strength is higher than the predetermined threshold, add edge connections to generate a task-resource dependency graph.

3. The log analysis task scheduling system based on a neural network according to claim 1, wherein The dependency resolution module includes: Relationship extraction sub-module: Based on the task-resource dependency graph, perform a node traversal operation. By reading the start node and end node identifiers of each edge in the graph and comparing the task numbers, obtain the corresponding relationships between task nodes, and extract the edge connection weight as the dependency strength value to construct a dependency relationship weight matrix and generate a node dependency weight set; Priority sorting sub-module: Based on the node dependency weight set, perform a numerical comparison operation of the dependency strength. By summing the weight values in the out-edge set of each node and sorting them by size, extract strongly dependent nodes and establish corresponding priority order labels, adjust the priority arrangement in task scheduling, and generate a task priority order mapping table; Execution Parity Sub-module: Based on the task priority order mapping table, determine the executable status of each task node. By comparing the hierarchical position of the task in the topological structure and verifying whether there are any outstanding dependencies for all prerequisite nodes, mark the executable and pending execution statuses, summarize the sorting nodes and status information, and generate a task execution dependency list.

4. The log analysis task scheduling system based on a neural network according to claim 1, wherein The sequence prediction module includes: Time Structure Analysis Sub-module: Based on the task execution dependency list, use a long short-term memory neural network to extract the priority, dependent node number, and time window boundary corresponding to each task node, perform a time interval comparison operation, filter overlapping interval tasks and calculate the priority difference, establish the time interaction structure of each task, and generate a set of time interaction matrices; Execution Duration Evaluation Sub-module: Based on the set of time interaction matrices, solve the difference between the start time and end time of each group of tasks, and combine the task resource usage density value to judge the actual duration of the task. Update and reallocate the task execution duration interval, and generate a task interval duration schedule; Scheduling Sequence Generation Sub-module: Based on the task interval duration schedule, jointly sort the priorities, time spans, and required resources of all tasks. Construct a scheduling start point identification table through time continuity and resource distribution sparsity, determine the order of task execution, establish an executable process chain, and generate a task execution sequence table.

5. The log analysis task scheduling system based on a neural network according to claim 4, characterized in that The long short-term memory neural network, according to the formula: Wherein: represents the task and the task the priority gap between, represents the task end time, represents the task end time, represents the task end time, represents the end time of the task, represents the task priority, represents the task priority, represents the weight coefficient of the overlapping part of the task time for calculating the priority difference, represents the weight coefficient of the influence of the task priority difference on the calculation result, represents the task and the task the weight coefficient of the task dependency between for the calculation result, represents the weight coefficient of the start time difference of the task for calculating the priority difference, represents the task and the task the task dependency between.

6. The log analysis task scheduling system based on a neural network according to claim 1, wherein The resource allocation module includes: Resource Requirement Extraction Sub-module: Based on the task execution sequence table, use a graph neural network to perform task index extraction and locate the task number. By reading the resource type identifier and resource quantity value of the corresponding task fields, perform a combination construction of the number and value, match the relationship between the task and the required resources, and register the matching results in a unified record table to generate a task resource requirement list; Allocation Condition Judgment Sub-module: Based on the task resource requirement list, read the system resource status snapshot and establish a resource remaining quantity mapping table. Sequentially calculate the numerical difference between the required resources of the task and the remaining resources, mark whether the current resources meet the task execution conditions, and append a queuing identifier to the insufficient resource items to generate a resource allocation feasibility marking set; Resource Priority Division Sub-module: Based on the resource allocation feasibility marking set, extract all schedulable task numbers and corresponding priority values, perform priority interval division and construct a resource level queue, map the remaining system resources to each task group in proportion and register the mapping relationship between the task and the resources, and generate a resource allocation optimization plan.

7. The neural network-based log analysis task scheduling system according to claim 6, wherein The graph neural network, according to the formula: Wherein: represents the task The total resource requirement calculated by the graph neural network, is the unit resource consumption value of the resource type ; is the requirement quantity of the task for the resource type ; is the quantity of all resource types, is the resource type for the task The weight coefficient of the resource requirement, is the task and the resource type The dependency between; is the adjustment weight coefficient of the time resource consumption on the total resource requirement, is the adjustment weight coefficient of the computing resource consumption on the total resource requirement, is the task The time resource requirement of, is the task The computing resource requirement of.

8. The neural network-based log analysis task scheduling system according to claim 1, wherein, The execution control module includes: Task Trigger Judgment Sub-module: Based on the resource allocation optimization plan, jointly check the task and resource lock status. By performing an availability judgment operation on the status field of the task-bound resources, mark the tasks with the resources in the non-occupied state as pending start status and establish a number list, and generate a task activation identification set; Status record execution sub-module: Based on the task activation identification set, traverse all task numbers to be executed, obtain the execution start time and resource usage status, record the task execution time period and write it into the task status field, set the resource to the occupied state, construct the mapping relationship between the number and time, and generate a task execution status record table; Execution result update sub-module: Based on the task execution status record table, poll to obtain the current execution status value of the task. By determining whether the task completion flag and the execution duration upper limit meet the termination conditions, release the occupied resources and update the execution status code and task completion time, complete the status write operation, and generate a task execution status report.

Images