Multi-dimensional data intelligent processing and analysis method

Abstract

The invention discloses a multi-dimensional data intelligent processing and analysis method, which relates to the technical field of data processing, and comprises the following steps of: performing static dependency analysis on a parallel task set, and constructing a directed resource dependency graph containing a semaphore occupancy relationship, a calling hierarchy and a recursive path; depth-first loop detection is performed based on the directed resource dependency graph, a potential deadlock closed loop is identified and divided into a plurality of edge loop-free sub-graphs, and a sequenced execution plan is generated for each sub-graph. According to the method, by introducing static dependency analysis, resource dependency graph construction, a token lease mechanism and a multi-level semaphore pool, active avoidance of task deadlock risks and automatic recovery of resource anomalies in the engineering machinery multi-dimensional data processing process are achieved. Through sequential scheduling, adaptive regulation and control and self-healing monitoring strategies, the task scheduling stability and the resource utilization efficiency in a high-concurrency scene are effectively improved, and closed-loop optimization of task scheduling is realized in cooperation with process traceability and evolutionary knowledge base feedback.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-dimensional data intelligent processing and analysis method. BACKGROUND

[0002] Multi-dimensional data intelligent processing and analysis refers to the whole process of efficiently, accurately and dynamically processing and comprehensively analyzing a large amount of, complex, and various data (including running state, performance parameters, alarm information, etc.) generated in the operation of engineering machinery by using various intelligent technology means. Multi-dimension not only embodies the heterogeneity of data sources (such as different equipment, different sensor types, and different time periods), but also includes the diversity of data structure (such as structured log, alarm record, and running curve). Intelligent processing embodies the realization of automatic and efficient raw data cleaning and organization through parallel file processing, adaptive thread scheduling, dynamic data deduplication and incremental update, and high-performance CSV parsing. Intelligent analysis includes intelligent classification and identification of alarm information, correlation aggregation and multi-dimensional fusion of key parameters, and finally supports real-time evaluation of engineering machinery equipment state, risk warning, and decision optimization.

[0003] The prior art has the following disadvantages: In the process of multi-dimensional data intelligent processing and analysis, in order to improve the processing efficiency of large-scale data files, a parallel processing mode based on asynchronous task and semaphore control mechanism is often used. This mode uses an asynchronous programming model to concurrently schedule multiple data processing tasks, and uses a semaphore to dynamically control the number of concurrent tasks, so as to realize the orderly allocation and efficient use of resources. However, in actual application, if some asynchronous tasks recursively call themselves or indirectly depend on other task resources whose semaphores have not been released during execution, it is easy to cause mutual waiting between tasks, thereby causing a typical deadlock recursion problem. Once this problem occurs, the processing thread cannot be released, the thread pool resource is continuously occupied, and finally the entire data processing flow is blocked, which seriously weakens the real-time processing capability of data, further causes monitoring data processing delay and alarm information uploading lag, and causes the running state of engineering machinery to be unable to be sensed and responded in time, and even seriously affects the effective play of the fault warning mechanism, which becomes a key technical obstacle restricting the stability of the existing parallel processing architecture.

[0004] The above information disclosed in the background section is only used to strengthen the understanding of the background of the present disclosure, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The application aims to provide a multi-dimensional data intelligent processing and analysis method, which introduces static dependency profiling, resource dependency graph construction, token lease mechanism and multi-level semaphore pool, to actively avoid task deadlock risk and automatically recover resource exceptions in the multi-dimensional data processing process of engineering machinery. Through sequential scheduling, adaptive control and self-healing monitoring strategies, the stability of task scheduling and resource utilization efficiency in high concurrency scenarios are effectively improved. With process tracing and evolution knowledge base feedback, closed-loop optimization of task scheduling is realized to solve the problems in the above background technology.

[0006] To achieve the above purpose, the application provides the following technical scheme: a multi-dimensional data intelligent processing and analysis method, comprising the following steps: Perform static dependency profiling on the parallel task set to be processed, construct a directed resource dependency graph containing the semaphore occupation relationship, call level structure and recursive potential path between tasks, and use it to identify resource competition and recursive dependency between tasks; Based on the directed resource dependency graph, perform depth-first loop detection to identify dependency loops with potential deadlock risk, and divide the loop into multiple edge acyclic subgraphs. For each edge acyclic subgraph, generate a corresponding sequential execution plan to avoid the occurrence of deadlock risk at the source stage; Introduce a token lease mechanism in the sequential execution plan, assign a semaphore token lease period with an automatic return mechanism to each asynchronous task in the plan, and configure the corresponding lease timeout callback logic to ensure that the semaphore resource can be actively released in the case of task execution exception or blocking state; Based on the token lease mechanism, construct a multi-level semaphore pool. According to the priority of the task, map high-priority data processing tasks to the upper layer semaphore pool, and map low-priority tasks to the lower layer semaphore pool, to reduce the conflict of semaphore competition between tasks and reduce the probability of cross-layer blocking; Deploy an adaptive scheduler on the layered semaphore pool. The scheduler continuously monitors the allocation and recovery of tokens in each layer of the semaphore pool. When it detects that the upper layer semaphore pool resource is saturated, it automatically promotes the unused tokens in the lower layer semaphore pool to the upper layer pool, achieving dynamic resource replenishment and utilization efficiency optimization; Embed a self-healing monitoring thread in the adaptive scheduler. The self-healing monitoring thread is used to periodically detect semaphore lease timeout non-return events and unbalanced token usage rate in each layer. When it detects resource usage abnormalities, it automatically triggers the corresponding task rollback mechanism, reconstructs the directed resource dependency graph, and regenerates the sequential execution plan of the task, to restore the resource balance of the system; Enabling runtime process trace recording, recording and writing each resource dependency graph reconstruction process and task rollback decision execution process into the evolution knowledge base, which serves as historical reference data for the next round of static dependency profiling, forming a continuous closed-loop optimization mechanism of profiling-scheduling-monitoring-feedback.

[0007] Preferably, the step of performing static dependency profiling on the parallel task set comprises: Extracting the input data source, processing logic, semaphore application and release information, and call path of each to-be-executed task, and constructing an initial mapping relationship between tasks and semaphores; On the basis of the mapping relationship, a task resource occupation order chain is established, and an initial directed graph with tasks as nodes and resource dependencies as edges is constructed; Performing depth-first traversal on the directed graph, using Tarjan algorithm, Kosaraju algorithm or back edge detection method to identify recursive calls and closed-loop dependency paths, and assigning weights to the edges in the graph in combination with task priority and execution time; Performing structural optimization and task classification on the constructed dependency graph, extracting a set of independent tasks, archiving recursive subgraphs, and generating a resource application and release record table for subsequent scheduling analysis.

[0008] Preferably, the step of performing loop detection based on the directed resource dependency graph and generating a sequential execution plan comprises: Performing depth-first traversal in the dependency graph, identifying dependency loops and recording semaphore number, task priority and path weight information; Breaking the loop into multiple edge acyclic subgraphs according to the boundary nodes, and recording the resource conflict information at the broken edges; Performing topological sorting on each acyclic subgraph, and generating a sequential execution plan according to the task dependency order and priority; Establishing a mapping relationship between the execution plan and the original dependency graph and persisting it as the basis for subsequent scheduling.

[0009] Preferably, the step of allocating a semaphore token lease period in the sequential execution plan comprises: Setting an independent resource entry for each asynchronous task, and allocating a semaphore token lease with a fixed period, defining the token usage time limit, after setting the lease, injecting a lease monitoring mechanism to detect the task state and token remaining time in real time, if the task is blocked or abnormal, triggering the lease timeout callback logic to forcibly release the semaphore; Combining the task execution context and resource usage state, dynamically adjusting the lease period limit to prevent abnormal interruption or resource waste, and writing the lease behavior throughout the process into the log as a reference basis for the next round of resource allocation strategy optimization and task behavior evaluation.

[0010] Preferably, the step of constructing a multi-level semaphore pool based on a token lease mechanism comprises: According to the task priority, the tasks to be scheduled are divided into high-priority tasks and low-priority tasks, which are mapped to the upper-layer and lower-layer semaphore pools, respectively; The semaphore resources are divided into the semaphore pools of each layer according to weights, and the tasks bind leases and join the corresponding layer's to-be-allocated queue, and when the upper-layer semaphore pool resource is tight, the recyclable tokens are temporarily allocated from the lower-layer pool, and the token ownership identifier and lease period information are updated.

[0011] Preferably, the step of deploying an adaptive scheduler on the basis of the semaphore pool comprises: Periodically monitor the allocation state, recycling condition and task waiting time of the tokens in each layer's semaphore pool, and judge whether the upper-layer resource is saturated; when detecting that the upper-layer resource is tight, select the idle tokens not bound to tasks from the lower-layer semaphore pool for cross-layer migration, and update the layer label and lease configuration thereof; After the migration, evaluate the influence of the allocation behavior on the resource utilization and task response of the upper and lower layers, dynamically adjust the allocation strategy, record the migration behavior and resource state change to the resource scheduling log, and use the same as the basis for subsequent strategy optimization and evolution analysis.

[0012] Preferably, the step of embedding a self-healing monitoring thread in the scheduler comprises: Periodically detect the lease state of the tokens and the task activity in each layer's semaphore pool, and identify the resource use imbalance event or the event that the lease is overdue and not returned; After detecting the abnormality, perform a task rollback operation to release the occupied tokens and update the state of the related edges in the directed resource dependency graph to be damaged; According to the updated resource state, reconstruct the dependency graph and regenerate a sequenced execution plan to replace the original execution path.

[0013] Preferably, the step of recording the directed resource dependency graph reconstruction and task rollback process as runtime trace data comprises: After the task rollback is completed, collect the dependency graph change, abnormal semaphore and task state trajectory, and establish a behavior record based on a time line; Archive the graph structure adjustment content and rollback decision process in a versioned manner, and bind and store the same as a parseable record unit; Input the record into an evolution knowledge base to perform mode clustering and risk labeling, and extract scheduling optimization suggestions for subsequent dependency profiling; In the next round of static dependency profiling, refer to the content of the knowledge base, correct the graph structure and optimize the task path, and realize closed-loop optimization.

[0014] In the above technical solutions, the present application provides technical effects and advantages: The application can identify the resource occupation path and recursive call relationship that may cause deadlock in advance before the multi-dimensional data processing task of the engineering machinery is executed by introducing the static dependency profiling and the directed resource dependency graph construction mechanism. By performing the depth-first loop detection on the dependency graph, and disassembling the potential risk path into multiple edge acyclic subgraphs, and combining the sequential scheduling strategy, the logical isolation and execution path rearrangement of the high-risk task are realized, so that the occurrence of resource interlocking is fundamentally avoided. In addition, the token lease mechanism and lease timeout callback logic are introduced, so that even if an abnormal block occurs during task running, the semaphore can be automatically recycled, avoiding resource hanging and thread exhaustion. This parallel processing structure with preventive and recovery functions makes the task scheduling process more controllable and reliable, effectively improves the processing stability in the high-concurrency environment such as remote operation and maintenance of engineering machinery and multi-threaded data cleaning, and solves the problem of deadlock recursion in the traditional asynchronous scheduling mode.

[0015] The application constructs a multi-level semaphore pool structure, and combines a task priority mapping mechanism, so that high-priority tasks are allocated to the upper layer semaphore pool and low-priority tasks are allocated to the lower layer according to the urgency of the data processing task, thereby reducing the resource contention and semaphore blocking between tasks. On this basis, an adaptive scheduling strategy is introduced, the token usage of each layer is dynamically monitored, and token migration and automatic replenishment across layers are realized when the high-priority task resource is in shortage, so as to ensure the efficient operation of the key task. At the same time, the embedded self-healing monitoring thread is used to detect resource usage abnormalities and lease expiration in real time, and once the resource imbalance is found, task rollback and resource dependency graph reconstruction are triggered to generate a new sequential execution path to restore the overall resource scheduling balance. During operation, all dependency reconstruction and scheduling behaviors are written into the evolution knowledge base through the process tracing mechanism, providing experience feedback for subsequent static profiling, realizing the closed-loop adaptive processing of task analysis, resource allocation, scheduling execution and historical optimization. This method significantly improves the resource usage efficiency and scheduling intelligence in the process of intelligent perception and state evaluation of engineering machinery, and is especially suitable for high-frequency data access and task-intensive operation environment. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0017] Figure 1 The method flowchart of the multi-dimensional data intelligent processing and analysis method of the present application. DETAILED DESCRIPTION

[0018] Example implementations are now described with reference to the drawings. Example implementations can, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these example implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0019] The present application provides a multi-dimensional data intelligent processing and analysis method as shown in Figure 1 The method comprises the following steps: The static dependency profiling is performed on the parallel task set to be processed to construct a directed resource dependency graph containing the semaphore occupation relationship, the calling hierarchy and the recursive potential path among the tasks, which is used to identify the resource competition and recursive dependency among the tasks. The specific implementation of this step is as follows: The definition information of all data processing tasks to be executed in parallel is obtained, including the input data source required by each task, the function structure of the processing logic, the semaphore resource identifier applied and released, and the calling chain in the execution process. The above task meta information is extracted by static scanning of the source code or a predefined task description file, and the initial mapping relationship between the tasks and the semaphores is established. In this process, special attention is paid to the function nesting structure and the task scheduling instructions, and the direct or indirect calling path among the tasks is determined, which lays a foundation for subsequent dependency relationship modeling.

[0020] Based on the extracted information, the application order and the release position of the semaphore resource in the logical execution process of each task are analyzed, and whether there is a shared semaphore or repeated occupation is identified. By constructing a semaphore application order chain, the situation that the same resource is requested by multiple tasks is sequentially arranged, and the possible blocking points of each task are recorded. On this basis, each task is regarded as a node in the graph, and the occupation or waiting relationship of the semaphore is regarded as an edge, and the entire task set is organized into an initial directed graph, wherein the direction of the edge represents the resource dependency relationship between the tasks.

[0021] The calling level analysis is performed on all paths in the above directed graph, and the calling path of each task is labeled by using a depth-first traversal algorithm, so as to identify the paths with recursive calling, cross calling or closed loop dependency. In particular, when a task re-directs itself in the calling chain, or indirectly forms a resource loop through other tasks, the recursive potential level of the path should be marked in the graph. In this process, further combined with dynamic attributes such as task priority and estimated execution time, a weight value is assigned to each edge in the graph, which is used to measure the dependency strength and blocking risk, so as to enhance the decision-making ability of the graph model in the subsequent processing; In the multi-dimensional data task dependency analysis, in order to identify recursive calls, cross calls or closed loop dependencies, the depth first traversal algorithm can be implemented according to a plurality of existing mature algorithms, including Tarjan's strongly connected component algorithm, Kosaraju's algorithm and the classical recursive depth first search (DFS) combined with back edge detection technology. Among them, Tarjan's algorithm is one of the most commonly used efficient algorithms, which can identify all strongly connected components in the graph in one depth first traversal process, and effectively identify all node sets constituting the loop through the construction of "stack structure + time stamp label + lowest reachable node index" mechanism. These sets may correspond to recursive or closed loop dependency paths in the task. Kosaraju's algorithm uses a double depth first traversal strategy, first performs a reverse ordering on the original graph, and then traverses the transpose graph, which can also identify strongly connected components, and is suitable for scenarios where the graph structure is relatively stable. For relatively simple structures or tasks with low dependency depth, basic recursive DFS combined with stack backtracking mechanism can be used to record visited nodes and backtracking paths in the traversal path. If a node already exists in the current path, it is determined that there is a back edge, thereby inferring potential recursion or cross call. The above algorithms are mature and feasible depth first traversal variants in the field of graph theory, with high accuracy and execution efficiency, and have wide engineering applicability and theoretical support in static task dependency graph analysis. The choice of algorithm depends on the size, density and performance requirements of the task graph.

[0022] Finally, the constructed directed resource dependency graph is optimized and classified, and the task set with no obvious dependency relationship is stripped as an independent parallel processing unit. At the same time, the subgraph with potential conflict or recursive dependency is numbered and archived, providing structured input for subsequent deadlock loop detection and execution path planning. On this basis, the dependency graph can be visualized and converted, and the atlas can be output for manual review and subsequent dependency learning. At the same time, the resource application, release location and possible blocking chain of each task in the graph are output as a record table to support risk prediction and behavior tracking during task scheduling.

[0023] Through the above method, the static analysis of complex resource dependency relationship in parallel task set is realized, which not only can identify potential deadlock paths and resource bottlenecks before execution, but also can provide comprehensive data support for intelligent optimization of dynamic scheduling strategy.

[0024] Based on the directed resource dependency graph, depth first loop detection is performed to identify dependency loops with potential deadlock risk, and the closed loop is divided into a plurality of edge acyclic subgraphs. For each edge acyclic subgraph, a corresponding sequential execution plan is generated to avoid the occurrence of deadlock risk at the source stage. The specific implementation method for this step is as follows: Building upon the previously constructed directed resource dependency graph, a depth-first search traversal is performed across the entire graph to identify any directed cycles that backtrack to the starting node. During the traversal, an access marker and backtracking path stack are set for each task node, and the order of resource acquisition and release nodes is recorded in the search path. If repeated access to a preceding node is found in the current traversal path, it is marked as a potential dependency loop. In the marking phase, the semaphore resource numbers, estimated request time windows, and priorities of participating tasks involved in the loop path are additionally recorded to construct a complete deadlock formation factor profile. This loop detection not only identifies explicit loop structures but also judges potential high-risk implicit loops through path weight differences, improving the comprehensiveness and foresight of the detection.

[0025] For the detected set of dependency loop paths, a boundary pruning strategy is executed. Each loop is split at its boundary nodes in the resource dependency graph, separating the original loop into multiple edge-acyclic subgraphs. Edge nodes are intermediate task nodes that are dependent on nodes within the loop and connected to nodes outside the loop. Through the loop-breaking strategy, concurrent access requests for shared resources are logically divided into different subgraphs, thereby transforming the original closed-loop structure into a controllable set of acyclic task fragments. Each subgraph is assigned an independent number for subsequent scheduling and ordering. Simultaneously, resource conflict attributes at the broken edges are recorded during the splitting process for dependency ordering evaluation during subsequent sequentialization, ensuring that the splitting result does not logically violate the semantic consistency of the original dependency chain.

[0026] Based on the multiple edge acyclic subgraphs formed, a directed acyclic graph topology sorting algorithm is executed according to the topological dependencies between tasks within them to generate corresponding sequential execution plans. This sorting process assigns a fixed execution order to each task based on the order of its resource requests and the execution position of the original call chain, ensuring that no task's semaphore request is earlier than the release node of its dependent tasks. If there are multiple task nodes with an in-degree of zero during the sorting, the scheduling order is determined based on a strategy that minimizes task priority or resource weight, thus balancing parallelism and deadlock risk control. In addition, this sequential plan establishes an explicit dependency chain between each task and its predecessor node, facilitating controllable execution of task activation and semaphore passing in actual scheduling. In multi-task dependency analysis, for multiple decomposed edge acyclic subgraphs, several existing directed acyclic graph (DAG) topological sorting algorithms are commonly used to generate sequential execution plans. Each algorithm has its own characteristics and is suitable for different task scales and execution efficiency requirements: One commonly used algorithm is Kahn's algorithm, which continuously removes nodes with an in-degree of zero from the graph and adds them to the sorted list until no nodes can be removed or a cycle is detected. It has the advantages of simple implementation, suitability for large-scale graph processing, and good support for multi-threaded concurrent processing. Another classic method is the depth-first search (DFS) based topological sorting algorithm, which recursively traverses the depth of each node in the graph, adding nodes to the result stack during backtracking. The final stack order is the topological sorting result. This method has the advantage of clear structure and ease of integration with recursive path analysis in the dependency graph. In addition, for large-scale task graphs and heterogeneous weighted dependency scenarios, an improved Kahn algorithm assisted by a priority queue can be used. This algorithm selects nodes with higher scheduling priority or lower resource dependency cost from those with zero in-degree in each round for execution, improving the system adaptability and resource scheduling efficiency of the sorting algorithm. If resource load balancing and shortest execution path are required in practical applications, sorting variants based on dynamic programming or greedy strategies, such as Minimum Path Cost Topological Sorting, can be extended. Depending on the structural complexity of the task dependency graph, the task granularity, and the priority weight distribution, one or more of the above topological sorting algorithms can be flexibly selected to generate a stable, efficient, and well-scheduled sequential execution plan.

[0027] Finally, all generated sequential execution plans are aggregated into an execution plan list, serving as the foundational input for subsequent parallel scheduling phases. Simultaneously, the mapping relationship between each execution plan and the original dependency graph is stored in the task scheduling history. To ensure rapid response under future task combination changes, the sequential result is persistently stored after each execution plan is generated, and a graphical dependency path and sorting result are output for manual review. After the plan list is generated, all edge acyclic subgraphs are back-checked to ensure there are no logical conflicts or time overlaps between tasks. Once verified, the plan is submitted to the subsequent scheduling process, achieving structural avoidance of potential deadlock dependencies from the source stage, effectively improving the stability and robustness of multi-task concurrent processing.

[0028] Introduce a token lease mechanism in the sequential execution plan, allocate a semaphore token lease period with an automatic return mechanism upon expiration to each asynchronous task in the plan, and configure corresponding lease timeout callback logic to ensure that semaphore resources can be actively released in the event of task execution abnormality or blocking. The specific implementation method for this step is as follows: Based on the sequential execution plan generated in the previous step, an independent resource allocation entry point is set for each asynchronous task within the plan, and a corresponding semaphore token is assigned to it. To prevent issues such as semaphores failing to release properly due to external dependency blocking, internal logic bottlenecks, or resource contention conflicts during task execution, a fixed-term token lease is bound to the semaphore during allocation. This lease defines the token's usage time limit, starting when the task starts and triggering automatic semaphore reclamation logic when the lease expires, thus avoiding indefinite resource occupation. The setting of this lease period needs to be dynamically adjusted based on multiple factors such as task type, historical execution time distribution, resource sensitivity, and priority, and a lease buffer threshold can be set for fault tolerance.

[0029] After token lease allocation, a lease monitoring mechanism is injected into the semaphore lease period bound to each task. This mechanism is responsible for real-time monitoring of the correspondence between the remaining validity period of the token and the current execution status of the task. When a task is in an active execution state, the mechanism only records progress information; once it is found that a task has not completed the predetermined semaphore release operation within its bound lease period and is in a blocked, suspended, or abnormal state, it is immediately marked as "potentially invalid" and the corresponding timeout handling process is scheduled. This process will trigger the lease timeout callback logic, forcibly returning the semaphore resource to the semaphore pool through a predefined safe release channel, ensuring that the overall resources are not hoarded and cannot be reclaimed.

[0030] Building upon the aforementioned lease duration monitoring and callback mechanisms, a further intelligent dynamic lease duration reassessment mechanism is designed. This mechanism supports real-time perception of the actual operational status of tasks during execution and allows for scaling up or down adjustments to the original lease duration within a reasonable range. For example, if a task is not completed by the lease expiration date, but its execution context shows it as active with continuous output, and the semaphore resource is idle, the lease duration of the task can be automatically extended to avoid unnecessary interruptions and rollbacks. Conversely, if resources are scarce and tasks fail to respond promptly, a lease termination signal can be triggered in advance, making the release operation more proactive and forward-looking. This reassessment mechanism relies on the task execution priority and path dependency data in the aforementioned sequential execution plan to prevent lease duration adjustments from causing execution order disruptions.

[0031] To achieve closed-loop recording of the entire token lease process, each lease allocation, use, recycling, and revaluation action is marked with a timestamp and a unique task identifier and written to the data processing behavior log, forming a complete resource lifecycle trajectory. These logs will serve as crucial inputs for the next round of sequential plan construction and semaphore resource allocation strategy optimization, enabling the entire data processing flow to possess self-learning and adaptive capabilities. Simultaneously, during system stability assessment, these behavior logs can also be used to diagnose frequently blocking tasks, evaluate the rationality of lease duration settings, and verify the response effectiveness of the automatic callback mechanism, providing data support for task scheduling security and processing efficiency.

[0032] By employing the above methods, the lifecycle behavior of semaphores can be finely controlled within the framework of sequential execution plans, minimizing the risk of resource consumption caused by asynchronous blocking and providing stable and reliable resource scheduling guarantees for the efficient processing of multi-dimensional data.

[0033] A multi-level semaphore pool is constructed based on the token lease mechanism. According to the priority of the tasks, high-priority data processing tasks are mapped to the upper-level semaphore pool, and low-priority tasks are mapped to the lower-level semaphore pool, so as to reduce semaphore contention conflicts between tasks and reduce the probability of cross-level blocking. The specific implementation method for this step is as follows: First, after completing token lease allocation and establishing a task lease management mechanism, all tasks to be executed are prioritized and categorized into at least two types based on their priority attributes in the sequential execution plan: high-priority tasks and low-priority tasks. This priority classification can be calculated comprehensively based on multiple dimensions such as the data type of the task, the urgency of the business, the processing latency sensitivity, and the historical blocking frequency. After classification, the total resource requirements and expected semaphore occupancy time for each type of task are statistically analyzed, and the corresponding semaphore pool capacity and resource distribution strategy are planned accordingly, serving as the parameter basis for constructing a multi-level semaphore pool.

[0034] Secondly, a multi-level semaphore pool structure is constructed, dividing semaphore resources into different pool levels according to preset weights. The upper-level semaphore pool prioritizes high-priority tasks, while the lower-level semaphore pools schedule low-priority tasks. Each semaphore pool is an independent resource set, not sharing semaphore tokens, to achieve resource isolation. Each task, while binding a semaphore token lease, is mapped to the designated semaphore pool's allocation queue based on its priority tag. This step ensures that high-priority tasks have higher allocation priority during resource scheduling, avoiding the problem of core tasks being blocked due to resource preemption by low-priority tasks. When creating the semaphore pool, in conjunction with the aforementioned token lease mechanism, different lease distribution strategies and default timeouts are set for each level to adapt to the execution characteristics of various tasks.

[0035] Subsequently, priority-inheritance-based concurrent scheduling logic is introduced into each semaphore pool. This ensures that when a short-term resource shortage occurs in the upper-level semaphore pool, the system can temporarily allocate reclaimable idle tokens from the lower-level pool to alleviate the waiting risk of high-priority tasks, provided that resources are safe. This cross-level allocation must be performed while the semaphore lease is still valid, and the token's ownership marker and lease duration timestamp must be updated synchronously to prevent reclamation chaos and lease duration overwriting errors. Simultaneously, a priority fallback protection mechanism is implemented between semaphore pools to prevent low-priority tasks from being delayed for extended periods due to temporary resource occupation, thus avoiding overall scheduling unfairness. This strategy balances resource utilization efficiency and task processing fairness, making semaphore allocation more flexible and adaptive in multi-task, high-concurrency environments.

[0036] Finally, to ensure the long-term stability and scheduling transparency of the multi-level semaphore pool, all task mapping records, semaphore allocation trajectories, cross-level allocation behaviors, and lease status changes are persistently stored in the form of event logs. Backtracking analysis is performed after subsequent task execution to determine whether resource usage distribution meets preset load balancing standards. Simultaneously, during resource scheduling, the resource utilization rate, allocation frequency, and task waiting time of each semaphore pool are periodically calculated to assess the rationality of the current resource allocation strategy. If necessary, a resource reallocation mechanism is triggered to dynamically adjust the semaphore pool capacity and lease term strategy.

[0037] By utilizing a multi-level semaphore pool structure based on priority mapping, combined with a token lease mechanism and cross-layer coordination scheduling strategy, the resource conflicts caused by semaphore contention between tasks are significantly reduced, the real-time scheduling capability of high-priority tasks is improved, and the risk of cross-layer blocking propagation is effectively suppressed. This provides a more efficient, secure, and scalable resource management solution for the parallel operation of complex tasks in multi-dimensional data processing scenarios.

[0038] An adaptive scheduler is deployed on top of the hierarchical semaphore pool. The scheduler continuously monitors the allocation and recycling of tokens in each semaphore pool. When the upper semaphore pool is found to be saturated, unused tokens in the lower semaphore pool are automatically moved to the upper pool, thereby achieving dynamic resource replenishment and utilization efficiency optimization. The specific implementation method for this step is as follows: Building upon the multi-level semaphore pool structure and task priority mapping, an adaptive scheduling mechanism is introduced into the semaphore pool system to further improve resource utilization efficiency and the responsiveness of high-priority tasks. This mechanism continuously monitors the token allocation status, usage frequency, and recycling interval within each semaphore pool layer using periodic or triggered scanning, and maintains a set of real-time metrics, including token allocation rate, lease recycling ratio, task waiting time, and the number of idle tokens. During monitoring, particular attention is paid to resource saturation in the upper-level semaphore pool caused by the concurrent aggregation of high-priority tasks; specifically, over multiple monitoring periods, the average waiting time for upper-level token allocation requests significantly increases, and the number of idle tokens drops below a threshold. This judgment mechanism provides a basis for subsequent scheduling actions, ensuring that resource allocation is conducted with sufficient data support.

[0039] When the scheduler identifies a resource shortage in the upper-layer semaphore pool, it initiates a cross-layer token allocation process. This automatically selects a batch of idle tokens from the lower-layer semaphore pool that are unallocated or reclaimed and currently not bound to any task, constructing a token migration candidate set. During the selection process, the token status is double-checked to ensure it is not in lease cooldown, expired waiting, or under abnormal rollback, preventing erroneous migration that could disrupt resource status. After selecting a token, the scheduler regenerates its token identifier and redefines its layer label, while updating the lease configuration so that it can be immediately used for resource allocation to high-priority tasks in the upper layer after migration. This migration process is completed at the memory level and does not involve task reordering, thus ensuring low latency and high real-time performance.

[0040] To prevent resource management imbalances caused by frequent cross-layer allocations, the adaptive scheduler automatically assesses the resource impact of each allocation operation after a migration, including its effect on task waiting status in the lower-level semaphore pool, the trend of resource vacancy rate changes after allocation, and the effect of upper-level response latency repair. The scheduler dynamically adjusts the allocation strategy for the next operation based on the assessment results, such as limiting the maximum number of allocations per cycle, introducing an allocation cooldown period, or adopting a weighted migration strategy. Furthermore, it uses a sliding window calculation method to determine the stability of the allocation behavior. If repeated migration rollbacks or upper-level saturation are detected after allocation, the issue is reported to the task scheduling strategy optimization stage to promote the formation of a better resource allocation structure and achieve coordinated evolution between system layers.

[0041] Finally, all token migration actions executed by the scheduler, along with their corresponding pre- and post-allocation status information, will be fully recorded and written to the resource scheduling behavior log for subsequent task scheduling tracking, anomaly analysis, and archiving in the evolutionary knowledge base. This log includes, but is not limited to, key fields such as the token's unique ID, original pool level, migration timestamp, lease configuration (new and old), allocation reason code, and affected task identifier. This method enables full traceability of resource allocation actions and gradually accumulates cross-layer allocation behavior data during system operation, providing data support for the future construction of more refined semaphore lease allocation, dynamic adjustment of layer capacity, and task priority reallocation mechanisms.

[0042] Through the above methods, flexible resource scheduling based on the hierarchical semaphore pool structure is realized, which effectively alleviates the resource bottleneck problem faced by high-priority tasks, improves the overall resource utilization efficiency and the response consistency of processing tasks, and further enhances the stable operation capability and resource adaptability of the multi-dimensional data processing system in high-concurrency and high-dynamic environments.

[0043] A self-healing monitoring thread is embedded in the adaptive scheduler. The self-healing monitoring thread is used to periodically detect semaphore lease timeout events and token usage imbalances at each level. When abnormal resource usage is detected, the corresponding task rollback mechanism is automatically triggered to reconstruct the directed resource dependency graph and regenerate the sequential execution plan of the task to restore the resource balance of the system. The specific implementation method for this step is as follows: To enhance the stability of resource scheduling and the overall fault tolerance of the system, a self-healing monitoring thread is introduced based on the aforementioned adaptive scheduling strategy. This thread is tightly coupled with the resource status of the semaphore pool and periodically scans the token usage status and lease fulfillment status of each layer of the semaphore pool. Within each monitoring cycle, the self-healing monitoring thread reads the remaining token lease time, semaphore return timestamp, and task activity flags for all running or suspended tasks in the current system. Based on the set lease threshold and resource return window, it determines whether a lease overdue event has occurred. If a task fails to release its semaphore after the lease expires, and the task is in an abnormal suspended, blocked, or invalid loop state, it is determined to be a resource usage failure event. The event occurrence point is recorded, and preparations are made to enter the fault handling process.

[0044] Upon detecting abnormal resource usage or semaphore retention, the self-healing monitoring thread initiates a task rollback mechanism. This involves cleaning up the execution context of the task currently holding the abnormal semaphore, including terminating the task thread, forcibly releasing the token it holds, reversing any pending data write operations, and removing the task from the current sequential execution plan to prevent it from interfering with subsequent tasks. During the rollback, the self-healing monitoring thread records the path of the task that caused the anomaly and its related dependent nodes, marking the edges connected to it in the original directed resource dependency graph as broken, indicating that subsequent graph structure update logic needs to recalculate this part. This step ensures that the system can quickly cut off the risk propagation chain in the event of local resource imbalance, laying the foundation for the system to return to stable operation.

[0045] Based on the aforementioned rollback markers, the self-healing monitor thread will immediately trigger the resource dependency graph reconstruction process. This process, based on the latest task status, token holding status, and semaphore pool hierarchy, re-establishes the resource request relationships, call order, and semaphore path mappings between task nodes, and updates the weight values ​​and direction identifiers of all edges, forming a repaired directed resource dependency graph that removes abnormal dependency paths. After the graph is constructed, combined with the existing resource status and task execution priorities, the self-healing monitor thread uses a topological sorting algorithm to regenerate a sequential execution plan, retaining completed tasks and prioritizing the scheduling of incomplete and non-conflicting tasks, ensuring minimal disruption to resource scheduling. This new plan will directly replace the original plan and become the basis for the scheduler's next cycle scheduling, achieving a "hot repair" of the entire task graph structure.

[0046] Finally, to ensure the traceability and optimization capabilities of the self-healing process, each self-healing action is fully recorded and written to the resource evolution log, including the task exception type, trigger time, rollback scope, graph structure change details, and planned update version number. After each round of repair, the self-healing monitoring thread will also reassess the resource utilization of each layer of semaphore pools to verify whether the current scheduling structure has reached a new equilibrium. If an imbalance in utilization is still found, the next round of optimized scheduling can be triggered by migrating tokens again or adjusting task priorities.

[0047] By relying on the self-healing monitoring thread embedded in the scheduler, the system can quickly detect abnormal resource states, actively intervene in task execution imbalances, and dynamically repair the overall system scheduling structure. This effectively maintains the resource scheduling balance and execution path continuity in a high-concurrency asynchronous task environment, and significantly improves the system's self-stabilization capability and resource adaptation efficiency in dynamic environments for multi-dimensional data processing tasks.

[0048] Enable runtime process tracing recording, record the process of each resource dependency graph reconstruction and task rollback decision execution and write it into the evolutionary knowledge base. The evolutionary knowledge base serves as historical reference data for the next round of static dependency analysis, forming a continuous closed-loop optimization mechanism of analysis-scheduling-monitoring-feedback. The specific implementation method for this step is as follows: After the aforementioned self-healing monitoring thread completes resource anomaly detection and task rollback processing, a runtime process tracing and recording mechanism is immediately activated to perform real-time data collection and behavior tracking of the entire process involved in this scheduling anomaly event. The collected data includes the initial structure of the resource dependency graph, the semaphore lease identifier that triggered the anomaly, the state transition trajectory of the corresponding task during execution, and the scope of the resulting resource imbalance. By accurately recording dependency chain path changes and abnormal resource lease behavior, the entire causal path of system state transitions can be reconstructed, providing a high-fidelity data foundation for subsequent behavior evaluation and scheduling strategy improvement. This tracing record uses the event timeline as an index to establish a multi-dimensional structured data view, ensuring the accurate reproduction of complex behavioral chains.

[0049] After the resource dependency graph is reconstructed and a new sequential execution plan is generated, all aspects of this graph structure change—including node adjustments, edge weight modifications, path switching, and rollback task mappings—are versioned and archived into an event archive set, bound to their corresponding rollback operation records. The execution process of each rollback decision, including trigger time, rollback scope, reclaimed token number, task state reset behavior, and subsequent dependency graph update commands, is solidified into parsable record units using standardized semantic modeling. This storage behavior not only achieves structured historical management of the graph structure evolution process but also provides clear data traceability for anomaly scheduling, enabling each system self-healing behavior to be auditable, comparable, and replayed.

[0050] To enable the extraction of generalizable experiences from historical events, the recorded resource dependency graph change data and task behavior data are input into an evolutionary knowledge extraction engine. Typical patterns are then clustered, attributed, and labeled. This engine analyzes recurring rollback factors, resource contention paths, and lease expiration trends across multiple cycles to generate a risk labeling table and rule suggestion set for subsequent static dependency analysis phases. For example, if a certain structure repeatedly appears in the task blocking graph, it is pre-marked as a "high-risk blocking path" in the next dependency graph construction phase to avoid it; or if a certain task combination frequently induces token retention, the knowledge base can provide suggestions for pre-scheduling order or adjustment of its semaphore lease configuration. Through knowledge evolution extraction, not only is the original operational data preserved, but it is also transformed into structural intelligent factors that can participate in future scheduling decisions.

[0051] In the next round of static dependency profiling, the evolutionary knowledge base is used as one of the core input sources. Combined with the current set of tasks to be executed, the resource pool status, and historical behavior models, the resulting directed resource dependency graph undergoes pre-correction and risk pruning operations. In this step, the static analysis logic prioritizes high-risk nodes and edges marked in the knowledge base, employing structural optimization strategies such as reducing topology nesting depth, delaying scheduling order, or imposing resource isolation constraints. Simultaneously, path scores, resource scheduling strategies, and lease allocation parameters in the profiling results can be directly inherited from optimization suggestions in the knowledge base, thereby achieving intelligent pre-scheduling planning driven by historical experience. Through this process, the resource behavior evolution data generated during operation is completely fed back to the analysis source, constructing a closed-loop system of profiling-scheduling-monitoring-feedback. This enables dynamic optimization of resource management, continuous evolution of scheduling paths, and adaptive adjustment of task execution behavior, fundamentally improving the resilience and efficiency of multi-dimensional data processing workflows in complex task scenarios.

[0052] This invention, by introducing static dependency analysis and a directed resource dependency graph construction mechanism, can comprehensively identify resource occupation paths and recursive call relationships that may cause deadlocks before the actual execution of tasks. Utilizing depth-first loop detection and acyclic subgraph partitioning techniques, potentially conflicting tasks can be isolated in advance, and a controllable execution plan can be generated according to the logical dependency order, eliminating the risk of resource interlocking at the source. Furthermore, this invention combines a token lease mechanism and an automatic timeout callback strategy to ensure that semaphore resources are reclaimed on schedule even if a task is blocked or abnormally terminated during execution, avoiding prolonged occupation or resource suspension. Through these proactive prevention and runtime self-recovery measures, the multi-task concurrent scheduling process becomes more controllable and robust, fundamentally improving the stability of the overall scheduling architecture and effectively overcoming the frequent deadlock and recursion problems in traditional asynchronous concurrent execution.

[0053] This invention constructs a hierarchical semaphore pool architecture and introduces a priority-aware adaptive scheduling mechanism and a self-healing monitoring thread with fault recovery capabilities. This allows semaphore resources to be dynamically managed hierarchically according to task importance, and enables cross-layer allocation of idle tokens under high load conditions, ensuring resource availability for high-priority tasks. When resource allocation anomalies occur, the self-healing mechanism can proactively initiate task rollback and dependency graph reconstruction to promptly restore the balance of resource allocation. Furthermore, this invention records the dependency evolution process and scheduling behavior results in an evolutionary knowledge base through runtime process tracing, providing experience-based data support in the subsequent static analysis phase and enabling continuous optimization of the scheduling strategy. By constructing a complete closed loop of task analysis, resource allocation, execution monitoring, and intelligent feedback, this invention significantly improves resource utilization efficiency and the intelligence level of task scheduling, making it particularly suitable for multi-dimensional data processing scenarios with large data volumes, high concurrency, and strict requirements for response timeliness.

[0054] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multi-dimensional data intelligent processing and analysis method, characterized in that, Includes the following steps: Static dependency analysis is performed on the set of parallel tasks to construct a directed resource dependency graph that includes semaphore occupancy relationships, call hierarchy, and recursion paths; Based on the directed resource dependency graph, depth-first loop detection is performed to identify potential deadlock loops and divide them into multiple edge-acyclic subgraphs, generating sequential execution plans for each subgraph; In the sequential execution plan, a semaphore token lease period with an automatic return mechanism upon expiration is allocated to each asynchronous task, and lease timeout callback logic is configured to actively release the semaphore in abnormal or blocking states. A multi-level semaphore pool is constructed based on the token lease mechanism, and high-priority tasks are mapped to the upper-level pool and low-priority tasks are mapped to the lower-level pool according to task priority. An adaptive scheduler is deployed on the basis of the semaphore pool to continuously monitor the token allocation and recycling status of each layer. When the upper layer pool is saturated, unused tokens from the lower layer pool are promoted to the upper layer to achieve dynamic resource replenishment. A self-healing monitoring thread is embedded in the scheduler to periodically check for lease timeouts and token usage imbalances. If an anomaly is detected, task rollback, dependency graph reconstruction, and regeneration of the sequential execution plan are triggered to restore resource balance. The process of reconstructing the directed resource dependency graph and rolling back the task is recorded as runtime traceability data and written into the evolutionary knowledge base. The knowledge base serves as a historical reference for subsequent static dependency analysis, forming a closed-loop optimization mechanism of analysis-scheduling-monitoring-feedback.

2. The multi-dimensional data intelligent processing and analysis method according to claim 1, characterized in that, The steps for static dependency profiling of a set of parallel tasks include: Extract the input data source, processing logic, semaphore acquisition and release information, and call path for each task to be executed, and construct the initial mapping relationship between tasks and semaphores; Based on the mapping relationship, a task resource occupancy order chain is established, and an initial directed graph is constructed with tasks as nodes and resource dependencies as edges. Perform depth-first traversal on the directed graph, using Tarjan's algorithm, Kosaraju's algorithm, or edge back detection to identify recursive calls and closed-loop dependency paths, and assign weights to edges in the graph based on task priority and execution time. The constructed dependency graph is structurally optimized and tasks are classified. The set of tasks without dependencies is extracted, the recursive subgraph is archived, and a resource request and release record table is generated for subsequent scheduling analysis.

3. The multi-dimensional data intelligent processing and analysis method according to claim 2, characterized in that, The steps for performing loop detection and generating a sequential execution plan based on a directed resource dependency graph include: Perform a depth-first traversal in the dependency graph to identify dependency loops and record semaphore numbers, task priorities, and path weights. The closed loop is divided into multiple edge-free subgraphs based on the breakage of the boundary nodes, and resource conflict information at the breakage points is recorded. Perform a topological sort on each acyclic subgraph and generate a sequential execution plan based on task dependency order and priority; The execution plan is mapped to the original dependency graph and persisted as the basic input for subsequent scheduling phases.

4. The multi-dimensional data intelligent processing and analysis method according to claim 1, characterized in that, The steps for allocating semaphore token leases in a sequential execution plan include: Set up an independent resource entry point for each asynchronous task and allocate a semaphore token lease with a fixed term. Define the token usage time limit. After the lease is set, inject a lease period monitoring mechanism to detect the task status and the remaining time of the token in real time. If the task is blocked or abnormal, trigger the lease timeout callback logic to force the release of the semaphore. By combining the task execution context and resource usage status, the lease period is dynamically adjusted to prevent abnormal interruptions or resource waste. The entire lease process is logged as a reference for the next round of resource allocation strategy optimization and task behavior evaluation.

5. The multi-dimensional data intelligent processing and analysis method according to claim 4, characterized in that, The steps for constructing a multi-level semaphore pool based on a token lease mechanism include: Based on task priority, the tasks to be scheduled are divided into high-priority tasks and low-priority tasks, and mapped to the upper-layer and lower-layer semaphore pools respectively. Semaphore resources are allocated to semaphore pools of different levels according to their weights. When a task is bound to a lease, it is added to the queue to be allocated at the corresponding level. When resources in the upper-level semaphore pool are scarce, reclaimable tokens are temporarily allocated from the lower-level pool, and the token ownership identifier and lease information are updated.

6. The multi-dimensional data intelligent processing and analysis method according to claim 1, characterized in that, The steps for deploying an adaptive scheduler based on a semaphore pool include: Periodically monitor the token allocation status, recycling status, and task waiting time in each layer of the semaphore pool to determine if upper-layer resources are saturated. When upper-layer resources are detected to be scarce, select idle tokens that are not bound to tasks from the lower-layer semaphore pool for cross-layer migration and update their layer labels and lease time configurations. After migration, the impact of the allocation behavior on the resource utilization and task response of the upper and lower layers is evaluated, the allocation strategy is dynamically adjusted, and the migration behavior and resource status changes are recorded in the resource scheduling log as the basis for subsequent strategy optimization and evolution analysis.

7. The multi-dimensional data intelligent processing and analysis method according to claim 6, characterized in that, The steps for embedding a self-healing monitor thread in the scheduler include: Periodically check the lease status and task activity of tokens in each layer of the semaphore pool to identify events such as overdue leases or unbalanced resource usage. Upon detecting an anomaly, a task rollback operation is performed, releasing the occupied tokens and updating the state of the relevant edges in the directed resource dependency graph to "damaged". The dependency graph is reconstructed based on the updated resource status, and a new sequential execution plan is generated to replace the original execution path.

8. The multi-dimensional data intelligent processing and analysis method according to claim 1, characterized in that, The steps for recording the directed resource dependency graph reconstruction and task rollback process as runtime traceability data include: After the task rollback is completed, collect dependency graph changes, abnormal semaphores and task state trajectories, and establish a timeline-based behavior record; Archive the graph structure adjustment content and rollback decision process in a versioned manner and bind them to storage as resolvable record units; The records are input into the evolutionary knowledge base for pattern clustering and risk labeling, and scheduling optimization suggestions are extracted for subsequent dependency analysis. In the next round of static dependency analysis, knowledge base content is referenced to correct the graph structure and optimize the task path, thus achieving closed-loop optimization.

Images