Task processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a task processing method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining a plurality of first tasks to be processed, task states of the first tasks and memories needed for processing the first tasks, wherein the task states are used for indicating stages where the first tasks are located; according to the plurality of task states, determining a memory resource pool corresponding to each first task, the memory resource pool being used for processing the first tasks with the same task state, and the memory resource pool comprising a memory unit for providing and executing the first tasks; determining a target memory unit in a plurality of memory units of a memory resource pool corresponding to the first task according to the memory corresponding to the first task, and processing the first task according to the target memory unit; after the first task is processed, the data in the target memory unit is emptied, and the emptied target memory unit is recycled into the corresponding memory resource pool, so that the task processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a task processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of large-scale computing clusters and distributed task processing platforms, task scheduling systems, as core components for coordinating and managing massive computing resources, determine the computing efficiency of the entire cluster through their scheduling performance. In practical applications, task scheduling systems are needed to manage the memory of tasks.

[0003] In related technologies, memory management typically employs an immediate allocation and release approach. This means that before processing a task, the corresponding memory unit needs to be allocated based on the memory required by the task, and the memory unit is released after the task is completed.

[0004] However, in the above process, when dealing with high-concurrency tasks, it is necessary to continuously and intensively allocate and release memory units, which prolongs the task processing chain and results in low task processing efficiency. Summary of the Invention

[0005] This application provides a task processing method, apparatus, electronic device, storage medium, and program product to solve the technical problem of low task processing efficiency.

[0006] Firstly, this application provides a task processing method, including:

[0007] Get multiple pending first tasks, the task status of each first task, and the memory required to process the first task. The task status is used to indicate the stage of the first task.

[0008] Based on multiple task states, a memory resource pool is determined for each first task. The memory resource pool is used to process first tasks with the same task state. The memory resource pool includes memory units required to execute the first task.

[0009] Based on the memory corresponding to the first task, determine the target memory unit from among the multiple memory units in the memory resource pool corresponding to the first task, and process the first task based on the target memory unit;

[0010] After the first task is completed, the data in the target memory unit is cleared, and the cleared target memory unit is recycled to the corresponding memory resource pool.

[0011] In this embodiment, a corresponding memory resource pool is determined based on the task status, and a target memory unit is identified within that pool to process the task. After the task is completed, the target memory unit is cleared and reclaimed into the memory resource pool. In this method, the terminal device can pre-divide the memory resource pool according to the task status, enabling dedicated memory area management for tasks in different states. It can also quickly match or allocate memory units based on the memory required by the task, effectively avoiding the system overhead of frequently requesting and releasing memory for each task. It can quickly obtain the memory space required for task processing without relying on the dynamic memory allocation of the underlying operating system, reducing the number of system calls and improving task processing efficiency. Furthermore, by clearing and reclaiming memory units to the corresponding resource pool, the circular reuse of memory resources is achieved, significantly reducing memory fragmentation. This method is applicable to high-concurrency, short-lifecycle task processing scenarios, improving overall throughput of task scheduling while increasing memory resource utilization.

[0012] Optionally, the method described above, based on the memory corresponding to the first task, determines the target memory unit from multiple memory units in the memory resource pool corresponding to the first task, including:

[0013] Determine the status of multiple memory units in the memory resource pool corresponding to the first task. The status can be available or unavailable.

[0014] Among the multiple memory units that are in an available state, determine the target memory unit that corresponds to the memory unit of the first task.

[0015] In this embodiment of the application, the accuracy of target memory unit determination and allocation efficiency are improved by filtering and matching based on the availability status of memory units.

[0016] Optionally, the method described above, in which the target memory unit corresponding to the memory corresponding to the first task is determined from among multiple memory units in an available state, includes:

[0017] Among the multiple memory units in the available state, determine whether there is a memory unit that matches the first task. The memory allocated to the matching memory unit is the same as the memory corresponding to the first task.

[0018] When it is determined that there is a memory cell that matches the first task, the matching memory cell is determined as the target memory cell;

[0019] When it is determined that there is any memory unit that does not match the first task, based on the memory of the first task, a memory unit that is the same as the memory of the first task is allocated in the memory resource pool and determined as the target memory unit.

[0020] In this embodiment of the application, a mechanism of priority matching rather than direct partitioning is used to optimize memory resource utilization and reduce unnecessary memory allocation operations.

[0021] Optionally, as described above, based on the memory of the first task, a memory unit identical to the memory of the first task is allocated from the memory resource pool, including:

[0022] Based on the memory of the first task, determine the memory units that need to be partitioned;

[0023] Within the memory resource pool, determine the starting and ending address locations of the memory.

[0024] Based on the start and end address locations, determine the memory unit that is the same as the memory of the first task.

[0025] In this embodiment, by accurately locating the start and end addresses of the memory, the size of the divided memory units is ensured to be accurate, thereby improving the accuracy of memory partitioning.

[0026] Optionally, the method described above applies to any task state; before determining the memory resource pool corresponding to each first task based on multiple task states, the method further includes:

[0027] Determine the memory resource pool corresponding to the task status;

[0028] Get the preset memory corresponding to the task status;

[0029] Based on the preset memory, create a preset number of memory units in the memory resource pool corresponding to the task state.

[0030] In this embodiment of the application, by pre-creating a preset number of memory units, the real-time allocation overhead during task processing is reduced, and the initial response speed of the system is improved.

[0031] Optionally, before obtaining multiple pending first tasks, the task status of each first task, and the memory required to process the first task, as described above, the following steps are included:

[0032] Receive a task processing request, which includes the first task to be processed, the task status of each first task, and the memory required to process the first task.

[0033] The task processing request is encapsulated to obtain the encapsulated task processing request.

[0034] Send a packaged task processing request to the asynchronous processing thread.

[0035] In this embodiment of the application, by encapsulating and asynchronously processing task requests, the system's responsiveness to high-concurrency requests and overall throughput are improved.

[0036] Optionally, the above method reclaims the cleared target memory units into the corresponding memory resource pool, including:

[0037] Determine the reclamation status of the target memory unit after it has been cleared. The reclamation status can be either reclaimable or non-reclaimable.

[0038] When the recycling status is recyclable, the cleared target memory unit will be recycled to the corresponding memory resource pool;

[0039] When the target memory unit is in an unrecoverable state, the cleared target memory unit is released, a memory unit corresponding to the target memory unit is allocated in the memory resource pool, and the memory unit is marked as available.

[0040] In this embodiment of the application, by judging the memory cell reclamation status and adopting different reclamation strategies, the resource pool capacity is kept stable, and the health and availability of the memory resource pool are ensured.

[0041] Secondly, this application provides a task processing apparatus, comprising:

[0042] The acquisition module is used to acquire multiple first tasks to be processed, the task status of each first task, and the memory required to process the first task. The task status is used to indicate the stage of the first task.

[0043] The first determining module is used to determine the memory resource pool corresponding to each first task based on multiple task states. The memory resource pool is used to process the first tasks with the same task state. The memory resource pool includes memory units for providing the memory units required to execute the first task.

[0044] The second determining module is used to determine the target memory unit from multiple memory units in the memory resource pool corresponding to the first task based on the memory corresponding to the first task, and to process the first task based on the target memory unit.

[0045] The processing module is used to clear the data in the target memory unit after the first task is completed, and to reclaim the cleared target memory unit into the corresponding memory resource pool.

[0046] Optionally, in the apparatus described above, based on the memory corresponding to the first task, a target memory unit is determined from multiple memory units in the memory resource pool corresponding to the first task, and the second determining module is specifically used for:

[0047] Determine the status of multiple memory units in the memory resource pool corresponding to the first task. The status can be available or unavailable.

[0048] Among the multiple memory units that are in an available state, determine the target memory unit that corresponds to the memory unit of the first task.

[0049] Optionally, in the apparatus described above, among a plurality of memory units in an available state, the target memory unit corresponding to the memory corresponding to the first task is determined, and the second determining module is specifically used for:

[0050] Among the multiple memory units in the available state, determine whether there is a memory unit that matches the first task. The memory allocated to the matching memory unit is the same as the memory corresponding to the first task.

[0051] When it is determined that there is a memory cell that matches the first task, the matching memory cell is determined as the target memory cell;

[0052] When it is determined that there is any memory unit that does not match the first task, based on the memory of the first task, a memory unit that is the same as the memory of the first task is allocated in the memory resource pool and determined as the target memory unit.

[0053] Optionally, in the apparatus described above, based on the memory of the first task, a memory unit identical to the memory of the first task is allocated from the memory resource pool, and the processing module is specifically used for:

[0054] Based on the memory of the first task, determine the memory units that need to be partitioned;

[0055] Within the memory resource pool, determine the starting and ending address locations of the memory.

[0056] Based on the start and end address locations, determine the memory unit that is the same as the memory of the first task.

[0057] Optionally, in the apparatus described above, for any given task state; before determining the memory resource pool corresponding to each first task based on multiple task states, the apparatus further includes a creation module, which is specifically used for:

[0058] Determine the memory resource pool corresponding to the task status;

[0059] Get the preset memory corresponding to the task status;

[0060] Based on the preset memory, create a preset number of memory units in the memory resource pool corresponding to the task state.

[0061] Optionally, in the apparatus described above, before acquiring multiple first tasks to be processed, the task status of each first task, and the memory required to process the first task, the apparatus further includes a receiving module, which is specifically used for:

[0062] Receive a task processing request, which includes the first task to be processed, the task status of each first task, and the memory required to process the first task.

[0063] The task processing request is encapsulated to obtain the encapsulated task processing request.

[0064] Send a packaged task processing request to the asynchronous processing thread.

[0065] Optionally, in the apparatus described above, the cleared target memory units are recycled back to the corresponding memory resource pool, and the processing module is specifically used for:

[0066] Determine the reclamation status of the target memory unit after it has been cleared. The reclamation status can be either reclaimable or non-reclaimable.

[0067] When the recycling status is recyclable, the cleared target memory unit will be recycled to the corresponding memory resource pool;

[0068] When the target memory unit is in an unrecoverable state, the cleared target memory unit is released, a memory unit corresponding to the target memory unit is allocated in the memory resource pool, and the memory unit is marked as available.

[0069] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0070] The memory stores instructions that the computer executes;

[0071] The processor executes computer-executable instructions stored in memory to implement the method described in the first aspect.

[0072] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a computer, are used to implement the method described in the first aspect.

[0073] The computer-readable storage medium provided in this application embodiment can execute the technical solutions in the above method embodiments, and its beneficial effects are similar, so they will not be described again here.

[0074] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a computer, is used to implement the method of the first aspect.

[0075] The computer program product provided in this application embodiment can execute the technical solutions in the above method embodiments, and its beneficial effects are similar, so they will not be described again here.

[0076] The task processing method, apparatus, electronic device, storage medium, and program product provided in this application determine a corresponding memory resource pool based on the task state, identify a target memory unit within that pool to process the task, and clear and reclaim the target memory unit from the memory resource pool after the task is completed. In this method, the terminal device can pre-divide the memory resource pool according to the task state, enabling dedicated memory area management for tasks in different states. It can also quickly match or allocate memory units based on the memory required by the task, effectively avoiding the system overhead of frequently requesting and releasing memory for each task. It can quickly obtain the memory space required for task processing without relying on the dynamic memory allocation of the underlying operating system, reducing the number of system calls and improving task processing efficiency. Furthermore, by clearing and reclaiming memory units to the corresponding resource pool, it achieves cyclic reuse of memory resources, significantly reducing memory fragmentation. This method is applicable to high-concurrency, short-lifecycle task processing scenarios, improving overall throughput of task scheduling while increasing memory resource utilization. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0078] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application;

[0079] Figure 2 A flowchart illustrating a task processing method provided in an embodiment of this application;

[0080] Figure 3 A flowchart illustrating the parallel processing method provided in an embodiment of this application;

[0081] Figure 4 A flowchart illustrating the asynchronous thread processing method provided in an embodiment of this application;

[0082] Figure 5 A schematic diagram illustrating the memory allocation and request process provided in this application embodiment;

[0083] Figure 6 A schematic diagram illustrating a method for reclaiming memory cells according to an embodiment of this application;

[0084] Figure 7 A schematic diagram illustrating the asynchronous cleanup process of memory units provided in an embodiment of this application;

[0085] Figure 8 A flowchart illustrating another task processing method provided in an embodiment of this application;

[0086] Figure 9This is a schematic diagram of the structure of a task processing device provided in an embodiment of this application;

[0087] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0088] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0089] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0090] In high-concurrency task scheduling scenarios, the performance limitations of traditional memory management methods are becoming increasingly apparent in large-scale computing clusters. Traditional dynamic memory management is characterized by on-demand allocation, zero memory reuse, and immediate return of memory units to the system after release. This typically leads to frequent system calls and significant overhead on the Central Processing Unit (CPU). This places higher demands on the processing efficiency of task scheduling, as memory operation latency directly affects task response time, and frequent system calls consume a large amount of computing resources that could otherwise be used for task scheduling.

[0091] In related technologies, task memory is typically managed using an immediate allocation and release method, which means that memory units need to be allocated separately for each received task and released back to the system immediately after the task is processed.

[0092] However, the above process requires a complete memory allocation and deallocation process to be executed separately for each task, and it relies on the underlying operating system's memory allocator. Although the memory unit specifications of different tasks are the same or similar, due to the lack of a reuse mechanism, relying solely on the system's default memory management will generate a large number of unnecessary system call overheads and memory fragmentation problems.

[0093] For example, the technical problem of traditional task memory management can be understood as the inability to establish an effective memory reuse mechanism in high-concurrency, short-cycle task scenarios. For instance, it is impossible to reuse memory units of the same specifications to reduce allocation overhead.

[0094] In the process of task memory management, it is usually necessary to allocate memory in real time based on task status and memory requirements. Specifically, this can be achieved by calling the memory allocation and deallocation system interfaces independently for each task.

[0095] However, traditional memory management methods have inherent flaws: First, each allocation and release is an independent operation, lacking batch and pooling management mechanisms; second, released memory is immediately returned to the system, making reuse at the application layer impossible; and finally, a unified memory allocator cannot be optimized for specific memory unit specifications. These factors make it difficult to efficiently manage massive amounts of task memory, resulting in difficulties in guaranteeing task processing efficiency and system throughput.

[0096] In addition, memory lifecycle needs to be managed independently for each task. Since the memory demand pattern of tasks cannot be predicted and the fixed allocation and release process cannot adapt to the dynamic load of high-concurrency scenarios, the overhead and fragmentation problem of memory management are significant.

[0097] Alternatively, it may rely on the operating system's general memory allocator, but what can be observed is the independent operation of each allocation and release. The reuse of memory units of the same size between consecutive tasks and the overall memory usage efficiency are difficult to optimize, and often require sacrificing processing performance to ensure the correctness of memory management.

[0098] Therefore, this application provides a task processing method that allocates a dedicated memory resource pool according to the task status, and creates memory units in batches within the pool for reuse. Through status matching and asynchronous recycling mechanisms, it achieves efficient management of task memory. By combining memory resource pooling, memory unit reuse, and asynchronous processing, the task processing logic and memory management overhead are effectively separated, thereby achieving high throughput and low latency in task scheduling. This method can adapt to task processing needs of different concurrency scales, avoiding performance degradation caused by frequent system calls, effectively improving memory resource utilization, and significantly enhancing task processing efficiency and overall system performance.

[0099] Figure 1 For a schematic diagram of the system architecture provided in this application embodiment, please refer to [link / reference]. Figure 1 This includes electronic devices and terminal devices. Electronic devices can be any device with on-device computing capabilities, such as servers and terminal devices.

[0100] The terminal device and the electronic device can communicate with each other. The terminal device can send tasks to be processed to the electronic device, and the electronic device can perform memory processing and other operations on the received tasks to obtain the processing results. The electronic device and the terminal device can be the same device, meaning it can acquire and process tasks during its own operation.

[0101] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0102] Figure 2 This is a flowchart illustrating a task processing method provided in an embodiment of this application. Specifically, as shown... Figure 2 As shown, the method includes the following steps:

[0103] S201. Obtain multiple pending first tasks, the task status of each first task, and the memory required to process the first task.

[0104] The execution subject of this application embodiment can be an electronic device or a task processing device installed in an electronic device. The task processing device can be implemented by software or by a combination of software and hardware.

[0105] The task status is used to indicate the stage of the first task, that is, the task status indicates the type of task to be processed, and the task status can refer to the transition stage of the first task in its lifecycle. In the embodiments of this application, the task status may include a pending submission status, a pending execution status, and a completed status.

[0106] The pending submission status can be the first task being in a state of waiting to be submitted.

[0107] The pending status indicates that the first task is in a state of being submitted and awaiting processing.

[0108] The completion status is used to indicate that the first task has been completed.

[0109] The first task can be a task of the user job in its life cycle (e.g., submission, running, etc.). For example, the first task can be a job submitted by the user, a job that has been successfully submitted by the user and is waiting to be processed, or a job that has been successfully submitted by the user and has been processed.

[0110] The memory required to process the first task can be the specification of the memory unit that holds the data of the first task; for example, the memory required to process the first task can be 1MB.

[0111] In some embodiments, to fundamentally alleviate the pseudo-parallelism caused by global resource lock contention in traditional architectures, horizontal decoupling of module logic can be used to restrict access to shared resources to their respective thread pools, thus avoiding global lock contention. For example, the job thread pool only accesses the job queue, and the node thread pool only accesses the node status table, thereby achieving fine-grained parallel processing and improving system throughput.

[0112] Specifically, the processing objects can be asynchronous messages from the job submission client, node monitoring system, and resource management module (e.g., job status updates, node heartbeat synchronization, partition resource judgment, etc.). Based on the message type (job-related, node-related, partition-related), messages are assigned to independent message processing channels, and each channel is bound to a dedicated thread pool. In this way, job-related messages, node status messages, and partition management messages are routed to their respective dedicated processing channels and processed in parallel by their respective thread pools. For example, when a job status update message is received, it can be assigned to the job thread pool; when a node heartbeat message is received, it can be assigned to the node thread pool. The job thread pool handles job scheduling logic, while the node thread pool updates the node status table, without interference. Here, a thread pool refers to a pre-created collection of threads used for efficient reuse of thread resources.

[0113] The above approach ensures that the processing flow of different types of tasks is physically isolated, and each of them operates on its own independent data structure, thereby completely eliminating lock contention across business types. This allows for full utilization of multi-core CPU resources and significantly improves the overall throughput in high-concurrency scenarios.

[0114] In some embodiments, while achieving horizontal parallel processing, it is necessary to ensure the vertical sequentiality of core state transitions within the lifecycle of a single job. Specifically, for a specific job (the first task), its lifecycle includes key stages such as submission, scheduling, startup, execution, and termination, which constitute a state machine for sequential execution.

[0115] Electronic devices can execute this sequence using an event-driven mechanism. This means that after each processing stage is completed, it doesn't directly trigger the next stage. Instead, it generates a specific event, which is submitted to the corresponding processing channel queue. Subsequent threads retrieve and execute these events sequentially. Furthermore, before the next stage begins, the completion status of the previous stage needs to be assessed. In this way, when processing massive amounts of different job messages, the internal state transitions of any given job always strictly follow a pre-defined lifecycle order. This avoids logical confusion and state errors that might result from concurrent execution, ensuring the correctness of scheduling semantics and the atomicity and consistency of critical operations.

[0116] Below, through Figure 3The process of optimizing message processing concurrency within each task state is explained in detail.

[0117] Figure 3 This is a flowchart illustrating the parallel processing method provided in an embodiment of this application. Please refer to... Figure 3 The figure uses the submission stage, start stage, and end stage (corresponding to the task status of pending submission, pending execution, and completed status respectively) as the horizontal division dimension. Taking three first tasks (Job1, Job2, and Job3) as examples, it is divided into upper and lower parts. The upper part of the figure shows the existing task processing method, and the lower part shows the optimized task processing method of this application embodiment.

[0118] Specifically, in the existing processing method, the order of each first task in each stage is sequential. For example, Job1 needs to complete all processing in its submission stage before Job2 can begin to enter the submission stage; after Job2 is completed, Job3 enters the submission stage. That is, multiple tasks in the same stage cannot be parallelized and need to queue up and wait for the previous task to complete.

[0119] The optimized task processing method in this application embodiment is parallel processing within a stage. That is, the first task is processed in parallel according to its current stage (e.g., submission stage, start stage, end stage). For example, tasks in the same stage (e.g., Job1, Job2, Job3) can be processed in parallel in the processing pipeline corresponding to that stage without waiting for each other to queue.

[0120] By grouping tasks in the same state into a dedicated resource pool for parallel processing, the problem of sequential queuing for tasks at the same stage in traditional methods is solved, significantly increasing the task processing volume (throughput) per unit time. Furthermore, it avoids global resource lock contention, significantly improving CPU utilization and system performance. It also shortens the overall task processing chain, with intra-stage parallelism directly compressing the completion time of batch tasks. The event-driven mechanism makes the flow between stages more efficient, thereby reducing end-to-end latency from task submission to completion and improving system response speed. Under this architecture, the system can fully leverage the advantages of multi-core processors to achieve parallel scheduling and processing of different message categories, laying the foundation for improving the efficiency of subsequent processes.

[0121] In some embodiments, the electronic device may, before obtaining a plurality of first tasks to be processed, the task status of each first task, and the memory required to process the first task, receive a task processing request, which includes the first tasks to be processed, the task status of each first task, and the memory required to process the first task, based on the following implementation: encapsulate the task processing request to obtain an encapsulated task processing request; and send the encapsulated task processing request to an asynchronous processing thread.

[0122] Specifically, to address the performance bottleneck caused by traditional synchronous memory operations (i.e., the serial process of allocation-use-release) when handling high-concurrency tasks, this application introduces an asynchronous memory management mechanism. The core of this mechanism lies in decoupling the lifecycle management of memory units (allocation and release) from the critical path of the main business processing thread, entrusting it to a dedicated asynchronous processing thread, thereby achieving decoupling and parallelism between computation and memory I / O. The specific process is as follows:

[0123] The scheduling server (electronic device) receives task processing requests from user job submission terminals or cluster computing nodes. These requests include the first task to be processed (e.g., a subtask in parallel computing, a job step, etc.), the current task status of each first task, and the memory unit specifications required for each task. To adapt to the unified parsing logic of asynchronous threads, the server performs standardized encapsulation processing on the request, integrating the task information, status identifiers, and memory requirements in the request into a structured data packet according to a preset format, ensuring that asynchronous threads can quickly identify key information. After encapsulation, the server directly sends the data packet to a pre-created asynchronous processing thread, which is responsible for subsequent request parsing and memory unit matching / allocation operations.

[0124] For example, after receiving a job submission request, the main thread only encapsulates the memory allocation task and submits it to the asynchronous thread queue. The background thread then performs the actual allocation, while the main thread continues processing subsequent logic. This significantly reduces the main thread's blocking time and improves the real-time performance of message processing. Simultaneously, the independent execution of asynchronous threads reduces the coupling between the main process and memory operations, enabling the system to maintain stable processing capabilities under high concurrency scenarios. Furthermore, asynchronous memory management optimizes resource scheduling, avoiding system performance fluctuations caused by memory operation delays and providing reliable guarantees for dynamic resource allocation for complex tasks.

[0125] Below, through Figure 4 This section provides a detailed explanation of the interaction process between the message processing thread and the background thread.

[0126] Figure 4 This is a flowchart illustrating the asynchronous thread processing method provided in an embodiment of this application. Please refer to... Figure 4 This includes message processing threads, shared containers between threads, and background threads. The message processing thread is the main thread of the scheduling server, responsible for the core business process of the task. The shared containers between threads are memory resource queues for different task states. For example, ListA corresponds to the memory unit queue for the pending submission state, ListB corresponds to the pending execution state, and ListC corresponds to the completed state. The background threads are asynchronous processing threads, responsible for non-core operations such as pre-creation and filling of memory units to avoid blocking the main thread.

[0127] Specifically, the message processing thread can obtain the first task to be processed and its task status (e.g., submitting a job). When a memory unit is needed to hold the task data, the thread does not directly call the system memory allocation operation. Instead, based on the task status, it directly obtains a pre-created, free memory unit from the corresponding shared container (e.g., ListA for the pending submission status) and proceeds with the subsequent process.

[0128] Background threads can pre-create and initialize blank memory units that conform to the specifications of each task's state based on configuration information (e.g., preset memory). They also monitor the status of each shared container in real time. If a memory unit is detected as insufficient in the memory resource pool, pre-created or reclaimed memory units are filled into the corresponding shared container (e.g., ListA). The main thread can continue processing subsequent messages without waiting for memory allocation to complete, significantly improving the real-time performance of message processing. Simultaneously, the independent execution of asynchronous threads reduces the coupling between the main process and memory operations, enabling the system to maintain stable processing capabilities under high concurrency scenarios.

[0129] S202. Based on the status of multiple tasks, determine the memory resource pool corresponding to each first task.

[0130] The memory resource pool is used to process the first task with the same task status, and the memory resource pool includes memory units required to execute the first task.

[0131] Memory units can be memory structures that are dynamically allocated and used during task execution, such as job description structures, node resource structures, and job statistics structures, used to store parameters, status, and context information required for task execution.

[0132] In some embodiments, the electronic device may be implemented as follows: for any task state; before determining the memory resource pool corresponding to each first task based on multiple task states, the method further includes: determining the memory resource pool corresponding to the task state; obtaining the preset memory corresponding to the task state; and creating a preset number of memory units in the memory resource pool corresponding to the task state based on the preset memory.

[0133] The default memory can be a pre-defined capacity specification for memory units, for example, the default memory can be 1MB.

[0134] The preset quantity can be the initial number of memory units to be created.

[0135] In some embodiments, the electronic device can pre-create memory units during the system initialization phase or in an idle state, i.e., before the scheduling server receives any processing requests for the first task. Specifically, this can be based on a pre-configured preset memory (i.e., a fixed size of the memory unit corresponding to each task state, for example, 1MB) and a preset quantity (i.e., the initial creation quantity of memory units of this specification, pre-calculated or empirically set, for example, 100). Independent memory regions can be allocated for each task state, and a corresponding number of formatted free memory units can be created in batches and linked to the free list of their respective resource pools. When subsequent first tasks (e.g., parallel computing subtasks) are submitted in a concentrated manner, there is no need to temporarily call the system's malloc interface to allocate memory; available memory units can be directly obtained from the resource pool, significantly shortening the initial processing latency of the task, especially suitable for scenarios such as computing clusters that need to cope with sudden large-scale task submissions.

[0136] Optionally, upon receiving the processing request for the first task, a large contiguous block of memory can be requested from the operating system at once, based on the preset memory unit specifications (preset memory) and initial capacity (preset quantity) of the current state, combined with the memory required by the first task. This block is then divided into multiple memory unit instances, initialized, and included in the resource pool for management. If, during processing, the idle memory units in a resource pool are about to be exhausted, a batch expansion is triggered again, creating a new batch of memory units to be added to the pool. This method uses memory on demand, reducing the initial memory overhead at system startup. It avoids memory resource idleness caused by creating too many memory units in advance, and can quickly replenish the resource pool capacity when task requests arrive, dynamically maintaining resource supply under actual load.

[0137] Optionally, during the processing of the first task, the specifications of the required memory units may change due to adjustments in task requirements. For example, the first task in the pending execution state may require an upgrade of the memory unit specification from 1MB to 2MB due to an expansion of the computational scope. In this embodiment, a model prediction method can be used, employing historical task specification change patterns and subsequent task submission volumes as training data to determine the pre-created memory unit requirements. Based on the prediction results, the target values ​​of the preset quantities for each resource pool are dynamically calculated and adjusted.

[0138] In this way, predictive batch creation of relevant resource pools can be carried out proactively and in advance before the predicted load peak arrives, replenishing a sufficient number of idle memory units and avoiding performance jitter caused by the temporary creation of memory units when traffic surges, thereby achieving a more stable and efficient memory supply guarantee.

[0139] S203. Based on the memory corresponding to the first task, determine the target memory unit from among the multiple memory units in the memory resource pool corresponding to the first task, and process the first task based on the target memory unit.

[0140] The target memory unit can be a specific memory unit instance selected and allocated from the corresponding memory resource pool to carry and process all the status information and intermediate data of the current first task. Specifically, the target memory unit can be a pre-created memory unit matched from the memory resource pool of the corresponding task status, which has the same memory specifications as the first task. It is a dedicated memory carrier to carry the task data and will be used to store task parameters, running status and other information.

[0141] In some embodiments, the electronic device may determine a target memory unit from multiple memory units in a memory resource pool corresponding to a first task based on the memory corresponding to the first task, according to the memory corresponding to the first task: determining the state of multiple memory units in the memory resource pool corresponding to the first task; and determining the target memory unit corresponding to the memory corresponding to the first task from multiple memory units in the available state.

[0142] The status can be either available or unavailable.

[0143] The availability status is used to indicate that the memory unit in the memory resource pool is not currently occupied by any task and can be directly used by the first task. This status is usually marked in the metadata of the memory unit and added to the availability queue of the memory resource pool.

[0144] The unavailable state indicates that the memory units in the memory resource pool have been occupied by other tasks or are in a state of waiting for reclamation / maintenance and cannot be used by the current first task; they are usually moved to the memory resource pool's occupancy queue.

[0145] In some embodiments, the electronic device may determine the target memory unit corresponding to the memory corresponding to the first task from among a plurality of memory units in an available state as follows: among the plurality of memory units in an available state, it is determined whether there is a memory unit that matches the first task, and the memory allocated to the matching memory unit is the same as the memory corresponding to the first task; when it is determined that there is a memory unit that matches the first task, the matching memory unit is determined as the target memory unit; when it is determined that there is any memory unit that does not match the first task, according to the memory of the first task, a memory unit that is the same as the memory of the first task is allocated in the memory resource pool, and it is determined as the target memory unit.

[0146] In some embodiments, the electronic device can filter out a queue of memory units in an available state from the memory resource pool corresponding to the first task; traverse each memory unit in the queue and read its pre-configured memory specifications (i.e., allocated memory capacity); compare the specifications with the memory required by the current first task one by one; if any memory unit has a memory specification that is completely consistent with the memory required by the first task, it is directly determined that there is a matching memory unit; if the specifications of all available memory units in the queue do not match the task memory, it is determined that there is no matching memory unit.

[0147] In the process of comparing the memory corresponding to the first task (i.e., the fixed memory size required by the task state) with the predefined specifications of the memory units in each available state in the resource pool, since memory units within the same resource pool have the same fixed size, the matching judgment can be logically simplified to checking whether any available memory unit exists in the memory resource pool. For example, if the memory resource pool includes 100 1MB memory units, and the first task requires 1MB of memory, then any 1MB memory unit can be selected directly from the memory resource pool.

[0148] In some embodiments, after determining that there is a memory unit that matches the first task, the matching memory unit is selected from the available queue and its status metadata is updated (from available to unavailable). At the same time, the memory unit is moved from the available queue of the resource pool to the occupied queue to avoid being called repeatedly by other tasks. The identification information (such as memory address and specifications) of the memory unit is bound to the current first task to complete the determination of the target memory unit, and task data can then be filled into the memory unit.

[0149] In some embodiments, the electronic device may, based on the memory of the first task, allocate memory units in the memory resource pool that are the same as the memory of the first task, according to the memory of the first task: determine the memory units to be allocated according to the memory of the first task; determine the start address position and end address position of the memory in the memory resource pool; and determine the memory units that are the same as the memory of the first task according to the start address position and end address position.

[0150] The starting address can be the starting position of the physical memory region corresponding to the new memory unit when dividing the memory resource pool. That is, the starting address can be the first available address of the free memory region of the resource pool, and it is necessary to ensure that the subsequent memory space is continuous and sufficient.

[0151] The end address can be the end position of the physical memory region corresponding to the memory unit when a new memory unit is partitioned in the memory resource pool. The end address can be calculated from the start address and the memory capacity required by the first task, and it marks the memory range boundary of the memory unit.

[0152] In some embodiments, when there are no memory units available in the memory resource pool (i.e., no matching items), new memory units need to be dynamically allocated. This situation typically occurs in two scenarios:

[0153] Scenario 1: In a high-concurrency scenario, tasks in a certain state arrive in a concentrated manner, causing all memory units pre-created for that state to become unavailable (i.e., they have been allocated to other tasks that are being processed), and the available capacity of the resource pool is temporarily exhausted.

[0154] Scenario 2: The same task state may correspond to multiple preset fixed specifications (for example, memory units of different sizes are subdivided according to job type in the pending execution state). The specific specifications required by the current task may not have any free instances in the resource pool. Although memory units of other specifications exist in the pool, they cannot be used because their sizes do not match.

[0155] The specific process is as follows: Based on the memory required by the current first task, determine the target capacity of the memory unit to be divided (the same as the memory required by the task); by detecting the free memory area of ​​the corresponding memory resource pool, confirm whether there is a continuous free space with a capacity not less than the target capacity (it needs to adapt to the cluster memory alignment rules); combine the historical load data of the task status to confirm the metadata format of the memory unit to be divided, in order to prepare for subsequent address division.

[0156] In some embodiments, a continuous free space is selected in the free memory region of the corresponding memory resource pool. The free space must have a capacity greater than or equal to the memory required by the task and meet the memory alignment requirements. The physical address corresponding to the first byte of the space is used as the starting address position. The ending address position is calculated by using the starting address position and the memory capacity required by the first task, thereby determining the memory address range of the new memory unit. This range ensures that the memory is continuous and fully matches the specifications required by the task.

[0157] In some embodiments, after determining the address range, the boundary of the contiguous memory region is marked in the memory resource pool. For example, the contiguous address range can be marked as allocated to prevent overlap during subsequent partitioning. Within the address range, the metadata of the memory unit is written according to a predefined layout. The memory unit is added to the available queue of the memory resource pool and bound to the current first task to complete the creation of a memory unit matching the specifications.

[0158] S204. After the first task is completed, clear the data in the target memory unit and reclaim the cleared target memory unit into the corresponding memory resource pool.

[0159] After parallelizing the message processing flow, improving the execution efficiency within each flow becomes crucial for further optimization. Analysis revealed that the scheduling server spends a significant amount of time allocating and releasing memory units when processing job or node messages.

[0160] In some embodiments, to reduce the system overhead caused by repeated memory allocation, this invention introduces a memory unit reuse strategy based on the asynchronous memory mechanism. In traditional designs, memory units are released immediately after use, and reallocated when the same type of message arrives again, which wastes time and causes memory fragmentation. To address this, a memory unit clearing and reclamation mechanism is adopted. After determining that the memory unit is no longer needed, it is not released directly, but a field-level clearing operation is performed, such as setting pointers to null, resetting states, and resetting counters to zero, before being placed in the memory resource pool for future reuse.

[0161] Below, through Figure 5 It provides a detailed explanation of the internal resource application and other execution processes at each stage.

[0162] Figure 5 This is a schematic diagram illustrating the memory allocation and request process provided in an embodiment of this application. Please refer to [link / reference]. Figure 5 The upper part of the figure shows the execution flow within the existing task phase, while the lower part shows the optimized execution flow of the embodiments of this application.

[0163] The existing execution flow within a task phase is as follows: each sub-step within a task (Job) independently performs resource allocation and release. Specifically, each sub-step within a Job (e.g., quota verification, cached data, etc.) and each sub-step within a job step (e.g., message authentication, resource selection, etc.) requires separate system memory allocation to obtain the necessary memory units. After the sub-step is completed, memory is released, node is released, and data is stored in the database. When the next sub-step starts, memory allocation, execution, and release must be repeated. However, the optimized execution flow in this embodiment does not require memory allocation and release every time.

[0164] In this scenario, especially for objects such as jobs, job steps, and job statistics, memory units need to be frequently created and destroyed with each submission, update, and completion, which severely slows down the processing speed. The memory operations of each sub-step are independent, and the repeated allocation and release not only consume a lot of CPU resources for heap memory management but also lengthen the overall execution chain of the task.

[0165] In this embodiment, the above process is progressive and complementary. Specifically, the first step improves the parallelism of message processing through module decoupling and thread separation, providing a foundation for system scalability in high-concurrency environments. The second step then optimizes the bottleneck within the process—memory operations—by replacing the blocking operations of the main process with asynchronous threads, thereby ensuring that each message can be processed efficiently under high concurrency. The third step further reduces memory processing costs by replacing the allocation / release of a large number of memory units with reusable "cleanup + reclamation" operations, fundamentally reducing resource consumption.

[0166] The task processing method, apparatus, electronic device, storage medium, and program product provided in this application classify asynchronous messages (jobs, nodes, partitions) and allocate them to independent thread pools through message classification and parallel processing. Asynchronous memory management involves the main thread submitting memory allocation / release tasks to a background thread queue while processing messages, with the actual operation performed by asynchronous threads. Memory unit reuse involves clearing fields and reclaiming memory units before release, and clearing fields and reclaiming them to the resource pool after a job ends, allowing subsequent jobs to reuse them. In this process, module decoupling and thread separation significantly alleviate the pseudo-parallelism problem caused by lock contention, enabling the system to fully utilize multi-core CPU resources in high-concurrency scenarios. The asynchronous memory management mechanism removes time-consuming memory operations from the main process, avoiding main thread blocking and improving message processing response speed. The memory unit reuse mechanism reduces redundant memory allocation through a "clear + reclamation" strategy, lowering CPU load and reducing fragmentation issues, especially effective in short-job-intensive scenarios. The synergistic effect of these three elements enables the scheduling system to maintain stable operation with low latency and high throughput even when facing millions of concurrent messages, providing a reliable guarantee for the resource scheduling efficiency and stability of ultra-large-scale clusters. Through the above steps, the system achieves efficient message processing capabilities in high-concurrency scenarios by synergistically optimizing module decoupling, asynchronous processing, and resource reuse.

[0167] Based on any of the above embodiments, the method by which the electronic device reclaims the cleared target memory unit into the corresponding memory resource pool in the above processing method will be described in detail.

[0168] Figure 6 This is a schematic diagram illustrating a method for reclaiming memory cells according to an embodiment of this application. Please refer to [link / reference]. Figure 6 The method includes the following steps:

[0169] S601. Determine the reclamation status of the target memory unit after clearing.

[0170] The recycling status can be either recyclable or non-recyclable. In other words, the recycling status is used to determine whether a target memory unit can be reused after it has been cleared.

[0171] The reclaimable status indicates that the target memory unit, after being cleared, can be relinked to the free list of the resource pool for direct reuse by subsequent tasks.

[0172] The "unreclaimable" status indicates that the internal structure of the target memory unit is too complex to be completely cleaned up, meaning it is not suitable to be put back into the resource pool for subsequent tasks.

[0173] In some embodiments, the electronic device can detect the memory integrity of the target memory unit (e.g., determine whether there is any residual data abnormality). If there is an integrity problem, it is directly determined to be in an unrecoverable state. If the integrity meets the requirements, the cumulative number of times the memory unit is used is counted (e.g., determine whether it exceeds a preset reuse limit). If it exceeds the limit, it is determined to be in an unrecoverable state. It can also calculate the memory region fragmentation rate (percentage of free sub-regions) corresponding to the memory unit. If the fragmentation rate is higher than a preset threshold, it is determined to be in an unrecoverable state. Otherwise, it is determined to be in a recoverable state.

[0174] S602. When the recycling status is recyclable, the cleared target memory unit is recycled to the corresponding memory resource pool.

[0175] In some embodiments, when the recycling status is recyclable, the status metadata of the cleared target memory unit can be updated from occupied to available; the memory unit can be moved from the occupied queue of the corresponding memory resource pool to the available queue of the same task status (for example, the memory unit in the pending execution state can be recycled to the available queue of the pending execution memory resource pool); the number of available memory units in the memory resource pool can be updated synchronously to ensure that the first task in the same state can quickly identify and use the memory unit, thus completing the closed loop of recycling and reuse.

[0176] S603. When the recycling status is non-recyclable, the cleared target memory unit is released, a memory unit corresponding to the target memory unit is allocated in the memory resource pool, and the memory unit is marked as available.

[0177] In some embodiments, the electronic device can perform a memory cell release operation. By calling the system memory management interface, the physical memory region corresponding to the unrecoverable target memory cell is marked as system free memory, the memory cell is released back to the operating system, and the metadata of the memory cell is removed from the management queue of the corresponding memory resource pool. The memory capacity of the released memory cell is obtained, and in the free memory region of the corresponding memory resource pool, the start and end addresses of consecutive memory cells with matching capacity are determined. New memory cells of the same specification are divided, the write status is set to available, and the new memory cells are added to the available queue of the memory resource pool, thereby maintaining the structure supply capacity of the resource pool.

[0178] Below, through Figure 7 It provides a detailed explanation of the lifecycle of memory units and the process of asynchronous data cleanup.

[0179] Figure 7 This is a schematic diagram illustrating the asynchronous cleanup process of memory units provided in an embodiment of this application. Please refer to... Figure 7 It includes a memory unit lifecycle module, a cache pool module, and an asynchronous data processing module. The cache pool module includes a memory allocation cache pool and a memory release cache pool.

[0180] The memory unit lifecycle module includes memory allocation, memory initialization, data filling, data usage, and memory release. The abstracted memory allocation and release indicate that, compared to existing technologies, this embodiment does not require repeated memory allocation and release. Specifically, when a new task needs to be processed, a pre-created blank memory unit is directly obtained from the memory allocation cache pool. Lightweight memory initialization (e.g., setting initial state flags), data filling (e.g., writing task parameters), and data usage (e.g., executing core business logic) are performed. After the business logic is completed, the memory unit is not immediately released but is temporarily moved to the memory release cache pool.

[0181] The asynchronous data processing module automatically performs data cleanup operations in the background, clearing business data from memory units to prepare for reuse. This process runs in parallel with the main business flow, does not block the execution of subsequent tasks, and sends the cleaned memory units back to the memory allocation cache pool, making them blank memory units that can be directly accessed again, thus forming a closed loop of access, use, temporary storage, cleanup, and reuse.

[0182] When the memory allocation cache pool resources are insufficient, the asynchronous data processing module performs memory allocation operations (e.g., batch pre-creating new memory units) to replenish the pool, ensuring a continuous supply of resources. This replenishment operation is also asynchronous and does not affect the performance of the main process. For frequently used memory units such as jobs, job steps, and job statistics, this mechanism can significantly improve the reuse rate, avoid frequent use of system-level malloc / free operations, and reduce CPU load and heap management pressure. Especially in clusters with short job density and frequent messages, this mechanism can stably reduce scheduling response time and improve the overall throughput of resource scheduling.

[0183] The task processing method, apparatus, electronic device, storage medium, and program product provided in this application determine the recycling status of target memory units after clearing. Recyclable memory units are recycled to the memory resource pool corresponding to the task status, while unrecyclable memory units are released and replenished with memory units of the same specification. In this method, the electronic device can accurately identify the reuse value of memory units through multi-dimensional detection (memory integrity, reuse count, fragmentation rate), achieving differentiated recycling management of memory resources. This effectively avoids the problems of damaged memory units mixed into the resource pool and continuously increasing memory fragmentation rate in traditional stateless recycling methods, ensuring the healthy availability of the memory resource pool. Furthermore, through the closed-loop process of recyclable reuse and unrecyclable replenishment, it ensures the stability of the memory resource pool capacity corresponding to the task status and provides resource supply capabilities adapted to high-concurrency task scenarios. It eliminates the need for additional memory defragmentation or frequent expansion operations to maintain resource pool availability, reducing system complexity and maintenance costs. Therefore, this method significantly improves the robustness of the memory resource pool and the continuity of task processing while increasing the memory unit reuse rate, and at the same time reduces the frequency of system-level memory operations, further enhancing the overall efficiency and stability of large-scale computing cluster scheduling.

[0184] Figure 8 This is a flowchart illustrating another task processing method provided in an embodiment of this application. Please refer to... Figure 8 The method includes the following steps:

[0185] S801, Receive task processing request.

[0186] The task processing request includes the first task to be processed, the task status of each first task, and the memory required to process the first task.

[0187] S802. Encapsulate the task processing request to obtain the encapsulated task processing request.

[0188] S803: Send the encapsulated task processing request to the asynchronous processing thread.

[0189] S804: Obtain multiple pending first tasks, the task status of each first task, and the memory required to process the first task.

[0190] The task status includes pending submission, pending execution, and completed status.

[0191] It should be noted that the execution process of S801-S804 above can be found in S201, and will not be repeated here.

[0192] S805. For any given task state, determine the memory resource pool corresponding to that task state.

[0193] S806, Get the preset memory corresponding to the task status.

[0194] S807. Based on the preset memory, create a preset number of memory units in the memory resource pool corresponding to the task state.

[0195] S808. Determine the memory resource pool corresponding to each first task based on the status of multiple tasks.

[0196] The memory resource pool is used to process the first task with the same task status, and the memory resource pool includes memory units required to execute the first task.

[0197] It should be noted that the execution process of S805-S808 can be found in S202, and will not be repeated here.

[0198] S809. Determine the status of multiple memory units in the memory resource pool corresponding to the first task.

[0199] The status can be either available or unavailable.

[0200] S810. Among the multiple memory units in the available state, determine whether there is a memory unit that matches the first task.

[0201] The memory allocated to the matched memory unit is the same as the memory corresponding to the first task.

[0202] S811. When it is determined that there is a memory unit that matches the first task, the matching memory unit is determined as the target memory unit.

[0203] S812. When it is determined that there is any memory unit that does not match the first task, determine the memory unit to be divided according to the memory of the first task.

[0204] S813. In the memory resource pool, determine the starting address and ending address of the memory.

[0205] S814. Based on the start address and end address, determine the memory unit that is the same as the memory of the first task, and designate it as the target memory unit.

[0206] S815. Process the first task according to the target memory unit.

[0207] It should be noted that the execution process of S809-S815 above can be found in S203, and will not be repeated here.

[0208] S816. After the first task is completed, clear the data in the target memory unit.

[0209] S817. Determine the reclamation status of the target memory unit after clearing.

[0210] The recycling status can be either recyclable or non-recyclable.

[0211] S818. When the recycling status is recyclable, the cleared target memory unit is recycled to the corresponding memory resource pool.

[0212] S819. When the recycling status is non-recyclable, the cleared target memory unit is released, a memory unit corresponding to the target memory unit is allocated in the memory resource pool, and the memory unit is marked as available.

[0213] It should be noted that the execution process of S817-S819 can be found in S601-S603, and will not be repeated here.

[0214] The task processing method, apparatus, electronic device, storage medium, and program product provided in this application effectively solve the bottleneck problems of existing systems in terms of concurrent processing capabilities, memory management efficiency, and memory unit reuse by comprehensively optimizing the message processing flow of the scheduling system server. Through logical decoupling of modules such as jobs, nodes, and partitions, combined with an event-driven mechanism, the system can support more granular concurrent processing, with different types of messages executed in parallel in independent threads, greatly improving the overall processing throughput. Furthermore, maintaining the execution order of the job lifecycle vertically ensures the accuracy and completeness of the scheduling logic, laying the foundation for system stability. Based on this, by introducing an asynchronous memory allocation and release mechanism, time-consuming dynamic memory management operations are separated from the main processing thread and handed over to a dedicated background thread for asynchronous completion, significantly reducing response latency during scheduling. The decoupling of the memory management process from the business logic processing process allows the scheduling system to maintain a smooth and stable operating state even in high-concurrency scenarios, effectively reducing the burden on the main thread and avoiding scheduling stuttering and delays caused by memory operation blocking. In addition, by adopting a memory resource pool and reuse mechanism, resource waste caused by frequent memory allocation and release is avoided. By clearing data and reclaiming it to the resource pool before releasing memory units, not only is memory management overhead saved, but memory utilization is also improved and fragmentation issues are reduced. This mechanism is particularly suitable for managing high-frequency objects such as jobs, job steps, and statistical data, further enhancing the system's operating efficiency and stability under high loads, and has good scalability and engineering application value.

[0215] Figure 9 This is a schematic diagram of the structure of a task processing device provided in an embodiment of this application, as shown below. Figure 9 As shown, the device 90 includes: an acquisition module 901, a first determination module 902, a second determination module 903, and a processing module 904.

[0216] The acquisition module 901 is used to acquire multiple first tasks to be processed, the task status of each first task, and the memory required to process the first task. The task status is used to indicate the stage of the first task.

[0217] The first determining module 902 is used to determine the memory resource pool corresponding to each first task based on multiple task states. The memory resource pool is used to process the first tasks with the same task state. The memory resource pool includes memory units for providing the memory units required to execute the first task.

[0218] The second determining module 903 is used to determine the target memory unit from multiple memory units in the memory resource pool corresponding to the first task based on the memory corresponding to the first task, and to process the first task based on the target memory unit.

[0219] The processing module 904 is used to clear the data in the target memory unit after the first task is completed, and to reclaim the cleared target memory unit into the corresponding memory resource pool.

[0220] The task processing device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0221] In one possible implementation, the second determining module 903 is specifically used for:

[0222] Determine the status of multiple memory units in the memory resource pool corresponding to the first task. The status can be available or unavailable.

[0223] Among the multiple memory units that are in an available state, determine the target memory unit that corresponds to the memory unit of the first task.

[0224] In one possible implementation, the second determining module 903 is specifically used for:

[0225] Among the multiple memory units in the available state, determine whether there is a memory unit that matches the first task. The memory allocated to the matching memory unit is the same as the memory corresponding to the first task.

[0226] When it is determined that there is a memory cell that matches the first task, the matching memory cell is determined as the target memory cell;

[0227] When it is determined that there is any memory unit that does not match the first task, based on the memory of the first task, a memory unit that is the same as the memory of the first task is allocated in the memory resource pool and determined as the target memory unit.

[0228] In one possible implementation, the processing module 904 is specifically used for:

[0229] Based on the memory of the first task, determine the memory units that need to be partitioned;

[0230] Within the memory resource pool, determine the starting and ending address locations of the memory.

[0231] Based on the start and end address locations, determine the memory unit that is the same as the memory of the first task.

[0232] In one possible implementation, the apparatus further includes a creation module, which is specifically used for:

[0233] Determine the memory resource pool corresponding to the task status;

[0234] Get the preset memory corresponding to the task status;

[0235] Based on the preset memory, create a preset number of memory units in the memory resource pool corresponding to the task state.

[0236] In one possible implementation, the apparatus further includes a receiving module, which is specifically used for:

[0237] Receive a task processing request, which includes the first task to be processed, the task status of each first task, and the memory required to process the first task.

[0238] The task processing request is encapsulated to obtain the encapsulated task processing request.

[0239] Send a packaged task processing request to the asynchronous processing thread.

[0240] In one possible implementation, the processing module 904 is specifically used for:

[0241] Determine the reclamation status of the target memory unit after it has been cleared. The reclamation status can be either reclaimable or non-reclaimable.

[0242] When the recycling status is recyclable, the cleared target memory unit will be recycled to the corresponding memory resource pool;

[0243] When the target memory unit is in an unrecoverable state, the cleared target memory unit is released, a memory unit corresponding to the target memory unit is allocated in the memory resource pool, and the memory unit is marked as available.

[0244] The task processing device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0245] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 10 As shown, the electronic device 10 may include at least one processor 1001 and a memory 1002.

[0246] The memory 1002 is used to store programs. Specifically, the program may include program code, which includes computer-executable instructions.

[0247] The memory 1002 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage.

[0248] The processor 1001 is used to execute computer execution instructions stored in the memory 1002 to implement the method described in the foregoing method embodiments. The processor 1001 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0249] Optionally, the electronic device 10 may also include a communication interface 1003. In specific implementations, if the communication interface 1003, memory 1002, and processor 1001 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an industry-standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0250] Optionally, in a specific implementation, if the communication interface 1003, memory 1002 and processor 1001 are integrated on a single chip, then the communication interface 1003, memory 1002 and processor 1001 can communicate through an internal interface.

[0251] The electronic device in this embodiment can be used to execute the technical solutions of the above method embodiments. The specific implementation methods and technical effects are similar, and will not be repeated here.

[0252] This application provides a computer-readable storage medium, which may include various media capable of storing computer-executable instructions, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), RAM, a disk, or an optical disk. Specifically, the computer-readable storage medium stores computer-executable instructions, which, when executed by a computer, cause the technical solution shown in the above method embodiment to be executed. The specific implementation and technical effects are similar and will not be repeated here.

[0253] This application provides a computer program product, including a computer program. When the computer program is executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar, and will not be repeated here.

[0254] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0255] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0256] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0257] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0258] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0259] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0260] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0261] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A task processing method, characterized in that, include: Obtain multiple pending first tasks, the task status of each first task, and the memory required to process the first task, wherein the task status is used to indicate the stage of the first task; Based on multiple task states, a memory resource pool is determined for each first task. The memory resource pool is used to process first tasks with the same task state. The memory resource pool includes memory units required to execute the first task. Based on the memory corresponding to the first task, a target memory unit is determined from multiple memory units in the memory resource pool corresponding to the first task, and the first task is processed based on the target memory unit; After the first task is completed, the data in the target memory unit is cleared, and the cleared target memory unit is recycled to the corresponding memory resource pool.

2. The method according to claim 1, characterized in that, Based on the memory corresponding to the first task, a target memory unit is determined from multiple memory units in the memory resource pool corresponding to the first task, including: Determine the status of multiple memory units in the memory resource pool corresponding to the first task, wherein the status can be an available status or an unavailable status; Among the multiple memory units that are in an available state, the target memory unit corresponding to the memory corresponding to the first task is determined.

3. The method according to claim 2, characterized in that, Determining the target memory unit corresponding to the memory corresponding to the first task from among the multiple memory units in the available state includes: Among the multiple memory units in the available state, it is determined whether there is a memory unit that matches the first task, and the memory allocated to the matching memory unit is the same as the memory corresponding to the first task. When it is determined that there is a memory unit that matches the first task, the matching memory unit is determined as the target memory unit; When it is determined that there is any memory unit that does not match the first task, based on the memory of the first task, a memory unit that is the same as the memory of the first task is allocated in the memory resource pool and determined as the target memory unit.

4. The method according to claim 3, characterized in that, Based on the memory of the first task, memory units identical to the memory of the first task are allocated from the memory resource pool, including: Based on the memory of the first task, determine the memory units that need to be divided. In the memory resource pool, the start address and end address of the memory are determined; Based on the start address and the end address, a memory unit that is the same as the memory of the first task is determined.

5. The method according to claim 1, characterized in that, For any given task state; before determining the memory resource pool corresponding to each first task based on multiple task states, the method further includes: Determine the memory resource pool corresponding to the task status; Obtain the preset memory corresponding to the task status; Based on the preset memory, a preset number of memory units are created in the memory resource pool corresponding to the task state.

6. The method according to claim 1, characterized in that, Before obtaining multiple pending first tasks, the task status of each first task, and the memory required to process the first task, the process includes: Receive a task processing request, the task processing request including a first task to be processed, the task status of each first task, and the memory required to process the first task; The task processing request is encapsulated to obtain the encapsulated task processing request; Send the encapsulated task processing request to the asynchronous processing thread.

7. The method according to claim 1, characterized in that, The cleared target memory units are then returned to the corresponding memory resource pool, including: Determine the reclamation status of the cleared target memory unit, which can be a reclaimable status or an unreclaimable status; When the recycling state is recyclable, the cleared target memory unit is recycled to the corresponding memory resource pool; When the recycling state is unrecyclable, the cleared target memory unit is released, a memory unit corresponding to the memory of the target memory unit is allocated in the memory resource pool, and the memory unit is marked as available.

8. A task processing device, characterized in that, include: The acquisition module is used to acquire multiple first tasks to be processed, the task status of each first task, and the memory required to process the first task. The task status is used to indicate the stage of the first task. The first determining module is used to determine the memory resource pool corresponding to each first task based on multiple task states. The memory resource pool is used to process first tasks with the same task state. The memory resource pool includes memory units for providing the memory units required to execute the first task. The second determining module is used to determine a target memory unit from multiple memory units in the memory resource pool corresponding to the first task based on the memory corresponding to the first task, and to process the first task based on the target memory unit; The processing module is used to clear the data in the target memory unit after the first task is completed, and to reclaim the cleared target memory unit into the corresponding memory resource pool.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images