Task processing method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a task processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of obtaining task information corresponding to a to-be-processed task, and obtaining current load information of a candidate task processing unit associated with the to-be-processed task; the candidate task processing unit at least comprises two thread units and two process units; determining a processing priority corresponding to the to-be-processed task according to the task information; determining a target task processing unit corresponding to the to-be-processed task from the candidate task processing units according to the processing priority and the current load information; distributing the to-be-processed task to a corresponding target task processing unit, and enabling the target task processing unit to process the to-be-processed task; the processing state of the to-be-processed task is used for updating the number of the candidate task processing units. By adopting the method, the task processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] At present, with the increasing amount of computer tasks, how to process the tasks efficiently becomes crucial.

[0003] In traditional technology, in the process of processing tasks, a single thread or process architecture is generally used to process tasks; however, this method is easily affected by blocking and consumes a lot of time, resulting in low efficiency in task processing. Summary of the invention

[0004] Based on this, it is necessary to provide a task processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of task processing in response to the above technical problems.

[0005] In a first aspect, the present application provides a task processing method, comprising:

[0006] Acquire task information corresponding to the task to be processed, and acquire current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units;

[0007] Determining the processing priority corresponding to the task to be processed according to the task information;

[0008] Determining, from each of the candidate task processing units, a target task processing unit corresponding to the task to be processed according to the processing priority and the current load information;

[0009] The to-be-processed tasks are allocated to corresponding target task processing units, so that the target task processing units process the to-be-processed tasks; and the processing status of the to-be-processed tasks is used to update the number of the candidate task processing units.

[0010] In one embodiment, determining the target task processing unit corresponding to the task to be processed from each of the candidate task processing units according to the processing priority and the current load information includes:

[0011] Obtaining the correspondence between the processing priority and the load information range;

[0012] According to the processing priority, query the corresponding relationship to obtain the target load information range corresponding to the processing priority;

[0013] From the candidate task processing units, a candidate task processing unit whose corresponding current load information is within the target load information range is determined as the target task processing unit corresponding to the task to be processed.

[0014] In one embodiment, after allocating the to-be-processed task to the corresponding target task processing unit so that the target task processing unit processes the to-be-processed task, the method further includes:

[0015] Monitoring the processing status of the task to be processed through a preset non-blocking input and output interface;

[0016] When it is monitored that the processing status is completed, obtaining log information corresponding to the task to be processed;

[0017] The number of the candidate task processing units is updated according to the log information.

[0018] In one embodiment, updating the number of the candidate task processing units according to the log information includes:

[0019] Extracting the task processing time in the log information;

[0020] The number of the candidate task processing units is updated according to the difference between the task processing time and the preset processing time.

[0021] In one embodiment, updating the number of the candidate task processing units according to the difference between the task processing time and the preset processing time includes:

[0022] When the task processing time is greater than the preset processing time, updating the number of the candidate task processing units;

[0023] or,

[0024] When the task processing time is less than or equal to the preset processing time, the number of the candidate task processing units is maintained.

[0025] In one embodiment, the method further comprises:

[0026] Writing the log information into a buffer;

[0027] Checking and processing the occupied space of the log information in the buffer to obtain a check result corresponding to the buffer;

[0028] When the inspection result meets the preset inspection result, the log information in the buffer is deleted.

[0029] In one embodiment, the obtaining task information corresponding to the task to be processed includes:

[0030] Obtaining the task type, task completion time and task data volume corresponding to the task to be processed;

[0031] The task type, the task completion time and the task data volume are all used as the task information.

[0032] In a second aspect, the present application further provides a task processing device, comprising:

[0033] An information acquisition module, used to acquire task information corresponding to the task to be processed, and to acquire current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units;

[0034] A priority determination module, used to determine the processing priority corresponding to the task to be processed according to the task information;

[0035] A unit determination module, used to determine the target task processing unit corresponding to the task to be processed from each of the candidate task processing units according to the processing priority and the current load information;

[0036] The task processing module is used to allocate the pending tasks to the corresponding target task processing units so that the target task processing units process the pending tasks; the processing status of the pending tasks is used to update the number of the candidate task processing units.

[0037] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0038] Acquire task information corresponding to the task to be processed, and acquire current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units;

[0039] Determining the processing priority corresponding to the task to be processed according to the task information;

[0040] Determining, from each of the candidate task processing units, a target task processing unit corresponding to the task to be processed according to the processing priority and the current load information;

[0041] The to-be-processed tasks are allocated to corresponding target task processing units, so that the target task processing units process the to-be-processed tasks; and the processing status of the to-be-processed tasks is used to update the number of the candidate task processing units.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0043] Acquire task information corresponding to the task to be processed, and acquire current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units;

[0044] Determining the processing priority corresponding to the task to be processed according to the task information;

[0045] Determining, from each of the candidate task processing units, a target task processing unit corresponding to the task to be processed according to the processing priority and the current load information;

[0046] The to-be-processed tasks are allocated to corresponding target task processing units, so that the target task processing units process the to-be-processed tasks; and the processing status of the to-be-processed tasks is used to update the number of the candidate task processing units.

[0047] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0048] Acquire task information corresponding to the task to be processed, and acquire current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units;

[0049] Determining the processing priority corresponding to the task to be processed according to the task information;

[0050] Determining, from each of the candidate task processing units, a target task processing unit corresponding to the task to be processed according to the processing priority and the current load information;

[0051] The to-be-processed tasks are allocated to corresponding target task processing units, so that the target task processing units process the to-be-processed tasks; and the processing status of the to-be-processed tasks is used to update the number of the candidate task processing units.

[0052] The above-mentioned task processing method, device, computer equipment, storage medium and computer program product first obtain the task information corresponding to the task to be processed, and obtain the current load information of the candidate task processing unit associated with the task to be processed, and then determine the processing priority corresponding to the task to be processed according to the task information, and then determine the target task processing unit corresponding to the task to be processed from each candidate task processing unit according to the processing priority and the current load information, and finally, assign the task to be processed to the corresponding target task processing unit, so that the target task processing unit processes the task to be processed. In this way, in the process of processing the task, the processing priority corresponding to the task to be processed and the current load information of the candidate task processing unit are comprehensively considered to allocate the tasks to be processed accordingly, so that each task can be processed in a relatively efficient state, which greatly reduces the time required for task processing and is conducive to improving the efficiency of task processing; moreover, the whole process uses the architecture of multiple threads and multiple processes to process the tasks, avoiding the method of using a single thread or process architecture to process the tasks, which is easily affected by blocking and consumes a lot of time, resulting in the defect of low efficiency of task processing, thereby improving the efficiency of task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a flowchart of a task processing method in one embodiment;

[0055] Figure 2 is a flowchart of a task processing method in another embodiment;

[0056] Figure 3 A flowchart of an efficient parallel audit tracking and non-blocking event processing method in one embodiment;

[0057] Figure 4 A schematic diagram of a process of system initialization and architecture configuration in one embodiment;

[0058] Figure 5 A schematic diagram of a process of non-blocking event processing and asynchronous I / O operation in one embodiment;

[0059] Figure 6 A schematic diagram of a process flow of an efficient log writing and memory buffering mechanism in one embodiment;

[0060] Figure 7 It is a schematic diagram of the process of distributed tracing and performance monitoring in one embodiment;

[0061] Figure 8 A flowchart of an audit trail and data optimization process in one embodiment;

[0062] Fig. 9 is a structural block diagram of a task processing device in one embodiment;

[0063] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0066] In an exemplary embodiment, Figure 1 As shown, a task processing method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones and tablet computers; the server can be implemented as an independent server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0067] Step S101, obtaining task information corresponding to the task to be processed, and obtaining current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units.

[0068] The pending tasks refer to tasks that need to be processed.

[0069] The task information includes the task type, task completion time and task data volume corresponding to the task to be processed.

[0070] The candidate task processing unit refers to a task processing unit to be selected, which includes at least two thread units and two process units. In an actual scenario, the candidate task processing units are 8 thread units and 4 process units.

[0071] The current load information is used to indicate the load status of the candidate task processing unit at the current time.

[0072] The thread unit is used to represent the smallest unit that the operating system can perform computation scheduling.

[0073] Among them, the process unit is used to represent the basic unit of resource allocation and scheduling of the operating system.

[0074] Exemplarily, the server obtains task information corresponding to the task to be processed from the database in response to a processing instruction corresponding to the task to be processed; then, the server obtains the load status of the candidate task processing unit associated with the task to be processed at the current time (for example, obtaining the load status of each of 8 thread units and 4 process units) as the current load information of the candidate task processing unit.

[0075] Step S102: determining the processing priority corresponding to the task to be processed according to the task information.

[0076] The processing priority is used to indicate the processing order of the tasks to be processed.

[0077] It should be noted that the processing priority can be determined by the task type in the task information. For example, for a financial system, fund transfer events may be set to the highest priority, while ordinary query operations may be set to a low priority.

[0078] It should be noted that the processing priority can also be determined by the task completion time in the task information. For example, some log events that need to be processed in real time (such as transaction completion notification) can be given a high priority.

[0079] It should be noted that the processing priority can also be determined by the task data volume in the task information. For example, for batch data processing tasks, the priority can be set according to the data volume, and events with large data volumes may need to be processed first to ensure system performance.

[0080] Exemplarily, the server queries the corresponding relationship between the task information and the processing priority according to the task information, and obtains the processing priority corresponding to the task to be processed.

[0081] Step S103 : determining a target task processing unit corresponding to the task to be processed from each candidate task processing unit according to the processing priority and the current load information.

[0082] The target task processing unit is used to represent a candidate task processing unit that processes the task to be processed among the candidate task processing units.

[0083] Exemplarily, the server determines, from among the candidate task processing units, a candidate task processing unit that processes the task to be processed as the target task processing unit according to the processing priority and the current load information.

[0084] Step S104, allocating the pending task to the corresponding target task processing unit, so that the target task processing unit processes the pending task; the processing status of the pending task is used to update the number of candidate task processing units.

[0085] The processing status is used to indicate whether the pending task has been completed.

[0086] Exemplarily, the server allocates the pending task to the target task processing unit corresponding to the pending task, so that the target task processing unit processes the pending task; then, the server updates the number of candidate task processing units according to the processing status of the pending task.

[0087] In the above-mentioned task processing method, firstly, the task information corresponding to the task to be processed is obtained, and the current load information of the candidate task processing unit associated with the task to be processed is obtained, and then the processing priority corresponding to the task to be processed is determined according to the task information, and then, according to the processing priority and the current load information, the target task processing unit corresponding to the task to be processed is determined from each candidate task processing unit, and finally, the task to be processed is assigned to the corresponding target task processing unit, so that the target task processing unit processes the task to be processed. In this way, in the process of processing the task, the processing priority corresponding to the task to be processed and the current load information of the candidate task processing unit are comprehensively considered to allocate the tasks to be processed accordingly, so that each task can be processed in a relatively efficient state, which greatly reduces the time required for task processing and is conducive to improving the efficiency of task processing; moreover, the whole process uses the architecture of multiple threads and multiple processes to process the tasks, avoiding the method of using a single thread or process architecture to process the tasks, which is easily affected by blocking and consumes a lot of time, resulting in the defect of low efficiency of task processing, thereby improving the efficiency of task processing.

[0088] In an exemplary embodiment, the above step S103, based on the processing priority and the current load information, determines the target task processing unit corresponding to the task to be processed from each candidate task processing unit, and specifically includes the following contents: obtaining the correspondence between the processing priority and the load information range; querying the correspondence based on the processing priority to obtain the target load information range corresponding to the processing priority; and determining, from each candidate task processing unit, the candidate task processing unit whose corresponding current load information is within the target load information range as the target task processing unit corresponding to the task to be processed.

[0089] Among them, the corresponding relationship between the processing priority and the load information range is used to indicate the associated information between the processing priority and the load information range. For example, the load information range corresponding to the high-priority task is that the CPU (Central Processing Unit) usage is less than 30% and the memory usage is less than 20%; the load information range corresponding to the medium-priority task is that the CPU usage is between 30%-60% and the memory usage is between 20%-50%; the load information range corresponding to the low-priority task is that the CPU usage is higher than 60% and the memory usage is higher than 50%.

[0090] The target load information range refers to the load information range corresponding to the processing priority.

[0091] Exemplarily, the server obtains the correspondence between the processing priority and the load information range from the database; queries the correspondence according to the processing priority to obtain the target load information range corresponding to the processing priority; and determines, from each candidate task processing unit, the candidate task processing unit whose corresponding current load information is within the target load information range as the target task processing unit corresponding to the task to be processed.

[0092] In this embodiment, by obtaining the correspondence between the processing priority and the load information range, and matching the target task processing unit for the task to be processed based on this relationship, system resources can be reasonably allocated based on priority factors such as the importance and urgency of the task, thereby avoiding idleness and waste of resources, and enabling tasks of different priorities to be carried out in an orderly manner under a suitable resource environment.

[0093] In an exemplary embodiment, the above step S104, after allocating the task to be processed to the corresponding target task processing unit and making the target task processing unit process the task to be processed, specifically includes the following contents: monitoring the processing status of the task to be processed through a preset non-blocking input and output interface; when it is monitored that the processing status is completed, obtaining the log information corresponding to the task to be processed; and updating the number of candidate task processing units according to the log information.

[0094] The preset non-blocking input / output interface refers to a non-blocking I / O (input / output) interface provided by the operating system.

[0095] The log information is used to represent the recorded data associated with the execution process of the task to be processed, such as the task processing time, task number, etc.

[0096] Exemplarily, the server monitors the processing status of the task to be processed through a preset non-blocking input and output interface according to set rules and time intervals; when it is monitored that the processing status is completed, the server obtains the log information corresponding to the task to be processed based on the task identifier corresponding to the task to be processed; then, the server updates the number of candidate task processing units based on the log information.

[0097] In this embodiment, a preset non-blocking input and output interface is used to monitor the processing status of the task to be processed. This non-blocking method does not hinder the task processing unit from executing tasks normally during the monitoring process, thereby avoiding the idle resources and reduced efficiency caused by traditional blocking monitoring, thereby improving the overall task processing efficiency.

[0098] In an exemplary embodiment, the number of candidate task processing units is updated according to the log information, specifically including the following contents: extracting the task processing time in the log information; and updating the number of candidate task processing units according to the difference between the task processing time and the preset processing time.

[0099] The task processing time refers to the processing time corresponding to the task to be processed.

[0100] The preset processing time refers to a pre-set processing time, which is used to determine the task processing time. It should be noted that the preset processing time depends on the situation.

[0101] Exemplarily, the server obtains format information corresponding to the log information; then, the server extracts the task processing time in the log information based on the format information corresponding to the log information; for example, when the format information corresponding to the log information is text information, the server records the corresponding timestamp (located at the beginning of the line or a fixed position in the line) for each line of the text information as the task processing time in the log information; then, the server updates the number of candidate task processing units based on the difference between the task processing time and the preset processing time.

[0102] In this embodiment, by extracting the task processing time in the log information and comparing it with the preset processing time, the number of candidate task processing units can be dynamically updated based on the difference between the actual and preset processing times, thereby achieving accurate allocation of resources, so that each task can obtain the amount of resources that matches its needs, avoiding waste of resources or affecting task progress due to insufficient resources.

[0103] In an exemplary embodiment, the number of candidate task processing units is updated according to the difference between the task processing time and the preset processing time, specifically including the following contents: when the task processing time is greater than the preset processing time, the number of candidate task processing units is updated; or, when the task processing time is less than or equal to the preset processing time, the number of candidate task processing units is maintained.

[0104] Exemplarily, the server compares the task processing time with the preset processing time to obtain a comparison result; when the comparison result indicates that the task processing time is greater than the preset processing time, the number of candidate task processing units is updated; when the comparison result indicates that the task processing time is less than or equal to the preset processing time, the number of candidate task processing units is maintained.

[0105] In this embodiment, by dynamically adjusting the number of candidate task processing units according to the task processing time and the preset processing time, it is possible to accurately and reasonably allocate resources according to the actual execution of the task, thereby optimizing the utilization of system resources and avoiding the situation where additional servers are added but there are no tasks to process, causing idle resources.

[0106] In an exemplary embodiment, the method also includes the following contents: writing log information into a buffer; checking and processing the space occupied by the log information in the buffer to obtain an inspection result corresponding to the buffer; and deleting the log information in the buffer when the inspection result meets the preset inspection result.

[0107] The buffer is used to represent the area where log information is stored, and is also called a log buffer or a memory buffer.

[0108] The occupied space is used to indicate the occupancy of the log information in the buffer, for example, the log information reaches 90% of the buffer capacity.

[0109] The check result is used to indicate the result obtained by checking the occupied space of the log information in the buffer.

[0110] The preset inspection result refers to a preset inspection result, which is used to judge the inspection result, such as the log information reaching 80% of the buffer capacity. It should be noted that the preset inspection result depends on the situation.

[0111] Exemplarily, the server determines the buffer corresponding to the log information and writes the log information to the buffer; then, the server checks the space occupied by the log information in the buffer, obtains the inspection result corresponding to the buffer, and obtains the current time information; when the inspection result meets the preset inspection result or the current time information meets the preset time information, the log information in the buffer is deleted; for example, when the log information reaches 80% of the capacity of the buffer, or the interval between the current time information and the last deletion time reaches 5 seconds, the server deletes the log information in the buffer.

[0112] In this embodiment, by checking and processing the space occupied by log information in the buffer periodically or according to specific conditions, the space occupied by logs that are no longer of immediate use value can be released in time, so that the buffer always has enough free space to accept new data, thereby achieving efficient use of storage space and avoiding problems such as data loss or system operation abnormality due to insufficient storage space.

[0113] In an exemplary embodiment, the above step S101, obtaining task information corresponding to the task to be processed, specifically includes the following contents: obtaining the task type, task completion time and task data volume corresponding to the task to be processed; and using the task type, task completion time and task data volume as task information.

[0114] The task type is used to indicate the type of task to be processed, such as data processing task, file transfer task, computing task, etc.

[0115] The task completion time is used to represent the total time from the start of the pending task to the successful completion of the pending task.

[0116] The task data volume is used to indicate the size of the task to be processed.

[0117] Exemplarily, the server obtains the task type, task completion time and task data volume corresponding to the task to be processed from the database; then, the server combines the task type, task completion time and task data volume according to a preset combination method to obtain the task information corresponding to the task to be processed.

[0118] In this embodiment, by taking the task type, task completion time and task data volume as the task information corresponding to the task to be processed, a multi-dimensional description of the task to be processed is formed, which is beneficial to improving the accuracy of determining the task information corresponding to the task to be processed and provides a data basis for the subsequent processing of the task to be processed.

[0119] In an exemplary embodiment, Figure 2 As shown, another task processing method is provided, which is described by taking the method applied to a server as an example, and includes the following steps:

[0120] Step S201, obtaining the task type, task completion time and task data volume corresponding to the task to be processed; taking the task type, task completion time and task data volume as the task information corresponding to the task to be processed.

[0121] Step S202, obtaining current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units.

[0122] Step S203: determining the processing priority corresponding to the task to be processed according to the task information.

[0123] Step S204: Acquire the corresponding relationship between the processing priority and the load information range.

[0124] Step S205: query the corresponding relationship according to the processing priority to obtain the target load information range corresponding to the processing priority.

[0125] Step S206 , determining, from among the candidate task processing units, a candidate task processing unit whose corresponding current load information is within the target load information range as the target task processing unit corresponding to the task to be processed.

[0126] Step S207 , allocating the pending task to the corresponding target task processing unit, so that the target task processing unit processes the pending task; the processing status of the pending task is used to update the number of candidate task processing units.

[0127] In the above-mentioned task processing method, in the process of processing tasks, the processing priorities corresponding to the tasks to be processed and the current load information of the candidate task processing units are comprehensively considered to allocate the tasks to be processed accordingly, so that each task can be processed in a relatively efficient state, which greatly reduces the time required for task processing and is conducive to improving the efficiency of task processing; moreover, the entire process adopts a multi-thread and multi-process architecture to process tasks, avoiding the use of a single thread or process architecture to process tasks, which is easily affected by blocking and consumes a lot of time, resulting in the defect of low task processing efficiency, thereby improving the efficiency of task processing.

[0128] In an exemplary embodiment, in order to more clearly illustrate the task processing method provided by the embodiment of the present application, the task processing method is specifically described below with a specific embodiment. Figure 3As shown, the present application also provides an efficient parallel audit tracking and non-blocking event processing method. In the process of processing tasks, the task information corresponding to the task to be processed is first obtained, and the current load information of the candidate task processing unit associated with the task to be processed is obtained. Then, according to the task information, the processing priority corresponding to the task to be processed is determined. Then, according to the processing priority and the current load information, the target task processing unit corresponding to the task to be processed is determined from each candidate task processing unit. Finally, the task to be processed is assigned to the corresponding target task processing unit, so that the target task processing unit processes the task to be processed. Specifically, it includes the following contents:

[0129] 1. System initialization and architecture configuration, such as Figure 4 As shown:

[0130] When the system starts, initialize the multi-threaded and multi-process parallel processing architecture. Set the threshold of the number of CPU cores and start the corresponding number of working thread pools and process pools. For example, on a server with an 8-core CPU, start 8 processing threads and 4 processes. Each thread and process is responsible for different audit tracking and log event processing tasks. Here it is understood that multiple processing threads / processes are assigned to each log event processing task. The task scheduler will dynamically adjust the task allocation according to the current running status and load conditions of each thread / process. For example, when a thread is processing a high-priority task, the scheduler can assign the new task to a thread with a lower load to ensure the processing efficiency and load balance of each thread.

[0131] Initialize the task scheduler, configure the task queue and load balancing rules. The task scheduler monitors the running status and load of each thread and process, and dynamically adjusts the distribution of tasks. Set the task queue as the priority queue to ensure that high-priority audit tasks are processed first.

[0132] Among them, task queue design: the task queue can be designed as a multi-level priority queue, and tasks of different priorities can be placed in different queues. For example, high-priority audit events (such as large financial transfers) can be placed in a high-priority queue. Ordinary events can be placed in a low-priority queue.

[0133] Among them, the load balancing rule is: use dynamic scheduling algorithms (such as polling, least connections, and least number of tasks) to balance the load of each thread / process. For example, the scheduler can prioritize tasks to threads with the lowest load to achieve more even task distribution. In addition, the scheduler can monitor indicators such as the CPU usage and processing speed of the thread and dynamically adjust the allocation strategy.

[0134] The priority determination method is as follows:

[0135] (1) Based on event type: For example, for a financial system, fund transfer events may be set to the highest priority, while general query operations may be set to a low priority.

[0136] (2) Event-based time requirements: Some log events that need to be processed in real time (such as transaction completion notifications) can be given high priority.

[0137] (3) Based on the size of the event: For batch data processing tasks, the priority can be set according to the size of the data. Events with large amounts of data may need to be processed first to ensure system performance.

[0138] It should be noted that these priority rules can be configured in the task scheduler to dynamically adjust task allocation.

[0139] 2. Non-blocking event processing and asynchronous I / O operations, such as Figure 5 As shown:

[0140] The system uses a main loop based on an event-driven model to register all audit events and log event handlers that need to be monitored. The main loop will continue to monitor the status of each event, and once an event occurs (such as a new log arrives or an audit event is triggered), the corresponding handler will be called immediately. It should be noted that the event-driven model is the Reactor mode.

[0141] The event handler performs data reading and writing through non-blocking I / O operations. The specific steps are as follows:

[0142] (1) Use non-blocking I / O interfaces provided by the operating system, such as epoll (event polling) or kqueue (kernel event queue), to monitor file descriptors and network connection status.

[0143] (2) When an event is triggered, check the event type (such as read, write, etc.) and call the corresponding callback function. For example, use aio_read (asynchronous input / output read operation) to asynchronously read audit log data, and mount the callback function at the same time to automatically notify the main loop when the data reading is completed.

[0144] Among them, events refer to audit events that need to be monitored in the system. Specifically, "events" here refer to changes in the status of file descriptors or network connections. For example, when there is data to read or write on the file descriptor monitored by the system, or when the network connection status changes (such as connection establishment or disconnection), the corresponding I / O events will be triggered. These I / O events are captured through non-blocking I / O interfaces (such as epoll or kqueue) and trigger subsequent processing flows. In this process, the handler will not be blocked by waiting for the I / O operation to complete, but will efficiently process I / O events through asynchronous I / O operations and callback mechanisms, thereby achieving efficient reading and writing of log data.

[0145] The role of the callback function: In non-blocking I / O operations, calling the callback function can immediately notify the event handler after the I / O operation is completed, thereby avoiding the system waiting during the I / O operation. Through the callback mechanism, the system can automatically start the corresponding processing logic when the event is triggered, while continuing to process other tasks, improving overall efficiency.

[0146] (3) The handler continues to process other tasks to avoid blocking the thread due to waiting for I / O operations.

[0147] 3. Efficient log writing and memory buffering mechanism, such as Figure 6 As shown:

[0148] The system creates a log buffer in memory (for example, set to 10MB in size) to temporarily store audit records and log data. Whenever a new log arrives, it is first written to the memory buffer. The "new log" here refers to the "audit log" data obtained after processing the log event processing task. That is, after the system processes a log event, the specific audit records generated will be temporarily stored in the memory buffer and written to the disk in batches when certain conditions are met.

[0149] Use a timer to trigger a regular check of the buffer status, and perform a batch write operation every 5 seconds or when the buffer reaches 80% capacity. The specific operations are as follows:

[0150] (1) Lock the buffer to prevent data contention, read all data in the buffer and flush the buffer.

[0151] (2) Pack the log data into large blocks and write them to disk at one time through the writev (scattered write) system call (which supports batch writing) to reduce the overhead and latency of a single write operation.

[0152] After the log is written, the buffer is unlocked to allow new log data to continue to be written.

[0153] 4. Distributed tracing and performance monitoring, such as Figure 7As shown:

[0154] Integrate distributed tracing tools and start the system's global performance monitoring module. Configure tracing code for each service and node, and record detailed service call information (such as request start time, end time, duration, call link, etc.) through interceptors and middleware.

[0155] During system operation, the tracking tool automatically generates audit logs and call link data and sends them to the centralized monitoring server for analysis. The monitoring server displays system performance indicators (such as request response time, throughput, error rate, etc.) in real time through a visual dashboard.

[0156] Among them, audit logs and call link data: audit logs and call link data can be generated through the integrated distributed tracing tool. The distributed tracing tool configures the tracing code in each service and node, and uses interceptors and middleware to record the detailed information of each request or operation (such as the start time, end time, duration, service link called, etc. of the request). This information is captured and converted into audit logs and call link data.

[0157] Among them, performance indicator analysis: The centralized monitoring server can extract and calculate system performance indicators from these audit logs and call link data. For example, the server can calculate indicators such as request response time, system throughput, and error rate by counting the time and success rate of requests. These indicators can reflect the overall performance of the system, especially under high load conditions.

[0158] When it is found that the response time of a service exceeds the preset threshold (such as 200 milliseconds), an alarm is triggered and the performance optimization process is started, dynamically adjusting the task allocation strategy or increasing the number of processing threads to cope with load changes. Here, the number of processing threads or processes can be dynamically adjusted according to the processing situation of each log event processing task (such as response time). If the response time of a service exceeds the set threshold (such as 200 milliseconds), the system will trigger an alarm and start the optimization process, such as alleviating the load by adding threads or processes to ensure the overall response speed and system performance.

[0159] 5. Audit tracking and data optimization processes, such as Figure 8 As shown:

[0160] When a new audit event is received, the system first records the event information in memory, such as user ID (identity), operation type, timestamp, etc., for quick query and analysis.

[0161] Audit events refer to specific activities or operation events captured by the system during operation (such as user login, data modification, etc.). When a new audit event occurs, the system will record the event information in memory (such as user ID, operation type, timestamp, etc.), and then write these event information in batches to the log file or database to form an audit log. Therefore, the audit log is the record or persistence of these audit events.

[0162] Each processing thread uses asynchronous I / O operations to write the processed audit data in batches to the local log file or database. To reduce the frequency of disk I / O operations, the system will regularly clean up and archive this data to long-term storage devices.

[0163] The system regularly performs background cleanup tasks, analyzes and compresses historical audit log data, ensures optimal utilization of audit storage space, and provides efficient audit query capabilities.

[0164] In the above embodiment, in the process of processing tasks, the processing priorities corresponding to the tasks to be processed and the current load information of the candidate task processing units are comprehensively considered to allocate the tasks to be processed accordingly, so that each task can be processed in a relatively efficient state, which greatly reduces the time required for task processing and is conducive to improving the efficiency of task processing; moreover, the entire process uses a multi-thread and multi-process architecture to process tasks, avoiding the defect of using a single thread or process architecture to process tasks, which is easily affected by blocking and consumes a lot of time, resulting in low efficiency of task processing, thereby improving the efficiency of task processing. At the same time, through the combination of multi-threading and multi-process, efficient parallel processing is achieved, which greatly improves the processing capacity and task response speed of the system; through the application of asynchronous I / O and event-driven programming models, the blocking problem of traditional synchronous I / O is avoided, so that the system can still respond quickly under high load conditions; through the integration of distributed tracking tools, it is ensured that the system can monitor and optimize performance in real time during operation, and timely discover and handle performance bottlenecks.

[0165] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0166] Based on the same inventive concept, the embodiment of the present application also provides a task processing device for implementing the task processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more task processing device embodiments provided below can refer to the limitations on the task processing method above, and will not be repeated here.

[0167] In an exemplary embodiment, Fig. 9 As shown, a task processing device is provided, including: an information acquisition module 901, a priority determination module 902, a unit determination module 903 and a task processing module 904, wherein:

[0168] The information acquisition module 901 is used to acquire task information corresponding to the task to be processed, and to acquire current load information of the candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units.

[0169] The priority determination module 902 is used to determine the processing priority corresponding to the task to be processed according to the task information.

[0170] The unit determination module 903 is used to determine the target task processing unit corresponding to the task to be processed from each candidate task processing unit according to the processing priority and the current load information.

[0171] The task processing module 904 is used to allocate the pending tasks to the corresponding target task processing units so that the target task processing units process the pending tasks; the processing status of the pending tasks is used to update the number of candidate task processing units.

[0172] In an exemplary embodiment, the unit determination module 903 is also used to obtain the correspondence between the processing priority and the load information range; according to the processing priority, the correspondence is queried to obtain the target load information range corresponding to the processing priority; from each candidate task processing unit, the candidate task processing unit whose corresponding current load information is within the target load information range is determined as the target task processing unit corresponding to the task to be processed.

[0173] In an exemplary embodiment, the task processing device also includes a quantity update module, which is used to monitor the processing status of the task to be processed through a preset non-blocking input and output interface; when it is monitored that the processing status is completed, obtain the log information corresponding to the task to be processed; and update the number of candidate task processing units according to the log information.

[0174] In an exemplary embodiment, the quantity updating module is further used to extract the task processing time in the log information; and update the quantity of the candidate task processing units according to the difference between the task processing time and the preset processing time.

[0175] In an exemplary embodiment, the quantity update module is also used to update the number of candidate task processing units when the task processing time is greater than the preset processing time; or to maintain the number of candidate task processing units when the task processing time is less than or equal to the preset processing time.

[0176] In an exemplary embodiment, the task processing device also includes an information deletion module, which is used to write log information into a buffer; check and process the occupied space of the log information in the buffer to obtain an inspection result corresponding to the buffer; when the inspection result meets the preset inspection result, delete the log information in the buffer.

[0177] In an exemplary embodiment, the information acquisition module 901 is also used to obtain the task type, task completion time and task data volume corresponding to the task to be processed; the task type, task completion time and task data volume are all used as task information.

[0178] Each module in the above task processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0179] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as task information and current load information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a task processing method is implemented.

[0180] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0182] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0183] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0185] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A task processing method, characterized in that: The method comprises: Acquire task information corresponding to the task to be processed, and acquire current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units; Determining the processing priority corresponding to the task to be processed according to the task information; Determining, from each of the candidate task processing units, a target task processing unit corresponding to the task to be processed according to the processing priority and the current load information; The to-be-processed tasks are allocated to corresponding target task processing units, so that the target task processing units process the to-be-processed tasks; and the processing status of the to-be-processed tasks is used to update the number of the candidate task processing units.

2. The method according to claim 1, characterized in that The step of determining a target task processing unit corresponding to the task to be processed from each of the candidate task processing units according to the processing priority and the current load information includes: Obtaining the correspondence between the processing priority and the load information range; According to the processing priority, query the corresponding relationship to obtain the target load information range corresponding to the processing priority; From the candidate task processing units, a candidate task processing unit whose corresponding current load information is within the target load information range is determined as the target task processing unit corresponding to the task to be processed.

3. The method according to claim 1, characterized in that After allocating the to-be-processed task to the corresponding target task processing unit so that the target task processing unit processes the to-be-processed task, the method further includes: Monitoring the processing status of the task to be processed through a preset non-blocking input and output interface; When it is monitored that the processing status is completed, obtaining log information corresponding to the task to be processed; The number of the candidate task processing units is updated according to the log information.

4. The method according to claim 3, characterized in that The updating of the number of the candidate task processing units according to the log information includes: Extracting the task processing time in the log information; The number of the candidate task processing units is updated according to the difference between the task processing time and the preset processing time.

5. The method according to claim 4, characterized in that The updating of the number of the candidate task processing units according to the difference between the task processing time and the preset processing time includes: When the task processing time is greater than the preset processing time, updating the number of the candidate task processing units; or, When the task processing time is less than or equal to the preset processing time, the number of the candidate task processing units is maintained.

6. The method according to claim 3, characterized in that: The method further comprises: Writing the log information into a buffer; Checking and processing the occupied space of the log information in the buffer to obtain a check result corresponding to the buffer; When the inspection result meets the preset inspection result, the log information in the buffer is deleted.

7. The method according to any one of claims 1 to 6, characterized in that: The obtaining of task information corresponding to the task to be processed includes: Obtaining the task type, task completion time and task data volume corresponding to the task to be processed; The task type, the task completion time and the task data volume are all used as the task information.

8. A task processing device, characterized in that: The device comprises: An information acquisition module, used to acquire task information corresponding to the task to be processed, and to acquire current load information of a candidate task processing unit associated with the task to be processed; the candidate task processing unit includes at least two thread units and two process units; A priority determination module, used to determine the processing priority corresponding to the task to be processed according to the task information; A unit determination module, used to determine the target task processing unit corresponding to the task to be processed from each of the candidate task processing units according to the processing priority and the current load information; The task processing module is used to allocate the pending tasks to the corresponding target task processing units so that the target task processing units process the pending tasks; the processing status of the pending tasks is used to update the number of the candidate task processing units.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images