Log task processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a log task processing method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a task pulling request sent by a task management node; wherein the task pulling request carries a target area identifier and allocated task information; according to the target region identifier and region identifiers corresponding to the candidate log tasks, selecting a region task matched with the target region identifier from the candidate log tasks; updating a heartbeat parameter of a first log task associated with the distributed task information in the regional task; wherein the parameter value change of the heartbeat parameter is used for representing the distribution condition of the task; according to the number of task management nodes, node identifiers of the task management nodes and task identifiers of other tasks except the first log task in the regional tasks, selecting a second log task from the other tasks; and issuing the first log task and the second log task to the task management node. By adopting the method, the log task can be responded in time.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log task processing method, device, equipment, storage medium and program product. Background Art

[0002] As cloud services continue to grow, the demand for high-performance elastic computing service nodes is increasing day by day, and more and more abnormal scenarios are coming. Log files are records of application system operation events or file collections, including the operation logs and fault logs of the entire application system, and are composed of multiple log records. Therefore, the recording and reading of log files are crucial for the maintenance of application systems and the detection of problems.

[0003] Currently, the efficiency of operations such as writing (recording) and reading (querying) log files is low, especially when the log files are large, which can easily cause serious queues of read and write tasks and system paralysis, thereby affecting tasks such as writing and reading log files. Summary of the invention

[0004] Based on this, it is necessary to provide a log task processing method, device, equipment, storage medium and program product to address the above technical problems, so as to respond to log tasks in a timely manner.

[0005] In a first aspect, the present application provides a log task processing method, which is applied to a log management server, and the method includes:

[0006] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0007] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0008] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0009] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0010] Send the first log task and the second log task to the task management node.

[0011] In one embodiment, selecting a second log task from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional task except the first log task includes:

[0012] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task, select the task to be assigned from the remaining tasks; update the heartbeat parameters of the task to be assigned, and use the task to be assigned whose heartbeat parameters are successfully updated as the second log task.

[0013] In one embodiment, selecting the task to be assigned from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional task except the first log task includes:

[0014] For each of the remaining tasks, the task identifier of the task is modulo the number of task management nodes to obtain a remainder; when the remainder is equal to the node identifier of the task management node, the task is regarded as a task to be assigned.

[0015] In one embodiment, updating the heartbeat parameter of the first log task associated with the assigned task information in the regional task includes:

[0016] From the regional tasks, select a first log task associated with the assigned task information; and update the value of the heartbeat parameter of the first log task to the receiving time of the task pull request.

[0017] In a second aspect, the present application provides a log task processing method, which is applied to a task management node, and the method includes:

[0018] A task pull request is sent to the log management server; wherein the task pull request carries the target region identifier and the assigned task information;

[0019] Receive the first log task and the second log task issued by the log management server; wherein the first log task is a task in the regional task associated with the assigned task information, and the regional task is a regional task in the candidate log tasks that matches the target regional identifier; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional task except the first log task.

[0020] In one embodiment, it also includes:

[0021] The local task list is updated according to the first log task and the second log task.

[0022] In one embodiment, it also includes:

[0023] Call the coroutine in the coroutine pool to execute the tasks in the local task list.

[0024] In one embodiment, it also includes:

[0025] Determine the ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool; if the ratio is greater than the set threshold, expand the capacity of the task management node.

[0026] In a third aspect, the present application further provides a log task processing device, the device comprising:

[0027] An acquisition module is used to acquire a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0028] A first selection module, configured to select a regional task matching the target regional identifier from the candidate log tasks according to the target regional identifier and the regional identifier corresponding to the candidate log tasks;

[0029] A heartbeat update module, used to update the heartbeat parameters of the first log task associated with the assigned task information in the regional task; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0030] A second selection module, configured to select a second log task from the remaining tasks according to the number of task management nodes, node identifiers of the task management nodes, and task identifiers of the remaining tasks in the regional tasks except the first log task;

[0031] The task sending module is used to send the first log task and the second log task to the task management node.

[0032] In a fourth aspect, the present application also provides a log task processing device, the device comprising:

[0033] A sending request module is used to send a task pull request to the log management server; wherein the task pull request carries a target area identifier and assigned task information;

[0034] The task receiving module is used to receive the first log task and the second log task issued by the log management server; wherein the first log task is a task associated with the assigned task information in the regional task, and the regional task is a regional task that matches the target regional identifier in the candidate log tasks; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifier of the task management node, and the task identifier of the remaining tasks in the regional task except the first log task.

[0035] In a fifth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0037] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0038] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0039] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0040] Send the first log task and the second log task to the task management node.

[0041] In a sixth aspect, the present application further provides another computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0042] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0043] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0044] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0045] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0046] Send the first log task and the second log task to the task management node.

[0047] In a seventh aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0048] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0049] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0050] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0051] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0052] Send the first log task and the second log task to the task management node.

[0053] In an eighth aspect, the present application further provides another computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0054] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0055] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0056] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0057] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0058] Send the first log task and the second log task to the task management node.

[0059] In a ninth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0060] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0061] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0062] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0063] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0064] Send the first log task and the second log task to the task management node.

[0065] In a tenth aspect, the present application further provides another computer program product, including a computer program, which implements the following steps when executed by a processor:

[0066] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0067] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0068] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0069] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0070] Send the first log task and the second log task to the task management node.

[0071] The above-mentioned log task processing method, device, equipment, storage medium and program product, by obtaining the task pull request sent by the task management node carrying the target area identifier and the assigned task information, lays a basis for issuing the log task to the task management node; further, according to the target area identifier and the area identifier corresponding to the candidate log task, the regional task matching the target area identifier can be selected from the candidate log tasks, and the heartbeat parameters of the first log task associated with the assigned task information in the regional task can be updated, and according to the number of task management nodes, the node identifier of the task management node, and the task identifier of the remaining tasks in the regional task except the first log task, the second log task can be selected from the remaining tasks, which provides a way to accurately issue the corresponding log task to the task management node; finally, the first log task and the second log task are issued to the task management node, so as to realize accurate and fast processing of the log tasks. This scheme provides a way to select log tasks suitable for task management nodes from a large number of candidate log tasks by introducing target area identifiers; secondly, by introducing heartbeat parameters, it provides a method for timely updating the processing status of each log task and facilitates the viewing of the processing status of each log task; finally, through information such as the number of task management nodes and the node identifiers of the task management nodes, it is possible to evenly distribute each log task to each task management node, so as to ultimately achieve timely response to user requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 An application environment diagram of a log task processing method provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of a log task processing provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of a process for selecting a second log task provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of another log task processing method provided in an embodiment of the present application;

[0077] Figure 5 A structural block diagram of a log task processing device provided in an embodiment of the present application;

[0078] Figure 6 A structural block diagram of another log task processing device provided in an embodiment of the application;

[0079] Figure 7 This is a diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0081] The log task processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in the figure, the log management server can accept the user's log processing requests, such as log delivery, log download and log processing, and save the processing request tasks of each user to the database; the task management node of each preset area can send a task pull request to the log management server to execute the processing request tasks of each user.

[0082] In an exemplary embodiment, Figure 2 As shown, a log task processing method is provided, which is applied to Figure 1 Taking the log management server in as an example, the following steps can be included:

[0083] S201, obtaining a task pull request sent by a task management node.

[0084] The task pull request carries the target area identifier and the assigned task information.

[0085] Optionally, the regional location of the task management node can be determined based on the target regional identifier, and the log tasks of the corresponding regional location can be assigned to the task management nodes of the same regional location to speed up the processing of the log tasks; the assigned task information can include the list of running tasks corresponding to the task management node, the current remaining executable space corresponding to the task management node, the node identifier of the task management node, and the number of task management nodes, etc.

[0086] Exemplarily, the communication module may receive the task pull request sent by each task management node, extract the information in the task pull request to obtain key information, and assign the corresponding log task to each task management node.

[0087] S202: Selecting a regional task matching the target regional identifier from the candidate log tasks according to the target regional identifier and the regional identifier corresponding to the candidate log tasks.

[0088] The candidate log tasks may represent the log tasks stored in the database of the log management server; the regional tasks may represent the tasks whose regional identifiers are the target regional identifiers among the candidate log tasks.

[0089] Exemplarily, the region identifier of each candidate log task may be compared with the target region identifier, and the candidate log tasks having the same region identifier as the target region identifier may be screened out and determined as the region tasks.

[0090] S203: Update the heartbeat parameter of the first log task associated with the assigned task information in the regional task.

[0091] The first log task may represent a task in a regional task that has been assigned to the task management node; and a parameter value change of the heartbeat parameter is used to characterize the distribution of the task.

[0092] Optionally, each task in the regional task can be compared with the list of running tasks corresponding to the task management node in the assigned task information, and the repeated task (that is, the task exists both in the regional task and in the list of running tasks corresponding to the task management node) can be determined as the first log task; for example, the heartbeat parameter is updateTime, which indicates the latest update time of the log task. If the time distance between updateTime and the current time exceeds a time threshold (such as 5 minutes), it indicates that the log task has not been assigned to any task management node; if the time distance between updateTime and the current time is less than or equal to the time threshold (such as 5 minutes), it indicates that the log task has been assigned.

[0093] Exemplarily, the tasks assigned to the task management node (ie, the first log task) are screened out from the regional tasks. To prevent the log tasks from being assigned repeatedly, the heartbeat parameters of the first log task are updated, such as updating the heartbeat parameters of the first log task to the current time.

[0094] S204, selecting a second log task from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task.

[0095] Among them, the number of task management nodes can represent the number of nodes executing log tasks; the node identifier of the task management node can uniquely identify the node, for example, it can be obtained by sorting the start time of each node; the second log task can represent the log task that has not yet been assigned in the regional task.

[0096] Exemplarily, the number of task management nodes, node identifiers of task management nodes, and information of other tasks in the regional tasks except the first log task can be input into the log allocation model, and the log allocation model allocates them to screen out the second log task again.

[0097] S205: Send the first log task and the second log task to the task management node.

[0098] Among them, the task management node can execute the log tasks issued by the log management server.

[0099] Exemplarily, the first log task and the second log task may be sent to the task management node through the communication module to quickly respond to the user's request.

[0100] The above-mentioned log task processing method, by obtaining the task pull request sent by the task management node carrying the target area identifier and the assigned task information, lays a basis for issuing the log task to the task management node; further, according to the target area identifier and the area identifier corresponding to the candidate log task, the regional task matching the target area identifier can be selected from the candidate log tasks, and the heartbeat parameter of the first log task associated with the assigned task information in the regional task can be updated, and according to the number of task management nodes, the node identifier of the task management node, and the task identifier of the remaining tasks in the regional task except the first log task, the second log task can be selected from the remaining tasks, which provides a way to accurately issue the corresponding log task to the task management node; finally, the first log task and the second log task are issued to the task management node, so as to realize accurate and fast processing of the log tasks. This scheme provides a way to select log tasks suitable for task management nodes from a large number of candidate log tasks by introducing target area identifiers; secondly, by introducing heartbeat parameters, it provides a method for timely updating the processing status of each log task and facilitates the viewing of the processing status of each log task; finally, through information such as the number of task management nodes and the node identifiers of the task management nodes, it is possible to evenly distribute each log task to each task management node, so as to ultimately achieve timely response to user requests.

[0101] Based on the above embodiment, the present embodiment of the present application explains the above embodiment S204 in detail. Specifically, the present embodiment of the present application involves the process of selecting the second log task, such as Figure 3 As shown, the specific steps include:

[0102] S301, selecting a task to be assigned from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task.

[0103] It should be noted that when each task management node is started, it registers its own information with Nacos (Dynamic Naming and Configuration Service), obtains and sorts all instance information of the node (such as node identification) through the Nacos API (Application Programming Interface), and injects the node sorting information into each task management node through the Kubernetes interface.

[0104] One possible implementation method is to perform a modulo operation on the task identifier of each of the remaining tasks and the number of task management nodes to obtain a remainder; when the remainder is equal to the node identifier of the task management node, the task is treated as a task to be assigned.

[0105] For example, in order to evenly distribute the log task to each node, the task identifier of the log task can be modulo the number of task management nodes, and the node to which it is assigned can be determined based on the remainder; for example, if a task identifier is 7, the number of task management nodes is 2, and the node identifiers of the task management nodes are 1 and 2 respectively, then the task identifier is 7 and the number of task management nodes is 2, and the remainder is 1, then the log task is assigned to task management node 1.

[0106] S302, updating the heartbeat parameters of the task to be assigned, and taking the task to be assigned with successfully updated heartbeat parameters as the second log task.

[0107] As shown in the above example, the heartbeat parameter of the task to be assigned can be directly updated to the current time, and the heartbeat parameter of the task to be assigned can also be updated to the time when the log management server processes the task pull request sent by the task management node to prevent other task management nodes from preempting the log task again.

[0108] In the embodiment of the present application, by introducing the heartbeat parameter, the status update of the log task is realized, laying the foundation for the efficient execution of the log task.

[0109] Based on the above embodiment, the embodiment of the present application explains the above embodiment S203 in detail. Specifically, the process of updating the heartbeat parameter of the first log task in the embodiment of the present application specifically includes: selecting the first log task associated with the assigned task information from the regional tasks; updating the value of the heartbeat parameter of the first log task to the receiving time of the task pull request.

[0110] Exemplarily, the time when the log management server receives the task pull request can be obtained, and the heartbeat parameter (such as the updateTime parameter) of the first log task can be directly updated to the receiving time of the task pull request.

[0111] In the embodiment of the present application, by determining the reception time of the task pull request as the update value of the heartbeat parameter of the first log task, a method for updating the heartbeat parameter is provided.

[0112] In an exemplary embodiment, Figure 4 As shown, a log task processing method is provided, which is applied to Figure 1 Taking the task management node in as an example, the following steps can be included:

[0113] S401, sending a task pull request to the log management server.

[0114] The task pull request carries the target area identifier and the assigned task information. Optionally, the target area identifier and the assigned task information can refer to the explanation of the above step S201.

[0115] Exemplarily, a task pull request may be sent to the log management server through a communication module or an internal dedicated line to quickly respond to the log request raised by the user.

[0116] S402, receiving a first log task and a second log task issued by a log management server.

[0117] Among them, the first log task is a task associated with the assigned task information in the regional task, and the regional task is a regional task that matches the target regional identifier in the candidate log tasks; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifier of the task management node, and the task identifier of the remaining tasks in the regional task except the first log task.

[0118] Exemplarily, the first log task and the second log task issued by the log management server can be received through the communication module or the internal dedicated line, and the log tasks can be processed in time according to the types of the log tasks (such as log delivery, log downloading and log processing, etc.).

[0119] In the embodiment of the present application, by introducing the interaction process between the task management node and the log management server, the processing process of the log task is made clearer and more efficient.

[0120] Based on the above embodiment, the present application embodiment explains the above embodiment S402 in detail. Specifically, the present application embodiment involves a process of updating a local task list, which specifically includes: updating the local task list according to the first log task and the second log task.

[0121] Exemplarily, the log tasks in the local task list can be updated by traversing the first log task and the second log task, filtering out the log tasks that are not in the local task list, and adding them to the end of the local task list in sequence.

[0122] In the embodiment of the present application, efficient processing of log tasks is achieved by updating the local task list.

[0123] Based on the above embodiment, the embodiment of the present application explains the above embodiment S402 in detail. Specifically, the embodiment of the present application involves the process of executing tasks in the local task list, which specifically includes: calling the coroutine in the coroutine pool to execute the tasks in the local task list.

[0124] Among them, the coroutine pool can be a coroutine pool in the Go language, that is, a golang coroutine pool, which can avoid excessive consumption of system resources (such as CPU and memory).

[0125] Optionally, if the number of coroutines in a node's coroutine pool is 10, the node will run a maximum of 10 tasks; then, the node will execute each task in the local task list by calling the coroutines in the coroutine pool according to the order of the tasks in the local task list.

[0126] In the embodiment of the present application, by calling the coroutine pool, the problem of filling up the running resource pool when the log tasks increase infinitely is avoided, the execution of the log tasks can be realized quickly, and the processing speed of the log tasks is accelerated.

[0127] Based on the above-mentioned embodiment, the embodiment of the present application explains the above-mentioned embodiment S402 in detail. Specifically, the embodiment of the present application involves the process of expanding the capacity of the task management node, which specifically includes: determining the ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool; when the ratio is greater than the set threshold, expanding the capacity of the task management node.

[0128] The threshold value can be pre-set according to the requirements of response speed, etc.; the expansion processing can increase the total number of coroutines in the coroutine pool by increasing the number of nodes to execute more log tasks to be processed.

[0129] Exemplarily, the ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool is calculated; if the ratio is greater than a set threshold, the task management node can be expanded through the Kubernetes interface to alleviate resource shortages or task starvation.

[0130] In the embodiments of the present application, a threshold is introduced to provide a data basis for judging resource shortage conditions, and by monitoring the number of tasks and the number of coroutines, a foundation is laid for discovering resource shortages in advance and taking timely countermeasures, thereby ultimately achieving timely response to user requests.

[0131] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0132] Based on the same inventive concept, the embodiment of the present application also provides a log task processing device for implementing the log task processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more log task processing device embodiments provided below can refer to the limitations of the log task processing method above, and will not be repeated here.

[0133] In an exemplary embodiment, Figure 5 As shown, a log task processing device 1 is provided, comprising: an acquisition module 10, a first selection module 20, a heartbeat update module 30, a second selection module 40 and a task issuing module 50, wherein:

[0134] The acquisition module 10 is used to acquire the task pull request sent by the task management node; wherein the task pull request carries the target area identifier and the assigned task information.

[0135] The first selection module 20 is used to select a regional task matching the target regional identifier from the candidate log tasks according to the target regional identifier and the regional identifier corresponding to the candidate log tasks.

[0136] The heartbeat update module 30 is used to update the heartbeat parameters of the first log task associated with the assigned task information in the regional task; wherein the parameter value change of the heartbeat parameter is used to characterize the task allocation situation.

[0137] The second selection module 40 is used to select a second log task from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task.

[0138] The task issuing module 50 is used to issue the first log task and the second log task to the task management node.

[0139] In one embodiment, the second selection module 40 is specifically used for:

[0140] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task, select the task to be assigned from the remaining tasks; update the heartbeat parameters of the task to be assigned, and use the task to be assigned whose heartbeat parameters are successfully updated as the second log task.

[0141] In one embodiment, the second selection module 40 is further configured to:

[0142] For each of the remaining tasks, the task identifier of the task is modulo the number of task management nodes to obtain a remainder; when the remainder is equal to the node identifier of the task management node, the task is regarded as a task to be assigned.

[0143] In one embodiment, the heartbeat update module 30 is specifically used for:

[0144] From the regional tasks, select a first log task associated with the assigned task information; and update the value of the heartbeat parameter of the first log task to the receiving time of the task pull request.

[0145] In an exemplary embodiment, Figure 6 As shown, a log task processing device 2 is provided, comprising: a request sending module 60 and a task receiving module 70, wherein:

[0146] A sending request module 60 is used to send a task pull request to the log management server; wherein the task pull request carries a target area identifier and assigned task information;

[0147] The task receiving module 70 is used to receive the first log task and the second log task issued by the log management server; wherein the first log task is a task associated with the assigned task information in the regional task, and the regional task is a regional task matching the target regional identifier in the candidate log task; the second log task is selected from the remaining tasks according to the number of task management nodes, the node identifier of the task management node, and the task identifier of the remaining tasks in the regional task except the first log task.

[0148] In one embodiment, the log task processing device 2 specifically further includes:

[0149] The updating module is used to update the local task list according to the first log task and the second log task.

[0150] In one embodiment, the log task processing device 2 specifically further includes:

[0151] The execution module is used to call the coroutines in the coroutine pool and execute the tasks in the local task list.

[0152] In one embodiment, the log task processing device 2 specifically further includes:

[0153] The capacity expansion module is used to determine the ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool; when the ratio is greater than the set threshold, the task management node is expanded.

[0154] Each module in the above-mentioned log task processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0155] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store log task data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log task processing method is implemented.

[0156] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0158] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0159] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0160] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0161] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0162] Send the first log task and the second log task to the task management node.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task, select the task to be assigned from the remaining tasks; update the heartbeat parameters of the task to be assigned, and use the task to be assigned whose heartbeat parameters are successfully updated as the second log task.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] For each of the remaining tasks, the task identifier of the task is modulo the number of task management nodes to obtain a remainder; when the remainder is equal to the node identifier of the task management node, the task is regarded as a task to be assigned.

[0167] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0168] From the regional tasks, select a first log task associated with the assigned task information; and update the value of the heartbeat parameter of the first log task to the receiving time of the task pull request.

[0169] In an exemplary embodiment, another computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0170] A task pull request is sent to the log management server; wherein the task pull request carries the target region identifier and the assigned task information;

[0171] Receive the first log task and the second log task issued by the log management server; wherein the first log task is a task in the regional task associated with the assigned task information, and the regional task is a regional task in the candidate log tasks that matches the target regional identifier; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional task except the first log task.

[0172] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0173] The local task list is updated according to the first log task and the second log task.

[0174] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0175] Call the coroutine in the coroutine pool to execute the tasks in the local task list.

[0176] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0177] Determine the ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool; if the ratio is greater than the set threshold, expand the capacity of the task management node.

[0178] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0179] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0180] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0181] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0182] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0183] Send the first log task and the second log task to the task management node.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0185] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task, select the task to be assigned from the remaining tasks; update the heartbeat parameters of the task to be assigned, and use the task to be assigned whose heartbeat parameters are successfully updated as the second log task.

[0186] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0187] For each of the remaining tasks, the task identifier of the task is modulo the number of task management nodes to obtain a remainder; when the remainder is equal to the node identifier of the task management node, the task is regarded as a task to be assigned.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] From the regional tasks, select a first log task associated with the assigned task information; and update the value of the heartbeat parameter of the first log task to the receiving time of the task pull request.

[0190] In one embodiment, another computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0191] A task pull request is sent to the log management server; wherein the task pull request carries the target region identifier and the assigned task information;

[0192] Receive the first log task and the second log task issued by the log management server; wherein the first log task is a task in the regional task associated with the assigned task information, and the regional task is a regional task in the candidate log tasks that matches the target regional identifier; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional task except the first log task.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] The local task list is updated according to the first log task and the second log task.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] Call the coroutine in the coroutine pool to execute the tasks in the local task list.

[0197] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0198] Determine the ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool; if the ratio is greater than the set threshold, expand the capacity of the task management node.

[0199] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0200] Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information;

[0201] According to the target region identifier and the region identifier corresponding to the candidate log task, select a region task matching the target region identifier from the candidate log tasks;

[0202] The heartbeat parameter of the first log task associated with the assigned task information in the regional task is updated; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task;

[0203] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks except the first log task in the regional tasks, a second log task is selected from the remaining tasks;

[0204] Send the first log task and the second log task to the task management node.

[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0206] According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task, select the task to be assigned from the remaining tasks; update the heartbeat parameters of the task to be assigned, and use the task to be assigned whose heartbeat parameters are successfully updated as the second log task.

[0207] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0208] For each of the remaining tasks, the task identifier of the task is modulo the number of task management nodes to obtain a remainder; when the remainder is equal to the node identifier of the task management node, the task is regarded as a task to be assigned.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0210] From the regional tasks, select a first log task associated with the assigned task information; and update the value of the heartbeat parameter of the first log task to the receiving time of the task pull request.

[0211] In one embodiment, another computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0212] A task pull request is sent to the log management server; wherein the task pull request carries the target region identifier and the assigned task information;

[0213] Receive the first log task and the second log task issued by the log management server; wherein the first log task is a task in the regional task associated with the assigned task information, and the regional task is a regional task in the candidate log tasks that matches the target regional identifier; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional task except the first log task.

[0214] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0215] The local task list is updated according to the first log task and the second log task.

[0216] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0217] Call the coroutine in the coroutine pool to execute the tasks in the local task list.

[0218] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0219] Determine the ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool; if the ratio is greater than the set threshold, expand the capacity of the task management node.

[0221] It should be noted that the information (including but not limited to log information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0223] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0224] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A log task processing method, characterized in that: Applied to a log management server, the method includes: Obtaining a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information; According to the target area identifier and the area identifier corresponding to the candidate log task, selecting an area task matching the target area identifier from the candidate log tasks; Updating the heartbeat parameter of the first log task associated with the assigned task information in the regional task; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task; According to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task, selecting a second log task from the remaining tasks; The first log task and the second log task are issued to the task management node.

2. The method according to claim 1, characterized in that The selecting a second log task from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task includes: Selecting a task to be assigned from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task; The heartbeat parameters of the task to be assigned are updated, and the task to be assigned whose heartbeat parameters are successfully updated is used as the second log task.

3. The method according to claim 2, characterized in that The selecting the task to be assigned from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task, comprises: For each of the remaining tasks, performing a modulo operation on the task identifier of the task and the number of task management nodes to obtain a remainder; When the remainder is equal to the node identifier of the task management node, the task is used as a task to be assigned.

4. The method according to claim 1, characterized in that: The updating of the heartbeat parameter of the first log task associated with the assigned task information in the regional task includes: From the regional tasks, select a first log task associated with the assigned task information; The value of the heartbeat parameter of the first log task is updated to the receiving time of the task pull request.

5. A log task processing method, characterized in that: Applied to a task management node, the method comprises: A task pull request sent to the log management server; wherein the task pull request carries a target area identifier and assigned task information; Receive a first log task and a second log task issued by the log management server; wherein the first log task is a task in a regional task associated with the assigned task information, and the regional task is a regional task in a candidate log task that matches the target regional identifier; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifier of the task management node, and the task identifier of the remaining tasks in the regional task except the first log task.

6. The method according to claim 5, characterized in that The method further comprises: The local task list is updated according to the first log task and the second log task.

7. The method according to claim 6, characterized in that The method further comprises: Call the coroutine in the coroutine pool to execute the tasks in the local task list.

8. The method according to claim 7, characterized in that The method further comprises: Determine a ratio between the number of tasks in the local task list and the number of coroutines in the coroutine pool; When the ratio is greater than a set threshold, the task management node is expanded.

9. A log task processing device, characterized in that: The device comprises: An acquisition module, used to acquire a task pull request sent by a task management node; wherein the task pull request carries a target area identifier and assigned task information; A first selection module, configured to select a regional task matching the target regional identifier from the candidate log tasks according to the target regional identifier and the regional identifier corresponding to the candidate log tasks; A heartbeat update module, used for updating the heartbeat parameters of the first log task associated with the assigned task information in the regional task; wherein the parameter value change of the heartbeat parameter is used to characterize the assignment of the task; a second selection module, configured to select a second log task from the remaining tasks according to the number of task management nodes, the node identifiers of the task management nodes, and the task identifiers of the remaining tasks in the regional tasks except the first log task; The task issuing module is used to issue the first log task and the second log task to the task management node.

10. A log task processing device, characterized in that: The device comprises: A sending request module, used for sending a task pull request to the log management server; wherein the task pull request carries a target area identifier and assigned task information; A task receiving module is used to receive a first log task and a second log task issued by the log management server; wherein the first log task is a task in a regional task associated with the assigned task information, and the regional task is a regional task in a candidate log task that matches the target regional identifier; the second log task is selected from the remaining tasks based on the number of task management nodes, the node identifier of the task management node, and the task identifier of the remaining tasks in the regional task except the first log task.

Images