Task processing method, device, computer equipment and storage medium

Abstract

The present application relates to the field of artificial intelligence technology, and provides a task processing method, device, computer equipment and storage medium, the method comprising: performing abnormal analysis on all servers contained in a server cluster to determine abnormal servers, and marking servers other than abnormal servers as normal servers; determining a target server from all normal servers; obtaining pending tasks from a data source through the target server; performing deduplication processing on all pending tasks based on a cache tool to obtain processed target tasks; storing the target tasks in a preset message queue, and assigning all target tasks in the message queue to each normal server in the server cluster through a message middleware. The present application can ensure the normal processing of tasks, improve the processing efficiency of tasks, and improve the intelligence and accuracy of task processing. The present application can also be applied to the field of blockchain, and the above-mentioned target tasks and other data can be stored on the blockchain.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a task processing method, apparatus, computer equipment and storage medium. Background Art

[0002] In the prior art, when it comes to large-scale task processing, the main method is to deploy and process on a single machine. With the development of business and the continuous expansion of data scale, the single-machine deployment task batch running solution has the following problems: a single server is used to execute large-scale tasks, resulting in slow task execution speed, long task execution time and low efficiency; and if the server fails, it will affect the execution of all tasks under the server, and all tasks executed on the server will fail, resulting in a waste of system resources and making it impossible to ensure the normal processing of tasks. Summary of the invention

[0003] The main purpose of this application is to provide a task processing method, device, computer equipment and storage medium, aiming to solve the technical problems of existing large-scale task processing methods, such as slow task execution speed, long task execution time, low efficiency, and the inability to ensure normal task processing.

[0004] The present application proposes a task processing method, which comprises the following steps:

[0005] Performing an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster, and marking the servers in the server cluster except the abnormal server as normal servers; wherein the number of the servers includes a plurality;

[0006] Determine a target server from all the normal servers;

[0007] Obtaining the tasks to be processed from a preset data source through the target server; wherein the number of the tasks to be processed includes multiple;

[0008] Deduplication processing is performed on all the tasks to be processed based on a preset cache tool to obtain a processed target task;

[0009] The target task is stored in a preset message queue, and all the target tasks in the message queue are distributed to the normal servers in the server cluster through the message middleware, so that the received target tasks are processed by the normal servers.

[0010] Optionally, the step of performing an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster includes:

[0011] Obtaining the operating parameters of each of the servers and obtaining a preset parameter threshold; wherein the preset parameter threshold is a comprehensive threshold corresponding to all of the servers; performing numerical comparison processing on the operating parameters of each of the servers and the preset parameter threshold, filtering out the first server whose operating parameters meet the preset parameter threshold from all of the servers, and marking all of the servers except the first server as the second server;

[0012] Acquire the operating parameters of each second server, and generate an operating abnormality probability value of each second server based on the operating parameters of each second server and a preset abnormality recognition model;

[0013] Obtaining a preset probability threshold; wherein the preset probability threshold is a comprehensive threshold corresponding to all the servers;

[0014] Compare the operation abnormality probability value of each second server with the preset probability threshold, and select a third server whose operation abnormality probability value is less than the preset probability threshold from all the second servers;

[0015] The first server and the third server are used as the abnormal servers.

[0016] Optionally, the step of obtaining the operating parameters of each second server and generating an operating abnormality probability value of each second server based on the operating parameters of each second server and a preset abnormality recognition model includes:

[0017] Calculating an average value of the operating parameters of all the second servers; and,

[0018] Calculating the variance of the operating parameters of all the second servers;

[0019] Obtaining designated operating parameters of a first designated server; wherein the first designated server is any one of all the second servers;

[0020] The mean value, the variance, and the designated operation parameter are input into the abnormality identification model, and an operation abnormality probability value corresponding to the first designated server is generated through the abnormality identification model.

[0021] Optionally, the step of performing deduplication processing on all the tasks to be processed based on a preset cache tool to obtain a processed target task includes:

[0022] Step A: sorting all the tasks to be processed in order from the beginning to the end according to the collection time of each task to be processed, and obtaining a corresponding sorting result;

[0023] Step B: obtaining the first task ranked first in the sorting result, and extracting the first task field included in the first task;

[0024] Step C: performing splicing processing on all the first task fields to obtain corresponding first splicing fields, storing the first splicing fields in the cache tool, and setting a first timeout period for the first splicing fields;

[0025] Step D: obtaining the second task ranked at the i-th position in the sorting result, performing splicing processing on all second task fields contained in the second task to obtain a corresponding second splicing field, and determining whether there is a first designated field identical to the second splicing field in the cache tool; wherein i is an integer and the initial value of i is 2;

[0026] Step E: if the first designated field does not exist, the second concatenated field is stored in the cache tool, and a second timeout period is set for the second concatenated field; otherwise, the second concatenated field is not stored;

[0027] Step F: let i=i+1, and repeat steps D-E until i=n, ​​so as to complete the processing of all tasks included in the sorting result, and obtain all target splicing fields stored in the cache tool; wherein n is the number of all the tasks to be processed;

[0028] Step G: Filter out the tasks corresponding to the target splicing field from all the tasks to be processed to obtain the target task.

[0029] Optionally, the step of storing the target task in a preset message queue, and distributing all the target tasks in the message queue to each of the normal servers in the server cluster through a message middleware includes:

[0030] Obtaining the collection time information, processing timeliness information and importance information of each target task;

[0031] Generate a processing order for each of the target tasks based on the collection time information, the processing timeliness information and the importance information;

[0032] storing all the target tasks in the message queue according to the processing order;

[0033] All the target tasks in the message queue are distributed to the normal servers in the server cluster through the message middleware according to the processing order.

[0034] Optionally, the step of generating a processing order of each of the target tasks based on the collection time information, the processing timeliness information and the importance information includes:

[0035] Acquire a first preset weight corresponding to the collection time information, a second preset weight corresponding to the processing timeliness information, and a third preset weight corresponding to the importance information;

[0036] Based on the collection time information, processing timeliness information, importance information, the first preset weight, the second preset weight, and the third preset weight, a priority value of each of the target tasks is calculated by a preset calculation formula;

[0037] Sorting all the target tasks in descending order according to the priority values ​​to obtain target sorting information corresponding to all the target tasks;

[0038] A processing order of each of the target tasks is generated based on all the target sorting information.

[0039] Optionally, after the step of storing the target task in a preset message queue and allocating all the target tasks in the message queue to each of the normal servers in the server cluster through a message middleware, the method further includes:

[0040] Receiving, through the message middleware, a task processing result obtained by processing the received designated task and sent by the second designated server; wherein the second designated server is any one of all the normal servers, and the designated task is any one of all the target tasks received by the designated server;

[0041] If the task processing result is normal, obtaining a specified splicing field corresponding to the specified task in the cache tool, and adjusting a specified timeout time corresponding to the specified splicing field so that the adjusted specified timeout time is greater than the specified timeout time;

[0042] If the task processing result is a processing exception, deleting the specified splicing field corresponding to the specified task from the cache tool;

[0043] The designated task is placed in a preset abnormal task list.

[0044] The present application also provides a task processing device, comprising:

[0045] A first determination module is used to perform an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster, and mark the servers in the server cluster except the abnormal server as normal servers; wherein the number of the servers includes multiple servers;

[0046] A second determination module is used to determine a target server from all the normal servers;

[0047] An acquisition module, used for acquiring tasks to be processed from a preset data source through the target server; wherein the number of the tasks to be processed includes multiple;

[0048] A first processing module, used for performing deduplication processing on all the tasks to be processed based on a preset cache tool to obtain a processed target task;

[0049] The second processing module is used to store the target task in a preset message queue, and distribute all the target tasks in the message queue to the normal servers in the server cluster through the message middleware, so that the received target tasks can be processed by each normal server.

[0050] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0051] The present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0052] The task processing method, apparatus, computer device and storage medium provided in this application have the following beneficial effects:

[0053] The task processing method, device, computer equipment and storage medium provided in the present application, before processing the task to be processed, first perform abnormal analysis on all servers contained in the preset server cluster and determine the abnormal server in the server cluster, then determine a target server from all normal servers in the server cluster except the abnormal server, and obtain the task to be processed from the preset data source through the target server, then perform deduplication processing on all the tasks to be processed based on the preset cache module to obtain the processed target task, and finally store the target task in the preset message queue, and distribute all the target tasks in the message queue to each of the normal servers in the server cluster through the message middleware, so that the received target tasks are processed by each normal server. The present application uses the normal servers in the server cluster to process the task to be processed, and the normal server can also execute the task while distributing the task, which can maximize the use of resources and improve the processing efficiency of the task. By using only normal servers for task processing, it is beneficial to avoid the situation where the task cannot be processed normally due to the use of abnormal servers to process the task, effectively improve the operation stability of the server cluster and ensure the normal processing of the task, and improve the user experience. Before processing a task, the system will use a cache tool to check whether there are duplicate tasks among all pending tasks, thereby accurately and quickly completing the deduplication processing of all pending tasks, effectively avoiding the repeated execution of the same task, reducing useless power consumption of the device, and improving the intelligence and accuracy of task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of a task processing method according to an embodiment of the present application;

[0055] Figure 2 is a structural diagram of a task processing device according to an embodiment of the present application;

[0056] Figure 3 It is a schematic diagram of the structure of a computer device according to an embodiment of the present application.

[0057] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and unless specifically defined as here, will not be interpreted with idealized or overly formal meanings.

[0059] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0061] Reference Figure 1 , a task processing method according to an embodiment of the present application includes:

[0062] S10: performing an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster, and marking the servers in the server cluster except the abnormal server as normal servers; wherein the number of the servers includes a plurality;

[0063] S20: Determine a target server from all the normal servers;

[0064] S30: Obtaining a task to be processed from a preset data source through the target server; wherein the number of the task to be processed includes multiple;

[0065] S40: performing deduplication processing on all the tasks to be processed based on a preset cache tool to obtain a processed target task;

[0066] S50: storing the target task in a preset message queue, and distributing all the target tasks in the message queue to the normal servers in the server cluster through the message middleware, so that the received target tasks are processed by the normal servers.

[0067] As described in the above steps S10 to S50, the execution subject of the embodiment of the present method is a task processing device. In practical applications, the above task processing device can be implemented by a virtual device, such as a software code, or by a physical device written or integrated with relevant execution codes, and can interact with the user through a keyboard, mouse, remote control, touchpad or voice control device. The task processing device in this embodiment can ensure the normal processing of tasks, improve the processing efficiency of tasks, and improve the intelligence and accuracy of task processing. Specifically, firstly, all servers included in the preset server cluster are analyzed for abnormalities to determine the abnormal servers in the server cluster, and the servers in the server cluster except the abnormal servers are marked as normal servers. Among them, the number of the servers includes multiple. In addition, the specific implementation process of performing abnormal analysis on all servers included in the preset server cluster to determine the abnormal servers in the server cluster will be further described in the subsequent specific embodiments of this application, and will not be repeated here.

[0068] Then a target server is determined from all the normal servers. Among them, the zookeeper distributed lock can be used to implement the election, and a server is generated from all the normal servers contained in the server cluster as the master, that is, the target server. As the data producer of the task, this target server is responsible for reading the corresponding task data from the data source storing the task data and putting it into the message queue. Other normal servers in the server cluster, except the target server, do not play the role of data production. Only when the target server is abnormal and the zookeeper distributed lock triggers the re-election of the master, other normal servers will have the opportunity to be selected as the master. When the target server fails, the zookeeper distributed lock will automatically trigger the master selection fault tolerance, so as not to affect the task batch running process of the entire server cluster, which is conducive to ensuring the normal processing of the task. After that, the target server obtains the pending tasks from the preset data source. Among them, the number of pending tasks includes multiple. In addition, the data source can be a related business system, such as a fixed investment business system. The pending tasks can be fixed investment tasks corresponding to the customer's fixed investment plan. Each pending task can be composed of multiple fields.

[0069] Subsequently, all the tasks to be processed are deduplicated based on the preset cache tool to obtain the processed target task. Wherein, the specific one may be redis cache. In addition, after obtaining all the tasks to be processed, the task fields contained in each task to be processed can be spliced ​​to generate spliced ​​fields corresponding to each task to be processed, and then all the tasks to be processed can be checked based on the use of spliced ​​fields and cache tools to see if there are duplicate tasks, so that the deduplication of all the tasks to be processed can be completed accurately and quickly, effectively avoiding the repeated execution of the same tasks. In addition, for the specific implementation process of deduplicating all the tasks to be processed based on the preset cache tool to obtain the processed target task, this application will be further described in the subsequent specific embodiments, which will not be repeated here. Finally, the target task is stored in a preset message queue, and all the target tasks in the message queue are distributed to each normal server in the server cluster through the message middleware, so that the received target tasks are processed by each normal server. Wherein, the message middleware may be RABBITMQ. After placing the pending tasks into the message queue, the message middleware RABBITMQ will evenly distribute the tasks in the message queue to the corresponding consumers, that is, all normal servers in the server cluster, according to the order of the tasks in the message queue, and each consumer will not be assigned duplicate tasks. In addition, when the processing speed needs to be increased, the server cluster can also be expanded. The startup of the new server will trigger the re-election of the master, and it will automatically connect to the message queue to become a new consumer and automatically participate in the consumption processing of the task.

[0070] In this embodiment, before processing the task to be processed, firstly, all servers included in the preset server cluster are analyzed for abnormality and the abnormal server in the server cluster is determined, then a target server is determined from all normal servers in the server cluster except the abnormal server, and the task to be processed is obtained from the preset data source through the target server, then all the tasks to be processed are deduplicated based on the preset cache module to obtain the processed target task, and finally the target task is stored in the preset message queue, and all the target tasks in the message queue are assigned to each of the normal servers in the server cluster through the message middleware, so that the received target tasks are processed by each normal server. In this embodiment, the normal server in the server cluster is used to process the task to be processed, and the normal server can also execute the task while distributing the task, so as to maximize the use of resources and improve the processing efficiency of the task. By using only normal servers to process the task, it is beneficial to avoid the situation that the task cannot be processed normally due to the abnormal server being used to process the task, effectively improving the operation stability of the server cluster and ensuring the normal processing of the task, and improving the user experience. Before processing a task, the system will use a cache tool to check whether there are duplicate tasks among all pending tasks, thereby accurately and quickly completing the deduplication processing of all pending tasks, effectively avoiding the repeated execution of the same task, reducing useless power consumption of the device, and improving the intelligence and accuracy of task processing.

[0071] Furthermore, in one embodiment of the present application, the above step S10 includes:

[0072] S100: Obtaining the operating parameters of each of the servers, and obtaining a preset parameter threshold; wherein the preset parameter threshold is a comprehensive threshold corresponding to all of the servers;

[0073] S101: performing numerical comparison processing on the operating parameters of each of the servers and the preset parameter threshold, selecting a first server whose operating parameters meet the preset parameter threshold from all the servers, and marking all the servers except the first server as second servers;

[0074] S102: Acquire operating parameters of each second server, and generate an operating abnormality probability value of each second server based on the operating parameters of each second server and a preset abnormality recognition model;

[0075] S103: Obtain a preset probability threshold; wherein the preset probability threshold is a comprehensive threshold corresponding to all the servers;

[0076] S104: performing numerical comparison processing on the operation abnormality probability value of each second server and the preset probability threshold, and selecting a third server whose operation abnormality probability value is less than the preset probability threshold from all the second servers;

[0077] S105: Use the first server and the third server as the abnormal servers.

[0078] As described in the above steps S100 to S105, the step of performing abnormal analysis on all servers included in the preset server cluster to determine the abnormal servers in the server cluster may specifically include: first obtaining the operating parameters of each server, and obtaining a preset parameter threshold. Wherein, the preset parameter threshold is a comprehensive threshold corresponding to all the servers. The operating parameters may only include the task processing error rate or only include the task processing timeout rate. Correspondingly, the preset parameter threshold may include an error threshold or a timeout threshold. Usually, according to the monitoring service, the total number of tasks processed by the server at each moment, the number of task processing errors and the task processing time can be recorded, and then the total number of tasks, the number of task processing errors and the task processing time of the server in a certain time period can be counted through the sliding window algorithm, and the task processing error rate and the task processing timeout rate are determined according to the total number of tasks, the number of task processing errors and the task processing time, and the task processing error rate and the task processing timeout rate are equal to the number of task processing errors divided by the total number of tasks of the server. In addition, the number of timed tasks can be obtained according to the task processing time and the preset standard task processing time, and the task processing timeout rate is equal to the number of timed tasks divided by the total number of tasks of the server. In addition, if the task processing time of the task is longer than the standard task processing time, the task is regarded as a timed-out task. Furthermore, the operating parameters may also include both the task processing error rate and the task request timeout rate. At this time, the task processing error rate and the task request timeout rate need to be judged respectively, and as long as one parameter meets the corresponding preset parameter threshold, the server corresponding to the parameter is regarded as an abnormal server. Then the operating parameters of each server are numerically compared with the preset parameter threshold, and the first server whose operating parameters meet the preset parameter threshold is selected from all the servers, and the servers except the first server in all the servers are marked as the second server. Among them, in the specific implementation, it is necessary to screen each server in the server cluster, first determine whether the operating parameters of the current server meet the parameter threshold, and if so, mark the current server as an abnormal server; then use the next server of the current server as the new current server, and continue to determine whether the operating parameters of the current server meet the parameter threshold until all the servers in the server cluster are judged. In addition, the operating parameter meeting the parameter threshold can be that the operating parameter is greater than the parameter threshold, or the operating parameter is less than the parameter threshold, etc. Then, the operating parameters of each second server are obtained, and the operating abnormality probability value of each second server is generated based on the operating parameters of each second server and a preset abnormality recognition model. The preset abnormality recognition model is a probability distribution model, which can be any one of a Gaussian model, a Poisson distribution model, or a Bernoulli distribution model, and preferably a Gaussian model can be used.The abnormal identification model can generally characterize the approximate distribution of the operating parameters of the servers in the server cluster. The indefinite parameters in the abnormal identification model can generally be obtained by fitting the operating parameters of all servers in the server cluster. Then, the operating parameters of the current server are input into the abnormal identification model after the indefinite parameters are determined, and the probability value of the normal operation of the current server can be output. Then, the probability value of the abnormal operation of the server is determined by the probability of normal operation, and the sum of the probability value of the normal operation of the server and the probability value of the abnormal operation is 1, so that the required probability value of the abnormal operation can be obtained by using 1-the probability value of normal operation. Subsequently, a preset probability threshold is obtained; wherein, the preset probability threshold is a comprehensive threshold corresponding to all the servers. Among them, the value of the preset probability threshold is not specifically limited and can be set according to actual needs. Finally, the abnormal operation probability value of each second server is numerically compared with the preset probability threshold, and a third server whose abnormal operation probability value is less than the preset probability threshold is selected from all the second servers, and the first server and the third server are used as the abnormal servers. In this embodiment, for each server in the server cluster, if the operating parameters of the first server meet the parameter threshold corresponding to the operating parameters, the first server will be used as an abnormal server. On this basis, the preset abnormality identification model will be used to predict the abnormality of the second server except the first server, and then the corresponding third server will be screened out from the second server as the abnormal server. By performing two abnormality identification screenings on the servers in the server cluster, and then determining all qualified servers obtained from the two abnormality identification screenings as abnormal servers, the accuracy of judging server abnormalities is improved, and the quality of normal servers in the service cluster is guaranteed to a certain extent, which is conducive to avoiding the situation where the task cannot be processed normally due to the use of abnormal servers to process tasks, effectively improving the operating stability of the server cluster, ensuring the normal processing of tasks, and improving the user experience.

[0079] Furthermore, in an embodiment of the present application, the above step S102 includes:

[0080] S1020: Calculate the average value of the operating parameters of all the second servers; and,

[0081] S1021: Calculate the variance of the operating parameters of all the second servers;

[0082] S1022: Obtain designated operating parameters of a first designated server; wherein the first designated server is any one of all the second servers;

[0083] S1023: Input the mean value, the variance, and the designated operation parameter into the abnormality identification model, and generate an operation abnormality probability value corresponding to the first designated server through the abnormality identification model.

[0084] As described in the above steps S1020 to S1023, the step of obtaining the operating parameters of each second server and generating the operating abnormality probability value of each second server based on the operating parameters of each second server and the preset abnormality recognition model may specifically include: first calculating the average value of the operating parameters of all second servers. Among them, the average value corresponding to the operating parameter can be obtained by adding up the operating parameters of all second servers and then averaging them. And calculate the variance of the operating parameters of all second servers. The variance of the operating parameters of each second server can be calculated by a preset formula based on the operating parameters of all second servers and the average values ​​corresponding to the operating parameters of all second servers. The preset formula is specifically: σ 2 is the variance of the operating parameters of all the second servers, p is the total number of all the second servers, x i represents the operating parameters of the i-th second server, and μ is the average value of the operating parameters of all the second servers. Then obtain the specified operating parameters of the first designated server. Among them, the first designated server is any one of all the second servers. Finally, the average value, the variance and the specified operating parameters are input into the abnormality recognition model, and the operating abnormality probability value corresponding to the first designated server is generated by the abnormality recognition model. Specifically, the abnormality recognition model is a Gaussian model. The formula of the Gaussian model is: In the formula of the Gaussian model, x represents the specified operating parameter of the first specified server, μ is the average value of the operating parameters of all the second servers, and σ 2 is the variance of the operating parameters of all the second servers, and the mean μ and variance σ corresponding to the operating parameters are 2, and the designated operating parameters of the first designated server are substituted into the formula of the Gaussian model, and the probability value of the corresponding first designated server operating normally can be output, and then the probability value of the normal operation is subtracted from 1, so as to obtain the probability value of the abnormal operation of the first designated server. In this embodiment, after the operating parameters of each of the second servers are obtained, the corresponding mean value and variance can be calculated according to the operating parameters, and then the mean value, the variance and the designated operating parameters of the first designated server can be input into the abnormal identification model, so that the abnormal operation probability value corresponding to the first designated server can be quickly and accurately generated through the abnormal identification model, which is conducive to the subsequent screening of the third server whose operating abnormality probability value is less than the preset probability threshold from all the second servers according to the obtained operating abnormality probability value, and the third server is used as the abnormal server, which effectively improves the accuracy of judging server abnormalities.

[0085] Furthermore, in an embodiment of the present application, the above step S40 includes:

[0086] S400: Step A: sorting all the tasks to be processed in order from the beginning to the end according to the collection time of each task to be processed, and obtaining a corresponding sorting result;

[0087] S401: Step B: Obtain the first task ranked first in the sorting result, and extract the first task field included in the first task;

[0088] S402: Step C: performing splicing processing on all the first task fields to obtain corresponding first splicing fields, storing the first splicing fields in the cache tool, and setting a first timeout period for the first splicing fields;

[0089] S403: Step D: Obtain the second task ranked at the i-th position in the sorting result, perform splicing processing on all second task fields contained in the second task to obtain a corresponding second splicing field, and determine whether there is a first designated field identical to the second splicing field in the cache tool; wherein i is an integer and the initial value of i is 2;

[0090] S404: Step E: If the first designated field does not exist, the second concatenated field is stored in the cache tool, and a second timeout period is set for the second concatenated field; otherwise, the second concatenated field is not stored;

[0091] S405: Step F: let i=i+1, and repeat steps D-E until i=n, ​​so as to complete the processing of all tasks included in the sorting result, and obtain all target splicing fields stored in the cache tool; wherein n is the number of all tasks to be processed;

[0092] S406: Step G: Filter out the tasks corresponding to the target splicing field from all the tasks to be processed to obtain the target task.

[0093] As described in the above steps S400 to S406, the step of performing deduplication processing on all the tasks to be processed based on the preset cache tool to obtain the processed target task may specifically include: Step A: Sort all the tasks to be processed in the order of the collection time of each task to be processed from front to back to obtain the corresponding sorting result. Wherein, the collection time may refer to the time when the target server collects each task to be processed. Then step B: Get the first task ranked first in the sorting result, and extract the first task field contained in the first task. Wherein, the task field may refer to all the fields contained in the task to be processed, or may also refer to the field at the specified position contained in the task to be processed. If it is a field at the specified position, it is necessary to ensure that the field extraction processing for each task to be processed is the field extraction at the specified position. Step C: Splice all the first task fields to obtain the corresponding first splicing field, store the first splicing field in the cache tool, and set a first timeout time for the first splicing field. Wherein, the cache tool may specifically be a redis cache. A timeout period can also be set for the spliced ​​fields generated by splicing. If the storage time of the spliced ​​field in the cache tool is greater than the timeout period, the spliced ​​field will be regarded as invalid information and removed from the cache tool, so that when a specific field identical to the spliced ​​field is received later, the specific field can be stored in the cache tool. In addition, for any current task, if the spliced ​​field corresponding to the task exists in the cache tool, the task can be regarded as a repeated task, and the spliced ​​field will not be stored in the cache tool later, and the task will also be removed to avoid repeated processing of the same multiple tasks, thereby improving the intelligence and accuracy of task processing. Step D: Obtain the second task ranked i-th in the sorting result, splice all the second task fields contained in the second task to obtain the corresponding second splicing field, and determine whether there is a first specified field identical to the second splicing field in the cache tool. Wherein, i is an integer and the initial value of i is 2. Step E: If the first designated field does not exist, the second splicing field is stored in the cache tool, and a second timeout period is set for the second splicing field, otherwise the second splicing field is not stored. Step F: Let i=i+1, and repeat steps D-E until i=n to complete the processing of all tasks included in the sorting result, and obtain all target splicing fields stored in the cache tool; wherein n is the number of all the tasks to be processed. The target splicing field refers to the field data that is different from each other after all the tasks to be processed are deduplicated based on the splicing field.For example, when i=3, the processing process may include: obtaining the third task ranked third in the sorting result, performing splicing processing on all the third task fields contained in the third task to obtain the corresponding third splicing field, and judging whether there is a second designated field identical to the third splicing field in the cache tool. If the second designated field does not exist, the third splicing field is stored in the cache tool, and a third timeout time is set for the second splicing field, otherwise the third splicing field is not stored. Step G: Filter out the task corresponding to the target splicing field from all the tasks to be processed to obtain the target task. In this embodiment, after all the tasks to be processed are obtained, the task fields contained in each task to be processed are spliced ​​to generate the splicing fields corresponding to each task to be processed, and then based on the use of the splicing fields and the cache tool, it is checked whether there are duplicate tasks in all the tasks to be processed, so that the deduplication processing of all the tasks to be processed can be completed accurately and quickly, effectively avoiding the repeated execution of the same task, reducing the useless power consumption of the device, and improving the intelligence and accuracy of task processing.

[0094] Furthermore, in an embodiment of the present application, the above step S50 includes:

[0095] S500: Acquire the collection time information, processing timeliness information and importance information of each target task;

[0096] S501: generating a processing order of each of the target tasks based on the collection time information, the processing timeliness information and the importance information;

[0097] S502: storing all the target tasks in the message queue according to the processing order;

[0098] S503: Allocate all the target tasks in the message queue to the normal servers in the server cluster according to the processing order through the message middleware.

[0099] As described in the above steps S500 to S503, the step of storing the target task in a preset message queue and assigning all the target tasks in the message queue to each normal server in the server cluster through the message middleware may specifically include: first obtaining the acquisition time information, processing timeliness information and importance information of each target task. Among them, the acquisition time information may refer to the time information of each target task acquired by the target server. In addition, the processing timeliness information of each task can usually be set according to the subsequent task requirement time of each business department. The more urgent the subsequent task requirement time is, the earlier the processing timeliness setting of the corresponding task is, that is, the smaller the value of the processing timeliness information is. In addition, the operation results obtained by processing each target task are usually required for each business department to process subsequent tasks. The more important the subsequent task is, the higher the importance of the corresponding target task is. Usually, the task importance recognition model can be obtained by analyzing historical data, obtaining a large number of sample tasks and sample importance information to train the convolutional neural network model, and the importance information of each target task can be obtained by inputting each target task into the task importance recognition model. Then, based on the acquisition time information, the processing timeliness information and the importance information, the processing order of each target task is generated. Among them, for the specific implementation process of generating the processing order of each target task based on the acquisition time information, the processing timeliness information and the importance information, this application will be further described in the subsequent specific embodiments, and will not be repeated here. After that, all the target tasks are stored in the message queue according to the processing order. Finally, all the target tasks in the message queue are assigned to each normal server in the server cluster according to the processing order through the message middleware. Among them, after all the target tasks are stored in the message queue according to the processing order, the process of assigning all the target tasks to each normal server in the server cluster may include: first, number each normal server according to the processing performance of each normal server, and the smaller the processing performance, the smaller the number value, such as the normal server with the highest processing performance is numbered 1, and the normal server with the second highest processing performance is numbered 2. Then, the order in which each normal server receives the target task is determined based on the number of the normal server. The smaller the number, the earlier the receiving order. For example, the normal server numbered 1 is used to receive and process the first target task ranked first in the message queue, and the normal server numbered 2 is used to receive and process the second target task ranked second in the message queue.In addition, it can be stipulated that each normal server can only receive one task at a time. When the number of target tasks is greater than the number of normal servers, after the first round of target tasks are allocated to each normal server, the remaining target tasks will be allocated again according to the number of the normal server until all target tasks are allocated to the normal servers in the server cluster. For example, if there are 6 target tasks in the message queue and there are 5 normal servers (server 1, server 2, server 3, server 4, server 5, and the number 1 after server 1 represents the number of the server), the target task ranked first in the message queue will be allocated to server 1 first, and then the target task ranked second will be allocated to server 2, the target task ranked third will be allocated to server 3, the target task ranked fourth will be allocated to server 4, and the target task ranked fifth will be allocated to server 5. After completing the first round of allocation of target tasks, there is still a target task ranked sixth. Then, according to the corresponding relationship between the number of the normal server and the order of task reception, the target task ranked sixth will be allocated to server 1, and finally the allocation of all target tasks in the message queue will be completed. In this embodiment, after the target tasks are screened out through deduplication processing, the processing order of each target task is further generated based on the collection time information, processing time information and importance information of each target task, and then all the target tasks are stored in the message queue according to the processing order, so that the target tasks in the message queue can be allocated to each normal server in the server cluster through the message middleware according to the processing order, so that the target tasks with earlier collection time, tight time and higher importance will be processed first, which effectively improves the processing intelligence of the target tasks and is conducive to the smooth connection of subsequent task processing of various business departments.

[0100] Furthermore, in an embodiment of the present application, the above step S501 includes:

[0101] S5010: Acquire a first preset weight corresponding to the collection time information, a second preset weight corresponding to the processing timeliness information, and a third preset weight corresponding to the importance information;

[0102] S5011: Calculate the priority value of each of the target tasks by a preset calculation formula based on the collection time information, the processing timeliness information, the importance information, the first preset weight, the second preset weight, and the third preset weight;

[0103] S5012: Sort all the target tasks in descending order according to the priority values ​​to obtain target sorting information corresponding to all the target tasks;

[0104] S5013: Generate a processing order for each of the target tasks based on all the target sorting information.

[0105] As described in the above steps S5010 to S5013, the step of generating the processing order of each of the target tasks based on the collection time information, the processing time information and the importance information may specifically include: first obtaining a first preset weight corresponding to the collection time information, a second preset weight corresponding to the processing time information, and a third preset weight corresponding to the importance information. Among them, there is no specific limitation on the values ​​of the first preset weight, the second preset weight and the third preset weight, which can be set according to actual usage requirements. Preferably, the second preset weight is greater than the third preset weight, the third preset weight is greater than the first preset weight, and the sum of the first preset weight, the second preset weight and the third preset weight is 1. Then, based on the collection time information, the processing time information, the importance information, the first preset weight, the second preset weight and the third preset weight, the priority value of each of the target tasks is calculated by a preset calculation formula. Specifically, the preset calculation formula is: S=M / a+N*b+O*c, where S is the priority value of the target task, M is the acquisition time information, a is the first preset weight, N is the processing time information, b is the second preset weight, O is the importance information, and c is the third preset weight. The larger the priority value S calculated by the preset calculation formula, the more urgent the demand for the corresponding target task and the higher the importance. Then, the processing order of each target task can be obtained by sorting from large to small according to the priority value S. The larger the priority value S, the higher the processing order of the corresponding target task. Then, all the target tasks are sorted in order from large to small according to the priority value, and the target sorting information corresponding to all the target tasks is obtained. Finally, the processing order of each target task is generated based on all the target sorting information. In this embodiment, after deduplication processing is performed on all pending tasks to screen out target tasks, a processing order for each target task is further generated based on the collection time information, processing timeliness information and importance information of each target task, and then all target tasks are stored in the message queue according to the processing order, so that the target tasks in the message queue can be allocated to each normal server in the server cluster according to the processing order through the message middleware, so that target tasks with earlier collection time, tight time and higher importance will be processed first, which effectively improves the intelligence of the processing of target tasks and is conducive to the smooth connection of subsequent task processing of various business departments.

[0106] Furthermore, in an embodiment of the present application, after the above step S50, the following steps are included:

[0107] S510: receiving, through the message middleware, a task processing result obtained by processing the received designated task and sent by the second designated server; wherein the second designated server is any one of all the normal servers, and the designated task is any one of all the target tasks received by the designated server;

[0108] S511: If the task processing result is normal, obtaining a specified splicing field corresponding to the specified task in the cache tool, and adjusting a specified timeout time corresponding to the specified splicing field so that the adjusted specified timeout time is greater than the specified timeout time;

[0109] S512: If the task processing result is a processing exception, deleting the specified splicing field corresponding to the specified task from the cache tool;

[0110] S513: placing the designated task into a preset abnormal task list.

[0111] As described in the above steps S510 to S513, after the step of storing the target task in a preset message queue and assigning all the target tasks in the message queue to each of the normal servers in the server cluster through the message middleware, the process includes: first, receiving the task processing result obtained by processing the received designated task sent by the second designated server through the message middleware. Among them, the second designated server is any one of all the normal servers, and the designated task is any one of all the target tasks received by the designated server. In addition, the task processing result may include normal processing or abnormal processing. If the task processing result is normal processing, the designated splicing field corresponding to the designated task is obtained in the cache tool, and the designated timeout time corresponding to the designated splicing field is adjusted so that the adjusted designated timeout time is greater than the designated timeout time. If the task processing result is abnormal processing, the designated splicing field corresponding to the designated task is deleted from the cache tool. And the designated task is placed in the preset abnormal task list. Among them, by placing the designated task in a preset abnormal task list, the currently occurring abnormal task can be quickly queried from the abnormal task list, and the subsequent processing of the abnormal task can be arranged in time. In this embodiment, when the task processing result of the designated task is normal processing, the designated timeout time corresponding to the designated splicing field will be intelligently adjusted, so that the timeout time of the normally processed designated task can be extended, so that the repeated processing of the normally processed designated task can be better avoided. In addition, when the task processing result of the designated task is abnormal processing, the designated splicing field corresponding to the designated task will be intelligently deleted from the cache tool, so that the designated splicing field corresponding to the designated task can be re-stored in the cache tool later, so that the designated task of the abnormal processing can be re-executed smoothly later.

[0112] The task processing method in the embodiment of the present application can also be applied to the field of blockchain, such as storing the above-mentioned target tasks and other data on the blockchain. By using blockchain to store and manage the above-mentioned target tasks, the security and non-tamperability of the above-mentioned target tasks can be effectively guaranteed.

[0113] The above-mentioned blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0114] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operation monitoring. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining public and private key generation (account management), key management, and the maintenance of the correspondence between the user's real identity and the blockchain address (authority management), etc., and, under authorization, supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records valid requests to storage after consensus is reached. For a new business request, the basic service first adapts the interface for parsing and authentication (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management). The smart contract module is responsible for the registration and issuance of contracts, as well as contract triggering and contract execution. Developers can define the contract logic in a programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, the key or other events are called to trigger the execution and complete the contract logic. It also provides the function of contract upgrade and cancellation. The operation monitoring module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and real-time status visualization output of the product during the product release process, such as alarm, network monitoring, node equipment health monitoring, etc.

[0115] Reference Figure 2 In one embodiment of the present application, a task processing device is also provided, including:

[0116] The first determination module 1 is used to perform an abnormality analysis on all servers included in a preset server cluster to determine abnormal servers in the server cluster, and mark the servers in the server cluster except the abnormal servers as normal servers; wherein the number of the servers includes multiple servers;

[0117] A second determination module 2 is used to determine a target server from all the normal servers;

[0118] An acquisition module 3 is used to acquire the tasks to be processed from a preset data source through the target server; wherein the number of the tasks to be processed includes multiple;

[0119] The first processing module 4 is used to perform deduplication processing on all the tasks to be processed based on a preset cache tool to obtain a processed target task;

[0120] The second processing module 5 is used to store the target task in a preset message queue, and distribute all the target tasks in the message queue to the normal servers in the server cluster through the message middleware, so that the received target tasks can be processed by each normal server.

[0121] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the task processing method in the above-mentioned implementation mode, and will not be described in detail here.

[0122] Furthermore, in an embodiment of the present application, the first determining module 1 includes:

[0123] A first acquisition unit is used to acquire the operating parameters of each of the servers and to acquire a preset parameter threshold; wherein the preset parameter threshold is a comprehensive threshold corresponding to all of the servers;

[0124] a first screening unit, configured to perform numerical comparison processing on the operating parameters of each of the servers and the preset parameter threshold, screen out a first server whose operating parameters meet the preset parameter threshold from all the servers, and mark all the servers except the first server as second servers;

[0125] A first generating unit, configured to obtain operating parameters of each of the second servers, and generate an operating abnormality probability value of each of the second servers based on the operating parameters of each of the second servers and a preset abnormality recognition model;

[0126] A second acquisition unit is used to acquire a preset probability threshold; wherein the preset probability threshold is a comprehensive threshold corresponding to all the servers;

[0127] A second screening unit is used to perform numerical comparison processing on the operation abnormality probability value of each second server and the preset probability threshold, and screen out third servers whose operation abnormality probability value is less than the preset probability threshold from all the second servers;

[0128] A determining unit is configured to use the first server and the third server as the abnormal servers.

[0129] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the task processing method in the above-mentioned implementation mode, and will not be described in detail here.

[0130] Furthermore, in one embodiment of the present application, the first generating unit includes:

[0131] a first calculation subunit, configured to calculate an average value of the operating parameters of all the second servers; and

[0132] A second calculation subunit, configured to calculate the variance of the operating parameters of all the second servers;

[0133] A first acquisition subunit is used to acquire a specified operating parameter of a first specified server; wherein the first specified server is any one of all the second servers;

[0134] The first generating subunit is used to input the mean value, the variance and the designated operation parameter into the abnormality identification model, and generate an operation abnormality probability value corresponding to the first designated server through the abnormality identification model.

[0135] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the task processing method in the above-mentioned implementation mode, and will not be described in detail here.

[0136] Furthermore, in one embodiment of the present application, the first processing module 4 includes:

[0137] A sorting unit, used to sort all the tasks to be processed in order from the beginning to the end according to the collection time of each task to be processed, to obtain a corresponding sorting result;

[0138] an extracting unit, configured to obtain a first task ranked first in the sorting result, and extract a first task field included in the first task;

[0139] A first processing unit, configured to perform splicing processing on all the first task fields to obtain corresponding first splicing fields, store the first splicing fields in the cache tool, and set a first timeout period for the first splicing fields;

[0140] a judgment unit, configured to obtain the second task ranked at the i-th position in the sorting result, perform splicing processing on all second task fields included in the second task to obtain a corresponding second splicing field, and judge whether there is a first designated field identical to the second splicing field in the cache tool; wherein i is an integer and the initial value of i is 2;

[0141] A second processing unit is configured to store the second concatenated field in the cache tool and set a second timeout period for the second concatenated field if the first designated field does not exist, and otherwise not perform storage processing on the second concatenated field;

[0142] A third acquisition unit is used to set i=i+1, and repeatedly execute the first judgment unit to the second processing unit until i=n, ​​so as to complete the processing of all tasks included in the sorting result, and obtain all target splicing fields stored in the cache tool; wherein n is the number of all the tasks to be processed;

[0143] The third screening unit is used to screen out the tasks corresponding to the target splicing field from all the tasks to be processed to obtain the target task.

[0144] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the task processing method in the above-mentioned implementation mode, and will not be described in detail here.

[0145] Furthermore, in one embodiment of the present application, the second processing module 5 includes:

[0146] A fourth acquisition unit is used to acquire the collection time information, processing timeliness information and importance information of each target task;

[0147] A second generating unit, configured to generate a processing order of each of the target tasks based on the collection time information, the processing timeliness information and the importance information;

[0148] A storage unit, used for storing all the target tasks in the message queue according to the processing order;

[0149] The allocating unit is used to allocate all the target tasks in the message queue to the normal servers in the server cluster according to the processing order through the message middleware.

[0150] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the task processing method in the above-mentioned implementation mode, and will not be described in detail here.

[0151] Furthermore, in one embodiment of the present application, the second generating unit includes:

[0152] A second acquisition subunit is used to acquire a first preset weight corresponding to the collection time information, a second preset weight corresponding to the processing timeliness information, and a third preset weight corresponding to the importance information;

[0153] A third calculation subunit is used to calculate the priority value of each of the target tasks through a preset calculation formula based on the collection time information, the processing timeliness information, the importance information, the first preset weight, the second preset weight and the third preset weight;

[0154] A sorting subunit is used to sort all the target tasks in descending order according to the priority values ​​to obtain target sorting information corresponding to all the target tasks;

[0155] The second generating subunit is used to generate a processing order of each of the target tasks based on all the target sorting information.

[0156] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the task processing method in the above-mentioned implementation mode, and will not be described in detail here.

[0157] Furthermore, in one embodiment of the present application, the task processing device includes:

[0158] A receiving module, used for receiving, through the message middleware, a task processing result obtained by processing the received designated task and sent by the second designated server; wherein the second designated server is any one of all the normal servers, and the designated task is any one of all the target tasks received by the designated server;

[0159] an adjustment module, for obtaining, in the cache tool, a designated concatenated field corresponding to the designated task, and adjusting a designated timeout period corresponding to the designated concatenated field, if the task processing result is normal processing, so that the adjusted designated timeout period is greater than the designated timeout period;

[0160] A deletion module, configured to delete a specified splicing field corresponding to the specified task from the cache tool if the task processing result is a processing exception;

[0161] The placement module is used to place the specified task in a preset abnormal task list.

[0162] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the task processing method in the above-mentioned implementation mode, and will not be described in detail here.

[0163] Reference Figure 3 In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface, a display screen, an input device and a database connected through a system bus. Among them, the processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a storage medium and an internal memory. The storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the storage medium. The database of the computer device is used to store tasks to be processed and target tasks. The network interface of the computer device is used to communicate with an external terminal through a network connection. The display screen of the computer device is an indispensable graphic output device in the computer, which is used to convert digital signals into optical signals so that text and graphics are displayed on the screen of the display screen. The input device of the computer device is the main device for information exchange between the computer and the user or other devices, and is used to transmit data, instructions and certain mark information to the computer. When the computer program is executed by the processor, a task processing method is implemented.

[0164] The above processor executes the steps of the above task processing method:

[0165] Performing an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster, and marking the servers in the server cluster except the abnormal server as normal servers; wherein the number of the servers includes a plurality;

[0166] Determine a target server from all the normal servers;

[0167] Obtaining the tasks to be processed from a preset data source through the target server; wherein the number of the tasks to be processed includes multiple;

[0168] Deduplication processing is performed on all the tasks to be processed based on a preset cache tool to obtain a processed target task;

[0169] The target task is stored in a preset message queue, and all the target tasks in the message queue are distributed to the normal servers in the server cluster through the message middleware, so that the received target tasks are processed by the normal servers.

[0170] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application scheme, and does not constitute a limitation on the devices and computer equipment to which the present application scheme is applied.

[0171] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a task processing method is implemented, specifically:

[0172] Performing an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster, and marking the servers in the server cluster except the abnormal server as normal servers; wherein the number of the servers includes a plurality;

[0173] Determine a target server from all the normal servers;

[0174] Obtaining the tasks to be processed from a preset data source through the target server; wherein the number of the tasks to be processed includes multiple;

[0175] Deduplication processing is performed on all the tasks to be processed based on a preset cache tool to obtain a processed target task;

[0176] The target task is stored in a preset message queue, and all the target tasks in the message queue are distributed to the normal servers in the server cluster through the message middleware, so that the received target tasks are processed by the normal servers.

[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in the present application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0178] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0179] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A task processing method, characterized in that: include: Performing an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster, and marking the servers in the server cluster except the abnormal server as normal servers; wherein the number of the servers includes a plurality; Determine a target server from all the normal servers; Obtaining the tasks to be processed from a preset data source through the target server; wherein the number of the tasks to be processed includes multiple; Deduplication processing is performed on all the tasks to be processed based on a preset cache tool to obtain a processed target task; The target task is stored in a preset message queue, and all the target tasks in the message queue are distributed to the normal servers in the server cluster through the message middleware, so that the received target tasks are processed by the normal servers; The step of performing an abnormality analysis on all servers included in a preset server cluster to determine abnormal servers in the server cluster includes: Obtaining the operating parameters of each of the servers, and obtaining a preset parameter threshold; wherein the preset parameter threshold is a comprehensive threshold corresponding to all of the servers; Comparing the operating parameters of each server with the preset parameter threshold, selecting a first server whose operating parameters meet the preset parameter threshold from all the servers, and marking all the servers except the first server as second servers; Acquire the operating parameters of each second server, and generate an operating abnormality probability value of each second server based on the operating parameters of each second server and a preset abnormality recognition model; Obtaining a preset probability threshold; wherein the preset probability threshold is a comprehensive threshold corresponding to all the servers; Compare the operation abnormality probability value of each second server with the preset probability threshold, and select a third server whose operation abnormality probability value is less than the preset probability threshold from all the second servers; The first server and the third server are used as the abnormal servers.

2. The task processing method according to claim 1, characterized in that: The step of obtaining the operating parameters of each second server and generating an operating abnormality probability value of each second server based on the operating parameters of each second server and a preset abnormality recognition model includes: Calculating an average value of the operating parameters of all the second servers; and, Calculating the variance of the operating parameters of all the second servers; Obtaining designated operating parameters of a first designated server; wherein the first designated server is any one of all the second servers; The mean value, the variance, and the designated operation parameter are input into the abnormality identification model, and an operation abnormality probability value corresponding to the first designated server is generated through the abnormality identification model.

3. The task processing method according to claim 1, characterized in that: The step of performing deduplication processing on all the tasks to be processed based on a preset cache tool to obtain the processed target tasks includes: Step A: sorting all the tasks to be processed in order from the beginning to the end according to the collection time of each task to be processed, and obtaining a corresponding sorting result; Step B: obtaining the first task ranked first in the sorting result, and extracting the first task field included in the first task; Step C: performing splicing processing on all the first task fields to obtain corresponding first splicing fields, storing the first splicing fields in the cache tool, and setting a first timeout period for the first splicing fields; Step D: obtaining the second task ranked at the i-th position in the sorting result, performing splicing processing on all second task fields contained in the second task to obtain a corresponding second splicing field, and determining whether there is a first designated field identical to the second splicing field in the cache tool; wherein i is an integer and the initial value of i is 2; Step E: if the first designated field does not exist, the second concatenated field is stored in the cache tool, and a second timeout period is set for the second concatenated field; otherwise, the second concatenated field is not stored; Step F: let i=i+1, and repeat steps D-E until i=n, ​​so as to complete the processing of all tasks included in the sorting result, and obtain all target splicing fields stored in the cache tool; wherein n is the number of all the tasks to be processed; Step G: Filter out the tasks corresponding to the target splicing field from all the tasks to be processed to obtain the target task.

4. The task processing method according to claim 1, characterized in that: The step of storing the target task in a preset message queue, and distributing all the target tasks in the message queue to each of the normal servers in the server cluster through a message middleware includes: Obtaining the collection time information, processing timeliness information and importance information of each target task; Generate a processing order for each of the target tasks based on the collection time information, the processing timeliness information and the importance information; storing all the target tasks in the message queue according to the processing order; All the target tasks in the message queue are distributed to the normal servers in the server cluster through the message middleware according to the processing order.

5. The task processing method according to claim 4, characterized in that: The step of generating a processing order of each of the target tasks based on the collection time information, the processing timeliness information and the importance information includes: Acquire a first preset weight corresponding to the collection time information, a second preset weight corresponding to the processing timeliness information, and a third preset weight corresponding to the importance information; Based on the collection time information, processing timeliness information, importance information, the first preset weight, the second preset weight, and the third preset weight, a priority value of each of the target tasks is calculated by a preset calculation formula; Sorting all the target tasks in descending order according to the priority values ​​to obtain target sorting information corresponding to all the target tasks; A processing order of each of the target tasks is generated based on all the target sorting information.

6. The task processing method according to claim 1, characterized in that: After the step of storing the target task in a preset message queue and allocating all the target tasks in the message queue to each of the normal servers in the server cluster through the message middleware, the method includes: Receiving, through the message middleware, a task processing result obtained by processing the received designated task and sent by the second designated server; wherein the second designated server is any one of all the normal servers, and the designated task is any one of all the target tasks received by the designated server; If the task processing result is normal, obtaining a specified splicing field corresponding to the specified task in the cache tool, and adjusting the specified timeout time corresponding to the specified splicing field so that the adjusted specified timeout time is greater than the specified timeout time; If the task processing result is a processing exception, deleting the specified splicing field corresponding to the specified task from the cache tool; The designated task is placed in a preset abnormal task list.

7. A task processing device, used to implement the method according to any one of claims 1 to 6, characterized in that: include: A first determination module is used to perform an abnormality analysis on all servers included in a preset server cluster to determine an abnormal server in the server cluster, and mark the servers in the server cluster except the abnormal server as normal servers; wherein the number of the servers includes multiple servers; A second determination module is used to determine a target server from all the normal servers; An acquisition module, used for acquiring tasks to be processed from a preset data source through the target server; wherein the number of the tasks to be processed includes multiple; A first processing module, used for performing deduplication processing on all the tasks to be processed based on a preset cache tool to obtain a processed target task; The second processing module is used to store the target task in a preset message queue, and distribute all the target tasks in the message queue to the normal servers in the server cluster through the message middleware, so that the received target tasks can be processed by each normal server.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images