Distributed asynchronous delay task processing method and system based on fragmented tasks

Abstract

The invention relates to the technical field of task processing, and relates to a distributed asynchronous delay task processing method and system based on fragmented tasks, and the method comprises the steps: carrying out task conversion and task storage operation on pre-obtained task parameters, and obtaining task information; performing triggering fragmentation and polling packaging on the task information to obtain a task message set; performing subscription transmission and message unpacking on the task message set to obtain an unpacked task set; performing task logic analysis and service execution on the unpacked task set to obtain an execution state set; and performing state feedback and task retry on the unpacked task set according to the execution state set to obtain a task processing log. According to the invention, the efficiency of distributed asynchronous delay task processing can be improved.

Description

Technical Field

[0001] The present invention relates to the field of task processing technology, and in particular to a distributed asynchronous delayed task processing method and system based on sharding tasks. Background Art

[0002] Distributed asynchronous delayed tasks are a task scheduling mechanism that involves scheduling tasks in a distributed system to be executed at a certain point in the future or after a specified delay. Distributed asynchronous delayed tasks are widely used in scenarios such as data backup, temporary data cleaning, and delayed push of notification information such as emails and text messages.

[0003] Existing distributed asynchronous delayed task processing methods include message queues, distributed locks, and timers, which put tasks into queues and trigger task processing based on delay time or specific conditions. In actual applications, message queue systems may cause message loss due to network partitions, node failures, and other problems, and the partitioning and load distribution of distributed environments are relatively complex, which may lead to low efficiency in task processing. Summary of the invention

[0004] The present invention provides a distributed asynchronous delayed task processing method and system based on sharding tasks, the main purpose of which is to solve the problem of low efficiency in task processing.

[0005] To achieve the above object, the present invention provides a distributed asynchronous delayed task processing method based on sharding tasks, comprising:

[0006] Perform task conversion and task storage operations on the pre-acquired task parameters to obtain task information;

[0007] Perform trigger feature segmentation and polling packaging on the task information to obtain a task message set;

[0008] Subscribe to the task message set and unpack the message to obtain an unpacked task set;

[0009] Performing task logic analysis and business execution on the unpacked task set to obtain an execution status set;

[0010] According to the execution status set, status feedback and task retry are performed on the unpacked task set to obtain a task processing log.

[0011] Optionally, performing task conversion and task storage operations on the pre-acquired task parameters to obtain task information includes:

[0012] Initialize the class of the pre-acquired task parameters to obtain the task information class;

[0013] Perform name configuration on the task parameters according to the task information class to obtain initial task parameters;

[0014] Performing delayed transformation on the initial task parameters to obtain delayed task parameters;

[0015] Calculating the execution time and configuring the execution state of the delayed task parameters to obtain execution task parameters;

[0016] The task information of the execution task parameters is stored in a database to obtain the task information.

[0017] Optionally, the triggering segmentation and polling packaging of the task information to obtain a task message set includes:

[0018] Performing polling scheduling scanning on the task message to obtain scheduling task information;

[0019] Perform trigger calculation on the scheduling task information to obtain a trigger task time;

[0020] Perform trigger screening on the scheduled task message according to the trigger task time to obtain trigger task information;

[0021] Dividing the trigger task information into units to obtain a trigger subtask information set;

[0022] Performing hash time-volume business feature extraction on the triggered subtask information set to obtain a subtask type feature set, a subtask time feature set, a subtask data feature set, a subtask priority feature set, and a subtask hash feature set;

[0023] Performing covariance feature dimensionality reduction on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a triggering subtask feature set;

[0024] Clustering and sharding the trigger subtask information set according to the trigger subtask feature set to obtain a sharded task information set;

[0025] The shard task information set is packaged into messages to obtain a task message set.

[0026] Optionally, the triggering subtask information set is subjected to hash time-volume business feature extraction to obtain a subtask type feature set, a subtask time feature set, a subtask data feature set, a subtask priority feature set, and a subtask hash feature set, including:

[0027] Selecting the trigger subtask information in the trigger subtask information set one by one as the target subtask information, extracting the task type of the target subtask information, and obtaining a subtask type feature set;

[0028] Extracting the execution time of the target subtask information to obtain a subtask time feature set;

[0029] Extracting data volume from the target subtask information to obtain a subtask data feature set;

[0030] Priority extraction is performed on the target subtask information to obtain a subtask priority feature set;

[0031] Perform hash feature extraction on the target subtask information to obtain a subtask hash feature set.

[0032] Optionally, performing covariance feature dimensionality reduction on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a triggering subtask feature set includes:

[0033] Performing global pooling dimensionality reduction and feature matrix operations on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a subtask feature matrix;

[0034] Calculating the feature mean of the subtask feature matrix to obtain a mean feature matrix;

[0035] The covariance matrix corresponding to the subtask feature matrix is ​​calculated according to the mean feature matrix using the following covariance feature formula:

[0036]

[0037] Wherein, S refers to the covariance matrix, n is the total number of samples of the subtask feature matrix, X refers to the subtask feature matrix, refers to the mean feature matrix, T is the transposed symbol;

[0038] Performing eigenvalue decomposition on the covariance matrix to obtain a subtask eigenvector group;

[0039] Generate a feature projection matrix according to the subtask feature vector group;

[0040] The projection variance corresponding to the feature projection matrix is ​​calculated according to the mean feature matrix and the subtask feature matrix using the following projection variance algorithm:

[0041]

[0042] Wherein, σ refers to the projection variance, d is the number of feature dimensions of the subtask feature matrix, X refers to the subtask feature matrix, refers to the mean feature matrix, T is the transposed symbol, and W refers to the feature projection matrix;

[0043] Iteratively screening and updating the feature projection matrix according to the projection variance to obtain a maximum projection matrix;

[0044] The subtask feature matrix is ​​projected with the maximum projection matrix to obtain a triggering subtask feature set.

[0045] Optionally, clustering and sharding the trigger subtask information set according to the trigger subtask feature set to obtain a sharded task information set includes:

[0046] Performing Gaussian distribution splitting on the trigger subtask feature set to obtain a plurality of subtask fragment feature groups;

[0047] The shard center features are selected from each subtask shard feature group, and the feature deviation values ​​between each shard center feature and each trigger subtask feature in the trigger subtask feature set are calculated using the following feature deviation algorithm:

[0048]

[0049] Wherein, P refers to the feature deviation value, μ is the preset adversarial coefficient, α refers to the shard center feature, β refers to the trigger subtask feature, · is the dot product symbol, and * is the cross product symbol;

[0050] Iteratively update each subtask slice feature group according to the feature deviation value in a manner of nearest grouping to obtain a slice subtask feature group set;

[0051] Splitting the trigger subtask information set into slice subtask information sets according to the slice subtask feature set;

[0052] Task information of each shard subtask information group in the shard subtask information group set is merged to obtain a shard task information set.

[0053] Optionally, the subscribing, transmitting and unpacking the task message set to obtain the unpacked task set includes:

[0054] Sending the task message set to a preset subscription message queue;

[0055] Performing message persistence on the task message set in the subscription message queue to obtain a persistent message queue;

[0056] Subscribing to the persistent message queue to obtain a subscription message set;

[0057] Parsing the subscription message set to obtain a subscription task object set;

[0058] The subscribed task object is extracted and task information is extracted to obtain an unpacked task set.

[0059] Optionally, performing task logic analysis and business execution on the unpacked task set to obtain an execution status set includes:

[0060] Extracting time-volume business features from the unpacking task set to obtain an unpacking task type feature set, an unpacking task time feature set, an unpacking task data feature set, and an unpacking task priority feature set;

[0061] Prioritizing the unpacking task set according to the unpacking task priority feature set to obtain an unpacking task sequence;

[0062] Allocating processors to the unpacking task sequence according to the unpacking task type feature set to obtain a type processor set;

[0063] Allocate resources to the unpacking task sequence according to the unpacking task time feature set and the unpacking task data feature set to obtain a task resource set;

[0064] The type processor set is used to perform task execution and execution record on the unpacked task sequence according to the task resource set to obtain an execution state set.

[0065] Optionally, performing status feedback and task retry on the unpacked task set according to the execution status set to obtain a task processing log includes:

[0066] Selecting unpacking tasks in the unpacking task set one by one as target unpacking tasks, and selecting execution states corresponding to the target unpacking tasks from the execution state set as target execution states;

[0067] Performing state feedback on the target execution state to obtain feedback control information;

[0068] Determine whether the target execution status is execution failure;

[0069] If not, generating a task processing log according to the target execution status;

[0070] If yes, retry the target unpacking task according to the feedback control information to obtain a target retry status, and generate a task processing log according to the target retry status;

[0071] The target execution status is updated using the target retry status, and the process returns to the step of determining whether the target execution status indicates execution failure.

[0072] In order to solve the above problems, the present invention also provides a distributed asynchronous delayed task processing system based on sharding tasks, the system comprising:

[0073] The task storage module is used to perform task conversion and task storage operations on pre-acquired task parameters to obtain task information;

[0074] A polling and packaging module, used for performing trigger feature segmentation and polling packaging on the task information to obtain a task message set;

[0075] A message unpacking module, used for subscribing to and transmitting the task message set and unpacking the message to obtain an unpacked task set;

[0076] A business execution module, used to perform task logic analysis and business execution on the unpacked task set to obtain an execution state set;

[0077] The task feedback module is used to provide status feedback and task retry for the unpacked task set according to the execution status set to obtain a task processing log.

[0078] The embodiment of the present invention can realize task initialization and persistence of task data by performing task conversion and task warehousing operations, facilitate retry after subsequent task failure, and facilitate task status management and log recording. By performing trigger sharding and polling packaging, a task message set is obtained, which can realize efficient task allocation and parallel processing, improve the efficiency of task scheduling and task execution, reduce task processing delays, and improve resource optimization efficiency. By performing subscription transmission and message unpacking, batch distributed acquisition of messages can be realized, efficient task allocation and execution can be realized, and the efficiency and reliability of task processing can be improved.

[0079] By performing task logic analysis and business execution, efficient execution and execution records for different types of tasks can be achieved, thereby effectively improving the efficiency and reliability of task processing. By performing status feedback and task retry, feedback processing of various abnormal situations can be performed at any time to improve the robustness of task processing, and task processing logs can be generated to facilitate subsequent unification and improve the efficiency of task processing. Therefore, the distributed asynchronous delayed task processing method based on sharding tasks proposed in the present invention can solve the problem of low efficiency when performing distributed task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 A schematic diagram of a process flow of a distributed asynchronous delayed task processing method based on sharding tasks provided by an embodiment of the present invention;

[0081] Figure 2 A schematic diagram of a process for generating an unpacking task set provided by an embodiment of the present invention;

[0082] Figure 3 A schematic diagram of a process for generating an execution state set provided by an embodiment of the present invention;

[0083] Figure 4 A functional module diagram of a distributed asynchronous delayed task processing system based on sharding tasks provided by an embodiment of the present invention;

[0084] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0085] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0086] The embodiment of the present application provides a distributed asynchronous delayed task processing method based on sharding tasks. The execution subject of the distributed asynchronous delayed task processing method based on sharding tasks includes but is not limited to at least one of the electronic devices such as the server, the terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the distributed asynchronous delayed task processing method based on sharding tasks can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0087] Reference Figure 1 FIG. 1 is a flow chart of a distributed asynchronous delayed task processing method based on sharding tasks provided by an embodiment of the present invention. In this embodiment, the distributed asynchronous delayed task processing method based on sharding tasks includes:

[0088] S1. Perform task conversion and task storage operations on the pre-acquired task parameters to obtain task information.

[0089] Specifically, the task parameters refer to data such as task type, task delay time, number of retries, and business data. The task parameters may be data input by a user and acquired by a client.

[0090] In detail, the task information refers to data such as task ID, task name, task type, task start time, task end time, task delay time, task status, number of retries, business data, and execution configuration.

[0091] In the embodiment of the present invention, the step of performing task conversion and task storage operations on the pre-acquired task parameters to obtain task information includes:

[0092] Initialize the class of the pre-acquired task parameters to obtain the task information class;

[0093] Perform name configuration on the task parameters according to the task information class to obtain initial task parameters;

[0094] Performing delayed transformation on the initial task parameters to obtain delayed task parameters;

[0095] Calculating the execution time and configuring the execution state of the delayed task parameters to obtain execution task parameters;

[0096] The task information of the execution task parameters is stored in a database to obtain the task information.

[0097] Specifically, the class initialization refers to the use of information classes to perform task configuration corresponding to task parameters. The class class can be used for class initialization. The name configuration refers to determining whether the task information class corresponding to the task parameters is repeated. If not, the task ID and task name are configured for the task parameters to achieve unique identification of the task.

[0098] In detail, the initial task parameters have additional parameters such as task ID and task name compared to the task parameters, the delay conversion refers to configuring the corresponding task delay time for the task parameters, the execution time calculation refers to calculating the task start time and the corresponding task end time based on the current time and the task delay time, and the execution status configuration refers to setting the task status to be executed, executing, completed, failed, etc. according to the corresponding time.

[0099] Specifically, the delayed task parameters have additional parameters such as task delay time compared to the initial task parameters, and the execution task parameters have additional parameters such as task end time, task delay time, task status and number of retries compared to the delayed task parameters. The task information storage refers to storing the execution task parameters as task information in a preset database.

[0100] In the embodiment of the present invention, by performing task conversion and task storage operations, task initialization and task data persistence can be achieved, which is convenient for retrying subsequent tasks after failure, and for task status management and log recording.

[0101] S2. Trigger segmentation and polling packaging are performed on the task information to obtain a task message set.

[0102] In detail, each task message in the task message set refers to a trigger message of each sliced ​​task after the task is sliced, and the trigger message is a data message for allocating to an executor for task execution.

[0103] In the embodiment of the present invention, the triggering and slicing and polling and packaging of the task information to obtain the task message set includes:

[0104] Performing polling scheduling scanning on the task message to obtain scheduling task information;

[0105] Perform trigger calculation on the scheduling task information to obtain a trigger task time;

[0106] Perform trigger screening on the scheduled task message according to the trigger task time to obtain trigger task information;

[0107] Dividing the trigger task information into units to obtain a trigger subtask information set;

[0108] Performing hash time-volume business feature extraction on the triggered subtask information set to obtain a subtask type feature set, a subtask time feature set, a subtask data feature set, a subtask priority feature set, and a subtask hash feature set;

[0109] Performing covariance feature dimensionality reduction on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a triggering subtask feature set;

[0110] Clustering and sharding the trigger subtask information set according to the trigger subtask feature set to obtain a sharded task information set;

[0111] The shard task information set is packaged into messages to obtain a task message set.

[0112] In detail, the polling scheduling scan refers to regularly checking in a polling manner whether the task storage database contains task messages that need to be processed. The polling scheduling scan can be performed using a timer or a cycle extreme value.

[0113] Specifically, the trigger calculation refers to determining the corresponding trigger task time based on the start time and end time of each task in the scheduled task information set, and the trigger filtering refers to filtering out the task message whose trigger task time is the current time in the scheduled task message as the trigger task information.

[0114] In detail, the unit segmentation refers to splitting the trigger task information into multiple subtasks according to the work unit, and aggregating each subtask as trigger subtask information into a trigger subtask information set; the message packaging refers to splitting the sliced ​​task information set into several task batches according to system performance, creating corresponding task messages, and aggregating all task messages into a task message set.

[0115] Specifically, the triggering subtask information set is subjected to hash time-volume business feature extraction to obtain a subtask type feature set, a subtask time feature set, a subtask data feature set, a subtask priority feature set, and a subtask hash feature set, including:

[0116] Selecting the trigger subtask information in the trigger subtask information set one by one as the target subtask information, extracting the task type of the target subtask information, and obtaining a subtask type feature set;

[0117] Extracting the execution time of the target subtask information to obtain a subtask time feature set;

[0118] Extracting data volume from the target subtask information to obtain a subtask data feature set;

[0119] Priority extraction is performed on the target subtask information to obtain a subtask priority feature set;

[0120] Perform hash feature extraction on the target subtask information to obtain a subtask hash feature set.

[0121] Specifically, the task type refers to the specific category or function of the task, such as data processing tasks, file transfer tasks, report generation tasks, etc. The execution time refers to the time required to execute the task unit corresponding to the target subtask information. The data volume refers to the size or quantity of the data designed for the target subtask information. The priority refers to the urgency and priority of the target subtask information. The hash feature extraction refers to the hash value obtained by hashing the attributes of the target subtask information such as the task ID and task name.

[0122] Specifically, the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set are subjected to covariance feature dimensionality reduction to obtain a triggering subtask feature set, including:

[0123] Performing global pooling dimensionality reduction and feature matrix operations on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a subtask feature matrix;

[0124] Calculating the feature mean of the subtask feature matrix to obtain a mean feature matrix;

[0125] The covariance matrix corresponding to the subtask feature matrix is ​​calculated according to the mean feature matrix using the following covariance feature formula:

[0126]

[0127] Wherein, S refers to the covariance matrix, n is the total number of samples of the subtask feature matrix, X refers to the subtask feature matrix, refers to the mean feature matrix, T is the transposed symbol;

[0128] Performing eigenvalue decomposition on the covariance matrix to obtain a subtask eigenvector group;

[0129] Generate a feature projection matrix according to the subtask feature vector group;

[0130] The projection variance corresponding to the feature projection matrix is ​​calculated according to the mean feature matrix and the subtask feature matrix using the following projection variance algorithm:

[0131]

[0132] Wherein, σ refers to the projection variance, d is the number of feature dimensions of the subtask feature matrix, X refers to the subtask feature matrix, refers to the mean feature matrix, T is the transposed symbol, and W refers to the feature projection matrix;

[0133] Iteratively screening and updating the feature projection matrix according to the projection variance to obtain a maximum projection matrix;

[0134] The subtask feature matrix is ​​projected with the maximum projection matrix to obtain a triggering subtask feature set.

[0135] Specifically, the feature matrixing refers to standardizing each feature in the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set and the subtask hash feature set, and vectorizing each feature set into a horizontal feature vector, and splicing all the horizontal feature vectors into a subtask feature matrix.

[0136] In detail, the covariance feature formula can capture the linear relationship between different features and reflect the correlation structure of the data, thereby improving the correlation of feature fusion. The projection variance algorithm can determine the maximum direction of data change and extract the main features, so that the fused trigger subtask feature set can retain the amount of information to the greatest extent.

[0137] Specifically, generating a feature projection matrix from the subtask feature vector group refers to selecting a number of subtask feature vectors from the subtask feature vector group and splicing them into a feature projection matrix; the iterative screening and updating refers to determining whether the projection variance is a maximum value; if so, taking the projection matrix corresponding to the projection variance as the maximum projection matrix; if not, returning to the step of generating a feature projection matrix according to the subtask feature vector group, and re-screening a number of subtask feature vectors to generate a feature projection matrix; the dimensionality reduction projection refers to multiplying the subtask feature matrix by the maximum projection matrix, and splitting the result into a triggering subtask feature set.

[0138] In detail, clustering and sharding the trigger subtask information set according to the trigger subtask feature set to obtain the sharded task information set includes:

[0139] Performing Gaussian distribution splitting on the trigger subtask feature set to obtain a plurality of subtask fragment feature groups;

[0140] The shard center features are selected from each subtask shard feature group, and the feature deviation values ​​between each shard center feature and each trigger subtask feature in the trigger subtask feature set are calculated using the following feature deviation algorithm:

[0141]

[0142] Wherein, P refers to the feature deviation value, μ is the preset adversarial coefficient, α refers to the shard center feature, β refers to the trigger subtask feature, · is the dot product symbol, and * is the cross product symbol;

[0143] Iteratively update each subtask slice feature group according to the feature deviation value in a manner of nearest grouping to obtain a slice subtask feature group set;

[0144] Splitting the trigger subtask information set into slice subtask information sets according to the slice subtask feature set;

[0145] Task information of each shard subtask information group in the shard subtask information group set is merged to obtain a shard task information set.

[0146] In detail, the Gaussian distribution splitting refers to screening out several trigger subtask features with dissimilar Gaussian distributions to form a subtask shard feature group according to the Gaussian distribution position corresponding to the trigger subtask feature set, and the shard center feature refers to the mean feature in each subtask shard feature group.

[0147] Specifically, the feature deviation algorithm can calculate the feature deviation based on the cosine similarity and Euclidean distance between the trigger subtask feature and the shard center feature, realizes the adjustability of the weight, can capture more complex feature relationships, and has greater flexibility and robustness. The iterative update method can be the k-means method.

[0148] In the embodiment of the present invention, by performing trigger sharding and polling packaging, a task message set is obtained, which can achieve efficient task allocation and parallel processing, improve the efficiency of task scheduling and task execution, reduce task processing delays, and improve resource optimization efficiency.

[0149] S3. Subscribe to the task message set and unpack the message to obtain an unpacked task set.

[0150] In detail, in order to achieve persistence of task data, maintain data consistency and enhance task retry management, tasks can be executed concurrently by using subscription.

[0151] In the embodiment of the present invention, refer to Figure 2 As shown, the subscribing, transmitting and unpacking the task message set to obtain the unpacked task set includes:

[0152] S21, sending the task message set to a preset subscription message queue;

[0153] S22, performing message persistence on the task message set in the subscription message queue to obtain a persistent message queue;

[0154] S23, subscribing to the persistent message queue to obtain a subscription message set;

[0155] S24, parsing the subscription message set to obtain a subscription task object set;

[0156] S25, extracting the subscribed task object and task information to obtain an unpacked task set.

[0157] Specifically, the subscription message queue refers to a message queue of a publish-subscribe (Pub / Sub) model, which may be a message queue such as RocketMQ or Kafka, and the message persistence refers to storing the task message set using the topic of the subscription message queue.

[0158] In detail, the queue subscription refers to using the subscriber of the subscription message queue to subscribe to a topic and receive the corresponding message. The message parsing method can be a parsing method of a JSON parsing library such as Gson, Jackson, or an XML parsing method. Task information can be extracted using information extraction methods such as taskId and taskType. The unpacked task set is a task information set corresponding to the sharded task information set.

[0159] In detail, by performing subscription transmission and message unpacking, batch distributed acquisition of messages can be achieved, efficient task allocation and execution can be achieved, and the efficiency and reliability of task processing can be improved.

[0160] S4. Perform task logic analysis and business execution on the unpacked task set to obtain an execution status set.

[0161] Specifically, each execution status in the execution status set represents the execution status of each unpacking task in the unpacking task set, such as execution failure or execution success. When the execution is successful, the execution time of the successful execution, the saving path of the task data and other execution results are also recorded.

[0162] In the embodiment of the present invention, refer to Figure 3 As shown, the unpacked task set is subjected to task logic analysis and business execution to obtain an execution state set, including:

[0163] S31, extracting time-volume business features from the unpacking task set to obtain an unpacking task type feature set, an unpacking task time feature set, an unpacking task data feature set, and an unpacking task priority feature set;

[0164] S32, prioritizing the unpacking task set according to the unpacking task priority feature set to obtain an unpacking task sequence;

[0165] S33, allocating processors to the unpacking task sequence according to the unpacking task type feature set to obtain a type processor set;

[0166] S34, performing resource allocation on the unpacking task sequence according to the unpacking task time feature set and the unpacking task data feature set to obtain a task resource set;

[0167] S35 , using the type processor set to perform task execution and execution record on the unpacked task sequence according to the task resource set, to obtain an execution status set.

[0168] Specifically, the priority sorting refers to sorting the unpacking tasks in the unpacking task set in order of priority, so that the unpacking tasks with higher priority levels are at the front of the unpacking task sequence.

[0169] In detail, the processor allocation refers to allocating different task processors to unpacking tasks with different unpacking task type characteristics in the unpacking task sequence, for example, allocating unpacking tasks with the unpacking task type characteristic of a data type to a data processor for execution.

[0170] Specifically, the resource allocation refers to determining a task resource attribute sequence corresponding to the unpacking task sequence according to the unpacking task time feature set and the unpacking task data feature set, and allocating resources to the unpacking task sequence according to the task resource attribute sequence.

[0171] In the embodiment of the present invention, by performing task logic analysis and business execution, efficient execution and execution records of different types of tasks can be achieved, thereby effectively improving the efficiency and reliability of task processing.

[0172] S5. Perform status feedback and task retry on the unpacked task set according to the execution status set to obtain a task processing log.

[0173] In detail, the task processing log records the status of task processing, including whether the task execution is successful, the number of task retries, the timestamp of task execution, the task execution result and the corresponding save path, etc.

[0174] In the embodiment of the present invention, the step of performing status feedback and task retry on the unpacking task set according to the execution status set to obtain a task processing log includes:

[0175] Selecting unpacking tasks in the unpacking task set one by one as target unpacking tasks, and selecting execution states corresponding to the target unpacking tasks from the execution state set as target execution states;

[0176] Performing state feedback on the target execution state to obtain feedback control information;

[0177] Determine whether the target execution status is execution failure;

[0178] If not, generating a task processing log according to the target execution status;

[0179] If yes, retry the target unpacking task according to the feedback control information to obtain a target retry status, and generate a task processing log according to the target retry status;

[0180] The target execution status is updated using the target retry status, and the process returns to the step of determining whether the target execution status indicates execution failure.

[0181] In detail, the status feedback refers to sending the target execution status to the user end, obtaining feedback control information from the user end regarding the target execution status feedback, and the feedback control information is used to determine whether the target unpacking task needs to be retried. The feedback control information includes retry and no retry.

[0182] Specifically, generating a task processing log refers to extracting execution results such as the time of successful execution, number of retries, and task data saving path from the target execution status or the target retry status, and generating the task processing log in XML encoding or YAML encoding.

[0183] In the embodiment of the present invention, by performing status feedback and task retry, feedback processing of various abnormal situations can be performed at any time, the robustness of task processing can be improved, and a task processing log can be generated to facilitate subsequent unification and improve the efficiency of task processing.

[0184] The embodiment of the present invention can realize task initialization and persistence of task data by performing task conversion and task warehousing operations, facilitate retry after subsequent task failure, and facilitate task status management and log recording. By performing trigger sharding and polling packaging, a task message set is obtained, which can realize efficient task allocation and parallel processing, improve the efficiency of task scheduling and task execution, reduce task processing delays, and improve resource optimization efficiency. By performing subscription transmission and message unpacking, batch distributed acquisition of messages can be realized, efficient task allocation and execution can be realized, and the efficiency and reliability of task processing can be improved.

[0185] By performing task logic analysis and business execution, efficient execution and execution records for different types of tasks can be achieved, thereby effectively improving the efficiency and reliability of task processing. By performing status feedback and task retry, feedback processing of various abnormal situations can be performed at any time to improve the robustness of task processing, and task processing logs can be generated to facilitate subsequent unification and improve the efficiency of task processing. Therefore, the distributed asynchronous delayed task processing method and system based on sharding tasks proposed in the present invention can solve the problem of low efficiency when performing distributed task processing.

[0186] like Figure 4 , which is a functional module diagram of a distributed asynchronous delayed task processing system based on sharding tasks provided by an embodiment of the present invention.

[0187] The distributed asynchronous delayed task processing system 100 based on sharding tasks of the present invention can be installed in an electronic device. According to the functions implemented, the distributed asynchronous delayed task processing system 100 based on sharding tasks can include a task storage module 101, a polling packaging module 102, a message unpacking module 103, a business execution module 104 and a task feedback module 105. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0188] In this embodiment, the functions of each module / unit are as follows:

[0189] The task storage module 101 is used to perform task conversion and task storage operations on pre-acquired task parameters to obtain task information;

[0190] The polling and packaging module 102 is used to perform trigger feature segmentation and polling packaging on the task information to obtain a task message set;

[0191] The message unpacking module 103 is used to subscribe to the task message set and unpack the message to obtain an unpacked task set;

[0192] The business execution module 104 is used to perform task logic analysis and business execution on the unpacked task set to obtain an execution state set;

[0193] The task feedback module 105 is used to perform status feedback and task retry on the unpacked task set according to the execution status set to obtain a task processing log.

[0194] In detail, each module described in the distributed asynchronous delayed task processing system 100 based on sharding tasks in the embodiment of the present invention is used in the same manner as described above. Figures 1 to 3 The distributed asynchronous delayed task processing method based on sharding tasks described in the invention has the same technical means and can produce the same technical effects, so it will not be repeated here.

[0195] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0196] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0198] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0199] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0200] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or systems stated in the system embodiment can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A distributed asynchronous delayed task processing method based on sharding tasks, characterized in that: The method comprises: Perform task conversion and task storage operations on the pre-acquired task parameters to obtain task information; Perform trigger feature segmentation and polling packaging on the task information to obtain a task message set; Subscribe to the task message set and unpack the message to obtain an unpacked task set; Performing task logic analysis and business execution on the unpacked task set to obtain an execution status set; According to the execution status set, status feedback and task retry are performed on the unpacked task set to obtain a task processing log.

2. The distributed asynchronous delayed task processing method based on sharding tasks according to claim 1, characterized in that: The task parameters acquired in advance are converted into tasks and stored in the task database to obtain task information, including: Initialize the class of the pre-acquired task parameters to obtain the task information class; Perform name configuration on the task parameters according to the task information class to obtain initial task parameters; Performing delayed transformation on the initial task parameters to obtain delayed task parameters; Calculating the execution time and configuring the execution state of the delayed task parameters to obtain execution task parameters; The task information of the execution task parameters is stored in a database to obtain the task information.

3. The distributed asynchronous delayed task processing method based on sharding tasks according to claim 1 is characterized in that: The triggering and slicing and polling and packaging of the task information to obtain a task message set includes: Performing polling scheduling scanning on the task message to obtain scheduling task information; Perform trigger calculation on the scheduling task information to obtain a trigger task time; Perform trigger screening on the scheduled task message according to the trigger task time to obtain trigger task information; Dividing the trigger task information into units to obtain a trigger subtask information set; Performing hash time-volume business feature extraction on the triggered subtask information set to obtain a subtask type feature set, a subtask time feature set, a subtask data feature set, a subtask priority feature set, and a subtask hash feature set; Performing covariance feature dimensionality reduction on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a triggering subtask feature set; Clustering and sharding the trigger subtask information set according to the trigger subtask feature set to obtain a sharded task information set; The shard task information set is packaged into messages to obtain a task message set.

4. The distributed asynchronous delayed task processing method based on sharding tasks as claimed in claim 3 is characterized in that: The step of performing hash time-volume business feature extraction on the trigger subtask information set to obtain a subtask type feature set, a subtask time feature set, a subtask data feature set, a subtask priority feature set, and a subtask hash feature set includes: Selecting the trigger subtask information in the trigger subtask information set one by one as the target subtask information, extracting the task type of the target subtask information, and obtaining a subtask type feature set; Extracting the execution time of the target subtask information to obtain a subtask time feature set; Extracting data volume from the target subtask information to obtain a subtask data feature set; Priority extraction is performed on the target subtask information to obtain a subtask priority feature set; Perform hash feature extraction on the target subtask information to obtain a subtask hash feature set.

5. The distributed asynchronous delayed task processing method based on sharding tasks as claimed in claim 3 is characterized in that: The performing covariance feature dimensionality reduction on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a triggering subtask feature set includes: Performing global pooling dimensionality reduction and feature matrix operations on the subtask type feature set, the subtask time feature set, the subtask data feature set, the subtask priority feature set, and the subtask hash feature set to obtain a subtask feature matrix; Calculating the feature mean of the subtask feature matrix to obtain a mean feature matrix; The covariance matrix corresponding to the subtask feature matrix is ​​calculated according to the mean feature matrix using the following covariance feature formula: Wherein, S refers to the covariance matrix, n is the total number of samples of the subtask feature matrix, X refers to the subtask feature matrix, refers to the mean feature matrix, T is the transposed symbol; Performing eigenvalue decomposition on the covariance matrix to obtain a subtask eigenvector group; Generate a feature projection matrix according to the subtask feature vector group; The projection variance corresponding to the feature projection matrix is ​​calculated according to the mean feature matrix and the subtask feature matrix using the following projection variance algorithm: Wherein, σ refers to the projection variance, d is the number of feature dimensions of the subtask feature matrix, X refers to the subtask feature matrix, refers to the mean feature matrix, T is the transposed symbol, and W refers to the feature projection matrix; Iteratively screening and updating the feature projection matrix according to the projection variance to obtain a maximum projection matrix; The subtask feature matrix is ​​projected with the maximum projection matrix to obtain a triggering subtask feature set.

6. The distributed asynchronous delayed task processing method based on sharding tasks as claimed in claim 3 is characterized in that: The clustering and sharding of the trigger subtask information set according to the trigger subtask feature set to obtain the sharded task information set includes: Performing Gaussian distribution splitting on the trigger subtask feature set to obtain a plurality of subtask fragment feature groups; The shard center features are selected from each subtask shard feature group, and the feature deviation values ​​between each shard center feature and each trigger subtask feature in the trigger subtask feature set are calculated using the following feature deviation algorithm: Wherein, P refers to the feature deviation value, μ is the preset adversarial coefficient, α refers to the shard center feature, β refers to the trigger subtask feature, · is the dot product symbol, and * is the cross product symbol; Iteratively update each subtask slice feature group according to the feature deviation value in a manner of nearest grouping to obtain a slice subtask feature group set; Splitting the trigger subtask information set into slice subtask information sets according to the slice subtask feature set; Task information of each shard subtask information group in the shard subtask information group set is merged to obtain a shard task information set.

7. The distributed asynchronous delayed task processing method based on sharding tasks according to claim 1, characterized in that: The step of subscribing, transmitting and unpacking the task message set to obtain the unpacked task set includes: Sending the task message set to a preset subscription message queue; Performing message persistence on the task message set in the subscription message queue to obtain a persistent message queue; Subscribing to the persistent message queue to obtain a subscription message set; Parsing the subscription message set to obtain a subscription task object set; The subscribed task object is extracted and task information is extracted to obtain an unpacked task set.

8. The distributed asynchronous delayed task processing method based on sharding tasks according to claim 1, characterized in that: The performing of task logic analysis and business execution on the unpacked task set to obtain an execution status set includes: Extracting time-volume business features from the unpacking task set to obtain an unpacking task type feature set, an unpacking task time feature set, an unpacking task data feature set, and an unpacking task priority feature set; Prioritizing the unpacking task set according to the unpacking task priority feature set to obtain an unpacking task sequence; Allocating processors to the unpacking task sequence according to the unpacking task type feature set to obtain a type processor set; Allocate resources to the unpacking task sequence according to the unpacking task time feature set and the unpacking task data feature set to obtain a task resource set; The type processor set is used to perform task execution and execution record on the unpacked task sequence according to the task resource set to obtain an execution state set.

9. The distributed asynchronous delayed task processing method based on sharding tasks according to claim 1, characterized in that: The performing status feedback and task retry on the unpacked task set according to the execution status set to obtain a task processing log includes: Selecting unpacking tasks in the unpacking task set one by one as target unpacking tasks, and selecting execution states corresponding to the target unpacking tasks from the execution state set as target execution states; Performing state feedback on the target execution state to obtain feedback control information; Determine whether the target execution status is execution failure; If not, generating a task processing log according to the target execution status; If yes, retry the target unpacking task according to the feedback control information to obtain a target retry status, and generate a task processing log according to the target retry status; The target execution status is updated using the target retry status, and the process returns to the step of determining whether the target execution status indicates execution failure.

10. A distributed asynchronous delayed task processing system based on sharding tasks, characterized in that: The system comprises: The task storage module is used to perform task conversion and task storage operations on pre-acquired task parameters to obtain task information; A polling and packaging module, used for performing trigger feature segmentation and polling packaging on the task information to obtain a task message set; A message unpacking module, used for subscribing to and transmitting the task message set and unpacking the message to obtain an unpacked task set; A business execution module, used to perform task logic analysis and business execution on the unpacked task set to obtain an execution state set; The task feedback module is used to provide status feedback and task retry for the unpacked task set according to the execution status set to obtain a task processing log.

Images