A method and system for dynamic task execution based on multithreading

Abstract

This invention discloses a multi-threaded dynamic task execution method and system. The method includes: receiving an original task processing request containing formatted task data; generating a task processing request to be executed based on the original task processing request; creating a thread for the formatted task data in the task processing request to be executed and enabling a thread lock; processing the formatted task data with the thread lock enabled to generate formatted task data to be segmented; when the number of formatted task data to be segmented exceeds a set threshold for the amount of task data to be executed by a single thread, segmenting the formatted task data to be segmented according to a pre-set task data segmentation rule to generate multiple task datasets; and executing the formatted task data in the multiple task datasets using multiple threads according to a set number of threads that can be enabled. This method and system reduce the number of interactions and improve response efficiency when there are a large number of concurrent user requests.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method and system for dynamic task execution based on multithreading. Background Technology

[0002] The core architecture of internet-based information systems is the browser / server architecture. The B / S architecture reduces the system's environmental requirements. However, due to browser performance limitations, system task processing is concentrated on the server side. In actual use, when the number of data requests is large, the system's response speed slows down. Furthermore, because different users have varying request volumes, the workload the system needs to handle is uneven. When some large data volume tasks take too long to process, it will negatively impact the user experience. Summary of the Invention

[0003] To address the problems in existing technologies, such as long concurrent response times and poor user experience caused by large data concurrency and uneven task distribution among different users when task processing is concentrated on the server side in a B / S architecture, embodiments of the present invention provide a dynamic task execution method and system based on multithreading.

[0004] According to one aspect of the present invention, a dynamic task execution method based on multithreading is provided, the method comprising:

[0005] Receive raw task processing requests containing formatted task data;

[0006] Generate a task processing request to be executed based on the original task processing request;

[0007] Create a thread for the formatted task data in the pending task processing request, and enable the thread lock;

[0008] Process the formatted task data that has the thread lock enabled to generate formatted task data to be split;

[0009] When the amount of data to be formatted and divided exceeds the set threshold for the amount of data in a single-threaded execution task, the data to be formatted and divided is divided according to the pre-set task data splitting rules to generate multiple task datasets.

[0010] Based on the set number of threads that can be enabled, multiple threads execute the formatted task data in the multiple task datasets.

[0011] Optionally, in the above-described method embodiments of the present invention, the method further includes, before receiving the original task processing request containing formatted task data:

[0012] Query the task processing request queue and take the task processing requests in the queue whose time length for sending the request is less than the preset call interval value as the original task processing requests.

[0013] Optionally, in the above-described method embodiments of the present invention, processing the formatted task data that enables the thread lock to generate formatted task data to be segmented includes:

[0014] Based on the date field contained in each task data, the formatted task data that enables thread locks is sorted in chronological order to generate the first formatted task data;

[0015] Perform a screening operation on the first formatted task data to generate formatted task data to be split, wherein the screening operation includes:

[0016] If one or more task data entries with missing data exist in the first formatted task data, delete the one or more task data entries with missing data.

[0017] If duplicate task data exists in the first formatted task data, only one task data record is retained, and the other duplicate task data is deleted.

[0018] Optionally, in the above-described method embodiments of the present invention, the date field of the formatted task data to be segmented is extracted as a data feature;

[0019] The data to be formatted is segmented according to the data characteristics, wherein:

[0020] The segmentation time interval J is calculated based on the data characteristics, and the calculation formula is as follows:

[0021]

[0022] X = T * Y / C T

[0023] In the formula, T is the total time length of the formatted task data to be segmented, determined based on the data characteristics; Y is a pre-set threshold for the amount of task data executed by a single thread; and C... T The total amount of data to be formatted and divided in the total time length T;

[0024] Based on the segmentation time interval J and the total time length T, the formatting task data to be segmented is divided into the maximum number of segments. There are several task datasets. If the i-th task dataset is empty, delete the i-th task dataset. If the number of formatted task data in the i-th task dataset is greater than Y, further split the formatted task data in the i-th task dataset into... A task dataset, C i Let be the total amount of formatted task data to be split in the i-th task dataset.

[0025] Optionally, in the above-described method embodiments of the present invention, the method further includes setting a thread lock operation flag and an execution result flag for formatting task data, wherein:

[0026] When generating a task processing request to be executed based on the original task processing request, if there is formatted task data with the execution result flag set to the third result flag in the original task processing request, the thread lock operation flag of the formatted task data with the execution result flag set to the third result flag is set to the first operation flag, and the thread lock operation flag of the formatted task data with the third result flag not present in the original task processing request is set to the first operation flag. The original task processing request with the thread lock flag set to the first operation flag for all formatted task data is taken as the task processing request to be executed.

[0027] When a thread is created for the formatted task data in the pending task processing request and a thread lock is enabled, the thread lock operation flag for the formatted task data in the pending task processing request is set to the second operation flag.

[0028] If the duration of the second operation flag in the formatted task data of the pending task processing request exceeds the set time threshold, the second operation flag in the formatted task data of the pending task processing request will be reset to the first operation flag.

[0029] When the set number of threads is enabled, multiple threads execute the formatted task data in the multiple task datasets until the task ends, wherein:

[0030] If the task is completed, set the execution result flag of the formatted task data as the first result flag;

[0031] If the task is not completed and the duration of the second operation flag for formatting task data is not greater than the set time threshold, the execution result flag for formatting task data is set to the second result flag.

[0032] If the task is not completed and the duration of the second operation flag for formatting task data is greater than the set time threshold, and the task result is empty, the execution result flag for formatting task data is set to the third result flag, and the formatting task data with the execution result flag set to the third result flag is put back into the task processing request queue.

[0033] According to another aspect of the present invention, a multi-threaded dynamic task execution system is provided, the system comprising:

[0034] The request invocation unit is used to invoke the original task processing request containing formatted task data.

[0035] The request processing unit is used to generate a task processing request to be executed based on the original task processing request.

[0036] Create a thread unit to create a thread for the formatted task data in the task processing request to be executed, and enable the thread lock;

[0037] The data processing unit is used to process the formatted task data that has the thread lock enabled, and generate the formatted task data to be segmented.

[0038] The data segmentation unit is used to segment the formatted task data to be segmented according to a preset task data segmentation rule to generate multiple task datasets when the amount of the formatted task data to be segmented is greater than the set threshold for the amount of data of the single-threaded execution task data.

[0039] The task execution unit is used to execute formatted task data in the multiple task datasets in a multi-threaded manner according to the set number of threads that can be enabled.

[0040] Optionally, in the above-described device embodiments of the present invention, the system further includes a request query unit, used to query the task processing request queue and take the task processing requests in the task processing request queue whose time length for sending the request is less than a preset call time interval value as the original task processing requests.

[0041] Optionally, in the above-described apparatus embodiments of the present invention, the data processing unit processes the formatted task data that unlocks the thread lock to generate formatted task data to be segmented, including:

[0042] Based on the date field contained in each task data, the formatted task data that enables thread locks is sorted in chronological order to generate the first formatted task data;

[0043] Perform a screening operation on the first formatted task data to generate formatted task data to be split, wherein the screening operation includes:

[0044] If one or more task data entries with missing data exist in the first formatted task data, delete the one or more task data entries with missing data.

[0045] If duplicate task data exists in the first formatted task data, only one task data record is retained, and the other duplicate task data is deleted.

[0046] Optionally, in the above-described apparatus embodiments of the present invention, the data segmentation unit segments the formatted task data to be segmented according to a pre-set task data segmentation rule to generate multiple task datasets, including:

[0047] Extract the date field from the formatted task data to be segmented as a data feature;

[0048] The data to be formatted is segmented according to the data characteristics, wherein:

[0049] The segmentation time interval J is calculated based on the data characteristics, and the calculation formula is as follows:

[0050]

[0051] X = T * Y / C T

[0052] In the formula, T is the total time length of the formatted task data to be segmented, determined based on the data characteristics; Y is a pre-set threshold for the amount of task data executed by a single thread; and C... T The total amount of data to be formatted and divided in the total time length T;

[0053] Based on the segmentation time interval J and the total time length T, the formatting task data to be segmented is divided into the maximum number of segments. There are 3 task datasets. If the i-th task dataset is empty, delete the i-th task dataset. If the number of formatted task data in the i-th task dataset is greater than Y, further split the formatted task data in the i-th task dataset into 3 sub-datasets. A task dataset, C i Let be the total amount of formatted task data to be split in the i-th task dataset.

[0054] Optionally, in the above-described apparatus embodiments of the present invention, the system further includes a flag setting unit, used to set a thread lock operation flag and an execution result flag for formatting task data, wherein:

[0055] When generating a task processing request to be executed based on the original task processing request, if there is formatted task data with the execution result flag set to the third result flag in the original task processing request, the thread lock operation flag of the formatted task data with the execution result flag set to the third result flag is set to the first operation flag, and the thread lock operation flag of the formatted task data with the third result flag not present in the original task processing request is set to the first operation flag. The original task processing request with the thread lock flag set to the first operation flag for all formatted task data is taken as the task processing request to be executed.

[0056] When a thread is created for the formatted task data in the pending task processing request and a thread lock is enabled, the thread lock operation flag for the formatted task data in the pending task processing request is set to the second operation flag.

[0057] If the duration of the second operation flag in the formatted task data of the pending task processing request exceeds the set time threshold, the second operation flag in the formatted task data of the pending task processing request will be reset to the first operation flag.

[0058] When the set number of threads is enabled, multiple threads execute the formatted task data in the multiple task datasets until the task ends, wherein:

[0059] If the task is completed, set the execution result flag of the formatted task data as the first result flag;

[0060] If the task is not completed and the duration of the second operation flag for formatting task data is not greater than the set time threshold, the execution result flag for formatting task data is set to the second result flag.

[0061] If the task is not completed and the duration of the second operation flag for formatting task data is greater than the set time threshold, and the task result is empty, the execution result flag for formatting task data is set to the third result flag, and the formatting task data with the execution result flag set to the third result flag is put back into the task processing request queue.

[0062] Based on the above embodiments of the present invention, the method and system for dynamic task execution based on multithreading include: receiving an original task processing request containing formatted task data; generating a task processing request to be executed based on the original task processing request; creating a thread for the formatted task data in the task processing request to be executed and enabling a thread lock; processing the formatted task data with the thread lock enabled to generate formatted task data to be segmented; when the number of formatted task data to be segmented is greater than a set threshold for the amount of task data to be executed by a single thread, segmenting the formatted task data to be segmented according to a pre-set task data segmentation rule to generate multiple task datasets; and executing the formatted task data in the multiple task datasets using multiple threads according to a set number of threads that can be enabled.

[0063] The beneficial effects of the method and system include:

[0064] 1. When the number of concurrent user requests is large, the network transmission time is long, and the user response time requirement is high, the data characteristics of the formatted task data are extracted, and the task data is divided into multiple task datasets by using pre-defined task data segmentation rules. This makes the amount of data for each executed task relatively reasonable, thereby making the consumption of system resources by task execution more reasonable and reducing the number of network interactions.

[0065] 2. The present invention also sets a thread lock operation flag and an execution result flag, which effectively avoids multiple threads operating on the same data and ensures data security. Furthermore, by using the execution result flag, it distinguishes between problems with the task data itself and execution failures caused by reasons other than the data, thereby ensuring that multi-threaded task execution is possible and that data that fails to execute due to abnormal reasons can be executed again.

[0066] 3. This invention also optimizes the use of thread pools. When creating threads based on task data, it limits the number of threads that each user can enable, thereby ensuring that each user has a corresponding process to execute tasks.

[0067] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0068] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0069] Figure 1 This is a flowchart illustrating a multi-threaded dynamic task execution method provided in an exemplary embodiment of the present invention.

[0070] Figure 2 This is a schematic diagram of the structure of a multi-threaded dynamic task execution system provided by an exemplary embodiment of the present invention. Detailed Implementation

[0071] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0072] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0073] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0074] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0075] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.

[0076] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0077] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0078] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0079] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0080] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0081] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0082] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0083] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0084] Exemplary methods

[0085] Figure 1 This is a flowchart illustrating a multi-threaded dynamic task execution method provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as… Figure 1 As shown, the method for incremental training and continuous deployment of the text classification model in this embodiment includes the following steps:

[0086] Step 101: Receive the raw task processing request containing formatted task data.

[0087] In one embodiment, this method can be started when a callable raw task processing request exists, receiving a raw task processing request containing formatted task data and preparing to start a thread to execute the task.

[0088] In step 102, a task processing request to be executed is generated based on the original task processing request.

[0089] In one embodiment, since the original task processing request may include task data that was not completed due to network anomalies or excessively long response times, it is necessary to process this type of data in the original task processing request to generate a task processing request to be executed. This embodiment effectively solves this problem by marking the above-mentioned situations with an execution result flag.

[0090] In step 103, a thread is created for the formatted task data in the task processing request to be executed, and a thread lock is enabled.

[0091] In one embodiment, to avoid delaying the task response time of other users due to the excessive amount of task data in a user's pending task processing request, this embodiment limits the number of threads that a user can enable. That is, when creating threads for the formatted task data in the pending task processing request, the number of threads cannot exceed the number of threads that can be enabled for that user.

[0092] In step 104, the formatted task data with the thread lock enabled is processed to generate formatted task data to be split.

[0093] Optionally, the formatted task data for which the thread lock is enabled is processed to generate formatted task data to be segmented, including:

[0094] Based on the date field contained in each task data, the formatted task data that enables thread locks is sorted in chronological order to generate the first formatted task data;

[0095] Perform a screening operation on the first formatted task data to generate formatted task data to be split, wherein the screening operation includes:

[0096] If one or more task data entries with missing data exist in the first formatted task data, delete the one or more task data entries with missing data.

[0097] If duplicate task data exists in the first formatted task data, only one task data record is retained, and the other duplicate task data is deleted.

[0098] In one embodiment, since the formatting task data for enabling thread locks may contain missing, erroneous, or redundant information, preprocessing is necessary to avoid wasting resources on formatting task data with such issues. Missing or duplicate data is deleted, and if necessary, the data can be annotated before being returned to the requester. Furthermore, to improve data preprocessing efficiency, authoritative versions of the formatting task data for enabling thread locks can be obtained from other sources. For example, for formatting task data submitted by taxpayers, standard data corresponding to the submitted formatting task data can be downloaded from the tax bureau system based on the taxpayer's relevant information, sorted by time, and then matched with the formatting task data for enabling thread locks to screen the data, thereby reducing matching time.

[0099] In step 105, when the number of data to be formatted and divided is greater than the set threshold for the amount of data in a single-threaded execution task, the data to be formatted and divided is divided according to the pre-set task data splitting rules to generate multiple task datasets.

[0100] Optionally, the formatted task data to be segmented is segmented according to a pre-set task data segmentation rule to generate multiple task datasets, including:

[0101] Extract the date field from the formatted task data to be segmented as a data feature;

[0102] The data to be formatted is segmented according to the data characteristics, wherein:

[0103] The segmentation time interval J is calculated based on the data characteristics, and the calculation formula is as follows:

[0104]

[0105] X = T * Y / C T

[0106] In the formula, T is the total time length of the formatted task data to be segmented, determined based on the data characteristics; Y is a pre-set threshold for the amount of task data executed by a single thread; and C... T The total amount of data to be formatted and divided in the total time length T;

[0107] Based on the segmentation time interval J and the total time length T, the formatting task data to be segmented is divided into the maximum number of segments. There are 3 task datasets. If the i-th task dataset is empty, delete the i-th task dataset. If the number of formatted task data in the i-th task dataset is greater than Y, further split the formatted task data in the i-th task dataset into 3 sub-datasets. A task dataset, C i Let be the total amount of formatted task data to be split in the i-th task dataset.

[0108] In one embodiment, the taxpayer provided tax data for the second quarter, totaling 90 days: 480 records for April 1st, 900 records for May 1st, 1200 records for May 30th, and 420 records for June 29th. If calculated in days, T represents 90 days. Within this 90-day time span, the total number of data entries to be formatted and segmented is 3000, while the threshold Y for the number of data entries to be processed in a single thread is set to 500. According to the formula for calculating the segmentation time interval J, J is 15, meaning the time interval for segmenting the task data is 15 days. Therefore, after arranging the tax data in sequence, it can be divided into 90 ÷ 15 = 6 task datasets. However, since there is no task data for April 16th to April 30th and June 1st to 15th, these two task datasets are empty and are therefore deleted. Since the data volume in the task dataset from May 16th to May 30th exceeds the set threshold of 500 for single-threaded task execution, this dataset needs to be further split. Specifically, the result of 1200 ÷ 500 is rounded up, dividing the data from May 16th to May 30th into three task datasets. For these three datasets, the goal is to ensure that the number of task data points in each dataset does not exceed 500. However, theoretically, an average split would be more efficient. As can be seen from the execution of the above task data splitting rules, by deleting task datasets with no task data and then further splitting task datasets whose data volume exceeds the set threshold for single-threaded task execution, execution of empty sets of tasks is avoided, and the number of task data points for each thread is within the thread's capacity, thus improving task execution efficiency in multiple ways.

[0109] It should be clearly pointed out that the data segmentation rules described in this embodiment only apply to the case where the amount of task data is greater than the preset threshold for the amount of task data to be executed by a single thread. When the amount of task data is greater than the preset threshold for the amount of task data to be executed by a single thread, there is no need to segment the task data at all, and the task data can be executed by a single thread directly.

[0110] In step 106, the formatted task data in the multiple task datasets is executed by multiple threads according to the set number of threads that can be enabled.

[0111] Optionally, before receiving the raw task processing request containing formatted task data, the following steps are also included:

[0112] Query the task processing request queue and take the task processing requests in the queue whose time length for sending the request is less than the preset call interval value as the original task processing requests.

[0113] In one embodiment, to avoid wasting resources by frequently invoking task processing requests while also preventing users from waiting too long for responses, a time interval value is set for invoking task processing requests, such as 20 minutes. When there are task processing requests in the task processing queue that are less than 20 minutes old, those requests are invoked as the original task processing requests.

[0114] Optionally, the method further includes setting a thread lock operation flag and an execution result flag for formatting task data, wherein:

[0115] When generating a task processing request to be executed based on the original task processing request, if there is formatted task data with the execution result flag set to the third result flag in the original task processing request, the thread lock operation flag of the formatted task data with the execution result flag set to the third result flag is set to the first operation flag, and the thread lock operation flag of the formatted task data with the third result flag not present in the original task processing request is set to the first operation flag. The original task processing request with the thread lock flag set to the first operation flag for all formatted task data is taken as the task processing request to be executed.

[0116] When a thread is created for the formatted task data in the pending task processing request and a thread lock is enabled, the thread lock operation flag for the formatted task data in the pending task processing request is set to the second operation flag.

[0117] If the duration of the second operation flag in the formatted task data of the pending task processing request exceeds the set time threshold, the second operation flag in the formatted task data of the pending task processing request will be reset to the first operation flag.

[0118] When the set number of threads is enabled, multiple threads execute the formatted task data in the multiple task datasets until the task ends, wherein:

[0119] If the task is completed, set the execution result flag of the formatted task data as the first result flag;

[0120] If the task is not completed and the duration of the second operation flag for formatting task data is not greater than the set time threshold, the execution result flag for formatting task data is set to the second result flag.

[0121] If the task is not completed and the duration of the second operation flag for formatting task data is greater than the set time threshold, and the task result is empty, the execution result flag for formatting task data is set to the third result flag, and the formatting task data with the execution result flag set to the third result flag is put back into the task processing request queue.

[0122] In one embodiment, for a system with asynchronous timer processing, there is a risk of multiple threads operating on the same data. Therefore, adding a thread lock operation flag field during data operation can effectively ensure that only one thread can execute a single piece of data. Furthermore, this embodiment also sets an execution result flag field to mark the task execution result. This ensures that even if the task data itself is correct, but the response time is too long due to network anomalies or other reasons, the incomplete task data can be returned to the task processing queue via the execution result flag, thus avoiding accidental manipulation of the task data.

[0123] Exemplary System

[0124] Figure 2 This is a schematic diagram of the structure of a multi-threaded dynamic task execution system provided in an exemplary embodiment of the present invention. For example... Figure 2 As shown, the multi-threaded dynamic task execution system described in this embodiment includes:

[0125] Request invocation unit 201 is used to invoke the original task processing request containing formatted task data;

[0126] Request processing unit 202 is used to generate a task processing request to be executed based on the original task processing request;

[0127] Create thread unit 203 to create a thread for the formatted task data in the task processing request to be executed, and enable the thread lock;

[0128] Data processing unit 204 is used to process the formatted task data that has the thread lock enabled, and generate formatted task data to be segmented;

[0129] Data segmentation unit 205 is used to segment the formatted task data to be segmented according to a preset task data segmentation rule to generate multiple task datasets when the number of data to be segmented and formatted task data is greater than the set data volume threshold for single-threaded execution task data.

[0130] The task execution unit 206 is used to execute formatted task data in the multiple task datasets in a multi-threaded manner according to the set number of threads that can be enabled.

[0131] Optionally, the system further includes a request query unit for querying the task processing request queue and taking task processing requests in the task processing request queue whose sending time is less than a preset call time interval value as the original task processing request.

[0132] Optionally, the data processing unit 204 processes the formatted task data that has the thread lock enabled to generate formatted task data to be segmented, including:

[0133] Based on the date field contained in each task data, the formatted task data that enables thread locks is sorted in chronological order to generate the first formatted task data;

[0134] Perform a screening operation on the first formatted task data to generate formatted task data to be split, wherein the screening operation includes:

[0135] If one or more task data entries with missing data exist in the first formatted task data, delete the one or more task data entries with missing data.

[0136] If duplicate task data exists in the first formatted task data, only one task data record is retained, and the other duplicate task data is deleted.

[0137] Optionally, the data segmentation unit 205 segments the formatted task data to be segmented according to a pre-set task data segmentation rule, generating multiple task datasets, including:

[0138] Extract the date field from the formatted task data to be segmented as a data feature;

[0139] The data to be formatted is segmented according to the data characteristics, wherein:

[0140] The segmentation time interval J is calculated based on the data characteristics, and the calculation formula is as follows:

[0141]

[0142] X = T * Y / C T

[0143] In the formula, T is the total time length of the formatted task data to be segmented, determined based on the data characteristics; Y is a pre-set threshold for the amount of task data executed by a single thread; and C... T The total amount of data to be formatted and divided in the total time length T;

[0144] Based on the segmentation time interval J and the total time length T, the formatting task data to be segmented is divided into the maximum number of segments. There are 3 task datasets. If the i-th task dataset is empty, delete the i-th task dataset. If the number of formatted task data in the i-th task dataset is greater than Y, further split the formatted task data in the i-th task dataset into 3 sub-datasets. A task dataset, C i Let be the total amount of formatted task data to be split in the i-th task dataset.

[0145] Optionally, the system further includes a flag setting unit for setting a thread lock operation flag and an execution result flag for formatting task data, wherein:

[0146] When generating a task processing request to be executed based on the original task processing request, if there is formatted task data with the execution result flag set to the third result flag in the original task processing request, the thread lock operation flag of the formatted task data with the execution result flag set to the third result flag is set to the first operation flag, and the thread lock operation flag of the formatted task data with the third result flag not present in the original task processing request is set to the first operation flag. The original task processing request with the thread lock flag set to the first operation flag for all formatted task data is taken as the task processing request to be executed.

[0147] When a thread is created for the formatted task data in the pending task processing request and a thread lock is enabled, the thread lock operation flag for the formatted task data in the pending task processing request is set to the second operation flag.

[0148] If the duration of the second operation flag in the formatted task data of the pending task processing request exceeds the set time threshold, the second operation flag in the formatted task data of the pending task processing request will be reset to the first operation flag.

[0149] When the set number of threads is enabled, multiple threads execute the formatted task data in the multiple task datasets until the task ends, wherein:

[0150] If the task is completed, set the execution result flag of the formatted task data as the first result flag;

[0151] If the task is not completed and the duration of the second operation flag for formatting task data is not greater than the set time threshold, the execution result flag for formatting task data is set to the second result flag.

[0152] If the task is not completed and the duration of the second operation flag for formatting task data is greater than the set time threshold, and the task result is empty, the execution result flag for formatting task data is set to the third result flag, and the formatting task data with the execution result flag set to the third result flag is put back into the task processing request queue.

[0153] Based on the above embodiments of the present invention, the method and system for dynamic task execution based on multithreading include: receiving an original task processing request containing formatted task data; generating a task processing request to be executed based on the original task processing request; creating a thread for the formatted task data in the task processing request to be executed and enabling a thread lock; processing the formatted task data with the thread lock enabled to generate formatted task data to be segmented; when the number of formatted task data to be segmented is greater than a set threshold for the amount of task data to be executed by a single thread, segmenting the formatted task data to be segmented according to a pre-set task data segmentation rule to generate multiple task datasets; and executing the formatted task data in the multiple task datasets using multiple threads according to a set number of threads that can be enabled.

[0154] Exemplary computer program products and computer-readable storage media

[0155] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods for incremental training and continuous deployment of a text classification model according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0156] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0157] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for incremental training and continuous deployment of a text classification model according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0158] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0159] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0160] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0161] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0162] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0163] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps are decomposable and / or recombinable. Such decomposition and / or recombination should be considered equivalent to the present disclosure. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0164] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A dynamic task execution method based on multithreading, characterized in that, The method includes: Query the task processing request queue and use task processing requests in the queue whose request duration is less than the preset call interval value as the original task processing requests; Receive raw task processing requests containing formatted task data; Generate a task processing request to be executed based on the original task processing request; Create a thread for the formatted task data in the pending task processing request, and enable the thread lock; Process the formatted task data that has the thread lock enabled to generate formatted task data to be split; When the amount of data to be formatted and segmented exceeds the set threshold for the amount of data in a single-threaded execution task, the data to be formatted and segmented is divided according to a pre-set task data segmentation rule to generate multiple task datasets, including: Extract the date field from the formatted task data to be segmented as a data feature; The data to be formatted is segmented according to the data characteristics, wherein: The segmentation time interval J is calculated based on the data characteristics, and the calculation formula is as follows: X=T*Y / C T In the formula, T is the total time length of the formatted task data to be segmented, determined based on the data characteristics; Y is a pre-set threshold for the amount of task data executed by a single thread; and C... T The total amount of data to be formatted and divided in the total time length T; Based on the segmentation time interval J and the total time length T, the formatting task data to be segmented is divided into the maximum number of segments. There are 3 task datasets. If the i-th task dataset is empty, delete the i-th task dataset. If the number of formatted task data in the i-th task dataset is greater than Y, further split the formatted task data in the i-th task dataset into 3 sub-datasets. For each task dataset, A task dataset, C i The total amount of formatted task data to be segmented in the dataset of the i-th task; Based on the set number of threads that can be enabled, multiple threads execute the formatted task data in the multiple task datasets.

2. The method according to claim 1, characterized in that, The formatted task data with the thread lock enabled is processed to generate the formatted task data to be split, including: Based on the date field contained in each task data, the formatted task data that enables thread locks is sorted in chronological order to generate the first formatted task data; Perform a screening operation on the first formatted task data to generate formatted task data to be split, wherein the screening operation includes: If one or more task data entries with missing data exist in the first formatted task data, delete the one or more task data entries with missing data. If duplicate task data exists in the first formatted task data, only one task data record is kept, and the other duplicate task data is deleted.

3. The method according to claim 1, characterized in that, The method further includes setting a thread lock operation flag and an execution result flag for formatting task data, wherein: When generating a task processing request to be executed based on the original task processing request, if there is formatted task data with the execution result flag set to the third result flag in the original task processing request, the thread lock operation flag of the formatted task data with the execution result flag set to the third result flag is set to the first operation flag, and the thread lock operation flag of the formatted task data with the third result flag not present in the original task processing request is set to the first operation flag. The original task processing request with the thread lock flag set to the first operation flag for all formatted task data is taken as the task processing request to be executed. When a thread is created for the formatted task data in the pending task processing request and a thread lock is enabled, the thread lock operation flag for the formatted task data in the pending task processing request is set to the second operation flag. If the duration of the second operation flag in the formatted task data of the pending task processing request exceeds the set time threshold, the second operation flag in the formatted task data of the pending task processing request will be reset to the first operation flag. When the set number of threads is enabled, multiple threads execute the formatted task data in the multiple task datasets until the task ends, wherein: If the task is completed, set the execution result flag of the formatted task data as the first result flag; If the task is not completed and the duration of the second operation flag for formatting task data is not greater than the set time threshold, the execution result flag for formatting task data is set to the second result flag. If the task is not completed and the duration of the second operation flag for formatting task data is greater than the set time threshold, and the task result is empty, the execution result flag for formatting task data is set to the third result flag, and the formatting task data with the execution result flag set to the third result flag is put back into the task processing request queue.

4. A dynamic task execution system based on multithreading, characterized in that, The system includes: The request query unit is used to query the task processing request queue and identify task processing requests in the queue whose request duration is less than a preset call interval value as original task processing requests. The request invocation unit is used to invoke the original task processing request containing formatted task data. The request processing unit is used to generate a task processing request to be executed based on the original task processing request. Create a thread unit to create a thread for the formatted task data in the task processing request to be executed, and enable the thread lock; The data processing unit is used to process the formatted task data that has the thread lock enabled, and generate the formatted task data to be segmented. A data segmentation unit is used to segment the formatted task data according to a pre-set task data segmentation rule when the amount of data to be segmented exceeds a set threshold for the amount of data in a single-threaded execution task, generating multiple task datasets, including: Extract the date field from the formatted task data to be segmented as a data feature; The data to be formatted is segmented according to the data characteristics, wherein: The segmentation time interval J is calculated based on the data characteristics, and the calculation formula is as follows: X=T*Y / C T In the formula, T is the total time length of the formatted task data to be segmented, determined based on the data characteristics; Y is a pre-set threshold for the amount of task data executed by a single thread; and C... T The total amount of data to be formatted and divided in the total time length T; Based on the segmentation time interval J and the total time length T, the formatting task data to be segmented is divided into the maximum number of segments. There are 3 task datasets. If the i-th task dataset is empty, delete the i-th task dataset. If the number of formatted task data in the i-th task dataset is greater than Y, further split the formatted task data in the i-th task dataset into 3 sub-datasets. A task dataset, C i The total amount of formatted task data to be segmented in the dataset of the i-th task; The task execution unit is used to execute formatted task data in the multiple task datasets in a multi-threaded manner according to the set number of threads that can be enabled.

5. The system according to claim 4, characterized in that, The data processing unit processes the formatted task data that has the thread lock enabled, generating the formatted task data to be segmented, including: Based on the date field contained in each task data, the formatted task data that enables thread locks is sorted in chronological order to generate the first formatted task data; Perform a screening operation on the first formatted task data to generate formatted task data to be split, wherein the screening operation includes: If one or more task data entries with missing data exist in the first formatted task data, delete the one or more task data entries with missing data. If duplicate task data exists in the first formatted task data, only one task data record is kept, and the other duplicate task data is deleted.

6. The system according to claim 4, characterized in that, The system also includes a flag setting unit for setting thread lock operation flags and execution result flags for formatting task data, wherein: When generating a task processing request to be executed based on the original task processing request, if there is formatted task data with the execution result flag set to the third result flag in the original task processing request, the thread lock operation flag of the formatted task data with the execution result flag set to the third result flag is set to the first operation flag, and the thread lock operation flag of the formatted task data with the third result flag not present in the original task processing request is set to the first operation flag. The original task processing request with the thread lock flag set to the first operation flag for all formatted task data is taken as the task processing request to be executed. When a thread is created for the formatted task data in the pending task processing request and a thread lock is enabled, the thread lock operation flag for the formatted task data in the pending task processing request is set to the second operation flag. If the duration of the second operation flag in the formatted task data of the pending task processing request exceeds the set time threshold, the second operation flag in the formatted task data of the pending task processing request will be reset to the first operation flag. When the set number of threads is enabled, multiple threads execute the formatted task data in the multiple task datasets until the task ends, wherein: If the task is completed, set the execution result flag of the formatted task data as the first result flag; If the task is not completed and the duration of the second operation flag for formatting task data is not greater than the set time threshold, the execution result flag for formatting task data is set to the second result flag. If the task is not completed and the duration of the second operation flag for formatting task data is greater than the set time threshold, and the task result is empty, the execution result flag for formatting task data is set to the third result flag, and the formatting task data with the execution result flag set to the third result flag is put back into the task processing request queue.

Images