Task processing system, method, device and computer-readable storage medium

Abstract

The present invention discloses a task processing system, method, device and computer-readable storage medium. It relates to the field of task processing. The method includes: when it is detected that the loading of the target batch unit is completed, obtaining the configuration information in the register; determining the task processing type of the target batch unit based on the configuration information; if the task processing type is the first processing type, updating the local task concurrent processor in the target batch unit to a remote task concurrent processor; if it is detected that there is a to-be-processed batch unit among multiple batch units, loading the to-be-processed batch unit to update the remote task concurrent processor in the to-be-processed batch unit to a local task concurrent processor. The present invention solves the technical problem of high labor costs caused by relying on manual implementation to switch between the local batch processing mode and the remote batch processing mode in the related art.

Description

Technical Field

[0001] The present invention relates to the field of task processing, and in particular, to a task processing system, method, device and computer-readable storage medium. Background Art

[0002] Typical batch processing features a large amount of data (ranging from millions to tens of millions or even hundreds of millions). Usually, tasks are partitioned (split into multiple subtasks) and processed locally using multi-threaded concurrent processing to improve batch processing efficiency. To distribute the system execution pressure, subtasks can also be distributed to remote batch units or machines for parallel processing to achieve batch remote partition processing. However, for the same batch unit, it cannot simultaneously achieve the distribution of local tasks (i.e., local batch processing mode) and the distribution of remote tasks (i.e., remote batch processing mode), and can only switch between the local batch processing mode and the remote batch processing mode.

[0003] Currently, with the lightweight and perfect Spring Batch batch processing framework, the task partitioning function has been satisfied. When it is necessary to switch the relevant batch unit from the local batch processing mode to the remote batch processing mode, it is necessary to manually extend Spring Batch based on Spring Intergration to Spring Batch Intergration to achieve the characteristics of batch remote partition execution. Among them, Spring-Intergration is used to provide an extension of the Spring programming model to support EIP (Enterprise Integration Patterns). Similarly, when it is necessary to switch the relevant batch unit from the remote batch processing mode to the local batch processing mode, relevant configurations or framework changes also need to be made manually, resulting in high labor costs and low switching efficiency.

[0004] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of the present invention provide a task processing system, method, device and computer-readable storage medium to at least solve the technical problem of high labor costs caused by relying on manual implementation of the switch between the local batch processing mode and the remote batch processing mode in related technologies.

[0006] According to one aspect of an embodiment of the present invention, a task processing system is provided, including: a plurality of batch units, each batch unit including a local task concurrent processor, wherein the local task concurrent processor is used to distribute local tasks; a register for storing the task processing type of each batch unit, and when it is determined that the task processing type corresponding to the target batch unit is the first processing type, updating the local task concurrent processor in the target batch unit to a remote task concurrent processor, wherein the target batch unit is any one of the plurality of batch units, the task processing type is the first processing type or the second processing type, the first processing type indicates that the current batch unit is used to distribute remote tasks to non-current batch units, the second processing type indicates that the current batch unit is used to process local tasks, and the remote task concurrent processor is used to distribute remote tasks; a loading module for detecting whether there is a batch unit to be processed among the plurality of batch units, and when there is a batch unit to be processed, loading the batch unit to be processed to update the remote task concurrent processor in the batch unit to be processed to a local task concurrent processor, wherein the batch unit to be processed is a batch unit whose task processing type is updated from the first processing type to the second processing type.

[0007] Further, each batch unit further includes: a task partitioning module for generating a plurality of local tasks or a plurality of remote tasks, and sending the plurality of local tasks to the local task concurrent processor, or sending the plurality of remote tasks to the remote task concurrent processor; wherein the local task concurrent processor is used to obtain the plurality of local tasks sent by the task partitioning module and schedule the sub-task processors in the current batch unit to process the plurality of local tasks; the remote task concurrent processor is used to obtain the plurality of remote tasks sent by the task partitioning module and distribute the task information corresponding to the plurality of remote tasks to at least one non-current batch unit, so that the non-current batch unit obtains the plurality of remote tasks according to the task information and processes the plurality of remote tasks.

[0008] Further, the remote task concurrent processor is used to obtain a plurality of remote tasks, allocate asynchronous threads to each remote task, and store the plurality of remote tasks in a target storage area, wherein each asynchronous thread is used to send the task information of the remote task corresponding to the asynchronous thread to a target message queue, and the target message queue is used to send the plurality of task information to at least one non-current batch unit, so that the non-current batch unit obtains the plurality of remote tasks according to the task information and processes the plurality of remote tasks.

[0009] Further, the remote task concurrent processor is further configured to query whether each task information has been acquired by a non-current batch unit after the target message queue sends multiple task information to at least one non-current batch unit, and when it is determined that there is target task information among the multiple task information, determine the sending time and the number of sending times of the target task information, and determine the processing method for the target task information based on the sending time and the number of sending times of the target task information, where the target task information is the information that has not been acquired by the non-current batch unit.

[0010] Further, the remote task concurrent processor is further configured to resend the target task information to the target message queue when the time interval between the sending time of the target task information and the current time is greater than or equal to a preset time interval and the number of sending times is less than the preset number of times; the remote task concurrent processor is further configured to prohibit resending the target task information to the target message queue when the time interval between the sending time of the target task information and the current time is greater than or equal to the preset time interval and the number of sending times is greater than or equal to the preset number of times.

[0011] Further, the target application service further includes: a remote task receiver, configured to acquire multiple task information sent by the target message queue, and after acquiring the multiple task information, acquire a remote task corresponding to each task information from the target storage area based on the multiple task information, and schedule a sub-task processor in at least one non-current batch unit to process the remote task.

[0012] According to another aspect of the embodiments of the present invention, there is also provided a task processing method, applied to the above task processing system, including: when it is detected that the target batch unit is loaded, acquire the configuration information in the register, where the target batch unit is any one of the multiple batch units; determine the task processing type of the target batch unit based on the configuration information; if the task processing type is the first processing type, update the local task concurrent processor in the target batch unit to a remote task concurrent processor; if it is detected that there is a to-be-processed batch unit among the multiple batch units, load the to-be-processed batch unit to update the remote task concurrent processor in the to-be-processed batch unit to a local task concurrent processor, where the to-be-processed batch unit is a batch unit whose task processing type is updated from the first processing type to the second processing type.

[0013] According to another aspect of the embodiments of the present invention, there is also provided a task processing device, including: an acquisition module, configured to acquire configuration information in a register when it is detected that a target batch unit is loaded completely, where the target batch unit is any one of a plurality of batch units; a determination module, configured to determine a task processing type of the target batch unit based on the configuration information; an update module, configured to update a local task concurrent processor in the target batch unit to a remote task concurrent processor if the task processing type is a first processing type; a loading module, configured to load a to-be-processed batch unit if it is detected that there is a to-be-processed batch unit among the plurality of batch units, so as to update a remote task concurrent processor in the to-be-processed batch unit to a local task concurrent processor, where the to-be-processed batch unit is a batch unit whose task processing type is updated from the first processing type to a second processing type.

[0014] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the above-mentioned task processing method when running.

[0015] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including one or more processors; a memory, configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement a program for running, where the program is configured to execute the above-mentioned task processing method when running.

[0016] In the above process, by storing the task processing type of each batch unit in the register, and when the batch unit is used to process remote tasks, updating the local task concurrent processor in the batch unit to a remote task concurrent processor, the rapid switching of the batch unit from the local task batch processing mode to the remote task batch processing mode is realized, and the problem of high labor cost caused by manual configuration change is avoided. Further, since the batch unit initially includes a local task concurrent processor, therefore, by detecting the update situation of the task processing type in the plurality of batch units through the loading module, and reloading the to-be-processed batch unit when there is a batch unit that needs to switch from the remote task batch processing to the local task batch processing, the rapid switching of the batch unit from the remote task batch processing mode to the local task batch processing mode is realized, thereby further reducing the labor cost.

[0017] It can be seen that the solution provided by the present application achieves the purpose of rapidly switching the batch unit between the remote task batch processing mode and the local task batch processing mode, thereby realizing the technical effect of reducing the labor cost, and further solving the technical problem of high labor cost caused by relying on manual operation to switch between the local batch processing mode and the remote batch processing mode in the related art. Brief Description of the Drawings

[0018] The drawings described herein are provided to further understand the present invention and form a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a schematic diagram of an optional task processing system according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of the local task batch processing mode and the remote task batch processing mode in the prior art;

[0021] Figure 3 is a schematic diagram of the processing of an optional remote task according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of the processing of a remote task in the prior art;

[0023] Figure 5 is a schematic diagram of an optional task processing method according to an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of an optional task processing device according to an embodiment of the present invention;

[0025] Figure 7 is a schematic diagram of an optional electronic device according to an embodiment of the present invention. Detailed Embodiments

[0026] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties.

[0029] Embodiment 1

[0030] According to an embodiment of the present invention, an embodiment of a task processing system is provided. As Figure 1 shown, the task processing system includes a target application service. Among them, the target application service includes:

[0031] A plurality of batch units. As Figure 1 shown, each batch unit includes a local task concurrent processor. Among them, the local task concurrent processor is used to distribute local tasks.

[0032] Optionally, the plurality of batch units are used to process tasks related to the target application service to ensure the normal operation of the target application service.

[0033] Specifically, each batch unit can be constructed based on the Spring Batch batch processing framework. Each batch unit includes at least a local task concurrent processor. When the application service starts, each batch unit will load the local task concurrent processor. Among them, as Figure 1 shown, the local task concurrent processor is used to distribute local tasks to sub-task processors in the same batch unit as the local task concurrent processor. A local task represents a task that needs to be processed in the batch unit where the local task is generated.

[0034] Optionally, as Figure 1As shown, the application service further includes a registrar. The registrar is used to store the task processing type of each batch unit, and when it is determined that the task processing type corresponding to the target batch unit is the first processing type, update the local task concurrent processor in the target batch unit to a remote task concurrent processor, where the target batch unit is any one of multiple batch units, the task processing type is the first processing type or the second processing type, the first processing type indicates that the current batch unit is used to distribute remote tasks to non-current batch units, the second processing type indicates that the current batch unit is used to process local tasks, and the remote task concurrent processor is used to distribute remote tasks.

[0035] Optionally, the task processing type of each batch unit can be pre-configured manually in the configuration information of the registrar. When it is detected that a batch unit in the task processing system has been loaded, the registrar can obtain the configuration information it stores, so as to determine the task processing type of the loaded batch unit based on the configuration information. Further, as Figure 1 shown, if the registrar determines that the task processing type corresponding to any one of multiple batch units (i.e., the aforementioned target batch unit) is to distribute remote tasks to non-current batch units, the registrar will unload the local task concurrent processor in this batch unit and load a remote task concurrent processor in this batch unit based on the preset configuration file in the application service, so as to update the local task concurrent processor in this batch unit to a remote task concurrent processor, that is, to achieve the Figure 1 configuration override in. Among them, the aforementioned remote task refers to a task that needs to be processed in a batch unit other than the batch unit that generates the remote task.

[0036] Among them, the remote task concurrent processor is used to distribute remote tasks to other batch units for processing, so as to relieve the execution pressure of the batch unit where the remote task concurrent processor is located and reduce resource occupancy.

[0037] Optionally, as Figure 1 shown, the target application service further includes a loading module. The loading module is used to detect whether there is a batch unit to be processed among multiple batch units, and in the case of the existence of a batch unit to be processed, load the batch unit to be processed to update the remote task concurrent processor in the batch unit to be processed to a local task concurrent processor, where the batch unit to be processed is a batch unit whose task processing type is updated from the first processing type to the second processing type.

[0038] Optionally, after multiple batch units are loaded, the staff can modify the configuration information of the batch units. Specifically, they can modify the task processing type corresponding to the batch units. Optionally, the target application service can also include a modification module. After multiple batch units are loaded, the modification module can modify the task processing type corresponding to the batch units at the corresponding time or under the corresponding conditions according to the preset configuration modification rules in the modification module.

[0039] Furthermore, the loading module can monitor the configuration information corresponding to the batch units in real time. When it detects that there is a batch unit in multiple batch units whose task processing type is updated from the first processing type to the second processing type (i.e., the aforementioned batch unit to be processed), it reloads the batch unit to be processed in the application service. Since the batch unit loads the local task concurrent processor by default during the loading process, it is possible to update the remote task concurrent processor in the batch unit to be processed to the local task concurrent processor.

[0040] In the related art, as Figure 2 shown, the related batch unit includes a task partition module, a local task concurrent processor, and a subtask processor. When it is necessary to switch the related batch unit from the local task batch processing mode to the remote task batch processing mode, that is, when Spring Batch is extended to Spring Batch Intergration based on Spring Intergration, the staff needs to independently configure a new remote task concurrent processor and a subtask processor bound to a specific business scenario in the batch unit, resulting in a high labor cost problem. Moreover, the remote task concurrent processor and the remote task receiver in the target application service can only be connected through the JMS (Java Message Service) interface, and the data transmission is slow. In addition, the batch unit in the related art is bound and coupled to a specific business partition scenario in the remote task batch processing mode, which is not conducive to the flexible addition and deletion of a wide range of business scenarios in the remote partitioning function.

[0041] It is easy to notice that in the above process, by storing the task processing type of each batch unit in the register and updating the local task concurrent processor in the batch unit to a remote task concurrent processor through the register when the batch unit is used to process remote tasks, a rapid switch of the batch unit from the local task batch processing mode to the remote task batch processing mode is achieved, avoiding the problem of high labor costs caused by manual configuration changes. Further, since the batch unit includes a local task concurrent processor initially, by the loading module detecting the update situation of the task processing type in multiple batch units and reloading the batch unit to be processed when there is a batch unit that needs to switch from remote task batch processing to local task batch processing, a rapid switch of the batch unit from the remote task batch processing mode to the local task batch processing mode is achieved, thereby further reducing the labor cost.

[0042] Thus, the solution provided by this application achieves the purpose of rapidly switching the batch unit between the remote task batch processing mode and the local task batch processing mode, thereby realizing the technical effect of reducing the labor cost, and further solving the technical problem of high labor cost caused by relying on manual implementation of the switch between the local batch processing mode and the remote batch processing mode in the related art.

[0043] In an alternative embodiment, as Figure 1 shown, each batch unit further includes: a task partitioning module, configured to generate multiple local tasks or multiple remote tasks, and send the multiple local tasks to the local task concurrent processor, or send the multiple remote tasks to the remote task concurrent processor. Among them, in the final stage of task partitioning, such as Figure 3 the task registration shown in Figure 3 , the task partitioning module may persistently store each sub-task information in a target storage area (i.e.,

[0044] the batch repository in

[0045] ), where the aforementioned target storage area may be JobRepository. Figure 1As shown in the figure, each batch unit further includes a sub-task processor. The local task concurrent processor is used to obtain multiple local tasks sent by the task partitioning module and asynchronously schedule the sub-task processors in the current batch unit to process the multiple local tasks in parallel.

[0046] Optionally, the remote task concurrent processor is used to obtain multiple remote tasks sent by the task partitioning module and distribute the task information corresponding to the multiple remote tasks to at least one non-current batch unit, so that the non-current batch unit can obtain multiple remote tasks according to the task information and process the multiple remote tasks. Among them, the aforementioned non-current batch unit can be in the same server as the current batch unit or in different servers from the current batch unit.

[0047] It should be noted that by using the local task concurrent processor to allocate multiple local tasks to the sub-task processors for execution, it is convenient to effectively process the multiple local tasks. By using the remote task concurrent processor to distribute the remote tasks to other batch units, it is convenient to effectively process the multiple remote tasks.

[0048] In an alternative embodiment, as Figure 3 shown, the remote task concurrent processor is used to obtain multiple remote tasks sent by the task partitioning module, and the remote task concurrent processor can pre-create an asynchronous thread pool. After the remote task concurrent processor obtains multiple remote tasks, the remote task concurrent processor can allocate asynchronous threads for each remote task and store the multiple remote tasks in the target storage area. For example, Figure 3 as shown in the figure, asynchronous thread 1 is allocated for task 1, asynchronous thread 2 is allocated for task 2, and asynchronous thread 3 is allocated for task 3.

[0049] After that, as Figure 3 shown, each asynchronous thread is used to send the task information of the remote task corresponding to the asynchronous thread to the target message queue. This process adopts an asynchronous mode and does not require the consumption result of receiving the message. Among them, it is preferably to introduce the spring-rabbit dependency in the batch unit and integrate the Rabbit MQ message middleware as the target message queue to achieve the information flow connection between the remote task concurrent processors and remote task receivers in different batch units.

[0050] Furthermore, the target message queue is used to send the multiple task information to at least one non-current batch unit through the remote task receiver, so that the non-current batch unit can obtain multiple remote tasks according to the task information and process the multiple remote tasks. Preferably, the non-current batch unit is an idle batch unit.

[0051] Among them, as Figure 1As shown, the target application service further includes a remote task receiver, which is used to obtain task information sent by each batch unit through the target message queue. Specifically, as Figure 3 shown, after the aforementioned remote task receiver obtains the task information sent by the current batch unit, it can mark the task information to indicate that the information has been consumed. At the same time, the marking result of the task information will be recorded in the aforementioned target storage area.

[0052] Furthermore, after the aforementioned remote task receiver marks the received task information, the remote task receiver can, based on the task information, obtain the corresponding remote tasks from the target storage area for each task information, and then schedule the sub-task processors in at least one non-current batch unit to process the remote tasks corresponding to the current batch unit, thereby completing the processing of the remote tasks. And after the processing is completed, as Figure 3 shown, each sub-task processor can store the execution results together in the target preset area for the batch unit that issued these remote tasks to obtain. Among them, in each batch unit, the sub-task processor for processing local tasks and remote tasks is the same processor.

[0053] It should be noted that by using the remote task concurrent processor to allocate remote tasks to at least one non-current batch unit by means of asynchronous threads, the processing efficiency of remote tasks can be effectively improved. The remote task receiver receives the remote tasks sent by other batch units and schedules the corresponding sub-task processors for processing, realizing the effective processing of remote tasks.

[0054] In an optional embodiment, the remote task concurrent processor is further used to, after the target message queue sends multiple task information to at least one non-current batch unit, the asynchronous thread can start polling, and the remote task concurrent processor can obtain the marking result of the task information by the remote task receiver in the non-current batch unit from the target storage area to query whether each task information has been obtained by the non-current batch unit.

[0055] Furthermore, when the remote task concurrent processor queries from the target storage area that there is unmarked task information among the multiple task information, it is determined that the task information has not been obtained by the non-current batch unit, that is, it is determined that the task information is the target task information. As Figure 3 shown, after the polling of the current round is completed, the remote task concurrent processor can determine the sending time and sending times of the target task information, and then, based on the sending time and sending times of the target task information, determine the processing method for the target task information. Among them, the processing method for the target task information can be to resend or cancel sending to the non-current batch unit.

[0056] It should be noted that in the related art, as Figure 4 shown, after the task partitioning module in the batch unit splits to obtain multiple remote tasks, it realizes task distribution through the remote task concurrent processor, sends the multiple remote tasks to the request queue, and sends the multiple remote tasks to other batch units (i.e., Figure 4 the non-current batch unit in ), and finally, obtains the response results of other batch units to implement the processing of remote tasks. That is, in the related art, problems such as message loss, duplication, and timeout in the message passing process are not considered, resulting in poor remote task processing effects. In this application, by tracking the sending results of remote tasks, the considerations of information loss, duplication, timeout, etc. in the message passing process are met, thereby improving the task processing effect.

[0057] In an alternative embodiment, the remote task concurrent processor is further configured to resend the target task information to the target message queue when the time interval between the sending time of the target task information and the current time is greater than or equal to a preset time interval and the number of sending times is less than the preset number of times, and is configured to prohibit resending the target task information to the target message queue when the time interval between the sending time of the target task information and the current time is greater than or equal to the preset time interval and the number of sending times is greater than or equal to the preset number of times.

[0058] For example, as Figure 3 shown, when the target task information times out and the number of retransmission times is less than the preset number of times, it can be considered that the non-current batch unit allocated previously is working saturated at this time, and the remote task concurrent processor can resend the target task information to the target message queue. Further, when the target task information times out and the number of retransmission times is greater than or equal to the preset number of times, it can be considered that there may be a problem with the target message queue at this time. Therefore, the remote task concurrent processor can no longer resend the target task information to the target message queue and register the task as failed in the target storage area. When all remote tasks are completed or registered as failed, it is determined that the processing of this remote task is completed.

[0059] It should be noted that by resending the target task information during the first few timeout processes of the target task information, the missed processing of remote tasks is avoided. By no longer resending the target task information when the number of retransmission times of the target task information is too large, the ineffective occupation of resources is avoided, thereby improving the task processing efficiency of this application.

[0060] It can be seen that in the present application, based on the Spring Batch batch processing framework, the Rabbit MQ message middleware is integrated to implement the function of batch remote partition execution. By decoupling the configuration in the remote batch processing mode from specific business scenarios, the flexible use of the remote partition function for various business scenarios is satisfied. By setting up a registrar and a loading module, the flexible switching of the partition configuration method is realized, and the ordinary partition (used to implement the local task batch processing mode) is conveniently switched to the remote partition (used to implement the remote task batch processing mode). In addition, the retry processing of the remote partition in scenarios such as loss, duplication, and timeout during the message passing process is also realized. Therefore, the solution provided by the present application achieves the purpose of quickly switching the batch unit between the remote task batch processing mode and the local task batch processing mode, thereby realizing the technical effect of reducing labor costs, and further solving the technical problem of high labor costs caused by relying on manual implementation of the switching between the local batch processing mode and the remote batch processing mode in the related art.

[0061] Embodiment 2

[0062] According to an embodiment of the present invention, an embodiment of a task processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0063] Figure 5 is a schematic diagram of an optional task processing method according to an embodiment of the present invention. As Figure 5 shown, this method is applied to the above-mentioned task processing system and includes the following steps:

[0064] Step S501, when it is detected that the loading of the target batch unit is completed, obtain the configuration information in the registrar, where the target batch unit is any one of the multiple batch units.

[0065] Optionally, when the application service is started, each batch unit will load a local task concurrent processor. Among them, as Figure 1 shown, the local task concurrent processor is used to distribute local tasks to the sub-task processors in the same batch unit as the local task concurrent processor. The local task represents the batch order in which the local task needs to be generated.

[0066] In step S501, the task processing type of each batch unit can be manually pre-configured in the configuration information of the registrar in advance. When it is detected that there is a batch unit (i.e., the aforementioned target batch unit) in the task processing system whose loading is completed, the registrar can obtain the configuration information stored in itself.

[0067] Step S502: Determine the task processing type of the target batch unit based on the configuration information.

[0068] In step S502, the register can determine the task processing type of the aforementioned loaded batch unit based on the configuration information.

[0069] Step S503: If the task processing type is the first processing type, change the local task concurrent processor to a remote task concurrent processor.

[0070] In step S503, if the register determines that the task processing type of any one of the multiple batch units (i.e., the aforementioned target batch unit) is for remote tasks, the register will unload the local task concurrent processor in this batch unit and load a remote task concurrent processor in this batch unit based on the preset configuration file in the application service, thereby realizing the update of the local task concurrent processor in this batch unit to a remote task concurrent processor, that is, realizing the Figure 1 configuration override. Among them, the aforementioned remote task represents a task that needs to be processed in a batch unit other than the batch unit where the remote task is generated. The remote task concurrent processor is used to distribute remote tasks to other batch units for processing to relieve the execution pressure of the batch unit where the remote task concurrent processor is located and reduce resource occupancy.

[0071] Step S504: If it is detected that there is a batch unit to be processed among the multiple batch units, load the batch unit to be processed to update the remote task concurrent processor in the batch unit to be processed to a local task concurrent processor, where the batch unit to be processed is a batch unit whose task processing type is updated from the first processing type to the second processing type.

[0072] Optionally, after the multiple batch units are loaded, the staff can modify the configuration information of the batch unit. Specifically, they can modify the task processing type corresponding to the batch unit. Optionally, the target application service can also include a modification module. The modification module can, after the multiple batch units are loaded, modify the task processing type corresponding to the batch unit at the corresponding time or condition according to the preset configuration modification rules in the modification module.

[0073] Furthermore, the loading module can monitor the configuration information corresponding to the batch unit in real time, and when it detects that there is a batch unit whose task processing type is updated from the first processing type to the second processing type (i.e., the aforementioned batch unit to be processed) among the multiple batch units, reload the batch unit to be processed in the application service. Since the batch unit loads the local task concurrent processor by default during the loading process, it can realize the update of the remote task concurrent processor in the batch unit to be processed to a local task concurrent processor.

[0074] Based on the solution defined in the above steps S501 to S503, it can be learned that in the embodiment of the present invention, by adopting a method of quickly switching the batch unit between the remote task batch processing mode and the local task batch processing mode, when it is detected that the target batch unit is loaded, the configuration information in the register is obtained, and then the task processing type of the target batch unit is determined based on the configuration information. Therefore, if the task processing type is the first processing type, the local task concurrent processor in the target batch unit is updated to a remote task concurrent processor. If it is detected that there is a batch unit to be processed among multiple batch units, the batch unit to be processed is loaded to update the remote task concurrent processor in the batch unit to be processed to a local task concurrent processor, where the target batch unit is any one of the multiple batch units, and the batch unit to be processed is a batch unit whose task processing type is updated from the first processing type to the second processing type.

[0075] It is easy to notice that in the above process, by storing the task processing type of each batch unit in the register, and when the batch unit is used to process remote tasks, updating the local task concurrent processor in the batch unit to a remote task concurrent processor, the quick switching of the batch unit from the local task batch processing mode to the remote task batch processing mode is realized, avoiding the problem of high labor cost caused by manual configuration changes. Further, since the batch unit initially includes a local task concurrent processor, by detecting the update situation of the task processing type in multiple batch units through the loading module, and reloading the batch unit to be processed when there is a batch unit that needs to switch from the remote task batch processing to the local task batch processing, the quick switching of the batch unit from the remote task batch processing mode to the local task batch processing mode is realized, thus further reducing the labor cost.

[0076] It can be seen that the solution provided by this application achieves the purpose of quickly switching the batch unit between the remote task batch processing mode and the local task batch processing mode, thus realizing the technical effect of reducing the labor cost, and further solving the technical problem of high labor cost caused by relying on manual operation to switch between the local batch processing mode and the remote batch processing mode in the related art.

[0077] Embodiment 3

[0078] According to an embodiment of the present invention, an embodiment of a task processing device is provided, where Figure 6 is a schematic diagram of an optional task processing device according to an embodiment of the present invention, as Figure 6 shown, the device includes:

[0079] An acquisition module 601, configured to obtain configuration information in a register when it is detected that the loading of a target batch unit is completed, where the target batch unit is any one of a plurality of batch units;

[0080] A determination module 602, configured to determine a task processing type of the target batch unit based on the configuration information;

[0081] An update module 603, configured to update a local task concurrent processor in the target batch unit to a remote task concurrent processor if the task processing type is a first processing type;

[0082] A loading module 604, configured to load a to-be-processed batch unit if it is detected that there is a to-be-processed batch unit among a plurality of batch units, so as to update a remote task concurrent processor in the to-be-processed batch unit to a local task concurrent processor, where the to-be-processed batch unit is a batch unit whose task processing type is updated from the first processing type to the second processing type.

[0083] It should be noted that the above acquisition module 601, determination module 602, update module 603, and loading module 604 correspond to steps S501 to S504 in the above embodiments. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment 1.

[0084] Embodiment 4

[0085] On the other hand, according to an embodiment of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and the computer program is configured to execute the above task processing method when running.

[0086] Embodiment 5

[0087] On the other hand, according to an embodiment of the present invention, there is also provided an electronic device, where Figure 7 is a schematic diagram of an optional electronic device according to an embodiment of the present invention, as Figure 7 shown, the electronic device includes one or more processors; a memory, configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement for running the program, where the program is configured to execute the above task processing method when running.

[0088] The above serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0089] In the above embodiments of the present invention, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0090] In several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings, direct couplings, or communication connections shown or discussed among each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0091] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. And the aforementioned storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0094] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A task processing system, characterized in that, Including a target application service, wherein the target application service includes: Multiple batch units, each batch unit including a local task concurrent processor, wherein the local task concurrent processor is used to distribute local tasks; A register for storing the task processing type of each batch unit, and when it is determined that the task processing type corresponding to the target batch unit is the first processing type, updating the local task concurrent processor in the target batch unit to a remote task concurrent processor, wherein the target batch unit is any one of the multiple batch units, the task processing type is the first processing type or the second processing type, the first processing type indicates that the current batch unit is used to distribute remote tasks to non-current batch units, the second processing type indicates that the current batch unit is used to process local tasks, and the remote task concurrent processor is used to distribute the remote tasks; A loading module for detecting whether there is a to-be-processed batch unit among the multiple batch units, and when there is a to-be-processed batch unit, loading the to-be-processed batch unit to update the remote task concurrent processor in the to-be-processed batch unit to the local task concurrent processor, wherein the to-be-processed batch unit is a batch unit whose task processing type is updated from the first processing type to the second processing type.

2. The task processing system according to claim 1, wherein Each of the batch units further includes: A task partitioning module for generating multiple local tasks or multiple remote tasks, and sending the multiple local tasks to the local task concurrent processor, or sending the multiple remote tasks to the remote task concurrent processor; Wherein the local task concurrent processor is used to obtain the multiple local tasks sent by the task partitioning module and schedule the sub-task processors in the current batch unit to process the multiple local tasks; The remote task concurrent processor is used to obtain the multiple remote tasks sent by the task partitioning module and distribute the task information corresponding to the multiple remote tasks to at least one non-current batch unit, so that the non-current batch unit obtains the multiple remote tasks according to the task information and processes the multiple remote tasks.

3. The task processing system according to claim 2, wherein: The remote task concurrent processor is used to obtain multiple remote tasks, allocate asynchronous threads to each remote task, and store the multiple remote tasks in a target storage area, wherein each asynchronous thread is used to send the task information of the remote task corresponding to the asynchronous thread to a target message queue, and the target message queue is used to send multiple task information to at least one non-current batch unit, so that the non-current batch unit obtains the multiple remote tasks according to the task information and processes the multiple remote tasks.

4. The task processing system according to claim 3, wherein: The remote task concurrent processor is further configured to, after the target message queue sends multiple task information to at least one non-current batch unit, query whether each task information has been acquired by the non-current batch unit, and when it is determined that there is target task information among the multiple task information, determine the sending time and the number of sending times of the target task information, and determine the processing method for the target task information based on the sending time and the number of sending times of the target task information, where the target task information is information that has not been acquired by the non-current batch unit.

5. The task processing system according to claim 4, wherein: The remote task concurrent processor is further configured to, when the time interval between the sending time of the target task information and the current time is greater than or equal to a preset time interval and the number of sending times is less than a preset number of times, re-send the target task information to the target message queue; The remote task concurrent processor is further configured to, when the time interval between the sending time of the target task information and the current time is greater than or equal to a preset time interval and the number of sending times is greater than or equal to a preset number of times, prohibit re-sending the target task information to the target message queue.

6. The task processing system according to claim 4, wherein The target application service further includes: A remote task receiver, configured to acquire multiple task information sent by the target message queue, and after acquiring the multiple task information, acquire a remote task corresponding to each target task information from the target storage area based on the multiple task information, and schedule a sub-task processor in at least one non-current batch unit to process the remote task.

7. A task processing method, characterized in that, Applied to the task processing system according to any one of claims 1 to 6, the method includes: When it is detected that a target batch unit is loaded, acquire configuration information in a register, where the target batch unit is any one of the multiple batch units; Determine the task processing type of the target batch unit based on the configuration information; If the task processing type is a first processing type, update the local task concurrent processor in the target batch unit to the remote task concurrent processor; If it is detected that there is a to-be-processed batch unit among the multiple batch units, load the to-be-processed batch unit to update the remote task concurrent processor in the to-be-processed batch unit to the local task concurrent processor, where the to-be-processed batch unit is a batch unit whose task processing type is updated from the first processing type to the second processing type.

8. A task processing device, characterized in that, Applied to the task processing system according to any one of claims 1 to 6, it includes: An acquisition module, configured to acquire configuration information in a register when it is detected that a target batch unit is loaded, where the target batch unit is any one of the multiple batch units; A determination module, configured to determine the task processing type of the target batch unit based on the configuration information; An update module, configured to, if the task processing type is a first processing type, update the local task concurrent processor in the target batch unit to the remote task concurrent processor; A loading module, configured to load the to-be-processed batch unit if a to-be-processed batch unit is detected in the multiple batch units, so as to update the remote task concurrent processor in the to-be-processed batch unit to the local task concurrent processor, where the to-be-processed batch unit is a batch unit whose task processing type is updated from a first processing type to a second processing type.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program is configured to execute the task processing method described in claim 7 when running.

10. An electronic device, characterized in that, The electronic device includes one or more processors; A memory, configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement running a program, where the program is configured to execute the task processing method described in claim 7 when running.

Images