Job execution system and method, electronic equipment and storage medium

Abstract

The invention provides a job execution system and method, electronic equipment and a storage medium, and the system comprises a job access subsystem, a job execution subsystem, a message queue and a completion queue. The operation access subsystem is used for acquiring a target request; the job access subsystem is also used for putting the target job identifier of the target job into the message queue; the job execution subsystem is used for consuming the target job identifier in the message queue; the job execution subsystem is also used for placing the target job identifier into a completion queue after executing the target operation on the target job corresponding to the target job identifier; and the job access subsystem is also used for determining that the target job is completed based on the target job identifier in the completion queue. Through application of the method and the device, the problem that the connections are held for a long time due to the fact that the execution time of the operation steps is relatively long in the prior art is solved; the long-term holding of the connection causes the occupation of system resources, and the concurrency cannot be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of distributed technology, and in particular to a job execution system and method, an electronic device and a storage medium. Background Art

[0002] At present, the core batch processing distributed system refers to a distributed system that uses computers to process data in batches every day. Due to the complexity of the business, the business will be scattered in many business systems. For example, there will be more than a dozen business systems on the enterprise side. Therefore, the execution of a job will involve multiple systems, and many interactions between systems are carried out through rpc or http. Network communication is also time-consuming, and the amount of data involved in the job is also large. Therefore, it takes a long time to execute a process. Long execution time means long-term occupation of system resources. And there are the following problems:

[0003] 1. Since the job execution time is relatively long, these connections are held for a long time;

[0004] 2. Long-term holding of connections causes system resources to be occupied and concurrency cannot be effectively improved.

[0005] Therefore, the above-mentioned problems exist in the related art. Summary of the invention

[0006] The present application provides a job execution system and method, an electronic device and a storage medium to solve at least one technical problem existing in the related art.

[0007] According to one aspect of an embodiment of the present application, a job execution system is provided, including: a job access subsystem, a job execution subsystem, a message queue, and a completion queue;

[0008] The job access subsystem is used to obtain a target request, wherein the target request is used to request execution of a target job;

[0009] The job access subsystem is further used to put the target job identifier of the target job into the message queue;

[0010] The job execution subsystem is used to consume the target job identifier in the message queue;

[0011] The job execution subsystem is further configured to place the target job identifier into the completion queue after executing a target operation on the target job corresponding to the target job identifier, wherein the target operation is at least one operation that the job execution subsystem needs to execute on the target job;

[0012] The job access subsystem is further configured to determine, based on the target job identifier in the completion queue, that the target job has been completed.

[0013] According to another aspect of the present application, a job execution method is also provided, which is applied to a job access subsystem, including:

[0014] Obtaining a target request, wherein the target request is used to request execution of a target job;

[0015] Putting the target job identifier of the target job into a message queue;

[0016] After the job execution subsystem consumes the target job identifier and performs a target operation on the target job and places the target job identifier into a completion queue, it is determined that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation that the job execution subsystem needs to perform on the target job.

[0017] Optionally, as in the aforementioned method, after placing the target job identifier of the target job into the message queue, the method further includes:

[0018] Generate an asynchronous future with the target job ID;

[0019] Put the asynchronous future in local memory;

[0020] The determining, based on the target job identifier in the completion queue, that the target job has been completed includes:

[0021] Compare the target job identifier in the asynchronous future with all candidate job identifiers in the completion queue to obtain a comparison result;

[0022] If the comparison result determines that the target job identifier exists among all the candidate job identifiers, it is determined that the target job has been completed.

[0023] Optionally, as in the aforementioned method, comparing the target job identifier in the asynchronous future with all candidate job identifiers in the completion queue to obtain a comparison result includes:

[0024] Establish a timed task to obtain the target;

[0025] By running the target scheduled task, the target job identifier in the asynchronous future is compared with all candidate job identifiers of the completion queue according to the target period to obtain the comparison result.

[0026] Optionally, as in the aforementioned method, the method further comprises:

[0027] Target status information corresponding to the target job is established in the database so that after the job execution subsystem has performed the target operation on the target job, the target status information is updated to status information indicating that the target operation has been performed on the target job, wherein the status information is used to indicate the operation that has been performed on the target job.

[0028] According to another aspect of the present application, a job execution method is also provided, which is applied to a job execution subsystem, and includes:

[0029] The target job identifier in the consumption message queue is placed in the message queue after the job access subsystem obtains the target request, and the target request is used to request the execution of the target job;

[0030] After executing the target operation on the target job, the target job identifier is placed in the completion queue so that the job access subsystem determines that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation required to be performed on the target job.

[0031] According to another aspect of the present application, there is also provided a work access device, comprising:

[0032] An acquisition module, used for acquiring a target request, wherein the target request is used for requesting execution of a target job;

[0033] An insert module, used for inserting the target job identifier of the target job into a message queue;

[0034] A determination module is used to determine that the target job has been completed based on the target job identifier in the completion queue after the job execution subsystem consumes the target job identifier and performs a target operation on the target job and puts the target job identifier into a completion queue, wherein the target operation is the operation that the job execution subsystem needs to perform on the target job.

[0035] According to another aspect of the present application, there is also provided a job execution device, including:

[0036] A consumption module, used for consuming a target job identifier in a message queue, wherein the target job identifier in the message queue is put into the message queue after the job access subsystem obtains a target request, and the target request is used to request execution of the target job;

[0037] The insertion module is used to place the target job identifier into a completion queue after executing a target operation on the target job corresponding to the target job identifier, so that the job access subsystem determines that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation that needs to be performed on the target job.

[0038] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps in any of the above embodiments by running the computer program stored in the memory.

[0039] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the method steps in any of the above embodiments when executed.

[0040] In an embodiment of the present application, by introducing a message queue and a completion queue, the job access subsystem can inform the job execution subsystem of the job that needs to be executed through the messages in the message queue, and the job execution subsystem can inform the job access subsystem of the completion status of each job through the messages in the completion queue. Therefore, there is no need to hold the connection corresponding to each executing job for a long time between the job access subsystem and the job execution subsystem, thereby effectively improving the concurrent execution of each job and improving resource utilization efficiency, and solving the technical problems existing in the related technology that these connections are held for a long time due to the long execution time of the job step; the long-term holding of the connection causes system resources to be occupied, and the concurrency cannot be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 is a flowchart of an optional job execution method applied to a job access subsystem according to an embodiment of the present application;

[0044] Figure 2 is a flowchart of another optional job execution method applied to a job access subsystem according to an embodiment of the present application;

[0045] Figure 3 is a flowchart of an optional job execution method applied to a job execution subsystem according to an embodiment of the present application;

[0046] Figure 4 It is a flowchart of an optional method for executing a job according to an application example of the present application;

[0047] Figure 5 is a structural block diagram of an optional operation access device according to an embodiment of the present application;

[0048] Figure 6 is a structural block diagram of an optional job execution device according to an embodiment of the present application;

[0049] Figure 7 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] According to one aspect of an embodiment of the present application, a job execution system is provided. Optionally, in this embodiment, the above-mentioned job execution system can be applied to a hardware environment consisting of a terminal and a server. The server is connected to the terminal through a network, and can be used to provide services (such as advertising push services, application services, etc.) to the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0053] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0054] The job execution system of the embodiment of the present application can be constructed by a server, can be constructed by a terminal, or can be constructed by a server and a terminal together.

[0055] Taking the job execution system in this embodiment constructed by the server as an example, Figure 1 The job execution system provided for the embodiment of the present application includes: a job access subsystem, a job execution subsystem, a message queue and a completion queue;

[0056] The job access subsystem is used to obtain a target request, wherein the target request is used to request execution of a target job.

[0057] The job execution system in this embodiment can be applied to scenarios where the business complexity is high and the business is dispersed across many business processing flows, so the execution of a job involves multiple business systems, such as: job processing scenarios of enterprise-side systems (for example, including more than a dozen business systems), job processing scenarios of e-commerce systems, etc., and can also be scenarios for identifying job processing of other systems. In the embodiments of this application, the application of the job execution system to the job processing of enterprise-side systems is used as an example for explanation. For other types of systems, the above-mentioned job execution system is also applicable if there is no contradiction.

[0058] The target request may be received by a node in the job access subsystem.

[0059] The target job is a job that needs to be processed in sequence by multiple business systems, and the target request is to execute the target job with a request.

[0060] The requester can actively send the target request to the job access subsystem, and then any node or idle node in the job access subsystem obtains the target request.

[0061] The job access subsystem is also used to put the target job identifier of the target job into the message queue.

[0062] After obtaining the target request, the job access subsystem may determine the target job carried in the target request, and parse and obtain the target job identifier of the target job.

[0063] Generally, each job has a unique corresponding job ID.

[0064] After obtaining the target job identifier, the job access subsystem puts the target job identifier into the message queue as a message, so that other execution nodes that process the target job (i.e., nodes located in the job execution subsystem) can determine that the target job needs to be executed by obtaining the target job identifier from the message queue.

[0065] The job execution subsystem is used to consume the target job identifier in the message queue.

[0066] The job execution subsystem can access the message queue and consume the messages in the message queue to obtain the target job identifier. Generally, each message can only be consumed once to prevent the nodes in the job execution subsystem from repeatedly consuming messages with a job identifier and repeatedly performing the same operation on the same job, thereby wasting resources.

[0067] The job execution subsystem is also used to place the target job identifier into a completion queue after executing a target operation on the target job corresponding to the target job identifier, wherein the target operation is at least one operation that the job execution subsystem needs to execute on the target job.

[0068] After obtaining the target job identifier, the job execution subsystem may obtain the corresponding target job to be executed, and then execute the target operation required to be executed for the target job.

[0069] The target operation may contain only one sub-operation or multiple sub-operations, and each sub-operation is executed through a specific job execution node. When there are multiple sub-operations, the sub-operations are executed on the target job in sequence through the corresponding multiple job execution nodes, that is, the job executed by the latter job execution node depends on the result of the execution of the previous job execution node.

[0070] For example, when the target operation includes executing sub-operations A (executed by job execution node a), B (executed by job execution node a), and C (executed by job execution node a) in sequence; then job execution node a obtains result I after executing sub-operation A on the target job, and then job execution node b obtains result II after executing sub-operation B on the target job based on result I; job execution node c obtains result III after executing sub-operation C on the target job based on result II, or based on result I and result II. After obtaining result III, it is determined that the target operation is completed.

[0071] After completing the target operation on the target job, the job execution subsystem may put the target job identifier into the completion queue as a message.

[0072] The job access subsystem is also used to determine that the target job has been completed based on the target job identifier in the completion queue.

[0073] The job access subsystem can access the completion queue and obtain each message therein. When it is found that one of the messages is the target job identifier, or includes the target job identifier, the job access subsystem can determine that the target job uniquely indicated by the target job identifier has been completed.

[0074] Through the system in this embodiment, by introducing a message queue and a completion queue, the job access subsystem can inform the job execution subsystem of the job that needs to be executed through the messages in the message queue, and the job execution subsystem can inform the job access subsystem of the completion status of each job through the messages in the completion queue. Therefore, there is no need for the job access subsystem and the job execution subsystem to hold the connection corresponding to each executing job for a long time, which can effectively improve the concurrent execution of each job and improve resource utilization efficiency, and solve the technical problems existing in the related technology that these connections are held for a long time due to the long execution time of the job step; the long-term holding of the connection causes system resources to be occupied, and the concurrency cannot be effectively improved.

[0075] like Figure 1 As shown, according to another aspect of the present application, a job execution method is also provided, which is applied to the job access subsystem and includes the following steps:

[0076] Step S101: obtaining a target request, wherein the target request is used to request execution of a target job.

[0077] The target request may be received by a node in the job access subsystem.

[0078] The target job is a job that needs to be processed in sequence by multiple business systems, and the target request is to execute the target job with a request.

[0079] The requester can actively send the target request to the job access subsystem, and then any node or idle node in the job access subsystem obtains the target request.

[0080] Step S102: putting the target job identifier of the target job into the message queue.

[0081] After obtaining the target request, the job access subsystem may determine the target job carried in the target request, and parse and obtain the target job identifier of the target job.

[0082] Generally, each job has a unique corresponding job ID.

[0083] After obtaining the target job identifier, the job access subsystem puts the target job identifier into the message queue as a message, so that other execution nodes that process the target job (i.e., nodes located in the job execution subsystem) can determine that the target job needs to be executed by obtaining the target job identifier from the message queue.

[0084] Step S103, after the job execution subsystem consumes the target job identifier and performs the target operation on the target job and puts the target job identifier into the completion queue, it is determined that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation that the job execution subsystem needs to perform on the target job.

[0085] The job execution subsystem can access the message queue and consume the messages in the message queue to obtain the target job identifier. Generally, each message can only be consumed once to prevent the nodes in the job execution subsystem from repeatedly consuming messages with a job identifier and repeatedly performing the same operation on the same job, thereby wasting resources.

[0086] After obtaining the target job identifier, the job execution subsystem may obtain the corresponding target job to be executed, and then execute the target operation required to be executed for the target job.

[0087] The target operation may contain only one sub-operation or multiple sub-operations, and each sub-operation is executed through a specific job execution node. When there are multiple sub-operations, the sub-operations are executed on the target job in sequence through the corresponding multiple job execution nodes, that is, the job executed by the latter job execution node depends on the result of the execution of the previous job execution node.

[0088] For example, when the target operation includes executing sub-operations A (executed by job execution node a), B (executed by job execution node a), and C (executed by job execution node a) in sequence; then job execution node a obtains result I after executing sub-operation A on the target job, and then job execution node b obtains result II after executing sub-operation B on the target job based on result I; job execution node c obtains result III after executing sub-operation C on the target job based on result II, or based on result I and result II. After obtaining result III, it is determined that the target operation is completed.

[0089] After completing the target operation on the target job, the job execution subsystem may put the target job identifier into the completion queue as a message.

[0090] The job access subsystem is also used to determine that the target job has been completed based on the target job identifier in the completion queue.

[0091] The job access subsystem can access the completion queue and obtain each message therein. When it is found that one of the messages is the target job identifier, or includes the target job identifier, the job access subsystem can determine that the target job uniquely indicated by the target job identifier has been completed.

[0092] Through the system in this embodiment, by introducing a message queue and a completion queue, the job access subsystem can inform the job execution subsystem of the job that needs to be executed through the messages in the message queue, and the job execution subsystem can inform the job access subsystem of the completion status of each job through the messages in the completion queue. Therefore, there is no need for the job access subsystem and the job execution subsystem to hold the connection corresponding to each executing job for a long time, which can effectively improve the concurrent execution of each job and improve resource utilization efficiency, and solve the technical problems existing in the related technology that these connections are held for a long time due to the long execution time of the job step; the long-term holding of the connection causes system resources to be occupied, and the concurrency cannot be effectively improved.

[0093] As an optional embodiment, as in the above method, after the target job identifier of the target job is placed in the message queue in step S102, the method further includes the following steps:

[0094] Step A101, generating an asynchronous future with a target job identifier;

[0095] Step A102, put the asynchronous future in local memory.

[0096] That is, the job access subsystem records a task handle of a target request locally; for example, through future-related operations of a Java program, an asynchronous future with a job identifier is added to the local memory.

[0097] The main function of future is that when performing certain operations, the operation process may be time-consuming, and sometimes the database may be checked, or heavy calculations such as compression and encryption may be performed. In this case, if you keep waiting for the method to return, the overall program efficiency will be greatly reduced. The operation process can be put into the child thread to execute, and then Future can be used to control the calculation process executed by the child thread, and finally the calculation result can be obtained. In this way, the operation efficiency of the entire program can be improved.

[0098] By establishing an asynchronous future, the current process (i.e., the process processing the target job) can be blocked, indicating that the process is not completed.

[0099] like Figure 2 As shown, the step S103, based on the target job identifier in the completion queue, determines that the target job has been completed, includes the following steps:

[0100] Step S201, compare the target job identifier in the asynchronous future with all candidate job identifiers in the completion queue to obtain a comparison result.

[0101] After the asynchronous future is established and saved locally, the target job identifier in the asynchronous future can be obtained in real time, and then the target job identifier in the asynchronous future is compared with all candidate job identifiers in the completion queue to obtain a comparison result.

[0102] Optionally, the target job identifier in the asynchronous future may be compared with each candidate job identifier in the completion queue in turn. After comparing with each candidate job identifier, a comparison result indicating whether the target job identifier exists in all candidate job identifiers may be obtained.

[0103] Step S202: when the comparison result determines that the target job identifier exists among all candidate job identifiers, it is determined that the target job has been completed.

[0104] When the comparison result determines that the target job identifier exists among all candidate job identifiers, it means that the job execution subsystem has completed processing the target job. Therefore, it can be determined that the target job has been completed.

[0105] Furthermore, when the comparison result determines that the target job identifier does not exist in all candidate job identifiers, the process continues to wait for the target job to be completed until it is determined that the target job identifier exists in the completion queue.

[0106] As an optional embodiment, as in the aforementioned method, step S201 compares the target job identifier in the asynchronous future with all candidate job identifiers in the completion queue to obtain a comparison result, including the following steps:

[0107] Step S301, establishing a target timed task;

[0108] Step S302 , by running the target scheduled task, the target job identifier in the asynchronous future is compared with all candidate job identifiers in the completion queue according to the target period to obtain a comparison result.

[0109] In this embodiment, a target timer task may be established, and the target timer task is used to regularly and continuously check whether the target job identifier of the target job associated with the asynchronous future in the memory is in the completion queue.

[0110] The target scheduled task can be executed according to a preset cycle, for example: once every 1 second, once every 0.1 second. The specific length of the cycle can be selected according to the actual business.

[0111] The target timing task can compare the target job identifier in the asynchronous future with all candidate job identifiers in the completion queue according to the preset target period to obtain a comparison result.

[0112] As an optional embodiment, as the aforementioned method, the method further includes the following steps:

[0113] Step S401, establish target status information corresponding to the target job in the database, so that after the job execution subsystem has executed the target operation on the target job, the target status information is updated to status information indicating that the target operation has been executed on the target job, wherein the status information is used to indicate the operation that has been executed on the target job.

[0114] That is, the job access subsystem may add a target status information corresponding to the target job in the database, and the target status information is used to write the flow record of the target job, that is, to characterize the executed operation of the target job.

[0115] For example, after the job execution subsystem has performed the target operation on the target job, the target status information is updated to status information indicating that the target operation has been performed on the target job. For example, when the job access subsystem obtains the target job, its target status information can be recorded as 10. After the job execution subsystem completes the target operation on the target job, its target status information can be recorded as 20.

[0116] Through the method in this embodiment, each node in each job execution subsystem can determine the execution progress of the target job in real time.

[0117] like Figure 3 As shown, according to another aspect of the present application, a job execution method is also provided, which is applied to the job execution subsystem and includes the following steps:

[0118] Step P101, consuming the target job identifier in the message queue, wherein the target job identifier in the message queue is placed in the message queue after the job access subsystem obtains the target request, and the target request is used to request execution of the target job;

[0119] Step P102, after executing the target operation on the target job, put the target job identifier into the completion queue so that the job access subsystem can determine that the target job has been completed based on the target job identifier in the completion queue, where the target operation is the operation required to be performed on the target job.

[0120] The target request may be received by a node in the job access subsystem.

[0121] The target job is a job that needs to be processed in sequence by multiple business systems, and the target request is to execute the target job with a request.

[0122] The requester can actively send the target request to the job access subsystem, and then any node or idle node in the job access subsystem obtains the target request.

[0123] After obtaining the target request, the job access subsystem may determine the target job carried in the target request, and parse and obtain the target job identifier of the target job.

[0124] Generally, each job has a unique corresponding job ID.

[0125] After obtaining the target job identifier, the job access subsystem puts the target job identifier into the message queue as a message, so that other execution nodes that process the target job (i.e., nodes located in the job execution subsystem) can determine that the target job needs to be executed by obtaining the target job identifier from the message queue.

[0126] The job execution subsystem can access the message queue and consume the messages in the message queue to obtain the target job identifier. Generally, each message can only be consumed once to prevent the nodes in the job execution subsystem from repeatedly consuming messages with a job identifier and repeatedly performing the same operation on the same job, thereby wasting resources.

[0127] After obtaining the target job identifier, the job execution subsystem may obtain the corresponding target job to be executed, and then execute the target operation required to be executed for the target job.

[0128] The target operation may contain only one sub-operation or multiple sub-operations, and each sub-operation is executed through a specific job execution node. When there are multiple sub-operations, the sub-operations are executed on the target job in sequence through the corresponding multiple job execution nodes, that is, the job executed by the latter job execution node depends on the result of the execution of the previous job execution node.

[0129] For example, when the target operation includes executing sub-operations A (executed by job execution node a), B (executed by job execution node a), and C (executed by job execution node a) in sequence; then job execution node a obtains result I after executing sub-operation A on the target job, and then job execution node b obtains result II after executing sub-operation B on the target job based on result I; job execution node c obtains result III after executing sub-operation C on the target job based on result II, or based on result I and result II. After obtaining result III, it is determined that the target operation is completed.

[0130] Furthermore, when there are multiple job execution nodes that perform multiple target operations on the target job in sequence, the job execution nodes that need to record status information can be determined in advance. For example, based on the above example, when only one completion progress needs to be recorded, the completion progress can be recorded only after the target job has been executed by the last job execution node. That is, the target job identifier is written into the completion queue. In this case, the job execution nodes a, b, and c are not decoupled. When the completion progress of each job execution node needs to be recorded, the target job identifier is written into the progress message queue corresponding to each job execution node, and then the job execution nodes a, b, and c can be decoupled from each other. When decoupling is not required between job execution nodes a and b, but decoupling is required between job execution nodes b and c, the progress message queue will not be written after job execution node a completes the corresponding sub-operation on the target job. The target job identifier will be written into the progress message queue corresponding to job execution node b only after job execution node b completes the corresponding sub-operation on the target job, to indicate that the two sub-operations corresponding to job execution nodes a and b have been completed for the target job, and the target job identifier will be written into the completion queue after job execution node c completes the corresponding sub-operation on the target job.

[0131] After completing the target operation on the target job, the job execution subsystem may put the target job identifier into the completion queue as a message.

[0132] The job access subsystem is also used to determine that the target job has been completed based on the target job identifier in the completion queue.

[0133] The job access subsystem can access the completion queue and obtain each message therein. When it is found that one of the messages is the target job identifier, or includes the target job identifier, the job access subsystem can determine that the target job uniquely indicated by the target job identifier has been completed.

[0134] Through the system in this embodiment, by introducing a message queue and a completion queue, the job access subsystem can inform the job execution subsystem of the job that needs to be executed through the messages in the message queue, and the job execution subsystem can inform the job access subsystem of the completion status of each job through the messages in the completion queue. Therefore, there is no need for the job access subsystem and the job execution subsystem to hold the connection corresponding to each executing job for a long time, which can effectively improve the concurrent execution of each job and improve resource utilization efficiency, and solve the technical problems existing in the related technology that these connections are held for a long time due to the long execution time of the job step; the long-term holding of the connection causes system resources to be occupied, and the concurrency cannot be effectively improved.

[0135] As an optional embodiment, the following steps may also be included:

[0136] Step P201, generating an alarm message when it is determined according to the target state information that the duration of the designated operation of the target job exceeds a preset duration;

[0137] Step P202: restart the designated execution node according to the alarm information, or enable a new execution node to perform the designated operation on the target job.

[0138] That is to say, when the task execution subsystem determines according to the target status information that the time for executing the specified operation on the target job exceeds the preset time, it generates an alarm message; restarts the specified execution node according to the alarm message, or enables a new execution node in the task execution subsystem to execute the specified operation on the target job.

[0139] like Figure 4 As shown, an application example of applying any of the above embodiments is provided:

[0140] The execution process of the job: The job is first sent to the job access cluster (i.e., the job access subsystem) through an http request. The job access cluster will add a flow record in the db (i.e., database) to record the status information of the job, and put the job-related information (i.e., the job identifier) ​​into the message queue (for example: redis, MQ, kafka, etc.). Then, through the asynchronous future-related operations of the java program, an asynchronous future with a job identifier is added (i.e., a new task handle is added) and put into the local memory. The other execution nodes of the job obtain relevant information from the message queue to perform various logical operations (i.e., target operations). After the operation is completed (which can be successful or failed), it is put into the message completion queue. The job access cluster uses a scheduled task to continuously check whether the job identifier of the job associated with the asynchronous future in the memory is in the completion queue. If the job identifier exists in the completion queue, it indicates that the job is completed, otherwise continue to wait.

[0141] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0143] According to another aspect of an embodiment of the present application, a job access device for implementing the above-mentioned job execution method is also provided, and the job access device is located in the job access subsystem. Figure 5 is a structural block diagram of an optional operation access device according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0144] An acquisition module 11 is used to acquire a target request, wherein the target request is used to request execution of a target job;

[0145] The putting module 12 is used to put the target job identifier of the target job into the message queue;

[0146] The determination module 13 is used to consume the target job identifier in the job execution subsystem, perform the target operation on the target job, put the target job identifier into the completion queue, and then determine that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation that the job execution subsystem needs to perform on the target job.

[0147] It should be noted that the acquisition module 11 in this embodiment can be used to execute the above step S101, the placement module 12 in this embodiment can be used to execute the above step S102, and the determination module 13 in this embodiment can be used to execute the above step S103.

[0148] It should be noted that the above modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can be run in the implementation of Figure 1 The hardware environment of the method shown may be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0149] According to another aspect of an embodiment of the present application, a job execution device for implementing the above-mentioned job execution method is also provided, and the job execution device is located in the job execution subsystem. Figure 6 is a structural block diagram of an optional job execution device according to an embodiment of the present application, such as Figure 6 As shown, the device may include:

[0150] The consumption module 21 is used to consume the target job identifier in the message queue, wherein the target job identifier in the message queue is put into the message queue after the job access subsystem obtains the target request, and the target request is used to request the execution of the target job;

[0151] The module 22 is used to place the target job identifier into the completion queue after executing the target operation on the target job corresponding to the target job identifier, so that the job access subsystem can determine that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation required to be performed on the target job.

[0152] It should be noted that the consumption module 21 in this embodiment can be used to execute the above step P101, and the placement module 22 in this embodiment can be used to execute the above step P102.

[0153] It should be noted that the above modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can be run in the implementation of Figure 3 The hardware environment of the method shown may be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0154] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned job execution method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0155] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 7 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0156] Memory 1503, used for storing computer programs;

[0157] The processor 1501 is used to implement the following steps when executing the program stored in the memory 1503:

[0158] Step S101: obtaining a target request, wherein the target request is used to request execution of a target job.

[0159] Step S102: putting the target job identifier of the target job into the message queue.

[0160] Step S103, after the job execution subsystem consumes the target job identifier and performs the target operation on the target job and puts the target job identifier into the completion queue, it is determined that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation that the job execution subsystem needs to perform on the target job.

[0161] Step P101, consume the target job identifier in the message queue, wherein the target job identifier in the message queue is placed in the message queue after the job access subsystem obtains the target request, and the target request is used to request execution of the target job.

[0162] Step P102, after executing the target operation on the target job, put the target job identifier into the completion queue so that the job access subsystem can determine that the target job has been completed based on the target job identifier in the completion queue, where the target operation is the operation required to be performed on the target job.

[0163] Optionally, in this embodiment, the above-mentioned communication bus can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0164] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0165] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0166] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0167] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0168] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0169] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0170] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0171] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0172] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0174] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A job execution system, characterized in that: include: Job access subsystem, job execution subsystem, message queue, and completion queue; The job access subsystem is used to obtain a target request, wherein the target request is used to request execution of a target job; The job access subsystem is further used to put the target job identifier of the target job into the message queue; The job execution subsystem is used to consume the target job identifier in the message queue; The job execution subsystem is further configured to place the target job identifier into the completion queue after executing a target operation on the target job corresponding to the target job identifier, wherein the target operation is at least one operation that the job execution subsystem needs to execute on the target job; The job access subsystem is further configured to determine, based on the target job identifier in the completion queue, that the target job has been completed.

2. A method for executing a job, characterized in that: Applicable to the job access subsystem, including: Obtaining a target request, wherein the target request is used to request execution of a target job; Putting the target job identifier of the target job into a message queue; After the job execution subsystem consumes the target job identifier and performs a target operation on the target job and places the target job identifier into a completion queue, it is determined that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation that the job execution subsystem needs to perform on the target job.

3. The method according to claim 2, characterized in that After placing the target job identifier of the target job into the message queue, the method further includes: Generate an asynchronous future with the target job ID; Put the asynchronous future in local memory; The determining, based on the target job identifier in the completion queue, that the target job has been completed includes: Compare the target job identifier in the asynchronous future with all candidate job identifiers in the completion queue to obtain a comparison result; If the comparison result determines that the target job identifier exists among all the candidate job identifiers, it is determined that the target job has been completed.

4. The method according to claim 3, characterized in that The step of comparing the target job identifier in the asynchronous future with all candidate job identifiers in the completion queue to obtain a comparison result includes: Establish a timed task to obtain the target; By running the target scheduled task, the target job identifier in the asynchronous future is compared with all candidate job identifiers of the completion queue according to the target period to obtain the comparison result.

5. The method according to any one of claims 2 to 4, characterized in that The method further comprises: Target status information corresponding to the target job is established in the database so that after the job execution subsystem has performed the target operation on the target job, the target status information is updated to status information indicating that the target operation has been performed on the target job, wherein the status information is used to indicate the operation that has been performed on the target job.

6. A method for executing a job, characterized in that: Applicable to the job execution subsystem, including: The target job identifier in the consumption message queue is placed in the message queue after the job access subsystem obtains the target request, and the target request is used to request the execution of the target job; After executing the target operation on the target job, the target job identifier is placed in the completion queue so that the job access subsystem determines that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation required to be performed on the target job.

7. A work access device, characterized in that: include: An acquisition module, used for acquiring a target request, wherein the target request is used for requesting execution of a target job; An insert module, used for inserting the target job identifier of the target job into a message queue; A determination module is used to determine that the target job has been completed based on the target job identifier in the completion queue after the job execution subsystem consumes the target job identifier and performs a target operation on the target job and puts the target job identifier into a completion queue, wherein the target operation is the operation that the job execution subsystem needs to perform on the target job.

8. A work execution device, characterized in that: include: A consumption module, used for consuming a target job identifier in a message queue, wherein the target job identifier in the message queue is put into the message queue after the job access subsystem obtains a target request, and the target request is used to request execution of the target job; The insertion module is used to place the target job identifier into a completion queue after executing a target operation on the target job corresponding to the target job identifier, so that the job access subsystem determines that the target job has been completed based on the target job identifier in the completion queue, wherein the target operation is the operation that needs to be performed on the target job.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is configured to execute the method steps of any one of claims 2 to 6 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method steps described in any one of claims 2 to 6 when run.

Images