Batch job execution method, Serverless platform, device, equipment and medium

Abstract

The invention discloses a batch job execution method, a Serverless platform, a device, equipment and a medium, and relates to the field of distributed technologies. The method comprises the following steps: receiving a batch job execution instruction sent by a scheduling node in a Serverless platform; calling a function component corresponding to the function component name to execute a batch job; when first-time starting success information or second-time starting success information sent by the function instance through the environment interaction component is received, real-time operation data for executing the batch jobs are obtained, and the takeover mode of the function instance is determined; and sending the real-time operation data and the takeover mode to the function instance, so that the function instance continues to execute the batch job based on the real-time operation data and the takeover mode. According to the technical scheme provided by the invention, the starting time for the function instance to reach the level of the runnable task is shortened, and the purpose of improving the task execution speed can be achieved.

Description

Technical Field

[0001] The present application relates to the field of distributed technology, and in particular to a batch job execution method, Serverless platform, device, equipment and medium. Background Art

[0002] Since its birth, Serverless function computing technology has been favored by the industry for its event-triggered, lightweight, efficient and elastic features, and is known as cloud computing 2.0. Lightweight, as one of the characteristics of the Serverless platform, is one of the important factors for the rapid startup of function instance containers (hereinafter referred to as function instances) in the platform. Due to the lightweight nature of functions, most of the function instances on the Serverless platform currently start up relatively quickly, with a startup time of a few seconds or more than ten seconds. However, in batch job execution scenarios, it is often necessary to start function instances with 4G or more memory. These function instances start up slowly, often taking two or three minutes, which affects the execution speed of the overall task. Summary of the invention

[0003] The present application provides a batch job execution method, Serverless platform, device, equipment and medium, which shortens the startup time of a function instance to reach a level where tasks can be run, thereby achieving the purpose of increasing the task execution speed.

[0004] In a first aspect, the present application provides a method for executing a batch job, which is applied to a sidecar instance in a Serverless platform. The method includes:

[0005] When receiving the secondary startup success information sent by the function instance through the environment interaction component, real-time operation data of executing the batch job is obtained, including:

[0006] When the secondary startup success information is received, the real-time operation data is acquired, and the takeover mode is determined to be direct takeover.

[0007] In a second aspect, the present application provides a method for executing a batch job, which is applied to a function instance in a Serverless platform, and the method includes:

[0008] Receive a scheduling instruction sent by a scheduling node in the Serverless platform, where the scheduling instruction is generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction includes a network address of a sidecar instance that executes the batch job;

[0009] Start necessary environment components and environment interaction components, and generate first start success information when the startup is successful;

[0010] Sending the first startup success information to the side car instance corresponding to the network address through the environment interaction component, so that the side car instance determines whether to obtain the real-time running data of executing the batch job when receiving the first startup success information;

[0011] When the real-time running data and the takeover method sent by the sidecar instance are received, the batch job continues to be executed based on the real-time running data and the takeover method.

[0012] In a third aspect, the present application provides a Serverless platform, including a running area of ​​a sidecar instance, a running area of ​​a function instance, a scheduling node, a sidecar instance, and a function instance, to execute the batch job execution method described in any embodiment.

[0013] In a fourth aspect, the present application provides a batch job execution device, which is integrated into a sidecar instance in a Serverless platform, and includes:

[0014] An execution instruction receiving module, used to receive a batch job execution instruction sent by a scheduling node in the Serverless platform, wherein the batch job execution instruction is generated by the Serverless platform based on an http trigger instruction sent from a distributed batch platform, and the batch job execution instruction includes a function component name for executing the batch job;

[0015] A first job execution module, used for calling the function component corresponding to the function component name to execute the batch job;

[0016] A data processing module, configured to obtain real-time operation data of executing the batch job and determine a takeover mode of the function instance when receiving first startup success information or second startup success information sent by the function instance through the environment interaction component, wherein the first startup success information is used to indicate that the function instance successfully starts the necessary environment component and the environment interaction component, and the second startup success information is used to indicate that the function instance successfully starts the secondary environment component;

[0017] The first data sending module is used to send the real-time running data and the takeover method to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover method.

[0018] In a fifth aspect, the present application provides a batch job execution device integrated into a function instance in a Serverless platform, the device comprising:

[0019] A scheduling instruction receiving module is used to receive a scheduling instruction sent by a scheduling node in the Serverless platform, wherein the scheduling instruction is generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction includes a network address of a sidecar instance that executes the batch job;

[0020] The component startup module is used to start the necessary environment components and environment interaction components, and generate the first startup success information when the startup is successful;

[0021] A second data sending module is used to send the first startup success information to the side car instance corresponding to the network address through the environment interaction component, so that the side car instance determines whether to obtain the real-time operation data of executing the batch job when receiving the first startup success information;

[0022] The second job execution module is used to continue executing the batch job based on the real-time operation data and the takeover method when receiving the real-time operation data and the takeover method sent by the side car instance.

[0023] In a sixth aspect, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the batch job execution method described in any embodiment of the present application.

[0024] In a seventh aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the batch job execution method described in any embodiment of the present application when executed.

[0025] In an eighth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the batch job execution method described in any embodiment of the present application.

[0026] In order to solve the defects of the prior art in the background technology, an embodiment of the present application provides a method for executing batch jobs, and executing this method can bring the following beneficial effects: the present application runs the full environment components such as JVM in the sidecar instance in advance, and configures the environment interaction components in the sidecar instance and the function instance respectively, so as to determine whether the function instance will take over the execution of the batch job through information interaction. The present application first executes the batch job by the sidecar instance, and at the same time, the full environment components of the function instance are started in two steps. For the first time, only the necessary environment components and the environment interaction components are started, and for the second time, the secondary environment components are started. Upon receiving the first startup success information or the second startup success information, the function instance takes over the execution of the batch job, realizing the ability of the function instance to quickly reach the level of executable tasks, shortening the startup time for the function instance to reach the level of executable tasks, and achieving the purpose of increasing the task execution speed.

[0027] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the batch job execution device, or may be packaged separately from the processor of the batch job execution device, which is not limited in this application.

[0028] The descriptions of the second, third, ..., and eighth aspects of the present application may refer to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, ..., and eighth aspects may refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

[0030] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A first flow chart of a method for executing a batch job provided in an embodiment of the present application;

[0033] Figure 2 A second flow chart of a method for executing a batch job provided in an embodiment of the present application;

[0034] Figure 3a-3c A data interaction diagram of a batch job execution method provided in an embodiment of the present application;

[0035] Figure 4 A system diagram of a Serverless platform provided in an embodiment of the present application;

[0036] Figure 5 A first structural schematic diagram of a batch job execution device provided in an embodiment of the present application;

[0037] Figure 6 A second structural schematic diagram of a batch job execution device provided in an embodiment of the present application;

[0038] Figure 7 The block diagram is a block diagram of an electronic device used to implement a batch job execution method of an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0040] It should be noted that the terms "first", "second", "target", and "original" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having", and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0041] Before executing the embodiments of the present application, it is necessary to first introduce the Serverless function computing platform (hereinafter referred to as the Serverless platform) architecture of the present application. The Serverless platform architecture of this embodiment is basically unchanged compared with the existing Serverless platform architecture, but the running area of ​​the sidecar instance will be increased. At the same time, by writing sidecar instance programs that can run various processes such as Java Virtual Machine (JVM) and network connections, and modifying the scheduling logic of the Serverless platform scheduling node, batch jobs can be pre-executed by sidecar instances, and the information interaction between function instances and sidecar instances can be coordinated to achieve the effect of reducing the first round of function instance startup time and the overall task running time.

[0042] The number and specification settings of the sidecar instances to be started are added to the configuration file of the Serverless platform for the scheduling node to read and use. After the Serverless platform is started, the scheduling node will synchronously read the configuration file as usual when it starts. After the scheduling node reads the content of the file, it starts multiple sidecar instances in the dedicated resource area of ​​the "sidecar instance running area" through the traditional function pull-up method, and keeps them running persistently. These sidecar instances run the full range of environment components (including hardware scheduling components, file management components, network management components, security protection components, etc.), the components required for the full range of function computing platform operations (including variable management components, instruction reception and processing components, container exception alarm components, etc.), JVM virtual machines, Java monitoring components and other necessary processes required for the operation of systems and Java codes, as well as newly added environment interaction components for interacting with function instances.

[0043] Figure 1 This is a first flow chart of a method for executing a batch job provided in an embodiment of the present application. This embodiment can be applied to the case where a sidecar instance and a function instance in a Serverless platform are used to process batch jobs. This embodiment provides a method for executing a batch job, which is applied to a sidecar instance in a Serverless platform and can be executed by a batch job execution device provided in an embodiment of the present application. The device can be implemented in software and / or hardware and integrated in an electronic device that executes this method.

[0044] See also Figure 1 The method of this embodiment includes but is not limited to the following steps:

[0045] S110. Receive batch job execution instructions sent by the scheduling node in the Serverless platform.

[0046] Among them, the Serverless platform is a platform that can run Java, Python, and Docker images, that is, a cloud computing service model. In this model, the cloud service provider provides users with an environment for running code, and users do not need to manage the configuration, maintenance, expansion, and other underlying infrastructure-related operations of the server. This embodiment uses the Serverless platform to execute batch jobs. The scheduling node is used to control batch jobs in the Serverless platform.

[0047] The batch job execution instruction is generated by the Serverless platform based on the http trigger instruction sent from the distributed batch platform. The batch job execution instruction includes the name of the function component used to execute the batch job.

[0048] In the embodiment of the present application, the distributed batch platform detects that the running time of a batch job (such as task A) has been reached, and then sends an http trigger instruction of task A to the Serverless platform, and the http trigger instruction contains important information such as the name of each job in the batch job and the job group identification number. The Serverless platform receives the http trigger instruction from the distributed batch platform, reads the http trigger instruction through the scheduling node, and finds the function component name corresponding to the batch job name and the job group identification number in the database according to the important information in the http trigger instruction, and generates a batch job execution instruction.

[0049] The scheduling node first selects a relatively idle sidecar instance from multiple sidecar instances and determines this sidecar instance as the instance container for executing batch jobs. The scheduling node sends the batch job execution instruction to this sidecar instance, and the sidecar instance receives the batch job execution instruction.

[0050] S120. Call the function component corresponding to the function component name to execute the batch job.

[0051] The sidecar instance pre-runs the full set of environment components, which include the environment interaction components that communicate with the function instance.

[0052] In the embodiment of the present application, after receiving the batch job execution instruction, the sidecar instance immediately starts the pre-execution of the job task. The sidecar instance calls the corresponding function component based on the function component name in the batch job execution instruction, and executes the batch job first through the function component in the sidecar instance. At the same time, the scheduling node synchronously pulls up a function instance, and the sidecar instance starts waiting for the first startup of this function instance to complete.

[0053] S130: When receiving the first start success information or the second start success information sent by the function instance through the environment interaction component, obtain the real-time operation data of the batch job and determine the takeover method of the function instance.

[0054] The startup of a function instance is divided into the first startup and the second startup, and the full amount of environment components in the function instance are divided into necessary environment components, secondary environment components, and environment interaction components. The first startup success information is used to indicate that the function instance successfully starts the necessary environment components and environment interaction components, and the second startup success information is used to indicate that the function instance successfully starts the secondary environment components.

[0055] Specifically, when receiving the first startup success information sent by the function instance through the environment interaction component, the real-time running data of the batch job is obtained and the takeover method of the function instance is determined, including: when the sidecar instance receives the first startup success information through the environment interaction component, the sidecar instance needs to first determine the resource load status during the batch job execution; determine whether the function instance takes over the batch job execution according to the resource load status and the necessary environment components; if taken over, obtain the real-time running data obtained by the current batch job execution, and determine the takeover method as direct takeover; if not taken over, continue to execute the batch job, and determine the takeover method as waiting for takeover. Among them, direct takeover means that the function instance takes over the execution of the batch job when the component is started for the first time; waiting for takeover means that it is necessary to wait until the function instance takes over the execution of the batch job when the component is started for the second time (that is, the startup of the full environment component is completed).

[0056] Further, judging whether the function instance takes over the execution of the batch job according to the resource load status and the necessary environment components includes: obtaining the number of waiting execution instructions, the sidecar instance's own load, and the resources required for executing the batch job from the resource load status; if the own load meets the preset operation standard and the number is not greater than the preset value, or the necessary environment components do not meet the required resources, indicating that the current sidecar instance is capable of continuing to execute the batch job, or the current function instance is not capable of executing the batch job, then it is determined that the function instance will not take over the execution of the batch job; if the own load does not meet the preset operation standard, the number is greater than the preset value (such as 2), or the necessary environment components meet the required resources, indicating that the current sidecar instance is not capable of continuing to execute the batch job, or the current function instance is capable of executing the batch job, then it is determined that the function instance will take over the execution of the batch job. Among them, the waiting execution instruction refers to other execution instructions subsequently assigned to this sidecar instance by the scheduling node. The preset operation standard refers to the operating conditions that can support the sidecar instance to continue to execute the batch job.

[0057] It is understandable that when the current sidecar instance is unable to continue executing the batch job, and the current function instance is able to execute the batch job, the function instance still takes over to execute the batch job, but the takeover mode is determined to be waiting for takeover.

[0058] Specifically, after the function instance is successfully started for the first time, the sidecar instance determines that the function instance should not take over the execution of the batch job according to the resource load status and the necessary environment components. At this time, the function instance starts the secondary environment component to achieve a secondary start. Then, when the sidecar instance receives the secondary start success information sent by the function instance through the environment interaction component, the real-time running data of the batch job is obtained, including: when the sidecar instance receives the secondary start success information through the environment interaction component, the sidecar instance obtains the real-time running data obtained by the current batch job execution, and determines the takeover mode as direct takeover.

[0059] S140: Send the real-time running data and the takeover method to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover method.

[0060] In an embodiment of the present application, the sidecar instance sends the real-time running data and the takeover method to the function instance through the environment interaction component, so that the function instance continues to execute the batch job based on the real-time running data and the takeover method.

[0061] The technical solution provided in this embodiment receives the batch job execution instruction sent by the scheduling node in the Serverless platform; calls the function component corresponding to the function component name to execute the batch job; when receiving the first startup success information or the second startup success information sent by the function instance through the environment interaction component, obtains the real-time operation data of the batch job execution and determines the takeover method of the function instance; sends the real-time operation data and the takeover method to the function instance, so that the function instance continues to execute the batch job based on the real-time operation data and the takeover method. This application runs the full environment components such as JVM in the sidecar instance in advance, and configures the environment interaction components in the sidecar instance and the function instance respectively, which are used for information interaction between the function instance and the sidecar instance, so as to determine whether the function instance takes over the execution of the batch job through information interaction. This application first executes the batch job by the sidecar instance, and at the same time, the full environment components of the function instance are started in two steps. For the first time, only the necessary environment components and the environment interaction components are started, and the secondary environment components are started for the second time, which can shorten the startup time of the function instance to reach the level of executable tasks. When receiving the first startup success message or the second startup success message, the function instance takes over to execute the batch job, so that the function instance can quickly reach the ability to run tasks, thereby achieving the purpose of improving the task execution speed.

[0062] Figure 2This is a second flow chart of a batch job execution method provided in an embodiment of the present application. A batch job execution method provided in this embodiment is applied to function instances in a Serverless platform and can be executed by a batch job execution device provided in an embodiment of the present application. The device can be implemented in software and / or hardware and integrated in an electronic device that executes this method.

[0063] See also Figure 2 The method of this embodiment includes but is not limited to the following steps:

[0064] S210. Receive a scheduling instruction sent by a scheduling node in the Serverless platform.

[0065] Among them, the scheduling instructions are generated by the Serverless platform based on the batch job execution instructions. The scheduling instructions include the network address of the sidecar instance that executes the batch job, which facilitates the matching of the sidecar instance and the function instance.

[0066] In the embodiment of the present application, after the serverless platform sends the batch job execution instruction to the sidecar instance, it obtains the network address of the sidecar instance and generates a scheduling instruction based on the network address and important information in the http trigger instruction. Then, the scheduling node synchronously pulls up a function instance and sends the scheduling instruction to the function instance, which receives the scheduling instruction.

[0067] S220: Start necessary environment components and environment interaction components, and generate first start success information when the start is successful.

[0068] In the embodiment of the present application, the scheduling node of the modified Serverless platform is different from the original method of directly starting a new function instance in the process of pulling up the function instance, that is, allocating the central processing unit (CPU) and memory, starting all system components (including hardware scheduling components, file management components, network management components, security protection components, etc.), starting the components required for the function computing platform to run (including variable management components, instruction receiving and processing components, container abnormality alarm components, etc.), starting the JVM virtual machine, starting batch programs, etc. Instead, after allocating CPU and memory, only the necessary environment components (including hardware scheduling components, file management components, network management components, etc.) and the newly added environment interaction components are started. When the necessary environment components and environment interaction components are successfully started, the function instance generates the first startup success information.

[0069] S230. Sending first startup success information to the side car instance corresponding to the network address through the environment interaction component, so that the side car instance determines whether to obtain real-time operation data for executing batch jobs when receiving the first startup success information.

[0070] In an embodiment of the present application, after the first startup of the component of the newly launched function instance is completed, the corresponding sidecar instance will be determined according to the network address in the scheduling instruction, and the first startup success information will be sent to the sidecar instance through the environment interaction component in the function instance to make it clear that the first startup has been completed.

[0071] When the sidecar instance receives the first startup success information through the environment interaction component, the sidecar instance needs to first determine the resource load status during the execution of the batch job; determine whether the function instance should take over the execution of the batch job based on the resource load status and the necessary environment components; if taken over, obtain the real-time running data obtained by the current batch job execution, and determine the takeover mode as direct takeover; if not taken over, continue to execute the batch job, and determine the takeover mode as waiting for takeover.

[0072] S240: When the real-time operation data and the takeover method sent by the sidecar instance are received, the batch job is continued to be executed based on the real-time operation data and the takeover method.

[0073] The takeover method is that the sidecar instance is determined by the required resources and necessary environment components for executing the batch job.

[0074] Specifically, the function instance continues to execute batch jobs based on real-time running data and the takeover method, including: if the takeover method is direct takeover, the function instance continues to execute batch jobs based on real-time running data; if the takeover method is waiting for takeover, the function instance also needs to start the secondary environment component; when the function instance successfully starts the secondary environment component, it continues to execute batch jobs based on real-time running data.

[0075] Furthermore, the batch job execution method also includes: when the sidecar instance determines that the current sidecar instance is capable of continuing to execute the batch job, or the current function instance is incapable of executing the batch job, based on the resource load status and the necessary environment components, it is determined that the function instance will not take over to execute the batch job, and the takeover mode is determined to be waiting for takeover. Then, at this time, the function instance will not receive real-time running data, and the function instance can only receive the information of waiting for takeover. At this time, the function instance starts the secondary environment component; when the secondary environment component is successfully started, a secondary startup success message is generated; the function instance sends the secondary startup success message to the sidecar instance, so that the sidecar instance obtains the real-time running data of executing the batch job when receiving the secondary startup success message; when the function instance receives the real-time running data sent by the sidecar instance, the function instance continues to execute the batch job based on the real-time running data.

[0076] The technical solution provided in this embodiment receives the scheduling instructions sent by the scheduling node in the Serverless platform; starts the necessary environment components and the environment interaction components, and generates the first startup success information when the startup is successful; sends the first startup success information to the sidecar instance corresponding to the network address through the environment interaction component, so that the sidecar instance determines whether to obtain the real-time operation data for executing the batch job when receiving the first startup success information; when receiving the real-time operation data and takeover method sent by the sidecar instance, the batch job is continued to be executed based on the real-time operation data and the takeover method. This application configures the environment interaction components in the sidecar instance and the function instance respectively, which are used for the information interaction between the function instance and the sidecar instance, so as to determine whether the function instance takes over the execution of the batch job through information interaction. This application first executes the batch job by the sidecar instance, and at the same time, the full environment components of the function instance are started in two steps. For the first time, only the necessary environment components and the environment interaction components are started, and the secondary environment components are started for the second time, which can shorten the startup time of the function instance to reach the level of the executable task. When the first startup success information or the second startup success information is received, the function instance takes over the execution of the batch job, realizing the ability of the function instance to quickly reach the executable task, and achieving the purpose of improving the task execution speed.

[0077] The following further describes the batch job execution method provided by the embodiment of the present application. Figure 3 is a data interaction diagram of a batch job execution method provided by the embodiment of the present application. The embodiment of the present application is optimized on the basis of the above embodiments, and the specific optimization is as follows: This embodiment explains in detail the data interaction process between the scheduling node, the sidecar instance and the function instance during the execution of the batch job.

[0078] See also Figure 3a The method of this embodiment includes but is not limited to the following steps:

[0079] S301, the scheduling node receives and reads the http trigger instruction.

[0080] The distributed batch platform detects that the running time of a batch job has arrived and sends the http trigger instruction of this batch job to the Serverless platform. The Serverless platform receives the http trigger instruction from the distributed batch platform and reads the http trigger instruction through the scheduling node.

[0081] S302: The scheduling node generates a batch job execution instruction.

[0082] The scheduling node searches the database for the function component name corresponding to the batch job name and the job group identification number according to the important information in the http trigger instruction, and generates a batch job execution instruction.

[0083] S303. The scheduling node sends the batch job execution instruction to the sidecar instance, generates a scheduling instruction, and sends the scheduling instruction to the function instance.

[0084] The scheduling node first selects a relatively idle sidecar instance from multiple sidecar instances and determines this sidecar instance as the instance container for executing batch jobs. The scheduling node then obtains the network address of this sidecar instance and generates a scheduling instruction based on the network address and important information in the http trigger instruction.

[0085] S304: The sidecar instance receives the batch job execution instruction and starts executing the batch job.

[0086] S305: The function instance receives the scheduling instruction and starts necessary environment components and environment interaction components.

[0087] S306. When the necessary environment components and the environment interaction components are successfully started, the function instance generates first start success information.

[0088] S307: The function instance sends the first startup success message to the sidecar instance.

[0089] The function instance sends the first startup success message to the sidecar instance corresponding to the network address through the environment interaction component.

[0090] S308. When receiving the first startup success information, the sidecar instance determines the resource load status during the execution of the batch job.

[0091] S309: The sidecar instance determines whether the function instance should take over and execute the batch job according to the resource load status and necessary environment components.

[0092] The sidecar instance analyzes the resource load status (the specific analysis process is explained in detail in the above embodiment and will not be repeated here). If it is determined that the current sidecar instance is capable of continuing to execute the batch job, or the current function instance is not capable of executing the batch job, then it is determined that the function instance will not take over to execute the batch job; at this time, the sidecar instance continues to execute the batch job, and the takeover mode is determined to be waiting for takeover. Execute the following steps S310-S316.

[0093] If it is determined that the current sidecar instance is incapable of continuing to execute the batch job, or the current function instance is capable of executing the batch job, then it is determined that the function instance will take over to execute the batch job; at this time, the sidecar instance obtains the real-time running data obtained by the current batch job execution, and determines the takeover mode as direct takeover. Execute the following steps S317-S322.

[0094] See also Figure 3b The method of this embodiment includes but is not limited to the following steps:

[0095] S310: The sidecar instance continues to execute the batch job and sends a takeover method including waiting for takeover to the function instance.

[0096] S311. The function instance receives the takeover method and starts the secondary environment components.

[0097] S312. When the secondary environment component is successfully started, the function instance generates secondary startup success information.

[0098] S313: The function instance sends a secondary startup success message to the sidecar instance.

[0099] S314. After receiving the secondary startup success information, the sidecar instance obtains the real-time running data of the batch job.

[0100] S315. The sidecar instance sends the current real-time running data to the function instance.

[0101] S316: The function instance receives the real-time running data and continues to execute the batch job based on the real-time running data.

[0102] See also Figure 3c The method of this embodiment includes but is not limited to the following steps:

[0103] S317. The sidecar instance suspends the execution of the batch job and obtains the real-time running data of the batch job.

[0104] S318. The sidecar instance sends real-time running data to the function instance, and sends a takeover method including direct takeover or waiting for takeover.

[0105] S319: The function instance receives real-time operation data and takeover mode.

[0106] S320: If the takeover mode is direct takeover, the function instance continues to execute the batch job based on the real-time running data.

[0107] S321. If the takeover mode is waiting for takeover, the function instance starts the secondary environment component.

[0108] S322. When the secondary environment component is started successfully, the function instance continues to execute the batch job based on the real-time running data.

[0109] The embodiment of the present invention also provides a Serverless platform, such as Figure 4The system diagram of a Serverless platform provided in an embodiment of the present application is shown, including a running area 401 of a sidecar instance, a running area 402 of a function instance, a scheduling node 403, a sidecar instance 404, and a function instance 405. The sidecar instance and the function instance are used to execute the batch job execution method as described in any of the embodiments of the present invention. For details, please refer to the description of the previous embodiment, which will not be repeated here.

[0110] Figure 5 A first structural diagram of a batch job execution device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device 500 is integrated into a sidecar instance in the Serverless platform and may include:

[0111] An execution instruction receiving module 510 is used to receive a batch job execution instruction sent by a scheduling node in the Serverless platform, wherein the batch job execution instruction is generated by the Serverless platform based on an http trigger instruction sent from a distributed batch platform, and the batch job execution instruction includes a function component name for executing the batch job;

[0112] A first job execution module 520, configured to call the function component corresponding to the function component name to execute the batch job;

[0113] The data processing module 530 is used to obtain the real-time operation data of executing the batch job and determine the takeover mode of the function instance when receiving the first startup success information or the second startup success information sent by the function instance through the environment interaction component, wherein the first startup success information is used to indicate that the function instance successfully starts the necessary environment component and the environment interaction component, and the second startup success information is used to indicate that the function instance successfully starts the secondary environment component;

[0114] The first data sending module 540 is used to send the real-time running data and the takeover method to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover method.

[0115] Furthermore, the above-mentioned data processing module 530 can be specifically used for: when receiving the first startup success information sent by the function instance through the environment interaction component, obtaining the real-time operation data of executing the batch job and determining the takeover method of the function instance, and when receiving the first startup success information, determining the resource load status during the execution of the batch job; judging whether the batch job is taken over by the function instance according to the resource load status and the necessary environment component; if taken over, obtaining the real-time operation data and determining the takeover method as direct takeover; if not taken over, continuing to execute the batch job, and determining the takeover method as waiting for takeover.

[0116] Furthermore, the above-mentioned data processing module 530 can also be specifically used for: when receiving the secondary startup success information sent by the function instance through the environment interaction component, obtaining the real-time operation data of executing the batch job; when receiving the secondary startup success information, obtaining the real-time operation data, and determining the takeover method as direct takeover.

[0117] Furthermore, the above-mentioned data processing module 530 can also be specifically used to: obtain the number of instructions waiting to be executed, the sidecar instance's own load and the resources required to execute the batch job from the resource load status; if the own load meets the preset operating standards and the number is not greater than the preset value, or the necessary environment components do not meet the required resources, determine that the function instance will not take over to execute the batch job; if the own load does not meet the preset operating standards, the number is greater than the preset value or the necessary environment components meet the required resources, determine that the function instance will take over to execute the batch job.

[0118] Optionally, the sidecar instance pre-runs a full environment component, wherein the full environment component includes an environment interaction component that communicates with the function instance.

[0119] The batch job execution device provided in this embodiment can be applied to the batch job execution method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0120] Figure 6 A second structural diagram of a batch job execution device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device 600 is integrated into a function instance in the Serverless platform, and may include:

[0121] A scheduling instruction receiving module 610 is used to receive a scheduling instruction sent by a scheduling node in the Serverless platform, where the scheduling instruction is generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction includes a network address of a sidecar instance that executes the batch job;

[0122] The component startup module 620 is used to start the necessary environment components and environment interaction components, and generate first startup success information when the startup is successful;

[0123] A second data sending module 630 is used to send the first startup success information to the side car instance corresponding to the network address through the environment interaction component, so that the side car instance determines whether to obtain the real-time operation data of executing the batch job when receiving the first startup success information;

[0124] The second job execution module 640 is used to continue executing the batch job based on the real-time operation data and the takeover method when receiving the real-time operation data and the takeover method sent by the sidecar instance.

[0125] Furthermore, the above-mentioned second job execution module 640 can be specifically used for: if the takeover mode is direct takeover, then continue to execute the batch job based on the real-time operation data, and the takeover mode is that the sidecar instance is determined by the required resources for executing the batch job and the necessary environment components; if the takeover mode is waiting for takeover, then start the secondary environment component; when the secondary environment component is successfully started, continue to execute the batch job based on the real-time operation data.

[0126] Furthermore, the batch job execution device may further include: a third job execution module;

[0127] The third job execution module is used to start the secondary environment component if the real-time operation data is not received; generate secondary startup success information when the secondary environment component is successfully started; send the secondary startup success information to the sidecar instance, so that the sidecar instance obtains the real-time operation data for executing the batch job when receiving the secondary startup success information; and continue to execute the batch job based on the real-time operation data when receiving the real-time operation data sent by the sidecar instance.

[0128] The batch job execution device provided in this embodiment can be applied to the batch job execution method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0129] Figure 7It is a block diagram of an electronic device for implementing a method for executing a batch job of an embodiment of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0130] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0132] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for executing a batch job.

[0133] In some embodiments, the batch job execution method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the batch job execution method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the batch job execution method in any other appropriate manner (for example, by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server) or a computing system that includes middleware components (e.g., an application server) or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein) or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0139] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0140] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. For example, those skilled in the art can use the various forms of processes shown above to reorder, add, or delete steps; the steps recorded in the present application can be performed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution of the present application can be achieved, and this document does not limit them here.

[0141] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for executing a batch job, characterized in that: Applied to the sidecar instance in the Serverless platform, including: Receive a batch job execution instruction sent by a scheduling node in the Serverless platform, where the batch job execution instruction is generated by the Serverless platform based on an http trigger instruction sent from a distributed batch platform, and the batch job execution instruction includes a function component name for executing the batch job; Calling the function component corresponding to the function component name to execute the batch job; When receiving the first startup success information or the second startup success information sent by the function instance through the environment interaction component, obtaining the real-time operation data of executing the batch job and determining the takeover mode of the function instance, the first startup success information is used to indicate that the function instance successfully starts the necessary environment component and the environment interaction component, and the second startup success information is used to indicate that the function instance successfully starts the secondary environment component; The real-time running data and the takeover method are sent to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover method.

2. The batch job execution method according to claim 1, characterized in that: When receiving the first start success information sent by the function instance through the environment interaction component, obtaining the real-time running data of executing the batch job and determining the takeover mode of the function instance include: When the first startup success information is received, determining a resource load status during the execution of the batch job; Determining whether the function instance should take over and execute the batch job according to the resource load status and the necessary environment components; If takeover is required, the real-time operation data is obtained, and the takeover mode is determined as direct takeover; If not, the batch job continues to be executed, and the takeover mode is determined as waiting for takeover.

3. The method for executing a batch job according to claim 2, characterized in that: When receiving the secondary startup success information sent by the function instance through the environment interaction component, obtaining the real-time running data of executing the batch job and determining the takeover mode of the function instance include: When the secondary startup success information is received, the real-time operation data is acquired, and the takeover mode is determined to be direct takeover.

4. The batch job execution method according to claim 2, characterized in that: Determining whether the function instance takes over the execution of the batch job according to the resource load status and the necessary environment components includes: Obtaining the number of instructions waiting to be executed, the load of the sidecar instance itself, and the resources required to execute the batch job from the resource load status; If the self-load meets the preset operation standard and the quantity is not greater than the preset value, or the necessary environment component does not meet the required resources, it is determined that the function instance will not take over the execution of the batch job; If the own load does not meet the preset operation standard, the quantity is greater than a preset value, or the necessary environment components meet the required resources, it is determined that the function instance will take over to execute the batch job.

5. The method for executing a batch job according to any one of claims 1 to 3, characterized in that: The sidecar instance pre-runs a full set of environment components, wherein the full set of environment components includes an environment interaction component that communicates with the function instance.

6. A method for executing a batch job, characterized in that: Applied to function instances in the Serverless platform, including: Receive a scheduling instruction sent by a scheduling node in the Serverless platform, where the scheduling instruction is generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction includes a network address of a sidecar instance that executes the batch job; Start necessary environment components and environment interaction components, and generate first start success information when the startup is successful; Sending the first startup success information to the side car instance corresponding to the network address through the environment interaction component, so that the side car instance determines whether to obtain the real-time running data of executing the batch job when receiving the first startup success information; When the real-time running data and the takeover method sent by the sidecar instance are received, the batch job continues to be executed based on the real-time running data and the takeover method.

7. The batch job execution method according to claim 6, characterized in that: Continuing to execute the batch job based on the real-time operation data and the takeover mode includes: If the takeover mode is direct takeover, the batch job continues to be executed based on the real-time running data, and the takeover mode is that the sidecar instance is determined by the resources required for executing the batch job and the necessary environment components; If the takeover mode is waiting for takeover, the secondary environment component is started; When the secondary environment component is successfully started, the batch job continues to be executed based on the real-time operation data.

8. The batch job execution method according to claim 6, characterized in that: The method further comprises: If the real-time operation data is not received, starting the secondary environment component; When the secondary environment component is successfully started, generating secondary startup success information; Sending the secondary startup success information to the sidecar instance, so that the sidecar instance obtains real-time running data of executing the batch job when receiving the secondary startup success information; When the real-time running data sent by the sidecar instance is received, the batch job continues to be executed based on the real-time running data.

9. A Serverless platform, characterized in that: It includes a running area of ​​a sidecar instance, a running area of ​​a function instance, a scheduling node, a sidecar instance and a function instance, and is used to execute the batch job execution method as described in any one of claims 1 to 5 and the batch job execution method as described in any one of claims 6 to 8.

10. A batch job execution device, characterized in that: The sidecar instance integrated into the Serverless platform includes: An execution instruction receiving module, used to receive a batch job execution instruction sent by a scheduling node in the Serverless platform, wherein the batch job execution instruction is generated by the Serverless platform based on an http trigger instruction sent from a distributed batch platform, and the batch job execution instruction includes a function component name for executing the batch job; A first job execution module, used for calling the function component corresponding to the function component name to execute the batch job; A data processing module, configured to obtain real-time operation data of executing the batch job and determine a takeover mode of the function instance when receiving first startup success information or second startup success information sent by the function instance through the environment interaction component, wherein the first startup success information is used to indicate that the function instance successfully starts the necessary environment component and the environment interaction component, and the second startup success information is used to indicate that the function instance successfully starts the secondary environment component; The first data sending module is used to send the real-time running data and the takeover method to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover method.

11. A batch job execution device, characterized in that: Function instances integrated into the Serverless platform include: A scheduling instruction receiving module is used to receive a scheduling instruction sent by a scheduling node in the Serverless platform, wherein the scheduling instruction is generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction includes a network address of a sidecar instance that executes the batch job; The component startup module is used to start the necessary environment components and environment interaction components, and generate the first startup success information when the startup is successful; A second data sending module is used to send the first startup success information to the side car instance corresponding to the network address through the environment interaction component, so that the side car instance determines whether to obtain the real-time operation data of executing the batch job when receiving the first startup success information; The second job execution module is used to continue executing the batch job based on the real-time operation data and the takeover method when receiving the real-time operation data and the takeover method sent by the side car instance.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the batch job execution method as described in any one of claims 1 to 5 and the batch job execution method as described in any one of claims 6 to 8.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the batch job execution method as described in any one of claims 1 to 5 and the batch job execution method as described in any one of claims 6 to 8 when executed.

Images