Batch job execution method, serverless platform, apparatus, device, and medium

Abstract

The application discloses a batch job execution method, a Serverless platform, an apparatus, an equipment and a medium, and relates to the technical field of distribution. The method comprises the following steps: receiving a batch job execution instruction sent by a scheduling node in a Serverless platform; calling a function component corresponding to a function component name to execute a batch job; when first startup success information or secondary startup success information sent by a function instance through an environment interaction component is received, acquiring real-time running data of the batch job execution and determining a takeover mode of the function instance; and sending the real-time running data and the takeover mode to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover mode. The technical scheme provided by the application shortens the startup time of the function instance reaching the level of a runnable task, and can achieve the purpose of improving the task execution speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distribution, in particular to a batch job execution method, a Serverless platform, a device, equipment and a medium. BACKGROUND

[0002] Since the birth of Serverless function computing technology, it has been favored by the industry with its event triggering, lightness, efficiency and flexibility, and is known as cloud computing 2.0. Lightness as one of the characteristics of the Serverless platform is one of the important factors for the rapid startup of the function instance container (hereinafter referred to as function instance) in the platform. Due to the lightness of the function, the startup of the function instance of most Serverless platforms is relatively fast, and the startup time is basically a few seconds or tens of seconds. However, in the batch job execution scenario, it is often necessary to start a function instance with 4G or more memory, and the startup of this part of the function instance is relatively slow, often reaching two or three minutes, which affects the execution speed of the overall task job. SUMMARY

[0003] The present application provides a batch job execution method, a Serverless platform, a device, equipment and a medium, which shortens the startup time of the function instance reaching the level of the executable task, and can achieve the purpose of improving the task execution speed.

[0004] In a first aspect, the present application provides a batch job execution method applied to a sidecar instance in a Serverless platform, which comprises:

[0005] When receiving the secondary startup success information sent by the environment interaction component of the function instance, real-time running data of executing the batch job is obtained, comprising:

[0006] When receiving the secondary startup success information, the real-time running data is obtained, and the takeover mode is determined as direct takeover.

[0007] In a second aspect, the present application provides a batch job execution method applied to a function instance in a Serverless platform, which comprises:

[0008] Receiving a scheduling instruction sent by a scheduling node in the Serverless platform, the scheduling instruction being generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction comprising a network address of a sidecar instance executing the batch job;

[0009] Starting necessary environment components and environment interaction components, and generating a first startup success information when the startup is successful;

[0010] The environment interaction component is configured to send the first startup success information to the sidecar instance corresponding to the network address, so that the sidecar instance determines whether to obtain real-time running data for executing the batch job when receiving the first startup success information.

[0011] When receiving the real-time running data and the takeover mode sent by the sidecar instance, the batch job is continued to be executed based on the real-time running data and the takeover mode.

[0012] In a third aspect, the present application provides a Serverless platform, comprising a running area of a sidecar instance, a running area of a function instance, a scheduling node, the sidecar instance and the function instance, and an execution method of the batch job according to any of the embodiments.

[0013] In a fourth aspect, the present application provides an execution device of a batch job, integrated in a sidecar instance in a Serverless platform, comprising:

[0014] An execution instruction receiving module is configured to receive a batch job execution instruction sent by a scheduling node in the Serverless platform, the batch job execution instruction being generated by the Serverless platform based on an http trigger instruction sent from a distributed batch platform, and the batch job execution instruction comprising a function component name for executing the batch job.

[0015] A first job execution module is configured to execute the batch job by calling a function component corresponding to the function component name.

[0016] A data processing module is configured to obtain real-time running data for executing the batch job and determine a takeover mode of the function instance when receiving first startup success information or secondary startup success information sent by the function instance through an environment interaction component, the first startup success information being used to indicate that the function instance successfully starts necessary environment components and the environment interaction component, and the secondary startup success information being used to indicate that the function instance successfully starts secondary environment components.

[0017] A first data sending module is configured to send the real-time running data and the takeover mode to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover mode.

[0018] In a fifth aspect, the present application provides an execution device of a batch job, integrated in a function instance in a Serverless platform, comprising:

[0019] The scheduling instruction receiving module is configured to receive a scheduling instruction sent by a scheduling node in the Serverless platform, the scheduling instruction being generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction including a network address of a sidecar instance that executes the batch job;

[0020] The component starting module is configured to start necessary environment components and environment interaction components, and generate first start success information when the starting is successful.

[0021] The second data sending module is configured to send the first start success information to the sidecar instance corresponding to the network address through the environment interaction component, so that the sidecar instance judges whether to acquire real-time running data of executing the batch job when the first start success information is received.

[0022] The second job execution module is configured to continue to execute the batch job based on the real-time running data and the takeover mode when the real-time running data and the takeover mode sent by the sidecar instance are received.

[0023] In a sixth aspect, the present application provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the batch job execution method described in any embodiment of the present application.

[0024] In a seventh aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the batch job execution method described in any embodiment of the present application when the processor executes the computer instructions.

[0025] In an eighth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program implements the batch job execution method described in any embodiment of the present application when the computer program is executed by a processor.

[0026] To solve the defects of the prior art in the background art, the embodiments of the present application provide a batch job execution method, and the execution of the method can bring the following beneficial effects: the present application runs full environment components such as JVM in the sidecar instance in advance, and configures environment interaction components in the sidecar instance and the function instance respectively, so as to determine whether the function instance takes over the execution of the batch job through information interaction. The present application executes the batch job by the sidecar instance first, and the full environment components of the function instance are started in two steps, the necessary environment components and the environment interaction components are started for the first time, and the secondary environment components are started for the second time. When the first-time starting success information or the second-time starting success information is received, the function instance takes over the execution of the batch job, the function instance quickly reaches the ability of a runnable task is realized, the starting time of the function instance reaching the level of a runnable task is shortened, and the purpose of improving the task execution speed is achieved.

[0027] It should be noted that the above computer instructions can be stored on a computer readable storage medium in whole or in part. The computer readable storage medium can be packaged together with the processor of the batch job execution device, or can be packaged separately from the processor of the batch job execution device, and the present application does not limit this.

[0028] The description of the second aspect, the third aspect, …, and the eighth aspect in the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second aspect, the third aspect, …, and the eighth aspect can refer to the beneficial effect analysis of the first aspect, which will not be described here.

[0029] It should be understood that the contents described in this part are not intended to identify the key or important features of the embodiments of the present application, nor are they used to limit the scope of the present application. Other features of the present application will become apparent through the following description.

[0030] It can be understood that before using the technical solutions disclosed in the embodiments of the present application, the type, scope of use and use scenario of the personal information involved in the present application should be informed to the user and the authorization of the user should be obtained in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating any inventive labor.

[0032] Figure 1 A first flowchart of a batch job execution method provided by the embodiments of the present application is shown in the figure;

[0033] Figure 2 A second flow diagram of a batch job execution method provided by an embodiment of the present application is shown in FIG. 7;

[0034] Figures 3a-3c A data interaction diagram of a batch job execution method provided by an embodiment of the present application is shown in FIG. 8;

[0035] Figure 4 A system diagram of a Serverless platform provided by an embodiment of the present application is shown in FIG. 9;

[0036] Figure 5 A first structural diagram of a batch job execution apparatus provided by an embodiment of the present application is shown in FIG. 10;

[0037] Figure 6 A second structural diagram of a batch job execution apparatus provided by an embodiment of the present application is shown in FIG. 11;

[0038] Figure 7 A block diagram of an electronic device for implementing a batch job execution method provided by an embodiment of the present application is shown in FIG. 12. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present application.

[0040] It should be noted that the terms “first”, “second”, “target” and “original” and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms “include”, “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Before the embodiment of the present application is executed, the Serverless function calculation platform (hereinafter referred to as Serverless platform) architecture of the present application needs to be introduced first. The Serverless platform architecture of the present embodiment is basically unchanged compared with the existing Serverless platform architecture, but the running area of the sidecar instance is increased, and by writing the sidecar instance program that can run Java Virtual Machine (JVM) and various processes such as network connection, and modifying the scheduling logic of the Serverless platform scheduling node, the pre-execution of batch jobs by the sidecar instance can be realized, and the information interaction between the function instance and the sidecar instance can be realized to reduce the first-round startup time of the function instance and the overall running time of the task.

[0042] The number and specification of the sidecar instance are added in the configuration file of the Serverless platform for the scheduling node to read and use. After the Serverless platform is started, the scheduling node will read the configuration file synchronously as usual when starting. After the scheduling node reads the file content, multiple sidecar instances are started in the "sidecar instance running area" dedicated resource area through the traditional function pulling mode, and are kept running persistently. These sidecar instances run full-environment components (including hardware scheduling components, file management components, network management components, security protection components, etc.), full-environment function calculation platform running required components (including variable management components, instruction receiving and processing components, container exception alarm components, etc.), JVM virtual machines, java monitoring components, and other necessary processes required by java code running, and the newly added environment interaction components for interacting with the function instance.

[0043] Figure 1 The first flowchart of the batch job execution method provided by the embodiment of the present application, the embodiment of the present application can be applied to the case of processing batch jobs by using the sidecar instance and the function instance in the Serverless platform. The batch job execution method provided by the embodiment of the present application is applied to the sidecar instance in the Serverless platform, and can be executed by the batch job execution device provided by the embodiment of the present application. The device can be realized by software and / or hardware, and integrated in the electronic device executing the method.

[0044] Referring to Figure 1 The method of the present embodiment includes but is not limited to the following steps:

[0045] S110, receiving the batch job execution instruction sent by the scheduling node in the Serverless platform.

[0046] The Serverless platform is a platform that can run Java, Python, and docker images, that is, a cloud computing service model. In this model, a cloud service provider provides a user with an environment for running code, and the user does not need to manage the configuration, maintenance, expansion, and other underlying infrastructure-related operations of a server. The batch job is executed by using the Serverless platform in this embodiment. The scheduling node is used to control the batch job in the Serverless platform.

[0047] The batch job execution instruction is generated by the Serverless platform based on the http trigger instruction sent by the distributed batch platform. The batch job execution instruction includes the function component name used to execute the batch job.

[0048] In this embodiment, the distributed batch platform detects that the running time of a certain batch job (for example, task A) has arrived, and then sends the http trigger instruction of task A to the Serverless platform. The http trigger instruction contains important information such as the name of each job in the batch job and the job grouping identification number. The Serverless platform receives the http trigger instruction from the distributed batch platform, reads the http trigger instruction through the scheduling node, and finds the function component name corresponding to the batch job name and the job grouping identification number in the database according to the important information in the http trigger instruction, and generates a batch job execution instruction.

[0049] The scheduling node first selects a relatively idle sidecar instance from a plurality of sidecar instances, and determines the sidecar instance as an instance container for executing the batch job. The scheduling node sends the batch job execution instruction to the sidecar instance, and the sidecar instance receives the batch job execution instruction.

[0050] S120, calling the function component corresponding to the function component name to execute the batch job.

[0051] The sidecar instance pre-executes the full-environment component, and the full-environment component includes an environment interaction component for communicating with the function instance.

[0052] In this embodiment, the sidecar instance starts pre-execution of the job task immediately after receiving the batch job execution instruction. The sidecar instance calls the corresponding function component based on the function component name in the batch job execution instruction, and executes the batch job through the function component in the sidecar instance. At the same time, the scheduling node synchronously pulls up a function instance, and the sidecar instance starts to wait for the first start of the function instance to be completed.

[0053] S130, when receiving the first start success information or the second start success information sent by the function instance through the environment interaction component, obtaining real-time running data of executing the batch job and determining the takeover mode of the function instance.

[0054] The start of the function instance is divided into first start and second start, and the full environment components in the function instance are divided into necessary environment components, secondary environment components and environment interaction components. The first start success information is used to represent that the function instance successfully starts the necessary environment components and the environment interaction components, and the second start success information is used to represent that the function instance successfully starts the secondary environment components.

[0055] Specifically, when receiving the first start success information sent by the function instance through the environment interaction component, the real-time running data of executing the batch job is obtained and the takeover mode of the function instance is determined, including: when the sidecar instance receives the first start success information through the environment interaction component, the sidecar instance needs to determine the resource load condition in the process of executing the batch job first; whether the function instance takes over the execution of the batch job is determined according to the resource load condition and the necessary environment components; if takeover, the real-time running data obtained by currently executing the batch job is obtained, and the takeover mode is determined as direct takeover; if not takeover, the batch job is continued to be executed, and the takeover mode is determined as waiting for takeover. Wherein, direct takeover refers to that the function instance takes over the execution of the batch job in the case of first start of the component; waiting for takeover refers to that the function instance needs to wait for takeover in the case of second start of the component (i.e. full environment component start is completed).

[0056] Further, whether the function instance takes over the execution of the batch job is determined according to the resource load condition and the necessary environment components, including: the number of waiting execution instructions, the self load of the sidecar instance and the required resources for executing the batch job are obtained from the resource load condition; if the self load meets the preset running standard and the number is not greater than the preset value, or the necessary environment components do not meet the required resources, it indicates that the current sidecar instance has the ability to continue to execute the batch job, or the current function instance has no ability to execute the batch job, and it is determined that the function instance does not take over the execution of the batch job; if the self load does not meet the preset running standard, the number is greater than the preset value (such as 2) or the necessary environment components meet the required resources, it indicates that the current sidecar instance has no ability to continue to execute the batch job, or the current function instance has the ability to execute the batch job, and it is determined that the function instance takes over the execution of the batch job. Wherein, the waiting execution instruction refers to other execution instructions subsequently assigned to this sidecar instance by the scheduling node. The preset running standard refers to the running condition that can support the sidecar instance to continue to execute the batch job.

[0057] It can be understood that, in the case that the current sidecar instance has no ability to continue to execute the batch job and the current function instance has the ability to execute the batch job, the function instance still takes over the execution of the batch job, but the takeover mode is determined to be waiting for takeover.

[0058] Specifically, after the function instance is successfully started for the first time, the sidecar instance determines not to take over the execution of the batch job by the function instance according to the resource load condition and the necessary environment components. At this time, the function instance starts the secondary environment components to realize secondary start. Then, when the sidecar instance receives the secondary start success information sent by the function instance through the environment interaction component, the real-time running data of executing the batch job is obtained, including that when the sidecar instance receives the secondary start success information through the environment interaction component, the sidecar instance obtains the real-time running data obtained by currently executing the batch job, and determines the takeover mode to be direct takeover.

[0059] S140, the real-time running data and the takeover mode are sent to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover mode.

[0060] In the embodiment of the application, the sidecar instance sends the real-time running data and the takeover mode to the function instance through the environment interaction component, so that the function instance continues to execute the batch job based on the real-time running data and the takeover mode.

[0061] The technical scheme provided in this embodiment comprises the following steps: receiving a batch job execution instruction sent by a scheduling node in a Serverless platform; executing the batch job by calling a function component corresponding to a function component name; when receiving first start success information or secondary start success information sent by a function instance through an environment interaction component, obtaining real-time running data of executing the batch job and determining a takeover mode of the function instance; and sending the real-time running data and the takeover mode to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover mode. In the application, JVM and other full environment components are pre-run in the sidecar instance, and environment interaction components are respectively configured in the sidecar instance and the function instance for information interaction between the function instance and the sidecar instance, so as to determine whether the function instance takes over the execution of the batch job through information interaction. The sidecar instance executes the batch job first, and the full environment components of the function instance are started in two steps, that is, the necessary environment components and the environment interaction components are started for the first time, and the secondary environment components are started for the second time, so that the startup time of the function instance reaching a runnable task level can be shortened. When the first start success information or the secondary start success information is received, the function instance takes over the execution of the batch job, so that the function instance quickly reaches a runnable task capability, and the purpose of improving the task execution speed is achieved.

[0062] Figure 2A second flowchart of a batch job execution method provided by the embodiment of the application is shown in FIG. 2. The batch job execution method provided by the embodiment of the application is applied to a function instance in a Serverless platform and can be executed by the batch job execution device provided by the embodiment of the application. The device can be implemented in the form of software and / or hardware and integrated in an electronic device for executing the method.

[0063] Referring to Figure 2 The method of the embodiment includes but is not limited to the following steps:

[0064] S210. Receiving a scheduling instruction sent by a scheduling node in a Serverless platform.

[0065] The scheduling instruction is generated by the Serverless platform based on a batch job execution instruction. The scheduling instruction includes a network address of a sidecar instance for executing the batch job, facilitating the matching of the sidecar instance and the function instance.

[0066] In the embodiment of the application, after the Serverless platform sends the batch job execution instruction to the sidecar instance, the network address of the sidecar instance is obtained, and the scheduling instruction is generated based on the network address and important information in the http trigger instruction. Then, the scheduling node synchronously pulls up a function instance, and sends the scheduling instruction to the function instance. The function instance receives the scheduling instruction.

[0067] S220. Starting necessary environment components and environment interaction components, and generating first start success information when the starting is successful.

[0068] In the embodiment of the application, during the pulling up of the function instance, the scheduling node of the modified Serverless platform is different from the original way of directly starting a new function instance, that is, allocating a central processing unit (CPU) and memory, starting full system components (including hardware scheduling components, file management components, network management components, security protection components, etc.), starting components required for the running of the function calculation platform (including variable management components, instruction receiving and processing components, container exception alarm components, etc.), starting a JVM virtual machine, and starting batch programs. Instead, after allocating the CPU and the memory, only the necessary environment components (including hardware scheduling components, file management components, network management components, etc.) and the newly added environment interaction components are started. When the necessary environment components and the environment interaction components are successfully started, the function instance generates first start success information.

[0069] S230. Sending the first start success information to the sidecar instance corresponding to the network address through the environment interaction components, so that the sidecar instance judges whether to obtain real-time running data for executing the batch job when receiving the first start success information.

[0070] In the embodiment of the present application, after the components of the newly pulled function instance complete the first start, the corresponding sidecar instance is determined according to the network address in the scheduling instruction, and the first start success information is sent to the sidecar instance through the environment interaction component in the function instance to explicitly indicate that the first start has been completed.

[0071] When the sidecar instance receives the first start success information through the environment interaction component, the sidecar instance needs to first determine the resource load condition in the process of executing the batch job; and determine whether to take over the execution of the batch job by the function instance according to the resource load condition and the necessary environment components; if so, the real-time running data obtained by the current execution of the batch job is acquired, and the takeover mode is determined as direct takeover; if not, the batch job is continued to be executed, and the takeover mode is determined as waiting for takeover.

[0072] S240, when receiving the real-time running data and the takeover mode sent by the sidecar instance, the batch job is continued to be executed based on the real-time running data and the takeover mode.

[0073] The takeover mode is determined by the sidecar instance according to the required resources and the necessary environment components for executing the batch job.

[0074] Specifically, the function instance continues to execute the batch job based on the real-time running data and the takeover mode, including: if the takeover mode is direct takeover, the function instance continues to execute the batch job based on the real-time running data; if the takeover mode is waiting for takeover, the function instance also needs to start the secondary environment component; when the function instance starts the secondary environment component successfully, the batch job is continued to be executed based on the real-time running data.

[0075] Further, the execution method of the batch job further includes: when the sidecar instance determines that the current sidecar instance has the ability to continue to execute the batch job or the current function instance does not have the ability to execute the batch job according to the resource load condition and the necessary environment components, it is determined that the function instance does not take over the execution of the batch job, and the takeover mode is determined as waiting for takeover. At this time, the function instance will not receive the real-time running data, and the function instance can only receive the information of waiting for takeover. At this time, the function instance starts the secondary environment component; when the secondary environment component starts successfully, the second start success information is generated; the function instance sends the second start success information to the sidecar instance, so that the sidecar instance acquires the real-time running data of the batch job when receiving the second start success information; when the function instance receives the real-time running data sent by the sidecar instance, the function instance continues to execute the batch job based on the real-time running data.

[0076] The technical scheme provided by the embodiment comprises the following steps: receiving a scheduling instruction sent by a scheduling node in a Serverless platform; starting necessary environment components and environment interaction components, and generating first start success information when the starting is successful; sending the first start success information to a sidecar instance corresponding to a network address through the environment interaction components, so that the sidecar instance judges whether to obtain real-time running data of the batch job when receiving the first start success information; and continuing to execute the batch job based on the real-time running data and the takeover mode when receiving the real-time running data and the takeover mode sent by the sidecar instance. The environment interaction components are configured in the sidecar instance and the function instance respectively, and are used for information interaction of the function instance and the sidecar instance, so as to determine whether the function instance replaces to execute the batch job through the information interaction. The batch job is executed by the sidecar instance first, and the full environment components of the function instance are started in two steps, that is, the necessary environment components and the environment interaction components are started first, and the secondary environment components are started secondly, so that the starting time of the function instance reaching the level of a runnable task can be shortened. When the first start success information or the second start success information is received, the function instance replaces to execute the batch job, the function instance quickly reaches the ability of a runnable task is realized, and the purpose of improving the task execution speed is achieved.

[0077] The execution method of the batch job provided by the embodiment of the application is further described below. FIG. 3 is a data interaction schematic diagram of an execution method of a batch job provided by an embodiment of the application. The embodiment of the application is optimized on the basis of the above-mentioned embodiments, and the specific optimization is that the embodiment explains and describes the data interaction process between the scheduling node, the sidecar instance and the function instance in detail during the execution of the batch job.

[0078] Referring to Figure 3a The method of the embodiment comprises but is not limited to the following steps:

[0079] S301, the scheduling node receives a read http trigger instruction.

[0080] The distributed batch platform detects that the running time of a certain batch job has arrived, and sends an http trigger instruction of the batch job to the Serverless platform. The Serverless platform receives the http trigger instruction from the distributed batch platform, and reads the http trigger instruction through the scheduling node.

[0081] S302, the scheduling node generates a batch job execution instruction.

[0082] The scheduling node finds the function component name corresponding to the batch job name and the job grouping identification number in the database according to the important information in the http trigger instruction, and generates a batch job execution instruction.

[0083] S303, the scheduling node sends the batch job execution instruction to the sidecar instance, and generates a scheduling instruction and sends the scheduling instruction to the function instance.

[0084] The scheduling node first selects a relatively idle sidecar instance from the plurality of sidecar instances, and determines the sidecar instance as an instance container for executing the batch job. The scheduling node further obtains a network address of the sidecar instance, and generates the scheduling instruction based on the network address and important information in the http trigger instruction.

[0085] S304, the sidecar instance receives the batch job execution instruction and starts executing the batch job.

[0086] S305, the function instance receives the scheduling instruction and starts the necessary environment component and the environment interaction component.

[0087] S306, the function instance generates first start success information when the necessary environment component and the environment interaction component are successfully started.

[0088] S307, the function instance sends the first start success information to the sidecar instance.

[0089] The function instance sends the first start success information to the sidecar instance corresponding to the network address through the environment interaction component.

[0090] S308, the sidecar instance determines a resource load condition in the process of executing the batch job when receiving the first start success information.

[0091] S309, the sidecar instance determines whether to take over the execution of the batch job by the function instance according to the resource load condition and the necessary environment component.

[0092] The sidecar instance analyzes the resource load condition (the specific analysis process is explained in detail in the above embodiment, and will not be repeated again). If it is determined that the current sidecar instance has the ability to continue executing the batch job, or the current function instance has no ability to execute the batch job, it is determined that the function instance does not take over the execution of the batch job. At this time, the sidecar instance continues to execute the batch job, and determines the takeover mode as waiting for takeover. The following steps S310-S316 are executed.

[0093] If it is determined that the current sidecar instance has no ability to continue executing the batch job, or the current function instance has the ability to execute the batch job, it is determined that the function instance takes over the execution of the batch job. At this time, the sidecar instance obtains real-time running data obtained by currently executing the batch job, and determines the takeover mode as direct takeover. The following steps S317-S322 are executed.

[0094] Referring to Figure 3b The method of the embodiment includes but is not limited to the following steps:

[0095] S310, the sidecar instance continues to execute the batch job and sends a takeover mode containing a takeover waiting to the function instance.

[0096] S311, the function instance receives the takeover mode and starts the secondary environment component.

[0097] S312, the function instance generates secondary start success information when the secondary environment component is successfully started.

[0098] S313, the function instance sends the secondary start success information to the sidecar instance.

[0099] S314, after the sidecar instance receives the secondary start success information, the real-time running data of executing the batch job is obtained.

[0100] S315, the sidecar instance sends the current real-time running data to the function instance.

[0101] S316, the function instance receives the real-time running data and continues to execute the batch job based on the real-time running data.

[0102] Referring to Figure 3c , the method of the embodiment includes but is not limited to the following steps:

[0103] S317, the sidecar instance suspends the execution of the batch job and obtains the real-time running data of executing the batch job.

[0104] S318, the sidecar instance sends the real-time running data to the function instance and sends a takeover mode containing a direct takeover or a takeover waiting.

[0105] S319, the function instance receives the real-time running data and the takeover mode.

[0106] S320, if the takeover mode is a direct takeover, the function instance continues to execute the batch job based on the real-time running data.

[0107] S321, if the takeover mode is a takeover waiting, the function instance starts the secondary environment component.

[0108] S322, when the secondary environment component is successfully started, the function instance continues to execute the batch job based on the real-time running data.

[0109] The embodiment of the application also provides a Serverless platform, such as Figure 4As shown is a system schematic diagram of a Serverless platform provided by an embodiment of the application, including a running area 401 of a sidecar instance, a running area 402 of a function instance, a scheduling node 403, a sidecar instance 404, and a function instance 405. The sidecar instance and the function instance are used to execute the batch job execution method as described in any of the embodiments of the application, and specific reference is made to the description of the foregoing embodiments, which will not be repeated here.

[0110] Figure 5 As shown is a first structural schematic diagram of a batch job execution device provided by an embodiment of the application, which is integrated in a sidecar instance in a Serverless platform, and can include: Figure 5

[0111] An execution instruction receiving module 510 is configured to receive a batch job execution instruction sent by a scheduling node in the Serverless platform, the batch job execution instruction being generated by the Serverless platform based on an http trigger instruction sent from a distributed batch platform, and the batch job execution instruction including a function component name used to execute the batch job.

[0112] A first job execution module 520 is configured to call a function component corresponding to the function component name to execute the batch job.

[0113] A data processing module 530 is configured to, when receiving first startup success information or second startup success information sent by a function instance through an environment interaction component, acquire real-time running data of executing the batch job and determine a takeover mode of the function instance, the first startup success information being used to represent that the function instance successfully starts necessary environment components and the environment interaction component, and the second startup success information being used to represent that the function instance successfully starts secondary environment components.

[0114] A first data sending module 540 is configured to send the real-time running data and the takeover mode to the function instance, so that the function instance continues to execute the batch job based on the real-time running data and the takeover mode.

[0115] ​Further, the data processing module 530 can be specifically configured to: when receiving first startup success information sent by an environment interaction component through a function instance, acquire real-time running data of executing the batch job and determine a takeover mode of the function instance; when receiving the first startup success information, determine a resource load condition in a process of executing the batch job; determine whether to take over execution of the batch job by the function instance according to the resource load condition and the necessary environment component; if yes, acquire the real-time running data, and determine the takeover mode as direct takeover; if not, continue to execute the batch job, and determine the takeover mode as waiting for takeover.

[0116] Further, the data processing module 530 can be specifically configured to: when receiving first startup success information sent by an environment interaction component through a function instance, acquire real-time running data of executing the batch job and determine a takeover mode of the function instance; when receiving the first startup success information, determine a resource load condition in a process of executing the batch job; determine whether to take over execution of the batch job by the function instance according to the resource load condition and the necessary environment component; if yes, acquire the real-time running data, and determine the takeover mode as direct takeover; if not, continue to execute the batch job, and determine the takeover mode as waiting for takeover.

[0117] Further, the data processing module 530 can be specifically configured to: when receiving first startup success information sent by an environment interaction component through a function instance, acquire real-time running data of executing the batch job and determine a takeover mode of the function instance; when receiving the first startup success information, determine a resource load condition in a process of executing the batch job; determine whether to take over execution of the batch job by the function instance according to the resource load condition and the necessary environment component; if yes, acquire the real-time running data, and determine the takeover mode as direct takeover; if not, continue to execute the batch job, and determine the takeover mode as waiting for takeover.

[0118] Optionally, the sidecar instance pre-executes full-amount environment components, and the full-amount environment components include environment interaction components in communication with the function instance.

[0119] The batch job execution apparatus provided by the embodiment can be applied to the batch job execution method provided by any of the above embodiments, and has corresponding functions and advantages.

[0120] Figure 6 A second structural schematic diagram of a batch job execution apparatus provided by an embodiment of the present application is shown in FIG. 6, which apparatus 600 is integrated in a function instance of a Serverless platform, and can include: Figure 6

[0121] ​The scheduling instruction receiving module 610 is configured to receive a scheduling instruction sent by a scheduling node in the Serverless platform, the scheduling instruction being generated by the Serverless platform based on a batch job execution instruction, and the scheduling instruction including a network address of a sidecar instance that executes the batch job;

[0122] The component starting module 620 is configured to start necessary environment components and environment interaction components, and generate first start success information when the starting is successful.

[0123] The second data sending module 630 is configured to send the first start success information to the sidecar instance corresponding to the network address through the environment interaction component, so that the sidecar instance judges whether to obtain real-time running data for executing the batch job when the first start success information is received.

[0124] The second job execution module 640 is configured to, when the real-time running data and the takeover mode sent by the sidecar instance are received, continue to execute the batch job based on the real-time running data and the takeover mode.

[0125] Further, the second job execution module 640 can be specifically configured to, if the takeover mode is direct takeover, continue to execute the batch job based on the real-time running data, and the takeover mode is determined by the sidecar instance based on required resources for executing the batch job and the necessary environment components; if the takeover mode is waiting takeover, start a secondary environment component; and when the secondary environment component is successfully started, continue to execute the batch job based on the real-time running data.

[0126] Further, the batch job execution device can further include a third job execution module.

[0127] The third job execution module is configured to, if the real-time running data is not received, start a secondary environment component; when the secondary environment component is successfully started, generate second start success information; send the second start success information to the sidecar instance, so that the sidecar instance obtains real-time running data for executing the batch job when the second start success information is received; and when the real-time running data sent by the sidecar instance is received, continue to execute the batch job based on the real-time running data.

[0128] The batch job execution device provided in the embodiment can be applied to the batch job execution method provided in any of the above embodiments, and has corresponding functions and advantages.

[0129] Figure 7This is a block diagram of an electronic device used to implement a batch job execution method according to embodiments of this application. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0130] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as batch job execution methods.

[0133] In some embodiments, the method of batch job execution can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the method of batch job execution described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method of batch job execution by other means, e.g., with the aid of firmware.

[0134] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0135] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0136] In the context of this application, a computer readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium can be a machine readable signal medium. More specific examples of the machine readable storage medium will include a one or more lines of a electrical connection, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0138] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain networks, and the Internet.

[0139] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0140] Note that the above merely describes preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, rearrangements and substitutions can be made by those skilled in the art without departing from the scope of the present application. For example, those skilled in the art can use the various forms of processes shown above to reorder, add or delete steps; can perform the steps described in the present application in parallel, sequentially or in different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0141] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for executing batch jobs, characterized in that, Sidecar instances used in serverless platforms include: The system receives batch job execution instructions sent by the scheduling node in the Serverless platform. The batch job execution instructions are generated by the Serverless platform based on HTTP trigger instructions sent from the distributed batch platform. The batch job execution instructions include the names of function components used to execute the batch jobs. The batch job is executed by calling the function component corresponding to the function component name. When the function instance receives the first startup success message or the second startup success message sent by the environment interaction component, the real-time running data of the batch job is obtained and the takeover method of the function instance is determined. The first startup success message is used to indicate that the function instance has successfully started the necessary environment components and the environment interaction component, and the second startup success message is used to indicate that the function instance has successfully started the secondary environment components. The real-time running data and the takeover method are sent to the function instance so that the function instance continues to execute the batch job based on the real-time running data and the takeover method. Specifically, when a function instance receives a successful initial startup message sent through the environment interaction component, the process of acquiring real-time running data of the batch job and determining the takeover method of the function instance includes: upon receiving the successful initial startup message, determining the resource load status during the execution of the batch job; determining, based on the resource load status and the necessary environment components, whether the function instance should take over the execution of the batch job; if it takes over, acquiring the real-time running data and determining the takeover method as direct takeover; if it does not take over, continuing to execute the batch job and determining the takeover method as waiting for takeover. When a secondary startup success message is received from a function instance via an environment interaction component, the real-time running data of the batch job and the takeover method of the function instance are obtained, including: when the secondary startup success message is received, the real-time running data is obtained, and the takeover method is determined to be direct takeover.

2. The batch operation execution method according to claim 1, characterized in that, Determining whether the function instance should take over the execution of the batch job based on the resource load status and the necessary environmental components includes: Obtain the number of instructions waiting to be executed, the load of the sidecar instance itself, and the resources required to execute the batch job from the resource load status; If the self-load meets the preset operating standard and the quantity is not greater than the preset value, or if the necessary environmental components do not meet the required resources, then it is determined that the batch job will not be taken over by the function instance. If the load itself does not meet the preset operating standard, the quantity is greater than the preset value, or the necessary environmental components meet the required resources, then it is determined that the function instance will take over the execution of the batch job.

3. The method for executing batch operations according to any one of claims 1 to 2, characterized in that, The sidecar instance pre-runs all environment components, which include environment interaction components that communicate with the function instance.

4. A method for executing batch jobs, characterized in that, Function examples applicable to Serverless platforms include: Receive scheduling instructions sent by the scheduling node in the Serverless platform. The scheduling instructions are generated by the Serverless platform based on batch job execution instructions. The scheduling instructions include the network address of the sidecar instance executing the batch job. Start the necessary environment components and environment interaction components, and generate a first-time startup success message when startup is successful; The environment interaction component sends the first startup success information to the sidecar instance corresponding to the network address, so that the sidecar instance can determine whether to obtain the real-time running data for executing the batch job when it receives the first startup success information; When the sidecar instance receives real-time running data and takeover mode, the batch job continues to be executed based on the real-time running data and the takeover mode, including: if the takeover mode is direct takeover, the batch job continues to be executed based on the real-time running data, wherein the takeover mode is determined by the sidecar instance from the resources required to execute the batch job and the necessary environment components; if the takeover mode is waiting for takeover, the secondary environment components are started; when the secondary environment components are successfully started, the batch job continues to be executed based on the real-time running data. The method further includes: if the real-time running data is not received, starting the secondary environment component; when the secondary environment component starts successfully, generating secondary startup success information; sending the secondary startup success information to the sidecar instance so that the sidecar instance can obtain real-time running data for executing the batch job when it receives the secondary startup success information; and continuing to execute the batch job based on the real-time running data when it receives the real-time running data sent by the sidecar instance.

5. A serverless platform, characterized in that, It includes the running area of ​​the sidecar instance, the running area of ​​the function instance, the scheduling node, the sidecar instance, and the function instance, and is used to execute the batch job execution method as described in any one of claims 1 to 4.

6. A batch operation execution device, characterized in that, A sidecar instance integrated into a serverless platform, used to implement the batch job execution method of claim 1, includes: The execution instruction receiving module is used to receive batch job execution instructions sent by the scheduling node in the Serverless platform. The batch job execution instructions are generated by the Serverless platform based on the HTTP trigger instructions sent from the distributed batch platform. The batch job execution instructions include the name of the function component used to execute the batch job. The first job execution module is used to call the function component corresponding to the function component name to execute the batch job; The data processing module is used to obtain real-time running data of the batch job and determine the takeover method of the function instance when it receives the first startup success information or the second startup success information sent by the function instance through the environment interaction component. The first startup success information is used to indicate that the function instance has successfully started the necessary environment components and the environment interaction component, and the second startup success information is used to indicate that the function instance has successfully started the secondary environment components. The first data sending module is used to send the real-time running data and the takeover method to the function instance, so that the function instance can continue to execute the batch job based on the real-time running data and the takeover method.

7. A batch operation execution device, characterized in that, A function instance integrated into the Serverless platform, used to implement the batch job execution method of claim 4, comprising: The scheduling instruction receiving module is used to receive scheduling instructions sent by the scheduling node in the Serverless platform. The scheduling instructions are generated by the Serverless platform based on batch job execution instructions, and the scheduling instructions include the network address of the sidecar instance executing the batch job. The component startup module is used to start the necessary environment components and environment interaction components. When the startup is successful, it generates the first startup success message. The second data sending module is used to send the first startup success information to the sidecar instance corresponding to the network address through the environment interaction component, so that the sidecar instance can determine whether to obtain the real-time running data for executing the batch job when it receives the first startup success information; The second job execution module is used to continue executing the batch job based on the real-time running data and takeover method sent by the sidecar instance when it receives the real-time running data and takeover method.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the batch job execution method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the batch job execution method as described in any one of claims 1 to 4.

Images