Distributed batch duplicate prevention method and apparatus based on serverless

Abstract

This invention provides a serverless distributed batch anti-duplicate method and apparatus, relating to the field of distributed architecture technology. The method includes: reading job execution information from ZooKeeper; obtaining a job ID using a hash algorithm based on the job execution information; sending the job ID to all slave nodes in the DBF management module cluster; determining whether the job ID status returned by the slave nodes and the job ID status of the slave nodes indicate a completed state; if not, triggering a function program instance from the Function Compute platform interface to execute the job based on the job execution information; if the job ID status indicates a completed state, modifying the status corresponding to the job ID to completed, and synchronizing the job ID status to all slave nodes. This invention provides a serverless batch SDK for sending status via the DBF management module, allocating unique job IDs and functions, which can significantly reduce the probability of duplicate job scheduling.

Description

Technical Field

[0001] This invention relates to the field of distributed architecture technology and can be used in the financial sector, particularly to a serverless-based distributed batch anti-duplicate method and apparatus. Background Technology

[0002] In the field of distributed technology, as mainframe services are increasingly moved offline, the Distributed Batch Framework (DBF) is carrying more and more core applications, and its importance is growing daily. Based on DBF runtime statistics, it can be inferred that the resource consumption of all applications is significant and shows a continuous upward trend. By functionalizing batch tasks, batch jobs can be started on demand and destroyed immediately after use, greatly improving the resource utilization of distributed batch programs.

[0003] Currently, distributed batch anti-duplicate measures are mainly implemented by checking the execution records of the application database using the distributed batch SDK of the function. However, this method has certain omissions in anti-duplicate measures when the database information of the application is lost. Summary of the Invention

[0004] In view of this, the present invention provides a serverless distributed batch anti-duplicate method and apparatus to solve at least one of the aforementioned problems.

[0005] To achieve the above objectives, the present invention adopts the following solution:

[0006] According to a first aspect of the present invention, a distributed batch anti-duplicate method based on Serverless is provided. The method is performed by the master node in the ServerlessDBF management module cluster. The method includes: reading job execution information in ZooKeeper; obtaining a job ID using a hash algorithm based on the job execution information; sending the job ID to all slave nodes in the ServerlessDBF management module cluster; determining whether the job ID status returned by the slave node and its own job ID status have a completed status. If not, the method triggers a function computation platform interface startup function program instance to execute the job based on the job execution information. If the job ID has a completed status, the method modifies the status corresponding to the job ID to completed and synchronizes the job ID status to all slave nodes.

[0007] As an embodiment of the present invention, the job execution information in the above method includes: application name, group information, batch number, job name and session number; the step of obtaining the job ID using a hash algorithm based on the job execution information includes: concatenating the application name, group information, batch number, job name and session number, and then obtaining the job ID through a hash algorithm.

[0008] As an embodiment of the present invention, the above method further includes: monitoring job execution information in Zookeeper; and reading the job execution information in Zookeeper in response to the presence of job execution information in Zookeeper.

[0009] As an embodiment of the present invention, in the above method, after receiving the job ID sent by the master node, the slave node in the ServerlessDBF management module cluster checks whether the job ID exists in its own memory. If it does, it returns the status of the job ID to the master node; if it does not, it returns a message that the job ID does not exist to the master node, and at the same time sets the status of the job ID in its own memory to the start job state.

[0010] As an embodiment of the present invention, after the function computing platform starts the function program instance to complete the job in the above method, it further includes: the Serverless Batch SDK of the function program instance sends the job completion message to the Zookeeper; at the same time, according to the ServerlessDBF management module domain name and port recorded in the program configuration file corresponding to the Serverless Batch SDK, the job completion message is sent to the corresponding ServerlessDBF management module node.

[0011] As an embodiment of the present invention, the job completion message in the above method includes: application name, group information, batch number, job name, and session number.

[0012] As an embodiment of the present invention, after the ServerlessDBF management module node receives the job completion message, it synchronizes the job completion message to all nodes in the ServerlessDBF management module cluster. Each ServerlessDBF management module node obtains the job ID corresponding to the job completion message by running a hash algorithm and updates the status of the job ID to the completed status.

[0013] According to a second aspect of the present invention, a serverless-based distributed batch anti-duplicate device is provided. The device includes: a reading unit for reading job execution information in Zookeeper; a job ID calculation unit for obtaining a job ID using a hash algorithm based on the job execution information; a sending unit for sending the job ID to all slave nodes in the ServerlessDBF management module cluster; a judging unit for judging whether the job ID status returned by the slave node and its own job ID status have a completed status; a triggering unit for triggering a function computation platform interface startup function program instance to execute the job based on the job execution information when the judging unit's judgment result is that the job ID does not exist; and a synchronization unit for modifying the status corresponding to the job ID to completed when the judging unit's judgment result is that the job ID exists, and synchronizing the job ID status to all slave nodes.

[0014] As an embodiment of the present invention, the job execution information in the above-mentioned device includes: application name, group information, batch number, job name and session number; the job ID calculation unit is specifically used to: concatenate the application name, group information, batch number, job name and session number, and then obtain the job ID through a hash algorithm.

[0015] As an embodiment of the present invention, the above-mentioned device further includes a listening unit for listening to job execution information in Zookeeper; in response to the presence of job execution information in Zookeeper, the reading unit reads the job execution information in Zookeeper.

[0016] As an embodiment of the present invention, after receiving the job ID sent by the master node, the slave node in the ServerlessDBF management module cluster checks whether the job ID exists in its own memory. If it does, it returns the status of the job ID to the master node; if it does not, it returns a message that the job ID does not exist to the master node, and at the same time sets the status of the job ID in its own memory to the start job state.

[0017] As an embodiment of the present invention, after the function computing platform starts the function program instance to complete the job, it further includes: the Serverless Batch SDK of the function program instance sending the job completion message to the Zookeeper; and at the same time, according to the ServerlessDBF management module domain name and port recorded in the program configuration file corresponding to the Serverless Batch SDK, sending the job completion message to the corresponding ServerlessDBF management module node.

[0018] As an embodiment of the present invention, the above-mentioned job completion message includes: application name, group information, batch number, job name, and session number.

[0019] As an embodiment of the present invention, after receiving the job completion message, the ServerlessDBF management module node synchronizes the job completion message to all nodes in the ServerlessDBF management module cluster. Each ServerlessDBF management module node obtains the job ID corresponding to the job completion message by running a hash algorithm and updates the status of the job ID to the completed status.

[0020] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0021] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0022] As can be seen from the above technical solutions, the Serverless-based distributed batch anti-duplicate method and device provided by the present invention can greatly reduce the probability of job duplication scheduling by allocating unique job IDs and function statuses through the ServerlessDBF management module's Serverless batch SDK. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0024] Figure 1 This is a structural diagram of a serverless-based distributed batch anti-duplicate system provided in an embodiment of this application;

[0025] Figure 2 This is a flowchart illustrating a serverless-based distributed batch anti-duplicate method provided in an embodiment of this application.

[0026] Figure 3 This is a flowchart illustrating a distributed batch anti-duplicate method based on Serverless provided in another embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the structure of a serverless-based distributed batch anti-duplicate device provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a distributed batch anti-duplicate device based on Serverless, provided in another embodiment of this application;

[0029] Figure 6 This is a schematic block diagram of the system configuration of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0031] Currently, distributed batch processing in function computing platforms primarily prevents duplicate job execution by checking application database execution records using the function's distributed batch SDK. This approach can lead to job duplication if application database data is lost. Therefore, this paper proposes a method to modify the ServerlessDBF management module and the Serverless batch SDK to significantly reduce the probability of duplicate job scheduling.

[0032] Before describing the technical solution of this application in detail, the technical names involved in this application are explained as follows:

[0033] "Application": The name of a group in a commercial bank. Groups are divided according to function, and one functional group is called an application.

[0034] "Function Compute Platform": A platform that can run Java, Python, and Docker images.

[0035] "Distributed Batch (DBF)": A platform within commercial banking systems that enables scheduled task execution and multi-task orchestration, with all tasks running on PaaS or virtual machines.

[0036] "Assignment": that is, DBF task.

[0037] “set”: This refers to the grouping information of the job in DBF.

[0038] "Serverless Batch": Another form of DBF, in which tasks no longer run on PaaS or virtual machines, but on the Function Compute platform.

[0039] like Figure 1The diagram shown is a structural diagram of a serverless-based distributed batch anti-duplicate system provided in an embodiment of this application. Figure 1 As can be seen, the system may include the following components: ServerlessDBF management module 110, Zookeeper module 120, function computing platform 130, and batch control module 140. The ServerlessDBF management module 110 is connected to both the Zookeeper module 120 and the function computing platform 130. The Zookeeper module 120 is connected to the batch control module 140. The batch control module 140 may include: batch console 141, batch database 142, and batch controller 143.

[0040] The ServerlessDBF management module 110 is a cluster containing one master node and multiple slave nodes, based on... Figure 1 The system architecture is described below, focusing on the master node side of the ServerlessDBF management module cluster, illustrating the process of the Serverless-based distributed batch anti-duplicate method of this application. Figure 2 The diagram shown is a schematic flowchart of a serverless-based distributed batch anti-duplicate method provided in an embodiment of this application. The method includes the following steps:

[0041] Step S201: Read the job execution information in Zookeeper.

[0042] In this embodiment, during the execution of distributed batch jobs, a node in the ServerlessDBF management module 110 first obtains the application's function configuration file and hierarchical parameter file; then, based on the function configuration file and hierarchical parameter file, it registers function information with the Zookeeper module 120 and calls the function computing platform 130 interface to create functions; at the same time, the batch controller 143 is also listening to the registered function information in Zookeeper. When the preset conditions for batch job execution are met, the batch controller 143 sends job execution information to the Zookeeper module 120 and marks the job status in the batch database as started.

[0043] This step involves the ServerlessDBF management module master node immediately reading the job execution information when it learns that there is job execution information in Zookeeper.

[0044] Preferably, the job execution information in this step may include: application name, grouping information, batch number, job name, and session number, which are the necessary information to locate a unique job.

[0045] Step S202: Obtain the job ID using a hash algorithm based on the job execution information.

[0046] Preferably, in this embodiment, the application name, group information, batch number, job name and session number in the above job execution information can be concatenated, and then a hash algorithm can be used to calculate the concatenated information to obtain a unique job ID.

[0047] Step S203: Send the job ID to all slave nodes in the ServerlessDBF management module cluster.

[0048] Step S204: Determine whether the job ID status returned by the node and the job ID status of the user are in a completed state. If not, proceed to step S205; if so, proceed to step S206.

[0049] After receiving the job ID information from the master node in step S203, the slave nodes in the ServerlessDBF management module cluster check whether the job ID exists in their own memory. If it does, they return the job ID status to the master node; if not, they return a message indicating that the job ID does not exist, and simultaneously set the job ID status to "Started Job" in their own memory. In this embodiment, to simplify the process, the job ID status can only include "Started Job" and "Completed". Of course, in other processes, it can include more statuses, such as "Not Started", "Ready", etc.

[0050] Step S205: Execute the job by triggering the function computing platform interface startup function program instance based on the job execution information.

[0051] After the function program instance is started, it can still perform job deduplication prevention according to the original DBF anti-duplicate mechanism, that is, by checking the execution records of the application database through the function's distributed batch SDK to determine whether the job is executed repeatedly.

[0052] Step S206: Modify the status corresponding to the job ID to "Completed" and synchronize the job ID status to all slave nodes. After receiving the job ID status, all slave nodes synchronize the job ID status to "Completed" in their respective memory.

[0053] As can be seen from the above, the Serverless-based distributed batch anti-duplicate method provided by the present invention can greatly reduce the probability of job duplication by allocating unique job IDs and function statuses through the Serverless DBF management module's Serverless batch SDK. In addition, this anti-duplicate method and the original DBF anti-duplicate mechanism can be well combined and do not conflict with each other, thus further reducing the probability of job duplication.

[0054] like Figure 3 The diagram shown is a flowchart of a serverless-based distributed batch anti-duplicate method according to another embodiment of this application. The method includes the following steps:

[0055] Step S301: The master node of the ServerlessDBF management module reads the job execution information in Zookeeper. This job execution information may include: application name, group information, batch number, job name and session number, which are the necessary information to locate a unique job.

[0056] Step S302: The ServerlessDBF management module master node obtains the job ID using a hash algorithm based on the job execution information. This involves concatenating the application name, group information, batch number, job name, and session number from the job execution information, and then using a hash algorithm to calculate a unique job ID.

[0057] Step S303: The master node of the ServerlessDBF management module sends the job ID to all slave nodes in the ServerlessDBF management module cluster.

[0058] Step S304: After receiving the job ID, the slave nodes of the ServerlessDBF management module check whether the job ID exists in their own memory. If it exists, they return the status of the job ID to the master node; if it does not exist, they return a message to the master node that the job ID does not exist, and at the same time set the status of the job ID in their own memory to "start job".

[0059] Step S305: The master node of the ServerlessDBF management module determines whether the job ID status returned by the slave node and its own job ID status are in a completed state. If not, proceed to step S306; if so, proceed to step S309.

[0060] Step S306: The ServerlessDBF management module master node triggers the function computing platform interface startup function program instance to execute the job based on the job execution information.

[0061] After the function program instance is started, it can still perform job deduplication prevention according to the original DBF anti-duplicate mechanism, that is, by checking the execution records of the application database through the function's distributed batch SDK to determine whether the job is executed repeatedly.

[0062] Step S307: After the Serverless Batch SDK of the function program instance completes the job, it sends the job completion message to Zookeeper. At the same time, based on the ServerlessDBF management module domain name and port recorded in the program configuration file corresponding to the Serverless Batch SDK, it also sends the job completion message to the corresponding ServerlessDBF management module node.

[0063] It should be noted that the Serverless Batch SDK does not necessarily send a job completion message to the master node after the job is completed. Instead, it sends it to the corresponding ServerlessDBF management module node according to the address information in the configuration file. That is, the node that receives the job completion message may be the master node or a slave node.

[0064] Preferably, the job completion message contains the same information as the job execution information, including: application name, group information, batch number, job name, and session number.

[0065] Step S308: The ServerlessDBF management module node that receives the job completion message synchronizes the job completion message to all other nodes in the ServerlessDBF management module cluster. Each ServerlessDBF management module node obtains the job ID corresponding to the job completion message by running a hash algorithm and updates the status of the job ID to the completed status.

[0066] Since the job completion message contains the same application name, group information, batch number, job name, and session number as the job execution message, each node can run its own hash algorithm to obtain the corresponding job ID and update the job ID status.

[0067] Preferably, after completing step S308, the master node of the ServerlessDBF management module can confirm the status of the job ID with all slave nodes. When the status of all nodes is consistent, the master node will update the status of the job ID in the application database to provide a basis for the original DBF anti-duplicate mechanism.

[0068] Step S309: The ServerlessDBF management module master node modifies the status corresponding to the job ID to "Completed" and synchronizes the job ID status to all slave nodes. After receiving the job ID status, all slave nodes synchronize the job ID status to "Completed" in their respective memory.

[0069] As can be seen from the above, the Serverless-based distributed batch anti-duplicate method provided by the present invention can greatly reduce the probability of job duplication by allocating unique job IDs and function statuses through the Serverless DBF management module's Serverless batch SDK. In addition, this anti-duplicate method and the original DBF anti-duplicate mechanism can be well combined and do not conflict with each other, thus further reducing the probability of job duplication.

[0070] like Figure 4 The diagram shown is a schematic diagram of a serverless distributed batch anti-duplicate device provided in an embodiment of this application. The device includes: a reading unit 410, a job ID calculation unit 420, a sending unit 430, a judgment unit 440, a triggering unit 450, and a synchronization unit 460. The reading unit 410, the job ID calculation unit 420, the sending unit 430, and the judgment unit 440 are connected in sequence, and the triggering unit 450 and the synchronization unit 460 are respectively connected to the judgment unit 440.

[0071] The reading unit 410 is used to read job execution information in Zookeeper.

[0072] The job ID calculation unit 420 is used to obtain the job ID using a hash algorithm based on the job execution information.

[0073] The sending unit 430 is used to send the job ID to all slave nodes in the ServerlessDBF management module cluster.

[0074] The judgment unit 440 is used to determine whether the job ID status returned from the node and its own job ID status have a completed status.

[0075] The triggering unit 450 is used to execute the job based on the job execution information when the judgment result of the judgment unit 440 is that the job does not exist.

[0076] The synchronization unit 460 is used to modify the status corresponding to the job ID to "completed" when the judgment result of the judgment unit 440 is "existent", and to synchronize the status of the job ID to all slave nodes.

[0077] Preferably, the job execution information in the above device includes: application name, group information, batch number, job name, and session number; the job ID calculation unit 420 is specifically used to: concatenate the application name, group information, batch number, job name, and session number, and then obtain the job ID through a hash algorithm.

[0078] Preferred, such as Figure 5As shown, the above device also includes a listening unit 470 for listening to job execution information in Zookeeper; in response to the presence of job execution information in Zookeeper, the reading unit 410 reads the job execution information in Zookeeper.

[0079] Preferably, after receiving the job ID sent by the master node, the slave node in the ServerlessDBF management module cluster checks whether the job ID exists in its own memory. If it does, it returns the status of the job ID to the master node; if it does not, it returns a message that the job ID does not exist to the master node, and at the same time sets the status of the job ID in its own memory to the start job state.

[0080] Preferably, after the function computing platform starts the function program instance to complete the job, it further includes: the Serverless Batch SDK of the function program instance sending the job completion message to the Zookeeper; and at the same time, according to the ServerlessDBF management module domain name and port recorded in the program configuration file corresponding to the Serverless Batch SDK, sending the job completion message to the corresponding ServerlessDBF management module node.

[0081] Preferably, the above-mentioned job completion message includes: application name, group information, batch number, job name, and session number.

[0082] Preferably, after receiving the job completion message, the ServerlessDBF management module node synchronizes the job completion message to all nodes in the ServerlessDBF management module cluster. Each ServerlessDBF management module node obtains the job ID corresponding to the job completion message by running a hash algorithm and updates the status of the job ID to the completed status.

[0083] For a detailed description of each of the above units, please refer to the corresponding description in the foregoing method embodiments, which will not be repeated here.

[0084] As can be seen from the above, the Serverless-based distributed batch anti-duplicate device provided by the present invention can greatly reduce the probability of job duplication by allocating unique job IDs and function statuses through the Serverless DBF management module's Serverless batch SDK. In addition, this anti-duplicate method and the original DBF anti-duplicate mechanism can be well combined and do not conflict with each other, thus further reducing the probability of job duplication.

[0085] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0086] This invention also provides a computer-readable storage medium storing a computer program for performing the above-described methods.

[0087] like Figure 6 As shown, the electronic device 600 may also include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily need to include these components. Figure 6 All components shown; in addition, the electronic device 600 may also include Figure 6 For components not shown, please refer to existing technologies.

[0088] like Figure 6 As shown, the central processing unit 100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operation of various components of the electronic device 600.

[0089] The memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 100 may execute the program stored in the memory 140 to perform information storage or processing, etc.

[0090] Input unit 120 provides input to central processing unit 100. Input unit 120 may be, for example, a keypad or touch input device. Power supply 170 provides power to electronic device 600. Display 160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0091] The memory 140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 140 can also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142 for storing application and function programs or processes for executing the operation of the electronic device 600 via the central processing unit 100.

[0092] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0093] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processing unit 100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0094] Based on different communication technologies, multiple communication modules 110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby enabling typical telecommunications functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 130 is coupled to a central processing unit 100, enabling on-device recording via the microphone 132 and on-device playback of stored audio via the speaker 131.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A serverless-based distributed batch anti-duplicate method, characterized in that, The method includes: The master node of the ServerlessDBF management module reads job execution information from ZooKeeper, obtains the job ID using a hash algorithm based on the job execution information, and sends the job ID to all slave nodes in the ServerlessDBF management module cluster. After receiving a job ID, the slave nodes of the ServerlessDBF management module check whether the job ID exists in their own memory. If it does, they return the status of the job ID to the master node; if not, they return a message to the master node that the job ID does not exist, and at the same time set the status of the job ID in their own memory to the start job state. The ServerlessDBF management module master node determines whether the job ID status returned by the slave node and its own job ID status have a completed status. If not, the ServerlessDBF management module master node triggers the function computing platform interface to start a function program instance to execute the job based on the job execution information. After the job is completed, the function program instance Serverless Batch SDK sends a job completion message to Zookeeper, and at the same time, according to the ServerlessDBF management module domain name and port recorded in the program configuration file corresponding to the Serverless Batch SDK, it also sends the job completion message to the corresponding ServerlessDBF management module node. The ServerlessDBF management module node that receives the job completion message synchronizes the job completion message to all other nodes in the ServerlessDBF management module cluster. Each ServerlessDBF management module node obtains the job ID corresponding to the job completion message by running a hash algorithm, and updates the status of the job ID to the completed status. If present, the ServerlessDBF management module master node modifies the status corresponding to the job ID to "completed" and synchronizes the job ID status to all slave nodes. After receiving the job ID status, all slave nodes synchronize the job ID status to "completed" in their respective memory.

2. The serverless-based distributed batch anti-duplicate method as described in claim 1, characterized in that, The job execution information includes: application name, group information, batch number, job name, and session number; The method of obtaining the job ID using a hash algorithm based on the job execution information includes: concatenating the application name, group information, batch number, job name, and session number, and then obtaining the job ID through a hash algorithm.

3. The serverless-based distributed batch anti-duplicate method as described in claim 1, characterized in that, The method further includes: monitoring job execution information in Zookeeper; and reading the job execution information in Zookeeper if job execution information is available in Zookeeper.

4. The serverless-based distributed batch anti-duplicate method as described in claim 1, characterized in that, The job completion message includes: application name, group information, batch number, job name, and session number.

5. A serverless-based distributed batch anti-duplicate device, applied to the method described in any one of claims 1 to 4, characterized in that, The device includes: The read unit is used to read job execution information from ZooKeeper. The job ID calculation unit is used to obtain the job ID using a hash algorithm based on the job execution information. The sending unit is used to send the job ID to all slave nodes in the ServerlessDBF management module cluster; The judgment unit is used to determine whether the job ID status returned from the node and its own job ID status have a completed status. The triggering unit is used to execute the job by triggering a function computing platform interface startup function program instance based on the job execution information when the judgment result of the judgment unit is that the job does not exist. The synchronization unit is used to modify the status corresponding to the job ID to "completed" when the judgment unit determines that the job ID exists, and to synchronize the status of the job ID to all slave nodes.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images