A method for scheduling system script synchronization and execution

Abstract

The application provides a scheduling system script synchronization and execution method, which comprises the following steps: obtaining a first scheduling system, wherein the first scheduling system comprises a server end and a worker end; obtaining first script development information submitted by a data development user according to the server end, and performing git resource configuration on the first script development information to obtain first configuration information; transmitting the first script development information to a first worker node based on a zookeeper to generate a first change instruction; pulling the first script development information at the server end according to the first change instruction by the first worker node, and generating a first execution instruction by the server end; and executing the first script development information by a data processing server on the first worker node according to the first execution instruction.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for synchronizing and executing scripts in a scheduling system. Background Technology

[0002] With the continuous development of computer technology and the deepening of enterprise digitalization, automated methods for data processing are also maturing. Using task scheduling combined with script development to achieve automated data processing is a commonly used approach. For example, the Linux crontab task scheduling and script development on the server are widely used methods.

[0003] However, in the process of implementing the technical solution of the invention in this application, it was discovered that the above-mentioned technology has at least the following technical problems:

[0004] Existing technologies only consider task scheduling, task dependencies, and script execution, which has technical problems such as the inability to guarantee that the script and the referenced script code are both the latest versions, and the inability to roll back the referenced script when task scheduling fails. Summary of the Invention

[0005] This application provides a method for script synchronization and execution in a scheduling system, solving the technical problems of existing technologies that only consider task scheduling, task dependencies, and script execution, resulting in the inability to guarantee that both the script and its referenced scripts are the latest versions simultaneously, and the inability to roll back referenced scripts when task scheduling fails. By dividing the task scheduling system into a server and worker side, the server retrieves script development information provided by developers and configures resources in the Git repository based on this information. The ZooKeeper program obtains the worker nodes corresponding to the changed script development information. Based on the changes, the worker nodes pull the script development information from the server, and then execute tasks according to the resource configuration results and the pulled script development information. This achieves the technical effect of ensuring that the script and its referenced code are the latest versions, and that due to the existence of the Git repository, both the script and its referenced scripts can be rolled back simultaneously during a rollback.

[0006] In view of the above problems, this application provides a method for scheduling system script synchronization and execution.

[0007] In a first aspect, this application provides a method for synchronizing and executing scripts in a scheduling system. The method includes: obtaining a first scheduling system, the first scheduling system including a server and a worker; obtaining first script development information submitted by a data development user based on the server, and configuring Git resources on the first script development information to obtain first configuration information; transmitting the first script development information to a first worker node based on ZooKeeper, and generating a first change instruction, wherein the ZooKeeper is communicatively linked to the server, and the first worker node is included in the worker; according to the first change instruction, the first worker node pulls the first script development information from the server, and the server generates a first execution instruction; according to the first execution instruction, a data processing server on the first worker node executes the first script development information.

[0008] On the other hand, this application provides a scheduling system script synchronization and execution system, the system comprising: a first obtaining unit, the first obtaining unit being used to obtain a first scheduling system, the first scheduling system including a server and a worker; a second obtaining unit, the second obtaining unit being used to obtain first script development information submitted by a data development user according to the server, and to configure the first script development information using Git resources to obtain first configuration information; a first generating unit, the first generating unit being used to transmit the first script development information to a first worker node based on ZooKeeper, and to generate a first change instruction, wherein the ZooKeeper is communicatively linked with the server, and the first worker node is included in the worker; a first processing unit, the first processing unit being used to, according to the first change instruction, have the first worker node pull the first script development information from the server, and the server generate a first execution instruction; and a first execution unit, the first execution unit being used to, according to the first execution instruction, have a data processing server on the first worker node execute the first script development information.

[0009] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in any of the first aspects.

[0010] Fourthly, this application provides a computer-readable storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the method described in any one of the first aspects.

[0011] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0012] By employing a technical solution that divides the task scheduling system into a server and a worker side, the server retrieves script development information provided by developers and configures resources in the Git repository based on this information. The ZooKeeper program is used to obtain the worker nodes corresponding to the changed script development information. Based on these changes, the worker nodes pull the script development information from the server, and the task is executed according to the resource configuration results and the retrieved script development information. This approach ensures that the script and its referenced code are the latest versions. Furthermore, due to the existence of the Git repository, a rollback can simultaneously revert the script and its referenced code.

[0013] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0014] Figure 1 This application provides a schematic diagram of a scheduling system script synchronization and execution method.

[0015] Figure 2 This application provides a schematic diagram of the update status retrieval method in the script synchronization and execution process of a scheduling system;

[0016] Figure 3 This application provides a schematic diagram of the file synchronization method before and after an abnormal working state occurs in the script synchronization and execution method of the scheduling system;

[0017] Figure 4 This application provides a schematic diagram of the architecture of a scheduling system script synchronization and execution system;

[0018] Figure 5 This is a schematic diagram of the structure of an exemplary electronic device of this application.

[0019] Explanation of reference numerals in the attached drawings: First obtaining unit 11, Second obtaining unit 12, First generating unit 13, First processing unit 14, First execution unit 15, Electronic device 300, Memory 301, Processor 302, Communication interface 303, Bus architecture 304. Detailed Implementation

[0020] This application provides a method for script synchronization and execution in a scheduling system, solving the technical problems of existing technologies that only consider task scheduling, task dependencies, and script execution, resulting in the inability to guarantee that both the script and its referenced scripts are the latest versions simultaneously, and the inability to roll back referenced scripts when task scheduling fails. By dividing the task scheduling system into a server and worker side, the server retrieves script development information provided by developers and configures resources in the Git repository based on this information. The ZooKeeper program obtains the worker nodes corresponding to the changed script development information. Based on the changes, the worker nodes pull the script development information from the server, and then execute tasks according to the resource configuration results and the pulled script development information. This achieves the technical effect of ensuring that the script and its referenced code are the latest versions, and that due to the existence of the Git repository, both the script and its referenced scripts can be rolled back simultaneously during a rollback.

[0021] Application Overview

[0022] With the continuous development of computer technology and the deepening of enterprise digitalization, automated methods for data processing are also maturing. Using task scheduling combined with script development to achieve automated data processing is a commonly used approach. However, existing technologies only consider task scheduling, task dependencies, and script execution, which leads to technical problems such as the inability to guarantee that the script and the referenced script code are both the latest versions, and the inability to roll back referenced scripts when task scheduling fails.

[0023] To address the aforementioned technical problems, the overall approach of the technical solution provided in this application is as follows:

[0024] This application provides a method for synchronizing and executing scripts in a scheduling system. The method includes: obtaining a first scheduling system, the first scheduling system including a server and a worker; obtaining first script development information submitted by a data development user based on the server, and configuring Git resources on the first script development information to obtain first configuration information; transmitting the first script development information to a first worker node based on ZooKeeper, and generating a first change instruction, wherein the ZooKeeper is communicatively linked to the server, and the first worker node is included in the worker; according to the first change instruction, the first worker node pulls the first script development information from the server, and the server generates a first execution instruction; according to the first execution instruction, the data processing server on the first worker node executes the first script development information.

[0025] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0026] Example 1

[0027] like Figure 1 As shown, this application provides a method for synchronizing and executing scripts in a scheduling system, wherein the method includes:

[0028] S100: Obtain a first scheduling system, which includes a server and a worker;

[0029] Specifically, the first scheduling system refers to a task scheduling system built for data processing, including:

[0030] The server side refers to the module that provides script and scheduling process management. The server side interacts with data developers and can obtain the latest version of the script and script code information from them. The server side also interacts with the Git repository and can configure the resources required for the script execution process.

[0031] The worker module refers to the module deployed on each data processing server, responsible for executing instructions issued by the server. The server is responsible for updating the Git service code and notifying worker nodes within the worker module of changed files via ZooKeeper. The worker nodes then pull the updated script code from the server and can use the latest version of the script to execute tasks. Multiple worker modules, relative to a single server, execute different tasks, while the server can schedule various tasks for execution.

[0032] Furthermore, Git repository refers to an open-source distributed version control system that can efficiently handle the management of various versions of scripts and script code of different sizes. When the task scheduling of one version of the script or script code fails, it can be rolled back to other versions of the script or script code for task scheduling. ZooKeeper refers to the management of task configuration items in a distributed service framework. It can listen for and notify the corresponding worker nodes when scripts change.

[0033] In summary, this achieves the technical effect of simultaneously executing tasks with the latest version of the script and script code, and simultaneously rolling back the script and script code when task scheduling fails.

[0034] S200: Based on the server, obtain the first script development information submitted by the data development user, and configure the first script development information using Git resources to obtain the first configuration information;

[0035] Furthermore, the method step S200 includes:

[0036] S210: Build a git repository, wherein the git repository includes webhook configuration information and resource type configuration information;

[0037] S220: Based on the resource type configuration information, configure the address and SSH public key of the git repository to generate the first configuration information;

[0038] S230: Configure the storage path for the data processing server according to the resource type configuration information.

[0039] Specifically, the first script development information refers to the latest version of script development data submitted by the data development user. After the data developer develops the script locally or on the system and submits it to the git repository, the server is notified of the script code update information through git webhook. Here, git webhook refers to a trigger that occurs when an operation is performed on the git repository, thereby achieving the purpose of notifying the server to update the script code.

[0040] Furthermore, after receiving the first script development information in the Git repository, resource configuration is performed on the first script development information in the Git repository. The configuration content includes, but is not limited to, webhook configuration information and resource type configuration information. For example, resource type configuration information includes address configuration, SSH public key configuration, and storage path configuration. The SSH public key refers to the encrypted data transmission process; therefore, the Git repository needs to generate an SSH public key for authentication. Address configuration refers to configuring the Git repository address for data processing of the first script information.

[0041] The address configuration and SSH public key configuration are stored as the first configuration information; the server-side script code update information is notified via webhook; and the storage path for the updated script's data processing information is configured. This facilitates quick retrieval of the configuration information during subsequent task scheduling.

[0042] S300: Based on ZooKeeper, the first script development information is transmitted to the first worker node to generate a first change instruction, wherein the ZooKeeper is connected to the server and the first worker node is included in the worker.

[0043] Specifically, the ZooKeeper program can monitor changes to configuration items in the Git repository. Different task execution flows correspond to different nodes within the worker nodes executing the script tasks, along with their corresponding script code information. When ZooKeeper detects a change in the configuration information of the first script development information, the server uses `git pull` to update the code and writes the script files that need updating to the ZooKeeper path registered on each worker node. ZooKeeper then notifies the corresponding first worker node, which generates a first change command. This command controls the first worker node to update the configuration information of the first script development information, ensuring that the scripts, script code, and configuration information are all up-to-date. The `git pull` command can retrieve the latest code from both local and remote branches of the Git repository.

[0044] S400: According to the first change instruction, the first worker node pulls the first script development information from the server, and the server generates the first execution instruction;

[0045] S500: According to the first execution instruction, the data processing server on the first worker node executes the first script development information.

[0046] Specifically, upon receiving the first change instruction, the first worker node retrieves the latest first script development information and code information from the server based on the ZooKeeper path registered to each worker node, and deletes the file information under the corresponding worker node path in ZooKeeper, thus indicating that the first script development information update is complete. Further, according to the first execution instruction, the data processing server on the first worker node executes the first script development information, completing task scheduling. This achieves the technical effect of ensuring both the script and its referenced script code information are up-to-date.

[0047] Furthermore, the method step S200 includes:

[0048] S240: After the data development user submits the first script development information, a webhook code update reminder instruction for the first script development information is generated according to the webhook configuration information;

[0049] Specifically, once the update dynamics of the script configuration information in the git repository are obtained through the git webhook, the webhook code update reminder instruction is generated to notify the server of the script code update.

[0050] S250: Based on the webhook code update reminder instruction, remind the server to perform a configuration update.

[0051] Furthermore, the method step S250 includes:

[0052] S251: Obtain local code branch information and Git remote code branch information;

[0053] S252: Call the git diff command to compare the local code branch information and the git remote code branch information to obtain the currently updated file;

[0054] S253: Enter the currently updated file into the database.

[0055] Specifically, after receiving the webhook code update notification command, the server uses the command `git diff {branch name}origin / {branch name}` to compare the local code branch with the remote Git branch. It identifies the updated file and records its current version information in the database, facilitating version rollback in case of task scheduling failure. Both `git diff` and `origin` are commands used to compare local and remote Git branches. By comparing the local and remote branches, the server obtains the necessary updates and updates the referenced script code, ensuring the script is the latest version. Simultaneously, the database records the script code content before and after the update, enabling version rollback of both the script and its contents in case of task scheduling failure. This achieves the technical effect of ensuring both the script and its contents are the latest versions and allowing for simultaneous rollback.

[0056] Furthermore, such as Figure 2 As shown, the execution of the first script development information based on the data processing server on the first worker node includes step S600:

[0057] S610: Search the historical synchronization files of ZooKeeper to determine whether there are any files that have not been synchronized.

[0058] S620: If the unsynchronized file does not exist, the first worker node pulls the first script development information from the server and switches to the directory where the first script development information is located to execute the first script development information.

[0059] Specifically, before executing the updated first script development information, it's necessary to check if there are any unsynchronized files in the ZooKeeper historical synchronization files. Since the first worker node retrieves the latest first script development information and code from the server and then deletes the files in the corresponding ZooKeeper worker node path after updating, signifying a complete update, by iterating through ZooKeeper, if any undeletable files exist in the corresponding ZooKeeper worker node path, the update is incomplete and continues; otherwise, if no such files exist, the update is complete, and development debugging and task execution proceed. By retrieving the ZooKeeper historical synchronization files, it ensures that the updated first script development information on the first worker node is the latest version.

[0060] Furthermore, such as Figure 3 As shown, the method includes step S700:

[0061] S710: Determine whether the data processing server is operating abnormally;

[0062] S720: If the data processing server has the abnormal working condition, generate a first comparison instruction;

[0063] S730: According to the first comparison instruction, compare the executable file after the exception with the executable file before the exception, and generate a first comparison result;

[0064] S740: Based on the first comparison result, if there are differences in the executable file after the anomaly, generate a first synchronization instruction;

[0065] S750: According to the first synchronization instruction, the first script development information is fully synchronized to the data processing server.

[0066] Specifically, when the data processing server experiences abnormal operation, such as a crash that interrupts task execution, upon restarting, it is necessary to compare the files before and after the abnormal operation and synchronize the first script development information based on the differences.

[0067] The first comparison instruction refers to the instruction generated after the data processing server restarts when there is an abnormal working condition, which calls the file before and after the abnormal working condition for comparison; the first comparison result refers to the result obtained by comparing the file before and after the abnormal working condition. An exemplary comparison method is: calling the updated first script development information from the database, calling the first script development information after restarting from the data processing server, and using the interface to compare whether the checksum file is the same, where the checksum is a verification algorithm for data integrity and accuracy.

[0068] Furthermore, when the data processing server experiences abnormal operation and there are discrepancies between the files before and after, the first script development information called from the database is fully synchronized to the data processing server based on the first synchronization instruction, ensuring the integrity and accuracy of the first script development information.

[0069] In summary, the scheduling system script synchronization and execution method provided in this application has the following technical effects:

[0070] 1. By dividing the task scheduling system into a server and a worker, the server retrieves script development information provided by the developers and configures resources in the Git repository based on this information. The ZooKeeper program obtains the worker nodes that have changed according to the script development information. Based on the changes, the worker nodes pull the script development information from the server and then execute the task according to the resource configuration results and the pulled script development information. This achieves the technical effect of ensuring that the script and the script's referenced code are the latest versions, and that due to the existence of the Git repository, the script and the referenced script code can be rolled back simultaneously during rollback.

[0071] 2. By comparing the local code branch with the remote Git code branch, the code content that needs to be updated is obtained and the referenced script code is updated to ensure that the script code is the latest version. At the same time, the database records the script code content before and after the update, which facilitates version rollback including the script code in the event of task scheduling failure. This achieves the technical effect of ensuring that the script code and script are the latest versions and that the script and script code can be rolled back at the same time.

[0072] 3. By retrieving historical synchronization files from ZooKeeper, ensure that the first script development information updated on the first worker node is not the latest version of the data.

[0073] 4. When the data processing server malfunctions and there are discrepancies between the files before and after, the first script development information called from the database is fully synchronized to the data processing server based on the first synchronization instruction, ensuring the integrity and accuracy of the first script development information.

[0074] Example 2

[0075] Based on the same inventive concept as the scheduling system script synchronization and execution method in the foregoing embodiments, such as Figure 4 As shown, this application provides a scheduling system script synchronization and execution system, wherein the system includes:

[0076] The first obtaining unit 11 is used to obtain a first scheduling system, which includes a server and a worker.

[0077] The second obtaining unit 12 is used to obtain the first script development information submitted by the data development user according to the server, and to configure the first script development information with git resources to obtain the first configuration information.

[0078] The first generation unit 13 is used to transmit the first script development information to the first worker node based on ZooKeeper and generate a first change instruction, wherein ZooKeeper is connected to the server and the first worker node is included in the worker.

[0079] The first processing unit 14 is configured to, according to the first change instruction, have the first worker node pull the first script development information from the server, and have the server generate a first execution instruction.

[0080] The first execution unit 15 is used to execute the first script development information on the data processing server on the first worker node according to the first execution instruction.

[0081] Furthermore, the system also includes:

[0082] The first building unit is used to build a Git repository, wherein the Git repository includes webhook configuration information and resource type configuration information;

[0083] The second generation unit is used to configure the address and SSH public key of the git repository according to the resource type configuration information, and generate the first configuration information.

[0084] The first configuration unit is used to configure the storage path of the data processing server according to the resource type configuration information.

[0085] Furthermore, the system also includes:

[0086] The third generation unit is used to generate a webhook code update reminder instruction for the first script development information based on the webhook configuration information after the data development user submits the first script development information.

[0087] The first reminder unit is used to remind the server to update the configuration based on the webhook code update reminder instruction.

[0088] Furthermore, the system also includes:

[0089] The third obtaining unit is used to obtain local code branch information and git remote code branch information;

[0090] The fourth obtaining unit is used to call the git diff command to compare the local code branch information and the git remote code branch information to obtain the currently updated file;

[0091] The first input unit is used to input the currently updated file into the database.

[0092] Furthermore, the system also includes:

[0093] The first judgment unit is used to search the historical synchronization files of the ZooKeeper and determine whether there are any files that have not been synchronized.

[0094] The second execution unit is used to, if the unsynchronized file does not exist, have the first worker node pull the first script development information from the server and switch to the directory where the first script development information is located to execute the first script development information.

[0095] Furthermore, the system also includes:

[0096] The second judgment unit is used to determine whether the data processing server is operating abnormally.

[0097] The fourth generation unit is used to generate a first comparison instruction if the data processing server has the abnormal working condition.

[0098] The fifth generation unit is used to compare the executable file after the exception with the executable file before the exception according to the first comparison instruction, and generate a first comparison result;

[0099] The sixth generation unit is used to generate a first synchronization instruction based on the first comparison result if there are differences in the executable file after the anomaly.

[0100] The third execution unit is used to fully synchronize the first script development information to the data processing server according to the first synchronization instruction.

[0101] Exemplary electronic devices

[0102] The following is for reference. Figure 5 To describe the electronic device of this application.

[0103] Based on the same inventive concept as the scheduling system script synchronization and execution method in the foregoing embodiments, this application also provides an electronic device, including: a processor coupled to a memory for storing a program, wherein when the program is executed by the processor, the system performs the method described in any one of the embodiments.

[0104] The electronic device 300 includes a processor 302, a communication interface 303, and a memory 301. Optionally, the electronic device 300 may also include a bus architecture 304. The communication interface 303, processor 302, and memory 301 can be interconnected via the bus architecture 304; the bus architecture 304 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus architecture 304 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0105] Processor 302 may be a CPU, microprocessor, ASIC, or one or more integrated circuits used to control the execution of programs according to the present application.

[0106] Communication interface 303 is used in any transceiver system for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), wired access network, etc.

[0107] Memory 301 can be ROM or other types of static storage devices capable of storing static information and instructions, RAM or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory can exist independently and be connected to the processor via bus architecture 304. Memory can also be integrated with the processor.

[0108] The memory 301 stores computer execution instructions for implementing the scheme of this application, and the processor 302 controls the execution. The processor 302 executes the computer execution instructions stored in the memory 301, thereby realizing the scheduling system script synchronization and execution method provided in the above embodiments of this application.

[0109] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the embodiments.

[0110] Optionally, the computer execution instructions in this application may also be referred to as application code, and this application does not specifically limit them.

[0111] This application divides the task scheduling system into a server and a worker. On the server, it retrieves script development information provided by the developers and configures resources in the Git repository based on the script development information. The ZooKeeper program obtains the worker nodes that have changed corresponding to the script development information. Based on the change information, the worker nodes pull the script development information from the server. Then, based on the resource configuration results and the pulled script development information, the task is executed. This achieves the technical effect of ensuring that the script and the script's referenced code are the latest versions. Moreover, due to the existence of the Git repository, the script and the referenced script code can be rolled back simultaneously during rollback.

[0112] Those skilled in the art will understand that the various numerical designations, such as "first," "second," etc., used in this application are merely for descriptive convenience and are not intended to limit the scope of this application, nor do they indicate a chronological order. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" refers to one or more. "At least two" refers to two or more. "At least one," "any one," or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0113] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0114] The various illustrative logic units and circuits described in this application can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic system, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing systems, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0115] The steps of the methods or algorithms described in this application can be directly embedded in hardware, a software unit executed by a processor, or a combination of both. The software unit can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other storage medium of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be disposed in an ASIC, which can be disposed in a terminal. Optionally, the processor and storage medium can also be disposed in different components within the terminal. These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of the application as defined herein, and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. A method for scheduling system script synchronization and execution, the method comprising: The method comprises: obtaining a first scheduling system, the first scheduling system comprising a server end and a worker end; obtaining first script development information submitted by a data development user according to the server end, and performing git resource configuration on the first script development information to obtain first configuration information, the first script development information referring to the latest version of script development data submitted by the data development user; transmitting the first script development information to a first worker node based on a zookeeper, generating a first change instruction at the first worker node when the zookeeper listens to a change in the first configuration information of the first script development information, wherein the zookeeper is in communication connection with the server end, and the first worker node is included in the worker end; pulling the latest first script development information under the server end zookeeper path according to the first change instruction, and generating a first execution instruction by the server end; executing the first script development information by a data processing server on the first worker node according to the first execution instruction.

2. The method of claim 1, wherein, The method comprises: building a git repository, wherein the git repository comprises webhook configuration information and resource type configuration information; address configuration and ssh public key configuration are performed on the git repository according to the resource type configuration information to generate the first configuration information; storage path configuration is performed on the data processing server according to the resource type configuration information.

3. The method of claim 2, wherein, The method comprises: after the data development user submits the first script development information, generating a webhook code update reminder instruction of the first script development information according to the webhook configuration information; performing configuration update on the server end according to the webhook code update reminder instruction.

4. The method of claim 3, wherein, The method comprises: obtaining local code branch information and git remote code branch information; calling a git diff command to compare the local code branch information and the git remote code branch information to obtain a current update file; entering the current update file into a database.

5. The method of claim 1, wherein, The data processing server on the first worker node executes the first script development information, which comprises: searching the history synchronization file of the zookeeper to determine whether there is an unsynchronized file; if there is no unsynchronized file, the first worker node pulls the first script development information from the server end, and switches to the directory where the first script development information is located to execute the first script development information.

6. The method of claim 4, wherein, The method comprises: determining whether the data processing server has an abnormal working condition; if the data processing server has the abnormal working condition, generating a first comparison instruction; According to the first comparison instruction, the post-exception execution file is compared with the pre-exception execution file to generate a first comparison result; According to the first comparison result, if the post-exception execution file has a difference, a first synchronization instruction is generated; According to the first synchronization instruction, the first script development information is fully synchronized to the data processing server.

7. A dispatch system script synchronization and execution system, characterized by, The system comprises: A first obtaining unit is configured to obtain a first scheduling system, wherein the first scheduling system comprises a server end and a worker end; A second obtaining unit is configured to obtain first script development information submitted by a data development user according to the server end, and perform git resource configuration on the first script development information to obtain first configuration information, wherein the first script development information refers to the latest version of script development data submitted by the data development user; A first generating unit is configured to transmit the first script development information to a first worker node based on a zookeeper, and generate a first change instruction in the first worker node when the zookeeper listens to a change in the first configuration information of the first script development information, wherein the zookeeper is communicatively connected to the server end, and the first worker node is included in the worker end; A first processing unit is configured to pull the latest first script development information under the zookeeper path of the server end according to the first change instruction, and generate a first execution instruction in the server end; A first execution unit is configured to execute the first script development information in a data processing server on the first worker node according to the first execution instruction.

8. An electronic device comprising: A processor is coupled to a memory, and the memory is configured to store a program, wherein when the program is executed by the processor, the system performs the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images