A method and device for synchronizing job scheduling and intervention of a data processing platform

Abstract

The application discloses a kind of data processing platform job scheduling and intervention synchronous method, comprising: setting lock service, delay message queue service and cache service;Predefine the type of job operation, and the mutual exclusion operation of each operation is configured;When data processing platform initiates job operation, operation message is put into delay message queue;Listen to delay message queue, and obtain lock when listening to operation message;Mutual exclusion operation check is carried out to current job operation: if mutual exclusion operation is being executed, message is set to delay preset time length and is put into queue again, and lock is released;Otherwise, current job operation is registered to cache, lock is released, and current job operation is executed;When current job operation is executed, lock is obtained again, current job operation is deleted, and lock is released again.The method shortens lock holding and waiting time, improves system concurrency, and can effectively control retry time interval, to avoid long transaction resource occupation.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and specifically relates to a synchronization method and apparatus for job scheduling and intervention in a data processing platform. Background Technology

[0002] A data processing platform is a system integrating data acquisition, storage, processing, analysis, and reporting. Its core includes a job scheduling system, used to schedule and execute Shell scripts, SQL statements, Python programs, and big data processing jobs using Spark, Flink, and other technologies. Job scheduling is performed automatically by the job scheduling system according to a preset cycle. Job intervention involves manual intervention to pause, terminate, or rerun jobs in the schedule. Both types of operations typically change the job status and are executed non-serially, which can easily lead to inconsistencies between the actual job status and the system-recorded status in concurrent scenarios.

[0003] For example, a job program might be starting up, and the process ID might have been updated, but the job status hasn't been returned to the main scheduling process yet. If a user initiates a termination operation at this point, the process will be killed. However, because the job started successfully, the main process will set the job status to "running," even though the process has actually been killed. This results in an inconsistency in job status. Another example is when a user initiates a kill operation before the job program fails. The kill operation detects the job's failure and considers it successful, changing the job status to "kill successful" and returning it to the user. However, the scheduling monitoring service also detects the job failure and changes the job status to "failed and retrying," resulting in a "failed and retrying" job status. Yet, the user still believes the job was successfully killed, leading to another inconsistency in job status.

[0004] To ensure job status consistency, existing technologies typically employ two synchronization control schemes: The first is a synchronization scheme that uses database status for pre-verification. Specifically, when updating the job status in the database, a `where status in()` statement is added to the update statement, listing the allowed pre-verified statuses within the `in` block. When two operations occur simultaneously, one will modify the database status first, but will not immediately commit the transaction. If the modification succeeds, the job operation continues, and the transaction is committed only after the operation is complete. If the second operation attempts to modify the status and finds it no longer meets the allowed pre-verified status, the operation will fail. The second is a synchronization scheme that locks the entire job operation process. Specifically, for each job operation, including those automatically scheduled by the system, the entire operation process is locked. The locking period includes both the database status modification and the job operation itself, and the lock is released only after the entire operation is completed.

[0005] However, the existing solutions described above have significant drawbacks. First, synchronizing via database state requires modifying the database state before execution and maintaining an uncommitted transaction until the actual operation is complete. This approach results in a long transaction retention time, leading to extended connection time and impacting database concurrency. Furthermore, controlling the retry interval after a failed state modification is difficult; immediately adding the data back to the queue might result in immediate retrying, leading to further failures and multiple retries. Second, locking the entire job operation process, which involves both database state modification and the job operation itself, can be time-consuming. This results in longer waiting times for later-initiated operations, consuming more thread time and impacting the scheduling service's concurrency. Initiating retries for failed operations is also inconvenient.

[0006] In summary, how to shorten lock holding and waiting time, improve system concurrency, and effectively control retry intervals when operation conflicts occur, while ensuring synchronization between job scheduling and manual intervention, has become an urgent technical problem to be solved in the field of job scheduling for data processing platforms. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a synchronization method and apparatus for job scheduling and intervention in a data processing platform. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, this invention proposes a synchronization method for job scheduling and intervention in a data processing platform, wherein the data processing platform supports both distributed and non-distributed environments, and the method includes the following steps: Configure lock service, delayed message queue service, and cache service; predefine job operation types and configure mutual exclusion operations for each operation; when the data processing platform initiates a job operation, call the delayed message queue service and put the job operation message into the delayed message queue; The scheduling service program listens to the delayed message queue, and when a job operation message is detected, it calls the lock service to acquire the lock for the current job. Call the cache service and perform a mutual exclusion operation check on the current job operation: If a mutual exclusion operation for the current job operation is being executed in the cache, set the job operation message to a preset delay time and put it back into the delayed message queue, while releasing the lock of the current job; if no mutual exclusion operation for the current job operation is being executed in the cache, register the current job operation in the cache, release the lock of the current job, and execute the current job operation. After the current job operation is completed, the lock service is invoked to reacquire the lock for the current job; at the same time, the cache service is invoked to delete the current job operation, and then the lock for the current job is released.

[0008] Secondly, this invention proposes a synchronization device for job scheduling and intervention in a data processing platform, used in the method proposed in the first aspect of this invention. The data processing platform supports both distributed and non-distributed environments. The device includes a lock service module, a delayed message queue service module, a cache service module, and a scheduling operation synchronization processing module. The scheduling operation synchronization processing module is communicatively connected to the lock service module, the delayed message queue service module, and the cache service module, respectively. The lock service module is used to provide job-level locking functionality; The delayed message queue service module is used to provide message delay functionality; The caching service module is used to cache currently executing job operations; The scheduling operation synchronization processing module is configured with the types of job operations and the mutual exclusion operations corresponding to each operation, and is used to implement the following steps: When the data processing platform initiates a job operation, it calls the delayed message queue service and puts the job operation message into the delayed message queue. The scheduling service program listens to the delayed message queue, and when a job operation message is detected, it calls the lock service to acquire the lock for the current job. Call the cache service and perform a mutual exclusion operation check on the current job operation: If a mutual exclusion operation for the current job operation is being executed in the cache, set the job operation message to a preset delay time and put it back into the delayed message queue, while releasing the lock of the current job; if no mutual exclusion operation for the current job operation is being executed in the cache, register the current job operation in the cache, release the lock of the current job, and execute the current job operation. After the current job operation is completed, the lock service is invoked to reacquire the lock for the current job; at the same time, the cache service is invoked to delete the current job operation, and then the lock for the current job is released.

[0009] Thirdly, the present invention proposes an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is configured to execute a program stored in memory to implement the method provided in the first aspect of the present invention.

[0010] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0011] The beneficial effects of this invention are: The synchronization method for job scheduling and intervention in a data processing platform provided by this invention pre-sets a lock service, a delayed message queue service, and a cache service, and defines the types of job operations and the mutual exclusion operations for each operation. When the data processing platform initiates a job operation, it puts the job operation message into the delayed message queue. When a job operation message is detected, the lock for the job is acquired first, and then a mutual exclusion operation check is performed. If a mutual exclusion operation is being executed, the job operation message is delayed, put back into the delayed message queue, and then the lock is released. If no mutual exclusion operation is being executed, the current operation is added to the cache, and then the lock is released. After the current job operation is completed, the lock is reacquired, the current job operation is deleted from the cache, and finally the lock is released. This method, by combining lightweight locking with delayed retries, significantly shortens the lock holding and waiting time, and significantly improves the system's concurrency capability. Database transactions can be committed immediately, avoiding long transactions occupying resources and ensuring accurate and consistent job states. It also supports distributed and single-process environments, and has strong versatility and high stability.

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0013] Figure 1 A flowchart illustrating a synchronization method for job scheduling and intervention in a data processing platform, provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the processing flow of a synchronization method for job scheduling and intervention in a data processing platform provided in an embodiment of the present invention when there are mutually exclusive operations in a distributed environment; Figure 3 A schematic diagram of the processing flow of a synchronization method for job scheduling and intervention in a data processing platform provided by an embodiment of the present invention when there are no mutually exclusive operations in a distributed environment; Figure 4 This is a structural block diagram of a synchronization device for job scheduling and intervention in a data processing platform, provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Before introducing the specific solution of the present invention, the four technical terms involved, namely data processing platform, operation, scheduling, and intervention, will be explained first.

[0016] A data processing platform is a system or tool that integrates data acquisition, storage, processing, analysis, and reporting. This platform typically includes a job scheduling system that supports the scheduling and execution of various data processing programs, such as executable programs developed and compiled in C / Java, shell / Python scripts, or big data processing programs such as Spark / Flink.

[0017] A job is a set of predefined operations scheduled and executed automatically by a task scheduling system to complete a specific data processing task. Common forms include shell scripts, SQL statements, Python programs, or task scripts of data processing frameworks (such as Spark and Flink).

[0018] Scheduling refers to the process of starting and executing job programs or scripts within the planned time frame according to the planned execution time and cycle configured for the job, and monitoring the execution status.

[0019] Intervention refers to human intervention in the scheduling process, such as pausing, terminating, or rerunning the operation.

[0020] The first aspect of this invention provides a synchronization method for job scheduling and intervention in a data processing platform, wherein the data processing platform supports distributed environments, non-distributed environments, or other single-process environments. For details, please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a synchronization method for job scheduling and intervention in a data processing platform provided by an embodiment of the present invention. The method mainly includes four core steps: algorithm initialization, message listening and locking, mutual exclusion operation verification and operation execution, and operation completion cleanup, which specifically correspond to the contents of steps 1 to 4.

[0021] Step 1, Algorithm Initialization: Set up lock service, delayed message queue service, and cache service; predefine the types of job operations and configure mutual exclusion operations for each operation; when the data processing platform initiates a job operation, it calls the delayed message queue service and puts the job operation message into the delayed message queue.

[0022] The method provided by this invention mainly relies on the joint implementation of several core components: a lock service, a delayed message queue service, a cache service, and a job scheduling operation synchronization processing module (hereinafter referred to as the scheduling processing module). Therefore, it is necessary to initialize these core components.

[0023] Specifically, firstly, a lock service is set up, which is mainly used to provide job-level locking functionality to enable locking and unlocking operations for jobs.

[0024] Then, a delayed message queue service is set up to provide message delay functionality, thereby controlling the retry time of operations and preventing messages from being retried too quickly, resulting in continuous invalid retries. This delayed message queue service effectively controls the time interval for job operation retry by controlling both immediate and delayed message delivery.

[0025] Next, a caching service is set up to cache the job operations being executed and to implement the functions of registering, querying and deleting job operation information.

[0026] Finally, the scheduling module defines the types of job operations and configures mutual exclusion operations for each operation.

[0027] In this embodiment, the types of job operations can include creation, start, pause, terminate, kill, rerun, status monitoring, and corresponding database operation transactions. Simultaneously, mutually exclusive operations that cannot be executed concurrently within the same job and would cause state conflicts can be configured; for example, creation and kill operations are mutually exclusive. Furthermore, parallel operations that can be executed concurrently within the same job without causing state conflicts can also be configured; for example, status monitoring operations can be executed in parallel with both creation and kill operations.

[0028] This completes the initialization of the algorithm's trusted module. When the data processing platform initiates a job operation, it calls the delayed message queue service and places the job operation message into the delayed message queue. At this point, the delay time can be set to 0 seconds, meaning there is no delay.

[0029] It is understood that the method provided by this invention can be implemented in data processing platforms in both distributed and non-distributed environments.

[0030] Optionally, as an implementation approach, when the data processing platform is a non-distributed environment, or when job operations are executed synchronously within the same process, the lock service is implemented using an in-process local lock, such as Java's ReentrantLock; the delayed message queue service is implemented using an in-process local delayed message queue, such as Java Delayed Message Queues; and the caching service is implemented using an in-process local cache, such as a memory Map or a local cache container.

[0031] Optionally, as another implementation, when the data processing platform is a distributed environment, the lock service adopts a distributed lock implementation based on ZooKeeper. ZooKeeper is a distributed, open-source distributed application coordination service that provides a distributed lock implementation mechanism, ensuring the orderly contention and acquisition of locks by multiple nodes in a distributed environment. The delayed message queue service adopts a distributed delayed message queue implementation based on RocketMQ. RocketMQ supports delayed message sending, and the message delay time can be set as needed to meet the requirements of operation retry time control in this invention. The caching service adopts a distributed cache implementation based on Redis. Redis is an open-source, in-memory data structure storage system that can be used as a database, cache, and message middleware. It features high performance and high concurrency, and can quickly store and retrieve operation information of jobs being executed.

[0032] Step 2, Message Listening and Locking: Use the scheduling service program to listen to the delayed message queue, and when a job operation message is detected, call the lock service to acquire the lock for the current job.

[0033] It should be noted that the method provided by this invention maintains the same flow across the three core steps—message listening and locking, mutual exclusion operation verification and operation execution, and operation completion cleanup—regardless of whether it uses local locks, local delayed message queues, and local caching in a non-distributed environment or distributed locks, distributed delayed message queues, and distributed caching in a distributed environment. This embodiment primarily uses a distributed environment as an example to illustrate the proposed synchronization method for job scheduling and intervention.

[0034] Specifically, after the scheduling service program starts, it begins to listen to the delayed message queue. When a job operation message is generated, it calls the locking function of the lock service to try to acquire the lock for the job. After successfully acquiring the lock, it executes the subsequent mutual exclusion operation verification steps.

[0035] Step 3, Mutual Exclusion Validation and Execution: Call the cache service and perform a mutual exclusion check on the current job operation: If a mutual exclusion operation for the current job operation is being executed in the cache, set the job operation message to a preset delay time and put it back into the delayed message queue, while releasing the lock of the current job; if no mutual exclusion operation for the current job operation is being executed in the cache, register the current job operation in the cache, release the lock of the current job, and execute the current job operation.

[0036] Specifically, taking a distributed environment as an example, please refer to [link / reference]. Figure 2 , Figure 2This invention provides a schematic diagram of the processing flow of a synchronization method for job scheduling and intervention in a data processing platform when mutual exclusion operations exist in a distributed environment. The complete flow can be described as follows: Before the algorithm begins, the scheduling module has configured mutual exclusion operations for job operations. First, the scheduling module obtains the job operation message, and upon successful acquisition, calls the distributed lock's locking service to acquire the lock for the current job. Then, it checks whether a mutual exclusion operation exists in the distributed cache. If it does, it sets the message to a 1-second delay (or another suitable duration) and sends it to the delayed message queue, and then releases the lock for the current job.

[0037] For further details, please see Figure 3 , Figure 3 This invention provides a schematic diagram of the processing flow of a synchronization method for job scheduling and intervention in a data processing platform under a distributed environment without mutual exclusion operations. The complete flow can be described as follows: Before the algorithm begins, the scheduling module has configured mutual exclusion operations for job operations. First, the scheduling module retrieves the job operation message and, upon successful retrieval, calls the distributed lock's locking service to acquire the lock for the current job. Then, it checks the distributed cache for a mutual exclusion operation; if none exists, it registers the current job operation in the distributed cache and releases the job lock. Next, the scheduling module executes the current job operation.

[0038] It should be noted that the lock service is only used for locking during the mutual exclusion operation check, job operation message registration, job operation deletion, job operation message delay setting, and re-entry queuing phases, and is not used during the job operation execution phase.

[0039] Furthermore, database operations can be committed immediately while the current job is being executed, without needing to keep the transactions uncommitted until the current job is completed.

[0040] Since there are no mutually exclusive operations, subsequent logical processing can promptly commit database operations to transactions, thereby avoiding prolonged occupation of database transaction resources.

[0041] Step 4: Operation Completion Cleanup: After the current job operation is completed, call the lock service to reacquire the lock for the current job; at the same time, call the cache service to delete the current job operation, and then release the lock for the current job.

[0042] Specifically, in a distributed environment, such as Figure 4 As shown, after execution, the distributed lock's locking service is invoked to acquire the lock for the current job; then, the registered current job operation is deleted from the distributed cache, and finally, the job lock is released. The processing flow in a non-distributed environment is the same as that in a distributed environment.

[0043] This achieves simultaneous processing of job scheduling and intervention.

[0044] This invention, through the collaborative design of job-level lock service, delayed message queue service, caching service, and scheduling operation synchronization processing, not only achieves reliable synchronization between job scheduling and manual intervention operations, but also fundamentally overcomes the shortcomings of existing technologies, possessing the following outstanding beneficial effects: 1. Significantly shortens lock holding and waiting time, greatly improving system concurrency capabilities.

[0045] This invention only locks lightweight, short-term operations such as mutual exclusion checks, operation registration or deletion, message delay settings, and re-entry queuing. No locks are applied during the actual execution of the job, and the lock holding time is extremely short, so it will not block threads. This effectively avoids the problem of excessive lock waiting time caused by full-process locking in existing technologies, and significantly improves the overall concurrent processing capability of the scheduling service.

[0046] 2. Reduce database resource consumption and improve database concurrency performance.

[0047] This invention can execute job operations after completing mutual exclusion verification, and can commit database transactions in real time when required according to logic, without having to keep transactions in an uncommitted state for a long time as in existing technologies. This avoids occupying database transaction resources for a long time, greatly reduces database resource consumption, reduces the impact of long transactions on database concurrency capabilities, and improves system stability.

[0048] 3. Precisely control the retry interval to avoid unnecessary and frequent retries.

[0049] This invention uses a delayed message queue to retry conflicting operations with a fixed time delay. This effectively avoids the waste of CPU and queue resources caused by immediate retries after an operation fails, making the retry mechanism more reasonable and efficient, and reducing unnecessary system overhead.

[0050] 4. It has strong environmental adaptability and flexible deployment.

[0051] This invention can be freely adapted between distributed and non-distributed or single-process environments, keeping the core process unchanged. Only the underlying locks, queues, and caches need to be replaced, without modifying the business logic. It is suitable for data processing platforms of different scales and deployment architectures, and has strong versatility.

[0052] In summary, the synchronization method for job scheduling and intervention in a data processing platform provided by this invention pre-sets a lock service, a delayed message queue service, and a cache service, and defines the types of job operations and the mutual exclusion operations for each operation. When the data processing platform initiates a job operation, it puts the job operation message into the delayed message queue. When a job operation message is detected, the lock for the job is acquired first, and then a mutual exclusion operation check is performed. If a mutual exclusion operation is being executed, the job operation message is delayed, put back into the delayed message queue, and then the lock is released. If no mutual exclusion operation is being executed, the current operation is added to the cache, and then the lock is released. After the current job operation is completed, the lock is reacquired, the current job operation is deleted from the cache, and finally the lock is released. In this way, by combining lightweight locking with delayed retry, reliable synchronization of job scheduling and manual intervention operations is achieved, while significantly shortening the lock holding and waiting time and significantly improving the system's concurrency capability. Moreover, database transactions can be committed immediately, avoiding long transactions occupying resources and ensuring accurate and consistent job states. It also supports distributed and single-process environments, has strong versatility, and high stability.

[0053] Based on the same inventive concept, a second aspect of the present invention also provides a synchronization device for job scheduling and intervention in a data processing platform. Please see [link to related document]. Figure 4 , Figure 4 This is a structural block diagram of a synchronization device for job scheduling and intervention in a data processing platform, provided in an embodiment of the present invention. The device includes a lock service module, a delayed message queue service module, a cache service module, and a scheduling operation synchronization processing module. The scheduling operation synchronization processing module is communicatively connected to the lock service module, the delayed message queue service module, and the cache service module, respectively. The lock service module is used to provide job-level locking functionality; The delayed message queue service module is used to provide message delay functionality; The caching service module is used to cache currently executing job operations; The scheduling operation synchronization processing module is configured with the types of job operations and the mutual exclusion operations corresponding to each operation, and is used to implement the following steps: When the data processing platform initiates a job operation, it calls the delayed message queue service and puts the job operation message into the delayed message queue. The scheduling service program listens to the delayed message queue, and when a job operation message is detected, it calls the lock service to acquire the lock for the current job. Call the cache service and perform a mutual exclusion operation check on the current job operation: If a mutual exclusion operation for the current job operation is being executed in the cache, set the job operation message to a preset delay time and put it back into the delayed message queue, while releasing the lock of the current job; if no mutual exclusion operation for the current job operation is being executed in the cache, register the current job operation in the cache, release the lock of the current job, and execute the current job operation. After the current job operation is completed, the lock service is invoked to reacquire the lock for the current job; at the same time, the cache service is invoked to delete the current job operation, and then the lock for the current job is released.

[0054] Based on the same inventive concept, a third aspect of the present invention also provides an electronic device, which includes a processor, a communication interface, a memory, and a communication bus; wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory is used to store computer programs; When the processor executes a program stored in memory, it implements the method steps provided in the first aspect of the present invention.

[0055] Based on the same inventive concept, a fourth aspect of the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method steps provided in the first aspect of the present invention.

[0056] It should be noted that the embodiments of the apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple, and relevant parts can be referred to in the descriptions of the method embodiments.

[0057] The apparatus, electronic device, and storage medium in the embodiments of the present invention are respectively the apparatus, electronic device, and storage medium for applying the above-described synchronization method for job scheduling and intervention of the data processing platform. Therefore, all embodiments of the above-described synchronization method for job scheduling and intervention of the data processing platform are applicable to the apparatus, electronic device, and storage medium, and can achieve the same or similar beneficial effects.

[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (devices), or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "apparatus." Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other forms of distribution, such as via the Internet or other wired or wireless telecommunications systems.

[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for synchronizing job scheduling and intervention in a data processing platform, characterized in that, The data processing platform supports both distributed and non-distributed environments, and the method includes the following steps: Configure lock service, delayed message queue service, and cache service; predefine job operation types and configure mutual exclusion operations for each operation; when the data processing platform initiates a job operation, call the delayed message queue service and put the job operation message into the delayed message queue; The scheduling service program listens to the delayed message queue, and when the job operation message is detected, it calls the lock service to acquire the lock for the current job. The cache service is invoked, and a mutual exclusion operation check is performed on the current job operation: if a mutual exclusion operation of the current job operation is being executed in the cache, the job operation message is set to a delay of a preset duration and re-added to the delayed message queue, while the lock of the current job is released; if no mutual exclusion operation of the current job operation is being executed in the cache, the current job operation is registered in the cache, the lock of the current job is released, and the current job operation is executed. After the current job operation is completed, the lock service is invoked to reacquire the lock for the current job; at the same time, the cache service is invoked to delete the current job operation and then release the lock for the current job.

2. The synchronization method for job scheduling and intervention in a data processing platform according to claim 1, characterized in that, When the data processing platform is a non-distributed environment, or when job operations are executed synchronously within the same process, the lock service is implemented using an in-process local lock; the delayed message queue service is implemented using an in-process local delayed message queue; and the caching service is implemented using an in-process local cache.

3. The synchronization method for job scheduling and intervention in a data processing platform according to claim 1, characterized in that, When the data processing platform is a distributed environment, the lock service is implemented using a distributed lock based on ZooKeeper; the delayed message queue service is implemented using a distributed delayed message queue based on RocketMQ; and the caching service is implemented using a distributed cache based on Redis.

4. The synchronization method for job scheduling and intervention in a data processing platform according to claim 1, characterized in that, The types of job operations include creation, start, pause, termination, rerun, status monitoring, and corresponding database operation transactions.

5. The synchronization method for job scheduling and intervention in a data processing platform according to claim 1, characterized in that, The preset duration is 1 second.

6. The synchronization method for job scheduling and intervention in a data processing platform according to claim 4, characterized in that, When the current job operation is executed, the database operation transaction is committed immediately, without needing to keep the transaction in an uncommitted state until the current job operation is completed.

7. The synchronization method for job scheduling and intervention in a data processing platform according to claim 1, characterized in that, The lock service is only used for locking during the mutual exclusion operation check, job operation message registration, job operation deletion, job operation message delay setting, and re-entry queuing stages, and is not used during the job operation execution stage.

8. A synchronization device for job scheduling and intervention in a data processing platform, used to implement the method described in any one of claims 1-7, characterized in that, The data processing platform supports both distributed and non-distributed environments. The device includes a lock service module, a delayed message queue service module, a cache service module, and a scheduling operation synchronization processing module. The scheduling operation synchronization processing module is communicatively connected to the lock service module, the delayed message queue service module, and the cache service module, respectively. The lock service module is used to provide job-level lock functionality; The delayed message queue service module is used to provide message delay functionality; The caching service module is used to cache currently executing job operations; The scheduling operation synchronization processing module is configured with job operation types and mutual exclusion operations corresponding to each operation, and is used to implement the following steps: When the data processing platform initiates a job operation, it calls the delayed message queue service and puts the job operation message into the delayed message queue. The scheduling service program listens to the delayed message queue, and when the job operation message is detected, it calls the lock service to acquire the lock for the current job. The cache service is invoked, and a mutual exclusion operation check is performed on the current job operation: if a mutual exclusion operation of the current job operation is being executed in the cache, the job operation message is set to a delay of a preset duration and re-added to the delayed message queue, while the lock of the current job is released; if no mutual exclusion operation of the current job operation is being executed in the cache, the current job operation is registered in the cache, the lock of the current job is released, and the current job operation is executed. After the current job operation is completed, the lock service is invoked to reacquire the lock for the current job; at the same time, the cache service is invoked to delete the current job operation and then release the lock for the current job.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor for executing a program stored in memory to implement the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method described in any one of claims 1-7.

Images