Cross-platform node automatic scheduling method, device and equipment based on message queue

Abstract

This invention discloses a cross-platform automatic node scheduling method, apparatus, and device based on message queues. The method is applied to a three-layer execution architecture including a triggering layer, a scheduling layer, and a base layer. The method includes: receiving multiple task requests triggered by various types of CI / CD platforms through the triggering layer; constructing a task message queue based on the multiple task requests; sequentially retrieving target tasks to be executed from the task message queue through the scheduling layer; obtaining candidate nodes from a pre-constructed machine message queue; and if the status of a candidate node relative to all CI / CD platforms meets the scheduling conditions, then the scheduling layer brings the candidate node online and executes the target task through the candidate node. The technical solution of this invention can avoid resource conflicts during cross-platform node scheduling, effectively improve the utilization rate of hardware computing resources, achieve full automation of cross-platform node scheduling, and greatly reduce operation and maintenance costs.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a cross-platform automatic node scheduling method, apparatus, and device based on message queues. Background Technology

[0002] In the software development process, continuous integration (CI) and continuous delivery (CD) business pipelines require a large number of node resources for compilation and testing.

[0003] In existing technologies, task requests triggered by CI / CD platforms mainly rely on the following node management methods: clustered mode, which uses Kubernetes, OpenStack, etc. to achieve cluster resource scheduling; static node pool mode, which controls nodes to be online for a long time and is fixedly allocated to specific pipelines; single CI / CD platform exclusive mode, such as JenkinsMaster managing its own Slave nodes, or GitLab Runner running independently; and self-developed CI / CD platform static configuration mode, which involves manually configuring nodes through the enterprise's self-developed CI / CD system.

[0004] However, existing node management methods have the following drawbacks when used collaboratively across multiple CI / CD platforms: node states are out of sync between multiple open-source CI / CD instances (such as Jenkins and GitLab CI) and the self-developed CI / CD platform, leading to the same node being scheduled by multiple platforms simultaneously, and the same node being preempted by different platforms, resulting in execution errors; nodes remain online for extended periods and cannot automatically go offline when idle, consuming hardware testing resources; nodes require manual configuration on each platform after going online, and cannot be dynamically allocated according to task requirements; when the number of tasks exceeds the number of available nodes, there is a lack of effective queuing and scheduling mechanisms; nodes appear online on the first type of CI / CD platform (such as Jenkins) but offline on the second type of CI / CD platform (such as the self-developed platform), leading to scheduling chaos. Summary of the Invention

[0005] This invention provides a method, apparatus, and device for automatic cross-platform node scheduling based on message queues, which can avoid resource conflicts in the cross-platform node scheduling process, effectively improve the utilization rate of hardware computing resources, realize full automation of cross-platform node scheduling, and greatly reduce operation and maintenance costs.

[0006] According to one aspect of the present invention, a cross-platform automatic node scheduling method based on message queues is provided. The method is applied to a three-layer execution architecture, which includes a triggering layer, a scheduling layer, and a base layer. The method includes: The triggering layer receives multiple task requests triggered by various types of CI / CD platforms, and constructs at least one task message queue based on the multiple task requests. The scheduling layer sequentially retrieves target tasks to be executed from the task message queue, obtains candidate nodes from the pre-built machine message queue, and verifies whether the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions. If the candidate node meets the scheduling conditions relative to the status of all CI / CD platforms, the candidate node is brought online through the scheduling layer, and the target task is executed through the candidate node. The base layer is used to provide platform support and infrastructure services to the triggering layer and the scheduling layer.

[0007] According to another aspect of the present invention, a cross-platform automatic node scheduling device based on a message queue is provided. The device is applied to a three-layer execution architecture, which includes a triggering layer, a scheduling layer, and a base layer. The device includes: The task triggering module is used to receive multiple task requests triggered by multiple types of CI / CD platforms through the triggering layer, and to construct at least one task message queue based on the multiple task requests. The task acquisition module is used to sequentially acquire target tasks to be executed from the task message queue through the scheduling layer, acquire candidate nodes from the pre-built machine message queue, and verify whether the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions. The task scheduling module is used to bring the candidate node online through the scheduling layer and execute the target task through the candidate node if the candidate node's status relative to all CI / CD platforms meets the scheduling conditions. The base layer is used to provide platform support and infrastructure services to the triggering layer and the scheduling layer.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to execute the message queue-based cross-platform automatic node scheduling method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the message queue-based cross-platform automatic node scheduling method described in any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the message queue-based cross-platform automatic node scheduling method described in any embodiment of the present invention.

[0011] The technical solution provided by this invention receives multiple task requests triggered by various types of CI / CD platforms through a triggering layer, constructs at least one task message queue based on the multiple task requests, and sequentially retrieves target tasks to be executed from the task message queue through a scheduling layer. Candidate nodes are obtained from a pre-constructed machine message queue, and the status of the candidate nodes relative to all CI / CD platforms is verified to meet scheduling conditions. If the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions, the scheduling layer brings the candidate nodes online and executes the target tasks through the candidate nodes. This technical approach avoids resource conflicts during cross-platform node scheduling, effectively improves the utilization rate of hardware computing resources, achieves full automation of cross-platform node scheduling, and greatly reduces operation and maintenance costs.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a cross-platform automatic node scheduling method based on a message queue, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the architecture applicable to a cross-platform automatic node scheduling method based on a message queue according to an embodiment of the present invention; Figure 3 This is a flowchart of another cross-platform automatic node scheduling method based on message queues provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a three-tier execution architecture according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a cross-platform node automatic scheduling device based on a message queue according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device that implements the cross-platform node automatic scheduling method based on message queues according to embodiments of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Figure 1 This is a flowchart illustrating a message queue-based cross-platform node automatic scheduling method provided in this embodiment of the invention. This embodiment is applicable to the unified scheduling of cross-platform triggered CI / CD tasks. The method is applied to a three-layer execution architecture, which includes a triggering layer, a scheduling layer, and a base layer. The method can be executed by a message queue-based cross-platform node automatic scheduling device, which can be implemented in hardware and / or software and configured in an electronic device, such as... Figure 1 As shown, the method includes: Step 110: Receive multiple task requests triggered by multiple types of CI / CD platforms through the triggering layer, and construct at least one task message queue based on the multiple task requests.

[0018] In this embodiment, optionally, Figure 2This can be a schematic diagram of an architecture applicable to a cross-platform automatic node scheduling method based on message queues, such as... Figure 2 As shown, the three-layer execution architecture in this embodiment is divided into a triggering layer, a scheduling layer, and a base layer from top to bottom, and the layers interact with each other through standardized interfaces. Figure 2 As shown, the triggering layer is used to receive and aggregate task requests from multiple types of CI / CD platforms. The triggering layer includes the following components: multiple open-source CI / CD platforms (such as...) Figure 2 The system includes open-source CI / CD platform 1, open-source CI / CD platform 2 (such as Jenkins, GitLab CI, etc.), and a self-developed CI / CD platform, which serve as the initiators of tasks. The triggering layer also includes a CI / CD task aggregation module, which is used to uniformly collect and standardize CI / CD task requests generated by various platforms to obtain at least one task message queue.

[0019] Step 120: The scheduling layer sequentially retrieves the target tasks to be executed from the task message queue, obtains candidate nodes from the pre-built machine message queue, and verifies whether the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions.

[0020] In this embodiment, as Figure 2 As shown, the scheduling layer is used to schedule tasks to be executed in the task message queue, allocate corresponding nodes, and manage the execution status of nodes and tasks. Specifically, the scheduling layer may include the following components: a task message queue, a core scheduling module, and a machine message queue.

[0021] The task message queue includes standardized task messages received and processed by the triggering layer. This task message queue can achieve queue isolation by label dimension based on a remote dictionary server (Redis). The core scheduling module is used to retrieve tasks from the task message queue, obtain available nodes from the machine message queue, execute scheduling decisions, and coordinate the basic layer to complete node operations. The machine message queue is used to store currently available compute node information, and queue isolation by platform and label dimension is achieved based on Redis.

[0022] The advantage of this setup is that, in the existing technology, nodes need to be manually configured on each platform and cannot be dynamically allocated according to task requirements. This embodiment, through a message queue-based decoupling design for triggering and scheduling, can achieve parallel scheduling of different types of resources and priority processing of urgent tasks, thereby improving the flexibility of the task scheduling process.

[0023] In this step, specifically, the scheduling layer can traverse the real-time status of all candidate nodes in the registered CI / CD platforms before each scheduling decision based on the cross-platform global status verification mechanism, so as to ensure that the same physical node is occupied by at most one task of one platform at any time. If a candidate node meets the above scheduling conditions, then step 130 is executed.

[0024] The advantage of this setup is that it can solve the resource contention problem caused by multiple CI / CD platforms managing nodes independently and their states being out of sync in existing technologies. By using a cross-platform global state verification mechanism, cross-platform resource conflicts can be fundamentally avoided.

[0025] In this embodiment, as Figure 2 As shown, the foundation layer can be understood as the basic tool layer required by the triggering layer and the scheduling layer. This foundation layer provides platform support and infrastructure services to the triggering layer and the scheduling layer. Specifically, the foundation layer may include the following components: an open-source CI / CD platform, a self-developed CI / CD platform, an ElasticSearch database, a Grafana visualization and monitoring analysis platform, a Redis caching component, a RocketMQ distributed message middleware, and an alarm notification component.

[0026] The open-source CI / CD platform (such as Jenkins, GitLab CI, etc.) provides operation interfaces for node online / offline status, tag management, and task status query. The self-developed CI / CD platform provides node management operation interfaces similar to those of the open-source CI / CD platform.

[0027] The ElasticSearch database stores historical data on node status and task queues, supporting retrospective analysis of node scheduling behavior and troubleshooting of anomalies. It also provides a data source for the Grafana monitoring dashboard. The Grafana visualization monitoring and analysis platform, based on historical data from ElasticSearch, provides a real-time visual monitoring dashboard for node status and task queues, allowing operations personnel to intuitively understand system operation and quickly locate abnormal nodes and queue backlog issues.

[0028] The Redis caching component provides underlying storage support for task message queues and machine message queues. It utilizes Redis's list data structure to implement a first-in-first-out queue, achieves physical isolation of queues by tag and platform through key naming rules, and supports the acquisition and release of distributed locks through Redis's SET NX atomic operation.

[0029] The distributed message middleware RocketMQ is used to provide message middleware services, supporting reliable asynchronous transmission of task messages and machine status messages (such as asynchronous node status notifications, alarm event transmission, etc.), ensuring that messages are not lost or duplicated in high-concurrency scenarios.

[0030] The alarm notification component is used to provide real-time alarm services for abnormal events, so as to promptly notify operation and maintenance personnel to handle abnormal issues.

[0031] The advantage of this setup is that, addressing the problem of deep coupling between scheduling logic and specific CI / CD platforms in existing technologies, making them difficult to scale, this embodiment uses a three-layer decoupling architecture to completely separate the scheduling logic from the specific platform. For any newly added CI / CD platform, there is no need to modify the core logic of the scheduling engine, thereby improving the scalability of the scheduling method. Furthermore, the entire lifecycle, from task triggering, node allocation, task execution, resource release, and exception response, can be completed automatically without manual intervention, achieving a fully automated scheduling closed loop and greatly reducing operation and maintenance costs.

[0032] Step 130: If the candidate node satisfies the scheduling conditions relative to the status of all CI / CD platforms, the candidate node is brought online through the scheduling layer, and the target task is executed through the candidate node.

[0033] In this embodiment, if the candidate node meets the scheduling conditions relative to the status of all CI / CD platforms, the candidate node is automatically brought online through the scheduling layer, and after the candidate node completes the target task, the node is automatically taken offline and released back to the available node pool.

[0034] The advantage of this setup is that it can solve the problem of resource waste caused by nodes being online for a long time and not being able to be automatically released when idle in existing technologies. By adopting an automatic node online / offline mechanism, that is, nodes are only activated online when there is a matching task to schedule, and are automatically offline and released back to the available node pool after the task is completed, the utilization rate of hardware computing resources can be effectively improved.

[0035] The technical solution provided by this invention receives multiple task requests triggered by various types of CI / CD platforms through a triggering layer, constructs at least one task message queue based on the multiple task requests, and sequentially retrieves target tasks to be executed from the task message queue through a scheduling layer. Candidate nodes are obtained from a pre-constructed machine message queue, and the status of the candidate nodes relative to all CI / CD platforms is verified to meet scheduling conditions. If the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions, the scheduling layer brings the candidate nodes online and executes the target tasks through the candidate nodes. This technical approach avoids resource conflicts during cross-platform node scheduling, effectively improves the utilization rate of hardware computing resources, achieves full automation of cross-platform node scheduling, and greatly reduces operation and maintenance costs.

[0036] Figure 3 A flowchart of another cross-platform automatic node scheduling method based on message queues provided in this embodiment of the invention is shown below. Figure 3 As shown, the method includes: Step 210: Receive multiple task requests triggered by various types of CI / CD platforms through the triggering layer, and convert the multiple task requests according to a unified standard task message format to obtain multiple task messages.

[0037] Step 220: Perform parameter verification on each of the multiple task messages, and store the verified task messages into at least one task message queue.

[0038] In this embodiment, optionally, Table 1 is a mapping table of core components and functions in the trigger layer: Table 1 Each task message queue corresponds to a queue label. The tmp_label is generated by the scheduling layer through the operation in step 260. The triggering layer can support manual triggering, timed triggering, or Webhook triggering; this embodiment does not impose any restrictions on this.

[0039] In one embodiment of this example, the task message queue and the machine message queue can be isolated according to the resource label dimension. Tasks and nodes with different labels are stored in their own independent Redis queues, and the queue key name adopts the formatting rule "task_queue_for_node_controller_{label}" to achieve physical isolation.

[0040] Optionally, each queue tag also corresponds to an independent task priority queue (i.e., a fast pass queue), which is used to store pending tasks with a priority higher than a preset level, so that the scheduling layer can schedule the pending tasks first.

[0041] In one specific embodiment, Table 2 is a mapping table of different queue types and their uses: Table 2 The advantage of this setup is that by maintaining an independent FastPass priority queue for each tag, high-priority tasks can directly enter the FastPass queue. This allows the scheduling engine to prioritize processing the queue in each round, ensuring that urgent tasks receive priority resource allocation and significantly improving the overall scheduling efficiency in high-concurrency scenarios.

[0042] Step 230: Create an independent scheduling thread for each queue label through the scheduling layer. The scheduling thread retrieves the target task to be executed from the corresponding task message queue according to the matched queue label and verifies the pipeline associated with the target task.

[0043] In this embodiment, the scheduling layer can continuously scan all message queues with pending tasks and create an independent scheduling thread for each tag of a pending task to run in parallel. Each scheduling thread is independent and does not block the others; the scheduling of different types of resources (such as GPU nodes and CPU nodes) does not interfere with each other.

[0044] In this step, specifically, the scheduling layer can continuously monitor the status of the target task to be executed in the task message queue and verify whether the pipeline associated with the target task is still running. If the pipeline has ended or been canceled, the target task is automatically discarded to avoid occupying node resources when the target task is ineffectively scheduled.

[0045] Step 240: Through the scheduling layer, traverse the offline and locked states of each candidate node on all registered CI / CD platforms from the pre-built machine message queue; when the candidate node is offline and unlocked on all CI / CD platforms, determine that the candidate node meets the scheduling conditions.

[0046] In this step, the scheduling layer can retrieve candidate nodes from the corresponding machine message queue based on the label and platform specified by the target task. After retrieving the candidate node, the scheduling layer can perform a global status check, that is, check the status of the candidate node on all registered CI / CD platforms. Only when the candidate node is offline and not locked by any platform will it be determined that the candidate node meets the scheduling conditions.

[0047] The advantage of this setup is that, compared to existing technologies where each CI / CD platform independently manages its own node state without a state synchronization mechanism, which easily leads to conflicts where the same node is scheduled by multiple platforms simultaneously, this embodiment fundamentally avoids resource conflicts caused by asynchronous node states between heterogeneous CI / CD platforms by performing a global state traversal verification across all registered platforms before scheduling decisions. This ensures that the same physical node is occupied by at most one task from one platform at any given time.

[0048] Step 250: Set a distributed lock for the candidate node by performing atomic operations.

[0049] In this embodiment, optionally, Figure 4 This can be a collaborative diagram of a three-tier execution architecture, such as... Figure 4 As shown, after the scheduling layer finds a candidate node that meets the scheduling conditions, in order to prevent multiple scheduling instances from operating on the same candidate node at the same time, it can set a distributed lock for the candidate node using Redis's atomic SET NX command.

[0050] In one specific implementation, the scheduling layer can utilize Redis's SET command in conjunction with the NX option to achieve atomic lock acquisition operations. The lock will only be successfully acquired if the specified key does not exist, thus ensuring that only one scheduling instance can acquire the operation right of a candidate node at any given time. Optionally, the lock key name follows the formatted naming rule of "node_scheduling:node name" to ensure that the granularity of the lock is precise down to a single node. The scheduling layer can also set a lock expiration time (e.g., 900 seconds) to prevent deadlocks caused by abnormal exits of scheduling instances.

[0051] The advantage of this setup is that by setting up a distributed lock, the security issue of multiple scheduling instances concurrently operating on the same candidate node can be resolved, effectively preventing resource contention and deadlocks during cross-platform node scheduling.

[0052] Step 260: Generate a temporary label corresponding to the candidate node; the temporary label includes a unique timestamp.

[0053] In this embodiment, as Figure 4 As shown, after the scheduling layer finds a candidate node that meets the scheduling conditions, it can also generate a temporary label for the candidate node to avoid the node preemption problem in a multi-task concurrent scenario.

[0054] In one specific embodiment, the scheduling layer can generate a globally unique temporary label (tmp_label) in the format "original label_timestamp" for candidate nodes that meet the scheduling conditions, and dynamically bind this temporary label to the candidate node by calling the target platform API through the platform adaptation interface. The temporary label is used to identify the candidate node for exclusive access. The pipeline associated with the target task can lock the candidate node by accurately matching the unique temporary label and perform corresponding operations through the candidate node.

[0055] The advantage of this setup is that, compared to existing technologies where node exclusivity typically relies on the locking mechanism of the CI / CD platform itself (i.e., node exclusivity can only be effective within a single platform), this embodiment can achieve cross-platform and cross-task node-level exclusivity control by overlaying a dynamic temporary tag layer on top of the platform's native tag system, thereby improving the flexibility of the cross-platform node scheduling process.

[0056] Step 270: Through the pre-built platform adaptation interface, call the application programming interface (API) of the target CI / CD platform to perform online processing on the candidate node, and execute the target task through the candidate node.

[0057] The platform adaptation interface predefines interface for node online operation, node offline operation, adding label operation, and deleting label operation.

[0058] In one specific embodiment, the platform adaptation interface can set specific adaptation subclasses for each CI / CD platform, and encapsulate the platform-specific API call logic and authentication mechanism in the subclasses. When the scheduling layer needs to operate on a node on a certain platform, it only needs to obtain the corresponding platform adapter instance through the platform adaptation interface, and can complete the node operation through a unified interface, thereby enabling the new adaptation subclass to quickly connect to the new CI / CD platform.

[0059] The advantage of this setup is that by building platform adaptation interfaces, the problem of inconsistent operation interfaces for multiple types of CI / CD platforms can be solved. When adding any new CI / CD platform, only a new adaptation class needs to be built, without modifying the core logic of the scheduling layer, thereby achieving complete decoupling between the scheduling layer and specific platforms.

[0060] Step 280: After the target task is completed, delete the temporary tag corresponding to the candidate node, take the candidate node offline, release the distributed lock corresponding to the candidate node, and then add the candidate node back to the machine message queue.

[0061] In this embodiment, the distributed lock can be actively released after the target task is completed normally; or the distributed lock can be automatically released based on the pre-set lock expiration time when the scheduling instance exits abnormally.

[0062] In one embodiment of this example, the method further includes: real-time monitoring of abnormal events during node scheduling, generating alarm information corresponding to the abnormal events, and pushing the alarm information to the operation and maintenance terminal.

[0063] The abnormal events include at least: node state change, node scheduling failure, the number of tasks to be executed in the task message queue exceeding a preset threshold, and the distributed lock corresponding to the candidate node timeout.

[0064] The technical solution provided in this invention receives multiple task requests triggered by various types of CI / CD platforms through a triggering layer. These requests are then converted into multiple task messages according to a unified standard task message format. Parameter validation is performed on each task message, and the validated task messages are stored in at least one task message queue. An independent scheduling thread is created for each queue, matching its corresponding tag. The scheduling thread retrieves the target task from the corresponding task message queue based on the matched queue tag. The pipeline associated with the target task is validated. The scheduling layer iterates through the pre-built machine message queue, checking the offline and locked states of all candidate nodes registered on the CI / CD platforms. When a candidate node is in all CI / CD platforms... When all nodes on the / CD platform are offline and unlocked, the following technical approach is used: First, if a candidate node meets the scheduling conditions, an atomic operation is performed to set a distributed lock for the candidate node, a temporary tag is generated for the candidate node, and the target CI / CD platform's API is called through a pre-built platform adaptation interface to bring the candidate node online. Then, the target task is executed through the candidate node. After the target task is completed, the temporary tag for the candidate node is deleted, the candidate node is taken offline, the distributed lock for the candidate node is released, and the candidate node is re-added to the machine message queue. This approach enables full lifecycle management from task reception to node release, ensuring the correctness, efficiency, and reliability of node scheduling in cross-platform scenarios.

[0065] Figure 5 This invention provides a schematic diagram of a cross-platform automatic node scheduling device based on a message queue. The device is applied to a three-layer execution architecture, which includes a triggering layer, a scheduling layer, and a base layer. The device includes: The task triggering module 310 is used to receive multiple task requests triggered by multiple types of CI / CD platforms through the triggering layer, and to construct at least one task message queue based on the multiple task requests. The task acquisition module 320 is used to sequentially acquire the target tasks to be executed from the task message queue through the scheduling layer, acquire candidate nodes from the pre-built machine message queue, and verify whether the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions. The task scheduling module 330 is used to bring the candidate node online through the scheduling layer and execute the target task through the candidate node if the candidate node's status relative to all CI / CD platforms meets the scheduling conditions. The base layer is used to provide platform support and infrastructure services to the triggering layer and the scheduling layer.

[0066] The technical solution provided by this invention receives multiple task requests triggered by various types of CI / CD platforms through a triggering layer, constructs at least one task message queue based on the multiple task requests, and sequentially retrieves target tasks to be executed from the task message queue through a scheduling layer. Candidate nodes are obtained from a pre-constructed machine message queue, and the status of the candidate nodes relative to all CI / CD platforms is verified to meet scheduling conditions. If the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions, the scheduling layer brings the candidate nodes online and executes the target tasks through the candidate nodes. This technical approach avoids resource conflicts during cross-platform node scheduling, effectively improves the utilization rate of hardware computing resources, achieves full automation of cross-platform node scheduling, and greatly reduces operation and maintenance costs.

[0067] Based on the above embodiments, each task message queue corresponds to a queue label; each queue label also corresponds to an independent task priority queue; the task priority queue is used to store pending tasks with a priority higher than a preset level, so that the scheduling layer prioritizes scheduling the pending tasks.

[0068] Task triggering module 310 includes: The request conversion unit is used to convert the multiple task requests into multiple task messages according to a unified standard task message format; to perform parameter verification on the multiple task messages respectively, and to store the verified task messages into at least one task message queue.

[0069] Task acquisition module 320 includes: The thread creation unit is used to create an independent scheduling thread for each queue label. The scheduling thread retrieves the target task to be executed from the corresponding task message queue according to the matched queue label and verifies the pipeline associated with the target task. The candidate node traversal unit is used to traverse the offline and locked states of all candidate nodes registered on the CI / CD platform from the pre-built machine message queue; when the candidate node is offline and unlocked on all CI / CD platforms, it is determined that the candidate node meets the scheduling conditions.

[0070] Task scheduling module 330 includes: The candidate node deployment unit is used to call the application programming interface (API) of the target CI / CD platform through a pre-built platform adaptation interface to perform deployment processing on the candidate node; Among them, the platform adaptation interface predefines the node online operation interface, the node offline operation interface, the tag addition operation interface, and the tag deletion operation interface; A distributed lock acquisition unit is used to set a distributed lock for the candidate node by performing atomic operations; A temporary label generation unit is used to generate temporary labels corresponding to the candidate nodes; the temporary label includes a unique timestamp. The task termination unit is used to delete the temporary tag corresponding to the candidate node, take the candidate node offline, release the distributed lock corresponding to the candidate node, and then add the candidate node back to the machine message queue after executing the target task through the candidate node.

[0071] The device further includes: The anomaly monitoring module is used to monitor abnormal events in the node scheduling process in real time, generate alarm information corresponding to the abnormal events, and push the alarm information to the operation and maintenance terminal. The abnormal events include at least: node state change, node scheduling failure, the number of tasks to be executed in the task message queue exceeding a preset threshold, and the distributed lock corresponding to the candidate node timeout.

[0072] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in the embodiments of the present invention can be found in the methods provided in all the foregoing embodiments of the present invention.

[0073] Figure 6A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0074] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) or a random access memory (RAM), communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from the storage unit 18 into the random access memory 13. The random access memory 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, the read-only memory 12, and the random access memory 13 are interconnected via a bus 14. Input / output (I / O) interfaces are also connected to the bus 14.

[0075] Multiple components in electronic device 10 are connected to input / output interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processing (DSP) processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a message queue-based cross-platform automatic node scheduling method.

[0077] In some embodiments, the message queue-based cross-platform automatic node scheduling method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via read-only memory 12 and / or communication unit 19. When the computer program is loaded into random access memory 13 and executed by processor 11, one or more steps of the message queue-based cross-platform automatic node scheduling method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the message queue-based cross-platform automatic node scheduling method by any other suitable means (e.g., by means of firmware).

[0078] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0082] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0083] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) in terms of management difficulty and weak business scalability.

[0084] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A cross-platform automatic node scheduling method based on message queues, characterized in that, The method is applied to a three-tier execution architecture, which includes a triggering layer, a scheduling layer, and a base layer; the method includes: The triggering layer receives multiple task requests triggered by various types of continuous integration and continuous delivery (CI / CD) platforms, and constructs at least one task message queue based on the multiple task requests. The scheduling layer sequentially retrieves target tasks to be executed from the task message queue, obtains candidate nodes from the pre-built machine message queue, and verifies whether the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions. If the candidate node meets the scheduling conditions relative to the status of all CI / CD platforms, the candidate node is brought online through the scheduling layer, and the target task is executed through the candidate node. The base layer is used to provide platform support and infrastructure services to the triggering layer and the scheduling layer.

2. The method according to claim 1, characterized in that, Each task message queue corresponds to a queue label; The target tasks to be executed are retrieved sequentially from the task message queue, including: Create an independent scheduling thread for each queue tag. The scheduling thread retrieves the target task to be executed from the corresponding task message queue according to the matched queue tag and verifies the pipeline associated with the target task.

3. The method according to claim 2, characterized in that, Each queue label also corresponds to an independent task priority queue; The task priority queue is used to store pending tasks with a priority higher than a preset level, so that the scheduling layer can prioritize scheduling the pending tasks.

4. The method according to claim 1, characterized in that, Construct at least one task message queue based on the multiple task requests, including: The multiple task requests are converted according to a unified standard task message format to obtain multiple task messages; The parameters of each of the multiple task messages are validated, and the task messages that pass the validation are stored in at least one task message queue.

5. The method according to claim 1, characterized in that, After determining that the candidate node's state relative to all CI / CD platforms satisfies the scheduling conditions, the process further includes: A distributed lock is set for the candidate node by performing atomic operations.

6. The method according to claim 5, characterized in that, After determining that the candidate node's state relative to all CI / CD platforms satisfies the scheduling conditions, the process further includes: Generate a temporary label corresponding to the candidate node; the temporary label includes a unique timestamp.

7. The method according to claim 6, characterized in that, The process of bringing the candidate nodes online includes: After generating the temporary tag corresponding to the candidate node, the application programming interface (API) of the target CI / CD platform is called through the pre-built platform adaptation interface to process the candidate node to go online. The platform adaptation interface predefines interface for node online operation, node offline operation, tag addition operation, and tag deletion operation.

8. The method according to claim 6, characterized in that, After executing the target task through the candidate node, the process also includes: Delete the temporary tag corresponding to the candidate node, take the candidate node offline, release the distributed lock corresponding to the candidate node, and then add the candidate node back to the machine message queue.

9. The method according to claim 6, characterized in that, The method further includes: Real-time monitoring of abnormal events during node scheduling, generation of alarm information corresponding to the abnormal events, and pushing of the alarm information to the operation and maintenance terminal; The abnormal events include at least: node state change, node scheduling failure, the number of tasks to be executed in the task message queue is greater than a preset threshold, and the distributed lock corresponding to the candidate node times out.

10. The method according to claim 1, characterized in that, Candidate nodes are retrieved from a pre-built machine message queue, and the status of the candidate nodes relative to all CI / CD platforms is verified to meet the scheduling conditions, including: Iterate through the offline and locked states of all candidate nodes registered on the CI / CD platform from the pre-built machine message queue. When the candidate node is offline and unlocked on all CI / CD platforms, the candidate node is determined to meet the scheduling conditions.

11. A cross-platform automatic node scheduling device based on message queues, characterized in that, The device is applied to a three-layer execution architecture, which includes a triggering layer, a scheduling layer, and a base layer; the device includes: The task triggering module is used to receive multiple task requests triggered by various types of continuous integration and continuous delivery (CI / CD) platforms through the triggering layer, and to construct at least one task message queue based on the multiple task requests. The task acquisition module is used to sequentially acquire target tasks to be executed from the task message queue through the scheduling layer, acquire candidate nodes from the pre-built machine message queue, and verify whether the status of the candidate nodes relative to all CI / CD platforms meets the scheduling conditions. The task scheduling module is used to bring the candidate node online through the scheduling layer and execute the target task through the candidate node if the candidate node's status relative to all CI / CD platforms meets the scheduling conditions. The base layer is used to provide platform support and infrastructure services to the triggering layer and the scheduling layer.

12. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the message queue-based cross-platform automatic node scheduling method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the message queue-based cross-platform automatic node scheduling method according to any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the cross-platform automatic node scheduling method based on message queues as described in any one of claims 1-10.

Images