Method for resource scheduling of data transmission system based on declarative api and storage medium

Abstract

The application discloses a data transmission system resource scheduling method based on a declarative API and a storage medium, belongs to the technical field of resource scheduling under a Kubernetes environment, and aims to solve the technical problem of how to utilize Kubernetes resource arrangement capability to realize convenient and quick scheduling of data transmission system resources.The technical scheme is as follows: the method is as follows: custom API resource type CRD: defining the resource type CRD of the collection end resource, the warehousing end resource and the message middleware resource of the data transmission system respectively; wherein, the group of the resource type CRD is dts; the resource description file corresponding to the resource type CRD is used to schedule the collection end resource, the warehousing end resource and the message middleware resource, and the corresponding resource scheduling work is realized through the resource file of the application captured by the custom controller; the definition of the resource file is used to complete the abstraction of the underlying physical resource and the business logic resource, and the scheduling of the business resource is realized through the custom controller.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology in the Kubernetes environment, specifically a method and storage medium for resource scheduling of a data transmission system based on a declarative API. Background Technology

[0002] Kubernetes, or K8s for short, is an open-source application used to manage containerized applications across multiple hosts in a cloud platform. Kubernetes aims to make deploying containerized applications simple and powerful, providing a mechanism for application deployment, planning, updating, and maintenance. Traditional application deployment involves installing applications via plugins or scripts. This approach ties the application's operation, configuration, management, and entire lifecycle to the current operating system, hindering upgrades, updates, and rollbacks. While virtual machines can achieve some functionality, they are very heavy and lack portability. The new approach uses container deployment, where each container is isolated, has its own file system, and processes within containers do not interfere with each other, allowing for the differentiation of computing resources. Compared to virtual machines, containers can be deployed quickly. Because containers are decoupled from the underlying infrastructure and machine file system, they can migrate between different clouds and operating system versions. Containers consume fewer resources and deploy quickly. Each application can be packaged into a container image, and the one-to-one relationship between each application and container gives containers a significant advantage. Using containers, container images can be created for applications during the build or release phase. Because each application does not need to be combined with other application stacks or depend on the production environment infrastructure, this provides a consistent environment from development to testing and production. Similarly, containers are lighter and more "transparent" than virtual machines, making them easier to monitor and manage.

[0003] Kubernetes is an open-source container orchestration engine from Google that supports automated deployment, massive scalability, and containerized application management. When deploying an application in a production environment, it's common to deploy multiple instances of that application to load balance application requests. In Kubernetes, multiple containers can be created, each running an application instance. Built-in load balancing strategies then manage, discover, and access this group of application instances, eliminating the need for complex manual configuration and handling by operations personnel.

[0004] When deploying a simple web server to a Kubernetes cluster, you usually need to first write a Deployment controller, then create a Service object, associate it with the Pod using its label, and finally expose the service through an Ingress or a Service of type NodePort. This process is a bit cumbersome.

[0005] Therefore, how to leverage Kubernetes resource orchestration capabilities to achieve convenient and rapid scheduling of data transmission system resources is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The technical objective of this invention is to provide a method and storage medium for scheduling data transmission system resources based on a declarative API, in order to solve the problem of how to utilize Kubernetes resource orchestration capabilities to achieve convenient and fast scheduling of data transmission system resources.

[0007] The technical objective of this invention is achieved as follows: a method for resource scheduling in a data transmission system based on a declarative API, the specific method of which is as follows:

[0008] Custom API Resource Type CRD: Defines the resource type CRDs for the data transmission system's acquisition end resources, database entry end resources, and message middleware resources; where the resource type CRD is grouped as dts;

[0009] The application uses the resource description file of the corresponding resource type CRD to schedule the resources of the collection end, the database end, and the message middleware, and uses a custom controller to capture the application's resource file to achieve the corresponding resource scheduling work;

[0010] The underlying physical resources and business logic resources are abstracted by defining resource files, and the scheduling of business resources is realized through a custom controller.

[0011] Preferably, the data transmission system includes a data acquisition component, a message middleware, and a data entry component. The data acquisition component reads data and writes it to the message middleware. The message middleware caches the read data to ensure data integrity. The data acquisition component and the message middleware are deployed in the same pod. The data acquisition component and the message middleware each define a resource type CRD of type dtsReader, and the data entry component defines a resource type CRD of type dtsWriter.

[0012] The acquisition component is used to read data from the source database.

[0013] Message middleware is used for data storage and transmission;

[0014] The data import component is used to write the collected data into the destination database.

[0015] More preferably, the specific fields of the resource type CRD of the dtsReader and dtsWriter are as follows:

[0016] taskId: A unique identifier for the data transfer task;

[0017] Version: Product version number. The version numbers of the backend database collection component and the data entry component are consistent with the product version number.

[0018] isAvailable: Whether the resource is available;

[0019] dbInfo: Data source information, connection information for the source or destination database;

[0020] task: Task configuration information, which includes table information, field information and related transmission strategy parameters;

[0021] Resources: Resource information, including CPU, memory, and storage information of the pod that needs to be started.

[0022] As a preferred option, a custom controller refers to creating a corresponding CDR controller for the resource type CRD, which triggers the business logic code within the CRD controller when the CRD object changes; specifically as follows:

[0023] The data transmission system creates a custom controller, dts-Operator;

[0024] The custom controller dts-Operator uses the ListAndWatch method of Reflector to get and listen for changes in object instances;

[0025] When any instance changes, Reflector will receive an event notification and, upon receiving the notification, will store the combination of the event and the object corresponding to the object instance into a first-in-first-out queue.

[0026] The custom controller dts-Operator uses Informer to continuously read objects from a first-in-first-out queue and determines whether the event type is dtsReader or dtsWriter:

[0027] After determining the type of event, create the resources required for dtsReader or dtsWriter based on the event type.

[0028] More specifically, the resources required to create a dtsReader are as follows:

[0029] The custom controller dts-Operator creates the pod and service resources of the message middleware nats that the collection component depends on;

[0030] In the message middleware NATS, create JetStream and Consumer resources;

[0031] After the NATS message middleware resources are deployed, deploy the pods, services, and storage resources of the data collection components.

[0032] By monitoring the corresponding events, physical resources and business configurations can be processed according to custom logic;

[0033] Once the processing is complete, the acquisition components required for the data transmission task will function normally.

[0034] The resources required to create dtsWriter are as follows:

[0035] The custom controller dts-Operator creates the physical resources of the pods and services required by the inbound component, and generates configuration files related to the data transfer task.

[0036] Write information related to the NATS message middleware to the configuration file;

[0037] By monitoring the corresponding events, physical resources and business configurations can be processed according to custom logic;

[0038] Once processing is complete, the data transfer task's required inbound components will function correctly.

[0039] Better still, the custom controller dts-Operator relies on the Informer code block to obtain dtsWriter and dtsReader objects from the Kubernetes APIServer. The Informer corresponds one-to-one with the API object. During the data transmission process, what is passed to the custom controller dts-Operator is an Informer of a Network object.

[0040] When creating the Informer factory, a networkClient is passed to the Informer factory. The NetworkInformer uses the networkClient to establish a connection with the APIServer. The connection between the Network Informer and the APIServer is maintained by the ListAndWatch mechanism in the Reflector package used by the Informer. The ListAndWatch mechanism is used to obtain and listen to changes in the Network object instance.

[0041] Under the ListAndWatch mechanism, once a new Network instance is created, deleted, or updated on the APIServer side, the Reflector will receive an event notification for the corresponding Network instance. The combination of this event and the corresponding API object is called the Delta increment and is placed in a Delta FIFO queue. At the same time, the Informer will continuously read increments from the Delta FIFO queue. Each time an increment is read, the Informer will determine the event type in the increment and create or update the buffer of local objects.

[0042] More preferably, Informer has the following features:

[0043] ① Synchronize local cache; Informer comes with its own caching and indexing mechanisms; the client library that triggers the Handler, the local cache is called Store in Kubernetes, and the index is called Index; Informer uses the Reflector package, which is a client wrapper that obtains and monitors changes to API objects through the ListAndWatch mechanism; the Reflector package and Informer coordinate with each other using an incremental first-in-first-out queue;

[0044] ② Based on the event type, trigger the registered ResourceEventHandler; through the monitored event changes, Informer updates the local cache in real time and calls the corresponding EventHandler. Every time the time specified by resyncPeriod elapses, the local cache maintained by Informer will be forcibly updated using the result returned by the most recent LIST, thereby ensuring the validity of the cache; in particular, when creating a custom controller dts-Operator, it is registered with the Informer corresponding to ResourceEventHandler;

[0045] Before writing the control loop, the informer uses a work queue for coordination. In practice, all code except the control loop is automatically generated by Kubernetes. The specific logic of the informer's control loop is as follows:

[0046] Waiting for Informer to complete a local cache data synchronization operation;

[0047] Start one or more infinite loop tasks using goroutines;

[0048] In each cycle, the data transmission business logic code is executed;

[0049] If the control loop cannot retrieve the information of the corresponding object in the Informer cache, it means that the deletion logic needs to be executed;

[0050] If object information is retrieved from the cache, the controller mode is executed to compare the expected state with the actual state. The expected state comes from etcd, and the actual state comes from the cluster itself.

[0051] As a preferred option, the custom API resource type CRD also includes the following:

[0052] Perform Initializer and Validation operations on API objects: The validation operation verifies the legality of each field in the object and saves it to the Registry data structure after validation. If the definition of an API object can be found in the Registry, it means that the API corresponds to a valid k8s API object.

[0053] The superversion object is converted into a serialized object of the user-submitted version and then saved to etcd.

[0054] Preferably, the data transmission system defines a management service dts-Service, which is used to receive user instructions sent by the front end and interact with the back-end custom controller dts-Operator through a declarative API to realize the creation of data transmission task resources and the control of task operation logic.

[0055] The creation of resources is detailed below:

[0056] Users create tasks through the front-end interface, which in turn calls the management service dts-Service via an HTTP request to create the data transmission task method.

[0057] The management service dts-Service accesses the Kubernetes API and applies the resource type CRDs of the data transfer tasks dtsReader and dtsWriter in sequence. The name field in the resource type CRD is consistent with the task number to distinguish the resource type CRDs of different tasks.

[0058] The application's resource type CRD will be detected by the custom controller dts-Operator and the corresponding resource creation logic will be executed.

[0059] After the resources are created, the database's acquisition component, storage component, and message middleware NATS automatically complete the initialization process.

[0060] After the management service dts-Service detects that the resources of the task's data collection component and data storage component have been successfully allocated and initialized normally, it returns a creation success message to the user.

[0061] The specific operational logic control for the business is as follows:

[0062] Users can start, stop, pause, resume, change passwords, and delete data transmission tasks through the front-end interface; the front-end interface calls the corresponding methods in the management service dts-Service via HTTP requests;

[0063] The management service dts-Service assembles the resource type CRDs of dtsReader and dtsWriter corresponding to the data transmission task. The name field of the resource type CRD is consistent with the task number to distinguish the resource type CRDs of different tasks.

[0064] In the assembled resource type CRD, the operation field corresponds to the user's operation. For example, when stopping an operation, the value of the operation field is stopTask.

[0065] The management service dts-Service accesses the Kubernetes API to apply the resource type CRD for the task dtsReader and dtsWriter respectively;

[0066] After the management service dts-Operator detects a change in the resource type CRD version, it analyzes the content of the operation field to determine the user's operation. Based on the operation type, it calls the corresponding gRPC interface of the data collection component and the data entry component to complete the user's operation.

[0067] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the method for resource scheduling of a data transmission system based on a declarative API as described above.

[0068] The method and storage medium for data transmission system resource scheduling based on declarative APIs of the present invention have the following advantages:

[0069] (i) In this invention, the data transmission system uses CRD to define the resources corresponding to the business model, captures the application's resource files through a custom controller, and completes the scheduling of the required resources according to the written rules.

[0070] (ii) This invention utilizes the powerful Kubernetes resource orchestration capabilities of the declarative API to abstract and define the business resources of the data transmission system and schedule the resources required by the data transmission system;

[0071] (III) This invention achieves resource scheduling for data transmission system services through the use of declarative APIs, allows for better description of service resources through custom resources, and enables more efficient use of Kubernetes project orchestration capabilities through custom controllers. This is mainly reflected in the following aspects:

[0072] ①This invention allows multiple API writers to modify API objects using a PATCH method without needing to concern themselves with the content of the original local YAML file;

[0073] ② This invention can complete the reconcile process between the "actual state" and the "desired state" based on adding, deleting, modifying and querying API objects without any external intervention. Attached Figure Description

[0074] The invention will be further described below with reference to the accompanying drawings.

[0075] Appendix Figure 1 This is a schematic diagram illustrating the structure of a data transmission system resource scheduling method based on a declarative API. Detailed Implementation

[0076] The method for resource scheduling of a data transmission system based on a declarative API and the storage medium of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0077] Example 1:

[0078] This embodiment provides a method for resource scheduling in a data transmission system based on a declarative API. The method is as follows:

[0079] S1. Custom API Resource Type CRD: Define the resource type CRDs for the data transmission system's acquisition end resources, database entry end resources, and message middleware resources respectively; where the resource type CRD is grouped as dts;

[0080] S2. The resource description file of the corresponding resource type CRD is used to schedule the resources of the collection end, the storage end and the message middleware, and the corresponding resource scheduling work is realized by capturing the application's resource file through a custom controller.

[0081] S3. By defining resource files, the underlying physical resources and business logic resources are abstracted, and the scheduling of business resources is implemented through a custom controller.

[0082] The data transmission system in this embodiment includes a data acquisition component, a message middleware, and a data entry component. After reading data, the data acquisition component writes it to the message middleware. The message middleware caches the read data to ensure data integrity. The data acquisition component and the message middleware are deployed in the same pod. The data acquisition component and the message middleware each define a resource type CRD of type dtsReader, and the data entry component defines a resource type CRD of type dtsWriter.

[0083] The acquisition component is used to read data from the source database.

[0084] Message middleware is used for data storage and transmission;

[0085] The data import component is used to write the collected data into the destination database.

[0086] In this embodiment, the specific fields of the resource type CRD for dtsReader and dtsWriter are as follows:

[0087] taskId: A unique identifier for the data transfer task;

[0088] Version: Product version number. The version numbers of the backend database collection component and the data entry component are consistent with the product version number.

[0089] isAvailable: Whether the resource is available;

[0090] dbInfo: Data source information, connection information for the source or destination database;

[0091] task: Task configuration information, which includes table information, field information and related transmission strategy parameters;

[0092] Resources: Resource information, including CPU, memory, and storage information of the pod that needs to be started.

[0093] In this embodiment, the custom controller refers to creating a corresponding CDR controller for the resource type CRD, which triggers the business logic code within the CRD controller when the CRD object changes; specifically as follows:

[0094] (I) Creating a custom controller, dts-Operator, in the data transmission system;

[0095] (ii) The custom controller dts-Operator uses the ListAndWatch method of Reflector to obtain and listen to changes in object instances;

[0096] (iii) When any instance changes, Reflector will receive an event notification and, upon receiving the notification, will store the combination of the event and the object corresponding to the object instance into a first-in-first-out queue.

[0097] (iv) The custom controller dts-Operator uses Informer to continuously read objects from the first-in-first-out queue and determines whether the event type is dtsReader or dtsWriter:

[0098] (v) After determining the type of event, create the resources required for dtsReader or dtsWriter according to the type of event.

[0099] In this embodiment, the resources required to create dtsReader are as follows:

[0100] (1) Create the pod and service resources of the message middleware nats that the collection component depends on, using the custom controller dts-Operator;

[0101] (2) Create jetstream and consumer resources in the message middleware nats;

[0102] (3) After the NATS message middleware resources are deployed, deploy the pods, services and storage resources of the collection components;

[0103] (4) By monitoring the corresponding events, physical resources and business configurations can be processed according to custom logic;

[0104] (5) Once the processing is complete, the acquisition components required for the data transmission task can work normally;

[0105] In this embodiment, the resources required to create dtsWriter are as follows:

[0106] (1) Create the physical resources of pods and services required by the custom controller dts-Operator and generate the configuration files related to data transfer tasks;

[0107] (2) Write the information related to the message middleware NATS into the configuration file;

[0108] (3) By monitoring the corresponding events, physical resources and business configurations can be processed according to custom logic;

[0109] (4) Once the processing is complete, the data transfer task's required data entry components will function normally.

[0110] In this embodiment, the custom controller dts-Operator relies on the Informer code block to obtain dtsWriter and dtsReader objects from the Kubernetes APIServer. The Informer corresponds one-to-one with the API object. During the data transmission process, what is passed to the custom controller dts-Operator is an Informer of a Network object.

[0111] When creating the Informer factory, a networkClient is passed to the Informer factory. The NetworkInformer uses the networkClient to establish a connection with the APIServer. The connection between the Network Informer and the APIServer is maintained by the ListAndWatch mechanism in the Reflector package used by the Informer. The ListAndWatch mechanism is used to obtain and listen to changes in the Network object instance.

[0112] Under the ListAndWatch mechanism, once a new Network instance is created, deleted, or updated on the APIServer side, the Reflector will receive an event notification for the corresponding Network instance. The combination of this event and the corresponding API object is called the Delta increment and is placed in a Delta FIFO queue. At the same time, the Informer will continuously read increments from the Delta FIFO queue. Each time an increment is read, the Informer will determine the event type in the increment and create or update the buffer of local objects.

[0113] The Informer in this embodiment has the following functions:

[0114] ① Synchronize local cache; Informer comes with its own caching and indexing mechanisms; the client library that triggers the Handler, the local cache is called Store in Kubernetes, and the index is called Index; Informer uses the Reflector package, which is a client wrapper that obtains and monitors changes to API objects through the ListAndWatch mechanism; the Reflector package and Informer coordinate with each other using an incremental first-in-first-out queue;

[0115] ② Based on the event type, trigger the registered ResourceEventHandler; through the monitored event changes, Informer updates the local cache in real time and calls the corresponding EventHandler. Every time the time specified by resyncPeriod elapses, the local cache maintained by Informer will be forcibly updated using the result returned by the most recent LIST, thereby ensuring the validity of the cache; in particular, when creating a custom controller dts-Operator, it is registered with the Informer corresponding to ResourceEventHandler;

[0116] Before writing the control loop, the informer uses a work queue for coordination. In practice, all code except the control loop is automatically generated by Kubernetes. The specific logic of the informer's control loop is as follows:

[0117] (1) Wait for Informer to complete a local cache data synchronization operation;

[0118] (2) Start one or more infinite loop tasks using goroutines;

[0119] (3) In each loop cycle, the data transmission business logic code is executed:

[0120] ① If the control loop cannot obtain the corresponding object information in the Informer cache, it means that the deletion logic needs to be executed;

[0121] ② If object information is obtained from the cache, the controller mode is executed to compare the expected state with the actual state. The expected state comes from etcd, and the actual state comes from the cluster itself.

[0122] In this embodiment, the custom API resource type CRD also includes the following:

[0123] ① Perform Initializer and Validation operations on the API object: The validation operation verifies the legality of each field in the object and saves it to the Registry data structure after validation. If the definition of an API object can be found in the Registry, it means that the API corresponds to a valid k8s API object.

[0124] ② Convert the superversion object into a serialized object of the user-submitted version and save it to etcd.

[0125] As attached Figure 1 As shown, the data transmission system in this embodiment defines a management service dts-Service. The management service dts-Service is used to receive user instructions sent by the front end and interact with the back end custom controller dts-Operator through a declarative API to realize the creation of data transmission task resources and the control of task operation logic.

[0126] The creation of resources in this embodiment is as follows:

[0127] (1) Users create tasks through the front-end interface, and the front-end interface calls the management service dts-Service to create data transmission task methods through HTTP requests;

[0128] (2) The management service dts-Service accesses the Kubernetes API and applies the resource type CRDs of the data transfer tasks dtsReader and dtsWriter in sequence. The name field in the resource type CRD is consistent with the task number to distinguish the resource type CRDs of different tasks.

[0129] (3) The application's resource type CRD will be detected by the custom controller dts-Operator and the corresponding resource creation logic will be executed.

[0130] (4) After the resource is created, the database acquisition component, the database entry component and the message middleware NATS automatically complete the initialization work.

[0131] (5) After the management service dts-Service detects that the resources of the task’s collection component and storage component have been successfully allocated and initialized normally, it returns a creation success message to the user.

[0132] The operational logic control for the services in this embodiment is as follows:

[0133] (1) Users can start, stop, pause, resume, change password and delete data transmission tasks through the front-end interface; the front-end interface calls the corresponding methods in the management service dts-Service through HTTP requests;

[0134] (2) The management service dts-Service assembles the resource type CRDs of dtsReader and dtsWriter corresponding to the data transmission task respectively. The name field of the resource type CRD is consistent with the task number to distinguish the resource type CRDs of different tasks.

[0135] (3) The operation field in the assembled resource type CRD corresponds to the user's operation. For example, when the operation is stopped, the value of the operation field is stopTask.

[0136] (4) The management service dts-Service accesses the Kubernetes API and applies the resource type CRD of the task dtsReader and dtsWriter respectively;

[0137] (5) After the management service dts-Operator detects the change in the resource type CRD version, it analyzes the content of the operation field to determine the user's operation. Based on the operation type, it calls the corresponding gRPC interface of the collection component and the database component to complete the user's operation.

[0138] Example 2:

[0139] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the data transmission system resource scheduling method based on a declarative API according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0140] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0141] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0142] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0143] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for resource scheduling in a data transmission system based on a declarative API, characterized in that, The method is as follows: Custom API Resource Type CRD: Defines the resource type CRDs for the data transmission system's acquisition end resources, database entry end resources, and message middleware resources; where the resource type CRD is grouped as dts; The application uses the resource description file of the corresponding resource type CRD to schedule the resources of the collection end, the database end, and the message middleware, and uses a custom controller to capture the application's resource file to achieve the corresponding resource scheduling work; The underlying physical resources and business logic resources are abstracted by defining resource files, and the scheduling of business resources is realized through a custom controller. The data transmission system includes a data acquisition component, a message middleware, and a data entry component. The data acquisition component reads data and writes it to the message middleware. The message middleware caches the read data to ensure data integrity. The data acquisition component and the message middleware are deployed in the same pod. The data acquisition component and the message middleware each define a resource type CRD of type dtsReader, and the data entry component defines a resource type CRD of type dtsWriter. The acquisition component is used to read data from the source database. Message middleware is used for data storage and transmission; The data import component is used to write the collected data into the destination database; The specific fields of the resource type CRD for dtsReader and dtsWriter are as follows: taskId: A unique identifier for the data transfer task; Version: Product version number. The version numbers of the backend database collection component and the data entry component are consistent with the product version number. isAvailable: Whether the resource is available; dbInfo: Data source information, connection information for the source or destination database; task: Task configuration information, which includes table information, field information and related transmission strategy parameters; resources: resource information, including CPU, memory and storage information of the pod that needs to be started; A custom controller refers to creating a corresponding CDR controller for a resource type CRD, which triggers the business logic code within the CRD controller when the CRD object changes; specifically as follows: The data transmission system creates a custom controller, dts-Operator; The custom controller dts-Operator uses the ListAndWatch method of Reflector to get and listen for changes in object instances; When any instance changes, Reflector will receive an event notification and, upon receiving the notification, will store the combination of the event and the object corresponding to the object instance into a first-in-first-out queue. The custom controller dts-Operator uses Informer to continuously read objects from a first-in-first-out queue and determines whether the event type is dtsReader or dtsWriter: After determining the type of event, create the resources required for dtsReader or dtsWriter based on the event type; The resources required to create a dtsReader are as follows: The custom controller dts-Operator creates the pod and service resources of the message middleware nats that the collection component depends on; In the message middleware NATS, create JetStream and Consumer resources; After the NATS message middleware resources are deployed, deploy the pods, services, and storage resources of the data collection components. By monitoring the corresponding events, physical resources and business configurations can be processed according to custom logic; Once the processing is complete, the acquisition components required for the data transmission task will function normally. The resources required to create dtsWriter are as follows: The custom controller dts-Operator creates the physical resources of the pods and services required by the inbound component, and generates configuration files related to the data transfer task. Write information related to the NATS message middleware to the configuration file; By monitoring the corresponding events, physical resources and business configurations can be processed according to custom logic; Once the processing is complete, the data transfer task's required data import components will function normally. The custom controller dts-Operator relies on the Informer code block to obtain dtsWriter and dtsReader objects from the Kubernetes APIServer. There is a one-to-one correspondence between Informer and API object. During the data transmission process, what is passed to the custom controller dts-Operator is an Informer of a Network object. When creating the Informer factory, a networkClient is passed to the Informer factory. The NetworkInformer uses the networkClient to establish a connection with the APIServer. The connection between the Network Informer and the APIServer is maintained by the ListAndWatch mechanism in the Reflector package used by the Informer. The ListAndWatch mechanism is used to obtain and listen to changes in the Network object instance. Under the ListAndWatch mechanism, once a new Network instance is created, deleted, or updated on the APIServer side, the Reflector will receive an event notification for the corresponding Network instance. The combination of this event and the corresponding API object is called the Delta increment and is placed in a Delta increment first-in-first-out queue. At the same time, the Informer will continuously read increments from the Delta increment first-in-first-out queue. Each time an increment is read, the Informer will determine the event type in the increment and create or update the local object buffer. Informer has the following functions: ① Synchronize local cache; Informer comes with its own caching and indexing mechanisms; the client library that triggers the Handler, the local cache is called Store in Kubernetes, and the index is called Index; Informer uses the Reflector package, which is a client wrapper that obtains and monitors changes to API objects through the ListAndWatch mechanism; the Reflector package and Informer coordinate with each other using an incremental first-in-first-out queue; ② Based on the event type, the registered ResourceEventHandler is triggered. Through the monitored event changes, the Informer updates the local cache in real time and calls the corresponding EventHandler. Every time the time specified by `resyncPeriod` elapses, the local cache maintained by the Informer is forcibly updated using the result returned by the most recent LIST, thus ensuring the validity of the cache. Specifically, when creating a custom controller (dts-Operator), it is registered with the Informer corresponding to the ResourceEventHandler. Before writing the control loop, the informer uses a work queue for coordination; the specific logic of the informer control loop is as follows: Waiting for Informer to complete a local cache data synchronization operation; Start one or more infinite loop tasks using goroutines; In each loop cycle, the data transmission business logic code is executed; If the control loop cannot retrieve the information of the corresponding object in the Informer cache, it means that deletion logic needs to be executed; If object information is obtained from the cache, the controller mode is executed to compare the expected state with the actual state. The expected state comes from etcd, and the actual state comes from the cluster itself. Custom API resource type CRDs also include the following: Perform Initializer and Validation operations on API objects: The validation operation verifies the legality of each field in the object and saves it to the Registry data structure after validation. If the definition of an API object can be found in the Registry, it means that the API corresponds to a valid k8s API object. The superversion object is converted into a serialized object of the user-submitted version and then saved to etcd; The data transmission system defines a management service dts-Service. The management service dts-Service is used to receive user commands sent by the front end and interact with the back end custom controller dts-Operator through a declarative API to realize the creation of data transmission task resources and the control of task operation logic. The creation of resources is detailed below: Users create tasks through the front-end interface, which in turn calls the management service dts-Service via an HTTP request to create the data transmission task method. The management service dts-Service accesses the Kubernetes API and applies the resource type CRDs of the data transfer tasks dtsReader and dtsWriter in sequence. The name field in the resource type CRD is consistent with the task number to distinguish the resource type CRDs of different tasks. The application's resource type CRD will be detected by the custom controller dts-Operator and the corresponding resource creation logic will be executed. After the resources are created, the database's acquisition component, storage component, and message middleware NATS automatically complete the initialization process. After the management service dts-Service detects that the resources of the task's data collection component and data storage component have been successfully allocated and initialized normally, it returns a creation success message to the user. The specific operational logic control for the business is as follows: Users can start, stop, pause, resume, change passwords, and delete data transmission tasks through the front-end interface; the front-end interface calls the corresponding methods in the management service dts-Service via HTTP requests; The management service dts-Service assembles the resource type CRDs of dtsReader and dtsWriter corresponding to the data transmission task. The name field of the resource type CRD is consistent with the task number to distinguish the resource type CRDs of different tasks. The operation field in the assembled resource type CRD corresponds to the user's operation; The management service dts-Service accesses the Kubernetes API to apply the resource type CRD for the task dtsReader and dtsWriter respectively; After the management service dts-Operator detects a change in the resource type CRD version, it analyzes the content of the operation field to determine the user's operation. Based on the operation type, it calls the corresponding gRPC interface of the data collection component and the data entry component to complete the user's operation.

2. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method for resource scheduling of a data transmission system based on a declarative API as described in claim 1.

Images