Method and system for synchronizing Kubernetes objects to database, storage medium and electronic equipment
By obtaining snapshots and incremental updates of Kubernetes objects, building a mapping relationship between objects and database tables, establishing a real-time synchronization link, the problem of consistent maintenance of Kubernetes and database data in the Web backend system is solved, and automated synchronization is achieved, reducing the Development and operation and maintenance costs are eliminated and functional development restrictions are eliminated.
Patent Information
- Application Number
- CN202411164687.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-12
AI Technical Summary
In the web backend system, the object interface provided by Kubernetes cannot be compatible with the SQL interface, resulting in increased development difficulty and limited functions. In the existing technology, maintaining data consistency between the two sides is prone to errors, which increases development and maintenance costs.
By obtaining snapshots and incremental updates of Kubernetes objects, the mapping relationship between object types and database tables is built, and Kubernetes objects are converted into database records are established to establish a real-time synchronization link to achieve automated and real-time data synchronization.
Reduces data consistency maintenance costs between web backend systems and Kubernetes and databases, allows the use of mature ORM and database connection pooling technologies, eliminates functional development limitations, and improves the reliability and efficiency of data synchronization.
Smart Images

Figure CN118939733A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of database data synchronization, and in particular to a method and system for synchronizing Kubernetes objects to a database. Background Art
[0002] MVC is a commonly used design pattern in Web backend development, in which the Model layer usually requires SQL and database to implement. In Web backend systems that need to interact with Kubernetes, since the interface provided by Kubernetes is object-based rather than SQL interface, this leads to the following problems:
[0003] 1. Web backend development cannot use widely used mature technologies such as ORM and database connection pool. It needs to be developed and packaged separately for Kubernetes, which greatly increases the difficulty and workload of development;
[0004] 2. The interface provided by Kubernetes cannot implement common query capabilities of the Web backend, such as association query, filtering, and sorting, which limits the functionality of the Web backend;
[0005] In order to effectively solve the above problems, in the existing development method, when performing write operations, a copy of the data is usually stored in the Web backend database and Kubernetes respectively, and the data consistency on both sides is maintained by the backend program. This method is prone to errors and ultimately leads to increased development and maintenance costs for the Web backend. Summary of the invention
[0006] The purpose of this application is to overcome the problems of data error and increased Web backend development and maintenance costs caused by storing a copy of data in the Web backend database and Kubernetes respectively and maintaining the consistency of data on both sides through backend programs in the prior art, and to provide a method and system for synchronizing Kubernetes objects to a database.
[0007] In a first aspect, a method for synchronizing a Kubernetes object to a database is provided, including:
[0008] Get snapshots and incremental updates of Kubernetes objects;
[0009] Build a mapping relationship between the object type of Kubernetes objects and the database table;
[0010] The Kubernetes objects in the snapshot and incremental update are converted into records in the database table through the synchronization link according to the mapping relationship.
[0011] Some possible implementations of taking snapshots and incremental updates of Kubernetes objects include:
[0012] Get the object type of the Kubernetes object that needs to be synchronized;
[0013] Traverse all Kubernetes objects in Kubernetes and filter out the first object of the object type;
[0014] Get a snapshot of the first object through the list interface;
[0015] sorting the first objects in the snapshot in ascending order according to the ResourceVersion field of the first objects;
[0016] Put the first object into the real-time synchronization queue in ascending order;
[0017] Use the watch interface to obtain the ResourceVersion value of the last first object in ascending order as the incremental update of the starting point;
[0018] In response to receiving the incremental update, the incrementally updated object is put into the real-time synchronization queue.
[0019] In some possible implementations, if the object type of the Kubernetes object to be synchronized is not obtained, all Kubernetes objects in Kubernetes are used as the first object.
[0020] In some possible implementations, the mapping relationship between the object type of the Kubernetes object and the database table is constructed, including:
[0021] Determine whether a database table with the same name as the object type already exists in the database table. If not, create a database table with the same name as the object type.
[0022] Obtaining the correspondence between the fields in the Kubernetes object and the columns in the database table. If the correspondence between the fields in the Kubernetes object and the columns in the database table is not obtained, expanding the Kubernetes object into secondary fields, and automatically mapping the expanded fields to a column in the database table;
[0023] Get the unique ID column name and value in the database table corresponding to Kubernetes. If the unique ID column name and value in the database table corresponding to Kubernetes are not obtained, a unique ID column name and value are automatically generated.
[0024] In some possible implementations, the Kubernetes objects in the snapshot and incremental update are converted into records in a database table through a synchronization link according to the mapping relationship, including: converting the Kubernetes object fields to columns in the database table one by one according to the mapping relationship, wherein data type conversion is performed for each data type during the conversion process.
[0025] In some possible implementations, the database table performs a write power operation when writing data.
[0026] In some possible implementations, if the synchronization link is interrupted, the synchronization link is automatically or manually restarted.
[0027] In a second aspect, a system for synchronizing Kubernetes objects to a database is provided, including:
[0028] The snapshot module is used to obtain snapshots and incremental updates of Kubernetes objects;
[0029] Configuration module, which is used to build the mapping relationship between the object type of Kubernetes objects and the database table;
[0030] A conversion module is used to convert the Kubernetes objects in the snapshot and incremental update into records in a database table through a synchronization link according to the mapping relationship.
[0031] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes steps for executing the method in any one of the implementations of the first aspect described above.
[0032] In a fourth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations in the first aspect above.
[0033] This application has the following beneficial effects: This application builds a data synchronization link that automatically and in real time synchronizes objects in Kubernetes to the database. All write operations of the Web backend system can interact directly with Kubernetes, and all read operations can interact with the database through SQL. The Web backend system no longer needs to maintain data consistency between Kubernetes and the database, so that the development and operation and maintenance costs of the Web backend system that interacts with Kubernetes are greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0036] Figure 1 is a flowchart of a method for synchronizing a Kubernetes object to a database according to Embodiment 1 of the present application;
[0037] Figure 2 This is a schematic diagram of the interaction between Kubernetes and the Web backend where the database is located in the method for synchronizing Kubernetes objects to a database in Example 1 of the present application;
[0038] Figure 3 This is a structural block diagram of a system for synchronizing Kubernetes objects to a database in Example 2 of the present application;
[0039] Figure 4 It is a schematic diagram of the internal structure of the electronic device of Example 4 of the present application.
[0040] Reference numerals:
[0041] 100, snapshot module; 200, configuration module; 300, conversion module. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] Example 1
[0044] A method for synchronizing Kubernetes objects to a database involved in Example 1 of the present application includes: obtaining snapshots and incremental updates of Kubernetes objects; building a mapping relationship between object types of Kubernetes objects and database tables; and converting Kubernetes objects in the snapshots and incremental updates into records in the database table through a synchronization link according to the mapping relationship.
[0045] Specifically, Figure 1 A flowchart of a method for synchronizing Kubernetes objects to a database in application embodiment 1 is shown, including:
[0046] S100, obtain snapshots and incremental updates of Kubernetes objects;
[0047] Specifically, Kubernetes provides two interfaces, list and watch. The list interface can obtain the stock snapshot of a single type of object, and the watch interface can obtain the incremental update of a single type of object. With the help of the list and watch interfaces, snapshots and incremental updates of all Kubernetes objects can be obtained through the following process:
[0048] S101, providing a configuration interface, allowing the user to configure the object type that needs to be synchronized;
[0049] S102. If the user has not configured, all types of objects will be synchronized by default. At this time, all object types are discovered through the Kubernetes discovery interface;
[0050] S103, traverse all object types obtained in step S101 or step S102, and perform the following operations on each object type:
[0051] S104, obtaining snapshots of all object types obtained in step S103 through the list interface;
[0052] S105. Sort the Kubernetes objects in the snapshot in step S104 in ascending order, based on the ResourceVersion field of each Kubernetes object. All Kubernetes objects in Kubernetes have a ResourceVersion field. All Kubernetes objects in Kubernetes are ultimately stored in the etcd metabase. etcd adds a timestamp of the last modification to each object. The timestamp is issued by a logical clock in the form of ResourceVersion: "234331", which is a strictly increasing number. Sorting in ascending order according to the ResourceVersion field can sort all Kubernetes objects by time.
[0053] S106. Put the Kubernetes objects into a real-time synchronization queue in sequence according to the sorted order. The queue is a Queue often mentioned in computer data structures. Real-time synchronization means that obtaining the object list and processing the list in sequence are done by two modules (or threads). Data is transferred (synchronized) between the two modules through a Queue. Real-time synchronization means that the Kubernetes objects will be processed immediately after being put into the queue.
[0054] S107. Obtain an incremental update of the object type through the watch interface and the ResourceVersion value of the last Kubernetes object sorted in step S105.
[0055] S108. If an incremental update is received, the object is put into a real-time synchronization queue.
[0056] In the above embodiment, all existing Kubernetes objects are obtained through the list interface, and the Kubernetes objects can be sorted by time, and then processed one by one in chronological order; after processing, the last Kubernetes object in the sorting is found according to the timestamp, and then the new changes after this time are further obtained through the watch interface, and continue to process them one by one.
[0057] S200, construct a mapping relationship between the object type of the Kubernetes object and the database table;
[0058] Specifically, the object type in Kubernetes is the basis for building the mapping relationship. The mapping relationship can be built through the following process:
[0059] S201, first check whether a table with the same name as the object type already exists in the database, if not, create it;
[0060] S202, providing a configuration interface, allowing the user to configure the correspondence between the fields in the object and the columns in the table;
[0061] S203. If the user has not configured, the object is expanded to the secondary field by default, and the expanded field is automatically mapped to a column in the table;
[0062] S204. Provide a configuration interface, and let the user configure the unique ID column name and value in the table corresponding to the Kubernetes;
[0063] S205. If the user has not configured, the default unique ID column name is "kubernetes_id", and the value is determined by a unique ID generation algorithm. This application does not limit the specific generation algorithm, but only requires that the algorithm can ensure the uniqueness of the generated ID.
[0064] S300. Convert the Kubernetes objects in the snapshot and incremental update into records in a database table through a synchronization link according to the mapping relationship, wherein the database where the database table is located is set in the Web backend.
[0065] Specifically, after the mapping relationship is established, the object fields can be converted to table columns one by one according to the mapping relationship. Because the data types and value ranges of the fields in the Kubernetes object do not completely correspond to the data types and value ranges supported by the database table, each data type needs to be processed during the conversion process to ensure the correctness of the data.
[0066] The present application does not limit the specific data type conversion algorithm, but only requires that the data type conversion algorithm ensure that the meaning of the data before and after the conversion remains unchanged.
[0067] In this embodiment, the consistency between the data in the database and the Kubernetes objects in Kubernetes is ensured through the following design:
[0068] 1. When obtaining stock snapshots and incremental updates, ResourceVersion is used to ensure accurate connection between the two, and object data is not lost or duplicated;
[0069] 2. The writing of table records is idempotent. If abnormal factors such as network jitter cause the converted table records to be delivered repeatedly, writing idempotent can ensure data consistency.
[0070] 3. Through the Kubernetes unique ID design, it is ensured that when objects in multiple Kubernetes are synchronized to the same database, objects with the same object name in different Kubernetes will not overwrite each other;
[0071] 4. In the event of a network interruption, machine downtime, or accidental human deletion, the synchronization link can be automatically or manually restarted after the interruption to restore the lost data.
[0072] like Figure 2 As shown, this embodiment builds a data synchronization link that automatically and in real time synchronizes objects in Kubernetes to the database, so that read operations can be directly read from the database, and write operations can be directly written to Kubernetes, thereby eliminating the need to write data to the database. This not only avoids the data error problem caused by double writing, but also greatly reduces the development and operation and maintenance costs of the Web backend system that interacts with Kubernetes, specifically including:
[0073] 1. Web backend development can reuse mature and widely used ORM, database connection pool and other technologies;
[0074] 2. Use SQL to query, filter, and sort Kubernetes objects, eliminating limitations on Web backend function development;
[0075] 3. The Web backend only writes one copy of the data, and there is no need to solve the consistency problem introduced by double writing of data. The Web backend no longer needs to maintain data consistency between Kubernetes and the database.
[0076] Example 2
[0077] like Figure 3 As shown, a system for synchronizing Kubernetes objects to a database according to Embodiment 2 of the present application includes:
[0078] A snapshot module 100 is used to obtain snapshots and incremental updates of Kubernetes objects;
[0079] Configuration module 200, used to build a mapping relationship between the object type of the Kubernetes object and the database table;
[0080] The conversion module 300 is used to convert the Kubernetes objects in the snapshot and incremental update into records in the database table through the synchronization link according to the mapping relationship.
[0081] It should be noted that other specific implementation methods of the system for synchronizing Kubernetes objects to a database in this embodiment can refer to the specific implementation methods of the method for synchronizing Kubernetes objects to a database mentioned above. To avoid redundancy, they are not described here.
[0082] Example 3
[0083] A computer-readable storage medium according to Embodiment 3 of the present application, wherein the computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing a method in any one of the implementations in Embodiment 1 of the present application;
[0084] Among them, the computer-readable storage medium can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM); the computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by the processor, the processor is used to execute the steps of the method in any one of the implementation methods in Example 1 of the present application.
[0085] Example 4
[0086] like Figure 4 As shown, an electronic device involved in Embodiment 4 of the present application includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, a method in any one of the implementations in Embodiment 1 of the present application is implemented;
[0087] Among them, the processor can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method in any one of the implementation methods in Example 1 of the present application.
[0088] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of Embodiment 1 of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in software form.
[0089] The above-mentioned processor may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the functions required to be performed by the unit included in the data processing device of the embodiment of the present application in combination with its hardware, or executes the method in any one of the implementation modes in the embodiment 1 of the present application.
[0090] The above are only preferred specific implementations of the present application; however, the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved ideas of the present application within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A method for synchronizing Kubernetes objects to a database, characterized in that: include: Get snapshots and incremental updates of Kubernetes objects; Build a mapping relationship between the object type of Kubernetes objects and the database table; The Kubernetes objects in the snapshot and incremental update are converted into records in the database table through the synchronization link according to the mapping relationship.
2. The method for synchronizing Kubernetes objects to a database according to claim 1, characterized in that: Get snapshots and incremental updates of Kubernetes objects, including: Get the object type of the Kubernetes object that needs to be synchronized; Traverse all Kubernetes objects in Kubernetes and filter out the first object of the object type; Get a snapshot of the first object through the list interface; sorting the first objects in the snapshot in ascending order according to the ResourceVersion field of the first objects; Put the first object into the real-time synchronization queue in ascending order; Use the watch interface to obtain the ResourceVersion value of the last first object in ascending order as the incremental update of the starting point; In response to receiving the incremental update, the incrementally updated object is put into the real-time synchronization queue.
3. The method for synchronizing Kubernetes objects to a database according to claim 2, characterized in that: If the object type of the Kubernetes object to be synchronized is not obtained, all Kubernetes objects in Kubernetes are used as the first object.
4. The method for synchronizing Kubernetes objects to a database according to claim 1, characterized in that: Construct the mapping relationship between the object type of the Kubernetes object and the database table, including: Determine whether a database table with the same name as the object type already exists in the database table. If not, create a database table with the same name as the object type. Obtaining the correspondence between the fields in the Kubernetes object and the columns in the database table. If the correspondence between the fields in the Kubernetes object and the columns in the database table is not obtained, expanding the Kubernetes object into secondary fields, and automatically mapping the expanded fields to a column in the database table; Get the unique ID column name and value in the database table corresponding to Kubernetes. If the unique ID column name and value in the database table corresponding to Kubernetes are not obtained, a unique ID column name and value are automatically generated.
5. The method for synchronizing Kubernetes objects to a database according to claim 1, characterized in that: The Kubernetes objects in the snapshot and incremental update are converted into records in a database table through a synchronization link according to the mapping relationship, including: converting the Kubernetes object fields to columns in the database table one by one according to the mapping relationship, wherein data type conversion is performed for each data type during the conversion process.
6. The method for synchronizing Kubernetes objects to a database according to claim 1, characterized in that: The database table performs a write power operation when writing data.
7. The method for synchronizing Kubernetes objects to a database according to claim 1, characterized in that: If the synchronization link is interrupted, the synchronization link is automatically or manually restarted.
8. A system for synchronizing Kubernetes objects to a database, characterized in that: include: The snapshot module is used to obtain snapshots and incremental updates of Kubernetes objects; Configuration module, which is used to build the mapping relationship between the object type of Kubernetes objects and the database table; A conversion module is used to convert the Kubernetes objects in the snapshot and incremental update into records in a database table through a synchronization link according to the mapping relationship.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing the method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the method as claimed in any one of claims 1 to 7 when executed by the processor.
