Database management system, model and data management system and method

Abstract

The invention provides a database management system, a model and a data management system and method. The database management system is used for providing a document management capability under the constraint of a YANG model and comprises a model management module and a data management module, and the model management module is used for providing a management function of the YANG model; the data management module is used for providing a transaction-based document management function; wherein the database management system takes a data container as a basic unit of model management and data management, and document data in the data container conforms to constraints of a YANG model in the data container. The management function of the YANG model and the management function of the document under the constraint of the YANG model can be integrated on one running entity, and the data container is used as a basic management unit, so that the integrity verification can be performed on the document in the data container based on the YANG model in the data container, and the integrity verification efficiency is improved. And the document in the data container always conforms to the constraint of the YANG model.

Description

Technical Field

[0001] This application relates to the field of network automation technology, and in particular, to a database management system, a model, and a data management system and method. Background Art

[0002] With the rapid development of network technology, various devices have emerged in the network, and the network scale has been continuously expanding. Network automation (such as automatic service provisioning on demand and automated operation and maintenance, etc.) has gradually become one of the key requirements for the network in the cloud era. To meet the needs of cloudified networks, a more secure and more operable network configuration protocol (Network Configuration Protocol, NETCONF) has gradually become the mainstream in the field of network automation. The NETCONF protocol usually uses the Yet Another Next Generation (YANG) data modeling language to define devices and operation processes, resulting in a large number of documents (such as XML documents, JSON documents, etc.) constrained (or described, defined) by the YANG model in the network automation process. It is necessary to manage these documents. Summary of the Invention

[0003] This application provides a database management system, a model, and a data management system and method for solving the problem of how to manage documents constrained by the YANG model.

[0004] To solve the above technical problems, this application is implemented as follows: In a first aspect, a database management system is provided for providing document management capabilities under the constraint of the YANG model, including: A model management module for providing management functions of the YANG model; A data management module for providing transaction-based document management functions; Wherein, the system uses a data container as the basic unit for model management and data management, and the document data in the data container conforms to the constraints of the YANG model in the data container.

[0005] In a second aspect, a model and data management system is provided, including a business application and the database management system described in the first aspect. The database management system takes the form of a function library and is embedded in the business application by means of linking to provide the business application with document management capabilities under the constraint of the YANG model.

[0006] In a third aspect, a model management method is provided, which is applied to the model and data management system described in the second aspect, including: Receive a model management request, where the model management request includes the name of a first data container, and the model management request is used to request the management of the YANG model in the first data container; Manage the YANG model in the first data container according to the model management request.

[0007] In a fourth aspect, a data management method is provided, which is applied to the model and data management system described in the second aspect, and includes: Receive a processing request for a first transaction, where the first transaction is used to manage the document data in a data container; Process the first transaction according to the processing request.

[0008] In a fifth aspect, an electronic device is provided, including: A processor; A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the instructions to implement the method described in the third aspect or the fourth aspect.

[0009] In a sixth aspect, a computer-readable storage medium is provided. When the instructions in the storage medium are executed by the processor of an electronic device, the electronic device can execute the method described in the third aspect or the fourth aspect.

[0010] In a seventh aspect, a computer program product is provided. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the method described in the third aspect, or execute some or all of the steps in the method described in the fourth aspect.

[0011] In the embodiments of the present application, since the database management system includes a model management module and a data management module, the model management module is used to provide the management function of the YANG model, and the data management module is used to provide the transaction-based document management function, that is, the management function of the YANG model and the management function of the documents under the constraints of the YANG model are integrated on one running entity. Therefore, the unified management and scheduling of the YANG model and the documents under the constraints of the YANG model can be realized. Since the database management system uses data containers as the basic units for document management and model management, when managing the documents under the constraints of the YANG model (such as document insertion, modification, or query, etc.), the integrity verification of the documents in the data container can be performed based on the YANG model in the data container, ensuring that the documents in the data container always comply with the constraints of the YANG model in the data container, and there is no problem of inconsistency between the YANG model and the document data. In addition, since the data management module can provide the transaction-based document management function, that is, it can provide the native ACID transactions designed for the documents under the constraints of the YANG model, the business side does not need to implement the relevant logic, and only needs to define the transactions according to the business process, with less intrusion into the business code and higher security. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] Figure 1 is a schematic diagram of the application environment for managing documents under the constraints of the YANG model in the related art; Figure 2 is a schematic diagram of the application environment for managing documents under the constraints of the YANG model in an embodiment of the present application; Figure 3 is a schematic diagram of the structure of the database management system in an embodiment of the present application; Figure 4 is a schematic diagram of the transaction manager in an embodiment of the present application; Figure 5 is a schematic diagram of the staging area in an embodiment of the present application; Figure 6 is a schematic diagram of the version chain in an embodiment of the present application; Figure 7 is a schematic diagram of the relationship between the components in the data management module in an embodiment of the present application; Figure 8Schematic diagram of an embodiment model and data management system of the present application; Figure 9 Schematic diagram of an embodiment model and data management system of the present application; Figure 10 Schematic flowchart of an embodiment model management method of the present application; Figure 11 Schematic flowchart of an embodiment model insertion process of the present application; Figure 12 Schematic diagram of an embodiment model insertion of the present application; Figure 13 Schematic flowchart of an embodiment model export process of the present application; Figure 14 Schematic flowchart of an embodiment mode modification process of the present application; Figure 15 Schematic flowchart of an embodiment data management method of the present application; Figure 16 Schematic flowchart of an embodiment general transaction execution process of the present application; Figure 17 Schematic flowchart of an embodiment transaction start operation execution process of the present application; Figure 18 Schematic flowchart of an embodiment document insertion operation execution process of the present application; Figure 19 Schematic flowchart of an embodiment document query operation execution process of the present application; Figure 20 Schematic flowchart of an embodiment transaction submission operation execution process of the present application; Figure 21 Schematic flowchart of an embodiment fault recovery operation execution process of the present application; Figure 22 Schematic diagram of an embodiment electronic device structure of the present application; Figure 23 Schematic diagram of an embodiment model management device structure of the present application; Figure 24 Schematic diagram of an embodiment electronic device structure of the present application; Figure 25 Schematic diagram of an embodiment data management device structure of the present application. Detailed implementation manners

[0014] With the rapid development of network technology, different types, manufacturers, and models of devices coexist in the network, and the network scale is also increasing continuously. Network automation (such as automatic service provisioning on demand or automated operation and maintenance, etc.) has gradually become one of the key requirements for networks in the cloud era. In this context, the traditional Simple Network Management Protocol (SNMP) can no longer meet the needs of cloudified networks, and the more secure and more operable NETCONF has gradually become the mainstream in the field of network automation. The NETCONF protocol usually uses the YANG data modeling language to define devices and operation processes, resulting in a large number of documents (which can be represented as YANG model modeling data, such as XML documents or JSON documents) constrained (or described) by the YANG model in the network automation process, thus giving rise to the management requirements for documents under the YANG model constraint.

[0015] In related technologies, when managing documents under the YANG model constraint, the current mainstream database management systems can be used for hosting, such as using a relational database management system (RDBMS) or a document database management system for hosting. However, these systems do not support defining data constraints using the YANG model, and business application programs have to additionally implement YANG model management and data verification. Taking the management of documents under the YANG model constraint using a relational database management system as an example, reference can be made to Figure 1 .

[0016] Figure 1 It is a schematic diagram of the application environment for managing documents under the YANG model constraint in related technologies. Figure 1 In, the network administrator uses the NETCONF protocol for network automation configuration, and the business application program saves the document data generated during the network automation configuration process in the RDBMS by interacting with the relational database management system (hereinafter abbreviated as RDBMS). Among them, the logical component for interaction between the business application program and the RDBMS can be abstracted as the YANG proxy layer. The YANG proxy layer can usually be implemented by a specific business application program. In different business application programs, the YANG proxy layer can be presented in different forms, but at least the following functions need to be provided: (1) Construct a YANG model schema tree according to the YANG model provided by the business application program, and use the constructed YANG model schema tree to complete the verification of the documents provided by the business application program; (2) Create a database and tables under the relational model in the RDBMS to store the verified document data.

[0017] However, in practical applications, the above solution for managing documents under the YANG model constraints using an RDBMS highly depends on the RDBMS. The RDBMS is a database storage system designed specifically for the storage and management of relational model data. It fails to fully consider the YANG model and its data characteristics during design. Therefore, it cannot provide a high enough read / write throughput in complex scenarios, which is mainly reflected in the following aspects: (1) The YANG agent layer responsible for model management and the RDBMS responsible for document management are physically separated. The system needs to periodically verify the consistency between the YANG model schema and the document data to ensure the normal operation of each module in the system; (2) It is difficult to convert the YANG model schema with complex hierarchies into a simple relational model, thus unable to provide native data definitions and integrity constraints. The data modeled by the YANG model usually needs to be converted into tuples with complex structures before being stored in the RDBMS, which will bring unnecessary performance losses; (3) The RDBMS does not have native ACID (atomicity, consistency, isolation, durability) transactions designed for the management of YANG-modeled data. The business side needs to implement the relevant logic by itself, which has a large intrusion on the business code.

[0018] In the embodiments of this application, reasonable abstractions are made for the key and repetitive processes in the YANG modeling data management process. These processes include: YANG model schema tree maintenance, integrity verification of XML or JSON documents, document reading and writing, and ACID transactions, etc. A database management system specifically designed for documents under YANG model constraints is proposed. The database management system includes a model management module and a data management module. The model management module is used to provide management functions for the YANG model, and the data management module is used to provide transaction-based document management functions. The system uses a data container as the basic unit for model management and data management, and the document data in the data container conforms to the constraints of the YANG model in the data container. In this way, since the management functions of the YANG model and the management functions of the documents under the YANG model constraints can be integrated on one running entity, the unified management and scheduling of the YANG model and the documents under the YANG model constraints can be realized; since the database management system uses a data container as the basic unit for document management and model management, when managing the documents under the YANG model constraints (such as document insertion, modification, or query, etc.), the integrity verification of the documents in the data container can be performed based on the YANG model in the data container to ensure that the documents in the data container always conform to the constraints of the YANG model in the data container, and there is no problem of inconsistency between the YANG model and the document data. In addition, since the data management module can provide transaction-based document management functions, that is, it can provide native ACID transactions designed for documents under the YANG model constraints, the business side does not need to implement the relevant logic, and only needs to define transactions according to the business process, which has less intrusion into the business code and higher security.

[0019] Based on the database management system provided in the embodiments of this application, the embodiments of this application also provide a model and data management system, including a business application program and the database management system provided in the embodiments of this application. The database management system can be in the form of a function library and be embedded into the business application program by linking to provide the business application program with the ability to manage document data under the YANG model constraints. In addition, the embodiments of this application also provide a model management method and a document management method applicable to the model and data management system, which can realize the management of the YANG model and the management of the documents under the YANG model constraints.

[0020] The database management system provided by the embodiments of the present application can specifically take the form of a C function library (or a C function library and its header files), and users can complete the document management process under the constraints of the YANG model based on the function library. In a specific implementation, the database management system provided by the embodiments of the present application can be embedded in a specific business application, and the business application can be deployed on a network device (such as a resource-constrained network device, and of course it can also be deployed on other network devices, which is not specifically limited here).

[0021] Figure 2 It is a schematic diagram of the application environment for managing documents under the constraints of the YANG model in an embodiment of the present application. As Figure 2 shown, the database management system provided by the embodiments of the present application can be deployed on a network device together with the business application to provide the business application with the management ability of document data under the constraints of the YANG model. The business application will work as a NETCONF server, and the network administrator can use the NETCONF protocol to complete configuration, distribution, and modification of the target device, and the relevant data processing will be completed by the database management system. The communication between the network administrator and the network device depends on the interconnected network. Among them, the business application can be written in C or C++ language. Assuming that the business application is developed in C language, during the coding, compilation, and linking stages, necessary development tools (such as Vscode), dependent function libraries (such as glibc), and compilation toolchains (such as gcc) need to be installed on the developer's host. Using these tools, an executable program that can run on the target device can be generated to achieve the management of documents under the constraints of the YANG model.

[0022] It should be noted that Figure 2 compared with the application environment shown in Figure 1 , the database management system replaces the original positions of the YANG proxy layer and RDBMS, and the business application can directly use the interfaces provided by the database management system to implement model and data management, which is more concise in form. However, it should be noted that this replacement is not simply organizing the original YANG proxy layer and RDBMS together, but a database management system with complete functions redesigned for the YANG model and its modeling data characteristics.

[0023] In the process of managing documents under the YANG model constraint, the technical solution provided by the embodiments of the present application does not require relational database management. Users no longer need to define the conversion from the YANG model schema to the relational model. They only need to correctly establish the YANG model schema by calling the model management interface of the database management system's library functions, and then they can manage XML documents or JSON documents under the YANG model constraint. This can effectively alleviate various difficulties existing in the execution of the YANG modeling data management process in the related art, greatly improve the data read / write throughput, and at the same time, have low resource requirements for devices, effectively alleviating the operation pressure on the devices.

[0024] In order to enable those skilled in the art to better understand the technical solutions in the present application, the following will clearly and completely describe the technical solutions in the present application with reference to the accompanying drawings in one or more embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application.

[0025] The terms "first", "second", etc. in the present application and the claims are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the present application and the claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0026] The following will describe in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

[0027] Figure 3 It is a schematic structural diagram of a database management system according to an embodiment of the present application. Figure 3 The shown database management system 30 is used to provide the document management ability under the YANG model constraint, including a model management module 31 and a data management module 32. The model management module 31 is used to provide the management function of the YANG model, and the data management module 32 is used to provide the transaction-based document management function. Among them, the database management system 30 uses a data container (which can also be expressed as a YANG data container, Figure 1 not shown) as the basic unit for model management and data management. The data container is an abstraction of a real network device. The data container can include the YANG model describing this type of device and accommodate the document data verified by the YANG model (such as XML or JSON document data). For any data container, the document data in the data container conforms to the constraints of the YANG model in the data container.

[0028] Since the management function of the YANG model and the management function of the documents under the YANG model constraints can be integrated on one running entity, the unified management and scheduling of the YANG model and the documents under the YANG model constraints can be realized; since the database management system uses data containers as the basic units for document management and model management, when managing the documents under the YANG model constraints, the integrity verification of the documents in the data containers can be performed based on the YANG model in the data containers, ensuring that the documents in the data containers always comply with the constraints of the YANG model in the data containers, and there is no problem of inconsistency between the YANG model and the document data. In addition, since the data management module can provide a transaction-based document management function, that is, it can provide native ACID transactions designed for the documents under the YANG model constraints, the business side can implement the relevant logic without having to do so, and only needs to define transactions according to the business process, which has less intrusion into the business code and higher security.

[0029] In some embodiments, when providing the model management function, the model management module is further configured to dynamically maintain the YANG file, the model schema tree, and the model metadata. Based on the YANG file, the model schema tree, and the model metadata, the model management module can better provide the management function for the YANG model. Among them, the YANG file can also be called the YANG model file or the model file, and can be used to describe or define the YANG model (or the model schema tree). The model schema tree can also be called the YANG model schema tree, which can be composed of or constructed from multiple associated YANG models, and can be used to verify the document data under the YANG model constraints to determine whether the document data complies with the YANG model constraints. The model metadata is used to store the basic information and storage path of the YANG file, and the basic information of the YANG file can include at least one of, but not limited to, the namespace, prefix, publisher organization, model description, and version of the YANG file.

[0030] The YANG file and the model metadata can be used to reconstruct the model schema tree when the data container restarts. When the YANG file changes or the model schema tree changes, the model metadata will also be updated accordingly.

[0031] Optionally, the model schema tree can be stored in memory. The model metadata can be stored in memory or external storage. The YANG file can be stored at a specified location under the data directory, which can avoid the problem of failure in reconstructing the model schema tree caused by the movement of the YANG file, and can also simplify the model export operation.

[0032] In some embodiments, the model management functions provided by the model management module may include at least one of model insertion, model export, and schema modification. Among them, model insertion may be inserting a YANG model into a data container. Model export may be exporting the YANG model (or model schema tree) in the data container to a specified location. Schema modification may be inserting a YANG file into the data container or deleting the definitions or constraints of the YANG file in the data container. Inserting a YANG file into the data container can be understood as inserting the YANG model described by the YANG file into the data container. Deleting the definitions or constraints of the YANG file in the data container can be understood as deleting the influence of the YANG file on the model schema tree in the data container (the model schema tree is composed of YANG models, and the YANG models come from the description of the YANG file, so the model schema tree will be affected by the YANG file). Specifically, it may be modifying the structure of the model schema tree or modifying the constraints on one or more nodes of the model schema tree.

[0033] In a specific implementation, when the model management module provides functions such as model insertion, model export, and / or schema modification externally, during the execution of these functions, the model management module will dynamically maintain relevant data organizations, including the model schema tree, model metadata, and YANG files described above. For example, after performing model insertion or schema modification in the data container, the model management module can dynamically update the model schema tree of the data container, store the updated model schema tree in memory, and at the same time transfer the YANG files related to the construction of the model schema tree to a specified location under the data directory, and then update the model metadata and write the updated model metadata to external storage. In this way, by dynamically maintaining YANG files, model schema trees, and model metadata, the management of YANG models can be conveniently realized.

[0034] In some embodiments, the document management functions provided by the data management module may include at least one of document insertion, document deletion, document modification, and document query. Document insertion may be inserting an XML document or a JSON document into the data container. Document deletion may be deleting the XML document or JSON document in the data container. Document modification may be modifying the XML document or JSON document in the data container. Document query may be performing a document query in the data container. This document query may be an exact query for a certain XML document or JSON document, or a range query for multiple XML documents or multiple JSON documents, or an XPATH query (the process of querying the documents hit by XPATH is called an XPATH query for the documents), etc.

[0035] The document management function provided by the data management module is implemented based on transactions. A transaction is a unit of concurrent control in the database management system of the embodiments of this application and is a sequence of operations. The execution process of a transaction satisfies the atomicity (or indivisibility), consistency, isolation (also known as independence), and durability (also known as independence) characteristics, collectively referred to as the ACID characteristics. Among them, atomicity means that all operations in a transaction are either all completed or all not completed, and will not end at some intermediate link. If an error occurs during the execution of a transaction, the database management system will be restored to the state before the transaction started, as if the transaction had never been executed. Consistency means that the database management system can only transition from one consistent state to another. It is agreed that when the database management system is in a consistent state, all documents in the database management system comply with the constraints of the YANG model schema of the data container where they are located. Isolation means that concurrently executed transactions cannot interfere with each other, that is, the operations and data used within a transaction are isolated from other concurrent transactions. The database management system of the embodiments of this application can achieve the isolation level of repeatable read. Durability means that once a transaction is committed, the change to the state of the database management system is permanent.

[0036] The transaction operation sequence consists of a series of transaction operations. Transaction operations can be predefined by the system. The transaction operations predefined by the system can be divided into two categories: transaction state maintenance operations and document read / write operations. In some embodiments, some of the transaction operations predefined by the system can be as shown in Table 1.

[0037] Table 1 Partial transaction operation sequences predefined by the database management system

[0038] When performing document management, the business application program can customize the transaction operation sequence according to the actual document management requirements. Taking document insertion in document management as an example, if it is necessary to insert document D and document E into data container C, the transaction operation sequence defined by the upper-layer business application program can be as follows: BEGIN -> INSERT_DOC(D, C) -> INSERT_DOC(E, C) -> COMMIT.

[0039] To better implement the transaction-based document management function, in some embodiments, the data management module may include a transaction manager, a thread pool, a staging area, an operation log, a B+ tree, and a commit log. The transaction manager is the core of the data management module and is used for transaction management. The transaction manager may include at least one of a globally default transaction manager and a custom transaction manager (which can be customized by a business application, i.e., the business application is allowed to customize the transaction manager). The data management module may include multiple transaction managers, and each transaction manager can be used to manage a transaction group (including one or more transactions). Different transaction managers are used to manage different transaction groups. Different transaction groups are logically isolated from each other, and different transactions within the same transaction group can also be isolated from each other. Among them, to ensure the transaction isolation within the transaction group, concurrent transactions within the transaction group can share the same thread pool, operation log, B+ tree, and commit log. At the same time, different transactions can maintain their own staging areas to save the document modifications of the transaction.

[0040] Figure 4 It is a schematic diagram of a transaction manager in an embodiment of the present application. Figure 4 The system default transaction manager and the transaction manager customized by the business application are shown. These two transaction managers are logically isolated from each other. Each transaction manager can manage a transaction group. Transactions within a transaction group can share the same thread pool, operation log, B+ tree, and commit log, and different transactions can maintain their own staging areas.

[0041] To better implement the transaction-based document management, in some embodiments, the transaction manager can maintain the execution context of all transactions within the transaction group it manages and control the transaction life cycle. The transaction execution context may include various types of data that need to be read and written during the transaction execution, including but not limited to transaction status, transaction ID, transaction operation sequence, staging area, and read view, etc. For details, please refer to Table 2.

[0042] Table 2 Variables in the Transaction Execution Context

[0043] The thread pool can be used to execute transactions. The thread pool includes multiple threads. The transaction manager can encapsulate a transaction as a task of the thread pool and submit it to the thread pool for processing. In the case of multiple transactions, multiple threads in the thread pool can implement the parallel execution of multiple transactions.

[0044] The staging area can be used to store the document data inserted or modified during the execution of a transaction and the corresponding object identifier (OID) of the document data. In some embodiments, the staging area can consist of a data area and a staging area index. The data area is used to store the document data inserted or modified during the execution of a transaction. The data area can be located in memory, specifically a dynamic array with a fixed number of bytes in memory. During the execution of a transaction, the data area can be expanded as document write operations are performed. When it is expanded to a specified size, the document data can be transferred and stored in a temporary file on external storage. For example, the data area is a 128-byte dynamic array in memory and can be expanded up to 4MB as document write operations are performed. When the data area is expanded to 4MB, the data on the staging area will be dumped into a temporary file on external storage, and all subsequent document modifications will be temporarily stored in this temporary file. In this way, through the combined storage of memory and external storage, document write operations can be better executed. Optionally, when recording the document data, the recording format of the document data in the data area can be {document write type|OID|new document}. The staging area index is used to store the OIDs of the document data inserted or modified during the execution of a transaction. Optionally, the staging area index can also record the position and length of the document in the data area. In this way, when performing document queries, according to the staging area index, the position and length of the latest version of the document with a certain OID in the data area can be determined, thus enabling fast document queries. Optionally, the staging area index can be stored only in memory without relying on external storage.

[0045] Figure 5 It is a schematic diagram of the staging area in an embodiment of the present application. Figure 5 The shown staging area includes a data area and a staging area index. The recording format of the document data in the data area is {document write type|OID|new document}, and the staging area index records the OID, offset (which can indicate the position of the document in the data area), and length of the document. When performing document queries, according to the staging area index, the position and length of the document in the data area can be determined, thus enabling fast queries.

[0046] The operation log is used to record the operations during the execution of a transaction, such as transaction start operation, transaction rollback operation, transaction commit operation, document insertion operation, document update operation, document deletion operation, etc. Since the operation log can record the operations during the execution of a transaction, the operation log can be used for fault recovery after the database management system crashes and restarts. Optionally, according to the characteristics of transaction execution, the operation log can be designed as a UNDO / REDO hybrid log. In some embodiments, the operation log can be encoded in binary form for storage, specifically including transaction ID, operation type, container name, OID, new value, and old value.

[0047] The B+ tree is used to store the document data verified by the YANG model. The B+ tree is an N-ary sorted tree on external storage, and its insertion and modification events have a relatively stable logarithmic time complexity. A typical B+ tree can provide common key-value query interfaces such as GET, PUT, and RANGE. To adapt to the B+ tree interface form, in some embodiments, the database management system can assign a unique identifier (such as the OID of the document) as the key for the document before insertion and use the tuple as the value. That is to say, the B+ tree can specifically be used to store the document data verified by the YANG model in the form of key-value pairs. Among them, the tuple can be obtained by encoding the document. In some embodiments, the tuple may include the following fields: Transaction ID, used to represent the ID of the transaction that writes the tuple into the B+ tree; Data container version, used to represent the data container version of the data fields of the last verified tuple; Identifier, used to mark whether the tuple passes the verification of the data container corresponding to the data container version; Tuple version, used to point to the previous old version of the tuple; Data, encoded in LV format, where L is the length of the encoded document and V is the encoded document.

[0048] In a specific implementation, the fields included in the tuple can be as shown in Table 3.

[0049] Table 3 Fields included in the tuple

[0050] In Table 3, TXID, CTR_VER, DFLAG, and VER_PTR are all fixed-length 8B integers (stored in little-endian format). These fields are jointly used for the implementation of the multi-version concurrency control mechanism and the re-verification of documents; DATA is the actually stored document, encoded in LV format, and the length field is encoded as a fixed-length 4B integer.

[0051] It should be noted that the above takes the B+ tree as the data storage structure of the document data as an example for illustration. In other possible embodiments, the B+ tree can also be replaced with other key-value storage data structures, such as LSM Tree, Bitcask, etc., which are not specifically limited here.

[0052] The commit log is used to record the document data eliminated from the B+ tree. In some embodiments, when the B+ tree stores the document data in the form of key-value pairs (the key is the OID of the document and the value is the tuple), the commit log can be used to record the old version tuples eliminated from the B+ tree. Among them, the logical format of the commit log for recording the old version tuples can be {OID of the document, a certain old version tuple}.

[0053] It should be noted that in the case where the B+ tree stores document data in the form of key-value pairs and the commit log is used to record old-version tuples, in the B+ tree and the commit log, tuples (or records) corresponding to the same OID can form a version chain. Optionally, tuples with the same OID on the B+ tree and the commit log can be linked through the tuple version field in the tuple (i.e., the VER_PTR field in Table 3) to form a version chain.

[0054] Figure 6 It is a schematic diagram of a version chain of an embodiment of the present application. Figure 6 In it, when modifying a certain document and storing the modified document data into the B+ tree, the B+ tree can record the OID (7) of the document and the corresponding tuple. There are three records with OID 7 stored in the commit log, and each record includes the OID (7) of the document and the corresponding old-version tuple. The 4 records with OID 7 in the B+ tree and the commit log can form a version chain.

[0055] The above explanations have been made for each component in the data management module (i.e., the transaction manager, thread pool, staging area, operation log, B+ tree, and commit log). When managing documents, these components can interact with each other, so as to realize the management of documents.

[0056] Figure 7 It is a schematic diagram of the relationship between components in a data management module of an embodiment of the present application. As Figure 7 shown, when managing documents, the transaction manager can encapsulate the transaction to be executed as a task of the thread pool and submit the task to the thread pool for processing. During the process of the thread pool executing the task, it can record the transaction operations in the operation log. If the transaction involves a document query operation, it can query the document from the B+ tree and the commit log. If the transaction involves a document write operation, it can store the document in the staging area, and the transaction manager manages the staging area and the operation log. The data in the staging area can be persisted to the B+ tree, and the eliminated document data (such as tuples) on the B+ tree can be stored in the commit log. Thus, through the interaction between components, the data management of documents can be realized.

[0057] It should be noted that the database management system of the embodiment of the present application is a document database under the YANG model constraint. When managing documents, the data container is used as the basic unit, and it is agreed that only when there is a model schema tree definition on the data container can specific document management operations be executed. In other words, it is impossible for a document management operation executed on a data container on which no model insertion operation has been performed to succeed.

[0058] The database management system provided by the embodiments of the present application can integrate the management function of the YANG model and the management function of the documents under the YANG model constraints on one running entity. Therefore, it can achieve unified management and scheduling of the YANG model and the documents under the YANG model constraints. Since the database management system uses data containers as the basic units for document management and model management, when managing the documents under the YANG model constraints, the integrity verification of the documents in the data containers can be performed based on the YANG model in the data containers, ensuring that the documents in the data containers always comply with the constraints of the YANG model in the data containers, and there is no inconsistency problem between the YANG model and the document data. In addition, since the data management module can provide a transaction-based document management function, that is, it can provide native ACID transactions designed for the documents under the YANG model constraints, the business side does not need to implement relevant logic, but only needs to define transactions according to the business process, which has less intrusion into the business code and higher security.

[0059] Based on the database management system provided by the embodiments of the present application, the embodiments of the present application also provide a model and data management system. Please refer to Figure 8 . Figure 8 As shown in the model and data management system 80 including a business application 81 and the database management system 30 of the embodiments of the present application. The database management system 30 takes the form of a function library and is embedded in the business application 81 in a linked manner to provide the business application 81 with the ability to manage documents under the YANG model constraints. That is to say, when the database management system 30 takes the form of a function library and is embedded in the business application 81 in a linked manner, the business application 81 can use the functions provided by the database management system 30 and manage the documents under the YANG model constraints through these functions.

[0060] In some embodiments, the database management system may further include an application programming interface. When the database management system takes the form of a function library and is embedded in the business application in a linked manner, the business application can access the database management system through the application programming interface, and thus can manage the documents under the YANG model constraints through the functions provided by the database management system. Optionally, the application programming interface provided by the database management system may include at least one of the following interfaces: Model management interface: The database management system can provide model management functions to the business application through the model management interface, such as model insertion, model export, schema modification, etc. The business application can manage the YANG model through the model management interface; Data Management Interface: The database management system can encapsulate document management functions into ACID transactions and provide a data management interface. Through the data management interface, it provides transaction-based document management functions to business applications, such as document insertion, document modification, document query, etc. Business applications can manage documents under the YANG model constraints through the data management interface; Transaction Definition and Execution Interface: The database management system allows business applications to customize the transaction operation sequence and encapsulate it into an ACID transaction for execution. It also allows business applications to query the transaction execution status and read the transaction execution results. That is to say, business applications can customize the transaction operation sequence through the transaction definition and execution interface, encapsulate it into an ACID transaction for execution, and query the transaction execution status and read the transaction execution results; System Management Interface: The database management system provides functions for creating, accessing, and destroying the running environment through the system management interface. That is, business applications can create, access, and destroy the running environment of the database management system through the system management interface. Among them, the running environment of the database management system refers to some global variables (such as a global transaction manager, path variables, etc.) created by the database management system during system initialization to maintain the normal operation of the database management system, and this running environment will be destroyed when the system shuts down.

[0061] Figure 9 It is a schematic diagram of an embodiment model and data management system of this application. Figure 9 In it, the database management system can provide the management function of the YANG model and the management function of documents under the YANG model constraints, and uses data containers as the basic units for model management and document management. The model management function includes, but is not limited to, model import, model export, and schema modification, and allows business applications to build a YANG model schema tree that meets the requirements in the data container. The data management function includes, but is not limited to, the addition, deletion, modification, and query of document data. The addition, deletion, modification, and query operations of document data are encapsulated into ACID transactions to ensure that the documents in the data container always conform to the verification of the YANG model schema tree in the data container. In addition, the database management system also includes an application programming interface, a runtime environment, and an external memory data structure. Business Application #1 and Business Application #2 can access the database management system through the application programming interface and manage the YANG model and documents under the YANG model constraints through the functions provided by the database management system. The runtime environment includes some global variables (such as a transaction manager, path variables, etc.), which are created during system initialization and destroyed when the system shuts down. The external memory data structure can be a B+ tree, and the verified document data can be stored on the B+ tree in the form of key-value pairs (the key is the OID of the document, and the value is the tuple generated according to the document).

[0062] Based on the model and data management system provided by the embodiments of the present application, the embodiments of the present application further provide a model management method applicable to the system, which can implement the management of the YANG model. Figure 10 It is a schematic flowchart of a model management method according to an embodiment of the present application, including the following steps.

[0063] Step S102: Receive a model management request, which includes the identifier of the first data container, and the model management request is used to request the management of the YANG model in the first data container.

[0064] As described above, the database management system provided by the embodiments of the present application uses the data container as the basic unit for model management. Therefore, when managing the model, the model management request can carry the identifier of the first data container to request the management of the model in the first data container. Among them, the model management request can be initiated by a business application program, and the identifier in the first data container can be the name of the first data container, and the name of the first data container can be pre-allocated by the database management system and can uniquely identify the first data container.

[0065] Step S104: Manage the YANG model in the first data container according to the model management request.

[0066] In the case of receiving the model management request, the YANG model in the first data container can be managed based on the model management request, thereby realizing the management of the YANG model.

[0067] In some embodiments, the model management request may include a model insertion request. In addition to carrying the identifier of the first data container, the model insertion request may also carry the first YANG file, which is used to request to insert the YANG model described by the first YANG file into the first data container. In this case, managing the YANG model in the first data container according to the model management request may include: Determine whether the model insertion condition is satisfied; In the case of satisfying the model insertion condition, merge the YANG model described by the first YANG file into the YANG model of the first data container, store the first YANG file in a specified location, and update the model metadata of the first data container.

[0068] The model insertion conditions may include that there is no data in the first data container, there are no errors in the first YANG file, and the YANG model described in the first YANG file is compatible with the YANG model in the first data container. That is to say, the model insertion process needs to occur when the data container does not contain data (i.e., the data container has not performed the operation of inserting document data). If the data container contains data, the model insertion process cannot be executed. In addition, when performing model insertion, it is also required that there are no errors in the first YANG file and the YANG model described in the first YANG file is compatible with the YANG model in the first data container, otherwise the model insertion process cannot be executed. Among them, the fact that there are no errors in the first YANG file may include that there are no lexical and syntactic errors in the first YANG file and the first YANG file conforms to the RFC definition. The compatibility between the YANG model described in the first YANG file and the YANG model in the first data container may include that none of the YANG models described in the first YANG file are defined in the first data container. In other words, the incompatibility (i.e., conflict) between the YANG model described in the first YANG file and the YANG model in the first data container may be that part or all of the YANG models described in the first YANG file are defined in the first data container (the first data container describes the model schema tree, and the first YANG file describes the YANG model. When performing model insertion, it is necessary to merge the YANG model described in the first YANG file into the model schema tree described by the first data container. If the first data container already contains part or all of the YANG models described in the first YANG file, then it is not necessary to merge the YANG model described in the first YANG file into the model schema tree described by the first data container. At this time, it can be considered that the YANG model described in the first YANG file is incompatible with the YANG model in the first data container).

[0069] When the model insertion conditions are met, the YANG model described in the first YANG file can be merged into the model schema tree (or YANG model) of the first data container. In addition, since the model insertion operation modifies the model schema tree in the first data container, it is also necessary to update the model metadata of the first data container and store the first YANG file at a specified location.

[0070] Optionally, if the model insertion conditions are not met, the model insertion request may not be responded to, that is, the YANG model described in the first YANG file is not merged into the model schema tree (or YANG model) of the first data container, and a model insertion failure message is returned.

[0071] Figure 11 It is a schematic diagram of the model insertion process in an embodiment of the present application. Figure 11The model insertion process shown takes the first data container as C and the first YANG file as F as an example, and includes the following steps: Step 1: Data check.

[0072] Check whether there is data on the data container C, whether there are errors in the YANG file F to be inserted, and whether the model schema trees on F and the data container C are compatible. Among them, the situations where F has errors include: F has lexical and syntactic errors and does not conform to the RFC definition. The situations where F is incompatible (or in conflict) with C include: part or all of the schema tree described by F is defined on C, etc.

[0073] If the check passes (that is, F has no errors and F is compatible with C), then execute Step 2. If the check fails (that is, F has errors or F is incompatible with C), then execute Step 4.

[0074] Step 2: Merge the YANG model described in F into C and dump F to the specified location.

[0075] Step 3: Modify the model metadata and dump it to the external storage, and feedback the successful operation to the business application, and the process ends.

[0076] As Figure 12 shown, after inserting the YANG file numbered 1 into the data container C, the data container C will change as follows: First, the model schema tree of C will be represented as the tree-shaped data structure numbered 2 in this figure; Second, a piece of metadata about this file will be inserted into the in-memory model metadata and then dumped to the external storage; Third, this YANG file will be dumped to the specified location.

[0077] Step 4: Print the error message and return a failure code to the business application, and the process ends.

[0078] Based on Figure 11 the model insertion process shown, the business application can construct the model schema tree it needs through model insertion. Among them, it should be noted that in the specific implementation, when performing model insertion, the business application can be allowed to pass in a certain YANG file path or a folder containing several YANG files. The latter is usually called model batch import, and will be serialized into multiple model insertion operations during execution.

[0079] In some embodiments, the model management request may include a model export request. In addition to carrying the identifier of the first data container, the model export request may also carry the export location (or export path) for requesting to export the model schema tree in the first data container to this export location. In this case, according to the model management request, managing the YANG model in the first data container may include: Determine whether the model export conditions are met; When the model export conditions are met, copy the model metadata of the first data container and the corresponding YANG file to the export location.

[0080] The model export conditions may include that the export location in the model export request corresponds to an empty directory. An empty directory can be understood as a directory that does not exist or a directory that does not contain any files.

[0081] If the model export conditions are met, the model export request can be responded to and the model export operation can be executed. Since all the information for reconstructing the model schema tree has been saved in the model metadata and the associated YANG file has also been dumped to the specified location, when performing the model export operation, the model metadata of the first data container and the corresponding YANG file can be copied to the export location. Based on the copied model metadata and YANG file, the model schema tree in the first data container can be reconstructed, thus achieving the purpose of model export. If the model export conditions are not met, the model export request can be not responded to, that is, the model metadata of the first data container and the corresponding YANG file are not copied to the export location, and a model export failure message is returned.

[0082] Figure 13 It is a schematic flow diagram of model export in an embodiment of the present application. Figure 13 The model export process shown takes the first data container as C and the export location as dest as an example, and includes the following steps: Step 1: Check the export location.

[0083] Check whether the export location dest corresponds to an empty directory. If dest points to an ordinary file or dest is a non-empty directory, execute Step 3; if dest points to an empty directory, execute Step 2.

[0084] Step 2: Copy the external storage model metadata of data container C and the model files it contains to dest, and return a success code to the application.

[0085] It should be noted that if dest does not exist, dest can be automatically created before performing the copy operation.

[0086] Step 3: Print an error message and return a failure code.

[0087] It should be noted that in specific implementation, to avoid the inconsistency between model metadata and model files during model export, the model export operation is mutually exclusive with the model insertion operation and the mode modification operation and cannot occur simultaneously. That is to say, before performing the model export operation, it is necessary to determine whether there is a model insertion operation or a mode modification operation. If so, the model export operation can be not executed, or wait until the model insertion operation or the mode modification operation ends and then execute the model export operation. If not, the model export operation can be executed.

[0088] In some embodiments, the model management request may include a mode modification request. In addition to carrying the identifier of the first data container, the mode modification request may also carry a mode modification sequence. The mode modification sequence may be composed of at least one mode modification operation. The mode modification operation includes constraints for inserting or deleting a YANG file. The mode modification request is used to request to perform the mode modification operations in the mode modification sequence on the first data container to modify the model mode tree in the first data container. In this case, according to the model management request, managing the YANG model in the first data container may include: Determine whether the mode modification condition is met; When the mode modification condition is met, process the YANG file of the first data container according to the mode modification sequence to obtain a set of YANG files; Create a second data container, which has a YANG model and model metadata independent of the first data container and points to the data storage of the first data container; Add the YANG model in the set of YANG files to the second data container and determine the second data container as the successor data container of the first data container.

[0089] The mode modification conditions may include the absence of other mode modification operations on the first data container and the successful execution of the model export operation for the first data container. In a specific implementation, when a mode modification operation is received, it can first be determined whether there are other mode modification operations currently on the first data container (multiple mode modification operations cannot be executed simultaneously on the same data container). If there are, it can be determined that the mode modification conditions are not met. If not, the model export operation can be executed on the first data container (the model export operation here is actually a backup of the first data container. If the subsequent mode modification operation fails, the first data container can be restored through this backup). If the model export operation for the first data container is successfully executed, it can be determined that the mode modification conditions are met. If the model export operation for the first data container fails to be successfully executed, it can be determined that the mode modification conditions are not met. Optionally, the mode modification conditions may also include the absence of model insertion operations and model export operations on the first data container (that is, model insertion operations, model export operations, and mode modification operations cannot be executed simultaneously on the same data container). If there is a model insertion operation or a model export operation on the first data container, the mode modification conditions are not met.

[0090] In the case where the mode modification conditions are not met, the mode modification request may not be responded to, and a mode modification failure message may be returned. In the case where the mode modification conditions are met, the YANG file of the first data container can be processed according to the mode modification sequence to obtain a set of YANG files. For example, if the mode modification sequence includes inserting YANG file F, YANG file F can be inserted into the YANG file of the first data container, and the obtained set of YANG files includes the YANG file of the first data container and YANG file F. Another example is that if the mode modification sequence includes deleting YANG file F, YANG file F can be deleted from the YANG file of the first data container, and the obtained set of YANG files includes the YANG file obtained after deleting YANG file F from the first data container.

[0091] After obtaining the YANG file set, a new data container, i.e., the second data container, can be created, and the YANG model (or model schema tree) described by the YANG file set is added to the second data container. Among them, the second data container has a YANG model and model metadata independent of the first data container and points to the data storage of the first data container. When the YANG model (or model schema tree) described by the YANG file set is successfully added to the second data container, the second data container can be used as the successor data container of the first data container. Subsequently, when performing model management and data management based on the first data container, it can be implemented based on the second data container. That is to say, the second data container can be used as a new container version of the first data container. Optionally, when adding the YANG model (or model schema tree) described by the YANG file set to the second data container, if the addition fails, the schema modification process can be ended, the second data container and the YANG file set are deleted, and a schema modification failure message is returned.

[0092] Figure 14 It is a schematic diagram of the process of modifying the schema in an embodiment of the present application. Figure 14 The illustrated schema modification process takes the first data container as B, the second data container as C, and the schema modification sequence as Q as an example for description, and includes the following steps: Step 1: Preparation stage.

[0093] First, check whether there is an ongoing schema modification operation on data container B. If not, continue to the next step. If so, the preparation stage fails and the process ends. Secondly, perform a model export operation on data container B. This model export operation essentially creates a model backup of data container B, which can be restored through this backup if the subsequent execution process fails. If the model export operation on data container B is successfully performed, continue to the next step. If the model export operation on data container B is not successfully performed, the preparation stage fails and the process ends. Thirdly, obtain all the YANG files of data container B and apply the schema operation sequence Q to obtain the YANG file set S. Finally, create an execution thread, and the execution thread will execute the schema modification process described in step 2. Step 2: Execution stage.

[0094] Create a new data container C. C has a model schema tree independent of B and metadata of the memory version (initialized to empty) and points to the data storage of B (here, the data storage refers to the storage of document data, and B and C share the same storage area for document data). Loop to add the YANG models in the YANG file set S to data container C. If no model insertion failure occurs in this process, the execution stage is successful; otherwise, the execution fails and data container C is deleted. Regardless of the result of the execution stage, step 3 will be executed.

[0095] Step 3: Recycling phase.

[0096] Generally speaking, the recycling phase still runs on the same thread as the execution phase. If the execution phase is successfully completed, the new data container C is taken as the successor data container of data container B. Then, the metadata of data container C is dumped to external storage, and the YANG file set S is used to replace the dumped model schema of B. After that, no schema modification process is allowed to occur on data container B. If an exception occurs during the execution phase, the YANG file set S is cleared.

[0097] The schema modification process can be initiated by the main thread of the business application. During the execution process, it does not block the data reading and writing process of the main thread on the original data container B, but no other schema modification process is allowed. The main thread can know the execution status of the schema modification process through methods such as thread waiting or status checking. If the schema modification process is completed, the main thread can switch to the new data container C and clear the original data container B. The switching process is not mandatory, and the main thread can still perform additional data reading and writing on the original data container B. Of course, it is not recommended to continue to perform new data insertion on the original data container B during or after the model modification execution, because this will prolong the time for data re-verification on the new data container C, thereby affecting the throughput of the system.

[0098] It should be noted that the schema modification process is designed to allow for schema-less data containers, that is, it can be based on data containers that do not contain data. Since such data containers do not contain any data, the schema modification process on such data containers is equivalent to the model insertion process. In addition, the schema modification process not only involves modifying the model schema tree but also involves re-verifying the document data in the data container. In this embodiment, the schema modification process does not force the re-verification of document data, but evenly distributes the re-verification operation of document data to subsequent data queries (note: this method is called asynchronous mode or verification on read). If you do not want the schema re-verification to cause a long-term impact on performance, you can manually perform a full-scale document query on the new data container (note: this method is called synchronous mode). Although the schema modification operation has a wide range of impacts and a long execution time, in actual applications, the YANG model on the device usually remains stable for a long time, and the schema modification operation is executed at a low frequency.

[0099] It should also be noted that the data container will iterate under the action of schema modification (for example Figure 14 the data container C shown is the new version of data container B). To distinguish different versions of data containers, the database management system will assign different version numbers to each data container with different schemas. The data container version number can be an 8-byte integer, which can uniquely identify a data container in a certain version during the iteration process.

[0100] The model management method provided by the embodiments of the present application, when managing the YANG model, since the model management request carries the identifier of the data container, it is possible to manage the YANG model based on the data container, including but not limited to inserting the YANG model into the data container, exporting the YANG model in the data container to a specified location, modifying the model schema tree or the YANG model in the data container.

[0101] Based on the model and data management system provided by the embodiments of the present application, the embodiments of the present application also provide a data management method applicable to the system, which can realize the management of document data under the YANG model constraint. Figure 15 It is a flowchart of the data management method according to an embodiment of the present application, including the following steps.

[0102] Step S152: Receive a processing request for a first transaction, where the first transaction is used to manage the document data in the data container.

[0103] As described above, the database management system provided by the embodiments of the present application uses the data container as the basic unit of data management and provides a transaction-based document management function. Therefore, when managing the data of the document under the YANG model constraint, a processing request for the first transaction can be initiated, where the first transaction is used to manage the document data in the data container, such as adding, deleting, modifying, and querying the document data in the data container. Among them, the first transaction can be defined by the business application program, and the processing request for the first transaction can be initiated by the business application program.

[0104] Step S154: Process the first transaction according to the processing request.

[0105] In the case of receiving a processing request for the first transaction, the first transaction can be processed, thereby realizing the management of the document data under the YANG model constraint.

[0106] In some embodiments, the processing request for the first transaction may include the identifier of the first transaction manager. Processing the first transaction according to the processing request may include: Determine whether the first transaction is a legal transaction; In the case where the first transaction is a legal transaction, submit the first transaction to the thread pool of the first transaction manager and sequentially execute each transaction operation in the first transaction; In the case where the first transaction is not a legal transaction, return the operation result that the first transaction is illegal.

[0107] In this embodiment, a legal transaction refers to a transaction that starts with a transaction start operation, ends with a transaction commit operation, and includes at least one document operation between the transaction start operation and the transaction commit operation. This document operation can be document query, document insertion, or document modification, etc. Before processing the first transaction, it is necessary to determine whether the first transaction is a legal transaction. If the first transaction is a legal transaction, the next step can be executed, that is, the first transaction is submitted to the thread pool of the first transaction manager, and the first transaction manager manages the first transaction, and the thread pool of the first transaction manager sequentially executes each transaction operation in the first transaction. If the first transaction is not a legal transaction, the first transaction can be not processed, and an operation result indicating that the first transaction is illegal is returned.

[0108] Optionally, in the case of sequentially executing each transaction operation in the first transaction, if each transaction operation in the first transaction is successfully executed, an operation result indicating successful execution of the first transaction can be returned. If the first transaction is not successfully executed, for example, a certain transaction operation in the first transaction is not successfully executed or the status of the first transaction is abnormal after a certain transaction operation in the first transaction is executed (such as being interrupted by the first transaction manager or the transaction operation returning an error code), an operation result indicating that the first transaction execution fails can be returned.

[0109] Figure 16 It is a schematic diagram of the execution process of a general transaction in an embodiment of the present application.

[0110] As Figure 16 shown, the transaction execution process needs to input transaction T (including a transaction operation sequence Q, and Q includes transaction status operations and read / write operations of documents), and a specified transaction manager M. Transaction T will be managed and controlled by transaction manager M. Before executing transaction T, the main thread will check whether transaction T is a legal transaction. If transaction T starts with a BEGIN operation (i.e., a transaction start operation) and ends with a COMMIT operation (i.e., a transaction commit operation), and there is at least one document operation between the BEGIN operation and the COMMIT operation, then T is a legal transaction. If transaction T does not start with a BEGIN operation, or does not end with a COMMIT operation, or there is no document operation between the BEGIN operation and the COMMIT operation, then transaction T is an illegal transaction.

[0111] In the case where transaction T is not a legal transaction, an operation result indicating that transaction T is illegal will be obtained. In the case where transaction T is a legal transaction, transaction T will be encapsulated as a task of the thread pool and submitted to the thread pool of M. When there is an idle thread in the thread pool, transaction T will be scheduled to a certain working thread. Thereafter, the upper-layer business application program will receive a message that the transaction is scheduled, and the main thread can continue to execute other operations, such as issuing other transactions or performing mode modifications, etc., or get blocked and wait to be awakened.

[0112] After the worker thread is scheduled for transaction T, it will sequentially fetch and execute instructions OP from the transaction operation sequence Q (i.e., each transaction operation included in Q) until all operations are executed or the transaction state becomes abnormal after a certain operation is executed (interrupted by the transaction manager or the operation returns an error code). After that, the worker thread will wake up the main thread to finalize the current transaction, and the worker thread will return to the idle state. When the main thread is woken up by the worker thread (or periodically polls and finds that the transaction execution has ended), the main thread will check the transaction operation result (such as transaction T is illegal, transaction T is successfully executed, or transaction T fails to execute), and return the transaction operation result to the upper-layer business application program.

[0113] As mentioned above, the first transaction may include multiple transaction operations (which may form a transaction operation sequence), such as a transaction start operation, document operations (including document insertion operations, document query operations, etc.), and a transaction commit operation. When sequentially executing each transaction operation in the first transaction, in the case where the transaction operation is a transaction start operation, executing the transaction start operation may include: Initializing the read view and the staging area in the transaction execution context; Adding the ID of the first transaction to the active transaction list of the first transaction manager.

[0114] The transaction execution context is maintained by the first transaction manager. Multiple variables involved in the execution process of the first transaction are recorded in the transaction execution context. Specifically, reference can be made to the records in Table 2 above, and details are not elaborated here. When executing the transaction start operation, the variables in the transaction execution context can be initialized, especially the two variables of the read view and the staging area in the transaction execution context, to prepare for subsequent document operations. Among them, the read view is used to record the current transaction ID, the active transaction ID list, the minimum transaction ID for concurrent execution, and the pre-allocated next transaction ID. The current transaction ID is the ID of the first transaction. The active transaction ID list includes the IDs of one or more currently executing transactions. The minimum transaction ID for concurrent execution refers to the ID of the transaction with the smallest ID among multiple concurrently executed transactions (the transaction ID is assigned by the transaction manager and increases with time, which can be used to determine the execution order of transactions). The pre-allocated next transaction ID is the ID of the transaction that needs to be executed after the first transaction is executed. The staging area is used to store the document data and the corresponding document OIDs inserted or modified during the transaction execution. The staging area includes a data area and a staging area index. Specifically, reference can be made to the description of the staging area in the data management module above, and details are not elaborated here. After initializing the transaction execution context, the ID of the first transaction can be added to the active transaction ID list in the read view.

[0115] Optionally, after initializing the transaction execution context, the transaction start event can be recorded in the operation log, and the status of the first transaction can be modified to running.

[0116] Figure 17 It is a schematic diagram of the execution process of the transaction start operation in an embodiment of the present application. Figure 17 The shown transaction start operation process includes the following steps.

[0117] Step 1: Initialize the read view. The read view consists of the current transaction ID, the list of active transaction IDs, the minimum transaction ID for concurrent execution, and the pre-allocated next transaction ID, and this information is provided by the transaction manager to which the transaction belongs.

[0118] Step 2: Initialize the staging area.

[0119] Step 3: Record the transaction start event in the operation log: BEGIN TX txid.

[0120] Step 4: Modify the transaction status to "running" and add the transaction to the active transaction list of the transaction manager.

[0121] If all the above operations are successful, return OK, otherwise return ERROR.

[0122] In the case where the transaction operation in the first transaction includes a document insertion operation, the document insertion operation can include the first document to be inserted and the identifier of the third data container (that is, the document insertion operation needs to specify the document to be inserted and the target data container), and this document insertion operation is used to insert the first document into the third data container. In this case, performing the document insertion operation can include: Verify the first document according to the model pattern tree in the third data container; In the case where the verification of the first document passes, assign an OID to the first document and write the first document into the staging area of the first transaction; In the case where the verification of the first document fails, record the failure of the document insertion operation in the transaction execution context.

[0123] To ensure that the document data in the data container always complies with the constraints of the model schema tree (or YANG model) in the data container, when inserting the first document into the third data container, it is necessary to first use the model schema tree in the third data container to verify the first document. If the verification passes, the first document can be inserted into the third data container. Specifically, an OID can be assigned to the first document, and then the first document can be written into the staging area of the first transaction for staging (persistence will be performed in the transaction commit operation). If the verification fails, the first document cannot be inserted into the first data container to avoid inconsistent problems caused by the document data in the third data container not conforming to the constraints of the model schema tree. In addition, the failure of the document insertion operation can also be recorded in the transaction execution context.

[0124] Figure 18 It is a schematic diagram of the execution process of the document insertion operation in an embodiment of the present application. Figure 18 The document insertion operation process shown takes the third data container as C and the first document to be inserted as document D as an example for illustration, and includes the following steps.

[0125] Step 1: Document loading and verification. Use the model schema tree of data container C to verify document D. If the verification is successful, convert document D into a computationally friendly binary form. Storing document data in binary format has the advantages of small storage volume and fast loading speed. Similar binary formats also include BSON, JSONB, etc.

[0126] Step 2: Document operation staging. Assign a document identifier OID to document D, and write {Insert|C|OID|D} into the staging area. It should be noted that when writing document D into the staging area, if the data area in the staging area is insufficient, document D can be transferred to an external file.

[0127] Step 3: Record the document insertion event in the operation log: INSERT DOC [TX, C, OID, D].

[0128] Step 4: Write the operation receipt. Record the OID assigned by the system into the transaction execution context.

[0129] If any of the above steps fails, an error is output and ERROR is returned. Otherwise, that is, if all steps are successfully executed, the document insertion operation is successful.

[0130] In the case where the transaction operations in the first transaction include document deletion operations or document modification operations, the specific implementation process is similar to the specific implementation process of the above document insertion, and will not be elaborated here.

[0131] When the transaction operation in the first transaction includes a document query operation, the document query operation can be an exact query based on OID (also known as point query), a range query based on OID, or an XPATH query. When the document query operation is an exact query based on OID, the first OID of the document to be queried and the identifier of the fourth data container can be carried in the document query operation (that is, the document query operation needs to specify the OID of the document to be queried and the data container to be queried), and the document query operation is used to query the document data corresponding to the first OID in the fourth data container. In this case, performing the document query operation may include the following steps S11 to S13: Step S11: Perform a staging area query and a version chain query based on the first OID to obtain a document query result.

[0132] The exact document query can be divided into a staging area query and a version chain query. Among them, the version chain query performs data query in the B+ tree and the commit log. Since the staging area stores the latest data, and the data in the B+ tree and the commit log is older than the data in the staging area, when performing an exact document query, the staging area query can be performed first, and then the version chain query.

[0133] In some embodiments, performing a staging area query and a version chain query based on the first OID to obtain a document query result may include the following steps S111 to S113: Step S111: Query whether there is a document corresponding to the first OID in the staging area.

[0134] When storing document data in the staging area, the OID of the document will be recorded in the staging area index. When performing a staging area query, it can be queried whether the first OID is recorded in the staging area index. If the first OID is recorded in the staging area index, it can be determined that there is a document corresponding to the first OID in the staging area. If the first OID is not recorded in the staging area index, it can be determined that there is no document corresponding to the first OID in the staging area.

[0135] When there is a document corresponding to the first OID in the staging area, step S112 can be executed. When there is no document corresponding to the first OID in the staging area, step S113 can be executed.

[0136] Step S112: When there is a document corresponding to the first OID in the staging area, determine the document corresponding to the first OID in the staging area as the document query result.

[0137] Step S113: When there is no document corresponding to the first OID in the staging area of the first transaction, query whether there is a first tuple corresponding to the first OID on the B+ tree.

[0138] When storing document data, the B+ tree stores it in the form of key-value pairs. The key is the document OID, and the value is the tuple generated based on the document data. When querying documents on the B+ tree, it is possible to query whether the first tuple corresponding to the first OID exists (i.e., is recorded) on the B+ tree according to the first OID. If the first tuple corresponding to the first OID does not exist on the B+ tree, it can be stated that the fourth data container has never written the document corresponding to the first OID. At this time, it can be determined that the document query result is empty. If the first tuple corresponding to the first OID exists on the B+ tree, it can be stated that the fourth data container has written the document corresponding to the first OID. At this time, it can be determined whether the first tuple is visible to the first transaction.

[0139] When determining whether the first tuple is visible to the first transaction, it can be achieved by comparing the transaction ID recorded in the read view of the transaction execution context and the transaction ID recorded in the first tuple. Among them, the read view is a snapshot of the data container at the moment when the transaction starts, ensuring that the transaction can only read the updates to the database by the committed transactions before the start. It consists of the current transaction ID, the list of active transaction IDs (m_ids), the minimum transaction ID (tx_min) of concurrent execution, and the pre-allocated next transaction ID (tx_max). The transaction ID (owner_txid) recorded in the first tuple is the ID of the transaction that wrote the first tuple into the B+ tree.

[0140] In some embodiments, determining whether the first tuple is visible to the first transaction may include: Determine the current transaction ID, the list of active transaction IDs, the minimum transaction ID of concurrent execution, and the pre-allocated next transaction ID recorded in the read view of the transaction execution context. The current transaction ID is the ID of the first transaction; Under the condition of satisfying the first condition, determine that the first tuple is visible to the first transaction; Under the condition of satisfying the second condition, determine that the first tuple is not visible to the first transaction; Among them, the first condition includes any one of the following: The transaction ID recorded in the first tuple is less than the minimum transaction ID; The transaction ID recorded in the first tuple is greater than or equal to the minimum transaction ID and less than the pre-allocated next transaction ID, and is not in the list of active transaction IDs; The second condition includes any one of the following: The transaction ID recorded in the first tuple is greater than or equal to the next transaction ID; The transaction ID recorded in the first tuple is greater than or equal to the minimum transaction ID and less than the pre-allocated next transaction ID, and is in the list of active transaction IDs.

[0141] That is to say, it is possible to determine whether the first tuple is visible to the first transaction through the following rules: Rule 1: When the owner_txid is less than tx_min, the first tuple is visible to the first transaction because the transactions before tx_min have been committed. Rule 2: When the owner_txid is greater than or equal to tx_max, the first tuple is not visible to the first transaction because the write transaction has not been started at the start time. Rule 3: When the owner_txid is greater than or equal to tx_min and less than tx_max, if the owner_txid is in m_ids, then the owner_txid has not been committed and the first tuple is not visible to the first transaction; otherwise, the first tuple is visible to the first transaction.

[0142] Suppose the current transaction ID is 5, that is, the transaction ID of the first transaction is 5. In the read view created at the start time of the first transaction, m_ids = {2, 3}, tx_min = 2, and tx_max = 6, which means that transactions 2 and 3 have not been committed yet, and transactions 1 and 4 have been executed and ended (assuming successful commit). Then, when the owner_txid of the first tuple read by the first transaction is the following values, the visibility of the first tuple to the first transaction is shown in Table 4.

[0143] Table 4 Visibility of the First Tuple to the First Transaction

[0144] After determining whether the first tuple is visible to the first transaction, if the first tuple is visible to the first transaction, the first tuple can be determined as the document query result. If the first tuple is not visible to the first transaction, the old version tuple corresponding to the first tuple can be queried in the commit log according to the first OID, and it is determined whether the old version tuple corresponding to the first tuple is visible to the first transaction. The specific implementation method is the same as determining whether the first tuple is visible to the first transaction and will not be described in detail here. In the case where there is a second tuple visible to the first transaction in the old version tuple corresponding to the first tuple, the second tuple can be determined as the document query result. In the case where there is no second tuple visible to the first transaction in the old version tuple corresponding to the first tuple, it can be determined that the document query result is empty.

[0145] Step S12: When the document query result is empty, return a query failure message.

[0146] Step S13: When the document query result is not empty, perform document re-verification on the document query result according to the fourth data container; when the verification of the document query result passes, return the document query result; when the verification of the document query result fails, return a query failure message.

[0147] When the document query result is not empty, in order to ensure that the document query result conforms to the constraint of the model schema tree in the fourth data container and avoid the problem of inconsistent models and data, the document query result can be re-verified according to the model schema tree in the fourth data container.

[0148] In some embodiments, when re-verifying the document query result according to the fourth data container, it can be first determined whether the data container version recorded in the document query result is consistent with the version of the fourth data container. Among them, based on the above-described staging area query process and version chain query process, the document query result can be the document data queried in the staging area or the tuple queried in the B+ tree or the tuple queried in the commit log. When the document query result is the document data queried in the staging area, the data container version recorded in the document query result is the version of the data container operated when modifying the staging area. When the document query result is the tuple queried in the B+ tree or the commit log, the data container version recorded in the document query result is the version of the data container recorded in the queried tuple (i.e., the CTR_VER field in Table 3 above).

[0149] When the data container version recorded in the document query result is consistent with the version of the fourth data container, it can be determined that the verification of the document query result passes, and the document query result is returned. When the data container version recorded in the document query result is inconsistent with the version of the fourth data container, the document query result needs to be re-verified according to the model schema tree in the fourth data container. When re-verifying the document query result, if the document query result conforms to the constraint of the model schema tree in the fourth data container, it can be determined that the verification of the document query result passes, and the document query result is returned. If the document query result does not conform to the constraint of the model schema tree in the fourth data container, it can be determined that the verification of the document query result fails, and a query failure message is returned.

[0150] Optionally, after re-verifying the document, regardless of whether the verification result of the document passes or fails, the verification result and the document query result can be encapsulated as a tuple and saved in the staging area. When performing a transaction commit later, the tuple saved in the staging area can be persisted to the B+ tree.

[0151] Figure 19 It is a schematic diagram of the execution process of the document query operation in an embodiment of the present application. Figure 19 The shown document query operation process includes the following steps.

[0152] Step 1: Staging area query. Query the document corresponding to the specified OID on the transaction staging area. If the query result is empty, execute Step 2. If the query result is not empty, obtain the document staged in the staging area and execute Step 4.

[0153] Step 2: Version chain query - B+ tree. Query the document corresponding to the specified OID on the B+ tree. If the query result is empty, it means that the data container has never written the document corresponding to this OID. At this time, Step 4 can be executed. If the query result is not empty, the tuple obtained from the query can be recorded as V0, and Step 3 is executed.

[0154] Step 3: Version chain query - commit log. Set V = V0 and determine whether V is visible to the current transaction. If V is visible to the current transaction, interrupt the version chain query and execute Step 4. If V is not visible to the current transaction, query the previous old version tuple in the commit log according to the address of the previous version recorded in V and assign it to V. Repeat this process until a visible tuple is found or it is determined that the oldest version tuple is still not visible to the current transaction.

[0155] Step 4: Version check and document re-verification. If the result V queried in Steps 1 to 3 is not empty and V = V0, check whether the data container version B recorded in V is consistent with the data container version C specified in the document query operation. If they are inconsistent, the document needs to be re-verified using the model schema tree on C, and the verification result (verification passed or verification failed) of the document and the document are encapsulated into a tuple and saved in the staging area. In this way, when this transaction is committed, the result of the document re-verification can be persisted to the B+ tree. If the result queried in Steps 1 to 3 is not equal to V0, or V is empty, version checking may not be required.

[0156] Step 5: If the document re-verification fails, set V to be empty.

[0157] Step 6: Write operation receipt. Record V in the transaction execution context and set the operation result. If V is empty, set the operation result to NOT_FOUND; otherwise, set it to SUCCESS.

[0158] If any of the above steps fails, an error is output and ERROR is returned. Otherwise, if all steps are successfully executed, the exact query operation is successful.

[0159] The above takes the exact document query as an example and details how to execute the document query operation. In the case where the document query operation is a range query or an XPATH query based on OID, the specific implementation process is similar to that of the exact document query. Specifically, in the case where the document query is a range query, taking the OID range as [start, end] (it can also be an open interval or a semi-open semi-closed interval) as an example, to execute the document query operation, it can include: Step 1: Staging area query. Find all documents in the staging area whose OIDs fall within [start, end], denoted as S.

[0160] Step 2: Version chain query. Search in the B+ tree for all documents whose OIDs fall within [start, end] and are not included in S. It should be noted that for each tuple retrieved from the B+ tree, the version V visible to this transaction needs to be retrieved along the version chain and added to S; If any of the above steps fails, an error is output and ERROR is returned; otherwise, the range query operation is successful.

[0161] In the case where the document query is an XPATH query, the document query operation can be performed, including: Step 1: Staging area query. Check whether the records in the staging area can be hit by XPATH, and add the hit documents to S; Step 2: Version chain query. Perform a full-range query on the B+ tree, but ignore the OIDs existing in the staging area. For each record, retrieve the version V visible to this transaction along the version chain. If V is hit by XPATH, add it to S.

[0162] If any of the above steps fails, an error is output and ERROR is returned; otherwise, the XPATH query is successful.

[0163] It should be noted that document exact query, range query, and XPATH query are all query operations occurring on the data container. For a business system, document exact query and range query are usually not used as the main query means because both rely on OIDs as the query basis, and OIDs are automatically generated by the database management system and have nothing to do with the business system. Optionally, it can be agreed that if a transaction only includes query operations on the data container, then this transaction is called a read-only transaction. A read-only transaction can ensure that the staging area in the transaction execution context is empty, thereby reducing the CPU cost of the XPATH query operation. In the absence of hardware failures, a read-only transaction can usually be successfully committed, and a read-only transaction does not conflict with other transactions and has good query performance. Based on this advantage of read-only transactions, the database management system recommends that the business system construct read-only transactions to perform XPATH query operations.

[0164] The transaction operations in the first transaction also include a transaction commit operation. Performing the transaction commit operation can include: Obtain the conflict list recorded in the transaction execution context. The conflict list is used to record the OIDs of the documents modified by concurrent transactions during the transaction execution process (specifically, the OIDs of the documents modified by concurrent committed transactions); Perform write-write conflict detection based on the document OIDs recorded in the staging area and the document OIDs recorded in the conflict list; In the case where the detection result indicates the existence of a write-write conflict, cancel the first transaction; In the case where the detection result indicates no write-write conflict, modify the data in the staging area and write it into the B+ tree in the form of a tuple, commit the first transaction, and return the transaction execution result.

[0165] When performing write-write conflict detection, it is possible to determine whether there is an intersection between the document OIDs recorded in the staging area and the document OIDs recorded in the conflict list. If there is an intersection between the document OIDs recorded in the staging area and the document OIDs recorded in the conflict list, it indicates that the documents modified by the current transaction have also been modified by other concurrent transactions, and at this time, it can be determined that there is a write-write conflict. If there is no intersection between the document OIDs recorded in the staging area and the document OIDs recorded in the conflict list, it can be explained that the documents modified by the current transaction have not been modified by other concurrent transactions, and at this time, it can be determined that there is no write-write conflict. Optionally, in the case where there is no write-write conflict, the OIDs recorded in the staging area can be written into the conflict list of the currently active and uncommitted transactions, so that other transactions can perform write-write conflict detection based on the conflict list when committing.

[0166] In the case where there is a write-write conflict, in order to avoid data conflicts, the first transaction can be revoked. Optionally, when revoking the first transaction, at least one of the following operations can be performed: Discard the staging area; record the transaction revocation event in the operation log and modify the transaction status to revoked; remove the first transaction from the first transaction manager.

[0167] In the case where there is no write-write conflict, the first transaction can be committed. When committing the first transaction, at least one of the following operations can be performed: Record the transaction pre-commit event in the operation log; Flush the operation log buffer to persist the operation records of the first transaction; Write the data modification in the staging area into the B+ tree in the form of a tuple. In the case where there is an old version tuple in the B+ tree, write the old version tuple into the commit log; Record the transaction commit event and the checkpoint event in the operation log; Flush the operation log buffer to persist the commit records of the first transaction; Modify the transaction status to committed and remove the first transaction from the first transaction manager.

[0168] After committing the first transaction, the transaction execution result can be returned. For example, if the first transaction includes a document query operation, the transaction execution result can include the document query result.

[0169] Figure 20 It is a schematic diagram of the execution process of the transaction commit operation in an embodiment of the present application. Figure 20 The shown transaction commit process includes the following steps.

[0170] Step 1: Modify the transaction status. Modify the transaction status to COMMITING.

[0171] Step 2: Write-write conflict detection. During the execution of a transaction, the OIDs of the documents modified by the committed transactions that are concurrently executed are saved in the conflict list of the transaction execution context, while the OIDs of the documents modified by the current transaction are recorded in the staging area index. When performing write-write conflict detection, the OIDs of the documents recorded in the staging area can be compared with the OIDs of the documents recorded in the conflict list. If there is an intersection between the OIDs recorded in the conflict list and the OIDs recorded in the staging area, the write-write conflict detection fails, that is, there is a write-write conflict. At this time, the transaction can be revoked and Step 9 can be executed. If there is no intersection between the OIDs recorded in the conflict list and the OIDs recorded in the staging area, the write-write conflict detection passes, that is, there is no write-write conflict. At this time, the system will traverse the active and uncommitted transactions and write all the document OIDs in the staging area of the current transaction into the conflict list of each traversed transaction. Then Step 3 can be executed.

[0172] Step 3: Record the transaction pre-commit in the operation log: PERSISTING TX txid.

[0173] Step 4: Flush the operation log buffer. This operation ensures that the operation log records written by the committed transaction are not lost.

[0174] Step 5: Apply the data modifications staged in the staging area. Traverse the staging area index and write the data in the staging area into the B+ tree (encode the data modifications in the staging area as tuples according to the OIDs stored in the staging area index, and write them into the B+ tree with the OID as the key and the tuple as the value). The tuples eliminated from the B+ tree will be written into the commit log.

[0175] Step 6: Record the transaction commit record in the operation log: COMMIT TX txid. Optionally, for every 5 transaction commits, an additional checkpoint record can be recorded: CHECKPOINT tx00 tx01 tx02, where tx00~tx02 is the list of transaction IDs that are currently in the "COMMITING" state. Step 7: Flush the operation log buffer. This operation ensures that the commit marker of the committed transaction is not lost.

[0176] Step 8: Modify the transaction status. Modify the transaction status to COMMITED and remove the current transaction from the active transaction list of the transaction manager.

[0177] Step 9: Return the transaction execution result. At this time, the conflict list and the staging area in the transaction execution context become invalid, and the operation results of document reading and writing are saved in the transaction operation sequence in the transaction execution context. The business application can check the transaction execution result and read the desired transaction execution result from the transaction execution context. Finally, the business application calls the transaction context destruction interface to clean up the transaction.

[0178] It should be noted that in the above Step 2 when performing write-write conflict detection, theoretically, regardless of whether the transaction has performed document insertion or modification operations, write-write conflict detection will be carried out. In particular, for a read-only transaction (where the document operations only include document query operations and do not include document insertion or modification operations), when performing write-write conflict detection, the detection result can be directly determined to be no write-write conflict (because it does not involve operations on document insertion or modification). In addition, after the write-write conflict detection of the transaction fails and the transaction is revoked, the transaction will not make any modifications to the B+ tree and the commit log, because the transaction revocation essentially does not require any IO operations and only needs to mark the transaction as revoked.

[0179] The above details how to execute various transaction operations in the first transaction. By executing various transaction operations in the first transaction, data management of documents under the YANG model constraints can be achieved. Among them, during the document insertion and query processes, the data management module needs to use the model schema tree dynamically maintained by the model management module to complete the document loading and verification, and through the staging area, write-write conflict detection, multi-version concurrency control (such as visibility rules, version chain query, etc. Optionally, the combination of the staging area, write-write conflict detection, and multi-version concurrency control can be replaced by other solutions with similar functions, such as the pessimistic concurrency control solution based on pessimistic locks), and the asynchronous document re-verification mechanism (re-verifying the document during document query), it is ensured that the documents read by the application must conform to the defined model schema tree, thus avoiding the phenomenon of inconsistency between the YANG schema and data in traditional RDBMS.

[0180] In some embodiments, during the execution of a certain transaction operation in the first transaction, a failure may occur. For example, a device restart due to power failure during the execution of a document insertion operation. In the case of a failure, it can be determined whether to perform failure recovery according to the execution status of the first transaction.

[0181] Taking the transaction manager M executing a complex transaction T as an example, this complex transaction T includes multiple document insertion and document query operations. Assume that at time t, the device restarts abnormally and the operation log and B+ tree of the transaction manager M are not damaged after the restart. Then, the following method can be used to determine whether to perform failure recovery: (1) At time t, transaction T is in the ready / running / aborted state. The operation log of M contains partial operation records of the transaction. Since the state of transaction T is ready, running, or aborted, transaction T has not yet reached (or passed) the write-write conflict detection (write-write conflict detection is performed when the transaction commits. In the case where transaction T is in the ready or running state, transaction T has not reached the write-write conflict detection. In the case where transaction T is in the aborted state, transaction T is aborted when it fails the write-write conflict detection). Correspondingly, transaction T has not yet modified the B+ tree and commit log of the transaction manager M. At the same time, the business application believes that transaction T has not been successfully committed, so the consistency guarantee still holds, and there is no need to perform a failure recovery operation at this time; (2) At time t, transaction T is in the "committing" state. When transaction T passes the write-write conflict detection, it will be in the "committing" state and modify the B+ tree and commit log of the transaction manager M. However, since transaction T fails during the commit, the modifications to the B+ tree and commit log are not all completed. At the same time, the application believes that transaction T has not been committed and should not modify the database, so the atomicity and consistency conditions are violated. Therefore, a transaction rollback operation is required, that is, a failure recovery needs to be performed to undo the modifications made by the transaction to the database; (3) At time t, transaction T is in the "committed" state. Transaction T has completed all persistent operations and is marked as completed. Since the business process after transaction T commits is not controlled by the database management system, whether the application completes the processing of the data returned by transaction T before the device restarts is unknown to the database management system. For simplicity, the database management system will not undo the updates made by transaction T to the database, but it is recommended that the business system recheck the overall consistency state after restarting.

[0182] It can be seen that in the event of a failure, for transactions in the "committing" state, a failure recovery is required to undo the modifications made by these transactions to the data blocks to ensure consistency.

[0183] When performing a failure recovery, in some embodiments, the failure recovery can be performed according to a failure recovery plan. Among them, the failure recovery plan is generated based on the operation log of the transaction manager and can include failure recovery records, which are used to undo the data modifications of the transaction.

[0184] In some embodiments, the failure recovery plan can be generated through the following steps S21 to S23: Step S21: Create a failure recovery context.

[0185] Step S22: Read the log records in the operation log in reverse order and update the failure recovery context according to the event type of the log records.

[0186] The fault recovery context may include a fault recovery staging area, a list of committed transactions, a list of transactions in progress, and a checkpoint flag bit. Among them, the fault recovery staging area is similar to the staging area of the transaction execution context and will not be described in detail here. The list of committed transactions is used to save the IDs of all transactions with commit records detected during the process of sequentially reading the log records in the operation log from back to front. The list of transactions in progress is used to save the IDs of all transactions that have not been detected with commit records but have been detected with pre-commit records during the process of sequentially reading the log records in the operation log from back to front. The checkpoint flag bit can indicate whether the checkpoint has been crossed when sequentially reading the log records in the operation log from back to front.

[0187] After creating the fault recovery context, during the process of sequentially reading the log records in the operation log in the order from back to front, the fault recovery context can be updated according to the event type of the log records.

[0188] In some embodiments, the event types of the log records may include transaction commit events or transaction pre-commit events. For example, if the form of the log record is COMMIT TX tx, the event type of this log record is a transaction commit event, and if the form of the log record is PERSISTING TX tx, the event type of this log record is a transaction pre-commit event. In this case, updating the fault recovery context according to the event type of the log record may include: When the event type of the log record is a transaction commit event, determine whether the checkpoint has been crossed; if the checkpoint has been crossed, ignore the log record (because if the checkpoint has been crossed, it means it is a committed transaction, so the log record can be ignored); if the checkpoint has not been crossed, add the transaction ID corresponding to the transaction commit event to the list of committed transactions (because if the checkpoint has not been crossed, it means it is an uncommitted transaction, so the ID of the transaction needs to be added to the list of committed transactions); When the event type of the log record is a transaction pre-commit event, determine whether the checkpoint has been crossed; if the checkpoint has been crossed, ignore the log record (because if the checkpoint has been crossed, it means it is a committed transaction, so the log record can be ignored); if the checkpoint has not been crossed, determine whether the list of committed transactions includes the transaction ID corresponding to the transaction pre-commit event; if the list of committed transactions does not include the transaction ID corresponding to the transaction pre-commit event, add the transaction ID corresponding to the transaction pre-commit event to the list of transactions in progress (because if the list of committed transactions does not include the transaction ID, it means the transaction is not committed, so the transaction ID needs to be added to the list of transactions in progress).

[0189] In some embodiments, the types of logged events may include transaction start events. For example, if the form of the log record is BEGIN TX tx, then the type of the logged event is a transaction start event. In this case, updating the failure recovery context according to the type of the logged event may include: Delete the transaction ID corresponding to the transaction start event from the list of committed transactions or the list of transactions in progress (since the start event of the transaction is detected, the operation log record of the transaction is closed, and no more transaction-related log records will appear when probing upward, so the ID of the transaction can be deleted from the list of committed transactions or the list of transactions in progress).

[0190] In some embodiments, the types of logged events may include document insertion events or document update events. For example, if the form of the log record is INSERT DOC [tx C OID D], then the type of the logged event is a document insertion event, indicating that transaction tx inserts document D with ID OID into data container C. If the form of the log is UPDATE[tx C OID DOC0 DOC1], then the type of the logged event is a document update event, indicating that transaction tx updates the document with ID OID in data container C from DOC0 to DOC1. In this case, updating the failure recovery context according to the type of the logged event may include: When the type of the logged event is a document insertion event, determine whether the list of transactions in progress includes the transaction ID corresponding to the document insertion event; when the list of transactions in progress includes the transaction ID corresponding to the document insertion event, write a first record in the failure recovery staging area of the failure recovery context, and the first record indicates deleting the data record inserted by the document insertion event; for example, {tx|C|OID|tombstone flag} can be written in the failure recovery staging area, which means deleting this record inserted by transaction tx; When the type of the logged event is a document update event, write a second record in the failure recovery staging area of the failure recovery context, and the second record indicates restoring the data updated by the document update event to its old value; for example, {tx|C|OID|DOC0} can be written in the failure recovery staging area, indicating restoring the document to its old value.

[0191] In some embodiments, the types of logged events may include checkpoint events. For example, if the form of the log record is CHECKPOINT tx01 tx02,…, then the type of the logged event is a checkpoint event, where tx01, etc. are the transactions in the "in progress" state at the time of the log record. In this case, updating the failure recovery context according to the type of the logged event may include: Determine whether the checkpoint has been passed; When the checkpoint has been passed, ignore the log record; When the checkpoint has not been passed, set the checkpoint flag to TRUE and determine whether the committed transaction list and the in - progress transaction list include the transaction ID corresponding to the checkpoint event; for any transaction ID corresponding to the checkpoint event, if the transaction ID is not included in both the committed transaction list and the in - progress transaction list, add the transaction ID to the in - progress transaction list (indicating that the transaction was not detected later and has been affecting the database at least since the checkpoint log record time until the failure occurred, and the transaction ID should be added to the in - progress transaction list); if the transaction ID is included in either the committed transaction list or the in - progress transaction list, ignore the log record.

[0192] Step S23: When the first log record of the operation log is read, or when the checkpoint has been passed and both the committed transaction list and the in - progress transaction list are empty, end the reading of the operation log and generate a failure recovery plan according to the failure recovery context.

[0193] During the process of reading the log records in the operation log from back to front, the above S22 can be executed in a loop until the first log record of the operation log is read, at which point the reading of the operation log ends, or until the checkpoint has been passed and both the committed transaction list and the in - progress transaction list are empty, at which point the reading of the operation log ends. After ending the reading of the operation log, a failure recovery plan can be generated according to the currently updated failure recovery context.

[0194] In some embodiments, generating a failure recovery plan according to the failure recovery context may include: Determine the cyclic redundancy check code of the failure recovery scratchpad area in the failure recovery context; Generate a failure recovery plan according to the cyclic redundancy check code and the records in the failure recovery scratchpad area.

[0195] After generating the failure recovery plan, in the case of needing to perform failure recovery, the failure recovery can be carried out according to the failure recovery plan. For example, all the failure recovery records in the failure recovery plan can be read in sequence, and according to the failure recovery records, the modification of the B + tree by the transaction can be undone.

[0196] Figure 21 It is a schematic diagram of the execution process of the failure recovery operation in an embodiment of the present application. Figure 21 The shown failure recovery operation process includes the following steps.

[0197] Step 1: Check whether there is a failure recovery plan in the specified directory. If it exists, load the failure recovery plan and execute step 12; if it does not exist, execute steps 2 to 11 to generate a failure recovery plan.

[0198] It should be noted that generally, when a failure occurs and recovery is carried out, there is generally no recovery plan under the specified target. The recovery plan needs to be generated through the following steps 2 to 11. In special cases, for example, if the device fails during or after the generation of the recovery plan, there may be a recovery plan in the specified directory at this time.

[0199] Step 2: Create the execution context for failure recovery. The execution context for failure recovery includes a failure recovery staging area, a committed transaction list L0, a transaction being committed list L1, a checkpoint flag bit, etc. Among them, the failure recovery staging area is similar to the staging area of the transaction execution context; L0 stores the transaction IDs of all detected transactions with commit records; L1 stores the transaction IDs of all transactions that have not been detected with commit records but have been detected with pre-commit records; the checkpoint flag bit indicates whether the checkpoint has been passed.

[0200] Step 3: Probe the records R in the operation log from back to front, and select the branch in Steps 4 to 9 according to the detected event type to maintain the failure recovery context. If R cannot match these situations, then ignore R (because in this case, it can be shown that R has nothing to do with failure recovery, so it can be ignored).

[0201] Step 4: The form of R is COMMIT TX tx, that is, it is detected that transaction tx has been committed. At this time, if the checkpoint flag bit has been passed, then ignore it; otherwise, add tx to the committed transaction list L0. Step 5: The form of R is PERSISTING TX tx, that is, it is detected that transaction tx has passed the write-write conflict detection. At this time, if the checkpoint flag bit has been passed, then ignore it. Otherwise, check whether L0 contains tx. If it does not contain it, it means that L0 has not been committed, and add tx to L1. Step 6: The form of R is BEGIN TX tx, that is, the start event of transaction tx is detected. Then the operation log record of tx is closed, and no more tx-related log records will appear when probing upward. Therefore, tx can be deleted from L0 or L1.

[0202] Step 7: The form of R is INSERT DOC [tx C OID D], that is, transaction tx inserts document D with ID OID into data container C. At this time, if transaction tx is in L1, then write {tx|C|OID|tombstone flag} to the failure recovery staging area, which means deleting the record inserted by transaction tx.

[0203] Step 8: The form of R is UPDATE [tx C OID DOC0 DOC1], that is, the transaction tx updates the document with the OID in the data container C from DOC0 to DOC1. Then, write {tx|C|OID|DOC0} to the fault recovery staging area, which means restoring the document to its old value.

[0204] Step 9: The form of R is CHECKPOINT tx01 tx02,…, that is, a checkpoint event is detected. tx01, etc. are the transactions that were in the "committing" state at the time of logging. At this time, if the checkpoint has been passed, no processing is required. Otherwise, set the checkpoint flag to TRUE and check one by one whether L0 and L1 contain the transaction ID (set as tx) in the record. If neither L0 nor L1 contains tx, it means that the transaction was not detected later. The transaction has affected the database at least since the checkpoint logging time until the failure occurred, and tx should be added to L1. If L0 or L1 contains tx, the transaction status is known and no processing is required.

[0205] Step 10: Loop through Steps 3 to 9 until the first record of the operation log is detected, or the checkpoint has been passed and L0 and L1 are empty.

[0206] Step 11: Calculate the CRC checksum of the data in the fault recovery staging area and dump the obtained checksum and the data in the fault recovery staging area to external storage. Thus, the operation log can be converted into a fault recovery plan.

[0207] Step 12: Read the records in the fault recovery staging area in sequence and apply the operations described in the records to the B+ tree to undo the modifications of the transaction to the B+ tree.

[0208] The data management method provided by the embodiments of this application can manage document data based on transactions, and uses data containers as the basic unit of data management. During the process of managing document data, it can ensure that the documents read by the business application from the data container must conform to the model schema tree defined by the data container, thus avoiding the phenomenon of YANG mode and data inconsistency in traditional RDBMS. In addition, since it can provide native ACID transactions designed for documents under YANG model constraints, on the business side, there is no need to implement relevant logic, and only need to define transactions according to the business process, which has less intrusion into business code and higher security.

[0209] The technical solutions provided by the embodiments of this application at least have the following technical effects: (1) Integrate the model management and data management functions into one, with data containers as the basic unit. The documents in the data container always conform to the model schema constraints, and there is no consistency problem between the model and the data.

[0210] (2) Users no longer need to define the conversion from the YANG model schema to the relational model, nor do they need to implement data verification additionally. The documents delivered to the system will be converted into a computationally friendly binary form for storage, with a small storage volume and fast loading speed.

[0211] (3) Provide native ACID transactions involved in YANG modeling data management, which are uniformly managed and scheduled by the system. The business side only needs to define transactions according to the business process.

[0212] (4) Provide a function library and its header files. Users can complete their YANG modeling data management process based on the function library, which is much lighter than common RDBMSs and more suitable for network devices with limited performance.

[0213] The database management system provided by the embodiments of this application provides all interfaces for the YANG model and its data management in the form of a function library, solves the problem of dependence on RDBMS in the traditional YANG modeling data management scenario, and greatly improves the read / write throughput of application programs accessing YANG constraint document data.

[0214] The application scenario of the embodiments of this application can be the access and storage of YANG modeling document data under network automation configuration. In addition, it can also be used in the following scenarios: (1) Offline YANG modeling document batch processing scenario. By integrating the function library function of the database management system provided by the embodiments of this application on the big data platform, the YANG modeling document data collected from the managed network devices is uniformly analyzed to discover the relationships and effective rules between the data, providing help for the construction of other applications within the organization.

[0215] (2) Existing mainstream RDBMS integration scenario. Inherited in the form of a plugin to the existing mainstream RDBMS, allowing access to the function library function through the session of the RDBMS, providing a more convenient ability to access and store YANG modeling document data.

[0216] (3) Distributed YANG modeling document data management scenario. By combining with distributed algorithms and protocols, a distributed version of the database management system cluster is designed based on the existing single-machine version to expand the YANG modeling document data management ability.

[0217] The above describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0218] Figure 22 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 22 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include internal memory, such as high-speed random access memory (Random-Access Memory, RAM), and may also include non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0219] The processor, network interface, and memory can be interconnected through an internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 22 only a bidirectional arrow is used in

[0220] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include internal memory and non-volatile memory, and provide instructions and data to the processor.

[0221] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a model management device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Receive a model management request, where the model management request includes the name of a first data container, and the model management request is used to request to manage the YANG model in the first data container; Manage the YANG model in the first data container according to the model management request.

[0222] The above is as in the present application Figure 22The method executed by the model management device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in the form of software. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with this application can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0223] The electronic device can also execute Figure 10 、 Figure 11 、 Figure 13 and Figure 14 the method, and implement the functions of the model management device in Figure 10 、 Figure 11 、 Figure 13 and Figure 14 the illustrated embodiment. Details are not described herein again in this application.

[0224] Of course, in addition to the software implementation method, the electronic device of this application does not exclude other implementation methods, such as a logic device or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.

[0225] This application also proposes a computer-readable storage medium that stores one or more programs. The one or more programs include instructions that, when executed by a portable electronic device including multiple application programs, can cause the portable electronic device to execute Figure 10 、Figure 11 , Figure 13 and Figure 14 the methods of the embodiments shown, and are specifically used to perform the following operations: Receive a model management request, where the model management request includes the name of a first data container, and the model management request is used to request management of the YANG model in the first data container; Manage the YANG model in the first data container according to the model management request.

[0226] Figure 23 is a schematic structural diagram of a model management device 230 according to an embodiment of the present application. Please refer to Figure 23 , in a software implementation manner, the model management device 230 may include: a receiving module 231 and a management module 232, where: The receiving module 231 receives a model management request, where the model management request includes the name of a first data container, and the model management request is used to request management of the YANG model in the first data container; The management module 232 manages the YANG model in the first data container according to the model management request.

[0227] The model management device 230 provided by the present application can also execute Figure 10 , Figure 11 , Figure 13 and Figure 14 the methods, and implement the functions of the model management device 230 in the Figure 10 , Figure 11 , Figure 13 and Figure 14 embodiments shown. The present application will not elaborate here.

[0228] Figure 24 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 24 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0229] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 24 only a bidirectional arrow is used in

[0230] Memory, used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0231] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data management device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Receive a processing request for a first transaction, where the first transaction is used to manage the document data in the data container; Process the first transaction according to the processing request.

[0232] The above as in this application Figure 24The method executed by the data management device disclosed in the embodiments shown can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with this application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0233] The electronic device can also execute Figures 15 to 21 the method and implement the functions of the model management device in Figures 15 to 21 the embodiments shown. Details are not described herein again in this application.

[0234] Of course, in addition to the software implementation, the electronic device of this application does not exclude other implementation manners, such as a logic device or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or a logic device.

[0235] This application also proposes a computer-readable storage medium. The computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by a portable electronic device including a plurality of application programs, can enable the portable electronic device to execute Figures 15 to 21 the method of the embodiments shown and specifically used to perform the following operations: Receive a processing request for a first transaction, where the first transaction is used to manage document data in a data container; Process the first transaction according to the processing request.

[0236] Figure 25 It is a schematic structural diagram of a data management device 250 according to an embodiment of the present application. Please refer to Figure 25 , in a software implementation manner, the data management device 250 may include: a receiving module 251 and a processing module 252, where: The receiving module 251 receives a processing request for a first transaction, where the first transaction is used to manage document data in a data container; The processing module 252 processes the first transaction according to the processing request.

[0237] The data management device 250 provided by the present application can also execute Figures 15 to 21 the method, and implement the functions of the data management device 250 in the Figures 15 to 21 illustrated embodiment. Details are not described herein again.

[0238] The present application also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. The computer program can be operated to cause a computer to execute some or all of the steps in the above-described method embodiments of model management, or execute some or all of the steps in the above-described method embodiments of data management.

[0239] In summary, the above are only the preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0240] The systems, devices, modules or units illustrated in the above embodiments may be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0241] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0242] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0243] Each embodiment in this application is described in a progressive manner. For the parts that are the same or similar among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the description of the method embodiment.

Claims

1. A database management system for providing document management capabilities under YANG model constraints, including: A model management module for providing management functions of the YANG model; A data management module for providing transaction-based document management functions; Wherein, the system uses data containers as the basic units for model management and data management, and the document data in the data containers conforms to the constraints of the YANG model in the data containers.

2. The system according to claim 1, wherein the management functions of the YANG model include at least one of model insertion, model export, and schema modification; The model management module is further configured to dynamically maintain YANG files, model schema trees, and model metadata; Among them, The YANG file is used to describe the YANG model; the model schema tree is composed of associated YANG models and is used to verify document data under the YANG model constraints; the model metadata is used to store the basic information and storage path of the YANG file, and the basic information of the YANG file includes at least one of the namespace, prefix, publisher organization, model description, and version of the YANG file.

3. The system according to claim 1, wherein the document management functions include at least one of document insertion, document deletion, document modification, and document query; The data management module includes a transaction manager, a thread pool, a staging area, an operation log, a B+ tree, and a commit log; Among them, The transaction manager is used for transaction management; The thread pool is used to execute transactions; the staging area is used to store the document data inserted or modified during the transaction execution and the corresponding document object identifier OID; the operation log is used to record the operations during the transaction execution; the B+ tree is used to store the document data verified by the YANG model; the commit log is used to record the document data eliminated from the B+ tree.

4. The system according to claim 3, wherein the transaction manager includes at least one of a default transaction manager and a custom transaction manager, and each transaction manager is used to manage a transaction group, and different transaction managers are used to manage different transaction groups; Among them, Different transaction groups are isolated from each other, different transactions within the same transaction group are isolated from each other, concurrent transactions within the same transaction group share the same thread pool, operation log, B+ tree, and commit log, and different transactions within the same transaction group each maintain their corresponding staging areas.

5. The system according to claim 3, wherein the B+ tree is used to store the document data verified by the YANG model in the form of key-value pairs, the key is the OID of the document, and the value is a tuple; The commit log is used to record the old version tuples eliminated from the B+ tree, and in the B+ tree and the commit log, the tuples corresponding to the same OID form a version chain; Among them, The tuple includes the following fields: A transaction ID, which is used to represent the ID of the transaction that writes the tuple into the B+ tree; A data container version, which is used to represent the data container version of the data field that finally verifies the tuple; An identifier, which is used to mark whether the tuple passes the verification of the data container corresponding to the data container version; A tuple version for pointing to the previous older version of the tuple; Data encoded in the LV format, where L is the length of the encoded document and V is the encoded document.

6. A model and data management system, comprising a business application and the database management system according to any one of claims 1 to 5. The database management system takes the form of a function library and is embedded into the business application in a linked manner to provide the business application with the document management ability under the YANG model constraint.

7. The model and data management system according to claim 6, wherein the database management system includes an application programming interface through which the business application accesses the database management system; wherein, The application programming interface includes: A model management interface for the business application to manage the YANG model; A data management interface for the business application to implement transaction-based document management; A transaction definition and execution interface for the business application to customize the transaction operation sequence and encapsulate it into transaction execution, as well as query the transaction execution status and read the transaction execution result; A system management interface for the business application to create, access, and destroy the operating environment of the database management system.

8. A model management method applied to the model and data management system according to claim 6 or 7, comprising: Receiving a model management request, which includes the identifier of the first data container and is used to request the management of the YANG model in the first data container; Managing the YANG model in the first data container according to the model management request.

9. The method according to claim 8, wherein the model management request includes a model insertion request, and the model insertion request carries a first YANG file. The managing the YANG model in the first data container according to the model management request includes: Determining whether the model insertion conditions are met, where the model insertion conditions include that there is no data in the first data container, there is no error in the first YANG file, and the YANG model described by the first YANG file is compatible with the YANG model in the first data container; When the model insertion conditions are met, merging the YANG model described by the first YANG file into the YANG model of the first data container, storing the first YANG file at a specified location, and updating the model metadata of the first data container.

10. The method according to claim 8, wherein the model management request includes a model export request, and the model export request includes an export location. The managing the YANG model in the first data container according to the model management request includes: Determining whether the model export conditions are met, where the model export conditions include that the export location corresponds to an empty directory; When the model export conditions are met, copying the model metadata and the corresponding YANG file of the first data container to the export location.

11. The method according to claim 8, wherein the model management request includes a schema modification request, the schema modification request includes a schema modification sequence, the schema modification sequence consists of at least one schema modification operation, and the schema modification operation includes a constraint for inserting or deleting a YANG file; The managing the YANG model in the first data container according to the model management request includes: Determine whether the pattern modification condition is satisfied, where the pattern modification condition includes that there is no other pattern modification operation on the first data container and the model export operation for the first data container is successfully executed; When the pattern modification condition is satisfied, process the YANG file of the first data container according to the pattern modification sequence to obtain a set of YANG files; Create a second data container, where the second data container has a YANG model and model metadata independent of the first data container and points to the data storage of the first data container; Add the YANG models in the set of YANG files to the second data container, and determine the second data container as the successor data container of the first data container.

12. A data management method applied to the model and data management system according to claim 6 or 7, including: Receive a processing request for a first transaction, where the first transaction is used to manage the document data in the data container; Process the first transaction according to the processing request.

13. The method according to claim 12, where the processing request includes an identifier of a first transaction manager; the processing the first transaction according to the processing request includes: Determine whether the first transaction is a legal transaction. A legal transaction starts with a transaction start operation, ends with a transaction commit operation, and includes at least one document operation between the transaction start operation and the transaction commit operation; When the first transaction is a legal transaction, submit the first transaction to the thread pool of the first transaction manager, and sequentially execute each transaction operation in the first transaction; When the first transaction is not a legal transaction, return the operation result that the first transaction is illegal.

14. The method according to claim 13, wherein the first transaction includes a transaction start operation; Executing the transaction start operation includes: Initialize the read view and the staging area in the transaction execution context. The read view is used to record the current transaction ID, the list of active transaction IDs, the minimum transaction ID for concurrent execution, and the pre-allocated next transaction ID. The current transaction ID is the ID of the first transaction. The staging area is used to store the document data inserted or modified during the transaction execution and the corresponding document OIDs; Add the ID of the first transaction to the list of active transaction IDs.

15. The method according to claim 13, wherein the first transaction includes a document insertion operation, and the identity of a first document and a third data container is carried in the document insertion operation; Executing the document insertion operation includes: Verify the first document according to the model pattern tree in the third data container; When the verification of the first document passes, assign an OID to the first document and write the first document into the staging area of the first transaction; When the verification of the first document fails, record the failure of the document insertion operation in the transaction execution context.

16. The method according to claim 13, wherein the first transaction includes a document query operation, and the first OID of the document to be queried and the identifier of the fourth data container are carried in the document query operation; Executing the document query operation includes: Perform a staging area query and a version chain query according to the first OID to obtain a document query result; When the document query result is empty, return a query failure message; When the document query result is not empty, perform document re-verification on the document query result according to the fourth data container; when the verification of the document query result passes, return the document query result; when the verification of the document query result fails, return the query failure information.

17. The method according to claim 16, wherein the querying the staging area and the version chain according to the first OID to obtain a document query result comprises: Querying whether there is a document corresponding to the first OID in the staging area; When there is a document corresponding to the first OID in the staging area, determining the document corresponding to the first OID in the staging area as the document query result; When there is no document corresponding to the first OID in the staging area of the first transaction, querying whether there is a first tuple corresponding to the first OID on the B+ tree; When the first tuple does not exist on the B+ tree, determining that the document query result is empty; When the first tuple exists on the B+ tree, determining whether the first tuple is visible to the first transaction; when the first tuple is visible to the first transaction, determining the first tuple as the document query result; when the first tuple is not visible to the first transaction, querying the old version tuple corresponding to the first tuple in the commit log according to the first OID; when there is a second tuple visible to the first transaction among the old version tuples corresponding to the first tuple, determining the second tuple as the document query result; when the second tuple does not exist among the old version tuples corresponding to the first tuple, determining that the document query result is empty.

18. The method according to claim 17, wherein the determining whether the first tuple is visible to the first transaction comprises: Determining the current transaction ID, the list of active transaction IDs, the minimum transaction ID for concurrent execution, and the pre-allocated next transaction ID recorded in the read view of the transaction execution context, where the current transaction ID is the ID of the first transaction; When a first condition is satisfied, determining that the first tuple is visible to the first transaction; When a second condition is satisfied, determining that the first tuple is not visible to the first transaction; wherein the first condition includes any one of the following: The transaction ID recorded in the first tuple is less than the minimum transaction ID; The transaction ID recorded in the first tuple is greater than or equal to the minimum transaction ID and less than the next transaction ID, and is not in the list of active transaction IDs; The second condition includes any one of the following: The transaction ID recorded in the first tuple is greater than or equal to the next transaction ID; The transaction ID recorded in the first tuple is greater than or equal to the minimum transaction ID and less than the next transaction ID, and is in the list of active transaction IDs.

19. The method according to claim 16, wherein the performing document re-verification on the document query result according to the fourth data container comprises: Determine whether the version of the data container recorded in the document query result is consistent with the version of the fourth data container; When the version of the data container recorded in the document query result is consistent with the version of the fourth data container, determine that the verification of the document query result passes; When the version of the data container recorded in the document query result is inconsistent with the version of the fourth data container, perform document re-verification on the document query result according to the model pattern tree in the fourth data container; when the document query result conforms to the constraints of the model pattern tree in the fourth data container, determine that the verification of the document query result passes; when the document query result does not conform to the constraints of the model pattern tree in the fourth data container, determine that the verification of the document query result fails; encapsulate the verification result of the document query result and the document query result into a tuple and save it in the staging area.

20. The method according to claim 13, wherein the first transaction includes a transaction commit operation; performing the transaction commit operation includes: Obtain a conflict list recorded in the transaction execution context, where the conflict list is used to record the OIDs of documents modified by concurrent transactions during transaction execution; Perform write-write conflict detection based on the document OIDs recorded in the staging area and the document OIDs recorded in the conflict list. Among them, when there is an intersection between the document OIDs recorded in the staging area and the document OIDs recorded in the conflict list, there is a write-write conflict; when there is no intersection between the document OIDs recorded in the staging area and the document OIDs recorded in the conflict list, there is no write-write conflict; When the detection result is that there is a write-write conflict, cancel the first transaction; When the detection result is that there is no write-write conflict, write the data modification in the staging area into the B+ tree in the form of a tuple, submit the first transaction and return the transaction execution result.

21. The method according to any one of claims 12 to 20, the method further includes: In the event of a failure, perform failure recovery according to a failure recovery plan, where the failure recovery plan includes failure recovery records, and the failure recovery records are used to cancel the data modification of the transaction; Among them, the failure recovery plan is generated through the following steps: Create a failure recovery context, where the failure recovery context includes a failure recovery staging area, a list of committed transactions, a list of transactions in progress, and a checkpoint flag; Read the log records in the operation log in reverse order, and update the failure recovery context according to the event type of the log records; When the first log record of the operation log is read, or when the checkpoint has been passed and both the list of committed transactions and the list of transactions in progress are empty, end the reading of the operation log, and generate the failure recovery plan according to the failure recovery context.

22. The method according to claim 21, wherein the event types of the log record include transaction commit events or transaction pre-commit events; The updating the failure recovery context according to the event type of the log record includes: When the event type of the log record is a transaction commit event, determine whether the checkpoint has been passed; if the checkpoint has been passed, ignore the log record; if the checkpoint has not been passed, add the transaction ID corresponding to the transaction commit event to the list of committed transactions; When the event type of the log record is a transaction pre-commit event, determine whether the checkpoint has been passed; if the checkpoint has been passed, ignore the log record; if the checkpoint has not been passed, determine whether the list of committed transactions includes the transaction ID corresponding to the transaction pre-commit event; if the list of committed transactions does not include the transaction ID corresponding to the transaction pre-commit event, add the transaction ID corresponding to the transaction pre-commit event to the list of in-progress transactions.

23. The method according to claim 21, wherein the event type of the log record includes a transaction start event; the updating the failure recovery context according to the event type of the log record includes: Delete the transaction ID corresponding to the transaction start event from the list of committed transactions or the list of in-progress transactions.

24. The method according to claim 21, wherein the event types of the log record include a document insertion event or a document update event; The updating the failure recovery context according to the event type of the log record includes: When the event type of the log record is a document insertion event, determine whether the list of in-progress transactions includes the transaction ID corresponding to the document insertion event; if the list of in-progress transactions includes the transaction ID corresponding to the document insertion event, write a first record in the failure recovery staging area, where the first record represents deleting the data record inserted by the document insertion event; When the event type of the log record is a document update event, write a second record in the failure recovery staging area, where the second record represents restoring the data updated by the document update event to the old value.

25. The method according to claim 21, wherein the event type of the log record includes a checkpoint event; the updating the failure recovery context according to the event type of the log record includes: Determine whether the checkpoint has been passed; If the checkpoint has been passed, ignore the log record; If the checkpoint has not been passed, set the checkpoint flag to TRUE, and determine whether the list of committed transactions and the list of in-progress transactions include the transaction ID corresponding to the checkpoint event; For any transaction ID corresponding to the checkpoint event, if neither the list of committed transactions nor the list of in-progress transactions includes the transaction ID, add the transaction ID to the list of in-progress transactions; If the list of committed transactions or the list of in-progress transactions includes the transaction ID, ignore the log record.

26. An electronic device, comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the method according to any one of claims 8 to 25.

27. A computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method according to any one of claims 8 to 25.

28. A computer program product, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to execute some or all of the steps of the method according to any one of claims 8 to 25.

Images