Data processing method for an object database

Abstract

The application belongs to the technical field of communication, and discloses a data processing method of an object database, wherein the object database contains a database table; the database table is a hash table, and data in the database table is stored in the form of classification and key-value pairs; the data is stored in the database table in the form of a binary sort tree; when the object database is started, the database table is mapped to a physical memory; and the object database provides API interfaces of multiple transactions to application program calling. The method is simple in design, good in data access performance, especially optimized for resource-intensive embedded devices, free from the restriction of open source software licenses, and capable of meeting the basic functional requirements of object access of embedded devices.

Description

Technical Field

[0001] This invention relates to the field of communication technology, specifically to a data processing method for an object database. Background Technology

[0002] For IoT device manufacturers, database solutions mainly fall into two categories:

[0003] Open source solutions: Existing open source solutions are usually quite large, consuming considerable storage space and memory, making them unsuitable for resource-constrained embedded devices. Furthermore, open source solutions must consider most use cases, sacrificing performance for versatility; the complex code presents a pain point for later maintenance and troubleshooting.

[0004] Existing self-developed solutions have the following drawbacks: low performance, inconvenient interface, outdated technology, and unreasonable design.

[0005] The technical problem addressed in this case is: how to optimize database access for existing embedded devices, improve access performance, and meet the basic functional requirements for object access in embedded devices. Summary of the Invention

[0006] The purpose of this invention is to provide a data processing method for an object database. This method is simple in design, has good data access performance, is especially optimized for resource-constrained embedded devices, and is not restricted by open-source software licenses. It can meet the basic functional requirements for object access in embedded devices.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a data processing method for an object database, wherein the object database contains database tables; the database tables are hash tables, and the data in the database tables is stored in the form of categories and key-value pairs; the data is stored in the database tables in the form of a binary search tree; when the object database starts, the database tables are mapped into physical memory; the object database provides various transaction API interfaces for application programs to call.

[0008] When an application wants to commit a transaction to an object database, the application calls the corresponding API interface. The database provides ping-pong and rollback caches for transactions on the database front end. The ping-pong cache is used to store the transaction request content and the processing result of the database back end. The rollback cache is used to store the position of the data modified by the transaction in the data table, as well as the content of the data before the modification.

[0009] The database backend processes transactions based on the request content and data in the rollback cache, and generates processing results to be returned to the ping-pong cache.

[0010] In the above-mentioned data processing method for object databases, the transaction refers to adding data, querying data, modifying data, and deleting data; the API interface is used to implement the addition, deletion, querying, and modification of data, as well as the submission and rollback of database tables.

[0011] In the above data processing method for the object database, when the database table in the request content needs to be submitted to the object database, the API interface for submitting the database table is called.

[0012] The database backend checks the data integrity of the database tables in the request content and whether the rollback cache is empty;

[0013] If the data in the database table in the request content is incorrect and the rollback cache is not empty, a rollback operation will be performed to restore all operations of the transaction on the database to the state at the beginning of the transaction, and an error message will be returned to the ping-pong cache.

[0014] If the data in the database table in the request content is correct and the rollback cache is not empty, then the commit operation is performed, and the database table in the object repository is overwritten with the database table in the request content.

[0015] In the above-described object database data processing method, when performing a rollback operation, the data in the rollback cache is compared with the corresponding data in the object database to confirm the accuracy of the data in the rollback cache; after confirming the accuracy of the data in the rollback cache, the data in the rollback cache is written to the corresponding position in the database table of the object database.

[0016] In the above-described object database data processing method, the transactions are processed in the form of queues; there is a processing queue for each application in the object database; the backend of the object database retrieves the request content from the processing queue for processing according to the first-in-first-out principle.

[0017] In the above-described object database data processing method, the binary search tree is a red-black tree or a weakly balanced binary tree.

[0018] In the above-described object database data processing method, the length of the hash table is calculated based on the available memory of the database during initialization and can be dynamically adjusted based on the available memory during operation. The adjustment is performed when the database program is idle.

[0019] In the above-described object database data processing method, the database assigns different permissions to each user; the permissions are permissions that exist in the form of invisible, read-only, or read-write for all or part of the categories in the database table.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] The advantages of this invention are as follows:

[0022] Object access: Stores and accesses data in units of objects, providing a simple and easy-to-use data access API. Input parameters are in key-value pair format, and values ​​are in JSON format. Its implementation is independent of the caller, achieving decoupling. Supports operations such as querying, adding, updating, deleting, committing, and rolling back.

[0023] As a preferred embodiment of the present invention, it has the following advantages:

[0024] Access control: All programs on the device access data only under controlled access permissions.

[0025] Concurrency support: Even when multiple processes access data simultaneously, the data's orderliness, integrity, correctness, and security are still guaranteed, and no performance bottlenecks will occur.

[0026] Resource consumption: It can dynamically adjust its own resource consumption based on the available hardware resources of the device (CPU usage, running memory, permanent storage space).

[0027] Transaction features: Supports simple transactions and allows rollback. Attached Figure Description

[0028] Figure 1 The data processing flow is shown in Embodiment 1 of the present invention;

[0029] Figure 2 This is the processing flow for submitting data to the database in Embodiment 1 of the present invention. Detailed Implementation

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example 1

[0032] refer to Figure 2A data processing method for an object database, wherein the object database contains database tables; the database tables are hash tables, and the data in the database tables is stored in the form of categories and key-value pairs; the data is stored in the database tables in the form of a binary search tree; when the object database starts, the database tables are mapped into physical memory; the object database provides various transaction API interfaces for application calls.

[0033] In this invention, the data storage mechanism is as follows: objects are stored in a data table (hash table + collision tree + rollback cache) in the form of category + key-value pairs. The data table is stored as a file on disk or in memory. Each category corresponds to an independent function (such as LAN, firewall, cellular network, SMS, etc.), and contains several key-value pairs. When the database starts, the data table in the storage device is mapped to physical memory through the operating system's memory mapping mechanism. If the data table in memory changes and a transaction needs to be committed, the entire data table is flushed to the storage device through the operating system's flush mechanism.

[0034] The data table structure is: a hash table + a collision tree. The length of the hash table is calculated based on available memory during database initialization and can be dynamically adjusted during operation based on available memory. The adjustment is performed when the database program is idle. The collision tree is used to store values ​​with the same hash value, using the key as an index. The structure can be a red-black tree or some kind of weakly balanced binary tree. The search speed should be fast, and the collapsible structure should be easy to store.

[0035] When an application wants to commit a transaction to an object database, the application calls the corresponding API interface. The database provides a ping-pong cache and a rollback cache for transactions on the database front end. The ping-pong cache is used to store the request content of the transaction and the processing result of the database back end. The rollback cache is used to store the position of the data modified by the transaction in the data table, as well as the content of the data before the modification.

[0036] The database backend processes transactions based on the request content and data in the rollback cache, and generates processing results to be returned to the ping-pong cache.

[0037] Ping-pong caching is used as a data cache, which can significantly improve data transmission efficiency. If two adjacent modules in the system have different operating speeds, with the former running faster than the latter, the results of the former cannot be immediately used by the latter. In this case, a cache is needed to temporarily store the results of the former, so that the operating speeds of the two levels can be independent of each other, and the operation of the former will not be affected by the latter.

[0038] Specifically, the transactions refer to adding, querying, modifying, and deleting data; the API interface is used to implement data creation, deletion, querying, modification, as well as database table commit and rollback. In this invention, the access logic and synchronization mechanism for the ping-pong cache are encapsulated into an independent, redistributable library file. The application only needs to call the APIs for creation, deletion, querying, modification, commit, and rollback, with the input parameter being the request and the output parameter being the result. The internal implementation is decoupled from the application. Besides API requests, the object database frontend does not accept any other form of request.

[0039] If new data needs to be retrieved, the application calls the API interface for retrieving new data. This request is submitted to the ping-pong cache as a transaction. The object database frontend scans the request, places the request content into the processing queue corresponding to the transaction, and the object database backend first calculates the hash value based on the key to quickly locate the hash table entry. Then, using the key, it searches the collision tree of the entry point to find the corresponding value and returns the value to the ping-pong cache. The application then retrieves the value from the ping-pong cache. If nested objects are encountered, the corresponding data should be obtained layer by layer from the innermost object, assembled into a complete nested object, and then returned.

[0040] For applications requiring data addition, deletion, and modification, the specific process is as follows: similar to a query process, but during the write process, the corresponding rollback cache for this process must be updated simultaneously. This cache records the position of the data involved in this transaction within the data table, as well as the content of the data before the modification. If the application calls the _commit() interface or starts a new transaction, the operating system's flush function is directly invoked to update the data table content to the storage device; if the application calls the _rollback() interface, the data table is rolled back through the rollback cache, and the storage device remains unchanged.

[0041] When performing the above operations, if data needs to be submitted, the API interface for submitting the database table is called; specifically, the steps are as follows:

[0042] Step 1: When a database table in the request content needs to be submitted to the object database, call the API interface for submitting the database table;

[0043] Step 2: The database backend checks the data integrity of the database tables in the request content and whether the rollback cache is empty. The method for verifying the integrity of the database table data is as follows: Before committing a new transaction, the database backend is responsible for ensuring strong consistency between the in-memory table and the disk data. Considering that the database table will gradually grow larger during use, it is extremely time-consuming and impractical to perform a full check of the target database table every time new data is committed. Therefore, only at each commit, the integrity check is performed on the data rows to be updated / added. If an error is found, the API interface returns an error message to the client; otherwise, only the in-memory data table is updated and the result is returned to the client, and the database backend flushes dirty pages according to the strategy.

[0044] Step 3: If the data in the database table in the request content is incorrect and the rollback cache is not empty, then perform a rollback operation to restore all operations of the transaction on the database to the state at the beginning of the transaction, and return an error message to the ping-pong cache;

[0045] Step 4: If the data in the database table in the request content is correct and the rollback cache is not empty, then perform the commit operation and overwrite the database table in the object repository with the database table in the request content.

[0046] When performing a rollback operation, the rollback cache is checked for correctness. This can be achieved by calculating the checksum of the entire cache when updating the rollback cache and checking the checksum for correctness before rolling back, thus ensuring that the data in the rollback cache is complete and correct throughout the process.

[0047] During rollback, the data in the cache is written back to the in-memory table on the storage device one by one. The purpose of the rollback cache is to save the database table data before any modifications are made at the start of the transaction, before the commit. After rollback, the contents of the in-memory table and the storage device will be consistent, so there's no need to update the storage device; otherwise, accessing the external device would be too costly. Similarly, after the commit is complete, the data in the in-memory table and the database table on the storage device will be consistent, and will contain the user-modified data. At this point, the contents of the rollback cache are meaningless and must be cleared.

[0048] If necessary, the data in the rollback cache can be compared with the data on the storage device. They should match; if they don't, the data is incorrect. The data table on the storage device should have a corresponding checksum, calculated when flushing the data table. If this value is incorrect, the data on the storage device is faulty.

[0049] Preferably, the transaction processing flow is as follows: Figure 1As shown, transactions are processed in the form of queues; each application has its own processing queue in the object database; the backend of the object database retrieves request content from the processing queue according to the first-in, first-out (FIFO) principle for processing. During data processing, the frontend of the object database scans the request, puts the request content into the processing queue (circular queue) corresponding to the transaction, and then transfers it to other request processing; the database backend retrieves the request from the circular queue, processes it accordingly, and returns the result to the ping-pong cache. The application waits in the ping-pong cache, reads the result, and continues its business process. This entire process occurs at the application layer, minimizing the risk of being pulled into the kernel.

[0050] In concurrent scenarios, multiple circular queues are used, with each ongoing transaction corresponding to one queue. The database frontend distributes requests to a specific queue based on the XAID in the request. Once the backend retrieves a request from a queue and begins processing it, it continues processing requests / transactions in that queue while simultaneously counting the transaction length until a transaction termination command is encountered. In extreme cases, excessively long transactions will result in an error. After the backend terminates (commits / rolls back) a transaction, it selects other transactions to process based on the first-in, first-out (FIFO) principle. This ensures the atomicity of each transaction and prevents time overlap between transactions, which could compromise data integrity.

[0051] In circular queues, frequent memory allocation (the frontend inserts requests into the queue) and deallocation (the backend retrieves requests from the queue) can easily lead to memory fragmentation and potentially cause unnecessary system calls. Therefore, a slab cache is introduced: at startup, one to several page frames are pre-allocated (using mmap), and these are sequentially divided into multiple memory blocks the size of queue nodes, which are then added to the available memory pool; when needed, a node is retrieved from the available pool (allocation), and after use, the node is returned to the available pool (deallocation). The use of the ptmalloc library is avoided as much as possible. This avoids unnecessary memory management mechanisms in the ptmalloc library, helping to improve efficiency; moreover, many complex details in ptmalloc are not visible, and this approach also reduces later maintenance costs.

[0052] Each application has its own independent ping-pong cache and request queue, ensuring isolation between applications; the database backend processes each request in units of transactions, ensuring the atomicity of transactions and the correctness of data; the database frontend and backend are in the same process / thread, and there is no contention between them when accessing the circular queue, ensuring the security of requests; all of the above processes are lock-free and try to avoid getting stuck in the kernel, ensuring that the database process can efficiently utilize CPU resources.

[0053] The application indicates that a request has been made by setting a flag on the "request" ping-pong cache; the database checks this flag, retrieves the request, and resets the flag. The application polls and waits on the "result" ping-pong cache, yielding some CPU time slices as appropriate, until the result flag is set. If this process takes too long, the client can terminate the wait and return a "request timeout" message.

[0054] In this invention, user permissions are optimized. Specifically, the object database is pre-configured with several accounts. Except for the administrator account, each account has invisible, read-only, and read-write permissions for each category. Each account has its own access view and can only read and write categories within that view according to its permissions; categories outside the view are invisible. The main process of permission management is performed at the front end of the object database.

[0055] The object database backend periodically checks memory and CPU usage. If it detects that usage exceeds configured thresholds, it uses the LRU mechanism to evict the oldest data, flush this data to storage, and shrink the data table to reduce memory usage. In cases of extreme memory shortage, the data table is converted to a weakly balanced tree structure to further reduce memory consumption. For CPU usage exceeding thresholds, a few CPU cycles are temporarily relinquished using `yeild` to alleviate CPU usage.

[0056] The backend backup subprocess of the object database backs up data periodically. When it finds that the data in the current database is incorrect, it restores the latest backup data to the current data table.

[0057] Compared with existing technical solutions, this invention has a simple design, good data access performance, and is especially optimized for resource-constrained embedded devices. It is also not restricted by open-source software licenses and can meet the basic functional requirements for object access in embedded devices.

Claims

1. A data processing method for an object database, characterized in that, The object database contains database tables; the database tables are hash tables, and the data in the database tables is stored in the form of categories and key-value pairs; the data is stored in the database tables in the form of a binary search tree; when the object database starts, the database tables are mapped into physical memory; The object database provides various transaction API interfaces for applications to call; When an application needs to commit a transaction to an object database, the application calls the corresponding API interface. The database provides ping-pong and rollback caches for transactions on the database front end. The ping-pong cache is used to store the transaction request content and the processing results from the database back end. The rollback cache is used to store the position of the modified data in the data table involved in the transaction, as well as the content before the data was modified; The database backend processes transactions based on the request content and data in the rollback cache, and generates processing results to be returned to the ping-pong cache. The transactions mentioned are adding data, querying data, modifying data, and deleting data; The API interface is used to implement data creation, deletion, querying, modification, as well as database table submission and rollback; If new data needs to be retrieved, the application calls the API interface for retrieving new data. At this time, the request is submitted to the ping-pong cache in the form of a transaction. The front end of the object database scans the request, puts the request content into the processing queue corresponding to the transaction, and the back end of the object database first calculates the hash value based on the key to quickly locate the hash table entry. Then, it searches for the corresponding value position from the collision tree of the entry using the key and returns the value content to the ping-pong cache. The application retrieves the value content from the ping-pong cache. For the application's needs to add, delete, and modify data, the specific process is as follows: In addition to the query process, the write process needs to simultaneously update the corresponding rollback cache of this process, which records the position of the data involved in this transaction in the data table, as well as the content of the data before the modification.

2. The data processing method for an object database according to claim 1, characterized in that, When the database table in the request content needs to be submitted to the object database, the API interface for submitting the database table is called; The database backend checks the correctness of the database table data in the request content and whether the rollback cache is empty; If the data in the database table in the request content is incorrect and the rollback cache is not empty, a rollback operation will be performed to restore all database operations of the transaction to the state at the beginning of the transaction, and an error message will be returned to the ping-pong cache. If the data in the database table in the request content is correct and the rollback cache is not empty, then the commit operation is performed, and the database table in the object repository is overwritten with the database table in the request content.

3. The data processing method for an object database according to claim 2, characterized in that, During the rollback operation, the data in the rollback cache is compared with the corresponding data in the object database to confirm the accuracy of the data in the rollback cache. After confirming the accuracy of the data in the rollback cache, the data in the rollback cache is written to the corresponding position in the database table of the object database.

4. The data processing method for an object database according to claim 1, characterized in that, The transactions are processed in the form of queues; there is a processing queue for each application in the object database; the backend of the object database retrieves the request content from the processing queue and processes it according to the first-in-first-out principle.

5. The data processing method for an object database according to claim 1, characterized in that, The binary search tree is a red-black tree or a weakly balanced binary tree.

6. The data processing method for an object database according to claim 1, characterized in that, The length of the hash table is calculated based on the available memory of the database during object database initialization, and can be dynamically adjusted during operation based on available memory. The adjustment is performed when the database program is idle.

7. The data processing method for an object database according to claim 1, characterized in that, The database assigns different permissions to each user; the permissions are permissions that exist in the form of invisible, read-only, or read-write permissions for all or part of the categories in the database table.

Images