Database, data searching method and data updating method
By using a database design that combines shared memory and a hash + tree hybrid algorithm, the access blocking problem caused by the TR069 database management process being managed independently was solved. This enabled multi-process parallel access and fast lookup, improving device smoothness and development efficiency.
Patent Information
- Application Number
- CN202410521933.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-28
- Publication Date
- 2025-10-28
AI Technical Summary
In complex databases such as TR069, the database management process of the gateway device manages the database independently, while other processes access it through inter-process communication, which is inefficient and prone to blocking, resulting in low access efficiency.
By employing shared memory technology and a hybrid algorithm combining hash tables and tree structures, the TR069 node is directly used as the database key to enable multi-process parallel access. Hash calculations facilitate fast lookup and dynamic node addition, eliminating the need for node conversion.
It improves access efficiency, reduces device memory costs, enhances development efficiency, and reduces device congestion. It is suitable for devices using the TR069 protocol, such as ONUs, gateways, and routers.
Smart Images
Figure CN120849422A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communication technology, specifically to a database, a data retrieval method, and a data update method. Background Technology
[0002] Databases are mainly used in large devices such as servers. For embedded devices such as gateways and routers, there are certain requirements for memory and performance. When dealing with complex databases such as TR069 and TR181, the gateway device mainly manages the database by a specific process, while other processes access the database indirectly through inter-process communication, which is inefficient and prone to blocking. Summary of the Invention
[0003] This application provides a database, a data retrieval method, and a data update method.
[0004] On one hand, embodiments of this application provide a database having shared memory, the database comprising:
[0005] The database attribute table, stored in the shared memory and non-volatile storage medium, is configured to store the database allocation attributes, which include: the number of basic data table entry pools, the table size and number of entries in each basic data table entry pool, the table size and number of entries in each extended table entry pool, and the hash table size.
[0006] The basic data table entry pool, stored in the shared memory, is configured to store real data, which includes: node attributes, attributes of all node parameters, and values of all node parameters;
[0007] Data table entries, including table header, key, attributes, and values;
[0008] Hash table entries are stored in the basic entry pool, and each hash table entry points to a data table entry.
[0009] The node table is configured to represent the relationships between nodes. The node table entries define the current node ID, parent node ID, number of child nodes, total number of nodes in the child node tree, node depth, and node name.
[0010] On the other hand, embodiments of this application provide a data retrieval method applied to the aforementioned database, the method comprising:
[0011] The hash value is obtained by performing a hash calculation based on the node URL;
[0012] The hash table entry is located based on the hash value to determine the hash table entry that matches the hash value.
[0013] Traverse the data table chain table pointed to by the hash table entry that matches the hash value, and compare the node URL with the URL of the data table entry in turn, until the node URL and the URL of the data table entry are completely consistent.
[0014] In another aspect, embodiments of this application provide a data update method applied to the aforementioned database, the method comprising: performing a hash calculation based on the node URL to obtain a hash value;
[0015] The hash table entry is located based on the hash value to determine the hash table entry that matches the hash value.
[0016] Traverse the data table chain table pointed to by the hash table entry that matches the hash value, and compare the node URL with the URL of the data table entry in turn, until the node URL and the URL of the data table entry are completely consistent;
[0017] The data table entries in the shared memory are fully read into the process memory space and then decompressed.
[0018] Modify the data in the unzipped table entries;
[0019] Copy the modified table entries to shared memory, overwriting the original data entries in shared memory.
[0020] According to the database, data search method, and data update method provided in the embodiments of this application, the database has shared memory and includes: a database, stored in the shared memory and a non-volatile storage medium, configured to store the database's allocation attributes, the database's allocation attributes including: the number of basic data table entry pools, the table size and number of entries in each basic data table entry pool, the table size and number of entries in each extended table entry pool, and the hash table size; a basic data table entry pool, stored in the shared memory, configured to store real data, the real data including: node attributes, attributes of all node parameters, and values of all node parameters; a data table entry, including an entry header, key, attributes, and values; a hash table entry, stored in the basic table entry pool, the hash table entry pointing to the data table entry; and a node table, configured to represent the relationships between nodes, the node table entry defining the current node ID, parent node ID, number of child nodes, total number of nodes in the child node tree, node depth, and node name. Compared to existing technologies, this method eliminates the need for configuration node conversion, allowing direct lookup using node keys. This significantly accelerates development when switching protocols. Shared memory enables parallel access by multiple processes, reducing blocking and improving access efficiency. A hybrid hash table + tree algorithm enables fast lookup and dynamic node addition. This effectively enhances the smoothness of device configuration for users, reduces device memory costs, and improves development efficiency. For time-sensitive requirements, it also reduces development difficulty. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a conceptual diagram of a tree node according to some embodiments of this application;
[0023] Figure 2 This is an illustration of database access provided according to some embodiments of this application. Figure 1 ;
[0024] Figure 3 This is an illustration of database access provided according to some embodiments of this application. Figure 2 ;
[0025] Figure 4 This is the overall structure of the database provided according to some embodiments of this application;
[0026] Figure 5 This is a schematic diagram of a data table entry pool in a database according to some embodiments of this application;
[0027] Figure 6 This is a schematic diagram of data table entries in a database provided according to some embodiments of this application;
[0028] Figure 7 This is a schematic diagram illustrating the relationship between hash table entries and data table entries in a database, according to some embodiments of this application.
[0029] Figure 8 This is a schematic diagram of a hash table entry in a database according to some embodiments of this application;
[0030] Figure 9 This is a schematic diagram of a node table in a database provided according to some embodiments of this application;
[0031] Figure 10 This is a schematic diagram illustrating the storage logic of a node table in a database according to some embodiments of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0033] Databases are primarily used in large devices such as servers. Embedded devices like Optical Network Units (ONUs), gateways, and routers have specific requirements for memory and performance. TR069 is a communication protocol for remote control terminal devices; similar protocols include TR098 and TR181. Saving configurations according to this protocol on the terminal side (e.g., ONU, gateway, or router) significantly speeds up access.
[0034] Figure 1 This is a conceptual diagram of a tree node provided according to some embodiments of this application.
[0035] Reference Figure 1 As shown in some examples, the configuration data defined by this type of protocol is typically defined in a tree node format.
[0036] For example, a node parameter can be represented in dotted fraction format as follows:
[0037] Object1.Object2.{i}.Object3.Object4.Object5.Parameter1
[0038] Here, Object1.Object2.{i}.Object3.Object4.Object5. represents the node where the final accessed parameter is located, and Parameter1 is the parameter 1 of that node.
[0039] In some examples, Objectx represents node x, {i} represents multiple instances (1-n), which can be dynamically added and deleted, and Parameterx represents parameter x. Nodes and child nodes, as well as nodes and parameters, are separated by a ".". For example:
[0040] <object ref="InternetGatewayDevice.XMpp.Connection.{i}.Server.{i}."
[0041] <parameter ref="Enable"requirement="readwrite" / >
[0042] <parameter ref="Priority"requirement="readwrite" / >
[0043] <parameter ref="Weight"requirement="readWrite" / >
[0044] <parameter ref="ServerAddress"requirement="readwrite" / >
[0045] <parameter ref="Port"requirement="readwrite" / >
[0046] Figure 2 This is a schematic diagram of database access provided according to some embodiments of this application.
[0047] Reference Figure 2 As shown, in some examples, gateway devices primarily manage the database independently, while other processes access the database indirectly through process-to-process communication.
[0048] For example, refer to Figure 2 As shown, TR069 management performs node conversion, and the database management process manages the database. Specifically, the first business process needs to communicate with the database management process to access the database, and the second business process also needs to communicate with the database management process to access the database.
[0049] Reference Figure 2 As shown, when the first business process communicates with the database management process and accesses the database, the database management process is occupied, and the second business process cannot communicate with the database management process, resulting in the second business process being blocked. The second business process needs to wait for the first business process to finish accessing the database before it can communicate with the database management process and access the database, resulting in low access efficiency.
[0050] Figure 3 This script application provides database access illustrations for some embodiments. Figure 2 .
[0051] In some examples, to address the issue of access blocking and low efficiency caused by communication between independent database management processes and business processes; refer to Figure 3 As shown, in some examples of embodiments of this application, the database management process is removed, which can significantly improve performance. In addition, the TR069 node is directly used as the database key, eliminating the node conversion process and facilitating protocol switching and upgrades.
[0052] Figure 4 This describes the overall structure of a database provided according to some embodiments of this application.
[0053] In some examples, refer to Figure 4 As shown, to match the TR069 tree nodes and enable shared access by multiple processes, file trees, i.e., shared memory technology, can be used.
[0054] In some examples, the file tree approach fits well with tree-like nodes, but it is less efficient, consumes more memory, and is insecure. While implementing a tree structure with variable data length using shared memory requires more advanced technology, it offers high security, high efficiency, and low memory consumption. Therefore, the databases provided in some embodiments of this application can be implemented based on shared memory technology.
[0055] In contrast to databases in non-shared memory, to match the characteristics of shared memory and satisfy the TR069 data characteristics, some examples of embodiments in this application may use a hash + tree hybrid approach.
[0056] In some examples, hash algorithms can be used to quickly set and look up the attributes and values of a single node, and trees can be used to assist hashing in traversing nodes and their children, as well as adding and deleting dynamic nodes and their bytes.
[0057] In some examples, refer to Figure 4 As shown, in order to achieve fast data access and save shared memory resources, in some examples of embodiments of this application, the database may have a key.
[0058] In some examples, the node defined in the TR069 protocol can be used directly as the key for hash calculation.
[0059] In some examples, the database may include hash tables.
[0060] In some examples, the hash table can generate a preset number of hash table entries based on the total number of nodes.
[0061] In some examples, the larger the number of hash table entries, the fewer hash collisions there are, but the larger the memory usage.
[0062] In some examples, the hash table entries can be 4096 in devices such as ONUs, gateways, or routers.
[0063] In some examples, the hash table entries can be 8192 in devices such as ONUs, gateways, or routers.
[0064] In some examples, hash table entries can point to the final data node.
[0065] In some examples, when different data nodes have the same hash table entries, multiple data nodes can be connected using a linked list.
[0066] Figure 5 This is a schematic diagram of a data table entry pool in a database according to some embodiments of this application.
[0067] In some examples, the database may have shared memory, and the database may include an underlying pool of data table entries.
[0068] In some examples, the underlying data table entry pool can include three.
[0069] In some examples, the underlying data table entry pool may include a first data table entry pool.
[0070] In some examples, the base data table entry pool may include a second data table entry pool.
[0071] In some examples, the base data table entry pool may include a third data table entry pool.
[0072] In some examples, dynamically allocating memory to each data node in shared memory can consume a large amount of shared memory resources, and the range of each access is complex; therefore, in some examples of embodiments of this application, the sizes of the three basic table entry pools can be different.
[0073] In some examples, refer to Figure 5 As shown, the first data table entry pool can have a first data table entry length.
[0074] In some examples, the length of the first data table entry can be 512 bytes.
[0075] In some examples, the second data entry pool may have a second data entry length.
[0076] In some examples, the length of the second data table entry can be 2048 bytes.
[0077] In some examples, the third data table entry pool can have a third data table entry length.
[0078] In some examples, the length of the third data table entry can be 8192 bytes.
[0079] It is understood that in some examples of the embodiments of this application, the length of the first data table entry, the length of the second data table entry, and the length of the third data table entry are only shown as specific examples and are not intended to limit the length of the data table entries.
[0080] In some examples, when the size required for the actual data node is within the range of the data table entry length of the three basic data table entry pools, a data table entry storage data node can be requested from the three basic data table entry pools.
[0081] In some examples, the data table entry pool can be configured to store actual data.
[0082] In some examples, the data table entry pool can be configured to store the attributes of nodes (defined by TR069), the attributes of all parameters of the node (defined by TR069), and the values of all parameters of the node.
[0083] In some examples, the allocation, release, and first access of shared memory require additional support, consume additional resources, and are inconvenient to manage.
[0084] In some examples, since the data length of each data entry is not the same according to the definition of TR069, and the length of the data entry may also change when the modifier parameter value is modified, multiple data entry pools of various lengths can be defined in some examples of embodiments of this application.
[0085] In some examples, the length of the data table entry pool can be defined based on the possible data length distribution.
[0086] In some examples, the first data table entry pool can have a first number of data table entries.
[0087] In some examples, the number of entries in the first data table can be 1500.
[0088] In some examples, the number of entries in the second data table can be 500.
[0089] In some examples, the number of third data table entries can be 100.
[0090] It is understood that in some examples of the embodiments of this application, the number of first data table entries, the number of second data table entries, and the number of third data table entries are only shown as specific examples and are not intended to limit the number of data table entries.
[0091] In some examples, initialization can be performed during system startup, and three segments of shared memory can be allocated.
[0092] In some examples, the three shared memory segments can be designated as a first shared memory, a second shared memory, and a third shared memory.
[0093] In some examples, the size of the first shared memory can be 512 × 1500 bytes. That is, the first shared memory is the first data table entry pool.
[0094] In some examples, the size of the second shared memory can be 2048 × 500 bytes. That is, the second shared memory is the second data table entry pool.
[0095] In some examples, the size of the third shared memory can be 8192 × 100 bytes. That is, the third shared memory can be a third data table entry pool.
[0096] In some examples, the length and number of the underlying data table item pool can be defined in the database attribute table.
[0097] In some examples, when a TR069 node data table entry needs to be created, a free data table entry can be requested from the first data table entry pool, the second data table entry pool, or the third data table entry pool, and the data table entry can be marked as used.
[0098] In some examples, when deleting a data table entry for the TR069 node, the entry can be released back into the data table entry pool and marked as free.
[0099] In some examples, if the length of valid data in a data entry is less than or equal to the length of the first data entry, a first free entry can be obtained from the first data entry pool. The data entry can be stored in the first free entry.
[0100] In some examples, if the length of valid data in a data entry is greater than the length of the first data entry and less than or equal to the length of the second data entry, a second free entry is obtained from the second data entry pool, and the data entry is stored in the second free entry.
[0101] In some examples, if the length of valid data in a data entry is greater than the length of the second data entry and the length of valid data is less than or equal to the length of the third data entry, a third free entry is obtained from the pool of third data entries, and the data entry is stored in the third free entry.
[0102] In some examples, an extended data entry pool can be used when the valid data in the actual entry exceeds the length of the third data entry. Data entries can be stored in extended data entries.
[0103] In some examples, the length of the extended data table entry pool can be requested based on the length of the valid data.
[0104] In some examples, the length of the extended data entry pool can be the effective data length rounded up to the power of 2, and the number is 20. For example, if the effective data is 10,000 bytes, then an extended data entry pool of 16384 * 20 bytes will be requested.
[0105] In some examples, the size and number of the base data entry pool can be adjusted based on the possible data lengths used by different protocols to ensure that the use of the extended data entry pool is minimized.
[0106] In some examples, when a node is deleted, the entry can be released back into the underlying data entry pool.
[0107] In some examples, the database may include an extended pool of data table entries.
[0108] In some examples, the extended data table entry pool can also be set in shared memory.
[0109] In some examples, when the resources of the three basic data table entry pools are exhausted, entries can be requested from the extended data table entry pool.
[0110] In some examples, the database may include node tables.
[0111] In some examples, the node table can store the relationships between nodes.
[0112] In some examples, the relationships between nodes can be a tree structure.
[0113] In some examples, the relationships between nodes can be stored as an array table. This facilitates the management of relationships between nodes within shared memory.
[0114] Figure 6 This is a schematic diagram of data table entries in a database provided according to some embodiments of this application.
[0115] In some examples, because table entries are allocated in a pooled manner, effective data can be compressed during storage to reduce memory usage.
[0116] In some examples, refer to Figure 6 As shown, a data table entry may include a table header.
[0117] In some examples, the table header may include the address information of the next table entry.
[0118] In some examples, the table header may include the offset of the next table entry.
[0119] In some examples, the shared memory number of the data entry pool and the offset index of the data entry pool can be used.
[0120] In some examples, the table header may include an indicator of whether it is occupied.
[0121] In some examples, the "occupied" flag can be configured to manage the allocation and release of entries in the entry pool. For example, the flag can be "occupied" when an entry pool allocation is being used. Alternatively, the flag can be "idle" when an entry pool is being released.
[0122] It is understood that in some examples of the embodiments of this application, the identifier of whether it is occupied is only shown as a specific example. In other examples, other types of identifiers can also be used to identify whether it is occupied. The embodiments of this application do not limit this.
[0123] In some examples, the table header may include the table length. The table length can represent the total length of the table entry.
[0124] In some examples, the table header may include the valid data length. The valid data length can represent the actual length of data used.
[0125] In some examples, the table header may include the key length. The key length can indicate the length of the key.
[0126] In some examples, the table header may include the attribute length.
[0127] In some examples, the table header may include the attribute length.
[0128] In some examples, valid data may include a key. The key can be the complete node URL, i.e., the fully qualified name of the node as defined by TR069.
[0129] In some examples, the key can be in the format Obj1.Obj2.Obj3.Obj4…Objn.
[0130] In some examples, the key can be used for hash calculations.
[0131] In some examples, the key can be used for exact match lookups after hash calculation.
[0132] In some examples, valid data may include an attribute section. The attribute section may contain node attributes, attributes of all parameters to which the node belongs, and values of all parameters to which the node belongs.
[0133] In some examples, since the length of the attribute does not change, but the length of the value may change, all parameter attributes and values can be stored in segments to improve efficiency.
[0134] In some examples, you can find specific parameters based on the length and the booking characteristics of the attributes.
[0135] In some examples, the content of the attribute can be defined by the TR069 protocol.
[0136] In some examples, valid data may include values. Values can be the values of all parameters.
[0137] In some examples, for ease of management, the values can be converted to strings and saved, separated by '\0'.
[0138] Figure 7 This is a schematic diagram showing the pointers between hash table entries and data table entries in a database according to some embodiments of this application.
[0139] In some examples, hash calculations can be performed using the key to obtain hash table entries.
[0140] In some examples, refer to Figure 7 As shown, when multiple keys have the same hash value after hash calculation, the data table entries corresponding to multiple keys can be associated with each other through the shared memory number of the next table entry.
[0141] In some examples, when looking up a table entry using a key, the hash table entry is first found through hash calculation, and then the data table entries pointed to by the hash table entries are traversed sequentially to compare whether the key in the data table entry is the key being looked up.
[0142] In some examples, the more hash table entries there are, the fewer data table entries the same hash table entry points to, and the higher the lookup efficiency.
[0143] In some examples, for the TR069 protocol, the number of hash table entries can be 4096 or 8192. This ensures that most data table entries pointed to by the same hash table entry are of the same type, with a maximum of three, resulting in high efficiency and low memory consumption.
[0144] It is understood that in some examples of the embodiments of this application, the number of hash table entries is only shown as specific examples and is not a limitation on the specific number of hash table entries.
[0145] Figure 8 This is a schematic diagram of a hash table entry in a database provided according to some embodiments of this application.
[0146] In some examples, a hash table entry may include a shared memory number pointing to the first data table entry.
[0147] In some examples, a hash table entry may include an offset pointing to the first data table entry.
[0148] In some examples, nodes need to support dynamic addition and deletion; therefore, locks are required to protect critical sections.
[0149] Figure 9 This is a schematic diagram of a node table in a database provided according to some embodiments of this application. Figure 10 This is a schematic diagram illustrating the storage logic of a node table in a database according to some embodiments of this application.
[0150] In some examples, the node table can store the relationships between nodes. The node table may not contain actual dynamic node instances or actual data. Therefore, it will not change.
[0151] In some examples, for suitability for use in shared memory, refer to Figure 9 As shown, the node table can use contiguous memory.
[0152] In some examples, each node table entry can be the same size. The number of node tables can be defined by the TR069 protocol.
[0153] In some examples, the node ID is the index of the node table entry in the table. A specific node can be quickly located using the ordered node table entry and the specified node ID.
[0154] In some examples, to facilitate reconstruction into a tree structure, node entries can include the current node ID, parent node ID, number of child nodes, total number of nodes in the child node tree, node depth, and node name. This allows for quick reconstruction into a tree structure.
[0155] In some examples, node depth can be the level of its child nodes, starting from the root node. The top-level node can be the root node, with a depth of 0. The depth of the root node's child nodes can be 1, the depth of the child nodes of the child nodes can be 2, and so on.
[0156] In some examples, refer to Figure 10 As shown, the following explanation uses three node URLs as an example:
[0157] obj1.obj2.obj4.obj6,
[0158] obj1.obj2.obj5.obj7.
[0159] obj1.obj3.
[0160] In this context, obj1 is the parent node of obj2 and obj3, and obj2 and obj3 are both child nodes of obj1; obj2 is the parent node of obj4 and obj5, and obj4 and obj5 are both child nodes of obj2; obj4 is the parent node of obj6, and obj6 is a child node of obj4; obj5 is the parent node of obj7, and obj7 is a child node of obj5.
[0161] obj2 and obj3 are sibling nodes, and obj4 and obj5 are sibling nodes.
[0162] In some examples, obj1 has 2 child nodes and the total number of nodes in the child node tree is 6; obj2 has 2 child nodes and the total number of nodes in the child node tree is 4; obj has 0 child nodes and the total number of nodes in the child node tree is 0.
[0163] In some examples, refer to Figure 10 As shown, the way node entries are stored in the node table can be as follows: first store the root node, then store the first child node of the root node, then store the first child node of the first child node, and so on; then store the second child node of the root node, and so on, until all nodes are saved.
[0164] In some examples, the database may include a database attribute table. The database attribute table can store some of the database's assignment properties.
[0165] In some examples, the database attribute table may include the number of underlying data table entry pools.
[0166] In some examples, the database attribute table may include the size and number of entries for a pool of basic data table entries.
[0167] In some examples, the database attribute table may include the size and number of entries for an extended pool of entries.
[0168] In some examples, the database attribute table may include the size of the hash table.
[0169] In some examples, database attribute tables can be stored in shared memory.
[0170] In some examples, database attribute tables can be stored in non-volatile storage media. Non-volatile storage media can include flash memory.
[0171] In some examples, when the system powers on, database property tables can be read from flash memory into shared memory.
[0172] In some examples, when the database property table is modified, the changes can be synchronized to flash.
[0173] In some examples, the database property table can be actively modified by the user.
[0174] In some examples, the database attribute table can be modified automatically.
[0175] In some examples, a database management initialization process can be started when the system powers on to initialize the shared-memory database. After the initialization process completes, it exits, and the database can be maintained by individual business processes directly calling database library functions.
[0176] In some examples, database property tables can be loaded from flash into shared memory space.
[0177] In some examples, user-modifiable data node files defined by TE069 are loaded and stored in node table format.
[0178] In some examples, the data node file can be defined by TE069.
[0179] In some examples, the data node file can be a file defined by the user using the TR069 format.
[0180] In some examples, the data node file can be in XML file format.
[0181] In some examples, when converting data node files into node tables, the data node files can be stored in local memory first, and then shared memory of the same size can be requested based on the final size of the node table. The node table in the local storage space can then be copied to the shared memory space.
[0182] In some examples, a pool of data entries can be pre-allocated from shared memory based on the data entry pool attributes defined in the database attribute table.
[0183] In some examples, a hash table can be pre-allocated from shared memory based on the hash table attribute defined in the database attribute table.
[0184] In some examples, a data entry can be generated in local memory space based on the initial data value defined in the node file, and a data entry can be requested from the data entry pool based on the size of the generated data entry; the hash value can be calculated based on the node URL, and the data entry index can be adjusted to the corresponding hash entry.
[0185] In some examples, since only the process location database is initialized during hash calculation, all locks can be closed directly, thereby improving efficiency.
[0186] In some examples of embodiments of this application, based on the above-mentioned database, data can be searched in the following two ways:
[0187] The first method is to search based on the complete node URL;
[0188] First, the node URL is hashed, and the hash value is used to find the hash table entry.
[0189] Lock the found hash table entry, then traverse the data table linked by the hash table entry, and compare the node URL with the data table entry URL to see if they are completely consistent.
[0190] The data lookup is successful if the node URL and the data table entry URL are exactly the same.
[0191] If the node URL does not exactly match the URL of the data table item, continue searching for the next data table item.
[0192] In some examples, an error is returned indicating that the lookup failed when the URLs of all data entries in the linked list do not match the node URLs.
[0193] In some examples, hash table entries can be unlocked after the lookup is complete.
[0194] The second method is to search based on partial URLs;
[0195] In some examples, a fuzzy search can be used to list all data entries that match partial URLs, making it convenient for users who don't remember the complete URL or find it inconvenient to type in a long URL.
[0196] In some examples, it may be possible to access only the complete URL of the matching data table item.
[0197] In some examples, the values of all parameters contained in a data table item can be listed at the same time.
[0198] In some examples, you can start from the first entry of the hash table and traverse all entries sequentially; if an entry does not have a data entry index, skip it and proceed to the next hash table entry; if an entry has a data entry index, lock that hash table entry.
[0199] In some examples, the locked hip-hop table entry points to the necklace table, and each table entry's URL is compared to see if it partially contains the URL to be searched. If it does, the table entry information is listed, and then the next entry is traversed.
[0200] In some examples, URL matching can employ fuzzy case matching.
[0201] In some examples, after a node is found, it can be read.
[0202] In some examples, to improve node reading efficiency, the following two methods can be used for reading:
[0203] The first method allows for a complete read;
[0204] In some examples, data table entries in shared memory can be completely copied to the process's memory space.
[0205] In some examples, data table entries can be decompressed in the process's memory space, and then parameter data can be read.
[0206] In some examples, although a full read requires copying a large amount of memory and has low read efficiency, all information can be obtained by searching sequentially, making it suitable for reading a large number of parameters and attributes of a node at the same time.
[0207] The second method allows for one-way reading;
[0208] In some examples, data can be parsed directly in shared memory, and specified parameter or attribute values can be copied to the process memory space.
[0209] In some examples, one-way read copying is efficient because it involves small amounts of data; however, it can only read one parameter value at a time, and each time it requires searching, locking, and unlocking. Therefore, it is suitable for searching a small number of parameters.
[0210] In some examples, nodes can be modified after they are found. To improve modification efficiency, nodes can be modified in the following two ways:
[0211] The first modification method allows for a complete modification;
[0212] In some examples, the data table entries in shared memory can be read completely into the process memory space first.
[0213] In some examples, data table entries can be decompressed in the process's memory space before being modified.
[0214] In some examples, the modified data table entries in the process memory space can be completely copied to the shared memory space, overwriting the original data table entries in the shared memory space.
[0215] The second modification method can be unidirectional modification;
[0216] In some examples, you can first read the data table item parameters or attribute values in the shared training in one direction, compare them with the length of the value to be written, and if the length is unchanged, you can directly parse the data in the shared memory and write the specified parameter value into the shared memory.
[0217] In some examples, if the length of the value to be written changes, you can jump to the complete modification process of the first method.
[0218] In some examples, to reduce memory usage, data entries in shared memory can be compressed. When the length changes, the compression needs to be reassembled. Therefore, when modifying a value whose length changes or when multiple parameters are written, the first modification method (complete modification) can be used to improve modification efficiency; while in other cases, unidirectional modification can be used to improve modification efficiency.
[0219] In some examples, the database can add nodes.
[0220] In some examples, regular nodes can be added only during initialization, while dynamic nodes can be added at any time.
[0221] In some examples, data table entries in local memory space can be generated first and then compressed into the format of data table entries in shared memory.
[0222] In some examples, shared training data entries can be requested from the shared memory data entry pool based on the size of the data entry.
[0223] In some examples, you can iterate through the shared memory data table entry pool, find the first entry whose occupancy flag is "free", and change the occupancy flag of the entry to "occupied" to indicate that it is occupied.
[0224] In some examples, you can iterate through the shared memory data entry pool, find the first entry with an occupancy flag of 0, and set the occupancy flag to 1 to indicate that the entry is occupied.
[0225] In some examples, data table entries in the local memory space can be copied to shared memory table entries.
[0226] In some examples, the database can calculate the hash value of the added node URL and find the hash table entry corresponding to the hash value.
[0227] In some examples, the database can lock hash table entries and traverse the data table associated with the hash table entries.
[0228] In some examples, after finding the table position of the linked list, the indexes of the requested shared memory data table entries (including table id and index) can be filled into the table position of the linked list.
[0229] In some examples, if it is the first time a hash table entry is used, the data table entry index can be populated into the hash table entry, and then the hash table entry can be unlocked.
[0230] In some examples, when adding dynamic nodes, the new node first copies all information from node 0 and all its child nodes. However, since the child nodes of the hash structure are scattered, a node table is needed to achieve this.
[0231] In some examples, the database can first look up the node table using the URL of the node to be added as the key, and then generate the URLs of the left and right child nodes.
[0232] In some examples, the database can find all data table entries for the child nodes of node 0 and completely modify them to the newly added data table entries that define the corresponding child nodes.
[0233] In some examples, you can eventually find the data table entry for node 0 itself and completely modify it to the data table entry for the newly added node. This completes the addition of the dynamic node.
[0234] In some examples, the database can delete nodes.
[0235] In some examples, the node to be deleted can be found based on the complete URL. The "occupied" position of the entry is set to 0, and then it is deleted from the list.
[0236] In some examples, for the deletion of dynamic nodes, you can first look up the node table using the URL of the node to be added as the key and generate all child node URLs.
[0237] In some examples, nodes can be found and deleted sequentially based on their child node URLs.
[0238] In some examples, database properties can be optimized automatically.
[0239] In some examples, after the system is first powered on, initial values can be used, with three data table entry pools of 512*1500, 2048*500, and 8192*100 respectively, and the number of hash tables is 4096.
[0240] In some examples, during database operation, it's possible to check whether an expanded data table entry pool is needed each time a node is added.
[0241] In some examples, if an extended data entry pool is needed, the quantity in the extended data entry pool is merged with the quantity in the base data entry pool and saved to the database properties table. This way, on the next startup, a new data entry pool can be created with the new data entry pool properties.
[0242] In some examples, the database can check the occupancy rate and collision rate of entries in a hash table.
[0243] In some examples, when the hash table entry occupancy rate and collision rate reach critical values, the hash table size attribute in the database properties is modified to twice its original value. For example, if the original value is 4096, it can be modified to 8192.
[0244] In some examples, the threshold for hash table entry occupancy can be 80%, and the collision rate can be defined as the number of hash tables with more than 3 chain tables exceeding 0.1% of the total, or the number of chain tables exceeding 4.
[0245] According to the database, data search method, and data update method provided in the embodiments of this application, the database has shared memory and includes: a database, stored in the shared memory and a non-volatile storage medium, configured to store the database's allocation attributes, the database's allocation attributes including: the number of basic data table entry pools, the table size and number of entries in each basic data table entry pool, the table size and number of entries in each extended table entry pool, and the hash table size; a basic data table entry pool, stored in the shared memory, configured to store real data, the real data including: node attributes, attributes of all node parameters, and values of all node parameters; a data table entry, including an entry header, key, attributes, and values; a hash table entry, stored in the basic table entry pool, the hash table entry pointing to the data table entry; and a node table, configured to represent the relationships between nodes, the node table entry defining the current node ID, parent node ID, number of child nodes, total number of nodes in the child node tree, node depth, and node name. Compared to existing technologies, this method eliminates the need for configuration node conversion, allowing direct lookup using node keys. This significantly accelerates development when switching protocols. Shared memory enables parallel access by multiple processes, reducing blocking and improving access efficiency. A hybrid hash table + tree algorithm enables fast lookup and dynamic node addition. This effectively enhances the smoothness of device configuration for users, reduces device memory costs, and improves development efficiency. For time-sensitive requirements, it also reduces development difficulty.
[0246] In addition, data compression reduces memory usage; and low-granularity locking reduces lock contention and improves access efficiency.
[0247] In some examples, the database can be used as a regular database even without using the TR069 protocol, thus expanding the scope of database usage.
[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
[0249] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.
Claims
1. A database having shared memory, characterized in that, The database includes: The database attribute table, stored in the shared memory and non-volatile storage medium, is configured to store the database allocation attributes, which include: the number of basic data table entry pools, the table size and number of entries in each basic data table entry pool, the table size and number of entries in each extended table entry pool, and the hash table size. The basic data table entry pool, stored in the shared memory, is configured to store real data, which includes: node attributes, attributes of all node parameters, and values of all node parameters; Data table entries, including table header, key, attributes, and values; Hash table entries are stored in the basic entry pool, and each hash table entry points to a data table entry. The node table is configured to represent the relationships between nodes. The node table entries define the current node ID, parent node ID, number of child nodes, total number of nodes in the child node tree, node depth, and node name.
2. The database according to claim 1, characterized in that, The basic data table entry pool includes: The first data table entry pool is formed in the first shared memory and has a first data table entry length and a first data table entry number; The second data entry pool is formed in the second shared memory and has a second data entry length and a second data entry number. The third data table entry pool is formed in the third shared memory and has the third data table entry length and the third data table entry number; Wherein, the length of the first data table entry is less than the length of the second data table entry, and the length of the second data table entry is less than the length of the third data table entry; The number of entries in the first data table is greater than the number of entries in the second data table, and the number of entries in the second data table is greater than the number of entries in the third data table.
3. The database according to claim 2, characterized in that, If the length of valid data in the data entry is less than or equal to the length of the first data entry, a first free entry is obtained from the first data entry pool, and the data entry is stored in the first free entry. If the length of valid data in the data entry is greater than the length of the first data entry, and the length of valid data is less than or equal to the length of the second data entry, a second free entry is obtained from the second data entry pool, and the data entry is stored in the second free entry. If the length of valid data in the data entry is greater than the length of the second data entry, and the length of valid data is less than or equal to the length of the third data entry, a third free entry is obtained from the third data entry pool, and the data entry is stored in the third free entry.
4. The database according to claim 3, characterized in that, Also includes: Expand the data table item pool; If the length of valid data in the data table entry is greater than the length of the third data table entry, the extended data table entry is obtained based on the length of the valid data, and the data table entry is stored in the extended data table entry.
5. The database according to claim 1, characterized in that, The table header includes: The address information of the next entry, including the shared memory number and offset, is configured as an index for the data entry pool; The identification information of whether an entry is occupied is configured for the application and release of entries in the entry pool management system; The data table entry length is configured to identify the total length of the data table entry. Valid data length is configured to identify the length of the data used. The key length, attribute length, and value length represent the lengths of the three fields, respectively. The key is the complete node URL, which is configured for hash calculation and matching lookup after hash calculation; The attributes include: node attributes, attributes of all parameters to which the node belongs, and the values of the parameters to which the node belongs; The value includes the values of all parameters, which are stored as strings.
6. The database according to claim 1, characterized in that, If multiple keys have the same hash value after hash calculation, the data table entries corresponding to the multiple keys are associated with each other through the shared memory number and offset of the next entry; The hash table entry points to multiple data table entries corresponding to the key.
7. The database according to claim 1, characterized in that, The node table includes a first node table entry and a second node table entry, and the first node table entry and the second node table entry use contiguous memory; the first node table entry and the second node table entry are of the same size. The node entries are stored in the node table according to the following sorting method: The root node, the first child node of the root node, the first child node of the first child node, and so on, until all the child nodes of the first child node are stored; Then store the second child node of the root node, and so on, until all nodes are stored.
8. A data retrieval method, characterized in that, Applied to the database according to any one of claims 1-7, the method comprises: The hash value is obtained by performing a hash calculation based on the node URL; The hash table entry is located based on the hash value to determine the hash table entry that matches the hash value. Traverse the data table chain table pointed to by the hash table entry that matches the hash value, and compare the node URL with the URL of the data table entry in turn, until the node URL and the URL of the data table entry are completely consistent.
9. The data retrieval method according to claim 8, characterized in that, After the step of searching for a hash table entry based on a hash value to determine the hash table entry that matches the hash value, the method further includes: Lock the hash table entry that matches the hash value.
10. A data update method, characterized in that, Applied to the database according to any one of claims 1-7, the method comprises: The hash value is obtained by performing a hash calculation based on the node URL; The hash table entry is located based on the hash value to determine the hash table entry that matches the hash value. Traverse the data table chain table pointed to by the hash table entry that matches the hash value, and compare the node URL with the URL of the data table entry in turn, until the node URL and the URL of the data table entry are completely consistent; The data table entries in the shared memory are fully read into the process memory space and then decompressed. Modify the data in the unzipped table entries; Copy the modified table entries to shared memory, overwriting the original data entries in shared memory.