Memory structure for storing Json data and its data processing method

Abstract

The present invention provides a memory structure for storing JSON data and a data processing method thereof. The memory data structure is a layered structure, with each layer corresponding to a layer of the JSON structure. JSON objects are implemented in the memory data structure using a hash table with a concurrency control mechanism, and arrays in the JSON are implemented in the memory data structure using a doubly linked list with a concurrency control mechanism. Connections between layers are implemented using memory pointers, and values ​​are implemented using a structure. By employing a memory data structure comprising a hash table and a doubly linked list with a concurrency control mechanism, the present invention solves the problem of concurrent reading and writing of values ​​in a single JSON structure. This significantly improves the processing speed of JSON structured data on computers with multiple CPUs or multi-core CPUs.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a memory structure for storing JSON data and a data processing method thereof. Background Art

[0002] The Json data structure parsed by the existing public Json parser does not support concurrent access at the object level, and there is no index mechanism within the Json structure.

[0003] Patent document CN105787128A (application number: CN201610188809.0) discloses a method for recovering Java serialized file data, including the following steps: S1: Analyze and record data type and structure identifiers; S2: Define an intermediate structure to store the data type name, field name, and value of each node; S3: Obtain the top-level intermediate result and maintain a list of class definition IDs; S4: Expand the intermediate result and convert it into a JSON string; S5: Extract the class structure and generate a class template for in-memory data recovery; S6: Restore the complete serialized data to memory. However, this method of reading and writing a JSON structure only uses a single thread, which is relatively slow.

[0004] Therefore, it is necessary to provide a Json memory structure with an index structure that supports concurrency and can concurrently process multiple objects in a Json structure, thereby improving processing speed. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a memory structure for storing JSON data and a data processing method thereof.

[0006] According to the memory structure for storing Json data provided by the present invention, the memory data structure is a layered structure, each layer corresponds to a layer of the Json structure, the objects in the Json are implemented in the memory data structure using a hash table with a concurrency control mechanism, the arrays in the Json are implemented in the memory data structure using a doubly linked list with a concurrency control mechanism, the connection between layers is implemented using memory pointers, and the value is implemented using a structure;

[0007] The value structure includes the value type, value and read-write lock;

[0008] The hash table with concurrency control mechanism has a fixed size and supports at least read and write concurrency control;

[0009] A doubly linked list with a concurrency control mechanism supports at least two types of concurrency control: read and write;

[0010] The key-value pair structure includes a key, a pointer, and a read-write lock. When the key-value pairs in a Json object exceed the hash table size * load factor, a complete hash table is added, and the newly added hash table is appended to the end of the doubly linked list where the original hash table is located;

[0011] Different value types store values ​​in different ways:

[0012] String type: The value in the structure is a pointer pointing to the location where the string is stored;

[0013] Integer: If the number can be saved in the 8-byte 64-bit space, it is saved in the location of this structure value; if it cannot be saved in 8 bytes, it is converted to a large integer;

[0014] Large integer: The value in the structure is a pointer to the location where the large integer is stored;

[0015] Decimal type: If the number can be saved as a double-precision floating-point type within the 8-byte 64-bit space, it is saved in the location of this structure value; if it cannot be saved within the 8-byte space, it is converted to a long decimal type;

[0016] Long decimal type: The value in the structure is a pointer to the location where the long decimal type is stored;

[0017] Object: The value in the structure is a pointer to a doubly linked list. Each node of this doubly linked list is a hash table.

[0018] Array: The value in the structure is a pointer to a doubly linked list;

[0019] The root node is the value structure. In any Json layer, there is only one value structure memory data structure layer. The layer above the memory data structure of objects and arrays is the value structure.

[0020] The data processing method for concurrently reading and writing memory data structures provided by the present invention includes: reading and writing memory data in a recursive manner, judging the types of the current layer and the next layer in the recursion, waiting for the lock holder to release the lock if there is a lock conflict, and implementing the modification operation by first deleting and then creating a new one; if there is no lock on a structure, a read lock or a write lock can be added; if there is a read lock on a structure, a read lock can be added again, but a write lock cannot be added again; if there is a write lock on a structure, a read lock or a write lock cannot be added again; structure locks can only be added from the upper layer to the lower layer, and the unlocking order is the reverse order of the locking order.

[0021] Preferably, when adding a new key-value pair:

[0022] Input parameters:

[0023] a) Pointer to the Json memory data structure;

[0024] b) A pointer to a doubly linked list whose nodes are hash tables, where the key-value pairs will be stored;

[0025] c) Pointer to the key-value pair structure;

[0026] Steps:

[0027] a) Set a read lock on the pointer to the Json memory data structure;

[0028] b) Set a write lock on the key-value pair structure;

[0029] c) Set a write lock on the doubly linked list whose nodes are hash tables;

[0030] d) Set a write lock on the last hash table in the doubly linked list;

[0031] e) Check whether the key-value pairs in the last hash table of the doubly linked list exceed the hash table size * load factor;

[0032] f) If the hash table size * load factor is exceeded, a new hash table is created and appended to the end of the doubly linked list. A write lock is set in the new hash table (the new hash table is the last hash table in the doubly linked list) and the write lock of the original hash table is released.

[0033] g) Calculate the hash value of the key in the key-value pair and insert the key-value pair into the hash table according to the hash table insertion algorithm;

[0034] h) Release all read locks and write locks set during the release process.

[0035] Preferably, when adding a new value:

[0036] Input parameters:

[0037] a) Pointer to the Json memory data structure;

[0038] b) A pointer to a memory data structure node for the value;

[0039] c) The type of the previous memory data structure node;

[0040] d) pointer to the value structure;

[0041] Steps:

[0042] a) Set a read lock on the pointer to the Json memory data structure;

[0043] b) Set the previous memory data structure node to a write lock;

[0044] c) If the previous node is a doubly linked list whose nodes are value structures, add a node to the end of the doubly linked list and point it to the parameter d) value structure;

[0045] d) If the previous node is a key-value pair, modify the pointer of the key-value pair structure to point to the parameter d) value structure;

[0046] e) Release all read locks and write locks set during the release process.

[0047] Preferably, when adding an empty object:

[0048] Input parameters:

[0049] a) Pointer to the Json memory data structure;

[0050] b) A pointer to a memory data structure node on the object;

[0051] Return value: pointer to the doubly linked list of the newly added node as hash table;

[0052] Steps:

[0053] a) Set a read lock on the pointer to the Json memory data structure;

[0054] b) Set the previous memory data structure node to a write lock;

[0055] c) Create a new doubly linked list;

[0056] d) Create a new hash table and append it to the doubly linked list;

[0057] e) Release all read locks and write locks set during the release process.

[0058] Preferably, when adding an empty array:

[0059] Input parameters:

[0060] a) Pointer to the Json memory data structure;

[0061] b) A pointer to a memory data structure node on the array;

[0062] Return value: pointer to a doubly linked list with the newly added node as the value structure;

[0063] Steps:

[0064] a) Set a read lock on the pointer to the Json memory data structure;

[0065] b) Set the previous memory data structure node to a write lock;

[0066] c) Create a new doubly linked list;

[0067] d) Release all read locks and write locks set during the release process.

[0068] Preferably, when deleting a key-value pair:

[0069] Input parameters:

[0070] a) Pointer to the Json memory data structure;

[0071] b) A pointer to a node on the key-value pair structure;

[0072] c) Pointer to the key-value pair structure;

[0073] Steps:

[0074] a) Set a read lock on the pointer to the Json memory data structure;

[0075] b) Set a write lock on the doubly linked list whose nodes are hash tables;

[0076] c) Add a write lock to the key-value pair structure and the value structure under it. If there is content under the value structure, add write locks layer by layer from top to bottom according to the logic of traversing the tree until all contents under the key-value pair structure are added with write locks;

[0077] d) Use the key in the key-value pair to search for this key in all hash tables in the doubly linked list. If the key is found, delete the key from the corresponding hash table.

[0078] e) Release the write lock of the key-value pair structure and all structures under it;

[0079] f) Release the memory of the key-value pair structure and all structures under it;

[0080] g) Release all locks set by a) and b)

[0081] Preferably, when deleting a value, the premise is that the upper layer of the value is an array:

[0082] Input parameters:

[0083] a) Pointer to the Json memory data structure;

[0084] b) A pointer to the previous memory data structure node of the value, where the previous node is a doubly linked list of value structures;

[0085] c) pointer to the value structure;

[0086] Steps:

[0087] a) Set a read lock on the pointer to the Json memory data structure;

[0088] b) Set the previous memory data structure node to a write lock;

[0089] c) Add a write lock to the value structure. If there is content under the value structure, add write locks layer by layer from top to bottom according to the logic of traversing the tree until all content under the key-value pair structure is written with a write lock;

[0090] d) Delete the node corresponding to the value structure from the doubly linked list;

[0091] e) Release the write lock of the value structure and all structures under it;

[0092] f) Release the memory of the value structure and all structures under it;

[0093] g) Release all locks set by a) and b)

[0094] Preferably, when performing a key-based value reading operation:

[0095] Input parameters:

[0096] a) Pointer to the Json memory data structure;

[0097] b) a pointer to a doubly linked list whose nodes are hash tables, the upper nodes of the key-value pairs;

[0098] c) key;

[0099] Output parameter: pointer to key-value pair structure;

[0100] Steps:

[0101] a) Set a read lock on the pointer to the Json memory data structure;

[0102] b) A pointer to a doubly linked list whose nodes are hash tables, with the upper node of the key-value pair set to a read lock;

[0103] c) Set read locks in all hash tables in the doubly linked list;

[0104] d) Search for the key in all hash tables of the doubly linked list and return a null pointer if the key-value pair is not found; if the key-value pair is found, set a read lock on the key-value pair;

[0105] e) Add a read lock to the value structure of the key-value pair. If there is content under the value structure, add read locks layer by layer from top to bottom according to the logic of traversing the tree until all content under the key-value pair structure is read locked;

[0106] f) Create a new key-value pair structure and copy the found key-value pair structure and all its underlying contents to the new key-value pair structure;

[0107] g) Release the read lock of the value structure of the key-value pair in the Json memory data structure and all structures under it;

[0108] h) Release the read locks applied in a), b), and c).

[0109] Preferably, when performing a copy operation:

[0110] Input parameter: pointer to Json memory data structure;

[0111] Output parameter: pointer to the memory data structure of the newly copied Json;

[0112] Steps:

[0113] a) Set a write lock on the pointer to the Json memory data structure;

[0114] b) Create a new Json structure and copy the data of the original structure to the new structure;

[0115] c) Release the lock.

[0116] Compared with the prior art, the present invention has the following beneficial effects:

[0117] The present invention solves the problem that the values ​​in a single Json structure cannot be read and written concurrently by adopting a memory data structure of a hash table and a bidirectional linked list with a concurrency control mechanism. On a computer with multiple CPUs or multi-core CPUs, the processing speed of Json structure data can be significantly improved. The memory data structure that can store Json data is formed by combining the two structures. Using this memory data structure, fast access and parallel reading and writing of the child nodes of a single Json can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0119] Figure 1 A diagram of the memory structure for storing Json data. DETAILED DESCRIPTION

[0120] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0121] Example:

[0122] The present invention provides a memory structure for storing Json data and a data processing method thereof, which are specifically as follows:

[0123] 1. Memory Data Structure

[0124] Json is a serialized object or array that can be nested multiple times.

[0125] A Json object can have multiple members, each of which is a key-value pair, where the key is a string.

[0126] The new in-memory data structure is layered. Each layer corresponds to a layer of the JSON structure. Objects in JSON are implemented in the in-memory data structure using a hash table with concurrency control. Arrays in JSON are implemented in the in-memory data structure using a doubly linked list with concurrency control. Connections between layers are implemented using memory pointers. Values ​​are implemented using a structure. The value structure corresponds to the value, the hash table with concurrency control corresponds to the object, and the doubly linked list with concurrency control corresponds to the array.

[0127] They are described as follows:

[0128] (1) Value structure, containing 3 members:

[0129] a. Value Type

[0130] b. Value

[0131] c. Read-write lock

[0132] Different value types store values ​​in different ways:

[0133] a) String type: The value in the structure is a pointer pointing to the location where the string is stored.

[0134] b) Integer: If the number can be stored within 8 bytes (64 bits), it is stored in the structure value. If it cannot be stored within 8 bytes, it is converted to a large integer.

[0135] c) Large integer: The value in the structure is a pointer to the location where the large integer is stored.

[0136] d) Decimal type: If the number can be stored in a double-precision floating-point type (double) within the 8-byte (64-bit) space, it is stored in the location of this structure value. If it cannot be stored within the 8-byte space, it is converted to a long decimal type.

[0137] e) Long decimal type: The value in the structure is a pointer pointing to the location where the long decimal type is stored.

[0138] f) Object: The value in the structure is a pointer to a doubly linked list. Each node of this doubly linked list is a hash table.

[0139] g) Array: The value in the structure is a pointer pointing to a doubly linked list.

[0140] The root node must be a value structure. Within any JSON layer, there must be one (and only one) value structure memory data structure layer. The layer above the memory data structure of objects and arrays must also be a value structure.

[0141] (2) Hash table with concurrency control mechanism:

[0142] The hash table described in this invention does not rely on a specific hash function; in theory, any hash function that supports a string as a parameter can be used. The hash table described in this invention does not rely on a specific method for resolving hash conflicts; in theory, any common method for resolving hash conflicts can be used.

[0143] The hash table described in this invention has the following points to emphasize compared to a standard hash table:

[0144] a) The size of the hash table described in the present invention is fixed and cannot be dynamically increased or decreased.

[0145] b) The hash table described in the present invention has a concurrency control mechanism on the entire table, and this concurrency control mechanism supports at least two types of concurrency control: read and write. The common implementation is a read-write lock.

[0146] c) In particular, if a linked list is used to resolve hash conflicts, a concurrency control mechanism must be implemented in the linked list and at each node within it, supporting at least read and write concurrency control. A common implementation is a read-write lock.

[0147] The address of this hash table comes from the key in the key-value pair calculated by the hash function. The value stored in the hash table address is a pointer to a key-value pair structure.

[0148] The key-value pair structure has 3 members:

[0149] a.Key (must be a string according to the Json specification)

[0150] b. Pointer to the value structure mentioned in (1)

[0151] c. Read-write lock

[0152] When the number of key-value pairs in a JSON object exceeds the hash table size * load factor, a new hash table is added, appended to the end of the doubly linked list containing the original hash table. In theory, it is also possible to add new hash tables using nested hashes, which is more time-consuming but requires more memory.

[0153] (3) Doubly linked list with concurrency control mechanism:

[0154] Doubly linked lists have a concurrency control mechanism that supports at least read and write concurrency control. The most common implementation is a read-write lock.

[0155] For example:

[0156] A Json:

[0157] {"a":1,"b":[{"c":5},{"d":"abcd"},{"e":1234.5678}]}

[0158] The outer layer (referred to as the first layer in this article) is a Json object, and the innermost layer (referred to as the nth layer in this article) is the value.

[0159] Then, the memory data structure is as follows Figure 1 :

[0160] Json layer 1: value structure: type → object storage data structure layer 1 value → pointer

[0161] |

[0162] The second layer of the doubly linked list memory data structure whose nodes are hash tables

[0163] |

[0164] Hash tables (possibly multiple)

[0165] |

[0166] Json layer 2: key-value pair structure (multiple) memory data structure layer 3

[0167] |

[0168] Value structure memory data structure level 4

[0169] |

[0170] The value of "b" is an array |

[0171] Json layer 3: nodes are value structures of the double-linked list memory data structure layer 5

[0172] |

[0173] Value structure memory data structure level 6

[0174] |

[0175] Array values ​​are objects |

[0176] Json layer 4: double linked list memory data structure layer 7 where the nodes are hash tables

[0177] |

[0178] Hash tables (possibly multiple)

[0179] |

[0180] Json layer 5: key-value pair structure (may have multiple) memory data structure layer 8

[0181] |

[0182] Value structure memory data structure layer 9

[0183] 2. Steps for concurrently reading and writing memory data structures

[0184] Reading and writing this memory structure has the following characteristics:

[0185] a. Can be read and written recursively (or in a logically equivalent way);

[0186] b. In one step of the recursion, it is necessary to determine the types of the current layer and the next layer;

[0187] c. All lock conflicts wait for the lock holder to release the lock;

[0188] d. To modify the operation, first delete and then create a new one.

[0189] About the use of locks:

[0190] a. If there is no lock on a structure, then a read lock or a write lock can be added;

[0191] b. If a read lock is already on a structure, then a read lock can be added, but a write lock cannot be added (and lock escalation is not allowed);

[0192] c. If a write lock is on a structure, a read lock or a write lock cannot be added;

[0193] d. Structs can only be locked from the top layer to the bottom layer. That is, if there are two structures that need to be locked, and these two structures are adjacent upper and lower nodes, the upper node must be locked first before the lower node can be locked;

[0194] e. The unlocking order is the reverse order of the locking order.

[0195] Describe them according to the operation type:

[0196] 1. Add a new key-value pair:

[0197] Input parameters:

[0198] a) Pointer to the Json memory data structure;

[0199] b) A pointer to a doubly linked list whose nodes are hash tables, where the key-value pairs will be stored;

[0200] c) Pointer to the key-value pair structure;

[0201] Steps:

[0202] a) Set a read lock on the pointer to the Json memory data structure;

[0203] b) Set a write lock on the key-value pair structure;

[0204] c) Set a write lock on the doubly linked list whose nodes are hash tables;

[0205] d) Set a write lock on the last hash table in the doubly linked list;

[0206] e) Check whether the key-value pairs in the last hash table of the doubly linked list exceed the hash table size * load factor;

[0207] f) If the hash table size * load factor is exceeded, a new hash table is created and appended to the end of the doubly linked list. A write lock is set in the new hash table (the new hash table is the last hash table in the doubly linked list) and the write lock of the original hash table is released.

[0208] g) Calculate the hash value of the key in the key-value pair and insert the key-value pair into the hash table according to the hash table insertion algorithm;

[0209] h) Release all read locks and write locks set during the release process.

[0210] 2. Add a new value:

[0211] Input parameters:

[0212] a) Pointer to the Json memory data structure;

[0213] b) A pointer to a memory data structure node for the value;

[0214] It may be a doubly linked list (Json array) with nodes as value structures, or it may be a key-value pair;

[0215] c) The type of the previous memory data structure node, see b)

[0216] d) pointer to the value structure;

[0217] Steps:

[0218] a) Set a read lock on the pointer to the Json memory data structure;

[0219] b) The previous memory data structure node is set to write lock;

[0220] c) If the previous node is a doubly linked list (JSON array) whose nodes are value structures, add a node to the end of the doubly linked list and point it to the value structure of parameter d);

[0221] d) If the previous node is a key-value pair, modify member b of the key-value pair structure to point to parameter d) value structure;

[0222] e) Release all read locks and write locks set during the release process.

[0223] 3. Add an empty object:

[0224] Input parameters:

[0225] a) Pointer to the Json memory data structure;

[0226] b) A pointer to a memory data structure node on the object;

[0227] The previous node must be a value structure;

[0228] Return value: pointer to the doubly linked list of the newly added node as hash table;

[0229] Steps:

[0230] a) Set a read lock on the pointer to the Json memory data structure;

[0231] b) The previous memory data structure node is set to write lock;

[0232] c) Create a new doubly linked list;

[0233] d) Create a new hash table and append (pushback) it to the doubly linked list;

[0234] e) Release all read locks and write locks set during the release process.

[0235] 4. Add an empty array:

[0236] Input parameters:

[0237] a) Pointer to the Json memory data structure;

[0238] b) A pointer to a memory data structure node on the array;

[0239] The previous node must be a value structure;

[0240] Return value: pointer to a doubly linked list with the newly added node as the value structure;

[0241] Steps:

[0242] a) Set a read lock on the pointer to the Json memory data structure;

[0243] b) The previous memory data structure node is set to write lock;

[0244] c) Create a new doubly linked list;

[0245] d) Release all read locks and write locks set during the release process.

[0246] 5. Delete a key-value pair:

[0247] Input parameters:

[0248] a) Pointer to the Json memory data structure;

[0249] b) A pointer to a node on the key-value pair structure (this node refers to the doubly linked list that stores the hash table);

[0250] c) Pointer to the key-value pair structure;

[0251] Steps:

[0252] a) Set a read lock on the pointer to the Json memory data structure;

[0253] b) Set a write lock on the doubly linked list whose nodes are hash tables;

[0254] c) Add a write lock to the key-value pair structure and the value structure under it. If there is content under the value structure, add write locks layer by layer from top to bottom according to the logic of traversing the tree until all content under the key-value pair structure is written.

[0255] d) Use the key in the key-value pair to search for this key in all hash tables in the doubly linked list. This search process can be concurrent. If the key is found, delete the key from the corresponding hash table;

[0256] e) Release the write lock of the key-value pair structure and all structures under it;

[0257] f) Release the memory of the key-value pair structure and all structures under it;

[0258] g) Release all locks set by a) and b)

[0259] 6. Delete a value:

[0260] This operation is only supported when the upper layer of the value is an array. If the upper layer of the value is a key-value pair, the steps described in 5 should be used to delete the entire key-value pair.

[0261] Input parameters:

[0262] a) Pointer to the Json memory data structure;

[0263] b) A pointer to a memory data structure node for the value;

[0264] The previous node is a doubly linked list of value structures (Json array);

[0265] c) pointer to the value structure;

[0266] Steps:

[0267] a) Set a read lock on the pointer to the Json memory data structure;

[0268] b) The previous memory data structure node is set to write lock;

[0269] c) Add a write lock to the value structure. If there is content under the value structure, add write locks layer by layer from top to bottom according to the logic of traversing the tree until all content under the key-value pair structure is written.

[0270] d) Delete the node corresponding to the value structure from the doubly linked list;

[0271] e) Release the write lock of the value structure and all structures under it;

[0272] f) Release the memory of the value structure and all structures under it;

[0273] g) Release all locks set by a) and b)

[0274] 7. Read the value by key:

[0275] Input parameters:

[0276] a) Pointer to the Json memory data structure;

[0277] b) a pointer to a doubly linked list whose nodes are hash tables, the upper nodes of the key-value pairs;

[0278] c) key;

[0279] Output parameters:

[0280] a) Pointer to the key-value pair structure;

[0281] Steps:

[0282] a) Set a read lock on the pointer to the Json memory data structure;

[0283] b) A pointer to a doubly linked list whose nodes are hash tables, with the upper node of the key-value pair set to a read lock;

[0284] c) Set read locks in all hash tables in the doubly linked list;

[0285] d) Search for the key in all hash tables of the doubly linked list (this step can be parallelized);

[0286] e) If not found, return a null pointer;

[0287] f) If a key-value pair is found, set a read lock on the key-value pair;

[0288] g) Add a read lock to the value structure of the key-value pair. If there is content under the value structure, add read locks layer by layer from top to bottom according to the logic of traversing the tree until all content under the key-value pair structure is read locked;

[0289] h) Create a new key-value pair structure (no locking required), and copy the found key-value pair structure and all its underlying contents to the new key-value pair structure;

[0290] This process can be referred to the tree replication algorithm, which will not be described here;

[0291] i) Release the read lock of the value structure of the key-value pair in the Json memory data structure and all structures under it;

[0292] j) Release the read locks imposed in a), b), and c).

[0293] 8. Copy:

[0294] Input parameters:

[0295] a) Pointer to the Json memory data structure;

[0296] Output parameters:

[0297] a) Pointer to the newly copied Json memory data structure;

[0298] Steps:

[0299] a) Set a write lock on the pointer to the Json memory data structure;

[0300] b) Create a new Json structure and copy the data of the original structure to the new structure;

[0301] This process can be referred to the tree replication algorithm, which will not be described here;

[0302] c) Release the lock.

[0303] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0304] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A memory structure for storing Json data, characterized in that: The memory data structure is a layered structure, each layer corresponds to a layer of the Json structure. The objects in the Json are implemented in the memory data structure using a hash table with a concurrency control mechanism. The arrays in the Json are implemented in the memory data structure using a doubly linked list with a concurrency control mechanism. The connection between layers is implemented using memory pointers, and the value is implemented using a structure. The value structure includes value type, value and read-write lock; The hash table with concurrency control mechanism has a fixed size and supports at least read and write concurrency control; A doubly linked list with a concurrency control mechanism supports at least two types of concurrency control: read and write; The key-value pair structure includes a key, a pointer, and a read-write lock. When the key-value pairs in a Json object exceed the hash table size * load factor, a complete hash table is added, and the newly added hash table is appended to the end of the doubly linked list where the original hash table is located; Different value types store values ​​in different ways: String type: The value in the structure is a pointer pointing to the location where the string is stored; Integer: If the value can be saved in the 8-byte 64-bit space, it is saved in the location of this structure value; if it cannot be saved in the 8-byte space, it is converted to a large integer; Large integer: The value in the structure is a pointer to the location where the large integer is stored; Decimal type: If the value can be saved as a double-precision floating-point type within the 8-byte 64-bit space, it is saved in the location of this structure value; if it cannot be saved within the 8-byte space, it is converted to a long decimal type; Long decimal type: The value in the structure is a pointer to the location where the long decimal type is stored; Object: The value in the structure is a pointer to a doubly linked list. Each node of this doubly linked list is a hash table. Array: The value in the structure is a pointer to a doubly linked list; The root node is the value structure. In any Json layer, there is only one value structure memory data structure layer. The layer above the memory data structure of objects and arrays is the value structure.

2. A data processing method for concurrently reading and writing memory data structures, characterized in that: The memory structure for storing Json data as described in claim 1 includes: reading and writing memory data in a recursive manner, judging the types of the current layer and the next layer in the recursion, waiting for the holder to release the lock if there is a lock conflict, and using the method of deleting first and then creating a new one during the modification operation; if there is no lock on a structure, a read lock or a write lock can be added; if there is a read lock on a structure, a read lock can be added again, but a write lock cannot be added again; if there is a write lock on a structure, a read lock or a write lock cannot be added again; structure locking can only be added from the upper layer to the lower layer, and the unlocking order is the reverse order of the locking order.

3. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When adding a new key-value pair: Input parameters: a) Pointer to the Json memory data structure; b) A pointer to a doubly linked list whose nodes are hash tables, where the key-value pairs will be stored; c) Pointer to the key-value pair structure; Steps: a) Set a read lock on the pointer to the Json memory data structure; b) Set a write lock on the key-value pair structure; c) Set a write lock on the doubly linked list whose nodes are hash tables; d) Set a write lock on the last hash table in the doubly linked list; e) Check whether the key-value pairs in the last hash table of the doubly linked list exceed the hash table size * load factor; f) If the hash table size is exceeded * load factor, a new hash table is created and appended to the end of the doubly linked list. A write lock is set in the new hash table, and the write lock of the original hash table is released. At this time, the new hash table is the last hash table in the doubly linked list. g) Calculate the hash value of the key in the key-value pair and insert the key-value pair into the hash table according to the hash table insertion algorithm; h) Release all read locks and write locks set during the release process.

4. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When adding a value: Input parameters: a) Pointer to the Json memory data structure; b) A pointer to a memory data structure node for the value; c) The type of the previous memory data structure node; d) pointer to the value structure; Steps: a) Set a read lock on the pointer to the Json memory data structure; b) Set the previous memory data structure node to a write lock; c) If the previous node is a doubly linked list whose nodes are value structures, add a node to the end of the doubly linked list and point it to the parameter d) value structure; d) If the previous node is a key-value pair, modify the pointer of the key-value pair structure to point to the parameter d) value structure; e) Release all read locks and write locks set during the release process.

5. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When adding an empty object: Input parameters: a) Pointer to the Json memory data structure; b) A pointer to a memory data structure node on the object; Return value: pointer to the doubly linked list of the newly added node as hash table; Steps: a) Set a read lock on the pointer to the Json memory data structure; b) Set the previous memory data structure node to a write lock; c) Create a new doubly linked list; d) Create a new hash table and append it to the doubly linked list; e) Release all read locks and write locks set during the release process.

6. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When adding an empty array: Input parameters: a) Pointer to the Json memory data structure; b) A pointer to a memory data structure node on the array; Return value: pointer to a doubly linked list with the newly added node as the value structure; Steps: a) Set a read lock on the pointer to the Json memory data structure; b) Set the previous memory data structure node to a write lock; c) Create a new doubly linked list; d) Release all read locks and write locks set during the release process.

7. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When deleting a key-value pair: Input parameters: a) Pointer to the Json memory data structure; b) A pointer to a node on the key-value pair structure; c) Pointer to the key-value pair structure; Steps: a) Set a read lock on the pointer to the Json memory data structure; b) Set a write lock on the doubly linked list whose nodes are hash tables; c) Add a write lock to the key-value pair structure and the value structure under it. If there is content under the value structure, add write locks layer by layer from top to bottom according to the logic of traversing the tree until all contents under the key-value pair structure are added with write locks; d) Use the key in the key-value pair to search for this key in all hash tables in the doubly linked list. If the key is found, delete the key from the corresponding hash table. e) Release the write lock of the key-value pair structure and all structures under it; f) Release the memory of the key-value pair structure and all structures under it; g) Release all locks set by a) and b) 8. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When deleting a value, the premise is that the upper layer of the value is an array: Input parameters: a) Pointer to the Json memory data structure; b) A pointer to the previous memory data structure node of the value, where the previous node is a doubly linked list of value structures; c) pointer to the value structure; Steps: a) Set a read lock on the pointer to the Json memory data structure; b) Set the previous memory data structure node to a write lock; c) Add a write lock to the value structure. If there is content under the value structure, add write locks layer by layer from top to bottom according to the logic of traversing the tree until all content under the key-value pair structure is written with a write lock; d) Delete the node corresponding to the value structure from the doubly linked list; e) Release the write lock of the value structure and all structures under it; f) Release the memory of the value structure and all structures under it; g) Release all locks set by a) and b) 9. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When reading values ​​by key: Input parameters: a) Pointer to the Json memory data structure; b) a pointer to a doubly linked list whose nodes are hash tables, the upper nodes of the key-value pairs; c) key; Output parameter: pointer to key-value pair structure; Steps: a) Set a read lock on the pointer to the Json memory data structure; b) A pointer to a doubly linked list whose nodes are hash tables, with the upper node of the key-value pair set to a read lock; c) Set read locks in all hash tables in the doubly linked list; d) Search for the key in all hash tables of the doubly linked list and return a null pointer if the key is not found; If a key-value pair is found, a read lock is set on the key-value pair; e) Add a read lock to the value structure of the key-value pair. If there is content under the value structure, add read locks layer by layer from top to bottom according to the logic of traversing the tree until all content under the key-value pair structure is read locked; f) Create a new key-value pair structure and copy the found key-value pair structure and all its underlying contents to the new key-value pair structure; g) Release the read lock of the value structure of the key-value pair in the Json memory data structure and all structures under it; h) Release the read locks applied in a), b), and c).

10. The data processing method for concurrently reading and writing memory data structures according to claim 2, characterized in that: When performing a copy operation: Input parameter: pointer to Json memory data structure; Output parameter: pointer to the memory data structure of the newly copied Json; Steps: a) Set a write lock on the pointer to the Json memory data structure; b) Create a new Json structure and copy the data of the original structure to the new structure; c) Release the lock.

Images