Method and apparatus for processing json text

Abstract

The present disclosure provides a method and device for processing a Json text. The method comprises: parsing the Json text to obtain a set of key-value pairs in the Json text, the set of key-value pairs comprising a plurality of keys and a plurality of values corresponding to the plurality of keys one by one; in the process of parsing the Json text, adding the plurality of values to a Json binary field in turn according to the parsing order of the plurality of values; after parsing the Json text, sorting the plurality of keys to obtain positioning data, the positioning data being used for positioning the positions of the plurality of values in the Json binary field according to the sorted keys; and adding the plurality of keys and the positioning data to the Json binary field.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and specifically to a method and apparatus for processing JSON text. Background Technology

[0002] Data exchange formats enable data exchange between different computer programs or between different programming languages. JavaScript Object Notation (JSON) is a lightweight data exchange format that uses text to represent the JavaScript language.

[0003] In some cases, such as when storing JSON text, it is necessary to convert the JSON text into a JSON binary field. Related technologies for converting JSON text into a JSON binary field suffer from high memory consumption. Summary of the Invention

[0004] In view of this, the present disclosure aims to provide a method and apparatus for processing JSON text to solve the problem of large memory consumption during JSON text conversion.

[0005] In a first aspect, a method for processing JSON text is provided, comprising: parsing the JSON text to obtain a set of key-value pairs in the JSON text, the set of key-value pairs including multiple keys and multiple values ​​corresponding one-to-one with the multiple keys; during the parsing of the JSON text, adding the multiple values ​​sequentially to a JSON binary field according to the parsing order of the multiple values; after parsing the JSON text, sorting the multiple keys to obtain positioning data, the positioning data being used to locate the positions of the multiple values ​​in the JSON binary field according to the sorted keys; and adding the multiple keys and the positioning data to the JSON binary field.

[0006] Optionally, the method further includes storing the plurality of keys in memory using a radix tree structure.

[0007] Optionally, the JSON binary field includes a first field, and after adding the plurality of keys and the location data to the JSON binary field, the method further includes: determining the position of the plurality of keys in the JSON binary field; and updating the first field in place such that the first field records the position of the plurality of keys in the JSON binary field.

[0008] Optionally, the location data includes a first array and a second array, wherein the first array is used to record the offset of the sorted keys, and the second array is used to record the offset of the value corresponding to the sorted keys.

[0009] Optionally, adding the plurality of keys to the JSON binary field includes: adding the plurality of keys to the JSON binary field in the parsing order of the plurality of keys.

[0010] Optionally, parsing the Josn text further includes: verifying the validity of the Josn text.

[0011] Secondly, an apparatus for processing JSON text is provided, comprising: a parsing unit configured to parse the JSON text to obtain a set of key-value pairs in the JSON text, the set of key-value pairs including multiple keys and multiple values ​​corresponding one-to-one with the multiple keys; a first adding unit configured to add the multiple values ​​sequentially to a JSON binary field in the parsing order of the multiple values ​​during the parsing of the JSON text; a sorting unit configured to sort the multiple keys after parsing the JSON text to obtain positioning data, the positioning data being used to locate the positions of the multiple values ​​in the JSON binary field according to the sorted keys; and a second adding unit configured to add the multiple keys and the positioning data to the JSON binary field.

[0012] Optionally, the device further includes a storage unit configured to store the plurality of keys in memory using a radix tree structure.

[0013] Optionally, the JSON binary field includes a first field, and the apparatus further includes: a determining unit configured to determine the position of the plurality of keys in the JSON binary field after the plurality of keys and the positioning data are added to the JSON binary field; and an updating unit configured to update the first field in place, such that the first field records the position of the plurality of keys in the JSON binary field.

[0014] Optionally, the location data includes a first array and a second array, wherein the first array is used to record the offset of the sorted keys, and the second array is used to record the offset of the value corresponding to the sorted keys.

[0015] Optionally, the adding unit is further configured to add the plurality of keys to the JSON binary field in the parsing order of the plurality of keys.

[0016] Optionally, the parsing unit is further configured to: verify the validity of the JSON text.

[0017] Thirdly, an apparatus for processing JSON text is provided, including a memory and a processor, wherein the memory stores executable code and the processor is configured to execute the executable code to implement the method as described in the first aspect.

[0018] Fourthly, a computer program product is provided, the computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in the first aspect.

[0019] Fifthly, a computer-readable medium is provided that stores program code, which, when run on a computer, causes the computer to perform the method described in the first aspect.

[0020] The present disclosure provides a solution for processing JSON text that encodes values ​​sequentially according to the parsing order of the JSON text, without waiting to parse all values ​​before encoding. In other words, the encoding of multiple values ​​in the JSON text is independent, and multiple parsed values ​​can be directly encoded without being cached in memory. Only the keys are cached in memory, avoiding the parsed values ​​from occupying memory space. This solves the problem of large memory consumption during the conversion of JSON text to JSON binary fields. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of a JSON binary field provided in an embodiment of this disclosure.

[0022] Figure 2 This is a schematic diagram of a process for processing JSON text provided in an embodiment of this disclosure.

[0023] Figure 3 This is a schematic diagram of another JSON binary field provided in an embodiment of this disclosure.

[0024] Figure 4 This is a flowchart illustrating a method for processing JSON text provided in an embodiment of this disclosure.

[0025] Figure 5 A method provided for embodiments of this disclosure and Figure 4 The diagram shows the JSON binary field corresponding to the method shown.

[0026] Figure 6 This is a schematic diagram of yet another JSON binary field provided in an embodiment of this disclosure.

[0027] Figure 7 This is a schematic diagram of yet another JSON binary field provided in an embodiment of this disclosure.

[0028] Figure 8 This is a schematic diagram of another process for processing JSON text provided in an embodiment of this disclosure.

[0029] Figure 9 This is a schematic diagram of the structure of an apparatus for processing JSON text provided in an embodiment of the present disclosure.

[0030] Figure 10 This is a schematic diagram of another apparatus for processing JSON text provided in an embodiment of this disclosure. Detailed Implementation

[0031] The technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments.

[0032] Data exchange formats describe data in a specific format, enabling modern computer languages ​​to support the data. Specific data formats make it possible to exchange data between different programming languages. Data exchange formats can organize data into a tree structure to describe the relationships between data.

[0033] Data exchange formats can include, for example, JavaScript object notation (JSON). JSON records the JavaScript language in text format and is a lightweight data exchange format.

[0034] JSON text can include objects and arrays. A JSON object can be a collection of key-value pairs, and an array can be a collection of values. A JSON object can begin with a left bracket "{" and end with a right bracket "}", with each key followed by a colon, and each key-value pair separated by a comma. For example, JSON text could be {"b":3,"a":"xyz"}, which contains two key-value pairs: b=3 and a=xyz. A JSON array can begin with a left square bracket "[" and end with a right square bracket "]", with each value separated by a comma. For example, JSON text could be {"b":["3","a","xyz"]}, which contains three key-value pairs: b=3, b=a, and b=xyz. The data type of the values ​​in the key-value pairs in JSON text can be one or more of the following: numbers, strings, objects, arrays, and booleans.

[0035] In JSON objects, the values ​​can be other arrays or other objects, making JSON a nested data type—a composite data type. JSON itself has a nested structure, which organizes it into a tree-like data structure. When the values ​​in JSON text are basic data types, they can be leaf nodes in the tree structure; when the values ​​are objects, they can be intermediate nodes. JSON has its own built-in data types, making it easy to write, read, and parse without requiring a special structure. Because JSON is a composite data type, JSON data is typically very large.

[0036] As mentioned in the text, JSON is a tree-like composite data structure, and accessing JSON is generally based on paths. However, JSON does not carry metadata describing this tree structure. When accessing key-value pairs in JSON, the JSON needs to be loaded into memory, parsed, and the tree structure data constructed before data access is performed. As an example, when accessing data in JSON text, the JSON string needs to be parsed into a JavaScript object. This can be achieved using a JSON parser, such as Gson or Jackson. Alternatively, a parsing function can be used, such as the `JSON parse` function.

[0037] During parsing, the JSON text needs to be loaded into memory and parsed according to JSON data rules to obtain a set of key-value pairs. When querying data within the JSON text, it's also necessary to load the JSON text into memory, expand it into a tree structure, and then perform data queries based on this tree structure. This tree structure could be, for example, a JSON Tree; that is, the JSON data in the JSON text is parsed into a JSON Tree in memory, and data queries are performed based on this JSON Tree.

[0038] As can be seen, accessing unparsed JSON text is extremely inefficient, as every access to data within the JSON text requires traversing and parsing the entire JSON. For example, when querying data from unparsed JSON text, the entire JSON text must be parsed before the query can be performed. Furthermore, repeated access to the JSON text necessitates repeated parsing before each access. To make JSON text easier to access and avoid repeated parsing for each access, the parsed JSON text can be serialized into a binary field, such as JSON binary encoding. Serialization, also known as encoding, involves encoding the key-value pairs in the parsed JSON text into a JSON binary field.

[0039] The relevant technologies provide two methods for processing JSON text, which are described below. Figures 1 to 3 This paper introduces two methods for processing JSON text provided in related technologies. In these technologies, JSON text can be '{"b":3,"a":"xyz"}', which includes two key-value pairs, b=3 and a=xyz, where b and a are keys, and 3 and xyz are values ​​that correspond one-to-one with the keys.

[0040] Figure 1 The diagram shown is a schematic of a JSON binary field. Figure 1 The binary fields shown include the JSON object's length, value type, key, value, and valuelength. From Figure 1 The JSON binary fields shown can be seen Figure 1 It is encoded according to the data order of the JSON text, that is, Figure 1 The encoding follows the parsing order of the data in the JSON text. The key-value pair b=3 is parsed first, followed by the key-value pair a=xyz. Figure 1 The key-value pairs are encoded independently according to the parsing order of the key-value pairs in the JSON text. Figure 1 The encoding method shown does not consider sorting the key-value pairs in the JSON. When querying data, it is necessary to traverse all elements in the entire encoded data, resulting in poor query performance.

[0041] To improve the queryability of JSON text, metadata describing the tree-like organization can be added to the JSON fields. During data retrieval, queries can be performed directly based on the JSON binary fields, avoiding multiple parsing steps. For efficient querying, the parsed key-value pairs are sorted during JSON text encoding. For example, the parsed keys can be sorted according to string order, and the encoding is based on this sorted data.

[0042] Figure 2 This diagram illustrates a workflow for processing JSON text. Figure 2 As shown, after obtaining the JSON text, it can be processed on the server side. First, the JSON text is parsed. The purpose of parsing is to convert the data in the JSON text into JavaScript objects, that is, to extract the key-value pairs from the JSON text. Based on the parsed key-value pairs, a tree-structured JSON tree is constructed in memory. After the complete tree structure is built, it is sorted, and the encoding is completed according to the sorted data.

[0043] Figure 3 To and Figure 2 A diagram showing the corresponding JSON binary field. Figure 3 The JSON binary fields shown can include JSON text type (type), number of key-value pairs (count), JSON object length (object length), key data length (key length), key offset (key offset), key, value, and value offset (value offset). It can be seen that, according to... Figure 3 When encoding JSON text using binary fields, it's necessary to sort the key data after all key-value pairs are parsed from the JSON text, and then encode the data based on the sorted keys. The purpose of this is to speed up subsequent data searches by allowing a binary search based on the keys. For example... Figure 3 As shown, the order of value offsets is kept consistent with the order of key offsets during encoding. This way, when performing a data query, after finding the key, the value offset can be directly located based on the array index, and the value data can be found through the value offset.

[0044] It is evident that the methods for processing JSON text provided by related technologies either cannot sort the key-value pairs in the JSON text, resulting in low data retrieval performance, such as... Figure 1 The JSON binary field can either be sorted for efficient searching by key-value pairs, but this requires parsing all key-value pairs from the JSON text, sorting them, and then encoding them, such as... Figure 2 The processing flow is as described above. As mentioned earlier, JSON is a tree-structured data with inherent nested relationships. Parsing all key-value pairs requires building this tree structure in memory, which consumes a significant amount of memory in the short term. Secondly, all key-value pairs in the JSON text need to be parsed, sorted, and then encoded. Before parsing is complete, the parsed key-value pairs are temporarily cached in memory, leading to memory consumption as data cannot be promptly written to disk. Furthermore, before parsing is complete, the corresponding JSON binary fields cannot be fully encoded and written to disk, resulting in both the JSON tree and the JSON binary fields existing in memory simultaneously. Especially in database systems, significant tenant memory consumption can affect access to other data types within the tenant. When multiple connections within a tenant access data concurrently, such as when multiple connections store different JSON texts simultaneously, it can consume a large amount of memory, degrading database performance.

[0045] Based on this, this disclosure provides a method and apparatus for processing JSON text to solve the problem of large memory consumption when converting JSON text into JSON binary fields.

[0046] Figure 4 This is a flowchart illustrating a method for processing JSON text according to an embodiment of this disclosure. The JSON text processing method provided in this embodiment is applicable to relational databases and NoSQL database products that support JSON.

[0047] In step S410, the JSON text is parsed to obtain the set of key-value pairs in the JSON text.

[0048] JSON text can be JavaScript data recorded in text format. In some embodiments, the JSON data in JSON text can be string data recorded in text. JSON text includes a set of key-value pairs, where each key contains multiple keys and multiple values ​​corresponding to those keys. As an example, JSON text could be '{"b":3,"a":"xyz"}', which includes two key-value pairs, b = 3 and a = xyz, where b and a are keys, and 3 and xyz are values ​​corresponding to those keys.

[0049] Parsing JSON text can be understood as converting JSON data into data that can be stored or accessed according to the JSON format. Specifically, by parsing JSON text, you can obtain the set of key-value pairs in the JSON text. Taking the data to be processed as '{"b":3,"a":"xyz"}' as an example, by parsing the JSON text, you can obtain the key-value pairs b=3 and a=xyz.

[0050] In some embodiments, a JSON parser can be used to parse JSON text; for example, a JSON parser could be Gson, Jackson, or similar. In other embodiments, a parsing function can be used to parse JSON text; for example, a JSON parse function.

[0051] In step S420, during the parsing of the JSON text, multiple values ​​are added to the JSON binary field in the order of parsing.

[0052] During the parsing of JSON text, the values ​​can be encoded in the order they are parsed, and the encoded data is then sequentially added to the JSON binary fields. In some embodiments, encoding the data can also be referred to as serializing the data.

[0053] It should be understood that the method for processing JSON text provided in this disclosure can encode values ​​immediately after parsing. That is, values ​​can be encoded synchronously in the parsing order. In some cases, values ​​can also be encoded after a certain number of parsed values ​​have been obtained, for example, after five values ​​have been parsed. In other words, the encoding of value data and the parsing of JSON text are performed synchronously in this disclosure.

[0054] Taking the JSON text '{"b":3,"a":"xyz"}' as an example, the first parsed key-value pair is b=3, so the value 3 in this key-value pair is retrieved and encoded. The next parsed key-value pair is a=xyz, so the value xyz in this key-value pair is retrieved and encoded. In other words, the values ​​in the key-value pairs can be encoded sequentially according to the order in which the key-value pairs are parsed from the JSON text.

[0055] Encoding multiple values ​​yields encoded data for those values. This encoded data can then be appended to a JSON binary field in the order the values ​​are parsed. The encoded data can include the data type (valuetype), the value itself, and its length. Data types can include integers, numbers, strings, etc.

[0056] For example, a JSON binary field could be 0x03 52 32 0x02 3xyz 0x01 3. Here, 0x03 represents the type of JSON data in the JSON text, which could be an object, for example. 52 represents the length of the JSON data. 32 represents location data, used to record the position of multiple keys in the JSON binary field. This location data can be left blank or filled with any data, and will be updated after parsing is complete. 0x02 3xyz represents the encoded data of the value xyz, where 0x02 represents the data type of xyz, which could be a string, 3 represents the length of xyz, and xyz is the value data. 0x01 3 represents the encoded data of the value 3, where 0x01 represents the data type of 3, which could be an integer, and 3 is the value data; the length of value 3 is omitted here. As can be seen, the order of multiple values ​​in the JSON binary field is consistent with the order of multiple values ​​in the JSON text, that is, the order of multiple values ​​in the JSON binary is consistent with the parsing order of the values.

[0057] In step S430, after parsing the JSON text, the multiple keys are sorted to obtain the location data.

[0058] After parsing the JSON text, a complete set of key-value pairs is obtained. Since the values ​​in the key-value pairs have been encoded in step S420, in step S430, multiple keys in the JSON text are sorted to obtain the location data.

[0059] The location data may include, for example, a second array that records the positions of multiple values ​​corresponding to sorted keys within a JSON binary field. In other words, the order of the location data matches the order of the sorted keys. The location data is used to locate the positions of multiple values ​​within the JSON binary field according to the sorted keys. The location data may be, for example, an offset; that is, the offset is used to locate the position of a value within the JSON binary field, and the location data can pinpoint the position of the value corresponding one-to-one with the sorted keys within the binary field.

[0060] Continuing with an example using a JSON binary field, such as 0x03 52 32 0x02 3xyz 0x01 3, let's illustrate this. In this JSON binary field, the offset of the value xyz within the binary field could be, for example, 9, meaning offsetting 9 bytes from the first byte to obtain the type of the inverse xyz value, and then reading it forward according to the encoding rules to get the value. Similarly, the offset of the value 3 within the binary field could be, for example, 17, meaning offsetting 17 bytes from the first byte to obtain the type of the value 3, and then reading it forward according to the encoding rules to get the value. In some embodiments, the offset of the value xyz within the binary field could be, for example, 14, meaning offsetting 14 bytes from the first byte to directly obtain the value; and the offset of the value 3 within the binary field could be, for example, 21, meaning offsetting 21 bytes from the first byte to directly obtain the value.

[0061] When sorting multiple keys, you can sort them by key size, such as sorting them in ascending order by key value; or you can sort them by character order, such as sorting them by the characters A through Z.

[0062] After sorting multiple keys, we obtain the location data, which will be used as offsets for explanation. As mentioned earlier, encoding the value gives the offset of the value in the JSON binary field. For example, the offset of the value xyz can be 9, and the offset of the value 3 can be 14. After sorting the keys character by character, key a comes before key b. Therefore, the value xyz corresponding to key a is located before the value 3 corresponding to key b in binary. Thus, the offset of the value corresponding to key a comes before the offset of the value corresponding to key b. That is, the location data of the value is an array [9, 17]. By using the location array, we can find the value corresponding to the sorted key in the JSON field.

[0063] In step S440, multiple keys and location data are added to the JSON binary field.

[0064] After obtaining the location data, multiple keys and location data can be added to the JSON binary field that includes the value data to obtain the corresponding JSON binary field of the JSON text.

[0065] You can first encode multiple keys and then add the encoded data of these keys to the binary field mentioned above. The encoded data of the keys may include, for example, the key length, the key value, and the key count.

[0066] When adding multiple keys to a JSON binary field, you can add them sequentially based on the sorted keys, or sequentially based on the parsing order of the keys.

[0067] Continuing with an example using a JSON binary field, such as 0x03 52 32 0x02 3xyz 0x01 3, let's illustrate this further. The sorted keys are in the order [a, b]. Multiple keys can be added to the binary code in sequence. The key encoding data could be, for example, 1a 1b 2, where 1a represents the length and key data of key a, 1b represents the length and key data of key b, and 2 represents the number of keys in the JSON text. The encoded data of these multiple keys can be added to the end of the binary field, resulting in the JSON binary field 0x03 52 32 0x02 3xyz 0x01 3 1a 1b 2. Then, the value positioning data [9, 17] from the previous example is added to the end of the binary field, resulting in the JSON binary field 0x03 52 32 0x02 3xyz 0x01 3 1a 1b 2 9 17. When encoding multiple keys in the order of key data, the location data of multiple keys can be obtained. The location data of multiple keys can be, for example, the offset of multiple keys. The offset of multiple keys can be added to the above JSON binary field to complete the serialization of JSON text.

[0068] The offsets of multiple keys can indicate the positions of multiple keys in a JSON binary field. The offset for key 'a' can be 26, meaning it's offset 26 bytes from the beginning of the binary field to obtain the length of key 'a', and then the key data is read from there. The offset for key 'b' can be 31, meaning it's offset 31 bytes from the beginning of the binary field to obtain the length of key 'b', and then the key data is read from there. The offset for key 'a' can also be 27, meaning it's offset 27 bytes from the beginning of the binary field to directly obtain the key data; the offset for key 'b' can be 32, meaning it's offset 32 ​​bytes from the beginning of the binary field to directly obtain the key data.

[0069] For example, the location data for multiple keys can be appended after the location data for the values ​​mentioned above, or it can be appended after the encoded data for the multiple keys mentioned above. For example, the binary field of the JSON text could be 0x03 52 32 0x023xyz 0x013 1a 1b 2 26 31 9 17.

[0070] As can be seen, the method for processing JSON text provided in this embodiment can encode values ​​sequentially according to the JSON text parsing order, without waiting to parse all values ​​before encoding. In other words, the encoding of multiple values ​​in the JSON text is independent, and the parsed multiple values ​​can be directly encoded without being cached in memory. Only the keys are cached in memory, avoiding the parsed values ​​from occupying memory space. This solves the problem of large memory consumption during the conversion of JSON text to JSON binary fields.

[0071] In some embodiments, the Lob append interface can be called to write the JSON binary fields to disk sequentially according to the parsing order, or the append function can be used to append multiple keys and location data to the end of the above binary fields to obtain the binary fields of the JSON text.

[0072] Since the keys in JSON text are typically string data with limited length (e.g., a maximum key size of 64KB in MySQL), they can be converted into a radix tree structure for storage. This reduces memory usage and frees up memory space. The radix tree could be a RadixTree, meaning the keys are stored in memory using a radix tree structure to further reduce memory consumption.

[0073] As mentioned earlier, the JSON binary field may contain a first field, which can be used to record the positions of multiple keys within the JSON binary field. In some embodiments, this first field also records the position of the number of multiple keys within the JSON binary field. Based on the first field, the number of multiple keys in the binary field can be quickly located. For example, the number of multiple keys can be located before the location data of the first key in the binary field. This allows for quick location of the number of keys in the JSON binary field based on the first field, and subsequent reading of the required key location data, thus quickly finding the key data. This location data can be, for example, a Sort-key offset.

[0074] The following example uses a JSON binary field of 0x03 52 32 0x02 3xyz 0x01 3 1a 1b 2 9 17 as an example. The first field, for instance, can be 32, meaning that offsetting 32 bytes from the beginning of the JSON binary field reveals the number of keys in the JSON text. In this example, offsetting 32 bytes from the beginning of the JSON binary field indicates two keys in the JSON text; reading further reveals the key data, enabling efficient data retrieval.

[0075] It can be seen that the first field can only be obtained after the entire JSON text has been parsed and multiple keys and location data have been added to the JSON binary field. In other words, the data in the first field can only be obtained after the JSON is parsed, the location data for multiple keys and values ​​is encoded, and added to the JSON binary field containing multiple values. To ensure the JSON binary field is written to disk promptly, arbitrary data can be added to the first field first, allowing the JSON binary field to be written to disk via appending. After adding multiple keys and location data to the JSON binary field, the positions of the multiple keys in the JSON binary field are determined, and the data in the first field is updated in-place.

[0076] For example, the in-place update command can be used to refill the data in the first field. The in-place update command is used to update fields whose data space usage has not changed before and after the change. That is, the new value directly overwrites the old value without moving other data or re-allocating space for writing, which is a very efficient update scenario.

[0077] In step S430, after parsing the JSON text, the multiple keys are sorted to obtain location data. This location data may include a first array and a second array. The first array can be the offsets of multiple keys, used to record the offsets of the sorted keys. The second array can be the offsets of multiple values, used to record the offsets of multiple values ​​that correspond one-to-one with the order of the sorted keys. Thus, when querying data, after finding a key, the offset of the value can be directly found in the second array based on the key's index in the first array, quickly locating the value and improving query efficiency.

[0078] When adding multiple keys to a JSON binary field, it is not necessary to add them according to the order of the keys, which further improves the efficiency of encoding.

[0079] After obtaining the first and second arrays, in step S440, multiple keys and the first and second arrays are added to the JSON binary field to obtain the binary field corresponding to the JSON text. This allows multiple keys to be added to the JSON binary field sequentially according to the parsing order, further improving the processing efficiency of the JSON text.

[0080] Continuing with an example using a JSON binary field, such as 0x03 52 32 0x02 3xyz 0x01 3, let's illustrate this further. The sorted key order is [a, b]. Based on the sorted key order, we obtain a second array [9, 17] that matches the key order. Following the parsing order and sorting order of the keys, we calculate the first array [31, 26] of the offsets of the sorted keys. That is, in the JSON binary field, according to the sorted key order, the offset of key a is 31, the offset of key b is 26, the offset of the value xyz corresponding to key a is 9, and the offset of the value 3 corresponding to key b is 17. Encode multiple keys, and append the multiple keys, the first array, and the second array to the end of the binary field to obtain the binary field corresponding to the JSON text: 0x03 52 32 0x02 3xyz 0x01 3 1b 1a 2 31 26 9 17. Combined with... Figure 5 This section introduces JSON binary fields. Figure 5 The binary field shown is 0x03 52 32 0x023xyz 0x01 3 1b 1a 231 26 9 17 as mentioned above.

[0081] Figure 5 The fields shown in the binary data have been described previously and will not be repeated here. For example... Figure 5 As shown, multiple values ​​are appended sequentially according to the parsing order of the JSON text, and multiple keys are also appended sequentially according to the parsing order of the JSON text. The offset order of the multiple keys is consistent with the sorted key order; that is, the offset order in the first array is consistent with the sorted key order. The first offset in the first array indicates the position of the first key after sorting, and the second offset in the first array indicates the position of the second key after sorting. Figure 5 Offset 36 indicates the position of key 'a', and offset 26 indicates the position of key 'b'. The order of the offsets for multiple values ​​is consistent with the sorted key order; that is, the order of the offsets in the second array is consistent with the sorted key order. The first offset in the second array indicates the position of the value corresponding to the first sorted key, and the second offset in the second array indicates the position of the value corresponding to the second sorted key. Figure 5 In the text, offset 9 indicates the position of the value xyz corresponding to key a, and offset 17 indicates the position of 3 corresponding to key b.

[0082] In some embodiments, the validity of the JSON text can also be verified when parsing the JSON text.

[0083] The following is combined with Figure 6The illustration of a JSON binary field provides an exemplary description of a method for processing JSON text provided in this disclosure. Figure 6 The JSON text shown is '{"d":4,"h":"x","a":"zxc","p":"5"}'.

[0084] When parsing JSON text, the values ​​are first encoded according to the parsing order. The encoded data is then added sequentially to a binary field in that order. For example, multiple encoded values ​​like 0x01 4 0x02 x 0x023zxc 0x01 5 are added to a binary field. This binary field can also include fields for the JSON data type, data length, and value position, as described earlier. Figure 6 Not shown in the figure. In the encoded data of multiple values ​​0x01 4 0x02x 0x02 3zxc 0x01 5, 0x01 4 indicates the data type and value of value 4, 0x02 x indicates the data type and value of value x, 0x02 3zxc indicates the data type, data length and value of value zxc, and 0x01 5 indicates the data type and value of value 5.

[0085] After parsing the JSON text, the multiple values ​​in the JSON text have been encoded, and only the keys need to be cached in memory. If we represent the offsets of the multiple values ​​in the binary field of the JSON as an array, before sorting the multiple keys, the value offset array can be, for example, [1, 6, 11, 19]. In this array, 1 is the offset of the value 4, 6 is the offset of the value x, 11 is the offset of the value zxc, and 19 is the offset of the value 5. Sort the multiple keys according to the character order Z to A, and the sorted multiple keys are [p, h, d, a]. Based on the sorted multiple keys, we obtain the positioning data [19, 6, 1, 11]. Here, the first data in the positioning data is the offset of the value corresponding to the first key in the sorted multiple keys in the binary field, that is, 19 indicates the position of the value 5 corresponding to the key p in the binary field, and the second data in the positioning data is the offset of the value corresponding to the second key in the sorted multiple keys in the binary field, that is, 6 indicates the position of the value x corresponding to the key h in the binary field. This is because key 'h' comes after key 'p', therefore, the offset of the value corresponding to key 'h' comes after the offset of the value corresponding to key 'p'. In other words, the order of the data in the value offset array is consistent with the order of the sorted keys. Adding multiple keys and the above-mentioned location data to the JSON binary field yields the following binary field: 0x01 4 0x02 x 0x02 3zxc 5 0x01 dh ap 196 1 11.

[0086] Next, add the location data for multiple keys to this binary field. The location data for multiple keys can be an array of key offsets, for example, [27, 25, 24, 26]. The first data in this array indicates the offset of the first key in the sorted array, i.e., 27 indicates the offset of key p in the binary field; the second data indicates the offset of the second key in the sorted array, i.e., 25 indicates the offset of key h in the binary field. This is because key h follows key p, therefore, the key offset of key h follows the key offset of key p. In other words, the order of the data in the key offset array is consistent with the order of the sorted keys. Adding the array of key offsets before the array of value offsets yields the final binary field: 0x01 4 0x02 x 0x02 3zxc 0x01 5d hap 27 25 24 26 19 61 11.

[0087] The following example, using the JSON binary field, demonstrates how to achieve fast key-value pair lookup. Suppose we need to find the key-value pair p = 5, using binary search. First, find the middle data in the key offset array, for example, the second offset in the array. The second offset is 25, and based on this offset, we find the key h. Comparing h with p, h is less than p. Therefore, we continue searching the key offset to the left, finding the first offset 27 in the key offset array, and based on this offset, we find the key p. When searching for the value corresponding to key p, because the order of the data in the key offset array is the same as the order in the value offset array, we can directly find the value based on the data's index. The offset of key p is the first data in the key offset array, so the offset of the value corresponding to key p is also the first data in the value offset array. Thus, we find the first value offset 19, and based on this value offset, we find the value 5.

[0088] The following is combined with Figure 7 Another JSON binary field is shown to exemplarily illustrate a method for processing JSON text provided in embodiments of this disclosure. Figure 7 The JSON text shown is '{"zt":"pq","wer":25,"gt":"bf"}'.

[0089] When parsing JSON text, the multiple values ​​are first encoded into the JSON binary string in the order they are parsed. For example, the encoded data for multiple values ​​could be 0x02 pq 0x01 25 0x02 bf, showing that the values ​​are encoded sequentially according to the parsing order. After parsing, the multiple values ​​in the JSON text are encoded, and only the keys need to be cached in memory. Before sorting the keys, the value offset array could be, for example, [1, 7, 13], where 1 is the offset of value pq, 7 is the offset of value 25, and 13 is the offset of value bf. Sorting the keys according to the character order A through Z results in [gt, wer, zt]. Based on the order of the keys, the location array [13, 7, 1] is obtained. The first data in the positioning array indicates the position of the value corresponding to the first key among the sorted keys; that is, 13 indicates the position of the value bf of key gt in the binary field. The second data in the positioning array indicates the position of the value corresponding to the second key among the sorted keys; that is, 7 indicates the position of the value 25 of key wer in the binary field. The third data in the positioning array indicates the position of the value corresponding to the third key among the sorted keys; that is, 1 indicates the position of the value pq of key zt in the binary field. It can be seen that the order of the data in the positioning array is consistent with the order of the sorted keys. For example, in the positioning array above, key zt is after key wer, therefore, the offset 1 of key zt's value is after the offset 7 of key wer's value; key gt is before key wer, therefore, the offset 13 of key gt's value is before the offset 7 of key wer's value. Adding multiple keys and the above positioning data to the JSON binary field results in the following binary field: 0x02 pq 0x01 25 0x02 bf zt wer gt 137 1.

[0090] When adding the offsets of multiple keys to a JSON binary field, the offsets of the keys can be, for example, an array of key offsets [24, 21, 19]. In this array of key offsets, the key zt is after the key wer, so the offset 19 of key zt is after the offset 21 of key wer; the key gt is before the key wer, so the offset 24 of key gt is before the offset 21 of key wer. Adding the array of key offsets before the array of value offsets results in the final binary field 0x02pq 0x01 25 0x02 bf zt wer gt 13 7 1 24 21 19.

[0091] When performing a key-value pair lookup, assuming we need to find the key-value pair zt = pq, we use binary search. First, we find the middle data in the key offset array, which is the key offset 21 in the middle of the array. Based on this key offset, we find the key wer. Comparing the key wer with the key zt, key zt is greater than key wer, so we continue searching to the right. We find the key offset 19, and based on this key offset, we find the key zt. When finding the value of key zt, because the order of the data in the key offset array is consistent with the order of the data in the value offset array, we can directly find the offset with the same index in the value offset array based on the index of key zt's offset in the key offset array. Key zt's key offset 19 is the third data in the key offset array, therefore, the third data in the value offset array is the offset of the value corresponding to this key. That is, 1 is the offset of the value corresponding to key zt, and based on this offset, we can find the value pq.

[0092] The following is combined with Figure 8 The method for processing JSON text provided in this disclosure is described by way of example. Figure 8 A flowchart illustrating the process of processing JSON text is shown.

[0093] like Figure 8 As shown, the obtained JSON text can be processed. JSON text can be text-formatted JSON data, such as string data, and can include one or more key-value pairs. The JSON text is parsed, for example, using the `JSON parse` function, to obtain multiple key-value pairs, each consisting of multiple keys and their corresponding values. During parsing, the values ​​are encoded according to the parsing order and added to the JSON binary field. Following the tree structure rules of JSON, a key-tree structure is constructed in memory, such as a sorted key-tree. Multiple keys are stored in memory using a radix tree structure to compress the data and further free up memory space. Finally, the multiple keys, key-location data, and value-location data are added to the JSON binary field, resulting in the JSON binary field corresponding to the JSON text.

[0094] like Figure 8As shown, the Lob's append interface can be called to write the JSON binary fields to disk sequentially according to the parsing order. Alternatively, the append function can be used to append multiple keys and location data to the end of the aforementioned binary fields to obtain the binary fields of the JSON text. In this way, the encoded value data can be written to disk / hard drive at any time, further reducing memory usage. In some embodiments, images, large binary objects, and composite data such as JSON / GIS data, which are too large to be fully stored in memory, can also be written to disk on demand using Lob.

[0095] like Figure 8 As shown, in-place update functions can be used to populate fields, for example... Figure 5 The example shows the first field and the data length field in the JSON binary field. This allows values ​​to be stored on disk immediately, without waiting for data parsing or key-value pair encoding. Once processing is complete, the in-place update function can be used to update the data length field and the key-value pair position field.

[0096] Therefore, the method for processing JSON text provided in this embodiment does not require storing values ​​in memory during the JSON text serialization process. In other words, it does not require reserving memory of unknown size to store the parsed key-value pairs, thus eliminating the uncertainty of memory space occupation during JSON text serialization and solving the problem of heavy memory dependence during JSON text encoding.

[0097] The above text combined Figures 1 to 8 The present disclosure describes in detail the method embodiments, which are then combined with the following. Figures 9 to 10 The present disclosure provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0098] Figure 9 This is a schematic diagram of the structure of an apparatus for processing JSON text provided in an embodiment of this disclosure. Figure 9 The data serialization device 900 shown includes a parsing unit 910, a first adding unit 920, a sorting unit 930, and a second adding unit 940. Each unit will be described in exemplary form below.

[0099] Parsing unit 910 is configured to parse JSON text to obtain a set of key-value pairs in the JSON text, the set of key-value pairs including multiple keys and multiple values ​​corresponding to the multiple keys;

[0100] The first adding unit 920 is configured to add the plurality of values ​​to the JSON binary field in the order of parsing the JSON text during the parsing process.

[0101] The sorting unit 930 is configured to sort the multiple keys after parsing the Json text to obtain positioning data, which is used to locate the positions of the multiple values ​​in the Json binary field according to the sorted keys.

[0102] The second adding unit 940 is configured to add the plurality of keys and the location data to the JSON binary field.

[0103] Optionally, the device further includes a storage unit 950 configured to store the plurality of keys in memory using a radix tree structure.

[0104] Optionally, the JSON binary field includes a first field, and the apparatus further includes: a determining unit 960 configured to determine the position of the plurality of keys in the JSON binary field after the plurality of keys and the positioning data are added to the JSON binary field; and an updating unit 970 configured to update the first field in place, such that the first field records the position of the plurality of keys in the JSON binary field.

[0105] Optionally, the location data includes a first array and a second array, wherein the first array is used to record the offset of the sorted keys, and the second array is used to record the offset of the value corresponding to the sorted keys.

[0106] Optionally, the adding unit is further configured to add the plurality of keys to the JSON binary field in the parsing order of the plurality of keys.

[0107] Optionally, the parsing unit is further configured to: verify the validity of the JSON text.

[0108] Figure 10 This is a schematic diagram of another apparatus for processing JSON text provided in an embodiment of this disclosure. Figure 10 The illustrated apparatus 1000 may include a memory 1010 and a processor 1020. The memory 1010 may be used to store executable code. The processor 1020 may be used to execute the executable code stored in the memory 1010 to implement the steps in the various methods described above. In some embodiments, the apparatus 1020 may further include a network interface 1030 through which the processor 1020 exchanges data with external devices.

[0109] It should be understood that in the embodiments of this disclosure, "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0110] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0111] It should be understood that in the various embodiments of this disclosure, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0112] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0115] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can read or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs, DVDs) or semiconductor media (e.g., solid-state disks, SSDs), etc.

[0116] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for processing JSON text, comprising: The JSON text is parsed to obtain a set of key-value pairs in the JSON text, the set of key-value pairs including multiple keys and multiple values ​​that correspond one-to-one with the multiple keys; During the parsing of the JSON text, the multiple values ​​are added to the JSON binary field in the order of parsing, wherein the parsing of the JSON text and the addition of the multiple values ​​are performed simultaneously; After parsing the JSON text to obtain all key-value pairs in the JSON text, and after all the multiple values ​​are added to the JSON binary field, the multiple keys are sorted to obtain positioning data. The positioning data is used to locate the position of the multiple values ​​in the JSON binary field according to the sorted keys. The positioning data includes a first array and a second array. The first array is used to record the offset of the sorted keys, and the second array is used to record the offset of the value corresponding to the sorted keys. Add the multiple keys and the location data to the JSON binary field; The encoded data of the plurality of values ​​can be written to the disk or hard drive before the JSON text parsing is completed and / or before the encoding of the plurality of key data is completed.

2. The method according to claim 1, further comprising: The multiple keys are stored in memory using a radix tree structure.

3. The method according to claim 1, wherein the JSON binary field includes a first field, and after adding the plurality of keys and the location data to the JSON binary field, the method further includes: Determine the positions of the plurality of keys in the JSON binary field; The first field is updated in place so that the first field records the position of the multiple keys in the JSON binary field.

4. The method according to claim 1, wherein adding the plurality of keys to the JSON binary field comprises: The multiple keys are added to the JSON binary field in the order they are parsed.

5. The method according to claim 1, wherein parsing the JSON text further comprises: Verify the validity of the JSON text.

6. An apparatus for processing JSON text, comprising: The parsing unit is configured to parse JSON text to obtain a set of key-value pairs in the JSON text, the set of key-value pairs including multiple keys and multiple values ​​corresponding to the multiple keys; The first adding unit is configured to add the plurality of values ​​to the Json binary field in the order of parsing the Json text, wherein the parsing of the Json text and the addition of the plurality of values ​​are performed synchronously. The sorting unit is configured to sort the keys after parsing the JSON text to obtain all key-value pairs in the JSON text and after all the multiple values ​​are added to the JSON binary field, to obtain positioning data. The positioning data is used to locate the position of the multiple values ​​in the JSON binary field according to the sorted keys. The positioning data includes a first array and a second array. The first array is used to record the offset of the sorted keys, and the second array is used to record the offset of the value corresponding to the sorted keys. The second adding unit is configured to add the plurality of keys and the location data to the JSON binary field; The encoded data of the plurality of values ​​can be written to the disk or hard drive before the JSON text parsing is completed and / or before the encoding of the plurality of key data is completed.

7. The apparatus according to claim 6, further comprising: The storage unit is configured to store the plurality of keys in memory using a radix tree structure.

8. The apparatus of claim 6, wherein the JSON binary field comprises a first field, and the apparatus further comprises: The determining unit is configured to determine the position of the plurality of keys in the JSON binary field after the plurality of keys and the positioning data are added to the JSON binary field; The update unit is configured to update the first field in place, such that the first field records the positions of the plurality of keys in the Json binary field.

9. The apparatus according to claim 6, wherein the adding unit is further configured to: The multiple keys are added to the JSON binary field in the order they are parsed.

10. The apparatus according to claim 6, wherein the parsing unit is further configured to: Verify the validity of the JSON text.

11. An apparatus for processing JSON text, comprising a memory and a processor, the memory storing executable code, the processor being configured to execute the executable code to implement the method of any one of claims 1-5.

Images