Data processing method, device, equipment, storage medium and program product

Abstract

The present application provides a data processing method, device, electronic device, computer-readable storage medium and computer program product; the method includes: serializing first data in text format to obtain second data in binary format, wherein the second data in binary format includes an ordered key lookup table; obtaining a query request and extracting a target field from the query request; determining a value query result matching the target field based on the ordered key lookup table, and reading a target storage address corresponding to the value query result to obtain a value reading result; wherein the target storage address is obtained by performing pointer conversion processing on the second data in binary format. Through the present application, data reading and query efficiency can be improved.

Description

Technical Field

[0001] The present application relates to big data technology, and in particular to a data processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] JSON (JavaScript Object Notation) is a lightweight data exchange format. It uses a text format that is completely independent of the programming language to store and represent data. The simplicity and clear hierarchical structure make JSON an ideal data exchange language that is easy for people to read and write, and easy for machines to parse and generate, which can effectively improve network transmission efficiency.

[0003] Related technologies convert text JSON into binary format, thereby achieving faster traversal speed and smaller occupied size. It is mainly used as data storage in databases and network transmission format. However, for network products that require faster parsing speed, smaller transmission volume and lower memory consumption, related technologies cannot meet the corresponding needs. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve data reading and parsing efficiency.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data processing method, including:

[0007] Serializing the first data in text format to obtain second data in binary format, wherein the second data in binary format includes an ordered key lookup table;

[0008] Obtaining a query request, and extracting a target field from the query request;

[0009] Determine a value query result matching the target field based on the ordered key lookup table, and perform a read process on a target storage address corresponding to the value query result to obtain a value read result;

[0010] The target storage address is obtained by performing pointer conversion processing on the second data in binary format.

[0011] The present application provides a data processing device, including:

[0012] A serialization module, used for serializing the first data in text format to obtain second data in binary format, wherein the second data in binary format includes an ordered key lookup table;

[0013] A query module, used to obtain a query request and extract a target field from the query request;

[0014] The query module is also used to determine a value query result that matches the target field based on the ordered key lookup table, and read the target storage address corresponding to the value query result to obtain a value reading result; wherein the target storage address is obtained by performing pointer conversion processing on the second data in binary format.

[0015] In the above scheme, the second data includes a leading bit and subsequent bits, and the serialization module is also used to: when the data type of the first data is a first basic type, perform binary encoding processing on the first basic type to obtain the leading bit of the first data; wherein, the first basic type includes at least one of the following: single-precision floating point, double-precision floating point, unsigned integer, signed integer, and the length of the leading bit is eight bits; perform binary encoding processing on the first data to obtain subsequent bits of the first data; wherein, when the first basic type is a single-precision floating point, the length of the subsequent bits is four bytes; when the first basic type is a double-precision floating point, the length of the subsequent bits is eight bytes; when the first basic type is an unsigned integer, the length of the subsequent bits is the number of bits of the unsigned integer; when the first basic type is a signed integer, the length of the subsequent bits is the number of bits of the signed integer.

[0016] In the above scheme, the second data includes a leading bit, and the serialization module is also used to: when the data type of the first data is a second basic type, perform binary encoding on the second basic type to obtain the leading bit of the first data, wherein the second basic type includes at least one of the following: Boolean type, empty object type, and set digital type.

[0017] In the above scheme, the second data includes a leading bit and subsequent bits, and the serialization module is also used to: when the data type of the first data is a third basic type, perform binary encoding on the third basic type to obtain the leading bit of the first data; wherein the third basic type includes a string type, and the length of the leading bit is three bits; perform binary encoding on the string offset of the first data to obtain the string offset in binary format; and determine the subsequent bits of the first data according to the string offset in binary format.

[0018] In the above scheme, the serialization module is also used to: when the string corresponding to the first data is not queried in the string pool, determine that the string offset is zero, and combine the string offset in binary format and the binary encoding of the string value of the first data into subsequent bits of the first data; when the string corresponding to the first data is queried in the string pool, determine the string offset in binary format as the subsequent bits of the first data.

[0019] In the above scheme, the second data includes a guide bit and a subsequent bit, and the serialization module is also used to: when the data type of the first data is the fifth basic type, binary encode the fifth basic type to obtain the guide bit of the first data; wherein the fifth basic type includes an object type, and the length of the guide bit is one bit; binary encode the object data amount of the first data to obtain the object data amount in binary format; binary encode the string pool offset of the first data to obtain the string pool offset in binary format; binary encode the key lookup table of the first data to obtain the key lookup table in binary format; binary encode the value lookup table of the first data to obtain the value lookup table in binary format; combine the binary format object data amount, the binary format string pool offset, the binary format key lookup table, the binary format value lookup table and the binary encoding of each data value of the first data into the subsequent bits of the first data.

[0020] In the above scheme, the serialization module is also used to: obtain the key identifier of each of the object data in the first data; sort the multiple key identifiers in ascending order, and obtain the identifier difference between each adjacent key identifier; obtain the maximum value from the key identifier sorted in the first place and at least one of the identifier differences; encode the number of bits of the maximum value as the table type bits of the key lookup table, and encode each of the identifier differences and the key identifier sorted in the first place as the key identifier bits of the key lookup table; wherein the length of the key identifier bits conforms to the number of bits corresponding to the table type bits of the key lookup table.

[0021] In the above scheme, the serialization module is also used to: obtain the data value of each object data in the first data; encode the number of bits of the maximum data value into the table type bits in the binary format of the value lookup table, and encode the data value of each object data into the value bits in the binary format of the value lookup table; wherein the length of the value bits in the binary format corresponding to each data value conforms to the number of bits corresponding to the table type bits of the value lookup table.

[0022] In the above scheme, the data type of the string offset is a variable-length integer, and the serialization module is also used to: obtain the binary code of the string offset; parse the binary code to obtain at least one parsing result; perform the following processing for each parsing result according to the position order of at least one parsing result: when the parsing result does not include the last bit of the binary code of the string offset, use zero as the control bit corresponding to the parsing result, and use the parsing result as the storage bit; when the parsing result includes the last bit of the binary code of the string offset, use one as the control bit corresponding to the parsing result, and use the parsing result as the storage bit; combine the control bit and storage bit corresponding to the same parsing result into bytes corresponding to the parsing result, and combine the bytes of each parsing result into the string offset in binary format.

[0023] In the above scheme, the second data includes a leading bit and subsequent bits, and the serialization module is also used to: when the data type of the first data is a fourth basic type, perform binary encoding on the fourth basic type to obtain the leading bit of the first data; wherein the fourth basic type includes an array type, and the length of the leading bit is two bits; perform binary encoding on the array capacity of the first data to obtain the array capacity in binary format; perform binary encoding on the array lookup table of the first data to obtain the array lookup table in binary format; and combine the array capacity in binary format, the array lookup table in binary format, and the binary encoding of the array value of the first data into the subsequent bits of the first data.

[0024] In the above scheme, the serialization module is also used to: obtain the array offset of each of the array values ​​in the first data; encode the number of bits of the maximum array offset into the table type bits in the binary format of the array lookup table, and encode the array offset of each of the array values ​​into the offset bits in the binary format of the array lookup table; wherein the length of the offset bits in the binary format corresponding to each of the array values ​​conforms to the number of bits corresponding to the table type bits in the binary format.

[0025] In the above scheme, the binary format string pool of the first data includes a guide bit and a subsequent bit. When the first data in text format is serialized to obtain the second data in binary format, the serialization module is further used to: binary encode the string pool type to obtain the guide bit corresponding to the string pool type; obtain a skip offset for skipping the string pool of the first data, and binary encode the skip offset to obtain the skip offset in binary format; binary encode the key string pool of the first data to obtain a key string pool in binary format; binary encode the value string pool of the first data to obtain a value string pool in binary format; and combine the skip offset in binary format, the key string pool in binary format, and the value string pool in binary format to form a subsequent bit corresponding to the string pool.

[0026] In the above scheme, the serialization module is also used to: obtain multiple key character strings in the first data; perform binary encoding on the number of the multiple key character strings to obtain the number code of the key character string pool; perform binary encoding on the key pool lookup table of the first data to obtain a key pool lookup table in binary format; perform binary encoding on each of the key character strings to obtain the binary code of each of the key character strings; wherein the binary code of the key character string includes the binary code of the number of bits of the key character string and the key character string in binary format; the number code of the key character string pool, the key pool lookup table in binary format, and the binary code of the key character string form the key character string pool in binary format.

[0027] In the above scheme, the serialization module is also used to: obtain the offset of each of the key strings; encode the number of bits of the maximum offset as the table type bits in the binary format of the key pool lookup table, and encode the offset of each of the key strings as the key offset bits in the binary format of the key pool lookup table; wherein the length of the key offset bits conforms to the number of bits corresponding to the table type bits in the binary format of the key pool lookup table.

[0028] In the above scheme, the serialization module is also used to: obtain multiple value strings in the first data; when there is no key string identical to the value string and the length of the value string is greater than a length threshold, determine the value string as a target value string; obtain the length of each target value string, and perform the following processing on each target value string: perform binary encoding processing on the number of bits of the target value string to obtain the binary encoding of the number of bits of the target value string, and perform binary encoding processing on the target value string to obtain a target value string in binary format; combine the binary encoding of the number of bits of the target value string and the target value string in binary format to form a binary encoding of the target value string; and concatenate the binary encodings of multiple target value strings into a value string pool in binary format.

[0029] In the above scheme, the query module is also used to: determine the target key string that matches the target field; perform a binary search in the key pool lookup table to obtain the first key identifier of the target key string in the key pool lookup table; perform a binary search in the key lookup table to obtain the second key identifier corresponding to the first key identifier in the key lookup table; and query the target storage address corresponding to the value string corresponding to the second key identifier in the value lookup table.

[0030] An embodiment of the present application provides an electronic device, including:

[0031] A memory for storing executable instructions;

[0032] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0033] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0034] The embodiments of the present application have the following beneficial effects:

[0035] The first data in text format is serialized to obtain the second data in binary format, wherein the second data in binary format includes a key lookup table; the second data in binary format is pointer converted to obtain multiple storage addresses corresponding to multiple fields, so that the data can be directly read in subsequent queries, reducing the parsing time and not requiring additional memory required for parsing, obtaining a query request, and extracting a target field from the query request; based on the key lookup table, a target storage address whose corresponding field matches the target field is queried from the multiple storage addresses, and the target storage address is read to obtain a query result. Querying based on the key lookup table can improve query efficiency and reduce query complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a structural diagram of a data processing system provided in an embodiment of the present application;

[0037] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0038] Figures 3A-3D It is a flowchart of a data processing method provided in an embodiment of the present application;

[0039] Figure 4 It is a method class diagram of the data processing method provided in the embodiment of the present application;

[0040] Figure 5 This is a schematic diagram of data in text JSON format provided in an embodiment of the present application;

[0041] Figure 6 It is a test result diagram of the data processing method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0043] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0046] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0047] 1) JSON (JavaScript Object Notation), a lightweight data exchange format. It uses a text format that is completely independent of the programming language to store and represent data. The simplicity and clear hierarchical structure make JSON an ideal data exchange language, which is easy for people to read and write, and easy for machines to parse and generate, and can effectively improve network transmission efficiency.

[0048] 2) RapidJSON is a C++ JSON parser and generator. RapidJSON is small but complete, and supports both SAX and DOM style APIs. RapidJSON is memory-friendly, and on most machines, each JSON value only takes up 16 bytes (except for strings). It uses a fast memory allocator by default, allowing the parser to allocate memory compactly.

[0049] 3) BSON is a computer data exchange format, mainly used as a data storage and network transmission format in databases. It is a binary representation that can be used to represent simple data structures, associative arrays, and various data types.

[0050] 4) DangoBinJson, a read-only binary JSON format optimized for parsing and querying in the data processing method provided in the embodiment of the present application. In DangoBinJson, for object data, a key lookup table is specially serialized, and this key lookup table is ordered, so that binary search can be used to reduce the search time to O(logn).

[0051] 5) First data: The first data includes development data. During the development stage, since files need to be stored in a way that humans can understand, the development data is stored in the text JSON format.

[0052] 6) Second data: The second data includes release data. During the release stage, the development data in text JSON format is serialized into release data in binary format provided in the embodiment of the present application (hereinafter referred to as DangoBinJson) through serialization in the Cook stage and released with the game version, thereby obtaining the smallest package size, the fastest parsing speed and excellent query time.

[0053] 7) Binary search: First, assuming that the elements in the table are arranged in ascending order, compare the key recorded in the middle position of the table with the search key. If the two are equal, the search is successful; otherwise, use the middle position record to divide the table into two sub-tables, front and back. If the key recorded in the middle position is greater than the search key, further search the front sub-table, otherwise further search the back sub-table. Repeat the above process until a record that meets the conditions is found, in which case the search is successful, or until the sub-table does not exist, in which case the search is unsuccessful.

[0054] 8) Guide bit: The guide bit is the front bit of the second data in binary format. The size of the guide bit is no more than 8 bits and the minimum is 1 bit. It is stored in the high bit of the first byte of the second data in binary format. The guide bit lengths of different types of nodes are different. Therefore, different guide bits are used to characterize the type of the second data.

[0055] 9) Subsequent bits: Subsequent bits are bits in the second data in binary format except the leading bit. The length of the subsequent bits is determined by the leading bit. The shortest length of the subsequent bits can be zero, and there is no upper limit for the longest length of the subsequent bits. The meanings expressed by the subsequent bits of different data types are different.

[0056] BSON is a computer data exchange format, mainly used as a data storage and network transmission format in databases. It is a binary representation that can be used to represent simple data structures, associative arrays, and various data types. It also achieves faster traversal speed and smaller footprint by converting text JSON into binary format. Similar to JSON, BSON also provides the ability to map string keys to values ​​(which can be numbers, strings, or another object). BSON objects consist of a field name, a type, and a value. These types can also be used to determine the type of each BSON object. Each BSON type has two identifiers: integer and string, as shown in Table 1:

[0057] type Integer String Double 1 "double" String 2 "string" Object 3 "object" Array 4 "array" Binary data 5 "binData" ObjectId 7 "objectId" Boolean 8 "bool" Date 9 "date" Null 10 "null" Regular Expression 11 "regex" JavaScript 13 "javascript" 32-bit integer 16 "int" Timestamp 17 "timestamp" 64-bit integer 18 "long" Decimal128 19 "decimal"

[0058] Table 1 BSON object composition table

[0059] In the related technology, BSON is not designed for games and cannot achieve the faster parsing speed, smaller transmission volume and lower memory consumption required by games. In addition, there are contents in BSON that have little to do with games, such as date type, decimal type, etc., which leads to data structure redundancy when applied to game products.

[0060] In view of the above technical problems, the embodiments of the present application provide a data processing method, device, electronic device, computer program product and computer-readable storage medium, which can improve the efficiency of data reading and querying. The exemplary application of the electronic device provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers. Below, an exemplary application when the device is implemented as a server will be described.

[0061] See also Figure 1 , Figure 1 It is a structural diagram of a data processing system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via the network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0062] In some embodiments, the function of the data processing system is implemented based on the server 200 and the terminal 400. In the development stage, the game is developed based on JSON in text format. The server 200 receives the development data in text format and serializes the development data in text format to obtain the publishing data in binary format, wherein the publishing data in binary format includes a key lookup table. In the publishing stage, the publishing data in binary format is published along with the game version. When the terminal 400 installs the game data packet, the publishing data in binary format is pointer converted to obtain multiple storage addresses corresponding to multiple fields one by one; the terminal 400 receives the query request and extracts the target field from the query request. The terminal 400 queries the target storage address whose corresponding field matches the target field from the multiple storage addresses of the database 500 based on the key lookup table, and reads the target storage address to obtain the query result.

[0063] In some embodiments, the function of the data processing system is implemented solely based on the terminal 400 or the server 200. The terminal 400 receives development data in text format and serializes the development data in text format to obtain publishing data in binary format, wherein the publishing data in binary format includes a key lookup table. The terminal 400 performs pointer conversion on the publishing data in binary format to obtain multiple storage addresses corresponding to multiple fields one by one; the terminal 400 receives a query request and extracts a target field from the query request. The terminal 400 queries the target storage address whose corresponding field matches the target field from multiple storage addresses of the database 500 based on the key lookup table, and reads the target storage address to obtain the query result.

[0064] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0065] Next, the structure of the electronic device for implementing the data processing method provided by the embodiment of the present application is described. As mentioned above, the electronic device provided by the embodiment of the present application may be Figure 1 The server 200 or the terminal 400 in FIG. Figure 2 , Figure 2 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device is taken as a server 200 as an example for explanation. Figure 2 The server 200 shown includes: at least one processor 210, a memory 250, and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0066] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0068] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing tasks based on hardware;

[0071] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).

[0072] In some embodiments, the data processing device provided in the embodiments of the present application can be implemented in software. Figure 2 The data processing device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a serialization module 2551, a pointer module 2552 and a query module 2553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0073] In some embodiments, the terminal or server can implement the data processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or an instant messaging APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0074] The data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0075] See also Figure 3A , Figure 3A is a flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Figure 3A Steps 101 to 103 are shown for explanation.

[0076] In step 101, development data in text format is serialized to obtain release data in binary format.

[0077] As an example, in the development phase, since files need to be stored in a way that humans can understand, the development data is stored in the text JSON format. In the release phase, the development data in the text JSON format is serialized into the binary format release data provided by the embodiment of the present application (hereinafter referred to as DangoBinJson) through the serialization of the Cook phase and released with the game version, thereby obtaining the smallest package size, the fastest parsing speed and excellent query time. By consuming the serialization time, data structures such as the key string pool and the lookup table are generated, thereby saving parsing time and memory usage when the game is running. The serialized C++ implementation code is as follows:

[0078] std::vector<uint8_t> data = dango::BinJson::SerializeFromRapidJson(rapidJsonNode); / / Serialize rapidJsonNode into binary data.

[0079] As an example, an ordered key lookup table is generated during the serialization phase, so the binary formatted published data includes an ordered key lookup table, which includes a key lookup table and a key pool lookup table. The first byte in the key pool lookup table indicates how many bytes each subsequent item occupies, which may be 1, 2, 4, or 8, indicating that each subsequent item is read according to this length. Each item after the first byte stores the offset to each key. The key pool lookup table is the lookup directory for the key. Through this key pool lookup table, you can quickly index to the key by identifier. The first byte in the key lookup table indicates how many bytes each subsequent item occupies, which may be 1, 2, 4, or 8, indicating that each subsequent item is read according to this length. Each item after the first byte stores the key identifiers of all keys in the key string pool. The ordered key lookup table can be used to speed up query processing for the target field, thereby improving query speed and reducing query time.

[0080] In step 102, a query request is obtained, and a target field is extracted from the query request.

[0081] As an example, the target field may include name, age, etc. When the target field is name, it indicates that the user needs to query the name. When the target field is age, it indicates that the user needs to query the age. The query request will also carry the target data type, so that the query result of the target field can be output according to the target data type. The class name implemented in C++ is dango::BinJson. This class can serialize the development data in text JSON format into DangoBinJson and provide the function of querying DangoBinJson, for example, query the target field through AsInt32, AsString and other functions and output it in the target data type.

[0082] In step 103, a value query result matching the target field is determined based on the ordered key lookup table, and a target storage address corresponding to the value query result is read to obtain a value read result.

[0083] As an example, the target storage address is obtained by converting the binary formatted data into a pointer. The C++ implementation code for deserialization is as follows:

[0084] const dango::BinJson&binJ = dango::BinJson::FromBin(data.data()); / / Construct BinJson class from binary data

[0085] In some embodiments, the deserialization process can be implemented by the following C++ code:

[0086]

[0087] When deserializing, call the const BinJson&BinJson::FromBin(const void*data) function to convert the data into a const reference of the BinJson class. The const reference of BinJson is the Bin Json pointer, which is used to represent multiple storage addresses. The FromBin() function is implemented as follows: After skipping the string pool (stringpool) in the binary data, the type conversion process is performed directly through the reinterpret_cast operator. After the data is converted into a BinJson pointer (pointing to the storage address), it can be read directly, thus saving parsing time and not requiring additional parsing memory allocation.

[0088] In some embodiments, the published data includes a leading bit and subsequent bits. In step 101, the development data in text format is serialized to obtain the published data in binary format, which can be achieved by the following technical solution: when the data type of the development data is a first basic type, the first basic type is binary-encoded to obtain the leading bit of the development data; wherein, the first basic type includes at least one of the following: single-precision floating point, double-precision floating point, unsigned integer, signed integer, and the length of the leading bit is eight bits; the development data is binary-encoded to obtain the subsequent bits of the development data; wherein, when the first basic type is a single-precision floating point, the length of the subsequent bits is four bytes; when the first basic type is a double-precision floating point, the length of the subsequent bits is eight bytes; when the first basic type is an unsigned integer, the length of the subsequent bits is the number of bits of the unsigned integer; when the first basic type is a signed integer, the length of the subsequent bits is the number of bits of the signed integer. The leading bit can be used to characterize the specific type of the first basic type, and the subsequent bits can be used to characterize the specific development data. After reading the leading bit, the data type of the published data in binary format and the reading length of the subsequent bits can be known. For example, when the leading bit characterizes that the first basic type is a single-precision floating point type, the subsequent four bytes can be read directly, thereby improving reading efficiency.

[0089] In some embodiments, the published data includes a guide bit, and in step 101, the development data in text format is serialized to obtain the published data in binary format, which can be achieved through the following technical solution: when the data type of the development data is a second basic type, the second basic type is binary encoded to obtain the guide bit of the development data, wherein the second basic type includes at least one of the following: Boolean type, empty object type, set digital type.

[0090] The guide bit can be used to represent the specific type of the second basic type. Since each second basic type only includes one data, for example, the true Boolean type is True, the false Boolean type is False, the digital type 1 is set to only include the number 1, and the digital type 16 is set to only include the number 16, the guide bit can simultaneously represent the data type and the specific data, thereby improving the encoding efficiency and reading efficiency.

[0091] As an example, there are several basic types: single-precision floating point type (Float), double-precision floating point type (Double), unsigned integer type (Uint), signed integer type (Sint), Boolean type (Boolean), null object type (Null) and string pool (String Pool). The leading bit length of the basic type is 8 bits, which is equivalent to filling up the first byte. The subsequent bit length and function are shown in Table 2 below:

[0092]

[0093]

[0094] Table 2 Description of the basic type of leading bit and subsequent bits

[0095] As an example, common numbers form a data table, and common data includes 16 common numbers such as 0.0, 1.0, 0, 1, and -1. During serialization, if this number exists in the data table, its serial number in the data table is directly written into the leading bit, and no subsequent bits are required. The encoding method of common numbers in the data table is shown in Table 3:

[0096]

[0097]

[0098] Table 3 Description of the leading and subsequent digits of common numbers

[0099] In some embodiments, the release data includes a leading bit and a subsequent bit, see Figure 3B , Figure 3B is a flow chart of the data processing method provided in the embodiment of the present application. In step 101, the development data in text format is serialized to obtain the release data in binary format, which can be obtained by Figure 3B Steps 1011 to 1013 are implemented as shown.

[0100] In step 1011, when the data type of the development data is the third basic type, binary encoding is performed on the third basic type to obtain a leading bit of the development data.

[0101] As an example, the third basic type includes a character string type, and the length of the leading bit is three bits.

[0102] In step 1012, the character string offset of the development data is binary-encoded to obtain the character string offset in binary format.

[0103] In some embodiments, the data type of the string offset is a variable-length integer, and the string offset is the offset of a specific string (development data) in the string pool. Since different strings have different positions in the string pool, the string offsets are different. In step 1012, the string offset of the development data is binary-encoded to obtain a string offset in binary format, which can be achieved by the following technical solution: obtaining the binary code of the string offset; parsing the binary code to obtain at least one parsing result, that is, splitting the string offset in units of 7 bits, and performing the following processing for each parsing result in the order of the positions of the parsing results: when the parsing result does not include the last bit of the binary code of the string offset, zero is used as the control bit of the corresponding parsing result, and the parsing result is used as the storage bit; when the parsing result includes the last bit of the binary code of the string offset, one is used as the control bit of the corresponding parsing result, and the parsing result is used as the storage bit; the control bit and the storage bit corresponding to the same parsing result are combined into bytes corresponding to the parsing result, and the bytes of each parsing result are combined into a string offset in binary format.

[0104] As an example, in a binary data stream, some places need to store data such as capacity (Size) and offset (Offset). This type of data varies in size and floats a lot, so it is not suitable to be represented by a fixed-length integer. The encoding method of variable-length numbers is as follows: the highest bit is used as the control bit, and the remaining bits in the byte are used as storage bits. If the control bit is 1, it means that the remaining storage bits of this byte can already store this number. If the control bit is 0, it means that the next byte will be used for storage. If the control bit of the next byte is 1, then the storage bits of these two bytes can already store this number. If the control bit is 0, then the storage bits of these two bytes cannot store this number, and the next byte will be used for storage. And so on. The encoding method is shown in Table 4:

[0105]

[0106] Table 4 Variable length integer encoding format table

[0107] As an example, the control bit of the 0th byte is 0, indicating that the remaining storage bits of the 0th byte are not enough to store the variable-length integer, and the 1st byte is used for storage. The control bit in the 1st byte is 1, indicating that the remaining storage bits of the 1st byte plus the remaining storage bits of one byte are enough to represent the variable-length integer, so the backward encoding stops. The variable-length integer shown in Table 4 can be represented by the following binary encoding 00010100101101.

[0108] In step 1013, the subsequent bits of the development data are determined according to the string offset in binary format.

[0109] In some embodiments, the above-mentioned determination of the subsequent bits of the development data according to the string offset in binary format can be implemented by the following technical solution: when the string in the development data is not queried in the string pool, the string offset is determined to be zero, and the binary encoding of the string offset in binary format and the string value of the development data are combined as the subsequent bits of the development data; when the string in the development data is queried in the string pool, the string offset in binary format is determined as the subsequent bits of the development data. If the string exists in the string pool, only the string offset is encoded, thereby reducing the encoding cost and improving the reading efficiency, but there are still a small number of strings that cannot be queried in the string pool, so the string itself can be directly encoded, thereby ensuring that all strings can be encoded, and also controlling the size of the string pool so as not to occupy too much space.

[0110] As an example, the string in data node A of string type found in the key string pool or value string pool is Key2 in the key string pool, which means that the length of data node A is greater than the length threshold. Then, the first byte written to data node A is the string offset of Key2. The string of data node A is not found in either the Key Pool or the String Value Pool, which means that the length of data node A is not greater than the length threshold and is not suitable for being placed in the value string pool. Then, the Offset variable length number will be equal to 0, and then the string of data node A will be directly stored in the bytes following the string offset.

[0111] In some embodiments, the release data includes a leading bit and a subsequent bit, see Figure 3C , Figure 3C is a flow chart of the data processing method provided in the embodiment of the present application. In step 101, the development data in text format is serialized to obtain the release data in binary format, which can be obtained by Figure 3C Steps 1014 to 1019 are implemented as shown.

[0112] In step 1014, when the data type of the development data is the fifth basic type, binary encoding is performed on the fifth basic type to obtain a leading bit of the development data.

[0113] In step 1015, binary encoding is performed on the object data volume of the development data to obtain the object data volume in binary format.

[0114] In step 1016, the string pool offset of the development data is binary-encoded to obtain the string pool offset in binary format.

[0115] As an example, the fifth basic type includes an object type. The length of the leading bit is one bit. The leading bit of the binary format of the data node of the object data type is only 1 bit, and then the remaining 7 bits and possible subsequent bits are a variable-length integer, which stores the amount of data in the data node (how many elements are in the data node), and then stores the string pool offset from the data node to the string pool. The string pool offset is also a variable-length integer, and the encoding method is similar to the encoding method of the string offset.

[0116] In step 1017, the key lookup table of the development data is binary-encoded to obtain a key lookup table in binary format.

[0117] In some embodiments, the key lookup table includes a table type bit in binary format and a key identification bit in binary format. The above-mentioned binary encoding processing of the key lookup table of the development data to obtain the key lookup table in binary format can be implemented by the following technical scheme: obtaining the key identification of each object data in the development data; sorting multiple key identifications in ascending order, and obtaining the identification difference between each key identification and the key identification ranked first; obtaining the maximum value from the key identification ranked first and at least one identification difference; encoding the number of bits of the maximum value into a table type bit in binary format, and encoding each identification difference and the key identification ranked first into a key identification bit in binary format; wherein the length of the key identification bit conforms to the number of bits corresponding to the table type bit of the key lookup table. In the embodiment of the present application, the encoding method for the key lookup table is based on the identification difference. By encoding the identification difference, it is not necessary to encode each key identification, which saves the encoding cost and improves the encoding efficiency.

[0118] As an example, the encoding structure of the key lookup table in the binary format of the data node of the object data type is shown in Table 5:

[0119]

[0120] Table 5 Encoding format of key lookup table

[0121] The first byte of the key lookup table is the table type bit type, which is used to represent the length of each subsequent item. It can be uint8 (1 byte), uint16 (2 bytes), uint32 (4 bytes) or uint64 (8 bytes). Then the key identifiers of all keys in the key string pool Key Pool are stored. Only id0 stores the complete id. After that, each id stores the difference between the complete id and id0, thereby reducing the size of the stored numbers and compressing the type to a smaller data type as much as possible, such as compressing it to uint8 (1 byte).

[0122] In step 1018, the value lookup table of the development data is binary-encoded to obtain a value lookup table in binary format.

[0123] In some embodiments, the above-mentioned binary encoding processing of the value lookup table of the development data to obtain the value lookup table in binary format can be achieved through the following technical scheme: obtaining the data value of each object data in the development data; encoding the number of bits of the largest data value into the table type bits in the binary format of the value lookup table, and encoding the data value of each object data into the value bits in the binary format of the value lookup table; wherein the length of the value bits in binary format corresponding to each data value conforms to the number of bits corresponding to the table type bits of the value lookup table.

[0124] In step 1019, the object data amount in binary format, the string pool offset in binary format, the key lookup table in binary format, the value lookup table in binary format, and the binary encoding of each data value of the development data are combined into subsequent bits of the development data.

[0125] As an example, there are two search acceleration structures in the binary format of the object data type. The encoding structure of the object data type is shown in Table 6:

[0126]

[0127] Table 6 Object data encoding format table

[0128] The leading bit of the binary format of the data node of the object data type is only 1 bit, and the remaining 7 bits and possible subsequent bits are a variable-length integer, which stores the amount of data in the data node (how many elements are in the data node), and then stores the offset Offset from the data node to the String Pool, which is also a variable-length integer. Subtracting this Offset from the this pointer can get the position of the String Pool, and then stores the key lookup table (Key Search Table, different from the key lookup table in the key string pool), and then stores the value lookup table (Value Search Table), which is similar to the array lookup table in the binary format of the array. It is used to quickly jump to the corresponding data value Value after finding the index in the key lookup table, and finally stores multiple data values ​​Value in sequence.

[0129] In some embodiments, the published data includes a leading bit and subsequent bits. In step 101, the development data in text format is serialized to obtain the published data in binary format, which can be achieved by the following technical solution: when the data type of the development data is the fourth basic type, the fourth basic type is binary-encoded to obtain the leading bit of the development data; wherein the fourth basic type includes an array type, and the length of the leading bit is two bits; the array capacity of the development data is binary-encoded to obtain the array capacity in binary format; the array lookup table of the development data is binary-encoded to obtain the array lookup table in binary format; the array capacity in binary format, the array lookup table in binary format, and the binary encoding of the array value of the development data are combined into the subsequent bits of the development data.

[0130] In some embodiments, the above-mentioned binary encoding processing of the array lookup table of the development data to obtain the array lookup table in binary format can be implemented by the following technical scheme: obtaining the array offset of each array value in the development data; encoding the number of bits of the largest array offset as the table type bit in the binary format of the array lookup table, and encoding the array offset of each array value as the offset bit in the binary format of the array lookup table; wherein the length of the offset bit in the binary format corresponding to each array value conforms to the number of bits corresponding to the table type bit in the binary format. By encoding the array lookup table, the array reading speed can be improved, that is, there is no need to read each array value in the array one by one in sequence, that is, the corresponding array value can be quickly jumped to through the array lookup table, thereby improving the reading efficiency.

[0131] As an example, the encoding format of an array is shown in Table 7:

[0132]

[0133] Table 7 Array encoding format table

[0134] The length of the leading bit of the array corresponding to the binary format is 2 bits. The subsequent bits of the array corresponding to the binary format first store the size of the array, which is a variable-length integer used to represent how many elements there are in the array. It is composed of the remaining 6 bits of the first byte and possible subsequent bytes. Then the array search table Array Search Table is stored. This is an acceleration structure. Next, the values ​​of each data in the array are stored one by one behind the array search table. The main function of the Array Search Table is to quickly jump to the corresponding data value when an index is given. The encoding structure of the Array Search Table is shown in Table 8:

[0135]

[0136] Table 8 Array encoding format table

[0137] Search Table Type is used to represent the size of each subsequent Offset, which can be uint8 (1 byte), uint16 (2 bytes), uint32 (4 bytes) or uint64 (8 bytes), and then stores the offset Offset from the this pointer of this data node to the corresponding data value value. When it is necessary to jump to a data value value at a certain index, first read the table type bit in binary format of the array lookup table, and then continue to read the offset bit in binary format of the array lookup table corresponding to the index according to the read data size (used to represent the offset Offset from the this pointer of this data node to the corresponding data value value), and then add the length of this offset Offset to the this pointer to get the storage address pointing to the corresponding data value Value, that is, after finding the index index in the Array Search Table, you can quickly jump to the corresponding data value Value.

[0138] In some embodiments, the binary format string pool of development data includes a guide bit and a subsequent bit. When the text format development data is serialized to obtain the binary format release data, the string pool type is binary-encoded to obtain the guide bit of the corresponding string pool type; a skip offset for skipping the string pool of development data is obtained, and the skip offset is binary-encoded to obtain the skip offset in binary format; the key string pool of development data is binary-encoded to obtain the key string pool in binary format; the value string pool of development data is binary-encoded to obtain the value string pool in binary format; the skip offset in binary format, the key string pool in binary format, and the value string pool in binary format are combined to form the subsequent bit of the corresponding string pool. When encoding the string pool, the skip offset is encoded, and the skip offset is the offset for skipping the current string pool, which is used to skip the current string pool. The key string pool and the value string pool are also encoded. The search time can be reduced by using the key string pool, and the string itself can be stored by using the value string pool. The efficiency of subsequent string search and reading can be improved by encoding the string pool.

[0139] As an example, in a data node tree in text JSON format, a string will exist in the following two places: the key string of the object data and the value string of the string node. If it is stored in text format, it will be stored once when a string appears. Figure 5 , Figure 5 This is a schematic diagram of data in text JSON format provided by an embodiment of the present application. By observing the data in text JSON format actually stored in the game, it can be found that many structures are repeated, for example Figure 5In the text, keys such as "chinese", "math", and "sport" will appear multiple times and be stored multiple times. The data node tree in the entire text JSON format will be traversed during the serialization phase, and some strings that meet the requirements (length greater than the length threshold) will be extracted and added to the string pool (String Pool). The String Pool consists of two parts, including an ordered key string pool (Key Pool) and an unordered value string pool (String Value Pool). Therefore, in the embodiment of the present application, all key strings are extracted to form a key string pool (Key Pool), and then when storing key strings, they can be replaced with simple string identifiers (StringID). In addition, it is observed that some value strings stored in string nodes are likely to have appeared in the key string pool, or a sentence whose length exceeds the length threshold. For strings that have appeared in the key string pool, the string identifier in the key string pool can be directly used. For those value strings whose length is greater than 2 and have not appeared in the key string pool, they are also extracted to form a value string pool (String Value Pool). The key string pool and the value string pool are combined together to form a string pool. The encoding format of the string pool is shown in Table 9:

[0140]

[0141] Table 9 String Pool encoding format table

[0142] The leading bit of the string pool is 8 bits. SkipSize is the offset (Offset) of skipping this string pool. The data type of the offset of the string pool is fixed-length uint64, that is, binary encoding is performed using the encoding method corresponding to the fixed-length uint64. SkipSize is used to help us quickly skip this StringPool when reading data. After SkipSize, the key string pool and value string pool are written to the subsequent bits.

[0143] In some embodiments, the above-mentioned binary encoding processing of the key string pool of development data to obtain a key string pool in binary format can be achieved through the following technical scheme: obtaining multiple key strings in the development data; binary encoding the number of multiple key strings to obtain the number code of the key string pool; binary encoding processing of the key pool lookup table of the development data to obtain a key pool lookup table in binary format; binary encoding processing of each key string to obtain the binary code of each key string; wherein the binary code of the key string includes the binary code of the number of bits of the key string and the key string in binary format; the number code of the key string pool, the key pool lookup table in binary format, and the binary code of the key string form a key string pool in binary format.

[0144] As an example, the key string pool carries an important task, which is to reduce the search time and keep the search complexity to (Ologn), so the key string pool is ordered. In addition, there is a search acceleration structure called the key pool search table (Key Search Table) in the key string pool. The encoding format of the key string pool is shown in Table 10:

[0145]

[0146] Table 10 Key Pool encoding format table

[0147] In some embodiments, the above-mentioned binary encoding processing of the key pool lookup table of the development data to obtain a key pool lookup table in binary format can be achieved through the following technical scheme: obtaining the offset of each key string; encoding the number of bits of the maximum offset as table type bits in the binary format of the key pool lookup table, and encoding the offset of each key string as key offset bits in the binary format of the key pool lookup table; wherein the length of the key offset bits conforms to the number of bits corresponding to the table type bits in the binary format of the key pool lookup table.

[0148] You can see that the composition of the key string pool is mainly composed of a key pool lookup table plus multiple keys. Each key is composed of a size Size and a string Raw String. Size is the variable-length integer mentioned earlier. The key pool lookup table is described below. The encoding format of the key pool lookup table is shown in Table 11:

[0149]

[0150] Table 11 Key Search Table encoding format

[0151] First, the first byte in the key pool lookup table indicates how many bytes each subsequent item occupies, which may be 1, 2, 4, or 8, indicating that each subsequent item is read according to this length. Then, each item stores the offset to each key. This key pool lookup table is the key search directory. Through this key pool lookup table, you can quickly index to the key through the identifier, and then store the identifier in the binary data stream of the corresponding object data type to achieve fast query.

[0152] In some embodiments, the above-mentioned binary encoding processing of the value string pool of development data to obtain the value string pool in binary format can be implemented by the following technical solutions: obtaining multiple value strings in the development data; when there is no key string identical to the value string and the length of the value string is greater than the length threshold, determining the value string as the target value string; obtaining the length of each target value string, and performing the following processing for each target value string: binary encoding the number of digits of the target value string to obtain the binary encoding of the number of digits of the target value string, and binary encoding the target value string to obtain the target value string in binary format; the binary encoding of the number of digits of the target value string and the target value string in binary format form the binary encoding of the target value string; and concatenating the binary encodings of multiple target value strings into a value string pool in binary format. By encoding the value string pool, the value string pool can be directly searched in the subsequent query string pool, which improves the query speed and reading speed.

[0153] As an example, the following continues to explain the value string pool (String Value Pool). The encoding format of the value string pool is shown in Table 12:

[0154]

[0155] Table 12 String Value Pool encoding format

[0156] Different from the key string pool, the strings stored in the value string pool are not ordered. Its function is to be used for data nodes of string data type.

[0157] In some embodiments, see Figure 3D , Figure 3D is a flow chart of the data processing method provided by the embodiment of the present application. In step 103, the value query result matching the target field is determined based on the key lookup table. Figure 3D Steps 1031 to 1034 are implemented as shown.

[0158] In step 1031, a target key string matching the target field is determined.

[0159] In step 1032, a binary search is performed in the key pool lookup table to obtain the first key identifier of the target key string in the key pool lookup table.

[0160] In step 1033, a binary search is performed in the key lookup table to obtain a second key identifier corresponding to the first key identifier in the key lookup table.

[0161] In step 1034, a target storage address corresponding to the value string corresponding to the second key identifier is searched in the value lookup table.

[0162] As an example, when performing a search, after obtaining the given string name (the target key string corresponding to the target field), first perform a binary search on the key pool lookup table in the key string pool to convert this string into an index in its key pool lookup table (the first key identifier in the key string pool). Then use this identifier to perform a binary search in the key lookup table of the object data to obtain its index in the key lookup table (the second key identifier). Then use the index in the key lookup table to jump to the target storage address of the corresponding Value (value string) in the value lookup table (Value Search Table), effectively improving the query speed.

[0163] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0164] In some embodiments, the function of the data processing system is implemented based on the server 200 and the terminal 400. In the development stage, the game is developed based on JSON in text format. The server 200 receives the development data in text format and serializes the development data in text format to obtain the publishing data in binary format, wherein the publishing data in binary format includes a key lookup table. In the publishing stage, the publishing data in binary format is published along with the game version. When the terminal 400 installs the game data packet, the publishing data in binary format is pointer converted to obtain multiple storage addresses corresponding to multiple fields one by one; the terminal 400 receives the query request and extracts the target field from the query request. The terminal 400 queries the target storage address whose corresponding field matches the target field from the multiple storage addresses of the database 500 based on the key lookup table, and reads the target storage address to obtain the query result.

[0165] The data processing method provided in the embodiment of the present application is widely used in various serialization bottom layers of self-developed engines. In the development stage, since files need to be stored in a way that humans can understand, the development data is stored in the text JSON format. In the release stage, the development data in the text JSON format is serialized into the binary format release data provided in the embodiment of the present application (hereinafter referred to as DangoBinJson) through the serialization of the Cook stage and released with the game version, thereby obtaining the smallest package size, the fastest parsing speed and excellent query time. By consuming serialization time, key string pools, lookup tables and other data structures are generated, thereby saving parsing time and memory usage when the game is running.

[0166] The following introduces the implementation of the data processing method provided in the embodiment of the present application through C++. The class name implemented through C++ is dango::BinJson. This class can serialize the development data in the text JSON format into DangoBinJson, and provide the function of querying DangoBinJson, for example, querying the target field through functions such as AsInt32, AsString, and outputting it in the target data type.

[0167] As an example, the C++ implementation code of serialization is as follows:

[0168] std::vector<uint8_t> data = dango::BinJson::SerializeFromRapidJson(rapidJsonNode); / / Serialize rapidJsonNode into binary data

[0169] As an example, the C++ implementation code for deserialization is as follows:

[0170] const dango::BinJson&binJ = dango::BinJson::FromBin(data.data()); / / Construct BinJson class from binary data

[0171] As an example, dango::BinJson is similar to a JSON node. It is like rapidjson::Value, which may be a number, string, array, object, etc. When deserializing, call the const BinJson&BinJson::FromBin(const void*data) function to convert the data into a const reference to BinJson. The const reference to BinJson is a BinJson pointer, which is used to represent multiple storage addresses. The FromBin() function is implemented as follows: After skipping the string pool in the binary data, the type conversion process is performed directly through the reinterpret_cast operator. After the data is converted into a BinJson pointer (pointing to the storage address), it can be read directly, thus saving parsing time and not requiring additional parsing memory allocation.

[0172] As an example, the C++ implementation code of query processing is as follows:

[0173] std::string name = binJ["name"].AsString(); / / Find the "name" node and convert it to string

[0174] uint32_t age=binJ["age"].AsSInt32(); / / Find the age node and convert it to Int32

[0175] In some embodiments, see Figure 4 , Figure 4 It is a method class diagram of the data processing method provided in an embodiment of the present application. The class diagram of dango::BinJson includes multiple method functions. The AsSInt32 method function is taken as an example for explanation. The data node obtained after serialization (hereinafter referred to as the BinJson node) may be an Int data type, a Float data type, a String data type, etc. The As series function can interpret the data node as the type expected by the user. For example, the BinJson node itself may be a Boolean type True, but if the AsSInt32 function is called to execute, it will return a value of 1 of the Int type.

[0176] In some embodiments, DangoBinJson is composed of "leading bits" and "subsequent bits". The first byte of any correct DangoBinJson node should contain at least one leading bit in the high position. The leading bit is no more than 8 bits and the minimum is 1 bit. It is stored in the high position of the first byte. The leading bit lengths of BinJson nodes of different data types are different. The length of the subsequent bits is determined by the leading bit. The shortest can be none, that is, the leading bit already contains all the information, such as 0, 1, -1 and other numbers in the data table. There is no upper limit for the longest length of the subsequent bits. For example, in the BinJson node of the object data type, in addition to the leading bit, it also includes subsequent bits with no upper limit.

[0177] In some embodiments, there are the following basic types: single-precision floating point type (Float), double-precision floating point type (Double), unsigned integer type (Uint), signed integer type (Sint), Boolean type (Boolean), null object type (Null) and string pool (String Pool). The leading bit length of the basic type is 8 bits, which is equivalent to filling up the first byte. The subsequent bit length and function are shown in Table 13 below:

[0178]

[0179] Table 13 Description of the leading bit and subsequent bits of the basic type

[0180] In some embodiments, common numbers form a data table, and the common data includes 16 common numbers such as 0.0, 1.0, 0, 1, and -1. During serialization, if the number exists in the data table, its serial number in the data table is directly written into the leading bit, and no subsequent bits are required. The encoding method of common numbers in the data table is shown in Table 14:

[0181]

[0182]

[0183] Table 14 Description of the leading and subsequent digits of common numbers

[0184] In some embodiments, in a binary data stream, some places need to store data such as capacity (Size) and offset (Offset). This type of data varies in size and floats a lot, so it is not suitable to be represented by a fixed-length integer. The encoding method of a variable-length number is as follows: the highest bit is used as a control bit, and the remaining bits in the byte are used as storage bits. If the control bit is 1, it means that the remaining storage bits of this byte can already store this number. If the control bit is 0, it means that the next byte will be used for storage. If the control bit of the next byte is 1, then the storage bits of these two bytes can already store this number. If the control bit is 0, then the storage bits of these two bytes cannot store this number, and the next byte will be used for storage. And so on. For the encoding method, see Table 15:

[0185]

[0186] Table 15 Variable length integer encoding format table

[0187] As an example, the control bit of the 0th byte is 0, indicating that the remaining storage bits of the 0th byte are not enough to store the variable-length integer, and the 1st byte is used for storage. The control bit in the 1st byte is 1, indicating that the remaining storage bits of the 1st byte plus the remaining storage bits of one byte are enough to represent the variable-length integer, so the backward encoding stops. The variable-length integer shown in Table 4 can be represented by the following binary encoding 00010100101101.

[0188] In some embodiments, strings are always a hotspot for storage. Since the length of strings is not fixed and usually takes up more space than data types such as numbers or booleans, DangoBinJson has made special optimizations for strings. During the serialization phase, it will traverse the entire data node tree in the text JSON format, extract some strings that meet the requirements (length greater than the length threshold), and add them to the string pool. The string pool consists of two parts, including an ordered key string pool (Key Pool) and an unordered value string pool (String Value Pool).

[0189] As an example, in a data node tree in text JSON format, a string will exist in the following two places: the key string of the object data and the value string of the string node. If it is stored in text format, it will be stored once when a string appears. Figure 5 , Figure 5 This is a schematic diagram of data in text JSON format provided by an embodiment of the present application. By observing the data in text JSON format actually stored in the game, it can be found that many structures are repeated, for example Figure 5Keys such as "chinese", "math", and "sport" will appear many times and be stored many times, so in the embodiment of the present application, all key strings are extracted to form a key string pool (Key Pool), and then when storing the key string, it can be replaced by a simple string identifier (String ID). It is also observed that some value strings stored in string nodes are likely to have appeared in the Key Pool, or a sentence whose length exceeds the length threshold. For strings that have appeared in the Key Pool, the String ID in the Key Pool can be used directly. For those value strings that are greater than 2 and have not appeared in the Key Pool, they are also extracted to form a value string pool (String Value Pool). The Key Pool and the StringValue Pool are combined together to form a String Pool. The encoding format of the String Pool is shown in Table 16:

[0190]

[0191] Table 16 String Pool encoding format table

[0192] In some embodiments, the leading bit of String Pool is 8 bits, SkipSize is the offset (Offset) of skipping this StringPool, and the data type of the offset of StringPool is fixed-length uint64, that is, binary encoding is performed using the encoding method corresponding to the fixed-length uint64. SkipSize is used to help us quickly skip this StringPool when reading data, and the Key Pool and String Value Pool are written to subsequent bits after SkipSize.

[0193] In some embodiments, the Key Pool carries an important task, which is to reduce the search time and keep the search complexity to (Ologn), so the Key Pool is ordered. In addition, there is a search acceleration structure called Key Search Table in the Key Pool. The encoding format of the Key Pool is shown in Table 17:

[0194]

[0195] Table 17 Key Pool encoding format table

[0196] It can be seen that the composition of the Key Pool is mainly composed of the Key Search Table plus multiple keys. Each key is composed of the size Size and the string Raw String. Size is the variable-length integer mentioned earlier. The following will continue to explain the KeySearch Table. The encoding format of the Key Search Table is shown in Table 18:

[0197]

[0198] Table 18 Key Search Table encoding format

[0199] First, the first byte in the Key Search Table indicates how many bytes each subsequent item occupies, which may be 1, 2, 4, or 8, indicating that each subsequent item is read according to this length. Then, each item stores the offset to each key. This Key Search Table is the search directory for keys. Through this Key Search Table, you can quickly index to the key through the identifier, and then store the identifier in the binary data stream of the corresponding object data type to achieve fast query.

[0200] In some embodiments, the value string pool (String Value Pool) is further described below, and the encoding format of the Key SearchTable is shown in Table 19:

[0201]

[0202] Table 19 String Value Pool encoding format

[0203] Different from Key Pool, the Strings stored in String Value Pool are not ordered. Its function is to be used for data nodes of string data type.

[0204] In some embodiments, the encoding format for the string data type is as shown in Table 20:

[0205]

[0206] Table 20 String encoding format table

[0207] String Offset is the offset to a specific string in the String Pool. String Offset is a variable-length integer consisting of the remaining 5 bits of the first byte and possible subsequent bytes. In the node of the string data type, it can be seen that the binary format corresponding to the node of the string data type consists of a 3-bit leading bit and subsequent bits. The subsequent bits first store the String Offset. In the serialization phase, the entire data node tree in the text JSON format will be traversed to build the Key Pool and String Value Pool. When serializing the node of the string data type, the Key Pool and String Value Pool will be searched for the string of this node. If the string of this node is found, the Offset of the string found in the pool can be obtained for this node, see Table 21:

[0208]

[0209] Table 21 The relationship between the key string pool, value string pool and the position of the string node

[0210] For example, if the string in the data node A of the string type is queried in the Key Pool, it is Key2. Then the first byte written to the data node A is the offset of Key2. When the AsString function is executed for the data node A, the this pointer of the data node A is subtracted by the distance of the offset, and the storage address pointing to the string is obtained. For example, if the string of the data node A is not queried in the KeyPool and the String Value Pool, it means that the length of the data node A is not greater than the length threshold and is not suitable for being placed in the String Value Pool. Then the Offset variable length number will be equal to 0, and then the string of the data node A is directly stored in the bytes following the String Offset. The encoding format of the string is shown in Table 22:

[0211]

[0212] Table 22 String encoding format table

[0213] When the AsString function is executed, the String Offset is read out first. If the String Offset is not 0, the position of the this pointer of the data node is subtracted from the length of the String Offset to get the required string. If the String Offset is equal to 0, the subsequent bits are read directly, and the string is connected after the first byte.

[0214] In some embodiments, the encoding format of the array (Array) is shown in Table 23:

[0215]

[0216] Table 23 Array encoding format table

[0217] The length of the leading bit of the array corresponding to the binary format is 2 bits. The subsequent bits of the array corresponding to the binary format first store the size of the array, which is a variable-length integer used to represent how many elements there are in the array. It is composed of the remaining 6 bits of the first byte and possible subsequent bytes. Then the array search table Array Search Table is stored. This is an acceleration structure. Next, the values ​​of each data in the array are stored one by one behind the array search table. The main function of the Array Search Table is to quickly jump to the corresponding data value when an index is given. The encoding structure of the Array Search Table is shown in Table 24:

[0218]

[0219] Table 24 Array encoding format table

[0220] Search Table Type is used to represent the size of each Offset that follows. It can be represented by uint8 (1 byte), uint16 (2 bytes), uint32 (4 bytes) or uint64 (8 bytes). It then stores the offset Offset from the this pointer of the data node to the corresponding data value value. When you need to jump to a data value value at a certain index, first read out the Search Table Type, and then continue to read out the offset Offset corresponding to the index in the Search TableType according to the size of the read data. Then, add the length of the offset Offset to the this pointer to get the storage address pointing to the corresponding data value Value. That is, after finding the index index in the Array Search Table, you can quickly jump to the corresponding data value Value.

[0221] In some embodiments, the object data type is the most complex structure. In order to achieve a search speed of O(logn), the binary format of the object data type has two search acceleration structures. The encoding structure of the object data type is shown in Table 25:

[0222]

[0223] Table 25 Object data encoding format table

[0224] In some embodiments, the leading bit of the binary format of the data node of the object data type is only 1 bit, and the remaining 7 bits and possible subsequent bits are a variable-length integer, which stores the amount of data in the data node (how many elements are in the data node), and then the offset Offset from the data node to the String Pool is stored, which is also a variable-length integer. The position of the String Pool can be obtained by subtracting the Offset from the this pointer, and then the key lookup table (Key Search Table, different from the key lookup table in the key string pool) is stored, and then the value lookup table (Value Search Table) is stored, which is similar to the array lookup table in the binary format of the array, and is used to quickly jump to the corresponding data value Value after finding the index in the key lookup table, and finally store multiple data values ​​Value in sequence.

[0225] In some embodiments, the key string pool Key Pool consists of a capacity Size, a key search table Key Searchtable and keys. When a user needs to query a name, first, the name key string is converted into an identifier in the Key Pool in the key search table of the key string pool. The identifier in the Key Pool is stored in the key search table Key Search Table in the binary format of the data node of the object data type. The encoding structure of the key search table in the binary format of the data node of the object data type is shown in Table 26:

[0226]

[0227] Table 26 Encoding format of key lookup table

[0228] The first byte of the Key Search Table is type, which is used to represent the size of each subsequent item. It can be represented by uint8 (1 byte), uint16 (2 bytes), uint32 (4 bytes) or uint64 (8 bytes). Then, the identifiers of all keys in the Key Pool are stored. Only id0 stores the complete id. After that, each id stores the difference between the complete id and id0, thereby reducing the size of the stored numbers and compressing the type to a smaller data type as much as possible, such as compressing it to uint8 (1 byte).

[0229] When searching, after obtaining the given string name (target field), first perform a binary search on the key pool lookup table in the Key Pool to convert the string into its index in the key pool lookup table (identifier in the Key Pool), then use this identifier to perform a binary search in the key lookup table of the object data to obtain its index in the key lookup table, and then use the index in the key lookup table to jump to the corresponding Value (value string) in the value lookup table, completing a logn level search.

[0230] See also Figure 6 , Figure 6 : is a test result diagram of the data processing method provided by the embodiment of the present application. The data processing method provided by the embodiment of the present application has the following advantages: 1. Parsing time is zero: due to the careful construction of the binary data structure, the data read from the disk is directly converted to a BinJson pointer and can be used immediately without parsing like BSON. According to the test results, DangoBinJson performs extremely well in the parsing stage, and the parsing time is about 0; 2. O(logn) level search time. When serializing DangoBinJson data, a key lookup table will be specially serialized for object data. This key lookup table is ordered, so binary search can be used to reduce the search time to O(logn), which is faster than R apidjson's search speed is 10 times faster. The key string will be added to the string pool. Only the identifier is stored in the key lookup table, so only a simple comparison of integers is required for binary search. 3. The serialized data size is about one-tenth of the text JSON format and one-third of the BSON format data. Due to the existence of the string pool, the text will be compressed into a variable-length integer, so it only needs to store the identifier, and for commonly used numbers, such as 0.0, 0.5, 1.0, 0, 1, -1, 2 and other 16 commonly used numbers, the type bit will be directly used for storage, and their size is only 1 byte (including type information); 4. Memory-friendly, no parsing is required, and no additional memory is required. After reading the data directly from the disk, the data can be used after the type conversion is obtained to obtain a pointer.

[0231] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0232] The following further describes an exemplary structure of the data processing device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2As shown, the software modules stored in the data processing device 255 of the memory 250 may include: a serialization module, which is used to serialize the first data in text format to obtain the second data in binary format; a pointer module 2552, which is used to perform pointer conversion processing on the second data in binary format to obtain multiple storage addresses corresponding to multiple fields one by one; a query module 2553, which is used to obtain a query request and extract the target field and the target data type of the target field from the query request; the query module 2553 is also used to query the corresponding field from the multiple storage addresses and the target storage address that matches the target field, and read the target storage address to obtain the query result.

[0233] In some embodiments, the second data includes a leading bit and subsequent bits, and the serialization module 2551 is also used for: when the data type of the first data is a first basic type, performing binary encoding processing on the first basic type to obtain the leading bit of the first data; wherein the first basic type includes at least one of the following: single-precision floating point, double-precision floating point, unsigned integer, signed integer, and the length of the leading bit is eight bits; performing binary encoding processing on the first data to obtain subsequent bits of the first data; wherein, when the first basic type is a single-precision floating point, the length of the subsequent bits is four bytes; when the first basic type is a double-precision floating point, the length of the subsequent bits is eight bytes; when the first basic type is an unsigned integer, the length of the subsequent bits is the number of bits of the unsigned integer; when the first basic type is a signed integer, the length of the subsequent bits is the number of bits of the signed integer.

[0234] In some embodiments, the second data includes a leading bit, and the serialization module 2551 is also used to: when the data type of the first data is a second basic type, perform binary encoding on the second basic type to obtain the leading bit of the first data, wherein the second basic type includes at least one of the following: Boolean type, empty object type, set number type.

[0235] In some embodiments, the second data includes a leading bit and subsequent bits, and the serialization module 2551 is also used for: when the data type of the first data is a third basic type, performing binary encoding processing on the third basic type to obtain a leading bit of the first data; wherein the third basic type includes a string type, and the length of the leading bit is three bits; performing binary encoding processing on the string offset of the first data to obtain a string offset in binary format; and determining subsequent bits of the first data according to the string offset in binary format.

[0236] In some embodiments, the serialization module 2551 is further used to: when the string corresponding to the first data is not found in the string pool, determine that the string offset is zero, and combine the string offset in binary format and the binary encoding of the string value of the first data into the subsequent bits of the first data; when the string corresponding to the first data is found in the string pool, determine the string offset in binary format as the subsequent bits of the first data.

[0237] In some embodiments, the second data includes a leading bit and subsequent bits, and the serialization module 2551 is also used for: when the data type of the first data is the fifth basic type, binary encoding the fifth basic type to obtain the leading bit of the first data; wherein the fifth basic type includes an object type, and the length of the leading bit is one bit; binary encoding the object data amount of the first data to obtain the object data amount in binary format; binary encoding the string pool offset of the first data to obtain the string pool offset in binary format; binary encoding the key lookup table of the first data to obtain the key lookup table in binary format; binary encoding the value lookup table of the first data to obtain the value lookup table in binary format; and combining the binary object data amount, the binary string pool offset, the binary key lookup table, the binary value lookup table, and the binary encoding of each data value of the first data into subsequent bits of the first data.

[0238] In some embodiments, the serialization module 2551 is also used to: obtain the key identifier of each object data in the first data; sort multiple key identifiers in ascending order, and obtain the identifier difference between each adjacent key identifier; obtain the maximum value from the key identifier sorted in the first place and at least one identifier difference; encode the number of bits of the maximum value as the table type bits of the key lookup table, and encode each identifier difference and the key identifier sorted in the first place as the key identifier bits of the key lookup table; wherein the length of the key identifier bits conforms to the number of bits corresponding to the table type bits of the key lookup table.

[0239] In some embodiments, the serialization module 2551 is also used to: obtain the data value of each object data in the first data; encode the number of bits of the largest data value into a table type bit in a binary format of a value lookup table, and encode the data value of each object data into a value bit in a binary format of the value lookup table; wherein the length of the value bit in the binary format corresponding to each data value conforms to the number of bits corresponding to the table type bit of the value lookup table.

[0240] In some embodiments, the data type of the string offset is a variable-length integer, and the serialization module 2551 is also used to: obtain the binary code of the string offset; parse the binary code to obtain at least one parsing result; perform the following processing on each parsing result according to the position order of at least one parsing result: when the parsing result does not include the last bit of the binary code of the string offset, use zero as the control bit of the corresponding parsing result, and use the parsing result as the storage bit; when the parsing result includes the last bit of the binary code of the string offset, use one as the control bit of the corresponding parsing result, and use the parsing result as the storage bit; combine the control bits and storage bits corresponding to the same parsing result into bytes of the corresponding parsing result, and combine the bytes of each parsing result into a string offset in binary format.

[0241] In some embodiments, the second data includes a leading bit and subsequent bits, and the serialization module 2551 is also used for: when the data type of the first data is a fourth basic type, performing binary encoding on the fourth basic type to obtain a leading bit of the first data; wherein the fourth basic type includes an array type, and the length of the leading bit is two bits; performing binary encoding on the array capacity of the first data to obtain an array capacity in a binary format; performing binary encoding on the array lookup table of the first data to obtain an array lookup table in a binary format; combining the array capacity in a binary format, the array lookup table in a binary format, and the binary encoding of the array value of the first data into subsequent bits of the first data.

[0242] In some embodiments, the serialization module 2551 is also used to: obtain the array offset of each array value in the first data; encode the number of bits of the maximum array offset as table type bits in the binary format of the array lookup table, and encode the array offset of each array value as offset bits in the binary format of the array lookup table; wherein the length of the offset bits in the binary format corresponding to each array value conforms to the number of bits corresponding to the table type bits in the binary format.

[0243] In some embodiments, the binary format string pool of the first data includes a leading bit and a subsequent bit. When the first data in text format is serialized to obtain the second data in binary format, the serialization module 2551 is also used to: binary-encode the string pool type to obtain the leading bit of the corresponding string pool type; obtain a skip offset for skipping the string pool of the first data, and binary-encode the skip offset to obtain the skip offset in binary format; binary-encode the key string pool of the first data to obtain a key string pool in binary format; binary-encode the value string pool of the first data to obtain a value string pool in binary format; and combine the skip offset in binary format, the key string pool in binary format, and the value string pool in binary format to form the subsequent bits of the corresponding string pool.

[0244] In some embodiments, the serialization module 2551 is also used to: obtain multiple key strings in the first data; perform binary encoding on the number of multiple key strings to obtain the number code of the key string pool; perform binary encoding on the key pool lookup table of the first data to obtain a key pool lookup table in binary format; perform binary encoding on each key string to obtain the binary code of each key string; wherein the binary code of the key string includes the binary code of the number of bits of the key string and the key string in binary format; the number code of the key string pool, the key pool lookup table in binary format, and the binary code of the key string form a key string pool in binary format.

[0245] In some embodiments, the serialization module 2551 is also used to: obtain the offset of each key string; encode the number of bits of the maximum offset as table type bits in the binary format of the key pool lookup table, and encode the offset of each key string as key offset bits in the binary format of the key pool lookup table; wherein the length of the key offset bits conforms to the number of bits corresponding to the table type bits in the binary format of the key pool lookup table.

[0246] In some embodiments, the serialization module 2551 is also used to: obtain multiple value strings in the first data; when there is no key string identical to the value string and the length of the value string is greater than a length threshold, determine the value string as a target value string; obtain the length of each target value string, and perform the following processing on each target value string: perform binary encoding processing on the number of bits of the target value string to obtain the binary encoding of the number of bits of the target value string, and perform binary encoding processing on the target value string to obtain a target value string in binary format; combine the binary encoding of the number of bits of the target value string and the target value string in binary format to form a binary encoding of the target value string; and concatenate the binary encodings of multiple target value strings into a value string pool in binary format.

[0247] In some embodiments, the query module 2553 is also used to: determine a target key string that matches the target field; perform a binary search in the key pool lookup table to obtain a first key identifier of the target key string in the key pool lookup table; perform a binary search in the key lookup table to obtain a second key identifier corresponding to the first key identifier in the key lookup table; and query the value lookup table for a target storage address corresponding to the value string corresponding to the second key identifier.

[0248] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method described above in the embodiment of the present application.

[0249] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data processing method provided by the present application embodiment, for example, Figures 3A-3D The data processing method is shown.

[0250] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0251] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0252] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0253] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0254] In summary, through the embodiment of the present application, the development data in text format is serialized to obtain the published data in binary format, wherein the published data in binary format includes a key lookup table; the published data in binary format is pointer converted to obtain multiple storage addresses corresponding to multiple fields, so that the data can be directly read during subsequent queries, reducing the parsing time and not requiring additional memory required for parsing, obtaining the query request, and extracting the target field from the query request; based on the key lookup table, the corresponding field is queried from multiple storage addresses for the target storage address that matches the target field, and the target storage address is read to obtain the query result. Querying based on the key lookup table can improve query efficiency and reduce query complexity.

[0255] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: When the data type of the first data is a fourth basic type, binary encoding is performed on the fourth basic type to obtain a guide bit of the first data, wherein the fourth basic type includes an array type, and the length of the guide bit is two bits; Performing binary encoding processing on the array capacity of the first data to obtain the array capacity in binary format, and performing binary encoding processing on the array lookup table of the first data to obtain the array lookup table in binary format; combining the array capacity in the binary format, the array lookup table in the binary format, and the binary encoding of the array value of the first data into subsequent bits of the first data; Combining the leading bit of the first data and the subsequent bits of the first data into second data in binary format, wherein the second data in binary format includes an ordered key lookup table; Obtaining a query request, and extracting a target field from the query request; Determine a value query result matching the target field based on the ordered key lookup table, and perform a read process on a target storage address corresponding to the value query result to obtain a value read result; The target storage address is obtained by performing pointer conversion processing on the second data in binary format.

2. The method according to claim 1, characterized in that The method further comprises: When the data type of the first data is a first basic type, performing binary encoding processing on the first basic type to obtain a leading bit of the first data; Wherein, the first basic type includes at least one of the following: single-precision floating point type, double-precision floating point type, unsigned integer type, signed integer type, and the length of the guide bit is eight bits; Performing binary encoding processing on the first data to obtain subsequent bits of the first data; Wherein, when the first basic type is a single-precision floating point type, the length of the subsequent bits is four bytes; when the first basic type is a double-precision floating point type, the length of the subsequent bits is eight bytes; when the first basic type is an unsigned integer type, the length of the subsequent bits is the number of bits of the unsigned integer type; when the first basic type is a signed integer type, the length of the subsequent bits is the number of bits of the signed integer type; The leading bit of the first data and the subsequent bits of the first data are combined into second data in a binary format.

3. The method according to claim 1, characterized in that The method further comprises: When the data type of the first data is a second basic type, binary encoding is performed on the second basic type to obtain a leading bit of the first data, wherein the second basic type includes at least one of the following: a Boolean type, an empty object type, and a set digital type; The leading bit of the first data is used as the second data.

4. The method according to claim 1, characterized in that: The method further comprises: When the data type of the first data is a third basic type, performing binary encoding processing on the third basic type to obtain a leading bit of the first data; Wherein, the third basic type includes a character string type, and the length of the guide bit is three bits; Performing binary encoding processing on the string offset of the first data to obtain the string offset in binary format; Determining subsequent bits of the first data according to the string offset in the binary format; The leading bit of the first data and the subsequent bits of the first data are combined into second data in a binary format.

5. The method according to claim 4, characterized in that The determining, according to the string offset in the binary format, subsequent bits of the first data includes: When a string corresponding to the first data is not found in the string pool, determining that the string offset is zero, and combining the string offset in binary format and the binary encoding of the string value of the first data into subsequent bits of the first data; When a character string corresponding to the first data is found in the character string pool, the character string offset in the binary format is determined as a subsequent bit of the first data.

6. The method according to any one of claims 4-5, characterized in that: The data type of the string offset is a variable-length integer, and the string offset of the first data is binary-encoded to obtain a string offset in binary format, including: Get the binary code of the string offset; Parsing the binary code to obtain at least one parsing result; The following processing is performed for each parsing result according to the position order of at least one parsing result: When the parsing result does not include the last bit of the binary code of the string offset, using zero as a control bit corresponding to the parsing result, and using the parsing result as a storage bit; When the parsing result includes the last bit of the binary code of the string offset, one is used as a control bit corresponding to the parsing result, and the parsing result is used as the storage bit; The control bits and storage bits corresponding to the same parsing result are combined into bytes corresponding to the parsing result, and each byte of the parsing result is combined into a string offset in the binary format.

7. The method according to claim 1, characterized in that The step of performing binary encoding processing on the array lookup table of the first data to obtain an array lookup table in a binary format includes: Obtaining an array offset of each of the array values ​​in the first data; Encoding the number of bits of the maximum array offset as table type bits of the array lookup table in binary format, and encoding the array offset of each of the array values ​​as offset bits of the array lookup table in binary format; The length of the offset bits in the binary format corresponding to each of the array values ​​conforms to the number of bits corresponding to the table type bits in the binary format.

8. The method according to claim 1, characterized in that The ordered key lookup table includes a key lookup table, and the method further includes: When the data type of the first data is the fifth basic type, performing binary encoding processing on the fifth basic type to obtain a leading bit of the first data; Wherein, the fifth basic type includes an object type, and the length of the guide bit is one bit; Performing binary encoding processing on the object data volume of the first data to obtain the object data volume in binary format; Performing binary encoding processing on the string pool offset of the first data to obtain a string pool offset in a binary format; Performing binary encoding processing on the key lookup table of the first data to obtain a key lookup table in binary format; Performing binary encoding processing on the value lookup table of the first data to obtain a value lookup table in binary format; Combine the amount of object data in the binary format, the string pool offset in the binary format, the key lookup table in the binary format, the value lookup table in the binary format, and the binary encoding of each data value of the first data into subsequent bits of the first data; The leading bit of the first data and the subsequent bits of the first data are combined into second data in a binary format.

9. The method according to claim 8, characterized in that The key lookup table in binary format includes a table type bit in binary format of the key lookup table and a key identification bit in binary format of the key lookup table. The key lookup table of the first data is binary-encoded to obtain the key lookup table in binary format, including: Obtaining a key identifier for each of the object data in the first data; Sorting the plurality of key identifiers in ascending order, and obtaining an identifier difference between each key identifier and the first key identifier in the sorting order; Obtain a maximum value from the key identifier sorted first and at least one of the identifier differences; Encode the number of bits of the maximum value as table type bits in a binary format of the key lookup table, and encode each of the identification differences and the key identification ranked first as key identification bits in a binary format of the key lookup table; The length of the key identification bit matches the number of bits corresponding to the table type bit of the key lookup table.

10. The method according to claim 8, characterized in that The value lookup table in binary format includes a table type bit in binary format of the value lookup table and a value bit in binary format of the value lookup table. The binary encoding process is performed on the value lookup table of the first data to obtain the value lookup table in binary format, including: Acquire a data value of each object data in the first data; Encoding the number of bits of the maximum data value into table type bits in a binary format of the value lookup table, and encoding the data value of each of the object data into value bits in a binary format of the value lookup table; The length of the value bits in the binary format corresponding to each of the data values ​​conforms to the number of bits corresponding to the table type bits of the value lookup table.

11. The method according to claim 1, characterized in that: The string pool in binary format of the first data includes a leading bit and a subsequent bit, and the method further includes: Binary-encode the string pool type to obtain a guide bit corresponding to the string pool type; Obtaining a skip offset of a string pool for skipping the first data, and performing binary encoding processing on the skip offset to obtain a skip offset in a binary format; Performing binary encoding processing on the key string pool of the first data to obtain a key string pool in binary format; Performing binary encoding processing on the value string pool of the first data to obtain a value string pool in binary format; The skip offset in binary format, the key string pool in binary format, and the value string pool in binary format are combined into subsequent bits corresponding to the string pool.

12. The method according to claim 11, characterized in that The binary encoding process is performed on the key string pool of the first data to obtain a key string pool in binary format, including: Obtain multiple key character strings in the first data; Performing binary encoding processing on the number of the plurality of key character strings to obtain a number code of the key character string pool; Performing binary encoding processing on the key pool lookup table of the first data to obtain a key pool lookup table in binary format; Performing binary encoding processing on each of the key strings to obtain a binary encoding of each of the key strings; The binary encoding of the key string includes the binary encoding of the number of bits of the key string and the key string in binary format; The number of the key string pool, the key pool lookup table in binary format, and the binary encoding of the key string are combined to form the key string pool in binary format.

13. The method according to claim 12, characterized in that The ordered key lookup table includes a key pool lookup table, and the key pool lookup table of the first data is binary-encoded to obtain a key pool lookup table in binary format, including: Get the offset of each of the key strings; Encode the maximum offset bit number as a table type bit in a binary format of the key pool lookup table, and encode the offset of each key string as a key offset bit in a binary format of the key pool lookup table; The length of the key offset bits in the binary format of the key pool lookup table conforms to the number of bits corresponding to the table type bits in the binary format of the key pool lookup table.

14. The method according to claim 11, characterized in that The binary encoding process is performed on the value string pool of the first data to obtain a value string pool in a binary format, including: Obtain multiple value strings in the first data; When there is no key string identical to the value string, and the length of the value string is greater than a length threshold, determining the value string as a target value string; The length of each target value string is obtained, and the following processing is performed on each target value string: Performing binary encoding processing on the number of digits of the target value string to obtain a binary encoding of the number of digits of the target value string, and performing binary encoding processing on the target value string to obtain a target value string in binary format; Combining the binary code of the target value string with the binary code of the number of bits of the target value string and the target value string in binary format to form the binary code of the target value string; Binary codes of the plurality of target value strings are concatenated into a value string pool in the binary format.

15. The method according to any one of claims 11 to 14, characterized in that The ordered key lookup table includes a key lookup table and a key pool lookup table, and determining a value query result matching a target field based on the ordered key lookup table includes: determining a target key string that matches the target field; Perform a binary search in the key pool lookup table to obtain a first key identifier of the target key string in the key pool lookup table; Perform a binary search in the key lookup table to obtain a second key identifier corresponding to the first key identifier in the key lookup table; A value string corresponding to the second key identifier is determined in the value lookup table as the value query result.

16. A data processing device, characterized in that: The device comprises: A serialization module, used for, when the data type of the first data is a fourth basic type, performing binary encoding processing on the fourth basic type to obtain a leading bit of the first data, wherein the fourth basic type includes an array type and the length of the leading bit is two bits; performing binary encoding processing on the array capacity of the first data to obtain an array capacity in a binary format, and performing binary encoding processing on the array lookup table of the first data to obtain an array lookup table in a binary format; combining the array capacity in the binary format, the array lookup table in the binary format, and the binary encoding of the array value of the first data into subsequent bits of the first data; forming the leading bit of the first data and the subsequent bits of the first data into second data in a binary format, wherein the second data in the binary format includes an ordered key lookup table; A query module, used to obtain a query request and extract a target field from the query request; The query module is also used to determine a value query result that matches the target field based on the ordered key lookup table, and read the target storage address corresponding to the value query result to obtain a value reading result; wherein the target storage address is obtained by performing pointer conversion processing on the second data in binary format.

17. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the data processing method according to any one of claims 1 to 15 when executing the executable instructions stored in the memory.

18. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 15 is implemented.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the data processing method according to any one of claims 1 to 15 is implemented.

Images