A new data protocol writing method and system

Abstract

The application relates to the technical field of database data transmission and data writing, in particular to a new data protocol writing method and system, which is characterized in that: a native short connection transmission protocol is encapsulated on the upper layer of a TCP protocol; a mapping structure object is constructed for each data table in a non-relational database in a server; the mapping structure object is used for data filling on a client side, is serialized, and the serialized data is transmitted to the server through the native short connection transmission protocol; the server uses the same mapping structure object to perform deserialization and analyzes the deserialized data into corresponding data for storage; the data for storage is obtained through analysis and a writing operation is performed; the client actively pulls the latest mapping structure object from the server according to a preset period, verifies and updates the local mapping structure object, and the application can meet the requirements of reducing the transmission data volume, standardizing the data analysis process and greatly improving the database data writing efficiency.

Description

Technical Field

[0001] This invention relates to the field of database data transmission and data writing technology, and in particular to a new data protocol writing method and system. Background Technology

[0002] Currently, various NoSQL databases widely support multi-protocol data writing, with HTTP-based data writing being the industry mainstream. While HTTP, as a general application layer transport protocol, boasts strong versatility and wide adaptability, making it suitable for most network data transmission scenarios, it exhibits significant technical limitations in database-specific data writing scenarios.

[0003] On the one hand, the HTTP protocol message structure is fixed, and a large number of common redundant request headers are automatically carried during transmission, including invalid fields such as request type, return type, data identifier, browser identifier, and cross-domain parameters. Such header information has no practical use for writing data to non-relational databases, which greatly increases the volume of network data transmission, occupies network bandwidth resources, and leads to increased transmission latency in high-frequency write scenarios.

[0004] On the other hand, existing HTTP data transmission lacks a fixed data format standard. Client-transmitted data is irregular in format and fields are unordered. Upon receiving data, the server needs to dynamically identify field names, data types, field lengths, and other information to perform dynamic parsing and format conversion. Because the data format cannot be predicted in advance, each data write requires significant computing power for dynamic parsing, resulting in lengthy parsing times. This severely limits the overall write performance of NoSQL databases and fails to meet the demands of high concurrency, high throughput, and low latency database write operations. Summary of the Invention

[0005] To overcome the problems of large transmission redundancy and low data parsing efficiency in existing database writing technologies, the purpose of this invention is to provide a new data protocol writing method and system that can significantly improve database data writing efficiency by simplifying the amount of transmitted data and standardizing the data parsing process.

[0006] This invention is implemented using the following scheme: A new data protocol writing method, the steps of which are as follows: Step 1: Based on the underlying TCP protocol, a native short connection transport protocol is encapsulated on top of it. The native short connection transport protocol removes all invalid request headers of the HTTP protocol and only configures fixed necessary identification information that can be recognized by non-relational databases. Step 2: For each data table in the non-relational database on the server, construct a mapping structure object. The mapping structure object records all fields of the corresponding data table. Simultaneously send the mapping structure object to the client and the server and maintain data consistency between the two ends. Step 3: The mapping structure object is used on the client side to populate data and serialize it. The serialized data is transmitted from the client side to the server side through the native short connection transmission protocol. The server side uses the same mapping structure object to deserialize and parse it into the corresponding data to be inserted into the database. The data to be inserted into the database is then parsed and written.

[0007] Step 4: The client actively retrieves the latest mapping structure object from the server according to a preset period, verifies and updates the local mapping structure object.

[0008] Furthermore, in the native short connection transmission protocol, each request needs to transmit data according to the mapping structure object already recorded by the client. If the data is not in the current mapping structure object, it is uniformly set to a map object. The map object is used to store dynamic fields that are not predefined in the field list, and finally the serialized data is transmitted. After the server receives the information, it first deserializes the mapping structure data to obtain the data to be put into the database. If the data in the corresponding map object is not empty, it performs the mapping of the deserialized data to be put into the database again.

[0009] Furthermore, the mapping structure object is a metadata model used to describe the database table structure, which is kept synchronized between the client and the server. The mapping structure object includes table name, field list, field index, version number and special mapping fields. The special mapping field is a Map object used to temporarily store dynamic fields that are not predefined in the field list.

[0010] Furthermore, the client's serialization process is as follows: during serialization, the client packages the data into a binary stream with a fixed format according to the order of the field list; the server performs deserialization according to the same field list order, and dynamic data not in the field list is uniformly placed into a special mapping field and serialized to the end of the data with a special marker.

[0011] Furthermore, the fixed header of the native short connection transmission protocol includes four fixed-length identification fields: 4 bytes for total message length, 1 byte for operation command type, 2 bytes for table mapping ID, and 1 byte for serialization version number. The native short connection transmission protocol adopts a short connection mechanism, which disconnects immediately after a single TCP request is completed and does not continuously maintain session state.

[0012] A novel data protocol writing system, comprising: a protocol generation module, a mapping module, a data transmission writing module, and a verification and update module; The protocol generation module is used to encapsulate a native short connection transmission protocol on top of the underlying TCP protocol. The native short connection transmission protocol removes all invalid request headers of the HTTP protocol and only configures fixed necessary identification information that can be recognized by non-relational databases. The mapping module is used to construct a mapping structure object for each data table in the database. The mapping structure object records all fields of the corresponding data table and synchronously sends the mapping structure object to the client and the server to maintain data consistency between the two ends. The data transmission writing module is used by the client to fill data and serialize the mapping structure object. The serialized data is transmitted from the server to the client through the native short connection transmission protocol. The server uses the same mapping structure object to deserialize and parse it into the corresponding data to be put into the database. The data to be put into the database is then parsed and the writing operation is performed. The verification and update module is used by the client to actively pull the latest mapping structure object from the server according to a preset period, verify and update the local mapping structure object.

[0013] Furthermore, in the native short connection transmission protocol, each request needs to transmit data according to the mapping structure object already recorded by the client. If the data is not in the current mapping structure object, it is uniformly set to a map object. The map object is used to store dynamic fields that are not predefined in the field list, and finally the serialized data is transmitted. After the server receives the information, it first deserializes the mapping structure data to obtain the data to be put into the database. If the data in the corresponding map object is not empty, it performs the mapping of the deserialized data to be put into the database again.

[0014] Furthermore, the mapping structure object is a metadata model used to describe the database table structure and is kept synchronized between the client and the server. The mapping structure object includes table name, field list, field index, version number and special mapping fields. The special mapping field is a Map object used to temporarily store dynamic fields that are not predefined in the field list.

[0015] Furthermore, the client's serialization process is as follows: during serialization, the client packages the data into a fixed-format binary stream according to the order of the field list; the server performs deserialization according to the same field list order, and dynamic data not in the field list is uniformly placed into a special mapping field and serialized to the end of the data with a special marker.

[0016] Furthermore, the fixed header of the native short connection transmission protocol includes four fixed-length identification fields: 4 bytes for total message length, 1 byte for operation command type, 2 bytes for table mapping ID, and 1 byte for serialization version number. The native short connection transmission protocol adopts a short connection mechanism, which disconnects immediately after a single TCP request is completed and does not continuously maintain session state.

[0017] The beneficial effects of this invention are as follows: This invention provides a new data protocol writing method and system that significantly reduces data transmission redundancy. Based on a custom TCP native protocol, this invention completely eliminates redundant HTTP request headers, general identifiers and other invalid data, retaining only the fixed header information necessary for database writing. This greatly reduces the message size of a single data transmission, saves network bandwidth, reduces network transmission latency, and is suitable for high-concurrency writing scenarios. This invention achieves data format standardization and ordering by pre-setting a database table mapping structure object. The client serializes the data into a binary stream according to a fixed field order. The server does not need to dynamically identify field information; it only needs to complete the deserialization according to a fixed offset. This completely solves the problem of long time consumption in the dynamic parsing of traditional unordered data and greatly improves the data parsing speed. This invention achieves standardized transmission and parsing of regular fields by mapping structure objects, while adding special Map objects to store dynamic unknown fields. This ensures writing efficiency while being compatible with scenarios of adding dynamic fields and writing temporary fields in the database, making it more adaptable. The Native short connection transmission protocol of this invention adopts a short connection mechanism, which disconnects immediately after a single request is completed, eliminating the need for continuous maintenance of session state, reducing server connection resource consumption, improving database concurrency carrying capacity, and optimizing the overall write performance of the database system. This invention designs a bidirectional synchronization mechanism between the client and server mapping structure objects to ensure complete consistency between the encoding and decoding models, effectively avoiding data parsing errors and write failures caused by model inconsistency, and improving the accuracy and stability of data writing. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural block diagram of the system of the present invention. Detailed Implementation

[0019] The invention will now be further described with reference to the accompanying drawings.

[0020] See Figure 1 A new data protocol writing method, the steps of which are as follows: A new data protocol writing method, the steps of which are as follows: Step 1: Based on the underlying TCP protocol, a native short connection transport protocol is encapsulated on top of it. The native short connection transport protocol removes all invalid request headers of the HTTP protocol and only configures fixed necessary identification information that can be recognized by non-relational databases. Step 2: For each data table in the non-relational database on the server, construct a mapping structure object. The mapping structure object records all fields of the corresponding data table. Simultaneously send the mapping structure object to the client and the server and maintain data consistency between the two ends. Step 3: The mapping structure object is used on the client side to populate data and serialize it. The serialized data is transmitted from the server to the client side through the native short connection transmission protocol. The server side uses the same mapping structure object to deserialize and parse it into the corresponding data to be inserted into the database. The data to be inserted into the database is then parsed and written. Step 4: The client actively retrieves the latest mapping structure object from the server according to a preset period, verifies and updates the local mapping structure object.

[0021] The present invention will be further described below with reference to a specific embodiment: A novel data protocol writing method, comprising the following steps: Step 1: Client initializes the mapping structure object Contents of the mapping structure object: Table name: "user" Field list: ["id" (integer), "name" (string, maximum 50), "age" (integer)] Field index: {"id":0, "name":1, "age":2} Version number: 1 Additional Map: Empty Step 2: Client constructs request data Command type: 0x01 (Write operation) Table mapping ID: Assuming it is 0x0001 Constructing the data body: Serialize the data sequentially according to the field list: OID: Integer value 1001 → 4 bytes [0x00, 0x00, 0x03, 0xE9] oname: string "Zhang San" → 2 bytes long 0x0006 + 6 bytes of UTF-8 encoding [E5 BC A0E4 B8 89] oage: Integer value 28 → 4 bytes [0x00,0x00,0x00,0x1C] Handling the extra field "remark": Construct a temporary Map with the key "remark" and the value "VIP". Serialize the Map: First, write the number of elements 1 (2 bytes [0x00, 0x01]), then write the key length 2 ([0x00, 0x02]) + the key content "re". The actual length of "remark" should be 6 bytes, but here's a correction: the key "remark" is 6 bytes long, and the content is the ASCII code of "remark"; the value "VIP" is 3 bytes long, and the content is "VIP". Actual serialization result: 2 bytes long 0x0006 + "remark" + 2 bytes long 0x0003 + "VIP".

[0022] The serialization results of the above fixed fields are concatenated with the serialization results of the Map to form a complete data body.

[0023] The total message length is calculated as follows: (Header fixed at 7 bytes (total length 4 + command 1 + mapping ID2)) + data body length.

[0024] Assemble the native short connection transport protocol message: first write the total length of 4 bytes, then write the command 0x01, then write the mapping ID 0x0001, and finally write the data body.

[0025] Step 3: The client sends messages via a short TCP connection. The client establishes a TCP connection to the server (IP: 192.168.1.100, port: 9999), sends the above binary message, and then closes the connection.

[0026] Step 4: Server receives and processes the data. The server listens on the same port and, upon receiving a message, first parses a fixed header: total length, command type, and mapping ID. Based on the mapping ID, it locates the locally stored mapping structure object (version 1).

[0027] Parsing the data body: First, read the fixed fields in the order of the field list of the mapped structure object: Read 4 bytes -> id=1001 Read 2 bytes of length L -> Read another L bytes -> name="Zhang San" Read 4 bytes -> age=28 Check if there are any remaining bytes in the data body (because it contains an additional Map area). If there are remaining bytes, parse it according to the Map format: read 2-byte elements N times, loop N times: read the key length KL, read KL bytes to get the key name, read the value length VL, read VL bytes to get the value content. In this example, the key is "remark" and the value is "VIP".

[0028] The server assembles the parsed data into internal record objects and stores the additional fields as dynamic fields in the database's JSON columns or extended columns (such as fields that support schema-less).

[0029] Step 5: Periodically synchronize and update the mapping structure Suppose the server administrator adds a field "email" (string) to the user table and upgrades the version number of the mapping structure object to 2. Every 30 seconds, the client calls the synchronization interface (sending a version number request via a native short-connection transport protocol). The server compares the version numbers; if the client's version number 1 is less than the server's version number 2, it returns the new mapping structure object (with "email" added to the field list). The client updates its local mapping structure object, and subsequent data writes will be serialized according to the new structure.

[0030] The above examples demonstrate that the native short-connection transmission protocol avoids HTTP header redundancy and reduces data volume; fixed field sequential serialization allows parsing to be read directly by offset, eliminating the need to traverse key names; and dynamic fields are extended through Map area compatibility, resulting in a significant improvement in overall write efficiency compared to the HTTP+JSON approach.

[0031] See Figure 2 A new data protocol writing system, the system comprising: a protocol generation module, a mapping module, a data transmission writing module, and a verification and update module; The protocol generation module is used to encapsulate a native short connection transmission protocol on top of the underlying TCP protocol. The native short connection transmission protocol removes all invalid request headers of the HTTP protocol and only configures fixed necessary identification information that can be recognized by non-relational databases. The mapping module is used to construct a mapping structure object for each data table in the database. The mapping structure object records all fields of the corresponding data table and synchronously sends the mapping structure object to the client and the server to maintain data consistency between the two ends. The data transmission writing module is used by the client to fill data and serialize the mapping structure object. The serialized data is transmitted from the server to the client through the native short connection transmission protocol. The server uses the same mapping structure object to deserialize and parse it into the corresponding data to be put into the database. The data to be put into the database is then parsed and the writing operation is performed. The verification and update module is used by the client to actively pull the latest mapping structure object from the server according to a preset period, verify and update the local mapping structure object.

[0032] In one embodiment of the present invention, in the native short connection transmission protocol, each request needs to transmit data according to the mapping structure object already recorded by the client. If the data is not in the current mapping structure object, it is uniformly set to a map object. The map object is used to store dynamic fields that are not predefined in the field list, and finally the serialized data is transmitted. After receiving the information, the server first deserializes the mapping structure data to obtain the data to be put into the database. If the data in the corresponding map object is not empty, the mapping of the deserialized data to be put into the database is performed again. The unique features of the native short-connection transport protocol are reflected in the following aspects: 1. The native short connection transport protocol is built directly on top of the TCP protocol, omitting the request line, header fields (such as Content-Type, User-Agent, Accept, etc.) and additional chunking or keep-alive mechanisms found in the HTTP protocol.

[0033] 2. The native short-connection transport protocol has a fixed message structure, consisting of a "fixed header + data body". The fixed header only contains the necessary fields: Total length (4 bytes): The total length of the message.

[0034] Command type (1 byte): such as write, query, update, etc.

[0035] Table mapping ID (2 bytes): Identifies the ID of the mapping structure object used.

[0036] Serialization version (1 byte): The version number of the corresponding mapping structure object.

[0037] 3. A TCP connection is established for each request and closed immediately after data is sent. No session state is maintained, thus avoiding the overhead of long HTTP connections.

[0038] 4. The data body of the native short-connection transmission protocol uses binary encoding with a predefined field order, eliminating the need to transmit field names or data type tags. For example, a record with two fields (id (integer) and name (string)) will be serialized as a 4-byte id + 2-byte length + name content, instead of the key-name format used in JSON or XML.

[0039] 5. If the data contains fields not defined in the mapping structure, these fields are packaged separately into the Map area at the end of the data body. The Map area is encoded in the format of "number of elements + key length of each element + key content + value length + value content", thereby ensuring the extensibility of the protocol.

[0040] These special designs enable the native short-connection transport protocol to reduce a significant amount of redundant header and field name transmission compared to the HTTP protocol. Furthermore, during parsing, there is no need to dynamically look up field names; it can simply read by offset, thus improving write efficiency and parsing speed.

[0041] In one embodiment of the present invention, the mapping structure object is a metadata model for describing the database table structure and is kept synchronized between the client and the server. The mapping structure object includes table name, field list, field index, version number and special mapping field. The table name (tableName) is the name of the database table that identifies the mapped object.

[0042] The field list `fieldList` is an ordered list that records the name, data type (such as string, integer, floating-point, timestamp, etc.), length, and nullability of all fields in the table.

[0043] The fieldIndexMap is a mapping from field names to serialization order indices, used to quickly locate field positions.

[0044] The version number identifies the current version of the mapped structure object and is used to determine whether an update is needed when synchronizing between the client and the server.

[0045] The special mapping field extraFieldMap is a Map object used to temporarily store dynamic fields that are not predefined in the field list.

[0046] During serialization, the client packages the data into a fixed-format binary stream according to the field list order; the server performs deserialization based on the same field list order. Dynamic data not in the field list is placed into a special mapping field extraFieldMap and serialized to the end of the data with a special marker.

[0047] As an extension of the present invention, the above-described scheme can also be used in data transmission from the server to the client. Those skilled in the art will know how to adjust it to apply the above-described scheme in data transmission from the server to the client.

[0048] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A novel data protocol writing method, characterized in that, The method steps are as follows: Step 1: Based on the underlying TCP protocol, a native short connection transport protocol is encapsulated on top of it. The native short connection transport protocol removes all invalid request headers of the HTTP protocol and only configures fixed necessary identification information that can be recognized by non-relational databases. Step 2: For each data table in the non-relational database on the server, construct a mapping structure object. The mapping structure object records all fields of the corresponding data table. Simultaneously send the mapping structure object to the client and the server and maintain data consistency between the two ends. Step 3: The mapping structure object is used on the client side to populate data and serialize it. The serialized data is transmitted from the server to the client side through the native short connection transmission protocol. The server side uses the same mapping structure object to deserialize and parse it into the corresponding data to be inserted into the database. The data to be inserted into the database is then parsed and written. Step 4: The client actively retrieves the latest mapping structure object from the server according to a preset period, verifies and updates the local mapping structure object.

2. The novel data protocol writing method according to claim 1, characterized in that, In the native short-connection transport protocol, each request needs to transmit data according to the mapping structure object already recorded by the client. If the data is not in the current mapping structure object, it is uniformly set to a map object. The map object is used to store dynamic fields that are not predefined in the field list, and finally the serialized data is transmitted. After receiving the information, the server first deserializes the mapping structure data to obtain the data to be put into the database. If the data in the corresponding map object is not empty, the deserialized data to be put into the database is mapped again.

3. The novel data protocol writing method according to claim 1, characterized in that, The mapping structure object is a metadata model used to describe the database table structure and is kept synchronized between the client and the server. The mapping structure object includes table name, field list, field index, version number and special mapping fields. The special mapping field is a Map object used to temporarily store dynamic fields that are not predefined in the field list.

4. A novel data protocol writing method according to claim 3, characterized in that, The client-side serialization process is as follows: During serialization, the client packages the data into a fixed-format binary stream according to the order of the field list; the server performs deserialization according to the same field list order, and dynamic data not in the field list is uniformly placed into a special mapping field and serialized to the end of the data with a special marker.

5. A novel data protocol writing method according to claim 1, characterized in that, The fixed header of the native short connection transmission protocol includes four fixed-length identification fields: 4 bytes for total message length, 1 byte for operation command type, 2 bytes for table mapping ID, and 1 byte for serialization version number. The native short connection transmission protocol adopts a short connection mechanism, which disconnects immediately after a single TCP request is completed and does not continuously maintain session state.

6. A new data protocol writing system, characterized in that, The system includes: a protocol generation module, a mapping module, a data transmission writing module, and a verification and update module; The protocol generation module is used to encapsulate a native short connection transmission protocol on top of the underlying TCP protocol. The native short connection transmission protocol removes all invalid request headers of the HTTP protocol and only configures fixed necessary identification information that can be recognized by non-relational databases. The mapping module is used to construct a mapping structure object for each data table in the database. The mapping structure object records all fields of the corresponding data table and synchronously sends the mapping structure object to the client and the server to maintain data consistency between the two ends. The data transmission writing module is used by the client to fill data and serialize the mapping structure object. The serialized data is transmitted from the server to the client through the native short connection transmission protocol. The server uses the same mapping structure object to deserialize and parse it into the corresponding data to be put into the database. The data to be put into the database is then parsed and the writing operation is performed. The verification and update module is used by the client to actively pull the latest mapping structure object from the server according to a preset period, verify and update the local mapping structure object.

7. A novel data protocol writing system according to claim 6, characterized in that, In the native short-connection transport protocol, each request needs to transmit data according to the mapping structure object already recorded by the client. If the data is not in the current mapping structure object, it is uniformly set to a map object. The map object is used to store dynamic fields that are not predefined in the field list, and finally the serialized data is transmitted. After receiving the information, the server first deserializes the mapping structure data to obtain the data to be put into the database. If the data in the corresponding map object is not empty, the deserialized data to be put into the database is mapped again.

8. A novel data protocol writing system according to claim 6, characterized in that, A mapping structure object is a metadata model used to describe the structure of a database table and is kept synchronized between the client and the server. The mapping structure object includes the table name, field list, field index, version number, and special mapping fields. The special mapping field is a Map object used to temporarily store dynamic fields that are not predefined in the field list.

9. A novel data protocol writing system according to claim 6, characterized in that, The client-side serialization process is as follows: During serialization, the client packages the data into a fixed-format binary stream according to the order of the field list; the server performs deserialization according to the same field list order, and dynamic data not in the field list is uniformly placed into a special mapping field and serialized to the end of the data with a special marker.

10. A novel data protocol writing system according to claim 6, characterized in that, The fixed header of the native short connection transmission protocol includes four fixed-length identification fields: 4 bytes for total message length, 1 byte for operation command type, 2 bytes for table mapping ID, and 1 byte for serialization version number. The native short connection transmission protocol adopts a short connection mechanism, which disconnects immediately after a single TCP request is completed and does not continuously maintain session state.

Images