Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining to-be-processed data, wherein the to-be-processed data comprises at least one data field; performing serialization processing on each data field to generate a field byte sequence of each data field so as to obtain a data byte sequence of the to-be-processed data; generating field metadata of each data field according to the field byte sequence of each data field to obtain a metadata mapping file of the to-be-processed data; serializing the metadata mapping file of the to-be-processed data to obtain a metadata byte sequence; and placing the metadata byte sequence in front of the data byte sequence to obtain a serialization result of the to-be-processed data. According to the embodiment, data serialization and analysis based on dynamic metadata driving are achieved, the dynamic data structure can be processed, limitation to the data structure is reduced, the processing mode is more flexible, implementation is easy, and the processing efficiency is high.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for data processing. Background Technology

[0002] In data transmission and communication, to ensure more efficient and reliable data transmission, the data sender typically serializes the data before transmission, and the data receiver deserializes the received serialized data to parse it. Currently, common data serialization and communication frameworks can be used for data serialization. However, these frameworks have limitations in handling dynamic data structures, lack flexibility, and suffer from complex implementation and poor timeliness. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data processing method and apparatus that can perform data serialization and parsing based on dynamic metadata, process dynamic data structures, reduce the limitations of data structures, make the processing method more flexible, and are simple to implement with high processing efficiency.

[0004] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising: Obtain data to be processed, wherein the data to be processed includes at least one data field; Each data field is serialized to generate a field byte sequence for each data field, thereby obtaining the data byte sequence of the data to be processed; Generate the field metadata for each data field based on the field byte sequence of each data field to obtain the metadata mapping file of the data to be processed; The metadata mapping file of the data to be processed is serialized to obtain a metadata byte sequence; The metadata byte sequence is placed before the data byte sequence to obtain the serialization result of the data to be processed.

[0005] Optionally, each data field has a field type; serializing each data field to generate a field byte sequence includes: for each data field, serializing the data field according to the field type of the data field to generate a field byte sequence of the data field.

[0006] Optionally, if the field type is a string type, the data field is serialized according to the field type of the data field to generate the field byte sequence of the data field, including: generating the field byte sequence of the data field according to the field number, field type, string length and string itself of the data field.

[0007] Optionally, when the field type is an integer type, the data field is serialized according to the field type to generate the field byte sequence of the data field, including: encoding the data field using an encoding algorithm corresponding to the integer type to obtain an encoded value; and generating the field byte sequence of the data field according to the field number, field type and encoded value of the data field.

[0008] Optionally, when the field type is a long integer, the data field is serialized according to its field type to generate a field byte sequence, including: encoding the data field using an encoding algorithm corresponding to the long integer type to obtain an encoded value; and generating the field byte sequence of the data field according to the field number, field type, and encoded value.

[0009] Optionally, generating field metadata for each data field based on the field byte sequence of each data field includes: for each data field, obtaining the field type and length of the field byte sequence based on the field byte sequence of the data field, and generating field metadata for the data field based on the field type, the length of the field byte sequence, and the starting position of the field byte sequence in the data byte sequence of the data to be processed.

[0010] Optionally, before serializing each data field to generate a field byte sequence for each data field, the method further includes: creating a first byte buffer to store the field byte sequence for each data field; serializing each data field to generate a field byte sequence for each data field to obtain the data byte sequence of the data to be processed, including: serializing each data field to generate a field byte sequence for each data field, and storing the field byte sequence of each data field in the first byte buffer; in response to the completion of serialization processing for all data fields of the data to be processed, using the data in the first byte buffer as the data byte sequence of the data to be processed; and the starting position of the field byte sequence in the data byte sequence of the data to be processed is obtained based on the starting offset of the field byte sequence in the first byte buffer.

[0011] Optionally, placing the metadata byte sequence before the data byte sequence to obtain the serialization result of the data to be processed includes: placing the metadata byte sequence before the first byte buffer corresponding to the data byte sequence to obtain the serialization result of the data to be processed.

[0012] Optionally, the metadata mapping file stores the mapping relationship between the field number of each data field and the field metadata; serializing the metadata mapping file of the data to be processed to obtain a metadata byte sequence includes: creating a second byte buffer to store the metadata byte sequence; setting the size of the second byte buffer according to the size of the metadata mapping file of the data to be processed; traversing the metadata mapping file and writing the mapping relationship between the field number of each data field and the field metadata into the second byte buffer according to a specified format to obtain the metadata byte sequence.

[0013] Optionally, the mapping relationship between the field number and the field metadata of each data field is written to the second byte buffer in a specified format, including: writing the mapping relationship between the field number and the field metadata of each data field to the second byte buffer in a specified order and with a specified field type.

[0014] Optionally, the serialization result of the data to be processed includes the metadata byte sequence, the data byte sequence, and the size of the metadata byte sequence, wherein the size of the metadata byte sequence is obtained based on the size of the second byte buffer.

[0015] Optionally, the method further includes: extracting the metadata byte sequence and the data byte sequence from the serialization result of the data to be processed according to the size of the metadata byte sequence; performing deserialization processing on the metadata byte sequence to parse and obtain the field metadata corresponding to each data field; and performing deserialization processing on the data byte sequence based on the parsed field metadata corresponding to each data field to parse and obtain each data field of the data to be processed.

[0016] Optionally, the field metadata includes the field type of the data field, the length of the field byte sequence, and the starting position of the field byte sequence in the data byte sequence; based on the field metadata corresponding to each data field obtained through parsing, the data byte sequence is deserialized to parse each data field of the data to be processed, including: for each data field, based on the starting position of the parsed field byte sequence in the data byte sequence and the length of the field byte sequence, obtaining the field byte sequence of the data field from the data byte sequence; based on the field type of the data field, the obtained field byte sequence of the data field is deserialized to parse the data field.

[0017] According to another aspect of the present invention, a data processing apparatus is provided, comprising: The data acquisition module is used to acquire data to be processed, wherein the data to be processed includes at least one data field; The serialization processing module is used to serialize each data field to generate a field byte sequence for each data field, so as to obtain the data byte sequence of the data to be processed; The metadata generation module is used to generate field metadata for each data field based on the field byte sequence of each data field, so as to obtain the metadata mapping file of the data to be processed; The metadata processing module is used to serialize the metadata mapping file of the data to be processed to obtain a metadata byte sequence; The result generation module is used to place the metadata byte sequence at the beginning of the data byte sequence to obtain the serialization result of the data to be processed.

[0018] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided in the embodiments of the present invention.

[0019] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the data processing method provided in the embodiments of the present invention.

[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method provided in the embodiments of the present invention.

[0021] An embodiment of the above invention has the following advantages or beneficial effects: by acquiring data to be processed, which includes at least one data field; performing serialization processing on each data field to generate a field byte sequence for each data field, thereby obtaining a data byte sequence of the data to be processed; generating field metadata for each data field based on the field byte sequence of each data field, thereby obtaining a metadata mapping file for the data to be processed; performing serialization processing on the metadata mapping file for the data to be processed to obtain a metadata byte sequence; and placing the metadata byte sequence at the beginning of the data byte sequence to obtain the serialization result of the data to be processed, the technical solution realizes data serialization and parsing based on dynamic metadata-driven processing, which can process dynamic data structures, reduces the limitations of data structures, makes the processing method more flexible, and is simple to implement with high processing efficiency.

[0022] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0023] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main steps of a data processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention; Figure 3 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0024] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0026] Currently, the data serialization and communication frameworks used for data transmission are mostly as follows: 1. Protocol Buffers (Protobuf): This is a high-efficiency binary data serialization protocol. This scheme enables cross-language and cross-platform data exchange through predefined structured data patterns, optimizing the data encapsulation, transmission, and parsing process. 2. Apache Thrift: As a cross-language service development framework, it supports efficient data serialization and remote procedure calls, and simplifies service integration between different languages ​​through compile-time generated code; 3. JSON (JavaScript Object Notation): A text-based, lightweight data interchange format, widely used in web development due to its ease of reading and writing. 4. XML (eXtensible Markup Language): A self-describing markup language used to define the structure and semantics of data. XML's extensibility and flexibility make it suitable for complex data exchange needs; 5. MessagePack: It is a binary serialization format designed to surpass JSON, providing a more compact data representation to save storage space and network bandwidth, and supports multiple languages ​​and data types.

[0027] While these data serialization and communication frameworks each have their own advantages in different aspects, they also have some disadvantages, mainly as follows: 1. Protocol Buffers (Protobuf): The use of Protobuf requires the definition of a strict data structure pattern, which has limitations in the processing of dynamic data structures; 2. Apache Thrift: It performs well in multi-language support, but its flexibility and ease of use in dynamic environments still need improvement. 3. JSON (JavaScript Object Notation): JSON is recognized for its readability and universality, but its text-based data representation is generally less efficient than binary formats. 4. XML (Extensible Markup Language): Its main drawbacks are its verbosity and parsing overhead, especially in scenarios with limited network bandwidth. 5. MessagePack: Performs poorly when handling very large data structures or very complex nested data.

[0028] In summary, these data serialization and communication frameworks have limitations in handling dynamic data structures, are not flexible enough, and suffer from drawbacks such as complex implementation and poor timeliness.

[0029] To address the aforementioned technical problems, this invention provides a data processing method and apparatus, which is essentially a dynamic metadata-driven network packet serialization and parsing technology. By utilizing dynamic metadata, it achieves more efficient and flexible data serialization and parsing. Compared to existing technologies, this technology not only optimizes data processing performance but also improves support for dynamic data structures, thus providing a more efficient and scalable solution for complex network applications.

[0030] Specifically, the network packet serialization and parsing technology solution driven by dynamic metadata of the present invention can achieve the following technical effects: 1. Improve data processing efficiency: Existing serialization technologies, such as JSON and XML, while performing well in terms of text readability, exhibit poor performance when processing large or complex data structures. This is because text format parsing and generation are relatively slow and consume more network bandwidth and storage space. This invention aims to significantly improve the speed and efficiency of data processing through a more efficient binary format and optimized serialization mechanism; 2. Support for Dynamic Data Structures: Many existing serialization frameworks (such as Protobuf and Thrift) require predefined static data structures, which become inflexible when dealing with dynamically changing data. This invention introduces the concept of dynamic metadata, allowing data structures to change dynamically at runtime, thus providing better support for serialization processing needs for rapidly changing data; 3. Reduced Development and Maintenance Costs: In multi-language and multi-platform environments, maintaining multiple serialization and deserialization code libraries increases the complexity of development and maintenance. This invention provides a unified serialization framework, making it easier for developers to share and process data across different programming languages ​​and platforms, thereby reducing development and maintenance costs. 4. Optimize network resource utilization: In environments with limited network bandwidth, transmit as little data as possible. Existing serialization methods, especially text-based methods, often generate large data packets. This invention reduces data packet size by employing a compact binary format and an effective data compression strategy, thereby optimizing network resource utilization. 5. Enhance data exchange security: Data security during transmission is a crucial consideration. This invention enhances data exchange security by incorporating built-in security mechanisms, such as data encryption and integrity verification. 6. Achieving Efficient Random Access to Data: Traditional serialization techniques typically require sequential access to data from the beginning of the data structure until the desired information is found. This approach is inefficient in applications processing large datasets or requiring frequent access to specific data segments. By incorporating random access functionality into this invention, any position within the data packet can be directly located, allowing for rapid reading of specific data elements without traversing the entire data structure. This significantly improves data access efficiency, particularly in the fields of big data and real-time data processing.

[0031] In summary, this invention addresses the limitations of existing data serialization and communication technologies in terms of processing efficiency, data structure flexibility, development and maintenance costs, efficient random access to data, network resource utilization, and data security through dynamic metadata-driven data serialization processing. It provides a more efficient, flexible, and secure data serialization and parsing solution for complex network applications.

[0032] According to embodiments of the present invention, a dynamic metadata structure is designed, wherein the design of the dynamic metadata structure aims to provide a mechanism to support the dynamism of the data structure by defining the characteristics of each data field (e.g., field type, length of the field byte sequence, storage location offset, etc.). This structure allows for the flexible addition, modification, or deletion of data fields without changing the existing data packet format, thus meeting the requirements for dynamic data structures.

[0033] The core idea of ​​dynamic metadata structures is to describe the characteristics of each data field in the data through metadata, such as field type, length of the field's byte sequence, and storage offset, rather than hard-coding this information. This approach provides extremely high flexibility, allowing the data structure to be dynamically adjusted without changing the existing code. The following explains how dynamic metadata structures work.

[0034] The dynamic metadata structure is mainly composed of the mapping structure "map[int]common.FieldMeta", where: the key is an integer type, representing the number of the data field; the value is of type common.FieldMeta, which contains the metadata information of the field, such as the field type Type, storage location offset Offset, and length of the field byte sequence Length.

[0035] The workflow of dynamic metadata structures mainly includes the following five parts.

[0036] 1. Define Metadata: Before data serialization, the metadata for each data field is first defined, including the field type, storage offset, and length of the field's byte sequence. This step is accomplished by dynamically calculating and updating the metadata mapping file (globalMetadata) during data serialization. For example, when a string data field is written using the writeString function (used for string serialization), the function calculates the storage offset of the current string data field's byte sequence within the corresponding byte sequence of the data, and stores the metadata information of that string data field (including field type, storage offset, and length of the field's byte sequence) in the globalMetadata mapping file.

[0037] 2. Data Serialization: During data serialization, the data is converted into a byte sequence, and a metadata mapping file corresponding to the data is generated based on the previously defined metadata. This metadata mapping file is then serialized. This step involves two main actions: first, the actual data fields are serialized, and then the metadata mapping file is serialized. The metadata mapping file is serialized into a byte sequence and placed at the beginning of the byte sequence obtained from the data serialization. This allows the receiver to first parse the metadata and understand how to parse the subsequent data.

[0038] 3. Sending data: Send the entire byte sequence containing metadata and actual data to the data receiver over the network.

[0039] 4. The data receiver receives the data and parses the metadata: The data receiver first reads and parses the metadata to understand the field type, storage location offset, and length of the field byte sequence for each data field.

[0040] 5. Parsing Data Based on Metadata: The data receiver uses the parsed metadata to parse the actual data fields. For example, if the metadata indicates that the first data field is a string, the data receiver will read the corresponding byte sequence based on the storage location offset and the length of the field's byte sequence, and parse it into a string. Of course, this application also supports parsing each data field sequentially from beginning to end during data parsing.

[0041] The following describes how to use dynamic metadata. Suppose you want to send a message containing multiple data fields, one of which is a string representing the API name (APIExample), and another is an integer representing the result code (ResultCode). After serializing these data fields, the metadata information for each field is calculated and updated, including its field type, storage location offset, and the length of its byte sequence. This information is then serialized into a byte sequence and placed at the beginning of the resulting serialized byte sequence.

[0042] After receiving the serialized result from the data sender, the data receiver first parses the metadata to understand the field type, storage location offset, and length of the field's byte sequence for each field. Then, based on this information, the receiver can correctly parse the values ​​of fields such as the API name and result code.

[0043] In this way, even after data fields are added, modified, or deleted, the receiver can still correctly parse the data as long as the metadata is updated correctly, which provides extremely high data structure dynamism and flexibility.

[0044] The data processing procedure of the present invention will be described below with reference to specific embodiments.

[0045] Figure 1 This is a schematic diagram illustrating the main steps of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the data processing method of this embodiment mainly includes the following steps S101 to S105.

[0046] Step S101: Obtain the data to be processed, which includes at least one data field. In data transmission scenarios, the data serialization process is usually performed by the data sender. When performing data serialization, the data to be processed must first be obtained. In embodiments of the present invention, the data to be processed may include at least one data field, and its structure is flexible and does not need to be limited to a fixed structure.

[0047] Step S102: Serialize each data field to generate a field byte sequence for each data field, thereby obtaining the data byte sequence of the data to be processed. In an embodiment of the present invention, when serializing the data, the data fields included in the data are serialized separately.

[0048] According to one embodiment of the present invention, each data field has a field type. The field type can be set when the user provides data to be processed, and can then be directly retrieved.

[0049] Serialization processing is performed on each data field to generate a field byte sequence for each data field. Specifically, this may include: for each data field, serialization processing is performed on the data field according to its field type to generate a field byte sequence for that data field. For different field types, this invention sets up different serialization processing functions to perform serialization processing for different field types respectively.

[0050] According to one embodiment of the present invention, when the field type is a string type, the data field is serialized according to the field type of the data field to generate the field byte sequence of the data field. Specifically, this may include generating the field byte sequence of the data field according to the field number, field type, string length and the string itself.

[0051] According to another embodiment of the present invention, when the field type is an integer type, the data field is serialized according to the field type of the data field to generate the field byte sequence of the data field. Specifically, this may include: encoding the data field using an encoding algorithm corresponding to the integer type to obtain an encoded value; and generating the field byte sequence of the data field according to the field number, field type and the encoded value of the data field.

[0052] According to another embodiment of the present invention, when the field type is a long integer, the data field is serialized according to the field type to generate the field byte sequence of the data field. Specifically, this may include: encoding the data field using an encoding algorithm corresponding to the long integer type to obtain an encoded value; and generating the field byte sequence of the data field according to the field number, field type and encoded value of the data field.

[0053] According to an embodiment of the present invention, before serializing each data field to generate a field byte sequence for each data field, the method further includes: creating a first byte buffer for storing the field byte sequence for each data field. Furthermore, serializing each data field to generate a field byte sequence to obtain the data byte sequence of the data to be processed may specifically include: serializing each data field to generate a field byte sequence for each data field, and storing the field byte sequence of each data field in the first byte buffer; in response to the completion of serialization processing for all data fields of the data to be processed, using the data in the first byte buffer as the data byte sequence of the data to be processed.

[0054] According to embodiments of the present invention, data can be written to a byte buffer (bytes.Buffer) in a predetermined format using functions such as writeString, writeInt, and writeLong. These functions are responsible not only for serializing data fields but also for dynamically updating the globalMetadata mapping file to record the metadata of each data field. These functions are the core of data serialization, allowing structured data (such as strings, integers, etc.) to be converted into byte streams for storage or network transmission. Simultaneously, updating the metadata mapping file ensures that data can be correctly deserialized.

[0055] The `writeString` function is used to write a string data field into a byte buffer. First, it writes a variable-length integer consisting of the field number and the field type. Then, it writes the length of the string and the string itself, as the byte sequence of the data field. Finally, it updates the globalMetadata mapping file, recording the field type (string), storage offset (i.e., the starting offset of the byte sequence in the byte buffer, hence also referred to as "starting offset" or "starting position"), and length of the byte sequence.

[0056] `writeInt` is used to write an integer value into a byte buffer. First, the field number and field type are written. Next, the integer value is encoded using the `zigzagEncode32` function (to support valid encoding of negative numbers), and the encoded value is written to the byte buffer to obtain the byte sequence of the data field of that integer type. Finally, the globalMetadata mapping file is updated to record the metadata information of the data field of that integer type.

[0057] `writeLong` is used to write a long integer data field into a byte buffer, similar to `writeInt`, but for handling 64-bit integers. First, the field number and field type are written. Then, the long integer is encoded using `zigzagEncode64`, and the encoded value is written to the byte buffer to obtain the byte sequence of the long integer data field. Finally, the globalMetadata mapping file is updated to record the metadata information of the long integer data field.

[0058] The functions `zigzagEncode32` and `zigzagEncode64` are used to encode 32-bit and 64-bit integers using ZigZag encoding. ZigZag encoding is an efficient encoding method used to map signed integers to unsigned integers, ensuring that small (positive and negative) integers also have relatively small encoded values, which is highly advantageous for subsequent use of variable-length encoding. The encoding method involves shifting the integer left by one bit, then inverting the sign bit of the original number and XORing it with the result of the left shift.

[0059] Each time a data field is serialized, the contents of the globalMetadata mapping file are updated, using the field number `fieldNumber` as the key and a `common.FieldMeta` structure as the value. This structure records the field type, its storage offset in the byte buffer, and the length of the field's byte sequence. This is done to ensure that the corresponding metadata can be located and correctly parsed during deserialization based on the field number.

[0060] This set of functions and encoding strategies together constitute a flexible serialization framework that allows common basic data types (and can be extended to support more data types) to be converted into byte streams, and supports efficient data transmission and storage by maintaining metadata information. In this way, structured data can be transmitted between different systems or networks while ensuring data integrity and parsability.

[0061] Step S103: Generate field metadata for each data field based on the field byte sequence of each data field to obtain the metadata mapping file of the data to be processed.

[0062] Each time a data field is serialized, the contents of the globalMetadata mapping file are updated, using the field number `fieldNumber` as the key and a `common.FieldMeta` structure as the value. This structure records the field type, its storage offset in the byte buffer, and the length of the field's byte sequence. This is done to ensure that the corresponding metadata can be located and correctly parsed during deserialization based on the field number.

[0063] According to one embodiment of the present invention, generating field metadata for each data field based on the field byte sequence of each data field includes: for each data field, obtaining the field type and the length of the field byte sequence based on the field byte sequence of the data field, and generating the field metadata of the data field based on the field type, the length of the field byte sequence, and the starting position of the field byte sequence in the data byte sequence of the data to be processed.

[0064] Furthermore, the starting position of the field byte sequence within the data byte sequence of the data to be processed is obtained based on the starting offset of the field byte sequence in the first byte buffer. See the specific details described in the preceding embodiments.

[0065] According to one embodiment of the present invention, the metadata mapping file stores the mapping relationship between the field number of each data field and the field metadata. After generating the field metadata of each data field, the field metadata of each data field can be saved in the globalMetadata mapping file to obtain the metadata mapping file of the data to be processed.

[0066] Step S104: Serialize the metadata mapping file of the data to be processed to obtain a metadata byte sequence.

[0067] According to an embodiment of the present invention, the metadata mapping file stores the mapping relationship between the field number of each data field and the field metadata. Serializing the metadata mapping file of the data to be processed to obtain a metadata byte sequence may specifically include: creating a second byte buffer to store the metadata byte sequence; setting the size of the second byte buffer according to the size of the metadata mapping file of the data to be processed; traversing the metadata mapping file and writing the mapping relationship between the field number of each data field and the field metadata into the second byte buffer according to a specified format to obtain the metadata byte sequence.

[0068] According to one embodiment of the present invention, the mapping relationship between the field number and the field metadata of each data field is written into the second byte buffer in a specified format, including: writing the mapping relationship between the field number and the field metadata of each data field into the second byte buffer in a specified order and with a specified field type.

[0069] Specifically, the serialization process for metadata mapping files can be performed as follows: 1. Create a byte buffer: Create a new byte buffer, bytes.Buffer, to temporarily store the serialized metadata; 2. Write Metadata Length: The size of the second byte buffer can be set according to the size of the metadata mapping file. Since the metadata byte sequence obtained after metadata serialization is added to the beginning of the data byte sequence, it is necessary to obtain the size of the metadata byte sequence. Therefore, the length of the second byte buffer needs to be set here. In the embodiment of this invention, the binary.Write function is used, and little-endian (binary.LittleEndian) is specified to store the integer value int32(len(metadata)). 3. Traverse and write the metadata for each data field (i.e., each metadata item): Traverse the metadata mapping file, and for each key-value pair (the mapping relationship between the field number and the corresponding metadata) in the mapping file, write it into the byte buffer in the following order and according to the following field types: Field number (fieldNumber): written as an integer (int32); Field type (meta.Type): Write as an integer (int32); Storage location offset (meta.Offset): written as an integer (int32); Metadata byte sequence length (meta.Length): written as an integer (int32); These four attributes are necessary for parsing the received data because they tell the receiver how to extract each data field from the received data; 4. Return the serialized byte slice: Finally, the function returns all bytes in the byte buffer as the serialized result—the metadata byte sequence.

[0070] In an embodiment of the present invention, the metadata byte sequence can be concatenated to the beginning of the data byte sequence to obtain the serialization result of the data to be processed.

[0071] In other embodiments, this metadata byte sequence can also be sent over the network so that the data receiver can use the corresponding deserialization logic to restore the original metadata mapping, and then parse the sent data based on the received data byte sequence.

[0072] The `serializeMetadata` function is part of the data serialization process, which is particularly useful when transmitting structured data between clients and servers. This ensures that the structure and type information of the data are preserved during network transmission, allowing the receiver to correctly parse and process the received data.

[0073] Step S105: Place the metadata byte sequence at the beginning of the data byte sequence to obtain the serialization result of the data to be processed. Then, the serialization result of the data to be processed can be encapsulated into a data packet for data transmission.

[0074] According to one embodiment of the present invention, placing the metadata byte sequence before the data byte sequence to obtain the serialization result of the data to be processed includes: placing the metadata byte sequence before the first byte buffer corresponding to the data byte sequence to obtain the serialization result of the data to be processed.

[0075] According to one embodiment of the present invention, the serialization result of the data to be processed includes the metadata byte sequence, the data byte sequence, and the size of the metadata byte sequence, wherein the size of the metadata byte sequence is obtained according to the size of the second byte buffer.

[0076] According to one embodiment of the present invention, encryption and signature mechanisms are typically used to ensure the confidentiality and integrity of data during transmission. Throughout the data serialization process, encryption algorithms are applied to the data fields requiring protection for security processing, ensuring data security during transmission. Furthermore, the serialized data and its corresponding dynamic metadata are encapsulated together in a data packet, and to enhance security, the metadata itself may also be encrypted. These measures collectively construct a secure and efficient data transmission framework.

[0077] Specifically, before sending data, the serialized data can be encrypted to ensure that the content of the data packets is not eavesdropped on or tampered with during network transmission. For example, the AES encryption algorithm can be used to encrypt the data. To ensure data integrity and prevent tampering, a digital signature can be added to the data packets. This is typically obtained by hashing the data packet content (including metadata and business data) and then encrypting the hash value using the sender's private key. The receiver can use the sender's public key to verify the signature, ensuring data integrity.

[0078] According to the technical solution of this invention, a key metadata index table is constructed during the data serialization process to directly link the field number (i.e., index number) of each data field with its specific location (storage location offset) in the data packet. Furthermore, to improve data processing efficiency, a random access method is provided, enabling users to directly locate and access specific data fields by field number, without needing to parse the entire data packet one by one. This method greatly optimizes the data access process and achieves highly efficient random access capabilities.

[0079] The construction and use of the metadata index table are achieved by defining the global variable `globalMetadata` and the function `serializeMetadata` for serializing metadata. `globalMetadata` acts as a global map, storing metadata information for each data field, including the field type, storage location offset, and the length of the field's byte sequence. This map allows a direct link between field numbers and their metadata to be established during the serialization phase. Then, during the deserialization phase, this information can be used to efficiently and randomly access any field in the data packet without parsing the entire packet one by one.

[0080] During the deserialization phase, after receiving the data packet and parsing the corresponding metadata, the globalMetadata mapping file can be used to efficiently and randomly access any data field. For example, the `getFieldValue` function can be used to directly retrieve the value of a field by its field number, without having to parse the entire data packet from scratch. The specific implementation of the `getFieldValue` function is as follows: First, it checks if `fieldNumber` exists in the globalMetadata mapping file; if it does, it uses the storage location offset and the length of the field's byte sequence stored in the metadata to directly access and parse the data field, without having to parse the entire data packet field by field. For example, suppose you want to access the value of data field number 5. After receiving the data packet and parsing the metadata, you can directly call the `getFieldValue` function. Using the previously built metadata index table (i.e., the globalMetadata mapping file), you can find the storage location offset and the length of the field's byte sequence based on field number 5, and then directly extract the value of the corresponding data field from the data packet. This method avoids parsing the entire data packet field by field from scratch, greatly improving the efficiency of data access.

[0081] In summary, by building a metadata index table during the serialization stage and using this index table for efficient random access during the deserialization stage, the performance of data processing can be significantly optimized, which is especially important for processing large amounts of data or transmitting structured data in network communication.

[0082] According to one embodiment of the present invention, the data processing method further includes: extracting the metadata byte sequence and the data byte sequence from the serialization result of the data to be processed based on the size of the metadata byte sequence; performing deserialization processing on the metadata byte sequence to parse and obtain field metadata corresponding to each data field; and performing deserialization processing on the data byte sequence based on the parsed field metadata corresponding to each data field to parse and obtain each data field of the data to be processed.

[0083] According to one embodiment of the present invention, the field metadata includes the field type of the data field, the length of the field byte sequence, and the starting position of the field byte sequence in the data byte sequence; based on the field metadata corresponding to each data field obtained by parsing, the data byte sequence is deserialized to parse each data field of the data to be processed, which may specifically include: for each data field, based on the starting position of the field byte sequence of the parsed data field in the data byte sequence and the length of the field byte sequence, obtaining the field byte sequence of the data field from the data byte sequence; based on the field type of the data field, the obtained field byte sequence of the data field is deserialized to parse the data field.

[0084] At the data receiving end, dynamic parsing of each data field is achieved by utilizing the dynamic metadata contained in the received data packets. Furthermore, to accelerate access to specific data fields, the receiving end also utilizes a metadata index table to quickly locate the specific positions of these data fields, thus realizing an efficient and flexible data access method. This method, combining dynamic parsing and rapid index table location, greatly improves the efficiency and flexibility of data processing.

[0085] The combination of data deserialization and dynamic parsing using metadata index tables provides an efficient and flexible data processing mechanism. This mechanism is extremely important in handling structured data transmission, especially in network communication and big data processing scenarios.

[0086] At the data receiving end, data packet processing is divided into two main stages: deserialization and dynamic parsing.

[0087] The goal of the deserialization stage is to convert the received byte stream back into its original data structure or data object. In this process, the receiving end first needs to parse the metadata information contained in the data packet. The metadata describes information such as the field type, the length of the field's byte sequence, and the starting position of each data field in the data packet, enabling the data receiver to understand the data structure.

[0088] In the dynamic parsing phase, after obtaining metadata information, the data receiver can dynamically parse the value of each field based on this information. This dynamic parsing capability allows the receiver to flexibly handle various data structures without needing to know the specific layout of the data in advance.

[0089] To expedite access to specific data fields, the data receiver utilizes a metadata index table to quickly locate the specific positions of these fields. This metadata index table, parsed from the data packet during the deserialization phase, provides a fast access path for each field. During the parsing of the data packet's metadata, the data receiver constructs an index table that maps field numbers to their storage offsets within the data packet and the length of the field's byte sequence. This process typically occurs once when the data packet arrives. With this index table, when a specific data field needs to be accessed, the data receiver can directly use the field number to find the field's location information in the index table and then jump directly to the corresponding position in the data packet to read the field value, without needing to traverse the entire data packet field by field.

[0090] The technical solution of the present invention, which combines dynamic parsing and index table fast location, has the following advantages: 1. High efficiency in data processing: By avoiding a full scan of the entire data packet, data processing time is significantly reduced, especially for complex data structures containing a large number of fields; 2. Flexibility of data format: Dynamic parsing enables the data receiver to process data packets of various formats, and can correctly parse data even without knowing the detailed structure of the data packets; 3. Scalability: Adding new fields or modifying existing fields will not affect the parsing of other fields, making data format upgrades and maintenance easier.

[0091] The implementation of the technical solution of this invention will provide a secure and flexible serialization and deserialization solution for network data transmission, which is particularly suitable for application scenarios where data structures change frequently or where there are high requirements for random access to data.

[0092] Figure 2 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention. Figure 2 As shown, the data processing apparatus 200 of this embodiment mainly includes a data acquisition module 201, a serialization processing module 202, a metadata generation module 203, a metadata processing module 204, and a result generation module 205.

[0093] Data acquisition module 201 is used to acquire data to be processed, wherein the data to be processed includes at least one data field; The serialization processing module 202 is used to perform serialization processing on each data field to generate a field byte sequence for each data field, so as to obtain the data byte sequence of the data to be processed; Metadata generation module 203 is used to generate field metadata for each data field based on the field byte sequence of each data field, so as to obtain the metadata mapping file of the data to be processed; Metadata processing module 204 is used to serialize the metadata mapping file of the data to be processed to obtain a metadata byte sequence; The result generation module 205 is used to place the metadata byte sequence at the beginning of the data byte sequence to obtain the serialization result of the data to be processed.

[0094] According to one embodiment of the present invention, each data field has a field type; the serialization processing module 202 can also be used to: perform serialization processing on each data field according to the field type of the data field to generate a field byte sequence of the data field.

[0095] According to another embodiment of the present invention, when the field type is a string type, the serialization processing module 202 can also be used to generate a field byte sequence of the data field based on the field number, field type, string length and the string itself of the data field.

[0096] According to another embodiment of the present invention, when the field type is an integer type, the serialization processing module 202 can also be used to: encode the data field using an encoding algorithm corresponding to the integer type to obtain an encoded value; and generate a field byte sequence of the data field according to the field number, field type and the encoded value of the data field.

[0097] According to another embodiment of the present invention, when the field type is a long integer, the serialization processing module 202 can also be used to: encode the data field using an encoding algorithm corresponding to the long integer type to obtain an encoded value; and generate a field byte sequence of the data field according to the field number, field type and the encoded value of the data field.

[0098] According to another embodiment of the present invention, the metadata generation module 203 can also be used to: for each data field, obtain the field type of the data field and the length of the field byte sequence according to the field byte sequence of the data field, and generate field metadata of the data field according to the field type of the data field, the length of the field byte sequence and the starting position of the field byte sequence in the data byte sequence of the data to be processed.

[0099] According to another embodiment of the present invention, the serialization processing module 202 can further be used to: create a first byte buffer for storing the field byte sequence of each data field before performing serialization processing on each data field to generate the field byte sequence of each data field. The serialization processing module 202 can also be used to: perform serialization processing on each data field to generate the field byte sequence of each data field, and store the field byte sequence of each data field in the first byte buffer; in response to the completion of serialization processing on all data fields of the data to be processed, use the data in the first byte buffer as the data byte sequence of the data to be processed. Furthermore, the starting position of the field byte sequence in the data byte sequence of the data to be processed is obtained based on the starting offset of the field byte sequence in the first byte buffer.

[0100] According to another embodiment of the present invention, the result generation module 205 can also be used to: place the metadata byte sequence in front of the first byte buffer corresponding to the data byte sequence to obtain the serialization result of the data to be processed.

[0101] According to another embodiment of the present invention, the metadata mapping file stores the mapping relationship between the field number of each data field and the field metadata; the metadata processing module 204 can also be used to: create a second byte buffer for storing the metadata byte sequence; set the size of the second byte buffer according to the size of the metadata mapping file of the data to be processed; traverse the metadata mapping file and write the mapping relationship between the field number of each data field and the field metadata into the second byte buffer according to a specified format to obtain the metadata byte sequence.

[0102] According to another embodiment of the present invention, when the metadata processing module 204 writes the mapping relationship between the field number of each data field and the field metadata into the second byte buffer in a specified format, it can specifically be used to write the mapping relationship between the field number of each data field and the field metadata into the second byte buffer in a specified order and with a specified field type.

[0103] According to another embodiment of the present invention, the serialization result of the data to be processed includes the metadata byte sequence, the data byte sequence, and the size of the metadata byte sequence, wherein the size of the metadata byte sequence is obtained according to the size of the second byte buffer.

[0104] According to another embodiment of the present invention, the data processing apparatus 200 may further include a data deserialization and parsing module (not shown in the figure), configured to: extract the metadata byte sequence and the data byte sequence from the serialization result of the data to be processed based on the size of the metadata byte sequence; perform deserialization processing on the metadata byte sequence to parse and obtain field metadata corresponding to each data field; and perform deserialization processing on the data byte sequence based on the parsed field metadata corresponding to each data field to parse and obtain each data field of the data to be processed.

[0105] According to another embodiment of the present invention, the field metadata includes the field type of the data field, the length of the field byte sequence, and the starting position of the field byte sequence in the data byte sequence; the data deserialization parsing module (not shown in the figure) performs deserialization processing on the data byte sequence based on the field metadata corresponding to each data field obtained by parsing to obtain each data field of the data to be processed, specifically: for each data field, based on the starting position of the field byte sequence of the parsed data field in the data byte sequence and the length of the field byte sequence, obtains the field byte sequence of the data field from the data byte sequence; based on the field type of the data field, performs deserialization processing on the obtained field byte sequence of the data field to parse to obtain the data field.

[0106] According to the technical solution of the present invention, the following steps are taken: acquiring data to be processed, which includes at least one data field; performing serialization processing on each data field to generate a field byte sequence for each data field, thereby obtaining a data byte sequence of the data to be processed; generating field metadata for each data field based on the field byte sequence of each data field, thereby obtaining a metadata mapping file for the data to be processed; performing serialization processing on the metadata mapping file for the data to be processed to obtain a metadata byte sequence; and placing the metadata byte sequence at the beginning of the data byte sequence to obtain the serialization result of the data to be processed. This technical solution realizes data serialization and parsing based on dynamic metadata, which can process dynamic data structures, reduces the limitations of data structures, makes the processing method more flexible, and is simple to implement with high processing efficiency.

[0107] Figure 3 An exemplary system architecture 300 is shown, in which a data processing method or data processing apparatus to which embodiments of the present invention can be applied is illustrated.

[0108] like Figure 3As shown, system architecture 300 may include terminal devices 301, 302, and 303, a network 304, and a server 305. Network 304 serves as the medium for providing communication links between terminal devices 301, 302, and 303 and server 305. Network 304 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0109] Users can use terminal devices 301, 302, and 303 to interact with server 305 via network 304 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 301, 302, and 303, such as data processing applications, data serialization applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0110] Terminal devices 301, 302, and 303 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0111] Server 305 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 301, 302, and 303 (for example only). The backend management server can process received data processing requests and other data by: acquiring data to be processed, the data to be processed including at least one data field; serializing each data field to generate a field byte sequence for each data field, thus obtaining a data byte sequence of the data to be processed; generating field metadata for each data field based on the field byte sequence of each data field, thus obtaining a metadata mapping file for the data to be processed; serializing the metadata mapping file for the data to be processed to obtain a metadata byte sequence; placing the metadata byte sequence at the beginning of the data byte sequence to obtain the serialization result of the data to be processed, and feeding back the processing result (e.g., the serialization result of the data to be processed—for example only) to the terminal device.

[0112] It should be noted that the data processing method provided in the embodiments of the present invention is generally executed by server 305, and correspondingly, the data processing device is generally disposed in server 305.

[0113] It should be understood that Figure 3 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0114] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing terminal devices or servers of the present invention. Figure 4 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0115] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0116] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0117] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined above in the system of this invention.

[0118] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a data acquisition module, a serialization processing module, a metadata generation module, a metadata processing module, and a result generation module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the result generation module can also be described as "a module for placing the metadata byte sequence before the data byte sequence to obtain the serialization result of the data to be processed."

[0121] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: acquiring data to be processed, the data to be processed including at least one data field; serializing each data field to generate a field byte sequence for each data field, to obtain a data byte sequence of the data to be processed; generating field metadata for each data field based on the field byte sequence of each data field, to obtain a metadata mapping file of the data to be processed; serializing the metadata mapping file of the data to be processed to obtain a metadata byte sequence; and placing the metadata byte sequence before the data byte sequence to obtain a serialization result of the data to be processed.

[0122] According to the technical solution of the present invention, the following steps are taken: acquiring data to be processed, which includes at least one data field; performing serialization processing on each data field to generate a field byte sequence for each data field, thereby obtaining a data byte sequence of the data to be processed; generating field metadata for each data field based on the field byte sequence of each data field, thereby obtaining a metadata mapping file for the data to be processed; performing serialization processing on the metadata mapping file for the data to be processed to obtain a metadata byte sequence; and placing the metadata byte sequence at the beginning of the data byte sequence to obtain the serialization result of the data to be processed. This technical solution realizes data serialization and parsing based on dynamic metadata, which can process dynamic data structures, reduces the limitations of data structures, makes the processing method more flexible, and is simple to implement with high processing efficiency.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of data processing, characterized by, The method comprises the following steps: obtaining to-be-processed data, wherein the to-be-processed data comprises no less than one data field; serializing each data field to generate a field byte sequence of each data field, so as to obtain a data byte sequence of the to-be-processed data; generating field metadata of each data field according to the field byte sequence of each data field, so as to obtain a metadata mapping file of the to-be-processed data; serializing the metadata mapping file of the to-be-processed data to obtain a metadata byte sequence; placing the metadata byte sequence in the front of the data byte sequence to obtain a serialization result of the to-be-processed data.

2. The method of claim 1, wherein, Each data field has a field type. Serializing each data field to generate a field byte sequence of each data field comprises: Serializing each data field according to the field type of the data field to generate a field byte sequence of the data field.

3. The method of claim 2, wherein, In the case that the field type is a string type, serializing each data field according to the field type of the data field to generate a field byte sequence of the data field comprises: generating the field byte sequence of the data field according to the field number, the field type, the string length and the string itself of the data field.

4. The method of claim 2, wherein, In the case that the field type is an integer type, serializing each data field according to the field type of the data field to generate a field byte sequence of the data field comprises: encoding the data field by using an encoding algorithm corresponding to the integer type to obtain an encoding value; generating the field byte sequence of the data field according to the field number, the field type and the encoding value of the data field.

5. The method of claim 2, wherein, In the case that the field type is a long integer type, serializing each data field according to the field type of the data field to generate a field byte sequence of the data field comprises: encoding the data field by using an encoding algorithm corresponding to the long integer type to obtain an encoding value; generating the field byte sequence of the data field according to the field number, the field type and the encoding value of the data field.

6. The method of claim 1, wherein, Generating field metadata of each data field according to the field byte sequence of each data field comprises: for each data field, obtaining the field type of the data field and the length of the field byte sequence according to the field byte sequence of the data field, and generating the field metadata of the data field according to the field type of the data field, the length of the field byte sequence and the starting position of the field byte sequence in the data byte sequence of the to-be-processed data.

7. The method of claim 6, wherein, Before serializing each data field to generate a field byte sequence of each data field, the method further comprises: creating a first byte buffer for storing the field byte sequence of each data field; Serializing each data field to generate a field byte sequence of each data field, so as to obtain a data byte sequence of the to-be-processed data, comprises: serializing each data field to generate a field byte sequence of each data field, and storing the field byte sequence of each data field into the first byte buffer; in response to completion of the serialization of all data fields of the to-be-processed data, taking the data in the first byte buffer as a data byte sequence of the to-be-processed data; and the starting position of the field byte sequence in the data byte sequence of the to-be-processed data is obtained according to the starting offset of the field byte sequence in the first byte buffer.

8. The method of claim 7, wherein, placing the metadata byte sequence at the front of the data byte sequence to obtain a serialization result of the to-be-processed data, including: placing the metadata byte sequence at the front of the first byte buffer corresponding to the data byte sequence to obtain a serialization result of the to-be-processed data.

9. The method according to claim 1 or 8, characterized in that, the metadata mapping file stores a mapping relationship between a field number and field metadata of each data field; serializing the metadata mapping file of the to-be-processed data to obtain a metadata byte sequence, including: creating a second byte buffer for storing the metadata byte sequence; setting a size of the second byte buffer according to a size of the metadata mapping file of the to-be-processed data; traversing the metadata mapping file, and writing the mapping relationship between the field number and the field metadata of each data field into the second byte buffer according to a specified format to obtain the metadata byte sequence.

10. The method of claim 9, wherein, writing the mapping relationship between the field number and the field metadata of each data field into the second byte buffer according to the specified format, including: writing the mapping relationship between the field number and the field metadata of each data field into the second byte buffer according to a specified order and a specified field type.

11. The method of claim 9, wherein, the serialization result of the to-be-processed data includes the metadata byte sequence, the data byte sequence, and a size of the metadata byte sequence, and the size of the metadata byte sequence is obtained according to the size of the second byte buffer.

12. The method of claim 11, wherein, the method further includes: extracting the metadata byte sequence and the data byte sequence from the serialization result of the to-be-processed data according to the size of the metadata byte sequence; performing deserialization processing on the metadata byte sequence to parse to obtain field metadata corresponding to each data field; based on the parsed field metadata corresponding to each data field, performing deserialization processing on the data byte sequence to parse to obtain each data field of the to-be-processed data.

13. The method of claim 12, wherein, the field metadata includes a field type of a data field, a length of a field byte sequence, and a starting position of the field byte sequence in a data byte sequence; based on the parsed field metadata corresponding to each data field, performing deserialization processing on the data byte sequence to parse to obtain each data field of the to-be-processed data, including: for each data field, based on the parsed starting position of the field byte sequence of the data field in the data byte sequence and the length of the field byte sequence, obtaining the field byte sequence of the data field from the data byte sequence; Based on the field type of the data field, the field byte sequence of the obtained data field is deserialized to obtain the data field.

14. An apparatus for data processing, characterized by The method comprises the steps of: The data acquisition module is configured to acquire the to-be-processed data, wherein the to-be-processed data comprises at least one data field. The serialization processing module is configured to serialize each data field to generate a field byte sequence of each data field, thereby obtaining a data byte sequence of the to-be-processed data. The metadata generation module is configured to generate field metadata of each data field according to the field byte sequence of each data field, thereby obtaining a metadata mapping file of the to-be-processed data. The metadata processing module is configured to serialize the metadata mapping file of the to-be-processed data to obtain a metadata byte sequence. The result generation module is configured to place the metadata byte sequence in the front of the data byte sequence to obtain a serialization result of the to-be-processed data.

15. An electronic device, comprising: The method comprises the steps of: One or more processors; Storage apparatus configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-13.

16. A computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1-13.

17. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1-13.

Images