Data processing method and system

Abstract

The embodiment of the present specification provides a data processing method and system, the data processing method is applied to a sending end, the sending end comprises a sending end software sequencer and a sending end hardware accelerator, wherein the sending end software sequencer, in response to a data call request of a receiving end, in the case of determining that the data structure is the same as that of the receiving end, serializes the data to be called based on the data call request, generates serialized data, and sends the serialized data to the sending end hardware accelerator; the sending end hardware accelerator encodes and compresses the serialized data based on a preset encoding compression method to generate a serialized byte stream, and sends the serialized byte stream to the receiving end; the subsequent receiving end software deserializer and the receiving end hardware accelerator will be relatively decoded and decompressed and deserialized to obtain the data to be called.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a data processing method and system. BACKGROUND

[0002] In data transmission and application between networks, it is usually required to convert objects into data streams to realize data transmission and application. However, under the existing object serialization and deserialization scheme, the object graph in the memory is directly traversed, and the object graph is completely copied for serialization / deserialization. The entire copying process is in units of the entire object, which will copy a large amount of metadata and invalid data, resulting in a very large byte stream after final encoding, which seriously affects the network transmission efficiency and increases the network transmission cost. SUMMARY

[0003] Therefore, the embodiments of the present specification provide a data processing method. One or more embodiments of the present specification also relate to a data processing system, a computing device, and a computer-readable storage medium to solve the technical defects in the prior art.

[0004] According to a first aspect of the embodiments of the present specification, a data processing method is provided, which is applied to a sending end including a sending end software serializer and a sending end hardware accelerator, wherein

[0005] The sending end software serializer serializes to-be-called data based on a data calling request of a receiving end to generate serialized data and sends the serialized data to the sending end hardware accelerator in a case where it is determined that the data structure of the receiving end is the same in response to the data calling request of the receiving end.

[0006] The sending end hardware accelerator encodes and compresses the serialized data based on a preset encoding compression method to generate a serialized byte stream and sends the serialized byte stream to the receiving end.

[0007] According to a second aspect of the embodiments of the present specification, a data processing system is provided, which includes a sending end software serializer and a sending end hardware accelerator, wherein

[0008] The sending end software serializer is configured to serialize to-be-called data based on a data calling request of a receiving end to generate serialized data and send the serialized data to the sending end hardware accelerator in a case where it is determined that the data structure of the receiving end is the same in response to the data calling request of the receiving end.

[0009] The sending end hardware accelerator is configured to encode and compress the serialized data based on a preset encoding compression method to generate a serialized byte stream and send the serialized byte stream to the receiving end.

[0010] According to a third aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0011] a memory and a processor;

[0012] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above data processing method.

[0013] According to a fourth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above data processing method.

[0014] According to a fifth aspect of the embodiments of the present specification, a computer program is provided, which, when executed in a computer, causes the computer to perform the steps of the above data processing method.

[0015] One embodiment of the present specification implements a data processing method and system, the data processing method is applied to a sending end, the sending end comprises a sending end software serializer and a sending end hardware accelerator, wherein the sending end software serializer, in response to a data call request of a receiving end, determines that the data structure of the receiving end is the same, serializes the data to be called based on the data call request, generates serialized data, and sends the serialized data to the sending end hardware accelerator; the sending end hardware accelerator encodes and compresses the serialized data based on a preset encoding compression method to generate a serialized byte stream, and sends the serialized byte stream to the receiving end.

[0016] The data processing method serializes the data to be called (i.e. the memory object) when it is determined that the data structure of the sending end is the same as the data structure of the receiving end, and encodes and compresses the serialized data to be called based on a preset encoding compression method, reduces the saving of metadata in the serialized data to be called, greatly reduces the encoding expansion problem, improves the network transmission efficiency, and reduces the network transmission cost. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a structure diagram of the data to be called in a data processing method provided by one embodiment of the present specification;

[0018] Figure 2 is a principle diagram of a serialization / deserialization technical solution taking kryo as an example provided by one embodiment of the present specification;

[0019] Figure 3is a principle diagram of a serialization / deserialization technical solution provided by an embodiment of the present specification using an on-chip accelerator;

[0020] Figure 4 is a principle diagram of a serialization / deserialization technical solution provided by an embodiment of the present specification taking Skyway as an example;

[0021] Figure 5 is a flow chart of a data processing method provided by an embodiment of the present specification;

[0022] Figure 6 is a diagram of different class definitions of a serialization end and a deserialization end in a data processing method provided by an embodiment of the present specification;

[0023] Figure 7 is a diagram of same class definitions of a serialization end and a deserialization end in a data processing method provided by an embodiment of the present specification;

[0024] Figure 8 is a diagram of serialization / deserialization of to-be-called data in a data processing method provided by an embodiment of the present specification;

[0025] Figure 9 is a process flow chart of the data processing method provided by an embodiment of the present specification;

[0026] Figure 10 is a structural diagram of a data processing system provided by an embodiment of the present specification;

[0027] Figure 11 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0028] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, set forth in this description. Those skilled in the art, in light of the description, can implement the present specification without limiting to the specific details disclosed in this description.

[0029] The terms used in one or more embodiments of the present specification are merely for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present specification. The singular forms "a", "an" and "the" used in one or more embodiments of the present specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present specification means and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a particular order or relationship between the information. The terms are used only to distinguish one type of information from another. For example, a first item can be termed a second item, and, similarly, a second item can be termed a first item, without departing from the scope of one or more embodiments of the present description. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining" or "in response to a determination."

[0031] First, the noun terms related to one or more embodiments of the present description are explained.

[0032] Serialization / deserialization: the process of writing an object in memory to a byte stream at the local end, and recovering the object in memory from the byte stream at the other end.

[0033] RPC: Remote Procedure Call.

[0034] In-network computing: network-in computing.

[0035] Hessian: Hessian is a remote communication library based on binary-RPC implementation provided by caucho, the process of converting Java objects into byte sequences is called object serialization; the process of restoring byte sequences into Java objects is called object deserialization.

[0036] Kryo: a fast and efficient object serialization framework based on Java, aiming to provide fast, efficient and easy-to-use API. Kryo can complete serialization at any time, whether it is a file, a database or network data. Kryo can also perform automatic deep copy (clone), shallow copy (clone), which is a direct copy of object to object, not object->byte->object copy.

[0037] Grpc: Java object serialization framework.

[0038] Due to the limited computing power of a single computing node, the server application is split into distributed services within the data center, and the distributed services communicate through RPC. The parameters and calculation results of RPC communication need to be serialized in order to be transmitted in the form of a byte stream over the network, and then the byte stream is constructed into a memory object friendly to the program code through deserialization.

[0039] Serialization / deserialization in distributed applications CPU consumption accounts for 10%-20%, so improving the efficiency of serialization / deserialization is essential to improve CPU utilization and application performance.

[0040] In the current serialization method, the strategy of directly traversing the object graph in memory and serializing / deserializing the complete copy of the object graph does not require additional encoding and is the most computationally efficient. However, since the entire copy process is based on the entire object, a large amount of metadata and invalid data will be copied, resulting in a very large byte stream after encoding, which will seriously affect network efficiency, so it is crucial to compress the encoded byte stream while ensuring computational efficiency.

[0041] Referring to Figure 1 , Figure 1 A structure diagram of data to be called in a data processing method according to an embodiment of the present specification is shown.

[0042] Among them, the data to be called can be understood as an object in memory. Specifically, taking a Java object as an example, the layout of the Java object in memory is introduced.

[0043] It can be seen from Figure 1 that each object has a fixed-length Header, followed by each field, and each field is a value or a pointer to another object. The Header consists of a Mark word and a Kclass pointer, where the Markword is used to identify the object and record the object state, and the Kclass is used to point to the Type descriptor type metadata, which contains the layout of the object (the representation of the object in memory) and the total size of the object.

[0044] Specifically, Kclass is a pointer that points to Type descriptor type metadata. First, it loads the offset of the pointer in the object, and the last block is a pointer field, which can find the position of the next pointer according to the offset. The position of the pointer is used to point to the next object, and so on, so that the representation of all objects in memory can be obtained. Among them, the pointer can be understood as a memory address pointing to another object.

[0045] In the embodiments of the present specification, two serialization / deserialization methods are proposed, wherein the first serialization / deserialization method is represented by Hessian, Kryo and Grpc (protobuffer), and the second serialization / deserialization method is represented by Skyway serialization technology.

[0046] Referring to Figure 2, Figure 2 Fig. 1 shows a schematic diagram of a serialization / deserialization technical solution according to an embodiment of the present specification.

[0047] The technical solution divides the fields of an object into three categories: reference, value and embedded class. The reference points to another object and is modified to the relative address of the object during serialization. The value represents a primitive data type such as integer, character, float, etc. and is saved in the serialized data stream after encoding. The embedded class is used for types such as list, map and hashmap in Java and includes these classes in the data stream using a variable-length array and a self-contained format.

[0048] For optimization purposes, these serialization / deserialization tools usually use classID to represent all field types and meta types to reduce the overhead of preserving strings.

[0049] As Figure 2 In Kryo, a type table is set up including the type name and the corresponding number of the type name. Then, during serialization, the type name is replaced by the number and during deserialization, the number is replaced back to realize the serialization / deserialization operation. For example, the type name objA corresponds to the number 11, the type name objB corresponds to the number 12, and the type name objC corresponds to the number 13.

[0050] The serialization / deserialization tool of the technical solution requires a large amount of computing power. In order to accelerate the serialization / deserialization operation, an on-chip accelerator method can also be used.

[0051] Referring to Figure 3 , Figure 3 Fig. 2 shows a schematic diagram of a serialization / deserialization technical solution using an on-chip accelerator according to an embodiment of the present specification.

[0052] Figure 3 The technical solution includes DRAM (memory), CPU Cores (CPU cores), an access interface and a hardware accelerator. The CPU Cores and the hardware accelerator are connected to the DRAM through the access interface and the serialization / deserialization of the memory object is completed through data transmission between them.

[0053] Specifically, as Figure 3 It can be seen that this kind of accelerator directly interacts with the memory system, sends a serialization or deserialization request through a simple software structure, the scheduler checks the request at the head of the queue, finds an available SEU (serialization unit) or DEU (deserialization unit), and performs serialization and deserialization according to the object graph.

[0054] However, the on-chip hardware accelerator needs to modify the CPU design, and a separate accelerator is needed on the CPU to perform serialization / deserialization; that is, the on-chip hardware accelerator needs to modify the structure of the CPU processor, and since the on-chip accelerator serializes by encoding the object and sending it to the deserialization end, the encoding is different from the format of the object in the memory, and when the deserialization end receives the encoding and restores it into an object, it needs to allocate a memory space of the original size in the memory for it, and then decode the encoding into the memory space to complete the deserialization, and the operation of allocating memory is also a very time-consuming operation.

[0055] Referring to Figure 4 , Figure 4 A principle diagram of a serialization / deserialization technical solution provided by an embodiment of the present specification is shown, taking Skyway as an example.

[0056] Figure 4 The left side of the figure includes a serialization end (Node A) and the right side includes a deserialization end (Node B), and the serialization / deserialization is completed through direct copying of Heap (heap memory) and Native (direct memory), as shown in Figure 4 In the figure, the serialization end directly copies B, C, and D in the Heap to the Native, or the deserialization end directly copies A, C, and D in the Heap to the Native.

[0057] Specifically, by Figure 4 It can be seen that object copying, Skyway transmits each object as a whole by modifying the JVM, which eliminates the need for independent access to each data field. Skyway uses an automatic global type numbering process to represent types-main nodes maintain registration of all types and their IDs, and each worker node obtains the ID of its type when loading the type by communicating with the main node. Skyway uses an efficient "relative value" technique to adjust pointers. When objects are copied to the output buffer, the pointers saved inside them are "taken relative values" in linear time-from "absolute values" to "relative values".

[0058] and Figure 2Compared with the scheme in the related art, Skyway only optimizes the software layer to improve the efficiency of serialization / deserialization. However, the scheme has the problems of encoding expansion and compatibility. The encoding expansion problem is that Skyway completely copies the entire object in the memory to the serialized byte stream, which causes a large amount of invalid data to be copied, resulting in the expansion of the encoded byte stream, which reduces the network efficiency, especially for cross-data center transmission. The compatibility problem is that when different versions, such as class member increase and decrease operations, occur, the object memory at the two ends of the serialization / deserialization is not completely consistent. At this time, direct memory copying will cause incorrect operation.

[0059] Based on this, further in the present specification, a data processing method is provided, and the present specification also relates to a data processing system, a computing device, and a computer readable storage medium, which are described in detail one by one in the following embodiments.

[0060] Referring to Figure 5 , Figure 5 A flowchart of a data processing method according to an embodiment of the present specification is shown, wherein the method is applied to a sending end including a sending end software serializer and a sending end hardware accelerator. Specifically, the data processing method specifically includes the following steps.

[0061] Step 502: The sending end software serializer, in response to a data call request of a receiving end, serializes to-be-called data based on the data call request to generate serialized data and sends the serialized data to the sending end hardware accelerator in a case where it is determined that the data structure of the receiving end is the same.

[0062] Specifically, the data processing method provided by the embodiment of the present specification is applied to the serialization / deserialization scenario of an object, so that the sending end can be understood as a serialization end, and the receiving end can be understood as a deserialization end. The sending end hardware accelerator and the receiving end hardware accelerator in the data processing method of the embodiment of the present specification are placed together with the network card for acceleration on the network, and acceleration is performed before sending or after receiving, without the need for modification at the CPU level, thereby protecting the hardware investment.

[0063] Wherein, the to-be-called data can be understood as a memory object, the data call request can be understood as an object call request, and the data structure can be understood as a class.

[0064] In specific implementation, the sending end software serializer, in response to an object call request of a receiving end, serializes a to-be-called object based on the object call request to generate serialized data and sends the serialized data to the sending end hardware accelerator in a case where it is determined that the class definition of the receiving end is the same.

[0065] In actual application, the sending end software serializer serializes the to-be-invoked object based on the object call request, and generates the serialized data in the following specific implementation manner:

[0066] The serialized data is generated by serializing the to-be-invoked data based on the data call request, and the method comprises the following steps of:

[0067] The sending end software serializer copies the to-be-invoked data to the data buffer according to the data structure of the receiving end based on the data call request, forms initial serialized data, modifies the pointer field of the to-be-invoked data to the relative pointer of the initial serialized data, completes serialization of the to-be-invoked data, and generates the serialized data.

[0068] In the embodiments of the present specification, when the class definition of the sending end software serializer is consistent with the class definition of the receiving end, the sending end software serializer directly copies the object (i.e. the to-be-invoked data) to the buffer (data buffer) when serializing the object, and modifies the pointer field in the object to the relative pointer of the object in the byte stream, thereby conveniently and efficiently completing serialization of the object.

[0069] Step 504: The sending end hardware accelerator encodes and compresses the serialized data based on a preset encoding and compression method to generate a serialized byte stream, and sends the serialized byte stream to the receiving end.

[0070] Specifically, after receiving the serialized data sent by the sending end software serializer, the sending end hardware accelerator encodes and compresses the serialized data based on a preset encoding and compression method to generate a serialized byte stream and sends the serialized byte stream to the receiving end.

[0071] In actual application, the receiving end also comprises a receiving end software deserializer and a receiving end hardware accelerator, and the sending of the serialized byte stream to the receiving end comprises the following steps of:

[0072] The sending end hardware accelerator sends the serialized byte stream to the receiving end hardware accelerator, wherein the sending end hardware accelerator and / or the receiving end hardware accelerator are arranged on a network card.

[0073] The data processing method provided by the embodiments of the present specification, the receiving end comprises a receiving end hardware accelerator and a receiving end deserializer, after the sending end hardware accelerator sends the serialized byte stream to the receiving end, the receiving end hardware accelerator can decode and decompress the serialized byte stream, so that the encoding expansion problem is solved without the awareness of the receiving end deserializer, and the sending end hardware accelerator and / or the receiving end hardware accelerator are arranged on a network card, that is, placed off-chip, and accelerated on the network, so that acceleration is realized before sending or after receiving, and no modification is needed at the CPU level, thereby protecting the hardware investment.

[0074] In actual implementation, after the receiving end receives the serialized byte stream sent by the sending end hardware accelerator, the receiving end hardware accelerator decompresses and decodes the received serialized byte stream, and the receiving end software deserializer deserializes the serialized decoded data after decompression and decoding to generate the to-be-called data for subsequent data processing. The specific implementation manner is as follows:

[0075] The receiving end hardware accelerator decompresses and decodes the received serialized byte stream based on the preset encoding compression method to generate serialized decoded data, comprising:

[0076] The receiving end hardware accelerator decompresses and decodes the received serialized byte stream based on the preset encoding compression method to generate serialized decoded data, and sends the serialized decoded data to the receiving end software deserializer.

[0077] The receiving end software deserializer deserializes the received serialized decoded data to generate the to-be-called data.

[0078] The preset encoding compression method is the same as the preset encoding compression method of the sending end.

[0079] Specifically, the sending end hardware accelerator uses a preset encoding compression method to compress and encode the serialized data, and the receiving end hardware accelerator uses the preset encoding compression method to decompress and decode the serialized byte stream, wherein the serialized byte stream is obtained by compressing and encoding the serialized data by the sending end hardware accelerator through the preset encoding compression method.

[0080] In actual application, since the binary uses JVM (virtual machine) object layout, that is, the layout of the serialized data and objects in the memory is the same, in this way, only memory copy operation is needed when serializing the objects, and only pointer adjustment is needed when deserializing, no useless objects are generated and the performance is excellent. However, the serialization / deserialization through copying will cause the problem of encoding expansion. The main sources of encoding expansion include:

[0081] • Metadata: mark, klass overhead, 12 bytes per object extra overhead

[0082] • boxing: int box expands from 4 bytes to 16 bytes as Integer

[0083] • String encoding: java string stored in utf-16, double space for all-english string

[0084] • Variable length integer: lack of integer variable length encoding mechanism, small integer kryo only occupies 1 byte

[0085] Data structure details: take ArrayList: [5, 12, 78] as an example

[0086] • Hessian serialization: ArrayList, len=3, 5, 12, 78

[0087] • Memory copy: ArrayList, len=3, array[offset_5, offset_12, offset_78, null, null], obj5, obj12, obj78

[0088] To solve this problem, the following solutions are adopted:

[0089] Metadata:

[0090] Cancel kclassID for integer, float, double, char, etc. Stateless recovery can be realized at the D end

[0091] For ordinary classes, only kclassID can be kept.

[0092] boxing:

[0093] string can be regarded as an array of char type

[0094] For integer, float, double, char, etc. Add corresponding action items in the action list to inform the hardware to encode these classes (such as varint encoding), which is combined with the compatibility solution introduced earlier.

[0095] String encoding:

[0096] java string stored in utf-16, double space for all-english string; hardware online encoding and decoding can be used to solve the encoding and decoding problem of string.

[0097] Variable-length integer:

[0098] The small integer only occupies 1 byte when using hardware for integer variable-length coding; the Double type appears more in e-commerce applications, and the coding efficiency is not great.

[0099] According to the above description, when the serialized data is encoded and compressed based on the preset encoding compression method, the data content in the serialized data is first analyzed, and in the case where metadata, boxing, string encoding and / or variable-length integer exist, the preset solution is used for processing, realizing the encoding compression of the serialized data, and solving the problem of encoding inflation in the serialized data.

[0100] In specific implementation, the serialized data is encoded and compressed based on the preset encoding compression method to generate a serialized byte stream, including:

[0101] The sending end hardware accelerator determines the attribute information of the serialized data, and in the case where it is determined based on the attribute information of the serialized data that the serialized data meets the preset encoding compression condition, determines the encoding compression method corresponding to the serialized data based on the attribute information of the serialized data.

[0102] The serialized data is encoded and compressed according to the encoding compression method corresponding to the serialized data to generate a serialized byte stream.

[0103] Among them, based on the attribute information of the serialized data, it can be determined whether the serialized data is metadata, boxing, string encoding and / or variable-length integer.

[0104] Specifically, in the case where it is determined based on the attribute information of the serialized data that the serialized data meets the preset encoding compression condition (i.e. belongs to metadata, boxing, string encoding and / or variable-length integer), the encoding compression method corresponding to the serialized data is determined based on the attribute information of the serialized data, i.e. the solution corresponding to metadata, boxing, string encoding and / or variable-length integer, and the serialized data is decoded and compressed according to the encoding compression method corresponding to the serialized data to generate a serialized byte stream. That is, the above encoding compression method of metadata, boxing, string encoding and / or variable-length integer is used to encode and compress the serialized data to generate a serialized byte stream.

[0105] Then, when deserializing, the receiving end hardware accelerator, after receiving the serialized byte stream sent by the sending end hardware accelerator, decompresses and decodes the serialized byte stream based on the above encoding inflation solution to obtain serialized decoding data. Among them, the serialized decoding data can be understood as the serialized data before the encoding compression of the sending end hardware accelerator.

[0106] In the data processing method provided by the embodiments of the present specification, the sending end hardware accelerator can compress and encode the serialized data based on the above-mentioned encoding expansion solution to generate a serialized byte stream, so as to effectively reduce the network bandwidth occupied when the serialized byte stream is transmitted to the receiving end, reduce the network cost, and improve the transmission efficiency; after receiving the serialized byte stream formed after encoding and compression, the receiving end hardware accelerator can continue to decode and decompress the received serialized byte stream based on the above-mentioned encoding expansion solution to obtain the serialized data before encoding and compression, so as to ensure that the subsequent receiving end deserializer can obtain the integrity of the to-be-called data after deserializing the serialized data.

[0107] In actual application, the class definitions of the serialization end and the deserialization end may be inconsistent. In order to solve the serialization compatibility problem, the serialization end needs to request the class definition of the deserialization end before serializing the object, and compare it with the class definition of the serialization end. The specific implementation manner is as follows:

[0108] The sending end software serializer serializes the to-be-called data based on the data calling request to generate serialized data in a case where the data structure of the receiving end is determined to be the same.

[0109] The sending end software serializer sends a data structure acquisition request to the receiving end software deserializer in response to the data calling request of the receiving end software deserializer, and receives the data structure of the receiving end returned by the receiving end software deserializer based on the data structure acquisition request.

[0110] The sending end software serializer serializes the to-be-called data based on the data calling request to generate serialized data in a case where the data structure of the receiving end is determined to be the same.

[0111] Specifically, the sending end software serializer sends a class definition acquisition request to the receiving end software deserializer in response to the object calling request of the receiving end software deserializer. The receiving end software deserializer returns the class definition of the receiving end to the sending end software serializer in a case where the class definition acquisition request is received. The sending end software serializer compares the class definition of the receiving end returned by the receiving end software deserializer with its own class definition. In a case where the class definition of the receiving end is determined to be the same as the class definition of the sending end, it is indicated that there is no serialization compatibility problem between the two, and at this time, the to-be-called object can be serialized based on the data calling request.

[0112] Referring to Figure 6 , Figure 6This diagram illustrates a data processing method according to an embodiment of the present specification, in which the class definitions of the serialization end and the deserialization end are different.

[0113] Figure 6 In this context, 'a' represents the data format after serialization, defined according to the class definition of the serialization endpoint. Figure 6 In this context, 'b' represents the data format required by the deserialization end based on the class definition of the deserialization end.

[0114] Depend on Figure 6 As can be seen from the comparison between a and b, since the class definitions of the serialization end and the deserialization end are different, the data format after serialization by the serialization end is different from the data format required by the deserialization end. Therefore, there will be a version incompatibility problem when serializing and deserializing objects.

[0115] To resolve serialization compatibility issues, the data layout of the buffer needs to be adjusted when the object layout changes.

[0116] See Figure 7 , Figure 7 A schematic diagram is shown in which the class definitions of the serialization end and the deserialization end are the same in a data processing method provided according to an embodiment of this specification.

[0117] Figure 7 In this context, 'a' represents the data format after serialization, defined according to the class definition of the serialization endpoint. Figure 7 In this context, 'b' represents the data format required by the deserialization end based on the class definition of the deserialization end.

[0118] Depend on Figure 7 As can be seen from the comparison between a and b, since the class definitions of the serialization end and the deserialization end are the same, the software serializer will directly copy the object into the data buffer during serialization.

[0119] Right now Figure 7 Region 1 in the code represents the metadata in the virtual machine, and Region 2 represents the data information. The object graph (i.e., the graph formed by the association between objects and other objects through pointers) is copied to a buffer (data buffer) according to the JVM (virtual machine) layout (the way objects are represented in memory), forming a byte stream. The pointer fields in the objects are modified to the relative pointers of the objects in the byte stream, thus completing the serialization.

[0120] On the deserialization side, the virtual machine interprets the data as an object, and modifies the pointer fields in the object from relative pointers to absolute address pointers, thus completing the deserialization.

[0121] Specifically, in the case of the same class definition at the serialization end and the deserialization end, the serialization / deserialization operation can be quickly completed without any memory movement.

[0122] In practical applications, Figure 6 and Figure 7 are the serialization / deserialization operations of the object in the case of the same or different class definitions at the serialization end and the deserialization end without a hardware accelerator.

[0123] In the embodiment of the present specification, in the case of different class definitions at the sending end software serializer and the receiving end software deserializer with a hardware accelerator, the operation sequence can be generated based on different class definitions to solve the version incompatibility problem. The specific implementation is as follows:

[0124] The sending end software serializer further comprises, in response to the data call request of the receiving end:

[0125] The sending end software serializer serializes the data to be called based on the data call request and generates serialized data in the case of determining that the data structure is not the same as that of the receiving end; and

[0126] generate an operation sequence based on the data structure of the sending end and the data structure of the receiving end, and send the serialized data and the operation sequence to the sending end hardware accelerator.

[0127] Specifically, in the case of different data structures at the sending end and the receiving end, the sending end software serializer also serializes the data to be called based on the data call request and generates serialized data; at the same time, an operation sequence is generated based on the data structure of the sending end and the data structure of the receiving end, and the serialized data and the operation sequence are sent to the sending end hardware accelerator.

[0128] The generation of the operation sequence can be calculated based on the data structure of the sending end and the different data structure of the receiving end and the sending end.

[0129] In specific implementation, when serializing, the operation needs to be encoded into a hardware executable operation (action), and the basic operation types are defined as follows:

[0130] action definition (the following len can be encoded by vaint, if the encoding overhead is acceptable):

[0131] 000 + len(16bit): skip len bytes

[0132] 001 + len(8bit) :insert len bytes

[0133] 010 + len(8bit) :delete len bytes

[0134] 011 + len(8bit) :move len bytes

[0135] 100 + len(8bit) :convert from utf-16 to utf-8 encoding

[0136] 101 + len(8bit) :convert from utf-8 to utf-16 encoding ....

[0137] 111 :class separator

[0138] Thus, the common operation types are mainly including:

[0139] • skip

[0140] • insert

[0141] • delete

[0142] • move

[0143] In addition, the length of the opcode can be encoded in the head of each class opcode, so that the hardware accelerator can directly jump to the next class opcode by retrieving the length of the opcode to achieve parallel operation of the class.

[0144] Thus, a complete action list should be similar to the following format:

[0145] opcode1 opcode1 len

[0146] opcode2 opcode2 len ...

[0147] end mark

[0148] Therefore, the action list for type 0 (Node) is: 000 0000 1000 001 0000 0100 111

[0152] After the operation sequence is generated in the above manner, the operation sequence is sent to the sending end hardware accelerator, so that the sending end hardware accelerator can generate a byte stream corresponding to the data structure of the receiving end according to the serialized data sent by the sending end software serializer and the operation sequence, and encode and compress the byte stream, and send the byte stream to the receiving end hardware accelerator, that is, the version compatibility problem can be ensured, and the sending end software serializer can be implemented without sensing. The specific implementation manner is as follows:

[0153] The sending end hardware accelerator encodes and compresses the serialized data based on a preset encoding compression method to generate a serialized byte stream, and sends the serialized byte stream to the receiving end, comprising:

[0154] The sending end hardware accelerator generates a byte stream matching the data structure of the receiving end according to the serialized data and the operation sequence, encodes and compresses the byte stream based on a preset encoding compression method to generate a serialized byte stream, and sends the serialized byte stream to the receiving end hardware accelerator.

[0155] The method of encoding and compressing the byte stream based on the preset encoding compression method is consistent with the method of encoding and compressing the serialized data based on the preset encoding compression method described above, and will not be described here.

[0156] Specifically, after receiving the serialized data and the operation sequence sent by the sending end software serializer, the sending end hardware accelerator generates a byte stream matching the data structure of the receiving end according to the serialized data and the operation sequence, encodes and compresses the byte stream based on a preset encoding compression method to generate a serialized byte stream, and sends the serialized byte stream to the receiving end hardware accelerator.

[0157] As can be known from the above description, when the byte stream is encoded and compressed based on the preset encoding compression method, the data content in the byte stream is first analyzed, and when it is determined that there is metadata, boxing, string encoding and / or variable-length integer, the preset solution is used to process it, so as to realize the encoding and compression of the byte stream and solve the problem of encoding expansion in the byte stream.

[0158] After receiving the serialized byte stream sent by the sending end hardware accelerator, the receiving end hardware accelerator decompresses and decodes the serialized byte stream based on the above encoding compression solution to obtain serialized decoding data. At this time, the serialized decoding data can be understood as the byte stream before the encoding and compression of the sending end hardware accelerator, that is, the byte stream matching the data structure of the receiving end generated according to the serialized data and the operation sequence.

[0159] Further, after the decompression and decoding of the byte stream received by the receiving end hardware software deserializer, the byte stream is deserialized according to its own data structure to obtain the to-be-called data. The specific implementation is as described below.

[0160] The receiving end software deserializer deserializes the received serialized and decoded data to generate the to-be-called data, including:

[0161] The receiving end software deserializer modifies the relative pointer of the received serialized and decoded data to an absolute pointer to complete deserialization and generate the to-be-called data.

[0162] Specifically, at the receiving end, when deserializing, the receiving end software deserializer modifies the pointer field in the serialized and decoded data (i.e., the object in the byte stream) from a relative pointer to an absolute address pointer to complete deserialization.

[0163] The data processing method provided by the embodiments of the present specification, in the case of determining that the data structure of the sending end is the same as that of the receiving end, serializes the to-be-called data (i.e., the memory object), and encodes and compresses the serialized to-be-called data based on a preset encoding and compression method, reduces the saving of metadata in the serialized to-be-called data, greatly reduces the encoding inflation problem, improves the network transmission efficiency, and reduces the network transmission cost.

[0164] Referring to Figure 8 , Figure 8 A schematic diagram of serialization / deserialization of to-be-called data in a data processing method according to an embodiment of the present specification is shown.

[0165] Among them, Figure 8 The a part in the a part represents the serialization of the to-be-called data, and the b part represents the deserialization of the serialized data sent by the sending end.

[0166] Figure 8 The a part in the a part includes source code, original serialization output, software-processed serialization output, operation sequence, and hardware-processed serialization output. The software processing can be understood as the processing of the sending end software serializer, and the hardware processing can be understood as the processing of the sending end hardware accelerator.

[0167] In implementation, the sending end software serializer requests the receiving end software deserializer for the Node class definition; the receiving end software deserializer associates the receiving end Node class definition with classID: 0; the receiving end software deserializer sends "Node:{value=I,node=LNode;expired=Z" to the sending end software serializer; the sending end software serializer calculates the operation sequence (action list) according to the difference between the local "Node:{value=I,node=LNode" and the receiving end software deserializer.

[0168] Meanwhile, the sending end software serializer processes the original serialization output obtained according to the source code to obtain the software-processed serialization output, i.e., the serialized byte stream.

[0169] The sending end software serializer sends the serialized byte stream and the operation sequence (including the start address and length of the operation sequence) to the sending end hardware accelerator; the sending end hardware accelerator performs hardware processing and encoding on the serialized byte stream and the operation sequence.

[0170] Specifically, the sending end hardware accelerator generates the serialized byte stream organized according to the layout format of the receiving end software deserializer according to the serialized byte stream and the operation sequence, and generates the hardware-processed serialization output by packeting and sending to the receiving end hardware accelerator. The packeting can be understood as encoding and compression of the generated serialized byte stream. The solution to the coding expansion problem in encoding and compression can be referred to the detailed description of the above embodiment.

[0171] Figure 8 The b part in the above formula includes the input received by the deserialization end hardware, the input received by the deserialization end software, and the output of the deserialization end software. The deserialization end hardware can be understood as the receiving end hardware accelerator, and the deserialization end software can be understood as the receiving end software deserializer.

[0172] In implementation, after the deserialization end hardware receives the serialized byte stream sent by the sending end hardware accelerator, the serialized byte stream is decoded and decompressed according to the above solution to the coding expansion problem to generate the serialized byte stream organized according to the layout format of the receiving end, and the serialized byte stream is sent to the deserialization end software. That is, the deserialization end hardware sends the decoded serialized byte stream to the deserialization end software as the input received by the deserialization end software for subsequent processing after the received input is decoded by hardware.

[0173] The deserialization end software copies the received serialized byte stream to the memory of the receiving end, and modifies the value of the pointer of the object in the serialized byte stream, to complete the deserialization. That is, the deserialization end software deserializes the received input to generate the deserialization end software output, to complete the deserialization.

[0174] The data processing method provided in the embodiments of the present specification integrates the calculation of the operation sequence and the encoding expansion solution together, and in the entire serialization process (part a of the data processing method) Figure 8 of the present specification), the calculation of the relative pointer value of the pointer field in the object is modified by using the definition of the class of the receiving end, wherein int enc and Double enc in the action list represent the integer and double type corresponding to the position, and the corresponding encoding conversion needs to be performed by the hardware accelerator; after the hardware processing, the serialized output will embed the special identified int and double type in the serialized byte stream, for the identification of the hardware accelerator of the receiving end; in the deserialization process (part b of the data processing method) Figure 8 of the present specification), the receiving end hardware accelerator sends the serialized byte stream to the deserialization software after restoring the serialized byte stream into the format of the layout of the receiving end according to the information embedded in the serialized byte stream; the deserialization software realizes the deserialization by modifying the relative pointer of the object in the serialized byte stream into the absolute pointer, to obtain the object. Through the acceleration strategy of in-network calculation, the compatibility of the serializer / deserializer and the offloading of the encoding expansion problem are realized, the overhead in the serialization / deserialization on the CPU is limited to a small number of functions such as object graph traversal and pointer modification, and the balance between the encoding efficiency and the calculation overhead is ensured.

[0175] Referring to Figure 9 , Figure 9 a processing process flow diagram of the data processing method is shown, and specifically includes the following steps.

[0176] The data processing method is applied to a serialization / deserialization system, and the serialization / deserialization system includes an S-end serializer (i.e., the above-mentioned sending end software serializer), an S-end accelerator (i.e., the above-mentioned sending end hardware accelerator), a D-end accelerator (i.e., the above-mentioned receiving end hardware accelerator), and a D-end deserializer (i.e., the above-mentioned receiving end software deserializer).

[0177] Step 902: The S-end serializer requests the D-end class definition from the D-end deserializer.

[0178] Step 904: The D-end deserializer returns the D-end class definition to the S-end serializer.

[0179] Step 906: In the case that the S-side class definition is different from the D-side class definition, the S-side sequencer generates an operation sequence, and sends the software-serialized data and the operation sequence to the S-side accelerator.

[0180] Step 908: The S-side accelerator performs compatibility and encoding compression processing on the software-serialized data and the operation sequence, to generate a serialized byte stream.

[0181] Step 910: The S-side accelerator sends the serialized byte stream to the D-side accelerator.

[0182] Step 912: The D-side accelerator decompresses and decodes the serialized byte stream.

[0183] Step 914: The D-side accelerator sends the decompressed and decoded data to the D-side deserializer.

[0184] Step 916: The D-side deserializer deserializes the decompressed and decoded data.

[0185] The data processing method provided by the embodiments of the present specification, when performing serialization / deserialization processing on data, in the entire serialization process, if the class definitions of the serialization side and the deserialization side are different, then in the serialization, the calculation of the relative pointer value of the pointer field in the object is modified using the definition of the D-side class, for example, in the a part of the above Figure 8 int enc and Double enc in the action list respectively represent that the integer and double types at the position need to be encoded and transformed by the hardware accelerator, and after the hardware processing, the serialized output will embed the special identified int and double types in the serialized byte stream, for the D-side accelerator to identify, and through this process, the serialization is completed. In the deserialization process, the D-side accelerator restores the serialized byte stream into the format of the D-side layout after embedding the information in the serialized byte stream, and sends it to the deserialization software, and the deserialization software can realize deserialization by modifying the relative pointer to an absolute pointer.

[0186] Moreover, in the data processing method provided by the embodiments of the present specification, the hardware accelerator is placed off-chip, and does not need to be protected at the CPU level, thereby protecting the hardware investment; at the same time, the method does not need to dynamically allocate memory for each object in the object graph at the deserialization side, but only needs to uniformly allocate memory for the entire object graph, thereby reducing the overhead of memory allocation and management; in addition, the method greatly reduces the encoding expansion problem through the reduction of the saving of metadata in the serialized byte stream, and maintains the efficiency of serialization / deserialization through hardware offloading, and also solves the compatibility problem through real-time hardware encoding.

[0187] Corresponding to the method embodiments described above, the present specification also provides data processing system embodiments, Figure 10 A structural schematic diagram of a data processing system provided by one embodiment of the present specification is shown. As shown in the figure, Figure 10 The system includes a sending-end software serializer and a sending-end hardware accelerator, wherein,

[0188] The sending-end software serializer 1002 is configured to, in response to a data call request of the receiving end, serialize data to be called based on the data call request to generate serialized data and send the serialized data to the sending-end hardware accelerator, in a case where it is determined that the data structure of the receiving end is the same.

[0189] The sending-end hardware accelerator 1004 is configured to encode and compress the serialized data based on a preset encoding and compression method to generate a serialized byte stream and send the serialized byte stream to the receiving end.

[0190] Optionally, the receiving end includes a receiving-end software deserializer and a receiving-end hardware accelerator.

[0191] Correspondingly, the sending-end hardware accelerator 1004 is further configured to send the serialized byte stream to the receiving-end hardware accelerator, wherein the sending-end hardware accelerator and / or the receiving-end hardware accelerator are arranged on a network card.

[0192] Optionally, the receiving-end hardware accelerator is configured to decompress and decode the received serialized byte stream based on the preset encoding and compression method, generate serialized decoded data, and send the serialized decoded data to the receiving-end software deserializer.

[0193] The receiving-end software deserializer is configured to deserialize the received serialized decoded data to generate the data to be called.

[0194] Optionally, the sending-end software serializer 1002 is further configured to, in response to a data call request of the receiving-end software deserializer, send a data structure acquisition request to the receiving-end software deserializer and receive a data structure of the receiving end returned by the receiving-end software deserializer based on the data structure acquisition request.

[0195] In a case where it is determined that the data structure of the receiving end is the same, serialize data to be called based on the data call request to generate serialized data.

[0196] Optionally, the sending-end software sequencer 1002 is further configured to, in a case where it is determined that the data structure of the receiving end is not the same, serialize the to-be-invoked data based on the data invocation request to generate serialized data; and

[0197] generate an operation sequence based on the data structure of the sending end and the data structure of the receiving end, and send the serialized data and the operation sequence to the sending-end hardware accelerator.

[0198] Optionally, the sending-end hardware accelerator 1004 is further configured to generate a byte stream matching the data structure of the receiving end according to the serialized data and the operation sequence, encode and compress the byte stream based on a preset encoding compression method to generate serialized byte stream, and send the serialized byte stream to the receiving-end hardware accelerator.

[0199] Optionally, the sending-end software sequencer 1002 is further configured to copy the to-be-invoked data to a data buffer according to the data structure of the receiving end based on the data invocation request to form initial serialized data, modify a pointer field of the to-be-invoked data to a relative pointer of the initial serialized data, complete serialization of the to-be-invoked data, and generate the serialized data.

[0200] Optionally, the receiving-end software deserializer is further configured to modify a relative pointer of the received serialized decoding data to an absolute pointer to complete deserialization and generate the to-be-invoked data.

[0201] Optionally, the sending-end hardware accelerator 1004 is further configured to determine attribute information of the serialized data, and in a case where it is determined based on the attribute information of the serialized data that the serialized data satisfies a preset encoding compression condition, determine an encoding compression method corresponding to the serialized data based on the attribute information of the serialized data.

[0202] encode and compress the serialized data according to the encoding compression method corresponding to the serialized data to generate serialized byte stream.

[0203] The data processing system provided by the embodiments of the present specification serializes to-be-invoked data (i.e., memory objects) in a case where the data structure of the sending end is the same as the data structure of the receiving end, and encodes and compresses the serialized to-be-invoked data based on a preset encoding compression method, thereby reducing the saving of metadata in the serialized to-be-invoked data, greatly reducing the encoding compression problem, improving network transmission efficiency, and reducing network transmission cost.

[0204] The above is a schematic solution of the data processing system according to the embodiment. It should be noted that the technical solution of the data processing system and the technical solution of the data processing method described above belong to the same concept, and the details of the technical solution of the data processing system that are not described in detail can be referred to the description of the technical solution of the data processing method.

[0205] Figure 11 A structural block diagram of a computing device 1100 according to one embodiment of the present specification is shown. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 through a bus 1130, and a database 1150 is used to save data.

[0206] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1060. Examples of these networks include the public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 can include one or more of any type of network interface (e.g., network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0207] In one embodiment of the present specification, the above-mentioned components of the computing device 1100 and other components not shown in the Figure 11 may be connected to each other, for example, through a bus. It should be understood that Figure 11 The structural block diagram of the computing device shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0208] The computing device 1100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 1100 can also be a mobile or stationary server.

[0209] The processor 1120 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned data processing method.

[0210] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0211] An embodiment of the present specification further provides a computer readable storage medium storing computer executable instructions, and the computer executable instructions are executed by a processor to implement the steps of the data processing method.

[0212] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0213] An embodiment of the present specification further provides a computer program, and when the computer program is executed in a computer, the computer executes the steps of the data processing method.

[0214] The above is a schematic scheme of the computer program of the embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the computer program that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0215] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order described in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or necessary.

[0216] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0217] It should be noted that for the foregoing method embodiments, the descriptions are each simply a combination of a series of acts for the sake of brevity, but those skilled in the art should know that the present application is not limited by the order of the acts described, because some steps can be performed in other orders or at the same time in accordance with the present application. In addition, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the acts and modules involved are not necessarily essential to the present application.

[0218] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0219] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, according to the content of the present application, many modifications and changes can be made. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, the method being applied at a sending end, the sending end comprising a sending end software serializer and a sending end hardware accelerator, wherein, The sending end software serializer, in response to the receiving end's data call request, and if it is determined that the data structure is the same as that of the receiving end, serializes the data to be called based on the data call request, generates serialized data, and sends the serialized data to the sending end hardware accelerator. The transmitting end hardware accelerator encodes and compresses the serialized data based on a preset encoding and compression method to generate a serialized byte stream, and then sends the serialized byte stream to the receiving end. The sending-end software serializer further includes, when it is determined that the data structure is different from that of the receiving end, serializing the data to be called based on the data call request to generate serialized data; Furthermore, an operation sequence is generated based on the data structure of the sending end and the data structure of the receiving end, and the serialized data and the operation sequence are sent to the hardware accelerator of the sending end; The transmitting end hardware accelerator further includes generating a byte stream that matches the data structure of the receiving end based on the serialized data and the operation sequence, encoding and compressing the byte stream based on a preset encoding and compression method to generate a serialized byte stream, and sending the serialized byte stream to the receiving end hardware accelerator.

2. The data processing method according to claim 1, wherein the receiving end includes a receiving end software deserializer and a receiving end hardware accelerator; Accordingly, sending the serialized byte stream to the receiving end includes: The transmitting hardware accelerator sends the serialized byte stream to the receiving hardware accelerator, wherein the transmitting hardware accelerator and / or the receiving hardware accelerator are located on the network interface card.

3. The data processing method according to claim 2, further comprising, after sending the serialized byte stream to the receiving end hardware accelerator: The receiving end hardware accelerator decompresses and deencodes the received serialized byte stream based on the preset encoding and compression method to generate serialized decoded data, and sends the serialized decoded data to the receiving end software deserializer. The receiving end software deserializer deserializes the received serialized and decoded data to generate the data to be called.

4. The data processing method according to claim 2, wherein the sending-end software serializer, in response to a data call request from the receiving end, serializes the data to be called based on the data call request, generating serialized data, provided that the data structure is the same as that of the receiving end, including: The sending software serializer, in response to the data call request of the receiving software deserializer, sends a data structure retrieval request to the receiving software deserializer, and receives the receiving software deserializer's data structure returned based on the data structure retrieval request. If the data structure is determined to be the same as that of the receiving end, the data to be called is serialized based on the data call request to generate serialized data.

5. The data processing method according to claim 1, wherein the step of serializing the data to be called based on the data call request to generate serialized data includes: The sending-end software serializer copies the data to be called to the data buffer according to the data structure of the receiving end based on the data call request, forming initial serialized data, and modifies the pointer field of the data to be called to a relative pointer of the initial serialized data, thereby completing the serialization of the data to be called and generating the serialized data.

6. The data processing method according to claim 3, wherein the receiving end software deserializer deserializes the received serialized-decoded data to generate the data to be called, comprising: The receiving software deserializer modifies the relative pointer of the received serialized and decoded data to an absolute pointer to complete the deserialization and generate the data to be called.

7. The data processing method according to claim 1 or 3, wherein encoding and compressing the serialized data based on a preset encoding compression method to generate a serialized byte stream includes: The transmitting end hardware accelerator determines the attribute information of the serialized data, and if it is determined that the serialized data meets the preset encoding and compression conditions based on the attribute information of the serialized data, it determines the encoding and compression method corresponding to the serialized data based on the attribute information of the serialized data. The serialized data is encoded and compressed according to the encoding and compression method corresponding to the serialized data to generate a serialized byte stream.

8. A data processing system, the system comprising a transmitting software serializer and a transmitting hardware accelerator, wherein, The sending-end software serializer is configured to, in response to a data call request from the receiving end, serialize the data to be called based on the data call request, generate serialized data, and send the serialized data to the sending-end hardware accelerator, provided that the data structure is the same as that of the receiving end. The transmitting end hardware accelerator is configured to encode and compress the serialized data based on a preset encoding and compression method to generate a serialized byte stream, and send the serialized byte stream to the receiving end; The sending-end software serializer further includes, when it is determined that the data structure is different from that of the receiving end, serializing the data to be called based on the data call request to generate serialized data; Furthermore, an operation sequence is generated based on the data structure of the sending end and the data structure of the receiving end, and the serialized data and the operation sequence are sent to the hardware accelerator of the sending end; The transmitting end hardware accelerator further includes generating a byte stream that matches the data structure of the receiving end based on the serialized data and the operation sequence, encoding and compressing the byte stream based on a preset encoding and compression method to generate a serialized byte stream, and sending the serialized byte stream to the receiving end hardware accelerator.

9. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1-7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-7.

Images