A data processing method, device and storage medium

Abstract

The application discloses a data processing method and device and a storage medium, wherein the data processing method comprises the following steps: obtaining first language structure data parsed based on a Proto description file, the first language structure data comprising a field in the Proto description file expressed in a first language and conversion limitation information between languages corresponding to a field type of the field in the Proto description file; calling a preset conversion function parsed based on the Proto description file; the preset conversion function representing a mapping relationship between languages determined based on the field type of the field in the Proto description file and the conversion limitation information between languages corresponding to the field type; converting the first language structure data into second language structure data according to the preset conversion function; and serializing the second language structure data to obtain Protobuf serialized data. The application can use Protobuf as a data exchange format of a protocol, can use shared memory in a service, and improves data transmission efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, and storage medium. Background Technology

[0002] Protocol Buffers, or Protobuf, is a lightweight and efficient structured data storage format. It can be used for structured data serialization, making it well-suited for data storage or RPC (Remote Procedure Call Protocol) data exchange. It's a language-independent, platform-independent, and scalable serialized structured data format applicable to communication protocols, data storage, and other fields. Compared to XML and JSON, Protobuf uses binary byte serialization, resulting in higher data transmission efficiency.

[0003] Shared memory is an efficient inter-process communication method that allows processes to directly read and write memory without copying any data. Shared memory is typically a fixed-size memory block, and the data within it is often in the form of statically typed language structures, such as those in C. However, Protobuf lacks a serialization interface for C structure data, preventing related services from using Protobuf as the data exchange format when using shared memory. This not only affects the compatibility between the front-end and back-end of the protocol but also results in low data transmission efficiency. Summary of the Invention

[0004] To address the problems of existing technologies, embodiments of the present invention provide a data processing method, apparatus, and storage medium. The technical solution is as follows:

[0005] On the one hand, a data processing method is provided, the method comprising:

[0006] Obtain first language structure data based on the parsing of the proto description file. The first language structure data includes fields in the proto description file represented in the first language, and language conversion constraint information corresponding to the field types of the fields in the proto description file; the language includes the first language and the second language.

[0007] Invoke the preset conversion function obtained by parsing the proto description file; the preset conversion function represents the mapping relationship between languages ​​determined based on the field types of the fields in the proto description file and the conversion constraint information between the languages ​​corresponding to the field types;

[0008] According to the preset conversion function, the first language structure data is converted into the second language structure data;

[0009] The second language structure data is serialized to obtain protobuf serialized data.

[0010] On the other hand, a data processing apparatus is provided, the apparatus comprising:

[0011] The first acquisition module is used to acquire first language structure data obtained by parsing a proto description file. The first language structure data includes fields in the proto description file represented in the first language, and language conversion constraint information corresponding to the field types of the fields in the proto description file. The language includes the first language and the second language.

[0012] The first calling module is used to call a preset conversion function obtained by parsing the proto description file; the preset conversion function represents the mapping relationship between languages ​​determined based on the field types of the fields in the proto description file and the conversion constraint information between the languages ​​corresponding to the field types;

[0013] The first conversion module is used to convert the first language structure data into second language structure data according to the preset conversion function;

[0014] The serialization module is used to serialize the second language structure data to obtain protobuf serialized data.

[0015] Optionally, the first conversion module includes:

[0016] The first determining module is used to determine a first target conversion function among the preset conversion functions that matches the field type of the field in the first language structure data; the first target conversion function represents converting the first language into the second language;

[0017] The first conversion submodule is used to convert the first language structure data into second language structure data according to the first target conversion function.

[0018] Optionally, the language conversion constraint information includes first conversion constraint information corresponding to a nested field type and second conversion constraint information corresponding to an array field type;

[0019] Accordingly, the apparatus further includes a parsing module for obtaining preset transformation functions based on the proto description file, the parsing module comprising:

[0020] The second acquisition module is used to acquire the proto description file;

[0021] The first traversal module is used to traverse the message structure in the proto description file;

[0022] The preset conversion function generation module is used to obtain the preset conversion function based on the assignment function during the traversal process when the message structure traversal in the proto description file ends.

[0023] The first traversal module includes:

[0024] The second traversal module is used to traverse the fields in the message structure and obtain the field type of the current field;

[0025] The first generation module is used to generate an assignment function to assign a value to the current field when the field type of the current field is a basic type;

[0026] The third acquisition module is used to acquire the message name of the nested message in the current field and the first conversion limitation information when the field type of the current field is a nested type;

[0027] The second generation module is used to generate an assignment function that calls the nested message based on the message name of the nested message and the first conversion limitation information.

[0028] The third generation module is used to generate an assignment function based on the second conversion constraint information and the field types of the subfields contained in the current field when the field type of the current field is an array type.

[0029] Optionally, the third generation module includes:

[0030] The third traversal module is used to traverse the subfields contained in the current field and obtain the field type of the current subfield;

[0031] The fourth generation module is used to generate an assignment function to assign values ​​to the current subfield when the field type of the current subfield is a basic type.

[0032] The fourth acquisition module is used to acquire the message name and first conversion limitation information of the nested message in the current sub-field when the field type of the current sub-field is a nested type;

[0033] The fifth generation module generates an assignment function that calls the nested message based on the message name of the nested message and the first conversion limitation information.

[0034] Optionally, the first conversion constraint information includes an identifier field for identifying the usage status of the field of the nested type, and the second conversion constraint information includes a length field for pointing to the actual length of the field of the array type;

[0035] Accordingly, when the first conversion submodule converts the first language structure data into second language structure data according to the first target conversion function, for fields in the first language structure data that are of nested type, the first target conversion function determines the target to be converted based on the identifier field in the field and converts the target to be converted into second language structure data; for fields in the first language structure that are of array type, the first target conversion function determines the target to be converted based on the length field in the field and converts the target to be converted into second language structure data.

[0036] Optionally, the device further includes:

[0037] The fifth acquisition module is used to acquire the protobuf serialized data;

[0038] The deserialization module is used to deserialize the protobuf serialized data to obtain the second language structure data;

[0039] The second calling module is used to call the preset conversion function and determine a second target conversion function among the preset conversion functions that matches the field type of the field in the second language structure data; the second target conversion function represents converting the second language into the first language;

[0040] The second conversion module is used to convert the second language structure data into the first language structure data according to the second target conversion function.

[0041] Optionally, the first conversion limitation information includes a first field extension item corresponding to each field in the nested type, the first field extension item pointing to the identifier field name of the internal state; the second conversion limitation information includes a second field extension item corresponding to the array type field, the second field extension item being used to identify the maximum length of the array type field.

[0042] Accordingly, when the second conversion module converts the second language structure data into first language structure data according to the second target conversion function, for fields in the second language structure data whose field type is nested, it converts the target field containing the first field extension item into first language structure data based on the second target conversion function; for fields in the second language structure whose field type is array, it determines whether the length of the field exceeds the second field extension item based on the second target conversion function, and converts the field into first language structure data if the result of the determination is no.

[0043] Optionally, the device further includes:

[0044] The writing module is used to write the first language structure data into shared memory after converting the second language structure data into first language structure data according to the second target conversion function;

[0045] Accordingly, the first acquisition module is used to acquire the first language structure data obtained from the shared memory based on the proto description file.

[0046] On the other hand, a computer device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the above-described data processing method.

[0047] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the data processing method described above.

[0048] In this embodiment of the invention, the first language structure data includes fields in a proto description file represented in the first language, and language conversion constraint information (i.e., first language and second language) corresponding to the field types in the proto description file. A preset conversion function characterizes the language mapping relationship determined based on the field types in the proto description file and the corresponding language conversion constraint information. Therefore, after obtaining the C language structure data (i.e., the first language structure data), the preset conversion function can be called to convert the C language structure data into C++ language structure data (i.e., the second language structure data). Since Protobuf provides a serialization interface for C++ language structure data, the serialization interface in Protobuf can be called to serialize the C++ language structure data, obtaining Protobuf serialized data. In practical applications, this Protobuf serialized data can be transmitted, achieving both the use of Protobuf as a data exchange format to ensure uplink compatibility between the front-end and back-end of the protocol, and the use of shared memory in the service to enhance service stability and reliability, while also improving data transmission efficiency. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1A This is a schematic diagram illustrating an application scenario of the data processing method provided in this embodiment of the invention;

[0051] Figure 1B This is a schematic diagram illustrating another application scenario of the data processing method provided in this embodiment of the invention;

[0052] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0053] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention;

[0054] Figure 4 This is a flowchart illustrating an optional method for obtaining preset transformation functions based on parsing a Proto description file, as provided in an embodiment of the present invention.

[0055] Figure 5 This is a flowchart illustrating an optional method for generating a corresponding assignment function when the field type is an array, as provided in an embodiment of the present invention.

[0056] Figure 6 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention;

[0057] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of another data processing device provided in an embodiment of the present invention;

[0059] Figure 9 This is a hardware structure block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0062] Please see Figure 1A The diagram shows an application scenario of a data processing method provided by an embodiment of the present invention. The application scenario may include shared memory 100 and database 200, wherein shared memory 100 and database 200 are transmitted based on the Protobuf protocol.

[0063] In this context, shared memory 100 refers to a large amount of memory in a multiprocessor computer system that can be accessed by different central processing units (CPUs). It is a communication method between multiple processes under Unix, and multiple programs can also exchange information through shared memory. In this embodiment, the data in shared memory 100 is first-language structure data, which can be C language, and the first-language structure data is obtained by parsing a user-uploaded Proto description file. Shared memory 100 can be memory in a terminal or server.

[0064] Database 200 can be a relational database, a non-relational database, or a key-value database. Database 200 can store data in shared memory 100, and shared memory 100 can also read data from database 200 and write data into shared memory 100.

[0065] As an optional implementation, the data processing method of this invention can also be applied to... Figure 1BThe data sharing system 300 shown refers to a system for data sharing between nodes. This system may include multiple nodes 301, which can refer to various clients within the system. Each node 301 receives input information during normal operation and maintains shared data within the system based on this information. To ensure interoperability within the system, information connections exist between nodes, enabling information transmission. For example, when any node in the system receives input information, other nodes obtain this input information according to a consensus algorithm and store it as part of the shared data, ensuring consistency of data stored across all nodes.

[0066] Each node in the data sharing system has a corresponding node identifier, and each node can also store the node identifiers of other nodes in the system. This allows for the subsequent broadcasting of generated blocks to other nodes in the data sharing system based on their node identifiers. The node identifier can be an IP (Internet Protocol) address or any other information that can be used to identify the node. Each node in the data sharing system stores the same blockchain. A blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores input information feature values, version number, timestamp, and difficulty value, while the block body stores the input information. The next block after the genesis block takes the genesis block as its parent block. The next block also includes a block header and a block body. The block header stores the input information feature values ​​of the current block, the block header feature values ​​of the parent block, version number, timestamp, and difficulty value, and so on. This ensures that the block data stored in each block is related to the block data stored in the parent block, guaranteeing the security of the input information in the blocks.

[0067] Please see Figure 2 The diagram shown is a flowchart of a data processing method provided in an embodiment of the present invention. This method can be applied to... Figure 1A as well as Figure 1B In the application scenarios shown, it should be noted that this specification provides the method operation steps as described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one of many possible execution orders and does not represent the only execution order. In actual system or product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0068] S201, Obtain the first language structure data obtained from parsing the Proto description file.

[0069] The Proto description file is the definition file for Protobuf Messages (or simply Messages), which defines the message structure. It's important to note that a single Proto description file can define multiple message structures and can also reference Messages defined in other Proto description files.

[0070] For each Message structure, the Proto description file provides description information for at least one field that constitutes the Message structure. The description information for each field includes at least: field type, data type, field name, and field number.

[0071] Field types can include: basic types, nested types, and array types. A nested type refers to one message nested within another message; the field type of the nested message is the nested type. An array type is the repeated type, meaning that the corresponding field can be repeated any number of times, including zero times, and the order of the repeated data will be stored in the protocol buffer. Apart from the nested types and array types mentioned above, the remaining field types are all basic types, which may include, but are not limited to, required types, optional types, etc.

[0072] The data types are consistent with those of various programming languages, including common atomic scalar types such as int32, string, and float.

[0073] Field names are basically the same as variable naming conventions in languages ​​such as C, C++, and Java; for example, they can be named page, num, etc.

[0074] The field number serves as an identifier for the field, similar to a primary key in a database. It is unique and cannot be repeated. The identifier is used to identify the field in the compiled binary message format, that is, the field number will be used in serialization and deserialization. Its range is [1, 536870911], where 19000 to 19999 are reserved numbers.

[0075] In the embodiments of this specification, in order to realize the mutual conversion between first language structure data, such as C language structure data, and second language structure data, such as C++ language structure data, the Proto description file also includes language conversion constraint information corresponding to the field type of the field. The language can be a first language (such as C language) and a second language (such as C++ language). Based on the language conversion constraint information, the mutual conversion between first language structure data and second language structure data can be realized.

[0076] Specifically, the language conversion constraint information includes first conversion constraint information corresponding to nested field types and second conversion constraint information corresponding to array field types. The first conversion constraint information may include first field extensions corresponding to each field in the identifier field and nested field, while the second conversion constraint information may include a length field and second field extensions corresponding to array field types.

[0077] The identifier field is used to identify the usage status of fields in a nested type. Usage status refers to the field currently in use within the nested type. In other words, the identifier field can determine the currently used field within a nested type. For example, this identifier field could be (use_field) = "type". This identifier field is used when converting first-language structured data to second-language structured data to identify the currently used field in the nested type, thereby determining which field to use as the conversion target.

[0078] Nested fields can include multiple fields, each corresponding to a first field extension. This first field extension points to the identifier field name of the internal state. This internal state can be understood as the activation state of the field corresponding to the first field extension. In other words, the identifier field name determines whether the field corresponding to the first field extension is the active field, i.e., the currently used field. For example, in the field: String home_num = 100[(union_field_value) = "phone_home"], [(union_field_value) = "phone_home"] is the first field extension, and "phone_home" is the identifier field name. This first field extension is used to identify the currently used field in the first language structure data after the second language structure data is converted to first language structure data.

[0079] The length field is used to indicate the actual length of an array-type field. When converting first-language structured data to second-language structured data, the length field can be used to determine the actual array-type field to be converted, i.e., the target to be converted. For example, the length field could be (num_field) = "num".

[0080] The second field extension is used to identify the maximum length of an array-type field. In the embodiments of this specification, the size of an array-type field in the first language structure data, such as C, is fixed, while the size of an array-type field in the second language structure data, such as C++, is variable. Therefore, when converting second-language structure data to first-language structure data, for array-type fields, it is necessary to use the second field extension to determine whether the field length exceeds the maximum length of an array-type field in the first language structure data. Conversion can only proceed if it does not exceed the maximum length.

[0081] In the embodiments of this specification, the length field and the second field extension item can be added as extension content of the array type field through the custom option. Similarly, the first field extension item can also be added as extension content of the nested type field through the custom option.

[0082] Based on the above description, the following is a specific example of a Proto description file in an embodiment of the present invention. It should be understood that this example is used to more clearly understand the language conversion limitation information in the embodiments of this specification, and does not constitute any limitation on the present invention.

[0083]

[0084] In this example, `enum` indicates that the field type is an enumeration type. An enumeration type field can only use one value from a specified set of constants as its value. By adding an enumeration type field to the message structure, it is possible to specify a value from a "predefined sequence of values" for a field in the message structure. This example defines a message structure named `Book`, which contains a nested message named `author_tel_num`. This nested message is of type `one of`. In this embodiment, `oneof author_tel_num{}` from Example 1 is used as a nested type field in the message `Book`. A `one of` field is similar to an optional field, meaning that at most one field from `one of` can be set simultaneously.

[0085] In Example 1 above, (use_field) = "type" is the identifier field. The value of "type" indicates the usage status of oneof, i.e., which field in oneof is used. [(union_field_value) = "phone_home"] and [(union_field_value) = "phone_telphone"] are the first field extensions of the corresponding fields, where "phone_home" and "phone_telphone" are the identifier field names.

[0086]

[0087] This example defines a message structure named Book, where “repeated” indicates an array type, (num_field) = “num” is the length field, which points to the actual length of the array type field; (max_len_1) = max_array_size is the second field extension, which identifies the maximum length of the array type field.

[0088] In the embodiments of this specification, the first language structure data is obtained based on the parsing of a Proto description file. The parsing of the Proto description file can be performed using a Protocol Buffer compiler. The compiler will generate code in the selected first language, such as C language code. This code can manipulate the message structures defined in the Proto description file, including getting and setting field values. By parsing the Proto description file, the first language structure data is obtained, which includes the fields in the Proto description file represented in the first language, such as C, and the language conversion constraint information corresponding to the field types in the Proto description file. When the first language is C and the second language is C++, the language conversion constraint information between C and C++, represented in the first language and corresponding to the field types, can be obtained.

[0089] Based on the above description, before step S201, as follows: Figure 3 As shown, the data processing method may further include step 301, parsing the Proto description file to obtain first-language structure data. Specifically, the Proto description file can be parsed into first-language structure data using a Protocol Buffer compiler, where the first language can be C.

[0090] In a specific embodiment, the data in the shared memory is C language structure data parsed from the Proto description file. Therefore, step S201 specifically involves obtaining the C language structure data parsed from the shared memory based on the Proto description file.

[0091] S203, call the preset conversion function obtained by parsing the Proto description file.

[0092] The preset conversion function represents the mapping relationship between languages ​​determined based on the field types in the Proto description file and the corresponding language conversion constraint information. In other words, in this embodiment, the same Proto description file is parsed in two directions: one direction is to parse the first language structure data (such as C language structure data) in step S201, and the other direction is to parse the preset conversion function representing the mapping relationship between the first and second languages.

[0093] Based on the above description, before step S203, as follows: Figure 3 As shown, the data processing method may further include step S303, parsing the Proto description file to obtain a preset conversion function.

[0094] Specifically, methods for parsing Proto description files to obtain preset transformation functions can include... Figure 4 The steps shown are as follows:

[0095] S401, obtain the Proto description file.

[0096] S403, Traverse the message structure in the Proto description file. When traversing the message structure in the Proto description file, execute the following steps S405-S413.

[0097] S405, iterate through the fields in the message structure and get the field type of the current field;

[0098] S407, when the field type of the current field is a basic type, generate an assignment function to assign values ​​to the current field.

[0099] S409, when the field type of the current field is a nested type, obtain the message name of the nested message in the current field and the first conversion constraint information.

[0100] S411, Generate an assignment function that calls the nested message based on the message name of the nested message and the first conversion constraint information.

[0101] S413, when the field type of the current field is an array type, generate an assignment function based on the second conversion constraint information and the field types of the subfields contained in the current field.

[0102] In practical applications, step S413 can include... Figure 5 The following steps:

[0103] S501, iterate through the subfields contained in the current field and get the field type of the current subfield;

[0104] S503, when the field type of the current subfield is a basic type, generate an assignment function to assign values ​​to the current subfield;

[0105] S505, when the field type of the current subfield is a nested type, obtain the message name of the nested message in the current subfield and the first conversion restriction information;

[0106] S507, Based on the message name of the nested message and the first conversion constraint information, generate and call the assignment function corresponding to the nested message.

[0107] S415, at the end of the message structure traversal in the Proto description file, the preset conversion functions are obtained based on the assignment functions obtained during the traversal. That is, the set of assignment functions obtained during the traversal is used as the preset conversion functions.

[0108] S205, convert the first language structure data into second language structure data according to the preset conversion function.

[0109] Specifically, a first target conversion function can be determined that matches the field type of the fields in the first language structure among the preset conversion functions. This first target conversion function represents the conversion from the first language to the second language. For example, the first target conversion function represents the conversion from C language to C++ language. Then, the first language structure data is converted to second language structure data according to the first target conversion function.

[0110] In practical applications, when the first conversion constraint information includes an identifier field to identify the usage status of a nested type field, and the second conversion constraint information includes a length field to point to the actual length of an array type field, when converting first language structure data to second language structure data according to the first target conversion function, for fields in the first language structure data that are of nested type, the first target conversion function determines the target to be converted based on the identifier field in that field and converts the target to be converted into second language structure data; for fields in the first language structure data that are of array type, the first target conversion function determines the target to be converted based on the length field in that field and converts the target to be converted into second language structure data.

[0111] Taking C as the first language and C++ as the second language as an example, when converting C language structured data to C++ language structured data, for nested type fields, the identifier field can be used to determine the fields used in the nested messages such as the aforementioned one of, and the fields used in one of are assigned to C++ through the assignment function; for array type fields, the length field can be used to determine the actual length of the field, and the actual length field is assigned to C++ through the assignment function.

[0112] S207 serializes the second language structure data to obtain Protobuf serialized data.

[0113] In the embodiments described in this specification, the second language is a language that provides a serialization interface for Protobuf and can be used in combination with the first language. For example, C language (first language) and C++ language (second language). Protobuf provides a serialization interface for C++ language structured data, and C language and C++ language can be used in combination.

[0114] Serialization refers to converting structured data or objects into a format that can be stored and transmitted (e.g., over a network), while ensuring that the serialized result can be reconstructed from the original structured data or object later (potentially in another computing environment). Specifically, second-language structured data can be serialized using the serialization interface (API) provided by the Protobuf language library (such as the C++ library) to obtain the corresponding Protobuf serialized data.

[0115] Unlike XML and JSON serialization, Protobuf uses binary byte serialization. It calculates the mapping between fields using field numbers and data types, achieving higher time and space efficiency. This makes it particularly suitable for applications sensitive to data size and transmission rate. In other words, Protobuf represents message structures as key-value pairs, where the key is obtained by calculating the field number and data type of each field in each message.

[0116] In practical applications, after obtaining the Protobuf serialized data, the Protobuf serialized data can be transmitted over the network, for example, sent to a database via shared memory, so that the corresponding data can be stored in the database.

[0117] Please see Figure 6 The diagram shown is a flowchart of another data processing method provided by an embodiment of the present invention. In this method, in Figure 2 The method embodiment shown may further include the following steps after step S207:

[0118] S601, obtain Protobuf serialized data;

[0119] Specifically, Protobuf serialized data can be obtained from a database.

[0120] S603, deserializes the Protobuf serialized data to obtain second language structure data;

[0121] Specifically, Protobuf serialized data, i.e. binary data, can be deserialized into language structure data corresponding to the language library through the deserialization interface (API) provided in the Protobuf language library (such as the C++ library). For example, Protobuf serialized data can be deserialized into C++ language structure data.

[0122] S605, call the preset conversion function to determine the second target conversion function that matches the field type of the field in the second language structure data.

[0123] The second objective transformation function represents the conversion from a second language to a first language. For example, the second objective transformation function represents the conversion from C++ to C.

[0124] S607, convert the second language structure data into the first language structure data according to the second target conversion function.

[0125] Specifically, when the first transformation constraint information in the Proto description file includes first field extensions corresponding to each field in a nested type field, and these first field extensions point to the identifier field name of the internal state, and the second transformation constraint information includes second field extensions corresponding to an array type field, and these second field extensions are used to identify the maximum length of the array type field, in step S607, for fields of nested type in the second language structure data, the target field containing the first field extension can be converted into first language structure data based on the second target transformation function. The target field is the field actually used in the nested type field. Since the target field contains the first field extension, after converting the target field into first language structure data, the first language can determine the field actually used in the nested type field based on the first field extension. For fields of array type in the second language structure, the second target transformation function is used to determine whether the field length exceeds the second field extension, and if the result is negative, the field is converted into first language structure data.

[0126] Taking C as the first language and C++ as the second language as an example, when converting C++ structured data to C structured data, for nested fields, the C structured data can know the field assigned by C++ based on the first field extension item; for array type fields, it can ensure that the actual length of the array type field in the C structured data does not exceed the fixed maximum length.

[0127] In practical applications, after converting C++ language structure data into C language structure data, the C language structure data can also be written into shared memory to facilitate the reading of data in shared memory by various processes.

[0128] As can be seen from the above technical solutions of the embodiments of the present invention, the embodiments of the present invention use first language structure data, including fields in a proto description file represented in the first language, and language conversion constraint information (i.e., first language and second language) corresponding to the field types in the proto description file. A preset conversion function characterizes the mapping relationship between languages ​​determined based on the field types in the proto description file and the language conversion constraint information corresponding to the field types. Therefore, after obtaining C language structure data (i.e., first language structure data), the preset conversion function can be called to convert the C language structure data into C++ language structure data (i.e., second language structure data). Since Protobuf provides a serialization interface for C++ language structure data, the serialization interface in Protobuf can be called to serialize the C++ language structure data to obtain Protobuf serialized data. In practical applications, this Protobuf serialized data can be transmitted, achieving both the use of Protobuf as a data exchange format to ensure uplink compatibility between the front-end and back-end of the protocol, and the use of shared memory in the service to enhance service stability and reliability, while also improving data transmission efficiency.

[0129] Corresponding to the data processing methods provided in the above embodiments, this embodiment of the invention also provides a data processing apparatus. Since the data processing apparatus provided in this embodiment of the invention corresponds to the data processing methods provided in the above embodiments, the implementation methods of the aforementioned data processing methods are also applicable to the data processing apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0130] Please see Figure 7 The diagram shown is a structural schematic of a data processing device provided in an embodiment of the present invention. This device has the function of implementing the data processing method described in the above-described method embodiments. This function can be implemented by hardware or by hardware executing corresponding software. Figure 7 As shown, the device may include:

[0131] The first acquisition module 710 is used to acquire first language structure data obtained by parsing a proto description file. The first language structure data includes fields in the proto description file represented in the first language, and language conversion constraint information corresponding to the field types of the fields in the proto description file. The languages ​​include the first language and the second language.

[0132] The first calling module 720 is used to call a preset conversion function obtained by parsing the proto description file; the preset conversion function represents the mapping relationship between languages ​​determined based on the field types of the fields in the proto description file and the conversion constraint information between the languages ​​corresponding to the field types;

[0133] The first conversion module 730 is used to convert the first language structure data into second language structure data according to the preset conversion function;

[0134] The serialization module 740 is used to serialize the second language structure data to obtain protobuf serialized data.

[0135] As an optional implementation, the first conversion module 730 may include:

[0136] The first determining module is used to determine a first target conversion function among the preset conversion functions that matches the field type of the field in the first language structure data; the first target conversion function represents converting the first language into the second language;

[0137] The first conversion submodule is used to convert the first language structure data into second language structure data according to the first target conversion function.

[0138] In one specific implementation, the inter-language conversion constraint information includes first conversion constraint information corresponding to a nested field type and second conversion constraint information corresponding to an array field type;

[0139] Accordingly, the above-mentioned apparatus may further include a parsing module for obtaining preset transformation functions based on the proto description file, the parsing module including:

[0140] The second acquisition module is used to acquire the proto description file;

[0141] The first traversal module is used to traverse the message structure in the proto description file;

[0142] The preset conversion function generation module is used to obtain the preset conversion function based on the assignment function during the traversal process when the message structure traversal in the proto description file ends.

[0143] The first traversal module may include:

[0144] The second traversal module is used to traverse the fields in the message structure and obtain the field type of the current field;

[0145] The first generation module is used to generate an assignment function to assign a value to the current field when the field type of the current field is a basic type;

[0146] The third acquisition module is used to acquire the message name of the nested message in the current field and the first conversion limitation information when the field type of the current field is a nested type;

[0147] The second generation module is used to generate an assignment function that calls the nested message based on the message name of the nested message and the first conversion limitation information.

[0148] The third generation module is used to generate an assignment function based on the second conversion constraint information and the field types of the subfields contained in the current field when the field type of the current field is an array type.

[0149] In one specific implementation, the third generation module may include:

[0150] The third traversal module is used to traverse the subfields contained in the current field and obtain the field type of the current subfield;

[0151] The fourth generation module is used to generate an assignment function to assign values ​​to the current subfield when the field type of the current subfield is a basic type.

[0152] The fourth acquisition module is used to acquire the message name and first conversion limitation information of the nested message in the current sub-field when the field type of the current sub-field is a nested type;

[0153] The fifth generation module generates an assignment function that calls the nested message based on the message name of the nested message and the first conversion limitation information.

[0154] In one specific implementation, the first conversion constraint information includes an identifier field for identifying the usage status of the field of the nested type, and the second conversion constraint information includes a length field for pointing to the actual length of the array type field.

[0155] Accordingly, when the first conversion submodule converts the first language structure data into second language structure data according to the first target conversion function, for fields in the first language structure data that are of nested type, the first target conversion function determines the target to be converted based on the identifier field in the field and converts the target to be converted into second language structure data; for fields in the first language structure that are of array type, the first target conversion function determines the target to be converted based on the length field in the field and converts the target to be converted into second language structure data.

[0156] As an optional implementation, such as Figure 8 A flowchart illustrating another data processing method is provided, and the apparatus may further include:

[0157] The fifth acquisition module 750 can be used to acquire the protobuf serialized data;

[0158] The deserialization module 760 can be used to deserialize the protobuf serialized data to obtain the second language structure data;

[0159] The second calling module 770 can be used to call the preset conversion function and determine a second target conversion function among the preset conversion functions that matches the field type of the field in the second language structure data; the second target conversion function represents converting the second language into the first language;

[0160] The second conversion module 780 can be used to convert the second language structure data into the first language structure data according to the second target conversion function.

[0161] In one specific implementation, the first conversion constraint information includes a first field extension item corresponding to each field in the nested type, the first field extension item pointing to the identifier field name of the internal state; the second conversion constraint information includes a second field extension item corresponding to the array type field, the second field extension item being used to identify the maximum length of the array type field.

[0162] Correspondingly, when the second conversion module 780 converts the second language structure data into first language structure data according to the second target conversion function, for fields in the second language structure data whose field type is nested, it converts the target field containing the first field extension item into first language structure data based on the second target conversion function; for fields in the second language structure whose field type is array, it determines whether the length of the field exceeds the second field extension item based on the second target conversion function, and converts the field into first language structure data if the result of the determination is negative.

[0163] As an optional implementation, the device may further include:

[0164] The writing module is used to write the first language structure data into shared memory after converting the second language structure data into first language structure data according to the second target conversion function;

[0165] Correspondingly, the first acquisition module 710 can be used to acquire the first language structure data obtained from the shared memory based on the proto description file.

[0166] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0167] The data processing device of this invention can use Protobuf as the data exchange format of the protocol to ensure the uplink compatibility of the front-end and back-end of the protocol, and can use shared memory in the service to enhance the stability and reliability of the service, while improving the data transmission efficiency.

[0168] This invention provides a computer device, the terminal including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data processing method provided in the above method embodiments.

[0169] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0170] The method embodiments provided in this invention can be executed in a computer terminal, server or similar computing device, that is, the computer device mentioned above may include a computer terminal, server or similar computing device. Figure 9 This is a hardware structure block diagram of a computer device for running a data processing method provided in an embodiment of the present invention, such as... Figure 9 As shown, the internal structure of this computer device may include, but is not limited to, a processor, a network interface, and a memory. The processor, network interface, and memory within the computer device can be connected via a bus or other means, as illustrated in the embodiments of this specification. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0171] The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device. The network interface may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.). Memory is the storage device in the computer device used to store programs and data. It is understood that the memory here can be a high-speed RAM storage device, or a non-volatile storage device, such as at least one disk storage device; optionally, it can also be at least one storage device located remotely from the aforementioned processor. The memory provides storage space, which stores the operating system of the electronic device, including but not limited to: Windows (an operating system), Linux (an operating system), Android (a mobile operating system), iOS (a mobile operating system), etc., which are not limited in this invention; and the storage space also stores one or more instructions suitable for being loaded and executed by the processor, which can be one or more computer programs (including program code). In the embodiments of this specification, the processor loads and executes one or more instructions stored in the memory to implement the data processing method provided in the above method embodiments.

[0172] Embodiments of the present invention also provide a computer-readable storage medium that can be disposed in a computer device to store at least one instruction or at least one program related to implementing a data processing method, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided in the above-described method embodiments.

[0173] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0174] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0176] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: First language structure data parsed from a Proto description file is obtained from shared memory. This first language structure data includes fields in the Proto description file represented in the first language, and language conversion constraint information corresponding to the field types in the Proto description file. The languages ​​include a first language and a second language. The language conversion constraint information includes first conversion constraint information corresponding to nested field types and second conversion constraint information corresponding to array field types. The first conversion constraint information includes an identifier field indicating the usage status of the nested field; the second conversion constraint information includes a length field pointing to the actual length of the array field. Obtain the Proto description file; traverse the message structure in the Proto description file, performing the following steps during the traversal: traverse the fields in the message structure and obtain the field type of the current field; if the field type of the current field is a basic type, generate an assignment function to assign values ​​to the current field; if the field type of the current field is a nested type, obtain the message name of the nested message in the current field and the first conversion constraint information; generate an assignment function to call the nested message based on the message name of the nested message and the first conversion constraint information; if the field type of the current field is an array type, generate an assignment function based on the second conversion constraint information and the field types of the subfields contained in the current field; when the traversal of the message structure in the Proto description file ends, obtain a preset conversion function based on the assignment functions during the traversal process. Invoke the preset conversion function obtained by parsing the Proto description file; the preset conversion function represents the mapping relationship between languages ​​determined based on the field types of the fields in the Proto description file and the conversion constraint information between the languages ​​corresponding to the field types; A first target conversion function is determined from the preset conversion functions, matching the field type of the field in the first language structure data. This first target conversion function represents the conversion of the first language to the second language. For fields in the first language structure data of nested type, the first target conversion function determines the target to be converted based on the identifier field in the field and converts the target to be converted into second language structure data. For fields in the first language structure data of array type, the first target conversion function determines the target to be converted based on the length field in the field and converts the target to be converted into second language structure data. Protobuf provides a serialization interface for the second language structure data. The second language structure data is serialized by calling the serialization interface in Protobuf to obtain Protobuf serialized data.

2. The data processing method according to claim 1, characterized in that, When the field type of the current field is an array type, generating an assignment function based on the second conversion constraint information and the field types of the subfields contained in the current field includes: Iterate through the subfields contained in the current field and obtain the field type of the current subfield; When the field type of the current subfield is a basic type, generate an assignment function to assign a value to the current subfield; When the field type of the current subfield is a nested type, obtain the message name of the nested message in the current subfield and the first conversion limitation information; Based on the message name of the nested message and the first conversion constraint information, an assignment function corresponding to the nested message is generated and invoked.

3. The data processing method according to claim 1, characterized in that, The method further includes: Obtain the Protobuf serialized data; The Protobuf serialized data is deserialized to obtain the second language structure data; The preset conversion function is invoked to determine a second target conversion function that matches the field type of the field in the second language structure data; the second target conversion function represents the conversion of the second language to the first language; The second language structure data is converted into the first language structure data according to the second target conversion function.

4. The data processing method according to claim 3, characterized in that, The first conversion constraint information includes a first field extension item corresponding to each field in the nested type, the first field extension item pointing to the identifier field name of the internal state; the second conversion constraint information includes a second field extension item corresponding to the array type field, the second field extension item being used to identify the maximum length of the array type field. Accordingly, converting the second language structure data into first language structure data according to the second target conversion function includes: For fields of nested type in the second language structure data, the target field containing the first field extension item is converted into the first language structure data based on the second target conversion function; For a field in the second language structure whose field type is array, the length of the field is determined based on the second target conversion function to see if it exceeds the second field extension item. If the result of the determination is negative, the field is converted into data of the first language structure.

5. The data processing method according to claim 3, characterized in that, After converting the second language structure data into first language structure data according to the second target conversion function, the method further includes: Write the first language structure data into shared memory.

6. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire first language structure data parsed from a Proto description file from shared memory. The first language structure data includes fields in the Proto description file represented in the first language, and language conversion constraint information corresponding to the field types of the fields in the Proto description file. The languages ​​include a first language and a second language. The language conversion constraint information includes first conversion constraint information corresponding to a nested field type and second conversion constraint information corresponding to an array field type. The first conversion constraint information includes an identifier field for identifying the usage status of the nested field, and the second conversion constraint information includes a length field for pointing to the actual length of the array field. The parsing module includes: a second acquisition module for acquiring the Proto description file; a first traversal module for traversing the message structure in the Proto description file; and a preset conversion function generation module for obtaining a preset conversion function based on the assignment function during the traversal process when the message structure traversal in the Proto description file ends. The first traversal module includes: a second traversal module for traversing the fields in the message structure and acquiring the field type of the current field; a first generation module for generating an assignment function to assign values ​​to the current field when the field type of the current field is a basic type; a third acquisition module for acquiring the message name of the nested message in the current field and the first conversion constraint information when the field type of the current field is a nested type; a second generation module for generating an assignment function to call the nested message based on the message name of the nested message and the first conversion constraint information; and a third generation module for generating an assignment function based on the second conversion constraint information and the field types of the subfields contained in the current field when the field type of the current field is an array type. The first calling module is used to call a preset conversion function obtained by parsing the Proto description file; the preset conversion function represents the mapping relationship between languages ​​determined based on the field types of the fields in the Proto description file and the conversion constraint information between the languages ​​corresponding to the field types; A first conversion module is used to convert the first language structure data into second language structure data according to the preset conversion function. The first conversion module includes: a first determining module, used to determine a first target conversion function among the preset conversion functions that matches the field type of a field in the first language structure data; the first target conversion function represents converting the first language into the second language; a first conversion submodule, used to convert the first language structure data into second language structure data according to the first target conversion function; when the first conversion submodule converts the first language structure data into second language structure data according to the first target conversion function, for fields in the first language structure data with a nested field type, the first target conversion function determines the target to be converted based on the identifier field in the field and converts the target to be converted into second language structure data; for fields in the first language structure data with an array field type, the first target conversion function determines the target to be converted based on the length field in the field and converts the target to be converted into second language structure data; Protobuf provides a serialization interface for the second language structure data. The serialization module is used to call the serialization interface in Protobuf to serialize the second language structure data and obtain Protobuf serialized data.

7. The apparatus according to claim 6, characterized in that, The third generation module includes: The third traversal module is used to traverse the subfields contained in the current field and obtain the field type of the current subfield; The fourth generation module is used to generate an assignment function to assign values ​​to the current subfield when the field type of the current subfield is a basic type. The fourth acquisition module is used to acquire the message name and first conversion limitation information of the nested message in the current sub-field when the field type of the current sub-field is a nested type; The fifth generation module generates an assignment function that calls the nested message based on the message name of the nested message and the first conversion limitation information.

8. The apparatus according to claim 6, characterized in that, The device further includes: The fifth acquisition module is used to acquire the Protobuf serialized data; The deserialization module is used to deserialize the Protobuf serialized data to obtain the second language structure data; The second calling module is used to call the preset conversion function and determine a second target conversion function among the preset conversion functions that matches the field type of the field in the second language structure data; the second target conversion function represents converting the second language into the first language; The second conversion module is used to convert the second language structure data into the first language structure data according to the second target conversion function.

9. The apparatus according to claim 8, characterized in that, The first conversion constraint information includes a first field extension item corresponding to each field in the nested type, the first field extension item pointing to the identifier field name of the internal state; the second conversion constraint information includes a second field extension item corresponding to the array type field, the second field extension item being used to identify the maximum length of the array type field. Accordingly, when the second conversion module converts the second language structure data into first language structure data according to the second target conversion function, for fields in the second language structure data whose field type is nested, it converts the target field containing the first field extension item into first language structure data based on the second target conversion function; for fields in the second language structure whose field type is array, it determines whether the length of the field exceeds the second field extension item based on the second target conversion function, and converts the field into first language structure data if the result of the determination is no.

10. The apparatus according to claim 8, characterized in that, The device further includes: The writing module is used to write the first language structure data into shared memory after converting the second language structure data into first language structure data according to the second target conversion function.

11. A computer-readable storage medium storing at least one instruction or at least one program, said at least one instruction or said at least one program being loaded and executed by a processor to implement the data processing method as described in any one of claims 1 to 5.

12. A computer device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data processing method as claimed in any one of claims 1 to 5.

Images