Universal serialization method, device and equipment and readable storage medium
By directly loading memory and generating common serialization interface code, the problem of inefficient serialization in the existing technology is solved, and the support and efficiency improvement of cross-core communication is achieved.
Patent Information
- Application Number
- CN202410969774.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-19
AI Technical Summary
The existing serialization methods are less efficient in cross-protocol, cross-core and cross-platform communication architectures. In particular, Protobuf does not support C language, resulting in the need to call other interfaces, which increases the complexity and efficiency of serialization.
By obtaining the original file, parsing structured data, and directly loading it into memory, a general serialized interface code is generated based on the serialized code model, avoiding data copying, supporting SOC and MCU cross-core communication, no need to call other interfaces, and improving execution efficiency.
It realizes that data is directly mapped to memory and serialized through zero copy method, improves the execution efficiency of serialization, and supports cross-core communication, solving the problem of inefficiency in the existing technology.
Smart Images

Figure CN118981487A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of serialization technology, and in particular to a universal serialization method, apparatus, device and readable storage medium. Background Art
[0002] Serialization is the process of converting the state information of an object into a form that can be stored or transmitted, so serialization is crucial in communication development.
[0003] Each programming language has its own corresponding serialization tool. For systems with large memory and processing power, Protobuf is usually used for serialization. Although Protobuf plays an important role in C++ projects, it does not support C language and cannot meet the requirements in cross-protocol, cross-core and cross-platform communication architectures. Using other tools requires calling other interfaces, which leads to low serialization execution efficiency.
[0004] Therefore, how to improve the execution efficiency of serialization is a problem that technical personnel in this field need to solve urgently. Summary of the invention
[0005] Based on the above problems, the present application provides a general serialization method, apparatus, device and readable storage medium, which directly maps data to memory. The serialization code model directly operates the memory address to read data, avoids data copying, and serializes the structured data of the original file in a zero-copy manner. The generated serialization code supports cross-core communication between SOC and MCU, without calling other interfaces, thereby improving execution efficiency.
[0006] In a first aspect, an embodiment of the present application provides a general serialization method, including:
[0007] Get the original file of the serialization code to be generated;
[0008] Parsing the original file and determining each structured data corresponding to the original file;
[0009] The structured data is loaded into the memory, and a universal serialization interface code is generated for each structured data corresponding to the original file based on a serialization code model; the universal serialization interface code can be used for cross-core communication.
[0010] Optionally, obtaining the original file of the serialization code to be generated includes:
[0011] Obtaining file parameters input by the user; the file parameters include the original file address;
[0012] An original file corresponding to the original file address is acquired based on the original file address.
[0013] Optionally, the file parameters also include: serialization type;
[0014] The general serialization method also includes:
[0015] If the serialization type specifies the Protobuf method, the Protoc command is executed to generate the Protobuf serialization code for the original file.
[0016] Optionally, the file parameters also include: a storage location for the generated file;
[0017] The general serialization method also includes:
[0018] Create a storage folder at the storage location;
[0019] The generated universal serialization interface code is saved in the storage folder.
[0020] Optionally, the generating of a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model includes:
[0021] Get the description information of each structured data;
[0022] Based on the description information, a universal serialization interface code is generated for each structured data through a serialization code model.
[0023] Optionally, before generating a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model, the method further includes:
[0024] The arrangement order of each structured data is adjusted based on the nested relationship so that the dependent structured data is prioritized.
[0025] Optionally, the method further includes:
[0026] Traversing a file list; the file list includes all original files to be generated into serialized codes;
[0027] If serialization code generation has been completed for all original files in the file list, the file list is marked.
[0028] In a second aspect, an embodiment of the present application provides a general serialization device, including:
[0029] An acquisition module is used to obtain the original file of the serialization code to be generated;
[0030] A parsing module, used for parsing the original file and determining each structured data corresponding to the original file;
[0031] A serialization module is used to load the structured data into memory and generate a universal serialization interface code for each structured data corresponding to the original file based on a serialization code model; the universal serialization interface code can be used for cross-core communication.
[0032] In a third aspect, an embodiment of the present application provides a universal serialization device, including:
[0033] Memory for storing computer programs;
[0034] A processor is used to implement the steps of the general serialization method as described above when executing the computer program.
[0035] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the general serialization method as described above are implemented.
[0036] It can be seen from the above technical solutions that compared with the prior art, the present application has the following advantages:
[0037] This application first obtains the original file of the serialization code to be generated.Then parse the original file, and determine the various structured data corresponding to the original file.Finally, the structured data is loaded into the memory, and a general serialization interface code is generated for each structured data corresponding to the original file based on the serialization code model, and the generated general serialization interface code can be used for cross-core communication.In this way, the data is directly mapped to the memory, and the serialization code model directly operates the memory address to read the data, avoids data copying, and serializes the structured data of the original file in a zero-copy manner, and the generated serialization code supports SOC and MCU cross-core communication, without calling other interfaces, and improves execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A flowchart of a general serialization method provided in an embodiment of the present application;
[0039] Figure 2 A schematic diagram of the association relationship of dependent components provided in an embodiment of the present application;
[0040] Figure 3 An execution flow chart of a serialization code generation tool provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of the structure of a general serialization device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] As mentioned above, existing serialization methods have the problem of low execution efficiency. Specifically, each programming language has its corresponding serialization tool. Protobuf is designed for systems with large memory and processing power. As an excellent and universal serialization method, it is of great use to C++ projects. However, Protobuf does not support C language and cannot meet the requirements in cross-protocol, cross-core and cross-platform communication architectures. It needs to call other interfaces and use other matching tools for serialization, which leads to the problem of low serialization execution efficiency.
[0043] To solve the above problems, an embodiment of the present application provides a general serialization method, including: first obtaining the original file for which the serialization code is to be generated. Then parsing the original file and determining each structured data corresponding to the original file. Finally, loading the structured data into the memory, and generating a general serialization interface code for each structured data corresponding to the original file based on the serialization code model, the generated general serialization interface code can be used for cross-core communication.
[0044] In this way, the data is directly mapped to the memory, and the serialization code model directly operates the memory address to read the data, avoiding data copying. The structured data of the original file is serialized in a zero-copy manner, and the generated serialization code supports cross-core communication between SOC and MCU without calling other interfaces, thereby improving execution efficiency.
[0045] It should be noted that the general serialization method, device, equipment and readable storage medium provided in the embodiments of the present application can be applied to the field of serialization technology. The above is only an example and does not limit the application field of the general serialization method, device, equipment and readable storage medium provided in the present application.
[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] Figure 1 A flowchart of a general serialization method provided in an embodiment of the present application. Figure 1 As shown, a general serialization method provided in an embodiment of the present application may include:
[0048] S101: Obtain the original file of the serialization code to be generated.
[0049] In practical applications, serialization is the process of converting the state information of an object into a form that can be stored or transmitted. The original file for which the serialization code is to be generated refers to the Proto file that needs to be serialized. Proto files usually refer to the file format of Protocol Buffer, which is used to serialize structured data.
[0050] In addition, since the methods of obtaining the original file are not the same, the embodiment of the present application can illustrate a possible acquisition method.
[0051] In one embodiment, obtaining the original file of the serialization code to be generated includes:
[0052] Obtaining file parameters input by the user; the file parameters include the original file address;
[0053] An original file corresponding to the original file address is acquired based on the original file address.
[0054] In practical applications, the present application provides a serialization code generation tool that interacts with users through a common interface and supports command line input of Proto file parameters. The user can input Proto file parameters to the serialization code generation tool in the form of a command line, which is received by the serialization code generation tool to perform the serialization process. The Proto file parameters may include a variety of parameters, such as the Proto file address, the save location of the generated file, and the serialization type, etc. When the Proto file parameter input by the user includes the Proto file address, the serialization code generation tool can parse the file parameters input by the user, extract the Proto file address therein, and obtain the corresponding Proto file from the file address, and then perform the serialization process.
[0055] S102: Parse the original file and determine each structured data corresponding to the original file.
[0056] In practical applications, Proto files can define a variety of data structure types, including basic data types (such as integer types, fixed-size integer types, floating-point types, Boolean types, and string types, etc.), composite data types, and user-defined data types. The serialization code generation tool provided in the embodiment of the present application needs to parse the Proto file to be serialized, and determine all the structured data (message data) therein, and then generate a unified serialization interface code for each message data. Message data is a custom data structure that can include multiple fields, each field has its own type and identifier (such as basic data type fields, enumeration type fields, nested message type fields, repeated fields, mapping type fields, Any type fields, and Bytes type fields, etc.).
[0057] S103: Load the structured data into the memory, and generate a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model; the universal serialization interface code can be used for cross-core communication.
[0058] In practical applications, the serialization code model uses no-copy technology to avoid unnecessary data duplication during data processing and transmission, thereby improving efficiency and performance. Share between different functions or modules through references or pointers, rather than creating copies of data. Specifically, the embodiment of the present application directly maps the message data to the memory, and then generates a unified serialization interface code (i.e., binary serialization method) for each message data through the serialization code model, such as the msg.hpp file. A possible execution command line for serialization by the serialization code model is as follows:
[0059] sf-msg-tool.exe
[0060] -o . / out
[0061] -p'binary,protobuf'. / msg.proto
[0062] Among them, the -t parameter is the serialization type to be supported; the -o parameter is the location to save the generated serialization code file; and the last is the Proto file name parameter, which can be multiple. In this way, the generated universal serialization interface code can be used for cross-core communication.
[0063] In addition, since serialization methods are not the same, the embodiments of the present application can illustrate a possible serialization method.
[0064] In one case, the generating of a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model includes:
[0065] Get the description information of each structured data;
[0066] Based on the description information, a universal serialization interface code is generated for each structured data through a serialization code model.
[0067] In practical applications, message data usually refers to the information unit transmitted between network communications, applications or systems. Its description information may include data type, structure, size, format, purpose, security and transmission method, etc. Among them, the data type includes multiple formats such as text, numbers, media files and binary data. The description information of the message data corresponding to different Proto files is different, and the general serialization interface code obtained through the serialization code model is also different. Therefore, to serialize a certain Proto file, it is necessary to pass the description information of the message data corresponding to the Proto file to the serialization code model, and the serialization code model generates the binary serialization code of the Proto file message, that is, the general serialization interface code.
[0068] In addition, since the order in which structured data is serialized is not the same, the embodiment of the present application can illustrate a possible sorting method.
[0069] In one case, before generating a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model, the method further includes:
[0070] The arrangement order of each structured data is adjusted based on the nested relationship so that the dependent structured data is prioritized.
[0071] In practical applications, a Proto file can contain multiple message data, and a type of message data can contain fields of other types of message data, thus forming a nested relationship. Proto files allow complex structure definitions, so this nesting can be done in multiple layers. When serializing such Proto files, all message data need to be sorted. The embodiment of the present application provides a means of sorting message data based on the nested relationship, which puts the dependent message data in front, and then passes the description information of the message data to the serialization code model in sequence according to this arrangement order, and generates the corresponding binary serialization code.
[0072] In addition, since the ways of completing the serialization of the mark are not the same, the embodiment of the present application can illustrate a possible mark.
[0073] In one embodiment, the method further comprises:
[0074] Traversing a file list; the file list includes all original files to be generated into serialized codes;
[0075] If serialization code generation has been completed for all original files in the file list, the file list is marked.
[0076] In actual applications, users can input Proto file parameters into the serialization code generation tool through a universal interface. The Proto file parameters input by the user at one time may indicate multiple Proto files that need to be serialized, so there are multiple Proto file addresses. The serialization code generation tool can obtain all Proto files according to the Proto file address, and these files will be recorded in the form of a file list and serialized as the same batch. This task is considered to be completed only when all Proto files indicated in the file list have been serialized. Therefore, the serialization code generation tool provided in the embodiment of the present application needs to traverse the file list. If the traversal result is that there are Proto files in the file list that have not been serialized, the serialization process continues to be executed until all Proto files in the file list are serialized; if the traversal result is that all Proto files in the file list have been serialized, the serialization code generation process is terminated and the file list is marked with a "completed" logo.
[0077] To summarize, first obtain the original file of the serialization code to be generated. Then parse the original file and determine the various structured data corresponding to the original file. Finally, load the structured data into the memory, and generate a general serialization interface code for each structured data corresponding to the original file based on the serialization code model. The generated serialization interface code can be used for cross-core communication. In this way, the data is directly mapped to the memory, and the serialization code model directly operates the memory address to read the data, avoiding data copying, and serializing the original file in a zero-copy manner. The generated serialization code supports SOC and MCU cross-core communication, without calling other interfaces, which improves execution efficiency.
[0078] In addition, since different file parameters correspond to different serialization methods, the embodiment of the present application can illustrate a possible serialization method.
[0079] In one embodiment, the file parameters further include: a serialization type;
[0080] The general serialization method also includes:
[0081] If the serialization type specifies the Protobuf method, the Protoc command is executed to generate the Protobuf serialization code for the original file.
[0082] In practical applications, users may expect to use other serialization tools for serialization, such as Protoc tools, etc. In order not to introduce redundant interfaces (generally each tool has an interactive interface with the user), the serialization code generation tool provided in the embodiment of the present application can directly call other serialization methods by parsing the Proto file parameters input by the user. Specifically, the serialization code generation tool provided in the embodiment of the present application only provides a general interface for the user, and the user can input the Proto file parameters to the serialization code generation tool through the general interface, and the serialization type can be included in the file parameter. For example, if the serialization type specifies the Protobuf method, the serialization code generation tool can directly execute the Protoc command to implement calling the Protoc tool to perform Protobuf method serialization, and generate the Protobuf serialization code corresponding to the Proto file, without introducing a new interface. It should be noted that whether or not the serialization type is specified, it is necessary to generate a general serialization interface code through the binary serialization method.
[0083] The Proto file parameters entered by the user can also include the serialization type.
[0084] In addition, since different file parameters correspond to different serialization methods, the embodiment of the present application can illustrate a possible serialization method.
[0085] In one case, the file parameters also include: a storage location for the generated file;
[0086] The general serialization method also includes:
[0087] Create a storage folder at the storage location;
[0088] The generated universal serialization interface code is saved in the storage folder.
[0089] In actual applications, the save location of the generated file can be specified by the user. Specifically, the user can write the save location of the generated file in the input Proto file parameters, and then input the save location file parameters including the generated file into the serialization code generation tool, the serialization code generation tool parses the file parameters, creates a storage folder in the save location written by the user, and then saves the general serialization interface code generated by the serialization code generation tool in the storage folder.
[0090] In summary, Figure 2A schematic diagram of the association relationship of dependent components provided in an embodiment of the present application. Figure 2 As shown, the user enters the Proto file parameters, and the serialization code generation tool (sf-msg-tool) obtains and parses the corresponding Proto file (msg.proto file). If the file parameters indicate the use of Protobuf, the Protoc command is executed, and the Protoc tool is called to generate Protobuf's own serialization code (Protobuf serialization code), such as protobuf_msg.h or protobuf_msg.cc. At the same time, binary serialization code is generated, such as msg.hpp file. MessageBus business can store or transmit structured data by calling the general serialization interface.
[0091] Figure 3 This is a flowchart of the execution of a serialization code generation tool provided in an embodiment of the present application. Figure 3 As shown, the user inputs the Proto file parameters to the serialization code generation tool, and the serialization code generation tool parses the parameter list and creates an output folder according to the save location of the generated file. Determine whether the Protobuf type serialization code needs to be generated according to the parsed serialization type. If necessary, use the Protoc tool to generate the Protobuf serialization code corresponding to all Proto files in the list. Regardless of whether other serialization methods are required, the message information in all Proto files in the list needs to be loaded into the memory, and the integrity of the serialization process is ensured by traversing the Proto file list. The serialization code generation tool obtains the message description information in the Proto file, and readjusts the order of the message according to the nested relationship to ensure that the dependent message is in the front. Finally, the message description information of the current Proto file is passed to the serialization code model, and the binary serialization code of the current Proto file message is generated by the model. The serialization code generation tool traverses the file list, and when there is no Proto file that has not been serialized, it is considered that the serialization process is over.
[0092] To summarize, first obtain the original file of the serialization code to be generated. Then parse the original file and determine the various structured data corresponding to the original file. Finally, load the structured data into the memory, and generate a general serialization interface code for each structured data corresponding to the original file based on the serialization code model. The generated general serialization interface code can be used for cross-core communication. In this way, the data is directly mapped to the memory, and the serialization code model directly operates the memory address to read the data, avoiding data copying, and serializing the original file in a zero-copy manner. The generated serialization code supports SOC and MCU cross-core communication, without calling other interfaces, thereby improving execution efficiency.
[0093] Based on the universal serialization method provided in the above embodiment, the embodiment of the present application further provides a universal serialization device. The universal serialization device is described below in conjunction with the embodiments and drawings.
[0094] Figure 4 A schematic diagram of the structure of a general serialization device provided in an embodiment of the present application. Figure 4 As shown, the general serialization device 400 provided in the embodiment of the present application includes:
[0095] The acquisition module 401 is used to acquire the original file of the serialization code to be generated;
[0096] A parsing module 402, configured to parse the original file and determine each structured data corresponding to the original file;
[0097] The serialization module 403 is used to load the structured data into the memory and generate a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model; the universal serialization interface code can be used for cross-core communication.
[0098] As an implementation method, regarding how to obtain the original file, the obtaining module 401 can be specifically used for:
[0099] Obtaining file parameters input by the user; the file parameters include the original file address;
[0100] An original file corresponding to the original file address is acquired based on the original file address.
[0101] As an implementation mode, in view of different compositions of the file parameters, the file parameters further include: a serialization type;
[0102] The serialization device 400 further includes: a first serialization submodule;
[0103] The first serialization submodule is used to execute the Protoc command to generate Protobuf serialization code for the original file if the serialization type specifies the Protobuf method.
[0104] As an implementation mode, according to different constitutions of the file parameters, the file parameters further include: a storage location of the generated file;
[0105] The serialization device 400 further includes: a storage module;
[0106] A storage module, used to create a storage folder at the storage location;
[0107] The generated universal serialization interface code is saved in the storage folder.
[0108] As an implementation method, regarding how to generate a universal serialization interface code, the serialization module 403 is specifically used to:
[0109] Get the description information of each structured data;
[0110] Based on the description information, a universal serialization interface code is generated for each structured data through a serialization code model.
[0111] As an implementation method, with respect to how to plan the serialization order, the serialization device 400 further includes: an adjustment module;
[0112] The adjustment module is used to adjust the arrangement order of each structured data based on the nested relationship so that the dependent structured data is queued first.
[0113] As an implementation method, how to determine whether to generate a serialization code is completed. Accordingly, the serialization device 400 further includes: an identification module;
[0114] An identification module, used to traverse a file list; the file list includes all original files to be generated into serialized codes;
[0115] If serialization code generation has been completed for all original files in the file list, the file list is marked.
[0116] To summarize, first obtain the original file of the serialization code to be generated. Then parse the original file and determine the various structured data corresponding to the original file. Finally, load the structured data into the memory, and generate a general serialization interface code for each structured data corresponding to the original file based on the serialization code model. The generated general serialization interface code can be used for cross-core communication. In this way, the data is directly mapped to the memory, and the serialization code model directly operates the memory address to read the data, avoiding data copying, and serializing the original file in a zero-copy manner. The generated serialization code supports SOC and MCU cross-core communication, without calling other interfaces, thereby improving execution efficiency.
[0117] In addition, the present application also provides a general serialization device, including:
[0118] Memory for storing computer programs;
[0119] A processor is used to implement the steps of the general serialization method as described in any one of the above items when executing the computer program.
[0120] In addition, the present application also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the general serialization method as described in any one of the above items are implemented.
[0121] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A general serialization method, characterized in that: The method comprises: Get the original file of the serialization code to be generated; Parsing the original file and determining each structured data corresponding to the original file; The structured data is loaded into the memory, and a universal serialization interface code is generated for each structured data corresponding to the original file based on a serialization code model; the universal serialization interface code can be used for cross-core communication.
2. The method according to claim 1, characterized in that The step of obtaining the original file of the serialization code to be generated includes: Obtaining file parameters input by the user; the file parameters include the original file address; An original file corresponding to the original file address is acquired based on the original file address.
3. The method according to claim 2, characterized in that The file parameters also include: serialization type; The general serialization method also includes: If the serialization type specifies the Protobuf method, the Protoc command is executed to generate the Protobuf serialization code for the original file.
4. The method according to claim 2, characterized in that: The file parameters also include: the storage location of the generated file; The general serialization method also includes: Create a storage folder at the storage location; The generated universal serialization interface code is saved in the storage folder.
5. The method according to claim 1, characterized in that The method of generating a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model includes: Get the description information of each structured data; Based on the description information, a universal serialization interface code is generated for each structured data through a serialization code model.
6. The method according to claim 1, characterized in that Before generating a universal serialization interface code for each structured data corresponding to the original file based on the serialization code model, the method further includes: The arrangement order of each structured data is adjusted based on the nested relationship so that the dependent structured data is prioritized.
7. The method according to claim 1, characterized in that The method further comprises: Traversing a file list; the file list includes all original files to be generated into serialized codes; If serialization code generation has been completed for all original files in the file list, the file list is marked.
8. A universal serialization device, characterized in that: include: An acquisition module is used to obtain the original file of the serialization code to be generated; A parsing module, used for parsing the original file and determining each structured data corresponding to the original file; A serialization module, used for loading the structured data into a memory, and generating a general serialization interface code for each structured data corresponding to the original file based on a serialization code model; The general serialization interface code can be used for cross-core communication.
9. A universal serialization device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the general serialization method as claimed in any one of claims 1 to 7 when executing the computer program.
10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the general serialization method according to any one of claims 1 to 7 are implemented.
