Method for coding and decoding field with unlimited length based on nanoopb

Abstract

The invention discloses a method for encoding and decoding a field with unlimited length based on nanoopb, and belongs to the technical field of computers. According to the method, after a nanoopb tool automatically generates a code according to a. Proto file, a coding and decoding process is improved through the following steps: firstly, identifying all fields with unlimited lengths in a message, and replacing the data type of the fields with a self-defined dynamic array type; then replacing an initial value in the initialization macro and a callback function macro; and finally, realizing a unified custom callback function to process coding and decoding operations of all the fields with unlimited lengths. According to the method disclosed by the present invention, by packaging bottom layer Protobuf coding and decoding details, developers can complete serialization / deserialization operation without mastering complex coding rules and callback APIs, so that the learning cost is effectively reduced, the readability and maintainability of codes are significantly improved, and the method is suitable for resource-limited embedded system development.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a method for encoding and decoding fields of unlimited length based on nanopb, which is applicable to data serialization and deserialization processing in embedded systems or resource-constrained environments. Background Technology

[0002] Nanopb is a lightweight Protocol Buffers (Protobuf) library written in C, designed for embedded systems and resource-constrained devices. It features small code size and low memory footprint, and is widely used in fields such as the Internet of Things and industrial control.

[0003] However, compared to Protobuf implementations in high-level languages ​​(such as C++ and Java), nanopb differs significantly in handling unlimited-length fields (such as strings, bytes, and repeated fields). In high-level languages, developers only need to define a .proto file and generate the corresponding code to directly manipulate objects without needing to understand the underlying encoding rules. In nanopb, however, developers must manually set callback functions for each unlimited-length field and implement the encoding / decoding logic within those functions. This requires developers to not only master Protobuf syntax but also be familiar with nanopb's callback mechanism and Protobuf's encoding rules, significantly increasing development difficulty and learning costs.

[0004] In existing technologies, the default code structure generated by nanopb uses the pb_callback_t type to represent fields of unlimited length. This type contains callback function pointers and parameters. Developers need to set callback functions for each field and implement encoding and decoding logic, which is cumbersome and error-prone, resulting in poor code readability and maintainability. Summary of the Invention

[0005] The purpose of this invention is to provide a method for encoding and decoding fields of unlimited length based on nanopb, so as to solve the problems of complex encoding and decoding processes, low development efficiency and difficult code maintenance in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for encoding and decoding fields of unlimited length based on nanopb, wherein after the code is automatically generated from the .proto file by the nanopb compilation tool, and before application development based on the generated code, the following steps are performed: S1. Based on the automatically generated code file, and combined with the original .proto file, identify all fields with unlimited length in the message, and replace the data type of the fields with unlimited length with a predefined custom dynamic array type; S2. Replace the initial value of the unlimited length field in the macro used to initialize the message in the automatically generated code file with the initial value of the custom dynamic array type; S3. Replace the callback function macros associated with messages containing fields of unlimited length in the automatically generated code file with a unified custom callback function; S4. Implement the unified custom callback function to perform encoding or decoding operations on all fields of unlimited length in the message.

[0007] Preferably, the unlimited length field mentioned in step S1 includes string type, bytes type, and repeating fields.

[0008] Preferably, the definition template for the custom dynamic array type mentioned in step S1 is: typedef struct st_dynamic_array_of_<field data type name> { <Field data type corresponding to the C language> *data; uint32_t used; uint32_t capacity; } st_dynamic_array_of_<field data type name>_t; The data member variable points to the storage space for the data to be encoded or the storage space for the decoded data; the used member variable indicates the amount of data to be encoded or the amount of usable data after decoding; and the capacity member variable indicates the maximum amount of data that the storage space pointed to by data can hold.

[0009] Preferably, the type name of the custom dynamic array type includes "dynamic_array" and the corresponding "field data type name information" to improve code intuitiveness.

[0010] Preferably, the initial value of the custom dynamic array type in step S2 is {NULL, 0, 0}.

[0011] Preferably, in the unified custom callback function of step S4, whether to enable encoding / decoding operation for the associated unlimited length field is determined based on whether the data member variable is NULL.

[0012] Preferably, in step S4, implementing the unified custom callback function specifically includes: Determine whether the current operation is encoding or decoding; According to the definition in the .proto file, for each field of unlimited length in the message, the dedicated encoding / decoding API interface of the callback function provided by nanopb, which corresponds to the field type, is called to perform encoding or decoding operations.

[0013] Preferably, when performing the encoding operation, a specified number of data points are extracted from the storage space pointed to by the data member variable of the custom dynamic array object associated with the field to be encoded, and the corresponding API interface in the Callback field encoders of nanopb is called to complete the encoding.

[0014] Preferably, when performing a decoding operation, the corresponding API interface in the Callback field decoders of nanopb is called to perform decoding, and the decoded data is stored in the storage space pointed to by the data member variable of the custom dynamic array object associated with the target field. At the same time, the used member variable of the dynamic array object is updated until decoding is completed or the value of the used member variable reaches the value of the capacity member variable.

[0015] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above methods.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention effectively hides the underlying Protobuf encoding and decoding details by uniformly replacing unlimited-length fields with a custom dynamic array type and setting a unified custom callback function to handle all encoding and decoding logic. Application developers do not need to master the complex Protobuf encoding rules and nanopb callback API design; they only need to perform simple assignment and association on the dynamic array object to complete the encoding and decoding operations. This achieves a development experience similar to that of high-level programming language developers, significantly reducing the technical threshold and learning cost of Protobuf application development in embedded environments.

[0017] This invention makes the code structure clearer and more intuitive by replacing the obscure pb_callback_t callback type with a custom dynamic array type with clear semantics; the unified custom callback function centrally handles the encoding and decoding logic of all fields with unlimited length, eliminating the duplicate code scattered in various callback functions in the traditional way, greatly improving the readability and maintainability of the code, and providing convenience for subsequent function expansion and code maintenance.

[0018] The dynamic array type of this invention can perform type checking during the compilation stage, effectively avoiding type mismatch errors; by introducing a capacity member variable and control logic, it ensures that no buffer overflow occurs during decoding, improving the stability and reliability of the system; at the same time, based on the judgment mechanism of whether the data member is NULL, it provides flexible encoding and decoding control capabilities, enhancing the robustness of the system. Attached Figure Description

[0019] Figure 1 This is an overall flowchart of the present invention; Figure 2 A flowchart for writing a custom callback function for this invention; Figure 3 This is a schematic diagram comparing the present invention with the default encoding process of nanopb; Figure 4 This is a schematic diagram comparing the present invention with the default decoding process of nanopb. Detailed Implementation

[0020] To further explain the technical solution of the present invention, a detailed description is provided below through specific embodiments.

[0021] Please see Figure 1 This invention provides a method for encoding and decoding fields of unlimited length based on nanopb. In the traditional nanopb usage process, after the nanopb compilation tool automatically generates code from the .proto file, and before application development based on the generated code, the following four core steps are inserted: S1. Replace the data type of the field with unlimited length; S2. Replace the initial value in the initialization macro; S3, Replace the callback function macro; S4. Implement a unified custom callback function.

[0022] S1, Replace the data type of the field with unlimited length: Based on the code file automatically generated by the nanopb tool, and combined with the original .proto file, it identifies all fields with unlimited length in the message, including string type, bytes type and repeated fields, and uniformly replaces the data type of these fields with a predefined custom dynamic array type.

[0023] The template for defining a custom dynamic array type is as follows: typedef struct st_dynamic_array_of_<field data type name> { <Field data type corresponding to the C language> *data; uint32_t used; uint32_t capacity; } st_dynamic_array_of_<field data type name>_t; In the above, the data member variable is a pointer type, used to associate the original data cache to be encoded, or the storage cache of the decoded data; the used member variable is used to indicate the number of original data caches to be encoded, or the number of available decoded data; the capacity member variable is used to indicate the maximum number that the cache space of the data member variable can hold.

[0024] Example 1: Processing student information messages: Taking the student information message defined in the .proto file as an example: / / Student Information Message Student { / / Number uint32 number = 1; / / Name string name = 2; } After processing in step S1, the following data type definitions will be generated: / / Custom dynamic array type definition typedef struct st_dynamic_array_of_char { char *data; uint32_t used; uint32_t capacity; } st_dynamic_array_of_char_t; / / Replaced message structure typedef struct Student { uint32_t number; st_dynamic_array_of_char_t name; Student_t; Application developers can intuitively perceive from the name st_dynamic_array_of_char_t that this member variable is a dynamic array type that stores char, which has the following advantages: 1. Improves code intuitiveness, allowing application developers to directly perceive from the type name that this member variable is a dynamic array storing char type; 2. Directly including data type information in type names increases the intuitiveness of the code and allows the compiler to perform automatic type checking, avoiding type mismatch errors.

[0025] Comparative analysis, comparing with the code generated by traditional nanopb: In comparison, if step S1 is not executed, the Student structure generated by nanopb by default is as follows: typedef struct Student { uint32_t number; pb_callback_t name; Student_t; The type definition of pb_callback_t includes a function pointer for encoding and decoding (funcs.encode / funcs.decode) and the parameter (arg) passed to the function pointer for encoding and decoding. This design is not as intuitive as a dynamic array object with data type, and the developer needs to manually set the callback function.

[0026] S2, replaces the initial value in the initialization macro: Replace the initial values ​​of the unlimited-length fields in the automatically generated xxx_init_default and xxx_init_zero macros (where xxx refers to the message name automatically generated by nanopb based on the .proto file) with the initial values ​​of the custom dynamic arrays.

[0027] The initial value of the custom dynamic array is: {NULL, 0, 0}.

[0028] Example 2, Replacement of initialization macro: For the Student message mentioned above, the initialization macro originally generated by nanopb might be: #define Student_init_default {0, {NULL, NULL}} #define Student_init_zero {0, {NULL, NULL}} After the replacement in step S2, it becomes: #define Student_init_default {0, {NULL, 0, 0}} #define Student_init_zero {0, {NULL, 0, 0}} S3, Replace the callback function macro: Replacing macros that callback functions for messages containing unlimited-length fields with custom callback functions ensures that all unlimited-length fields use a uniform callback handling mechanism, rather than nanopb's default approach of setting callbacks for each field individually.

[0029] S4 enables unified custom callback functions: like Figure 2 As shown, write a custom callback function to implement encoding and decoding operations for fields of unlimited length. The specific implementation includes the following steps: S41. Determine the operation type: First, determine whether the current operation is an encoding operation or a decoding operation.

[0030] S42. Call dedicated APIs based on field type: Based on the original .proto file definition, write encoding or decoding programs for each of the unlimited length fields; based on the actual unlimited length field type (string type, bytes type, repeated field), call the API interfaces specified in the dedicated encoding / decoding function set (Callback fieldencoders and Callback field decoders) provided by nanopb.

[0031] The specific implementation of the encoding operation is as follows: When performing encoding operations on a specified field: automatically retrieve the data from the storage space specified by the data member variable in the custom dynamic array object associated with the field; extract the corresponding number of data from the used member variable; and call the API interface specified in nanopb's Callbackfield encoders to perform the encoding.

[0032] The specific implementation of the decoding operation is as follows: When performing a decoding operation on a specified field: the API interface specified in the Callback field decoders of nanopb is called to perform decoding; the decoded data is stored one by one into the storage space specified by the data member variable in the custom dynamic array object associated with the field; the used member variable is incremented from 0 until the decoding is completed, or the number of used values ​​equals the capacity.

[0033] The logic for initial value processing is as follows: In step S4, whether to enable encoding / decoding operations is determined by whether the data member variable is NULL. When data is NULL, the encoding / decoding process for that field is skipped; when data is not NULL, the normal encoding / decoding process is executed.

[0034] Application development process comparison: like Figure 3 and Figure 4 As shown, this invention is compared with the default development process of nanopb: Comparison of coding processes ( Figure 3 ): Traditional approach: Requires setting the arg and funcs.encode of the pb_callback_t callback type, and writing funcs.encode encoding callback functions for all fields with unlimited length; This invention replaces the obscure pb_callback_t callback type with a dynamic array object containing data types, avoiding the need to set callback parameters and function pointers, and eliminating the need to concern oneself with Protobuf's encoding rules and nanopb's Callback fieldencoders API design.

[0035] Decoding process comparison ( Figure 4 ): Traditional approach: Requires setting arg and funcs.decode for pb_callback_t callback type, and writing decoding callback functions for funcs.decode for all fields with unlimited length; This invention replaces the obscure pb_callback_t callback type with a dynamic array object containing data types, avoiding the need to set callback parameters and function pointers, and eliminating the need to concern oneself with Protobuf's encoding rules and nanopb's Callback fielddecoders API design.

[0036] Practical application example: In practical application development, developers only need to perform the following simple operations: Encoding operations: Student_t student = Student_init_zero; char name_buffer[] = "Zhang San"; student.number = 1; student.name.data = name_buffer; / / Associate the data to be encoded student.name.used = strlen(name_buffer); / / Set the data length student.name.capacity = sizeof(name_buffer); / / Set the buffer capacity / / Serialization can be completed by calling the encoding API. pb_ostream_t stream = ...; pb_encode(&stream, Student_fields, &student); Decoding operations: Student_t student = Student_init_zero; char name_buffer

[100] ; student.name.data = name_buffer; / / Associate decoding storage space student.name.used = 0; / / Initialize to 0 student.name.capacity = sizeof(name_buffer); / / Set the maximum capacity / / Deserialization can be completed by calling the decoding API. pb_istream_t stream = ...; pb_decode(&stream, Student_fields, &student); / / After decoding, student.name.used is the actual length of the decoded data. printf("Name: %.*s\n", student.name.used, student.name.data); The following effects are achieved through the implementation of this invention: 1. Reduced learning cost: Application developers do not need to master the coding rules of Protobuf and the details of the nanopb callback API; 2. Improved development efficiency: Eliminates the tedious process of setting callback functions for each field individually; 3. Enhance code readability: Using dynamic array types with clear semantics makes the code intent clearer; 4. Improved code maintainability: Unified encoding and decoding processing logic facilitates subsequent maintenance and expansion; 5. Maintain compatibility: The underlying codec interface still uses nanopb to ensure compatibility with existing systems.

[0037] The present invention also relates to a computer-readable storage medium storing a computer program that performs the above-described methods and can run in an embedded platform that supports nanopb.

[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of encoding a variable length field based on nanopb, characterized in that, After the code is automatically generated from the.proto file by the nanopb compiling tool, and before the application development based on the generated code, the following steps are performed: S1. Based on the automatically generated code file, in combination with the original.proto file, all unlimited-length fields in the message are identified, and the data type of the unlimited-length fields is replaced with a predefined custom dynamic array type; S2. The initial value of the unlimited-length field in the macro for initializing the message in the automatically generated code file is replaced with the initial value of the custom dynamic array type; S3. The callback function macro associated with the message containing the unlimited-length field in the automatically generated code file is replaced with a unified custom callback function; S4. The unified custom callback function is implemented to perform encoding or decoding operations on all unlimited-length fields in the message.

2. The method of claim 1, wherein, In step S1, the unlimited-length fields include string type, bytes type, and repeated fields.

3. The method of claim 1, wherein, In step S1, the definition template of the custom dynamic array type is: typedef struct st_dynamic_array_of_<field data type name> { <field data type corresponding to the type in C language> *data; uint32_t used; uint32_t capacity; } st_dynamic_array_of_<field data type name>_t; Wherein, the data member variable is used to point to the storage space of the data to be encoded or the storage space of the decoded data; the used member variable is used to indicate the number of data to be encoded or the number of available data after decoding; the capacity member variable is used to indicate the maximum number of data that the storage space pointed to by data can accommodate.

4. The method of claim 3, wherein, The type name of the custom dynamic array type contains dynamic_array and the corresponding field data type name information.

5. The method of claim 3, wherein, In step S2, the initial value of the custom dynamic array type is {NULL, 0, 0}.

6. The method of claim 5, wherein, In the unified custom callback function of step S4, it is determined whether to start the encoding and decoding operation on the associated unlimited-length field according to whether the data member variable is NULL.

7. The method of claim 1, wherein, In step S4, the implementation of the unified custom callback function specifically includes: Determine whether the current operation is encoding or decoding; According to the definition of the.proto file, for each unlimited-length field in the message, call the special encoding and decoding API interface provided by nanopb corresponding to the field type to perform encoding or decoding operation.

8. The method of claim 7, wherein, When performing encoding operation, the data specified by the used member variable is extracted from the storage space pointed to by the data member variable of the custom dynamic array object associated with the field to be encoded, and the corresponding API interface in the Callback field encoders of nanopb is called to complete the encoding.

9. The method of claim 7, wherein, When performing the decoding operation, the corresponding API interface in the Callback field decoders of nanopb is called for decoding, and the decoded data is stored in the storage space pointed by the data member variable of the custom dynamic array object associated with the target field, and the used member variable of the dynamic array object is updated until the decoding is completed or the value of the used member variable reaches the value of the capacity member variable.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 9.

Images