Structured storage fixed-length serialization method and system based on Rust process macro
By parsing declarative properties using Rust procedural macros to obtain the fixed length of variable-length fields, automatically allocating memory, and employing a fixed-length serialization algorithm combined with compile-time type checking, this approach solves the problems of low efficiency, complex manual coding, and insufficient type safety in existing serialization schemes, achieving efficient and stable fixed-length serialization processing.
Patent Information
- Application Number
- CN202511199977.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-26
- Publication Date
- 2025-12-12
AI Technical Summary
In existing technologies, serialization schemes suffer from problems such as low random access efficiency, heavy manual coding burden, insufficient type safety guarantees, and inefficient memory management. In particular, Rust lacks a solution for fixed-length serialization.
A structured storage fixed-length serialization method based on Rust procedural macros is adopted. The preset fixed length of variable-length fields is obtained by parsing declarative attributes, memory capacity is automatically configured, and a fixed-length serialization algorithm is used to ensure that the total length of the byte sequence is fixed. Combined with compile-time type checking, redundant operations at runtime are avoided.
It achieves efficient fixed-length serialization processing, improves the execution efficiency of the serialization process, simplifies the development process, enhances the stability of memory usage and data consistency, and is suitable for scenarios with high performance and security requirements.
Smart Images
Figure CN121117079A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data serialization, and particularly relates to a structured storage fixed-length serialization method and system based on a Rust process macro. BACKGROUND
[0002] In modern data storage and transmission systems, serialization is the process of converting data structures in memory into byte sequences for the purpose of data storage and transmission. Current mainstream serialization schemes such as JSON, XML, Protocol Buffers and MessagePack, etc. usually generate variable-length data representations. Although this approach has advantages in terms of space efficiency, it has obvious shortcomings in structured storage scenarios that require random access. The main problems in the prior art include:
[0003] Low random access efficiency: variable-length serialization formats require scanning the entire file or maintaining additional index structures to locate specific records, and cannot achieve constant-time direct access.
[0004] Heavy manual coding burden: existing fixed-length serialization schemes usually require developers to manually write a large amount of code to manage the length and offset of each field, which is prone to errors and has high maintenance costs.
[0005] Type safety is not guaranteed: Many serialization libraries lack compile-time type checking, which can easily lead to runtime errors.
[0006] Complex handling of variable-length types: For string, dynamic array and other variable-length types, existing technologies are difficult to maintain API friendliness while ensuring the fixed-length nature of the serialization result.
[0007] Inefficient memory management: Frequent memory allocation and reallocation during serialization and deserialization processes affect performance.
[0008] Rust language has been widely used in system programming due to its memory safety and zero-cost abstraction features, but there is currently a lack of solutions specifically designed for fixed-length serialization needs. Therefore, there is an urgent need for a Rust language-based technical solution to address the problems in the prior art. SUMMARY
[0009] In order to overcome the shortcomings of the prior art, the present application provides a structured storage fixed-length serialization method and system based on a Rust process macro to solve the problems in the prior art.
[0010] One embodiment of the present application provides a structured storage fixed-length serialization method based on a Rust process macro, comprising the following steps:
[0011] S10, acquiring a preset fixed length of a variable length field by resolving a declarative attribute in a structured data definition through a Rust process macro;
[0012] S20, automatically configuring a memory capacity for the variable length field based on the preset fixed length to avoid dynamic expansion;
[0013] S30, converting the structured data into a byte sequence by using a fixed-length serialization algorithm, so that a total length of the byte sequence is fixed and irrelevant to an actual value of the field;
[0014] S40, when an actual data length of the variable length field exceeds the preset fixed length, performing intelligent truncation on the actual data and maintaining data integrity;
[0015] S50, using a type checking mechanism of a Rust compiler to perform type checking on a serialization process at a compilation stage;
[0016] S60, generating, by the Rust process macro, an execution logic of steps S20 to S50 at the compilation stage, so that there is no redundant operation at a runtime.
[0017] In one of the embodiments, the declarative attribute in step S10 is configured as:
[0018] for specifying the preset fixed length of the variable length field, and the declarative attribute contains a numerical parameter N, which is a preset fixed length value of the variable length field.
[0019] In one of the embodiments, the automatic configuration of the memory capacity for the variable length field in step S20 includes:
[0020] allocating, when a specific data object is created according to the structured data definition, a memory space equal to the preset fixed length for the variable length field to prevent dynamic expansion at a runtime;
[0021] wherein the variable length field includes a string type and a vector type, and the vector type can contain elements of any data type.
[0022] In one of the embodiments, when the actual data length of the variable length field is less than the preset fixed length in step S30, zero bytes are filled in a difference part between the actual data and the preset fixed length in the serialized byte sequence.
[0023] In one of the embodiments, after step S30, the following step is further included:
[0024] S301, for a plurality of stored serialized records, a storage position of a target record is calculated by the following formula to realize direct access with constant time complexity, and the specific formula is:
[0025] Target record position = start position + record index x total length of fixed byte sequence.
[0026] In one of the embodiments, after step S30, further comprising the following steps:
[0027] S302, generating deserialization logic through the Rust procedural macro, parsing corresponding fields from the byte sequence according to the preset fixed length, and using the type checking mechanism of the Rust compiler to ensure that the parsing result matches the type of the original structured data, and the deserialization process has no runtime redundant operation.
[0028] In one of the embodiments, in step S40, the maintaining data integrity comprises:
[0029] When truncating the actual data, the complete boundary of the character is detected and maintained to avoid the character being split; wherein the character adopts the UTF-8 encoding format.
[0030] The application also relates to a structured storage fixed-length serialization system based on a Rust procedural macro, comprising:
[0031] The procedural macro parsing module is configured to parse the declarative attribute in the structured data definition through the Rust procedural macro, and obtain the preset fixed length of the variable-length field.
[0032] The memory configuration module is configured to automatically configure the memory capacity for the variable-length field based on the preset fixed length, and avoid dynamic expansion.
[0033] The serialization first execution module is configured to convert the structured data into a byte sequence by using a fixed-length serialization algorithm, so that the total length of the byte sequence is fixed and independent of the actual value of the field.
[0034] The serialization second execution module is configured to perform intelligent truncation on the actual data and maintain data integrity when the actual data length of the variable-length field exceeds the preset fixed length.
[0035] The compile-time guarantee module is configured to use the type checking mechanism of the Rust compiler to perform type checking on the serialization process at the compile time, and to generate the execution logic of steps S20 to S50 at the compile time through the Rust procedural macro, so that there is no redundant operation at the runtime.
[0036] The application also relates to a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned structured storage fixed-length serialization method based on a Rust procedural macro when executing the computer program.
[0037] The application also relates to a computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned structured storage fixed-length serialization method based on a Rust procedural macro.
[0038] The structured storage fixed-length serialization method and system based on a Rust procedural macro provided by the above-mentioned embodiments have the following beneficial effects:
[0039] The Rust procedural macro can be used to parse declarative attributes and generate execution logic at the compilation stage, accurately obtain the preset fixed length of the variable-length field, and automatically configure the memory capacity, thus fundamentally avoiding memory fragmentation and performance fluctuations caused by dynamic expansion; at the same time, the fixed-length serialization algorithm ensures that the total length of the byte sequence is fixed and independent of the actual value of the field, thus guaranteeing the format consistency of data storage and transmission; when processing long data, the data integrity is maintained through intelligent truncation, and the problems of type mismatch or illegal conversion are avoided in advance by combining the type checking at the compilation stage, thus reducing runtime errors. Overall, the scheme significantly improves the execution efficiency of the serialization process, enhances the stability and predictability of memory usage, simplifies the development process, reduces the complexity of manual operation, is suitable for scenarios with high requirements for performance, security and data consistency, and optimizes the fixed-length serialization processing of structured data. BRIEF DESCRIPTION OF DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the drawings shown.
[0041] Figure 1 A flowchart of the structured storage fixed-length serialization method based on a Rust procedural macro provided by the embodiments of the present application;
[0042] Figure 2 A principle block diagram of a computer device provided by the embodiments of the present application. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0044] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture, and if the certain posture changes, the directional indications will also change accordingly.
[0045] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In addition, "and / or" or "and / or" appearing throughout the text means that the three parallel schemes include A scheme, or B scheme, or A and B are satisfied at the same time. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.
[0046] Referring to Figure 1 One embodiment of the present application provides a structured storage fixed-length serialization method based on a Rust process macro, comprising the following steps:
[0047] S10, the declarative attribute in the structured data definition is parsed by the Rust process macro to obtain the preset fixed length of the variable length field; wherein the declarative attribute is configured to specify the preset fixed length of the variable length field, and the declarative attribute contains a numerical parameter N, and the numerical parameter N is the preset fixed length value of the variable length field.
[0048] In the embodiment, the step is to parse the declarative attribute in the structured data definition by the Rust procedural macro to obtain the preset fixed length of the variable-length field. The declarative attribute is specifically used to specify the preset fixed length of the variable-length field, and contains a numerical parameter N which is the preset fixed length value of the variable-length field. Specifically, in the definition of the structured data, the developer adds the declarative attribute to the variable-length field (such as a string, a vector, etc.) that needs to set the fixed length, and the Rust procedural macro scans and identifies these fields with the declarative attribute, and then parses the numerical parameter N in the declarative attribute to determine the preset fixed length of the corresponding variable-length field and record it. This process does not require the developer to manually write code to specify the length, but directly defines it in a declarative manner, which simplifies the operation process, reduces the errors that may occur in manual input, and provides accurate basic data for subsequent steps of configuring the memory capacity according to the preset length and performing serialization.
[0049] S20, based on the preset fixed length, automatically configuring the memory capacity for the variable-length field to avoid dynamic expansion.
[0050] In the embodiment, after obtaining the preset fixed length of the variable-length field, the memory capacity for the variable-length field is automatically configured according to the length to avoid dynamic expansion during runtime. Specifically, when creating an actual data object according to the definition of the structured data, a memory space completely matching the preset fixed length is allocated to the string, vector and other variable-length fields. For example, if the preset fixed length of a string field is set to 50 by the declarative attribute, the system will directly allocate a memory capable of accommodating 50 bytes when creating the string object. For a vector type field, if the preset fixed length corresponds to 10 elements, a memory space capable of storing 10 elements of the type will be allocated. This automatic configuration method allows the variable-length field to have sufficient memory in the initialization stage, and even if the data volume increases in the subsequent use process, there is no need to dynamically apply more memory, thereby reducing the memory allocation and reallocation operations, reducing the possibility of memory fragmentation, and improving the efficiency of the serialization and deserialization process. This process relies on the Rust procedural macro and does not require the developer to manually write code to manage memory, which simplifies the development process and ensures the stability of memory usage.
[0051] S30, using a fixed-length serialization algorithm to convert the structured data into a byte sequence, so that the total length of the byte sequence is fixed and independent of the actual value of the field.
[0052] In this embodiment, step S30 is to convert the structured data into a byte sequence by using a fixed-length serialization algorithm, and the total length of the byte sequence is fixed and is not affected by the actual values of the fields. Specifically, for various data types in the structured data, whether they are basic types such as integers and floating-point numbers, or composite types such as strings and vectors, the byte length occupied after serialization is fixed. Basic types are encoded according to their own characteristics using a preset fixed byte length, for example, a certain type of integer occupies 4 bytes, and a certain type of floating-point number occupies 8 bytes, and this length will not change due to different specific values. For variable-length types, even if the actual data length changes, the serialized byte length strictly follows the preset fixed length obtained in step S10. When the actual data length is less than the preset fixed length, zero bytes will be filled in the byte sequence; when the actual data length exceeds the preset fixed length, the intelligent truncation mechanism will be used to process it to ensure the fixed length of the byte.
[0053] In this way, the total length of the byte sequence formed after the serialization of the entire structured data always remains constant and will not change due to the differences in the actual values of the fields, providing a stable basis for subsequent storage and random access operations.
[0054] According to the needs, in step S30, the fixed-length serialization algorithm includes the processing of basic types including but not limited to integers, floating-point numbers, and Boolean values, which includes encoding using a preset fixed byte length to make the byte length of each basic data type after serialization fixed and unique, and the byte length is independent of the specific value of the type.
[0055] Specifically, for the processing of basic types, the fixed-length serialization algorithm converts the structured data by pre-setting a unique and fixed byte length for each type for integers, floating-point numbers, and Boolean values, and the encoding process strictly follows this length. For example, the u32 type occupies 4 bytes, and the f64 type occupies 8 bytes, and regardless of the specific value of these types, the corresponding byte length after serialization remains unchanged. This processing method ensures that the byte length of each basic type after serialization is unique and stable, and is not affected by the specific value, which is an important basis for the fixed total length of the entire structured data.
[0056] According to the needs, in step S30, the fixed-length serialization algorithm also includes byte sequence configuration: specifying the byte sequence by declarative attributes, and the fixed-length serialization algorithm encodes the basic types according to the specified byte sequence to ensure compatibility when transmitting across systems.
[0057] Specifically, for byte order configuration, the fixed-length serialization algorithm supports specifying the byte order (such as big-endian or little-endian) through a declarative attribute. When encoding a basic type, the algorithm performs the encoding operation according to the specified byte order rule. For example, when specified as big-endian, a multi-byte basic type is stored with the high byte first; when specified as little-endian, it is stored with the low byte first. This mechanism enables the serialized byte sequence to be correctly parsed when transmitted between different systems, effectively avoiding data parsing errors caused by system byte order differences, thereby ensuring cross-system transmission compatibility.
[0058] S40, when the actual data length of the variable-length field exceeds the preset fixed length, performing intelligent truncation on the actual data and maintaining data integrity.
[0059] In this embodiment, step S40 is the process of performing intelligent truncation on the actual data and maintaining data integrity when the actual data length of the variable-length field exceeds the preset fixed length. Specifically, when the actual data length of a variable-length field such as a string or vector exceeds the preset fixed length obtained in step S10, the system automatically triggers the truncation mechanism, and takes measures to protect the integrity of the data during the truncation process.
[0060] For string type, the truncation operation maintains the UTF-8 character boundary, ensuring that the truncated data is still a valid UTF-8 encoded string. For example, if the preset fixed length of a string is 5 and the actual content is "ábcde" containing multi-byte characters (where "á" is 2 bytes), when the actual length exceeds the preset length, the system will search forward from the preset length position to find the nearest valid character boundary for truncation, avoiding splitting a multi-byte character into two parts, and ensuring that the truncated string can be correctly parsed.
[0061] At the same time, when performing the truncation operation, the system outputs a warning message to prompt the developer about the situation that the data length exceeds the preset length, so that the developer can adjust the preset length or process the data according to the actual needs in a timely manner. This intelligent truncation mechanism not only ensures that the length of the serialized variable-length field meets the preset fixed length requirement, but also protects the integrity and validity of the data, avoiding data corruption or parsing errors caused by blind truncation.
[0062] S50, using the type checking mechanism of the Rust compiler to perform type checking on the serialization process at compile time.
[0063] In this embodiment, step S50 is a process of type checking of the serialization process in the compilation phase using the powerful type checking mechanism of the Rust compiler. Specifically, when the developer defines the structured data and uses the related serialization features, the Rust compiler will perform a matching check on the types of the fields in the data structure and the types required by the serialization logic at the compilation time.
[0064] For example, if a field is defined as an integer type, but the serialization logic expects to process a string type, the compiler will identify the type mismatch at the compilation stage and throw an error message to prompt the developer to correct it. This checking mechanism covers the entire process from data structure definition to serialization code generation, ensuring that the types of the fields involved in serialization comply with the preset rules, avoiding runtime errors caused by type incompatibility.
[0065] By completing type checking at the compilation time, there is no need for additional type checking operations at runtime, which not only ensures the type safety of the serialization process, but also complies with the characteristics of zero-cost abstraction, without additional overhead on runtime performance, further improving the reliability and efficiency of the entire serialization technology.
[0066] According to the needs, the type checking in step S50 further includes checking the matching of the field type with the preset fixed length and the conversion legality of different types of fields in the serialization process, and all the checking results are fed back at the compilation stage.
[0067] Specifically, for the matching check of the field type and the preset fixed length, the system verifies whether the preset fixed length set for the field is compatible with the data type of the field. For example, if a 32-bit integer type field is preset with a fixed length of 8 bytes, the compiler will identify the mismatch at the compilation time and report an error, because a 32-bit integer usually only needs 4 bytes to store, and an excessively long preset length may cause resource waste or logical errors; if a string field that needs to store long text is only preset with a fixed length of 2 bytes, it will also be detected as mismatch, prompting that it may not meet the actual data storage requirements.
[0068] For the conversion legality check of different types of fields in the serialization process, it mainly aims at the possible type conversion scenarios. For example, if an attempt is made to directly convert the data of a string type field to an integer type for serialization, or an incompatible conversion occurs between the child type and the parent type in a composite type nesting, the compiler will determine that the conversion is illegal at the compilation stage and give an error prompt, avoiding data corruption or program exceptions caused by type conversion errors at runtime.
[0069] Through these detailed checks completed in the compilation phase, further ensure the type safety and logical correctness of the serialization process, and provide more comprehensive protection for the reliability of the entire fixed-length serialization technology.
[0070] S60, through the Rust process macro, generates the execution logic of steps S20 to S50 at the compilation stage, so that there is no redundant operation at runtime.
[0071] In this embodiment, step S60 is to generate the execution logic of steps S20 to S50 at the compilation stage through the Rust process macro, to ensure that there is no redundant operation at runtime. Specifically, the Rust process macro will automatically generate the specific code logic corresponding to steps S20 (configuring the memory capacity for the variable-length field), S30 (fixed-length serialization), S40 (intelligent truncation), and S50 (type checking) during code compilation according to the structured data and related attributes defined by the developer (such as #[fixlen(len=N)]), rather than dynamically building these logics at runtime.
[0072] For example, for the logic of presetting the capacity of the variable-length field in step S20, the process macro will analyze the preset fixed length of the field at the compilation stage and generate code for directly allocating memory of the corresponding capacity; for the fixed-length serialization algorithm of step S30, the process macro will generate code for encoding according to the fixed byte length according to the type and length of each field; the intelligent truncation mechanism of step S40 and the type checking logic of S50 will also be converted into code segments executable at the compilation stage and embedded into the program in advance.
[0073] Since all execution logics are generated and fixed at the compilation stage, the runtime only needs to directly execute these pre-generated codes without additional parsing, judgment, or dynamic code generation operations, thereby realizing "zero-cost abstraction" - maintaining the simplicity and development efficiency of the code, avoiding the performance loss at runtime, and ensuring the efficiency and no redundant overhead of the serialization process.
[0074] Based on step S60, according to the needs, the no-redundant operation at runtime specifically means that there is no need for dynamic memory allocation, no need for parsing format identification, and the generation and reading of byte sequences are realized through direct memory operations in the serialization process.
[0075] Specifically, the dynamic memory allocation is not required because, at the compile time, the procedural macro has preset sufficient memory capacity for the variable-length type according to the preset fixed length, and the runtime does not trigger additional memory allocation or reallocation operations due to the change of data length, thereby reducing the overhead of memory management. The format identification is not required to be parsed because the fixed-length serialization format has a fixed field length and layout, and the position and occupied byte number of each field are clear at the compile time, and the runtime does not need to parse additional format identifiers (such as parentheses, tags, etc.) to determine the field boundary like JSON, XML and other formats, and can directly locate the target field.
[0076] The generation and reading of the byte sequence are implemented through direct memory operation, which means that the data is directly written into the preset memory buffer according to the fixed length during serialization, and the corresponding length of bytes is directly read from the specified memory location and converted into the target type during deserialization, thereby avoiding complex format conversion and intermediate steps and further improving the running efficiency.
[0077] This design fully utilizes the advantage of the Rust procedural macro in generating code at the compile time, and eliminates the redundant operations that may be generated at the runtime in advance, thereby ensuring that the serialization and deserialization processes are lightweight and efficient.
[0078] In one of the embodiments, the step S20 of automatically configuring the memory capacity for the variable-length field includes:
[0079] In the creation of the specific data object according to the structured data definition, the memory space equal to the preset fixed length is allocated for the variable-length field, so as to prevent the dynamic expansion at the runtime;
[0080] The variable-length field includes a string type and a vector type, and the vector type can contain elements of any data type.
[0081] In this embodiment, specifically:
[0082] The step S20 of automatically configuring the memory capacity for the variable-length field includes that, in the creation of the specific data object according to the structured data definition, the memory space equal to the preset fixed length is allocated for the variable-length field, so as to fundamentally prevent the dynamic expansion at the runtime.
[0083] The variable-length field in this embodiment mainly includes a string type and a vector type. The string type is a field for storing text data, and the vector type is a collection type field that can contain multiple elements, and the vector type can contain elements of any data type, such as integer, floating point number, Boolean value, or other structured data.
[0084] For example, if the preset fixed length of a string field is 50, 50 bytes of memory space are directly allocated when the string object is created. If the preset fixed length of a vector field corresponds to 8 elements, and the elements contained in the vector are floating-point numbers, memory space that can exactly store 8 floating-point numbers is allocated when the vector object is created. In this way, the variable-length field is ensured to have sufficient memory when it is initialized, and there is no need to dynamically expand the memory due to an increase in data volume in subsequent use, further clarifying the operation timing, specific method, and applicable variable-length field type range of automatic memory capacity configuration.
[0085] In one embodiment, in step S30, when the actual data length of the variable-length field is less than the preset fixed length, zero bytes are filled in the serialized byte sequence for the difference between the actual data and the preset fixed length.
[0086] In this embodiment, when the structured data is processed using the fixed-length serialization algorithm in step S30, for a variable-length field, when the actual data length is less than the preset fixed length, zero bytes are filled in the serialized byte sequence for the difference between the actual data and the preset fixed length. For example, if the preset fixed length of a string field is 50, and the actual stored text occupies only 10 bytes after encoding, when the field is serialized, the 10 bytes of actual data are first written, and then 40 bytes of zero are filled to ensure that the total length of the serialized field is exactly 50 bytes. This processing method ensures that the variable-length field can strictly follow the preset fixed length in any case, is an important link to realize the fixed total length of the entire serialized structured data, and provides a consistent structural basis for data storage and random access.
[0087] In one embodiment, after step S30, the following steps are further included:
[0088] S301. For the stored multiple serialized records, the storage position of the target record is calculated by the following formula to realize direct access with constant time complexity, and the specific formula is:
[0089] Target record position = start position + record index × total length of fixed byte sequence.
[0090] In this embodiment, after the fixed-length serialization of the structured data is completed in step S30, step S301 provides a mechanism to realize direct access with constant time complexity for the stored multiple serialized records. Specifically, the storage position of the target record can be accurately calculated by the formula "target record position = start position + record index × total length of fixed byte sequence":
[0091] The start position refers to the start byte address of the first record in the storage medium (such as a file or a database), for example, the first byte position after the file header metadata;
[0092] The record index is the serial number (counting from 0) of the target record in all records, for example, the index of the third record is 2.
[0093] The fixed byte sequence total length refers to the fixed byte number (such as 102 bytes) of each record after serialization in step S30.
[0094] For example, if the start position is 200 bytes, the index of a certain record is 8, and the fixed length of each record is 100 bytes, the position of the record is 200 + 8 x 100 = 1000 bytes. Through this formula, the target record can be directly located without scanning the previous text or relying on additional index structures, achieving constant time complexity random access, greatly improving the efficiency of record retrieval in structured storage systems, especially suitable for database and file system scenarios that require high-frequency random access.
[0095] In one embodiment, after step S30, the following steps are further included:
[0096] S302, generating deserialization logic through the Rust process macro, parsing the corresponding field from the byte sequence according to the preset fixed length, and using the type checking mechanism of the Rust compiler to ensure that the parsing result matches the type of the original structured data, and the deserialization process has no runtime redundant operation.
[0097] In this embodiment, after step S30, step S302 generates targeted deserialization logic through the Rust process macro to complete the restoration from the fixed length byte sequence to the original structured data, and the specific process includes:
[0098] Field parsing: the deserialization logic extracts byte segments of the corresponding length from the byte sequence in order according to the preset fixed length of each field determined in step S10. For example, 50 bytes of continuous bytes are extracted for a string field with a preset length of 50 bytes, and 40 bytes are extracted for a vector field with a preset length of 10 elements (each f32 occupies 4 bytes), ensuring accurate field boundary.
[0099] Type matching verification: using the type checking mechanism of the Rust compiler, the consistency of the parsing result and the type of the original structured data is verified at the compilation stage. For example, it is ensured that the numerical value parsed from the byte sequence can only be assigned to the original integer type field, avoiding type mismatch problems such as incorrect conversion of string bytes to integers, and avoiding runtime errors in advance.
[0100] High efficiency without redundancy: Since the deserialization logic is generated by the procedural macro at compile time, the parsing is completed directly through memory operations such as byte slice conversion and type coercion, without the need for runtime dynamic type judgment, format verification or additional memory allocation, so there is no redundant operation, ensuring that the deserialization performance is consistent with the serialization process.
[0101] This mechanism not only guarantees the accuracy of data restoration, but also maintains the characteristics of zero-cost abstraction, forming a complete and efficient processing loop for serialization and deserialization.
[0102] In one embodiment, in step S40, maintaining data integrity includes:
[0103] When truncating the actual data, the complete boundary of the character is detected and maintained to avoid the character being split; wherein the character adopts a UTF-8 encoding format.
[0104] In this embodiment, in step S40, when the actual data length of the variable-length field exceeds the preset fixed length, the maintenance of data integrity specifically represents that: in the process of performing the truncation operation on the actual data, the character in the UTF-8 encoding format is specially processed, and the complete boundary of the character is detected and maintained to avoid the character being split.
[0105] Specifically, a UTF-8 encoded character can be composed of 1 to 4 bytes, and if it is directly truncated at the preset fixed length, part of the bytes of a multi-byte character may be retained and part of the bytes may be discarded, resulting in the remaining character becoming an invalid UTF-8 encoding. Therefore, the system will start from the preset fixed length position and look back to find the nearest valid character boundary (i.e. the bytes before a certain position form a complete UTF-8 character), and take this position as the actual truncation point. For example, if the preset fixed length is 7, and the 6th to 7th bytes belong to a 2-byte UTF-8 character, the system will adjust the truncation point to before the 6th byte to ensure that the retained characters are all complete and valid UTF-8 encodings, avoiding garbled characters or errors during subsequent parsing.
[0106] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0107] In one embodiment, a structured storage fixed-length serialization system based on a Rust procedural macro is provided, which corresponds to the structured storage fixed-length serialization method based on a Rust procedural macro in the above embodiment. The structured storage fixed-length serialization system based on a Rust procedural macro includes:
[0108] a process macro parsing module, configured to parse a declarative attribute in a structured data definition by a Rust process macro, to obtain a preset fixed length of a variable-length field;
[0109] a memory configuration module, configured to automatically configure a memory capacity for the variable-length field based on the preset fixed length, to avoid dynamic expansion;
[0110] a serialization first execution module, configured to convert the structured data into a byte sequence by using a fixed-length serialization algorithm, so that a total length of the byte sequence is fixed and irrelevant to an actual value of a field;
[0111] a serialization second execution module, configured to perform intelligent truncation on actual data of the variable-length field and maintain data integrity when an actual data length of the variable-length field exceeds the preset fixed length;
[0112] a compilation period guarantee module, configured to perform type checking on a serialization process at a compilation period by using a type checking mechanism of a Rust compiler, and configured to generate execution logic of steps S20 to S50 at the compilation period by the Rust process macro, so that there is no redundant operation at a running time.
[0113] Further, the system further comprises:
[0114] a random access module, configured to calculate a storage position of a target record by the following formula for a plurality of stored serialized records, to realize direct access with constant time complexity.
[0115] Further, the system further comprises:
[0116] a deserialization module, configured to generate deserialization logic by the Rust process macro, to parse a corresponding field from a byte sequence according to the preset fixed length, to utilize a type checking mechanism of a Rust compiler to ensure that a parsing result matches a type of original structured data, and to ensure that there is no redundant operation at a running time in a deserialization process.
[0117] For specific limitations of the structured storage fixed-length serialization system based on the Rust process macro, refer to the limitations of the structured storage fixed-length serialization method based on the Rust process macro in the foregoing, which will not be described herein. Each module in the structured storage fixed-length serialization system based on the Rust process macro can be realized by software, hardware, and a combination thereof, in whole or in part. Each module can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to each module.
[0118] In one embodiment, a computer device can be provided, which can be a server, and an internal structure diagram of the computer device can be as shown inFigure 2 As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store data, data processing, data analysis, etc. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a structured storage fixed-length serialization method based on Rust process macro.
[0119] In one of the embodiments, a computer device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement a structured storage fixed-length serialization method based on Rust process macro.
[0120] In one of the embodiments, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to implement a structured storage fixed-length serialization method based on Rust process macro.
[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.
[0123] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A structured storage fixed-length serialization method based on Rust procedure macros, characterized in that, Includes the following steps: S10. Parse the declarative properties in the structured data definition using Rust procedural macros to obtain the preset fixed length of the variable-length field; S20. Based on the preset fixed length, automatically configure the memory capacity for the variable-length field to avoid dynamic expansion; S30. The structured data is converted into a byte sequence using a fixed-length serialization algorithm, so that the total length of the byte sequence is fixed and independent of the actual value of the field; S40. When the actual data length of the variable-length field exceeds the preset fixed length, intelligent truncation is performed on the actual data and data integrity is maintained. S50. Utilize the type checking mechanism of the Rust compiler to perform type verification during the serialization process at compile time; S60. The execution logic of steps S20 to S50 is generated during compilation using the Rust process macro, so that there are no redundant operations at runtime.
2. The structured storage fixed-length serialization method based on Rust procedure macros as described in claim 1, characterized in that, The declarative property in step S10 is configured as follows: This is used to specify the preset fixed length of a variable-length field, and the declarative property contains a numerical parameter N, which is the preset fixed length value of the variable-length field.
3. The structured storage fixed-length serialization method based on Rust procedure macros as described in claim 1, characterized in that, In step S20, automatically configuring the memory capacity for the variable-length field includes: When creating a specific data object based on the structured data definition, memory space equal to the preset fixed length is allocated for the variable-length field to prevent dynamic expansion at runtime. The variable-length field includes string type and vector type, and the vector type can contain elements of any data type.
4. The structured storage fixed-length serialization method based on Rust procedure macros as described in claim 3, characterized in that, In step S30, when the actual data length of the variable-length field is less than the preset fixed length, zero bytes are filled into the difference between the actual data and the preset fixed length in the serialized byte sequence.
5. The structured storage fixed-length serialization method based on Rust procedure macros as described in claim 1, characterized in that, Following step S30, the following steps are also included: S301. For multiple serialized records in storage, the storage location of the target record is calculated using the following formula, achieving direct access with constant time complexity. The specific formula is as follows: Target record position = starting position + record index × total length of fixed byte sequence.
6. The structured storage fixed-length serialization method based on Rust procedure macros as described in claim 5, characterized in that, Following step S30, the following steps are also included: S302. The Rust process macro generates deserialization logic, parses the corresponding field from the byte sequence according to the preset fixed length, and uses the Rust compiler's type checking mechanism to ensure that the parsing result matches the type of the original structured data, and the deserialization process has no runtime redundancy.
7. The structured storage fixed-length serialization method based on Rust procedure macros as described in claim 1, characterized in that, In step S40, maintaining data integrity includes: When truncating the actual data, the complete boundaries of the characters are detected and preserved to prevent the characters from being split; wherein the characters are in UTF-8 encoding format.
8. A structured storage fixed-length serialization system based on Rust procedure macros, used to implement the steps of the structured storage fixed-length serialization method based on Rust procedure macros as described in any one of claims 1-7, characterized in that, include: The procedural macro parsing module is used to parse declarative properties in structured data definitions using Rust procedural macros to obtain the preset fixed length of variable-length fields; The memory configuration module is used to automatically configure the memory capacity for the variable-length field based on the preset fixed length, so as to avoid dynamic expansion. The first serialization execution module is used to convert the structured data into a byte sequence using a fixed-length serialization algorithm, so that the total length of the byte sequence is fixed and independent of the actual value of the field; The second execution module for serialization is used to intelligently truncate the actual data and maintain data integrity when the actual data length of the variable-length field exceeds the preset fixed length. The compile-time guarantee module is used to perform type verification during the serialization process at compile time by utilizing the type checking mechanism of the Rust compiler. And the execution logic for generating steps S20 to S50 at compile time through the Rust process macro, so that there are no redundant operations at runtime.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the structured storage fixed-length serialization method based on Rust procedure macros as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the structured storage fixed-length serialization method based on Rust procedure macros as described in any one of claims 1-7.