Data serialization method and device, computer readable storage medium and electronic equipment
Through the improved protobuf protocol and Base128 encoding method, the original data is serialized and encoded, solving the problems of low serialization efficiency and large memory footprint in the existing technology, and achieving efficient data serialization and caching.
Patent Information
- Application Number
- CN202411917261.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-04-15
AI Technical Summary
In the prior art, serialization efficiency is low and memory footprint cannot be reduced, especially when processing large amounts of Chinese data.
The improved protobuf protocol is used to serialize data, the original data is encoded using GB18030 encoding, and the serialized binary data is encoded through Base128 encoding, and it is encoded within the ASCII code range.
It improves the encoding efficiency of Chinese characters, reduces memory footprint, and improves caching efficiency, achieving efficient data serialization and caching.
Smart Images
Figure CN119829668A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data serialization method, device, computer-readable storage medium, and electronic device. Background Art
[0002] Data caching refers to temporarily storing data in memory or other high-speed storage media to facilitate quick access and reading of data. Caching data in memory into a cache system often requires the selection of a serialization scheme.
[0003] At present, among the common serialization schemes, although some of them take up less space after serialization, the efficiency of serializing a large amount of Chinese is very low. In addition, when the serialized binary data is written into the cache system, the space occupied will be much higher than the binary length.
[0004] Therefore, the serialization efficiency in the related technology is low and the memory usage cannot be reduced. Summary of the invention
[0005] The main purpose of the present disclosure is to provide a data serialization method, device, computer-readable storage medium and electronic device to solve the problem in the related art that the serialization efficiency cannot be improved and the memory occupied space cannot be reduced.
[0006] In order to achieve the above object, the first aspect of the present disclosure provides a data serialization method, comprising:
[0007] The original data is converted into binary data through the improved binary serialization protocol protobuf, and the improved protobuf is encoded using the GB18030 encoding method;
[0008] The improved protobuf serialized binary data is encoded by Base128 encoding to obtain Base128 encoded binary data; wherein the Base128 encoding adopts all 128 characters of ASCII as a character set for encoding, and the Base128 encoded binary data corresponds one to one with the ASCII code;
[0009] The characters or strings corresponding to the Base128-encoded binary data are written into Redis.
[0010] Optionally, further, encoding the improved protobuf serialized binary data by Base128 encoding to obtain Base128 encoded binary data includes:
[0011] Divide the binary data after the improved protobuf serialization into a group for every 7 bytes, wherein if the binary data after the improved protobuf serialization is less than 7 bytes, the binary data after the improved protobuf serialization is taken as a group;
[0012] For each group of binary data, perform the following operations to obtain Base128-encoded binary data:
[0013] Convert the binary data in the current group into a plurality of new bytes; wherein the number of the new bytes is equal to the number of bytes corresponding to the binary data in the current group plus 1;
[0014] The binary data of each new byte corresponding to the current group is the binary data in the ASCII code table, and the characters corresponding to the binary data in the ASCII code table are used as the characters corresponding to the Base128.
[0015] Optionally, further, converting the binary data in the current group into a plurality of new bytes includes:
[0016] Make sure the first bit of each new byte is 0;
[0017] Divide the binary data in the current group into multiple subgroups with 7 binary bits as one subgroup; wherein the number of the multiple subgroups is the same as the number of the new bytes, and if the number of binary bits of the last subgroup divided in sequence is less than 7, add 0 after the binary data of the last subgroup until the number of binary bits of the last subgroup is equal to 7;
[0018] For each subgroup, the first leading 0 of a new byte and the 7 binary data of the subgroup are encoded into the new byte.
[0019] Optionally, further, the converting the original data into binary data by using an improved binary serialization protocol protobuf includes:
[0020] By using the improved protobuf, a string type field in the original data is identified;
[0021] The string type field is encoded using the GB18030 encoding method in the improved protobuf to obtain binary data.
[0022] Optionally, further, the method further includes:
[0023] Read the characters corresponding to the Base128 in the Redis;
[0024] The original data is obtained according to the characters corresponding to the Base128.
[0025] Optionally, further, writing the characters or strings corresponding to the Base128-encoded binary data into Redis also includes:
[0026] According to the Base128-encoded binary data, determining a corresponding character from a character set corresponding to the Base128 encoding method;
[0027] Write the corresponding characters to Redis.
[0028] A second aspect of the present disclosure provides a data serialization device, comprising:
[0029] The processing unit is used to convert the original data into binary data through an improved binary serialization protocol protobuf, and the improved protobuf is encoded in a GB18030 encoding method;
[0030] An encoding unit, used for encoding the improved protobuf serialized binary data by Base128 encoding to obtain Base128 encoded binary data; wherein the Base128 encoding adopts all 128 characters of ASCII as a character set for encoding, and the characters corresponding to the Base128 correspond to the characters corresponding to the ASCII code one by one;
[0031] The cache unit is used to write the characters or character strings corresponding to the Base128-encoded binary data into Redis.
[0032] A third aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data serialization method provided by any one of the first aspects.
[0033] The fourth aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the data serialization method provided in any one of the first aspects.
[0034] A fifth aspect of the present disclosure provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the data serialization method provided in any one of the first aspects.
[0035] In the data serialization method provided in the embodiment of the present disclosure, the original data is encoded and converted into binary data by adopting an improved protobuf serialization method. Since the improved protobuf adopts the GB18030 encoding method, the average number of bytes occupied by the utf-8 encoding adopted by the traditional protobuf is less, which makes the serialized binary data not too large, thereby effectively improving the encoding efficiency of Chinese characters; then, by adopting the Base128 encoding method in which all 128 characters of ASCII are encoded as the character set, the binary data after the improved protobuf serialization is encoded, and the binary code can be within the ASCII code range, which can effectively improve the cache efficiency while reducing the memory occupied space, so as to achieve the purpose of data serialization and caching, thereby realizing the technical effect of efficient caching and reducing the memory occupied space, thereby solving the technical problem that the serialization efficiency cannot be improved and the memory occupied space cannot be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0037] Figure 1 A schematic diagram of a scenario of a data serialization method provided in an embodiment of the present disclosure;
[0038] Figure 2 A flowchart of a data serialization method provided by an embodiment of the present disclosure;
[0039] Figure 3 A block diagram of a data serialization device provided in an embodiment of the present disclosure;
[0040] Figure 4 A block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION
[0041] In order to enable those skilled in the art to better understand the scheme of the present disclosure, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present disclosure.
[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0043] In the present disclosure, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present disclosure and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.
[0044] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this disclosure can be understood according to specific circumstances.
[0045] In addition, the terms "installed", "set", "provided with", "connected", "connected", and "socketed" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be an internal connection between two devices, elements, or components. For those of ordinary skill in the art, the specific meanings of the above terms in this disclosure can be understood according to specific circumstances.
[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0047] At present, in the related technologies, some common serialization schemes take up less space after serialization, but the efficiency of serializing a large number of Chinese characters is very low. Moreover, when the serialized binary data is written into the cache system, the space occupied will be much higher than the binary length. Therefore, the serialization efficiency in the related technologies is low, and the memory space occupied cannot be reduced at the same time.
[0048] In order to solve the above problems, the technical concept of the present invention is to use GB18030 encoding method for traditional protobuf, and to encode the original data through an improved protobuf serialization method to convert it into binary data. Since the improved protobuf adopts GB18030 encoding method, most Chinese characters only need 2 bytes, and a small part of the characters take up 3 bytes, so the encoding efficiency of Chinese characters can be effectively improved; at the same time, in order to reduce the occupied space, the Base128 encoding method can be adopted. Since the Base128 encoding method adopts all 128 characters of ASCII as the character set to encode the binary data serialized by the improved protobuf, the binary code can be within the ASCII code range, so that each binary byte (i.e., byte) in the encoded binary data or the character corresponding to each binary byte in the encoded binary data (or the encoded binary character, the binary character is: the character represented by the binary encoding scheme) in Redis only occupies 1 byte of memory, which can effectively improve the cache efficiency while reducing the memory occupied space.
[0049] In practical applications, the execution subject of the present disclosure may be a data serialization device, which may be deployed in an electronic device, such as a terminal device, a server, etc. A user may use an electronic device with a data serialization device to perform data serialization processing on the original data (e.g., memory data) through an improved protobuf, and perform Base128 encoding on the serialized binary data to cache the encoded characters in a database (e.g., Redis). The original data may be a character or a string, and the string may also contain 1 character, which is not specifically limited here.
[0050] For example, see Figure 1 As shown, Figure 1A scenario diagram of a data serialization method provided for an embodiment of the present disclosure. The scenario includes a data serialization device 101 and a memory-based key-value pair database (Remote DictionaryServer, Redis) 102; wherein the data serialization device first obtains the original data, and encodes the original data through an improved protobuf serialization method using GB18030 encoding, and converts it into binary data, and then encodes the improved protobuf serialized binary data through a Base128 encoding method using all 128 characters of ASCII as a character set, so that the binary code can be within the ASCII code range, and then the characters corresponding to the encoded binary data are cached in Redis, and the corresponding characters in the Base128 character set are cached in Redis, which corresponds one to one with the characters corresponding to the ASCII code.
[0051] In this way, each binary byte or encoded binary character in the encoded binary data (here refers to the character corresponding to each binary byte (i.e. byte) in the encoded binary data) only occupies 1 byte of memory in Redis, which can effectively improve the cache efficiency while reducing the memory usage, achieve the purpose of data serialization and caching, and realize the improvement of serialization efficiency, thereby achieving efficient caching and reducing the memory usage.
[0052] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of data and other information involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0053] The technical solution of the present disclosure is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0054] The present disclosure provides a data serialization method. Figure 2 As shown, the method includes the following steps S201 to S203:
[0055] Step S201: converting the original data into binary data through the improved binary serialization protocol protobuf, and the improved protobuf is encoded using the GB18030 encoding method.
[0056] Step S202: Encode the improved protobuf serialized binary data through Base128 encoding to obtain Base128 encoded binary data; wherein the Base128 encoding method uses all 128 characters of ASCII as a character set for encoding, and the Base128 encoded binary data corresponds one-to-one to the ASCII code.
[0057] Step S203: Write the characters or strings corresponding to the Base128-encoded binary data into Redis.
[0058] In the embodiment of the present disclosure, based on the improved protobuf serialization method, the original data is encoded and converted into binary data. Since the improved protobuf adopts the GB18030 encoding method, the average number of bytes occupied by the utf-8 encoding adopted by the traditional protobuf is less, which makes the serialized binary data not too large, so the encoding efficiency of Chinese characters can be effectively improved; then, the improved protobuf serialized binary data is encoded by using the Base128 encoding method that uses all 128 characters of ASCII as the character set for encoding, and the binary code can be within the ASCII code range. In this way, each binary byte or encoded binary character in the encoded binary data (here refers to the character corresponding to each binary byte (i.e., byte) in the encoded binary data) only occupies 1 byte of memory in Redis, compared with when the binary does not belong to the ASCII code range, it only occupies 1 byte in UTF-8 encoding and 2 bytes in redis memory. It can effectively improve the cache efficiency while reducing the memory space occupied, so as to achieve the purpose of data serialization and caching, thereby realizing efficient caching and reducing the memory space occupied.
[0059] Among them, the improved protobuf is an improved binary serialization protocol protobuf or an improved protobuf protocol. The original data here can be source data or source string, which is not specifically limited here; the original data may include string type fields. The string type field in the original data usually refers to the field represented in text form during data storage or transmission. These fields may contain a combination of characters such as letters, numbers, symbols or spaces, and are used to represent various types of information, such as name, address, email address, telephone number, etc.
[0060] Exemplarily, the original data here may be memory data, which is not specifically limited here. Memory data refers to data stored in a computer memory (RAM, Random Access Memory).
[0061] Specifically, the first step is to serialize the memory data into a string / binary, and the second step is to write the string / binary into Redis. In the first step of serialization, since protobuf uses UTF-8 encoding, a Chinese character usually occupies 3 to 4 bytes, and an average of 3.5 bytes. This makes the serialized binary too large.
[0062] In order to solve the problem of low efficiency of protobuf Chinese encoding, the disclosed embodiment introduces GB18030 encoding in the encoding process. GB18030 only requires 2 bytes for most Chinese characters, and a small number of characters occupy 3 bytes, which can effectively improve the encoding efficiency of Chinese characters.
[0063] In the second step, the binary is written to Redis. Since Redis uses the text protocol and UTF-8 encoding, each binary byte serialized by protobuf occupies 1 or 2 bytes in Redis. When the binary byte is within the ASCII code range, it only occupies 1 byte in the UTF-8 encoding and 1 byte in the Redis memory. When the binary byte is not within the ASCII range, it only occupies 1 byte in the UTF-8 encoding and 2 bytes in the Redis memory.
[0064] In order to solve the problem of being unable to reduce the occupied space, the disclosed embodiment proposes a new Base128 encoding scheme. Binary can be encoded within the ASCII code range, so that each binary byte or encoded binary character in the encoded binary data (here refers to the character corresponding to each binary byte (i.e. byte) in the encoded binary data) only occupies 1 byte of memory in Redis, which can greatly improve the cache efficiency. Compared with the related Base64 encoding, the use of Base128 encoding can save a larger storage space on average (for example, by calculation, compared with the related Base64 encoding, the use of Base128 encoding can save an average of 14.3% of storage space).
[0065] Optionally, encoding the improved protobuf serialized binary data by Base128 encoding to obtain Base128 encoded binary data includes:
[0066] Divide the binary data after the improved protobuf serialization into a group for every 7 bytes, wherein if the binary data after the improved protobuf serialization is less than 7 bytes, the binary data after the improved protobuf serialization is taken as a group;
[0067] For each group of binary data, perform the following operations to obtain Base128-encoded binary data:
[0068] Convert the binary data in the current group into a plurality of new bytes; wherein the number of the new bytes is equal to the number of bytes corresponding to the binary data in the current group plus 1;
[0069] The binary data of each new byte corresponding to the current group is the binary data in the ASCII code table, and the characters corresponding to the binary data in the ASCII code table are used as the characters corresponding to the Base128.
[0070] In the disclosed embodiment, there is a big difference between Base128 encoding and Base64 encoding. Among them, the related Base64 encoding uses 64 visible characters as the character set, and divides the binary data into groups of 3 bytes (24 bits). Divide these 24 bits into 4 groups of 6 bits, and then find the corresponding characters in the Base64 encoding table. If the length of the binary data is not a multiple of 3, add 0 at the end and add '=' sign at the end of the encoded string for padding.
[0071] Among them, Base128 encoding can re-divide the serialized binary data into binary data that conforms to the ASCII code range (128 characters), and then determine the corresponding characters. Specifically, Base128 encoding uses all 128 characters of ASCII (one byte (8 binary bits, i.e., 8 bits) represents one character) as the character set. Binary data is divided into groups of every 7 bytes (56 bits). These 56 bits are divided into 8 groups of 7 bits (here referred to as subgroups), and then the new bytes are determined. Then based on the new bytes, the corresponding characters can be found in the ASCII code table, that is, the binary data corresponding to the new bytes is the same binary data in the ASCII code table, and the binary data (or characters, etc.) in the ASCII code table and the binary data encoded by Base128 (or the characters corresponding to Base128) are one-to-one corresponding (for example, the binary data encoded by Base128 is 01111100, which is 01111100 in the ASCII code table). If the length of the binary data is not a multiple of 7, 0 is added to the end, and it is only added to the end of the last byte, and it is not filled to a multiple of 7.
[0072] Optionally, converting the binary data in the current group into a plurality of new bytes includes:
[0073] Make sure the first bit of each new byte is 0;
[0074] Divide the binary data in the current group into multiple subgroups with 7 binary bits as one subgroup; wherein the number of the multiple subgroups is the same as the number of the new bytes, and if the number of binary bits of the last subgroup divided in sequence is less than 7, add 0 after the binary data of the last subgroup until the number of binary bits of the last subgroup is equal to 7;
[0075] For each subgroup, the first leading 0 of a new byte and the 7 binary data of the subgroup are encoded into the new byte.
[0076] The binary bit here is the bit.
[0077] In the embodiment of the present disclosure, the following operations are performed for the binary data in each group (taking the current group as an example, the current group here refers to the group that currently needs to be encoded):
[0078] Encode each byte corresponding to the binary data in the current group in sequence, assuming that the length of the binary data in the current group (the length is less than or equal to 56, that is, the length of the binary data in each group is less than 56 bits, that is, 56 bits) is a multiple of 7. For example, the binary data in the current group includes at least N complete bytes (for example, the length is 16 bits, and the binary data in the group corresponds to 2 complete bytes + 2 bits), where N is greater than or equal to 1, and N is less than or equal to 7 (if equal to 7, the length of the binary data is 56 bits). Specifically, for the first byte, the first fixed bit is 0, plus the first 7 bits of the first byte to form the first new byte, and the last bit is left for the next byte; for the second byte, the first fixed bit is 0, the second bit is left over from the previous byte (here refers to the first byte), plus the first 6 bits of the second byte to form the second new byte, and the last 2 bits are left for the next byte; and so on, until the Nth byte, for the Nth byte, the first fixed bit is 0, the second to Nth bits are all left over from the previous byte (here refers to the (N-1)th byte), plus the first (8-N) bits of the Nth byte to form the Nth new byte, and the last N bits are left for the next byte (here refers to the (N+1)th byte); for the (N+1)th byte, the first fixed bit is 0, the second to -(N+1)th bits are all left over from the previous byte (here refers to the Nth byte) to form the new (N+1)th byte. Among them, the new bytes correspond one-to-one to the binary data in the ASCII code table.
[0079] It should be noted that if the length of the binary data is not a multiple of 7, 0 is added to the end, and it is only added to the end of the last byte, and it is not filled to a multiple of 7. For example, the length of the binary data is 16 bits (the binary data in this group corresponds to 2 complete bytes + 2 bits). For the first byte, the first fixed bit is 0, plus the first 7 bits of the first byte to form the first new byte, and the last bit is left for the next byte; for the second byte, the first fixed bit is 0, the second bit is left from the previous byte (here refers to the first byte), plus the first 6 bits of the second byte to form the second new byte, and the last 2 bits are left for the next byte (here refers to the third byte); for the third byte, the first fixed bit is 0, the second bit is left from the previous byte (here refers to the second byte), plus the first 6 bits of the third byte: Since the third byte has only 2 bits at this time, all the bits of the third byte are added here, and the remaining bits are padded with 0 to form the third new byte. Therefore, the binary data in this group is finally encoded into 3 new bytes, which correspond one to one with the binary data in the ASCII code table.
[0080] For example, taking the string abcdefg as an example, the encoding process is as shown in Table 1:
[0081] Table 1 Coding process
[0082]
[0083]
[0084]
[0085] Among them, "0b" is used as a prefix of binary numbers, that is, when a number starts with "0b", the numbers following it should be interpreted as binary numbers.
[0086] After using Base128 encoding, 7 bytes of data need to be represented by 8 bytes. It will take up (8-7) / 7=14.3% of space. But it is worth it, because each character after Base128 encoding only takes up one byte in Redis, while other characters need to take up 1 to 2 bytes. Based on the average occupancy of 1.5 bytes, Base128 can save an average of 1-(8 / 7) / 1.5=23.8% of memory.
[0087] Compared with the native protobuf protocol (here refers to the traditional protobuf protocol), a Chinese character occupies an average of 3.5 bytes after serialization, and occupies 1.5 times the memory after being written to Redis:
[0088] 3.5*1.5=5.25 bytes of memory. However, the improved or improved protobuf disclosed in the present invention, combined with the Base128 encoding method, only 2.5*8 / 7, ≈ 2.86 bytes of memory are occupied, and an average of 1-2.5*8 / 7 / (3.5*1.5) ≈ 45.6% of the memory space can be saved.
[0089] Therefore, compared with the native protobuf serialization solution, the data serialization method provided by the embodiment of the present disclosure can significantly reduce the memory usage stored in Redis.
[0090] In addition, the data serialization method proposed in the present disclosure is not only applicable to the serialization framework protobuf and the database Redis, but also to other binary serialization schemes and cache systems, which will not be repeated here.
[0091] Optionally, the converting the original data into binary data by using an improved binary serialization protocol protobuf includes:
[0092] By using the improved protobuf, a string type field in the original data is identified;
[0093] The string type field is encoded using the GB18030 encoding method in the improved protobuf to obtain binary data.
[0094] In the disclosed embodiment, Protobuf uses GB18030 encoding to encode raw data into binary data. The specific process of Protobuf serialization is as follows: 1) calling the serialization method. 2) processing different types of fields (for example, string type fields such as Chinese characters and numeric characters). 3) processing Unknown fields. 4) generating a byte stream (processing raw data through GB18030 encoding and converting the encoding into binary data).
[0095] Optionally, the method further comprises:
[0096] Read the characters corresponding to the Base128 in the Redis;
[0097] The original data is obtained according to the characters corresponding to the Base128.
[0098] In the disclosed embodiment, data is read from Redis in order to quickly obtain previously stored data so that the data can be used in the application to improve performance, implement real-time data analysis, manage session status, etc.
[0099] Specifically, the main purpose of reading data from Redis is to obtain previously stored key-value data so that the data can be used in the application. For example, 1) Cache data access: Applications often need to query data from the database. In order to improve the query speed, the query results can be cached in Redis. When the same data is needed next time, it can be read directly from Redis without querying the database again. 2) Session management: Web applications often use sessions to track the status of users. Redis can be used to store session data, so that when the user accesses the application, the session information can be quickly read from Redis to restore the user's status. 3) Real-time data analysis: Redis supports various data structures, such as lists, sets, ordered sets, etc., which are very suitable for real-time data analysis. Applications can read this data from Redis for real-time statistics and analysis. 4) Message queues: Redis's list and publish / subscribe functions can be used to implement message queues. Producers can send messages to Redis, and consumers can read messages from Redis for processing.
[0100] The steps of reading data from Redis include:
[0101] 1) Connect to the Redis server: Use a Redis client library (such as redis-py, Jedis, etc.) to connect to the Redis server.
[0102] 2) Send a read command: Read data from Redis using Redis commands such as GET, HGET, etc. These commands require specifying the key and (optional) field to read.
[0103] 3) Processing read results: The Redis client library returns the read data. The application needs to process this data, such as converting it to the appropriate type and performing error handling.
[0104] 4) Close the connection: After reading the data, you can close the connection to the Redis server (if the connection is not persistent).
[0105] Then, according to the read data, such as the characters corresponding to Base128, the corresponding original data, such as a string (Chinese characters, numbers, a string formed by Chinese characters and numbers, etc.), is obtained or output.
[0106] Optionally, writing the characters or strings corresponding to the Base128-encoded binary data into Redis further includes:
[0107] According to the Base128-encoded binary data, determining a corresponding character or string from a character set corresponding to the Base128 encoding method;
[0108] Write the corresponding characters to Redis.
[0109] The corresponding characters here may be one or more, and a plurality of characters may constitute a character string, and no specific limitation is made here on the determined corresponding characters.
[0110] In the disclosed embodiment, the original data is serialized into binary data, and then the serialized binary data is encoded by Base128 encoding. The characters corresponding to the Base128-encoded binary data (the number of characters is not limited here, and multiple characters can form a string in sequence or order) are characters (or strings) in the character set corresponding to the Base128 encoding (that is, the Base128 character set here, for example, the character set includes all 128 characters of ASCII); wherein, the characters corresponding to the Base128-encoded binary data can be one character or multiple characters, and multiple characters are combined into a string. If it is a character, the character is written, and if it is a string, the string is written. It needs to be determined according to the specific original data and is not specifically limited here. Take the above-mentioned original data as the string "abcdefg" as an example. After serialization and Base128-encoded binary data, each byte in the Base128-encoded binary data corresponds to a character as the new character corresponding to the original data. In this way, the string "abcdefg" is a new string corresponding to the serialization and Base128 encoding, and the new string is written into Redis.
[0111] Optionally, the method further comprises:
[0112] When the binary data after the improved protobuf serialization falls within the ASCII code range, the binary data after the improved protobuf serialization is encoded by using the ASCII encoding method to obtain characters corresponding to the ASCII code;
[0113] When the binary data after the improved protobuf serialization does not fall within the ASCII code range, encoding the binary data after the improved protobuf serialization by using the Base128 encoding method to obtain Base128-encoded binary data;
[0114] In the Base128 encoding process, one bit is added, and the bit is used to indicate the encoding method used.
[0115] In the embodiment of the present disclosure, if the original data is serialized into binary data and the binary data is within the ASCII code range, the UTF-8 encoding method can continue to be used, because when the binary byte is within the ASCII code range, it only occupies 1 byte in the UTF-8 encoding and only occupies 1 byte in the Redis memory. The Base128 encoding method can also be used. The specific steps are shown in the above embodiment and will not be repeated here.
[0116] If the original data is serialized into binary data and the binary data is not within the ASCII code range, Base128 encoding can be performed only on the binary data that is not within the ASCII code range to convert it into a binary number within the ASCII code range, and then the corresponding characters determined by the Base128-encoded binary data are written into Redis.
[0117] Since the improved protobuf is used when the original data is serialized into binary data, the purpose of improving efficiency can be achieved. Therefore, the present disclosure can improve the serialization efficiency, thereby improving the cache efficiency, and at the same time increasing the memory footprint.
[0118] From the above description, it can be seen that the present disclosure achieves the following technical effects: using an improved protobuf based on the GB18030 encoding method to encode the original data and convert it into binary data can effectively improve the encoding efficiency, thereby improving the serialization efficiency; and by using the Base128 encoding method that uses all 128 characters of ASCII as the character set for encoding, the binary data serialized by the improved protobuf is encoded, and the binary code can be within the ASCII code range, so that each binary byte or encoded binary character in the encoded binary data (here refers to the character corresponding to each binary byte (i.e., byte) in the encoded binary data) only occupies 1 byte of memory in Redis, which can effectively improve the cache efficiency while reducing the memory footprint.
[0119] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0120] The present disclosure also provides a data serialization device for implementing the above method embodiment. Figure 3 As shown, the data serialization device 30 includes:
[0121] The processing unit 301 is used to convert the original data into binary data through the improved binary serialization protocol protobuf, and the improved protobuf is encoded in the GB18030 encoding method;
[0122] The encoding unit 302 is used to encode the improved protobuf serialized binary data through Base128 encoding to obtain Base128 encoded binary data; wherein the Base128 encoding method uses all 128 characters of ASCII as a character set for encoding, and the Base128 encoded binary data corresponds to the ASCII code one by one;
[0123] The cache unit 303 is used to write the characters or character strings corresponding to the Base128-encoded binary data into Redis.
[0124] Optionally, the encoding unit 302, when encoding the improved protobuf serialized binary data by Base128 encoding to obtain Base128 encoded binary data, specifically includes:
[0125] Divide the binary data after the improved protobuf serialization into a group for every 7 bytes, wherein if the binary data after the improved protobuf serialization is less than 7 bytes, the binary data after the improved protobuf serialization is taken as a group;
[0126] For each group of binary data, perform the following operations to obtain Base128-encoded binary data:
[0127] Convert the binary data in the current group into a plurality of new bytes; wherein the number of the new bytes is equal to the number of bytes corresponding to the binary data in the current group plus 1;
[0128] The binary data of each new byte corresponding to the current group is the binary data in the ASCII code table, and the characters corresponding to the binary data in the ASCII code table are used as the characters corresponding to the Base128.
[0129] Optionally, the encoding unit 302, when converting the binary data in the current group into a plurality of new bytes, specifically includes:
[0130] Make sure the first bit of each new byte is 0;
[0131] Divide the binary data in the current group into multiple subgroups with 7 binary bits as one subgroup; wherein the number of the multiple subgroups is the same as the number of the new bytes, and if the number of binary bits of the last subgroup divided in sequence is less than 7, add 0 after the binary data of the last subgroup until the number of binary bits of the last subgroup is equal to 7;
[0132] For each subgroup, the first leading 0 of a new byte and the 7 binary data of the subgroup are encoded into the new byte.
[0133] Optionally, the processing unit 301 is used to convert the original data into binary data by executing the improved binary serialization protocol protobuf, specifically including:
[0134] By using the improved protobuf, a string type field in the original data is identified;
[0135] The string type field is encoded using the GB18030 encoding method in the improved protobuf to obtain binary data.
[0136] Optionally, the data serialization device is further used to perform the following operations:
[0137] Read the characters corresponding to the Base128 in the Redis;
[0138] The original data is obtained according to the characters corresponding to the Base128.
[0139] Optionally, when the cache unit 303 writes the characters or strings corresponding to the Base128-encoded binary data into Redis, it specifically includes:
[0140] According to the Base128-encoded binary data, determining a corresponding character or string from a character set corresponding to the Base128 encoding method;
[0141] Write the corresponding characters to Redis.
[0142] Optionally, the data serialization device is further used to perform the following operations:
[0143] When the binary data after the improved protobuf serialization falls within the ASCII code range, the binary data after the improved protobuf serialization is encoded by using the ASCII encoding method to obtain characters corresponding to the ASCII code;
[0144] When the binary data after the improved protobuf serialization does not fall within the ASCII code range, the binary data after the improved protobuf serialization is encoded using the Base128 encoding method to obtain Base128-encoded binary data.
[0145] The specific manner in which each unit in the above device embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.
[0146] The present disclosure also provides an electronic device, such as Figure 4 As shown, the electronic device includes one or more processors 41 and a memory 42. Figure 4 A processor 41 is taken as an example.
[0147] The controller may further include: an input device 43 and an output device 44 .
[0148] The processor 41, the memory 42, the input device 43 and the output device 44 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.
[0149] The processor 41 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips. The general-purpose processor may be a microprocessor or any conventional processor.
[0150] The memory 42 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the control method in the embodiment of the present disclosure. The processor 41 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 42, that is, the data serialization method of the above method embodiment is implemented.
[0151] The memory 42 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of a processing device operated by the server, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 42 may optionally include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to a network connection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0152] The input device 43 can receive input digital or character information, and generate key signal input related to user settings and function control of the processing device of the server. The output device 44 can include display devices such as a display screen.
[0153] One or more modules are stored in the memory 42, and when executed by one or more processors 41, the execution is as follows: Figure 2 The method shown.
[0154] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute and implement the data serialization method as described above.
[0155] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the data serialization method as described above when executed by a processor.
[0156] Those skilled in the art can understand that the implementation of all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FM), a hard disk drive (HDD), or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.
[0157] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A data serialization method, characterized in that: include: The original data is converted into binary data through the improved binary serialization protocol protobuf, and the improved protobuf is encoded using the GB18030 encoding method; The improved protobuf serialized binary data is encoded by Base128 encoding to obtain Base128 encoded binary data; wherein the Base128 encoding adopts all 128 characters of ASCII as a character set for encoding, and the Base128 encoded binary data corresponds one to one with the ASCII code; The characters or strings corresponding to the Base128-encoded binary data are written into Redis.
2. The method according to claim 1, characterized in that The improved protobuf serialized binary data is encoded by Base128 encoding to obtain Base128 encoded binary data, including: Divide the binary data after the improved protobuf serialization into a group for every 7 bytes, wherein if the binary data after the improved protobuf serialization is less than 7 bytes, the binary data after the improved protobuf serialization is taken as a group; For each group of binary data, perform the following operations to obtain Base128-encoded binary data: Convert the binary data in the current group into a plurality of new bytes; wherein the number of the new bytes is equal to the number of bytes corresponding to the binary data in the current group plus 1; The binary data of each new byte corresponding to the current group is the binary data in the ASCII code table, and the characters corresponding to the binary data in the ASCII code table are used as the characters corresponding to the Base128.
3. The method according to claim 2, characterized in that The process of converting the binary data in the current group into a plurality of new bytes includes: Make sure the first bit of each new byte is 0; Divide the binary data in the current group into multiple subgroups with 7 binary bits as one subgroup; wherein the number of the multiple subgroups is the same as the number of the new bytes, and if the number of binary bits of the last subgroup divided in sequence is less than 7, add 0 after the binary data of the last subgroup until the number of binary bits of the last subgroup is equal to 7; For each subgroup, the first leading 0 of a new byte and the 7 binary data of the subgroup are encoded into the new byte.
4. The method according to any one of claims 1 to 3, characterized in that: The improved binary serialization protocol protobuf is used to convert the original data into binary data, including: By using the improved protobuf, a string type field in the original data is identified; The string type field is encoded using the GB18030 encoding method in the improved protobuf to obtain binary data.
5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Read the characters corresponding to the Base128 in the Redis; The original data is obtained according to the characters corresponding to the Base128.
6. The method according to any one of claims 1 to 3, characterized in that: Writing the characters or strings corresponding to the Base128-encoded binary data into Redis also includes: According to the Base128-encoded binary data, determining a corresponding character or string from a character set corresponding to the Base128 encoding method; Write the corresponding character or string to Redis.
7. A data serialization device, characterized in that: include: The processing unit is used to convert the original data into binary data through an improved binary serialization protocol protobuf, and the improved protobuf is encoded in a GB18030 encoding method; An encoding unit, used for encoding the improved protobuf serialized binary data by Base128 encoding to obtain Base128 encoded binary data; wherein the Base128 encoding adopts all 128 characters of ASCII as a character set for encoding, and the Base128 encoded binary data corresponds to the ASCII code one by one; The cache unit is used to write the characters or character strings corresponding to the Base128-encoded binary data into Redis.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data serialization method described in any one of claims 1 to 6.
9. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the data serialization method described in any one of claims 1 to 6.
10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the data serialization method described in any one of claims 1 to 6.