Data serialization method, deserialization method, device, equipment and storage medium

Abstract

Embodiments of the application disclose a data serialization method, a deserialization method, a device, equipment and a storage medium. The method comprises: obtaining to-be-serialized data; performing field matching on the to-be-serialized data to obtain a matching result; in a case where the matching result indicates that a field is an empty field, determining a first count value corresponding to the empty field; and when the first count value reaches a preset lower limit value, writing the first count value into a serialization result. In this way, for the to-be-serialized data, especially the empty field in sparse data, only the count value of the empty field is written into the serialization result through a self-defined serialization rule, so that effective compression of a data stream can be realized and transmission efficiency in a data ETL process can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data serialization method, a deserialization method, a device, an apparatus and a storage medium. BACKGROUND

[0002] In the process of data extraction, transformation and loading (ETL), specific data usually needs to be transmitted to a computing engine for calculation and finally landed. In this process, the data needs to be serialized and deserialized multiple times. Common serialization tools include Protobuf, XML, SOAP, YAML, JSON, Thrift, Avro, etc.

[0003] Currently, related technologies usually optimize the serialization and deserialization methods of data in the serialization and deserialization requirements of data transmission and storage universal scenarios, and do not optimize the multiple serialization and deserialization scenarios in the data ETL process. In particular, in the data ETL scenario of sparse data, the transmission efficiency cannot be improved. SUMMARY

[0004] The present application provides a data serialization method, a deserialization method, a device, an apparatus and a storage medium, which can improve the transmission efficiency of data in the sparse data scenario.

[0005] The technical solution of the present application is implemented as follows:

[0006] In a first aspect, the present application provides a data serialization method, which comprises:

[0007] obtaining to-be-serialized data;

[0008] performing field matching on the to-be-serialized data to obtain a matching result;

[0009] in a case where the matching result indicates that the field is an empty field, determining a first count value corresponding to the empty field;

[0010] when the first count value reaches a preset lower limit value, writing the first count value into a serialization result.

[0011] In some embodiments, the method further comprises:

[0012] in a case where the matching result indicates that the field is a non-empty field, determining a second count value corresponding to the non-empty field, and storing the non-empty field after serialization to a preset cache area;

[0013] When the second count value reaches a preset upper limit value, the second count value and the cached data in the preset cache area are written into the serialization result.

[0014] In some embodiments, when the matching result indicates that the field is an empty field, the determination of the first count value corresponding to the empty field comprises:

[0015] determining whether the preset cache area is empty;

[0016] If the preset cache area is not empty, the current count value of the counter and the cached data in the preset cache area are written into the serialization result, and after the counter is cleared and the preset cache area is emptied, the first count value is obtained by performing a minus 1 operation on the counter.

[0017] If the preset cache area is empty, the first count value is directly obtained by performing a minus 1 operation on the counter.

[0018] In some embodiments, when the matching result indicates that the field is a non-empty field, the determination of the second count value corresponding to the non-empty field comprises:

[0019] determining whether there is an unstored first count value;

[0020] If there is an unstored first count value, the first count value is written into the serialization result, and after the counter is cleared, the second count value is obtained by performing a plus 1 operation on the counter;

[0021] If there is no unstored first count value, the second count value is directly obtained by performing a plus 1 operation on the counter.

[0022] In some embodiments, the method further comprises:

[0023] The method further comprises:

[0024] determining whether all the rows to which the data to be serialized belong have completed serialization processing;

[0025] If the determination result is no, the step of obtaining data to be serialized is executed again;

[0026] If the determination result is yes, the count value is written into the serialization result, and it is determined whether the preset cache area is empty;

[0027] If the preset cache area is not empty, after the cached data in the preset cache area is written into the serialization result, an end symbol is written at the end of the serialization result;

[0028] If the preset buffer is empty, a terminator is directly written at the end of the serialization result.

[0029] In a second aspect, the embodiments of the present application provide a data deserialization method, which comprises:

[0030] obtaining a serialization result;

[0031] reading a current byte value from the serialization result;

[0032] If the current byte value is greater than zero, the field of the current byte value quantity after the current byte value is deserialized to obtain a deserialization result.

[0033] If the current byte value is less than zero, the absolute value of the current byte value is determined, and the absolute value quantity of empty fields is deserialized in the deserialization result.

[0034] In some embodiments, after the serialization result is obtained, the method further comprises:

[0035] splitting the serialization result to determine a temporary object;

[0036] parsing a preset expression to obtain target field information and target semantic information;

[0037] selectively deserializing the temporary object according to the target field information to obtain target data;

[0038] calculating the target data according to the target semantic information to obtain a calculation result.

[0039] In some embodiments, after the calculation result is obtained, the method further comprises:

[0040] after the calculation result is serialized, storing the calculation result to the temporary object according to the target semantic information;

[0041] writing the temporary object into the serialization result.

[0042] In a third aspect, the embodiments of the present application provide a data serialization device, which comprises a first obtaining unit, a matching unit and a first determining unit; wherein

[0043] The first obtaining unit is configured to obtain to-be-serialized data.

[0044] The matching unit is configured to perform field matching on the to-be-serialized data to obtain a matching result.

[0045] The first determining unit is configured to determine a first count value corresponding to the empty field when the matching result indicates that the field is an empty field, and write the first count value into a serialization result when the first count value reaches a preset lower limit value.

[0046] In a fourth aspect, an embodiment of the present application provides a data deserialization apparatus, which comprises a second obtaining unit, a reading unit and a deserialization unit.

[0047] The second obtaining unit is configured to obtain a serialization result.

[0048] The reading unit is configured to read a current byte value from the serialization result.

[0049] The deserialization unit is configured to perform deserialization processing on fields with a quantity of the current byte value after the current byte value to obtain a deserialization result if the current byte value is greater than zero, and determine an absolute value of the current byte value and deserialize empty fields with a quantity of the absolute value in the deserialization result if the current byte value is less than zero.

[0050] In a fifth aspect, an embodiment of the present application provides an electronic device, which comprises a memory and a processor.

[0051] The memory is configured to store a computer program capable of running on the processor.

[0052] The processor is configured to execute the data serialization method in the first aspect or the data deserialization method in the second aspect when the computer program is running.

[0053] In a sixth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. The computer program is executed by a processor to implement the data serialization method in the first aspect or the data deserialization method in the second aspect.

[0054] The data serialization method, the deserialization method, the device, the equipment and the storage medium provided by the embodiment of the application, in the process of data serialization, the data to be serialized is obtained; the field matching is performed on the data to be serialized, and a matching result is obtained; in the case that the matching result indicates that the field is an empty field, a first count value corresponding to the empty field is determined; when the first count value reaches a preset lower limit value, the first count value is written into a serialization result. In the process of data deserialization, the serialization result is obtained; the current byte value is read from the serialization result; if the current byte value is greater than zero, the fields of the current byte value quantity after the current byte value are deserialized to obtain a deserialization result; if the current byte value is less than zero, the absolute value of the current byte value is determined, and the absolute value quantity of empty fields is deserialized in the deserialization result. In this way, when the data to be serialized is serialized, especially the empty fields in the sparse data, only the count value of the empty field is written into the serialization result by defining the serialization rule; when the serialization result is deserialized, for the positive digital byte value read, only the fields of the byte value quantity after the current byte value need to be deserialized in sequence; and for the negative digital byte value read, the absolute value quantity of empty fields of the byte value is directly deserialized; by this method, the serialization and deserialization processes of the sparse data in the data ETL process are optimized, so that the effective compression of the data stream can be realized, and finally the transmission efficiency in the data ETL process is improved. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The flowchart of the data serialization method provided by the embodiment of the application is shown in the figure.

[0056] Figure 2 The flowchart of another data serialization method provided by the embodiment of the application is shown in the figure.

[0057] Figure 3 The flowchart of the data deserialization method provided by the embodiment of the application is shown in the figure.

[0058] Figure 4 The flowchart of another data deserialization method provided by the embodiment of the application is shown in the figure.

[0059] Figure 5 The flowchart of another data deserialization method provided by the embodiment of the application is shown in the figure.

[0060] Figure 6 The composition structure diagram of the data serialization device provided by the embodiment of the application is shown in the figure.

[0061] Figure 7 The composition structure diagram of the data deserialization device provided by the embodiment of the application is shown in the figure.

[0062] Figure 8 A hardware structure schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing the embodiments of the present application only and not intended to limit the present application.

[0065] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0066] It should be noted that the terms "first", "second", "third" involved in the embodiments of the present application are only to distinguish similar objects, and do not represent the specific order of the objects. It can be understood that "first", "second", "third" can be interchanged in specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0067] Before the embodiments of the present application are further described in detail, the terms and terms involved in the embodiments of the present application are explained, and the terms and terms involved in the embodiments of the present application are applicable to the following explanations:

[0068] ETL: is the abbreviation of English Extract-Transform-Load, used to describe the process of extracting data from the source end, transforming, loading to the destination end. The word ETL is more commonly used in data warehouse, but its object is not limited to data warehouse.

[0069] Sparse data: in the database, sparse data refers to data containing a large number of null values in a two-dimensional table; in other words, sparse data refers to data in which most values are missing or zero in a data set. Sparse data is not useless data at all, but only incomplete information, and a large amount of useful information can be mined through appropriate means.

[0070] Serialization: The process of converting an object's state information into a form that can be stored or transmitted. During serialization, the object writes its current state to temporary or persistent storage. Later, the object can be recreated by reading or deserializing its state from storage. For example, serialization can refer to the process of converting a data structure into a binary data stream or a text stream. Serialized data can be easily transmitted over a network and stored on disk.

[0071] Deserialization is the process of reading a serialized object and deserializing it back into the original object. For example, deserialization, the opposite of serialization, can be the process of converting a binary data stream or text stream into a data structure that is easy to process and read.

[0072] In related technologies, by introducing the concept of type pointers, the type pointers are serialized along with the data during the serialization process. In this way, during data deserialization, it is no longer necessary to rely on external objects to specify the target type, thus enabling accurate deserialization of objects. Alternatively, by using preset identifiers, the problem that traditional serialization techniques cannot effectively deserialize serialized graph data is solved, realizing the serialization and deserialization of graph data. Furthermore, by parsing the target code, determining the serialization function signature and structure type definition, and generating the target serialization function to serialize the data, the serialization of user-defined target data of non-specified data structure types is realized.

[0073] While these related technologies have significantly complemented traditional methods, most are geared towards serialization and deserialization requirements in general data transmission and storage scenarios. However, data ETL typically involves multiple serialization and deserialization processes, and these technologies are not optimized for scenarios requiring multiple serializations and deserializations, especially for sparse data ETL, where they fail to achieve satisfactory results.

[0074] Based on this, this application provides a data serialization method. The basic idea of ​​the method is: to obtain data to be serialized; to perform field matching on the data to be serialized to obtain a matching result; when the matching result indicates that the field is an empty field, to determine a first count value corresponding to the empty field; when the first count value reaches a preset lower limit value, to write the first count value into the serialization result.

[0075] Correspondingly, this application also provides a data deserialization method. The basic idea of ​​this method is: to obtain a serialization result; to read the current byte value from the serialization result; if the current byte value is greater than zero, to perform deserialization processing on the field after the current byte value that is equal to the current byte value, and to obtain a deserialization result; if the current byte value is less than zero, to determine the absolute value of the current byte value, and to deserialize the number of empty fields equal to the absolute value in the deserialization result.

[0076] In this way, during data serialization, for the data to be serialized, especially empty fields in sparse data, a custom serialization rule is used to write only the count value of the empty field into the serialization result. During data deserialization, byte values ​​are read first. If a positive number of bytes is read, it indicates that non-empty fields need to be deserialized; therefore, the field counting the number of bytes is deserialized sequentially. If a negative number of bytes is read, it indicates that empty fields need to be deserialized; therefore, the field counting the absolute value of the bytes is directly deserialized. This method optimizes the serialization and deserialization processes of sparse data in data ETL, thereby achieving effective compression of the data stream and ultimately improving the transmission efficiency of the data ETL process.

[0077] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0078] In one embodiment of this application, see [link to embodiment]. Figure 1 This illustrates a flowchart of a data serialization method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0079] S101. Obtain the data to be serialized.

[0080] It should be noted that the data serialization method provided in this application embodiment can be applied to a data serialization device or a device integrated with such a device. Here, the device can be such as a computer, smartphone, tablet computer, laptop computer, handheld computer, personal digital assistant (PDA), navigation device, server, etc., and this application embodiment does not specifically limit it in this regard.

[0081] It should be noted that the data serialization method provided in this application embodiment counts empty fields and non-empty fields separately and serializes them according to a custom serialization rule. Simply put, the custom serialization rule here means that when serializing empty fields, only the count value of the empty field needs to be stored in the serialization format; when serializing non-empty fields, both the count value of the non-empty field and the number of count values ​​of the non-empty field are stored in the serialization format.

[0082] In some embodiments, before obtaining the data to be serialized, the method may further include:

[0083] Initialize the counter; initialize the preset buffer.

[0084] It should be noted that this embodiment uses a counter to count the fields in the data to be serialized. Therefore, before obtaining the data to be serialized, the counter needs to be initialized to a value of 0 so that the fields can be counted subsequently. This embodiment establishes a preset buffer to temporarily store the non-empty fields after serialization (hereinafter referred to as serialized fields), and writes the serialized fields stored in the preset buffer into the serialization result when the trigger condition is met.

[0085] S102. Perform field matching on the data to be serialized to obtain the matching result.

[0086] It should be noted that this embodiment determines the field type of the data to be serialized by matching each field individually. For example, if the data to be serialized is a row consisting of five fields (A, B, C, D, E), during serialization, the fields are processed sequentially. First, field A is matched, and the corresponding serialization process is performed based on the matching result (details of the processing are provided in subsequent steps). Then, field B is matched, and the corresponding serialization process is performed based on the matching result; this continues until all fields have been processed. During the sequential processing of fields, if a trigger condition is met, the result of the current processing is written into the serialization result (details are provided in subsequent steps). In other words, the entire process is equivalent to traversing and processing the fields in the data to be serialized according to certain rules.

[0087] It should also be noted that the purpose of performing field matching on the serialized data is to determine the field type of the current field. The field type can include: an empty field or a non-empty field. An empty field is a field where the matching result is an empty value (null), while a non-empty field is a field where the matching result is not null.

[0088] S103. If the matching result indicates that the field is an empty field, determine the first count value corresponding to the empty field.

[0089] It should be noted that after obtaining the matching result for a field, it is necessary to count that field. If the matching result indicates that the field is empty, then the empty field is counted to determine the first count value corresponding to the empty field. The first count value represents the count value of the empty field counted by the counter. In addition, in this embodiment of the application, the count value of the non-empty field counted by the counter is referred to as the second count value.

[0090] In some embodiments, when the matching result indicates that the field is an empty field, determining the first count value corresponding to the empty field may include:

[0091] Check if the preset cache is empty;

[0092] If the preset buffer is not empty, the current count value of the counter and the cached data in the preset buffer are written into the serialization result. After clearing the counter and emptying the preset buffer, the first count value is obtained by decrementing the counter by 1.

[0093] If the preset buffer is empty, the first count value is obtained by directly decrementing the counter by 1.

[0094] It should be noted that the empty field matched at this point may or may not be the first field in the data to be serialized. Regardless of whether it is the first field or not, it can be processed in this way. That is:

[0095] Determine if the preset buffer is empty, that is, determine if there are still serialized fields in the preset buffer after serializing non-empty fields.

[0096] If, before the current match of an empty field, the previous one or multiple consecutive matches were all non-empty fields, then the counter value at this time is the count value of the previous one or multiple consecutive matches of non-empty fields (i.e., the second count value). At this time, the preset buffer will cache the serialized fields corresponding to the one or multiple consecutive non-empty fields, that is, the preset buffer is non-empty. Then, the current count value of the counter (at this time, the current count value represents the second count value) is written into the serialization result in serialization format (the writing of data into the serialization result as described in this application embodiment refers to writing it into the serialization result in serialization format), and simultaneously, the cached data in the preset buffer is written into the serialization result. It can be understood that the content written into the serialization result this time is a serialized field containing a second count value in serialization format and the number of serialized fields containing the second count value. After writing the second count value and the serialized field into the serialization result, the counter is cleared to zero and the preset buffer is cleared. Then, the counter is decremented by 1 to obtain the first count value. It can be understood that, since the counter is cleared to zero first, the first count value obtained after this decrementing process is -1. In this embodiment, negative numbers are used to count empty fields, and their absolute values ​​represent the number of empty fields that have been counted in this counting process.

[0097] As can be seen, one of the triggering conditions for writing the processing result into the serialization result is: the field type changes from a non-empty field to an empty field. At this time, it is necessary to write the second count value of the non-empty field and the serialization field of the number of second count values ​​into the serialization result, then clear the counter, clear the read and write cache, and count the empty field.

[0098] If the preset buffer is empty, the counter is decremented by 1 to obtain the first count value. This can be understood as follows: an empty preset buffer indicates that the field being matched might be the first field in the data to be serialized; or the matching result has consistently been an empty field from the first field to this field; or the matching result has consistently been an empty field since the preset buffer was cleared after the serialization result was written to it, etc. Therefore, if the empty field is the first field in the data to be serialized, the first count value is -1.

[0099] It should also be noted that after performing field matching on the data to be serialized, the matching result may also indicate that the field is not empty. Therefore, in some embodiments, the method may further include:

[0100] If the matching result indicates that the field is a non-empty field, determine the second count value corresponding to the non-empty field, and serialize the non-empty field and store it in a preset cache area.

[0101] It should be noted that when the matching result indicates that the field is not empty, it is necessary to count the non-empty field to determine the second count value corresponding to the non-empty field. At the same time, the content of the non-empty field is serialized into a serialized field and stored in the preset cache. The second count value represents the count value of the non-empty field by the counter.

[0102] In some embodiments, when the matching result indicates that the field is a non-empty field, determining the second count value corresponding to the non-empty field may include:

[0103] Determine if there is a first count value that is not stored;

[0104] If there is a first count value that is not stored, the first count value is written into the serialization result, and after the counter is cleared, the second count value is obtained by incrementing the counter by 1.

[0105] If there is no unstored first count value, the second count value is obtained by directly incrementing the counter by 1.

[0106] It should be noted that, similar to the case where the matching result is an empty field, the matched non-empty field may or may not be the first field in the data to be serialized. Regardless of whether it is the first field in the data to be serialized, it can be processed in this way. That is:

[0107] To determine whether there is an unstored first count value, specifically, since the embodiments of this application count empty fields with negative numbers, it is possible to determine whether there is an unstored first count value by judging the sign of the current counter value.

[0108] If the current counter value is negative, it indicates that there is an unstored first count value, meaning that before the current matching result is a non-empty field, the previous one or multiple consecutive matches were all empty fields. It should be noted that in this embodiment, processing empty fields only requires counting them using a counter (no data is cached in a preset buffer). Therefore, the current counter value (representing the first count value) is simply written into the serialization result in a serialized format. It can be understood that the content written into the serialization result is a serialized first count value. After writing the first count value into the serialization result, the counter is cleared to zero, and then incremented by 1 to obtain the second count value. It can be understood that since the counter is cleared first, the second count value obtained after incrementing by 1 is 1. This embodiment uses positive numbers to count non-empty fields, representing the number of non-empty fields that have been counted in the current counting process.

[0109] As can be seen, another triggering condition 2 for writing the processing result into the serialization result is: the field type changes from an empty field to a non-empty field. At this time, the first count value of the empty field needs to be written into the serialization result, then the counter is cleared, the non-empty field is counted, and the content of the non-empty field is serialized into a serialized field and stored in the preset buffer.

[0110] S104. When the first count value reaches the preset lower limit, the first count value is written into the serialization result.

[0111] It should be noted that the counter is not infinitely countable, but has a lower limit. Therefore, once the counter reaches the preset lower limit for counting empty fields, it can no longer count. At this point, the first count value is written into the serialization result in serialized format, and then the counter is reset to zero. This first count value is the preset lower limit. Preferably, the preset lower limit is -128. For example, if there are 130 consecutive empty fields in a row of serialized fields, but the lower limit of the counter is -128, meaning it can only count a maximum of 128 consecutive empty fields, after the counter reaches -128, -128 is first written into the serialization result in serialized format, then the counter is reset to zero, and then the remaining empty fields are counted again.

[0112] As can be seen, another trigger condition 3 for writing the processing result into the serialization result is: the first count value corresponding to the empty field reaches the preset lower limit value. At this time, the first count value is written into the serialization result and the counter is cleared.

[0113] It should also be noted that the counter has an upper limit on the value it can count; therefore, in some embodiments, the method may further include:

[0114] When the second count value reaches the preset upper limit, the second count value and the cached data in the preset buffer are written into the serialization result.

[0115] It should be noted that once the counter reaches the preset upper limit for counting non-empty fields, it can no longer count. At this point, the second count value is written to the serialization result in serialized format, then the counter is cleared and the read / write buffer is emptied. This second count value is the preset upper limit. Preferably, the preset upper limit is 127. For example, if there are 130 consecutive non-empty fields in a row of serialized fields, but the counter's upper limit is 127 (meaning it can only count a maximum of 127 consecutive non-empty fields), after the counter reaches 127, 127 is first written to the serialization result in serialized format, and the cached data in the preset buffer is written to the serialization result. Then the counter is cleared, and the remaining non-empty fields are counted again, serialized, and stored in the preset buffer.

[0116] As can be seen, another triggering condition 4 for writing the processing result into the serialization result is: the second count value corresponding to the non-empty field reaches the preset upper limit value. At this time, the second count value and the cached data in the preset buffer are written into the serialization result.

[0117] It should also be noted that, in this embodiment of the application, each line of data in the data to be serialized is serialized using this method. When a line of data is serialized, the processing of that line of data needs to be stopped first, and then the next line of data is serialized using the method of this embodiment of the application. Therefore, in some embodiments, the method may further include:

[0118] Determine whether all rows of the data to be serialized have been serialized.

[0119] If the judgment result is negative, then return to the step of obtaining the data to be serialized;

[0120] If the determination result is yes, then the count value is written into the serialization result, and it is determined whether the preset buffer is empty;

[0121] If the preset buffer is not empty, the cached data in the preset buffer is written into the serialization result, and an end character is written at the end of the serialization result;

[0122] If the preset buffer is empty, an end character is written at the end of the serialization result.

[0123] It should be noted that, in cases such as the following:

[0124] (1) When the first count value reaches the preset lower limit value, the first count value is written into the serialization result and the counter is cleared;

[0125] (2) When the second count value reaches the preset upper limit value, the second count value and the serialization field are written into the serialization result, the counter is cleared and the preset buffer is cleared;

[0126] (3) When the first count value does not reach the preset lower limit value;

[0127] (4) When the second count value does not reach the preset upper limit value.

[0128] In situations like those described above, the data in a given row may or may not have been processed. Therefore, it's necessary to determine whether the entire row containing the data to be serialized has been processed. If the result is negative, meaning the row hasn't been fully processed, the process returns to retrieving the data to be serialized and continues matching the next field. If the result is positive, meaning the row has been fully processed, the current count is written to the serialization result, and it's checked whether the preset buffer is empty. If the preset buffer is not empty, the cached data in the preset buffer is written to the serialization result, and an end-of-line character is written at the end of the serialization result. If the preset buffer is empty, an end-of-line character is directly written at the end of the serialization result.

[0129] In this embodiment of the application, preferably, the terminator is 0.

[0130] It should also be noted that if the processing of a line of data is determined to be complete when the first count value reaches the lower limit or the second count value reaches the upper limit, since the previous steps have already reset the counter to zero and cleared the preset buffer, the count value written to the target end at this time is 0, and the preset buffer is necessarily empty. Since the preferred end-of-line character in this embodiment is also 0, the count value written at this time is the end-of-line character. In both cases, after writing the count value to the serialization result, the serialization of a line of data can be completed, and the serialization process for the next line of data can begin or end.

[0131] As can be seen, another triggering condition 5 for writing the processing result into the serialization result is: all rows of the data to be serialized have completed data processing. If there is a first count value, or a second count value and a serialization field that have not been written into the serialization result, then they are written into the serialization result, and an end character is written at the end of the serialization result.

[0132] This application provides a data serialization method, which involves: acquiring data to be serialized; performing field matching on the data to be serialized to obtain a matching result; determining a first count value corresponding to the empty field when the matching result indicates that the field is empty; and writing the first count value into the serialization result when the first count value reaches a preset lower limit. In this way, for data to be serialized, especially empty fields in sparse data, by customizing serialization rules, only the count value of the empty field is written into the serialization result, thereby achieving effective compression of the data stream and improving the transmission efficiency during the data ETL process.

[0133] In another embodiment of this application, see Figure 2 This illustrates a detailed flowchart of a data serialization method provided in an embodiment of this application. Figure 2 As shown, the method may include:

[0134] S201. Initialize counter i = 0; initialize read / write cache.

[0135] It should be noted that the counter here is used to count the fields of the data to be serialized, and the read-write buffer (i.e., the preset buffer in the aforementioned embodiment) temporarily buffers the data that has been counted but not yet written to the target end (also called serialized data, i.e., the serialized fields in the aforementioned embodiment). Therefore, before serializing the data to be serialized, the counter is initialized so that the counter value (i.e., the count value) i = 0, and the read-write buffer is initialized at the same time so that it can be used to temporarily buffer the serialized data that has not been written to the target end in subsequent steps. Preferably, the value that the counter can record is defined in one byte, so the counter can record values ​​in the range of -128 to 127.

[0136] It should also be noted that the target end is the storage area where the results obtained after data serialization are stored, such as local storage location or database, depending on the actual application requirements. This application embodiment does not make specific limitations on this.

[0137] S202. Read in the data to be serialized and perform field matching.

[0138] It should be noted that the data serialization method provided in this application is a complete process of serializing a row of data in the data to be serialized. This involves matching and processing the row of data field by field, which can also be understood as a process of traversing and processing the fields of the data to be serialized. Therefore, after reading the data to be serialized, the fields are first matched according to the order of the data. In one execution of this step, only one field of the data to be serialized is matched to determine its type, and the next step is determined based on the matching result. For example, assuming the data to be serialized is a row consisting of five fields A, B, C, D, and E, if this matching is the first match in this process, the matched field is A. Then, the matching result for field A is processed, and then B, C, D, and E are matched and processed sequentially according to the process.

[0139] S203. Determine if a field is null.

[0140] It should be noted that matching the field to determine whether the field is null is used to confirm whether the field is an empty field (i.e., an empty field in the aforementioned example).

[0141] If the judgment result is yes, then proceed to step S204; if the judgment result is no, then proceed to step S2010.

[0142] S204. Determine if the read / write cache is empty.

[0143] It should be noted that the process in this embodiment can be viewed as a loop process, meaning that before a row of data is serialized, this process will continue to process each field in the serialized data. Since it has been determined that the field is null, there are possibilities: the read / write cache contains serialized data that has not yet been written to the target end, i.e., the read / write cache is not empty; or the read / write cache has not cached any data before, or although data was cached, it has been cleared, etc. For example, if this field is the first field to be processed, then naturally no data has been stored in the read / write cache, or no new data has been stored since the read / write cache was last cleared, i.e., the read / write cache is empty.

[0144] If the judgment result is yes, then proceed to step S207; if the judgment result is no, then proceed to step S205.

[0145] S205. Write the count value to the target end; write the cached data to the target end.

[0146] It should be noted that if the read / write cache is not empty, it means that the read / write cache contains serialized data that has not yet been written to the target. In this case, the counter value represents the number of non-empty fields that have not yet been written to the target. The cached data in the read / write cache is the data that has been serialized from the non-empty fields and cached in the read / write cache, but has not yet been written to the target. Therefore, the counter value is written to the target, and the cached data in the read / write cache is also written to the target.

[0147] S206, Reset the counter to zero (i = 0); Clear the read / write cache.

[0148] It should be noted that after writing the count value and the data in the read / write cache to the target end, it is necessary to count the null value fields matched this time. At this time, the counter is reset to zero (i=0), and the read / write cache is cleared to complete the operation on the cached data in the read / write cache.

[0149] S207, i = i-1.

[0150] It should be noted that at this time, there is no need to write the null value field to the read / write cache; it is only necessary to count the null value field. Preferably, a negative number is used to count the null value field, i.e., i = i - 1, where i before the equals sign represents the current count value of the null value field, and i after the equals sign represents the previous count value of the null value field.

[0151] It is understandable that if step S207 follows step S204, i.e., the read / write cache is empty, for example, if the field is the first field in this data serialization process, then i after the equal sign = 0, and after counting the null value field, i = 0 - 1 = -1; if the field is not the first field in this data serialization process, then i after the equal sign ≠ 0, and the calculated i should be an integer less than -1. Here, the absolute value of i represents the number of null values ​​counted, and the negative sign indicates that the count value represents the number of null value fields.

[0152] It can also be understood that if step S207 is followed by step S206, since the counter has been reset to zero and the read / write cache has been cleared in step S206, i.e., the count value after the equal sign i = 0, the read / write cache is empty, and after counting this empty value field, i = 0 - 1 = -1.

[0153] S208. Determine if i has reached BYTE_MIN.

[0154] It should be noted that the counter is not infinitely countable, but has a lower limit BYTE_MIN based on the actual settings. Therefore, it is necessary to determine whether the counter has reached the lower limit. Since i is preferably defined in bytes in this embodiment, that is, the count value will be stored in one byte, the lower limit BYTE_MIN is preferably -128.

[0155] If the judgment result is yes, then proceed to step S209; if the judgment result is no, then proceed to step S2016.

[0156] S209. Write the count value to the target terminal; reset the counter to zero i = 0.

[0157] It should be noted that since the count of null fields has reached its lower limit, the count needs to be written to the target terminal, and the counter reset to zero. This means the count written to the target terminal is now a negative number, and its absolute value represents the number of null fields counted.

[0158] After step S209, step S2016 is executed.

[0159] S2010, Determine if there are any unrecorded null values.

[0160] It should be noted that this step follows step S203, which is executed when the field is not null, i.e., the field is a non-null field. Since this process is a loop, it is possible that the counter has already counted a certain number of null value fields but has not yet written the count value to the target end (i.e., there are unrecorded null values; here, "recording" refers to writing the count value, or the count value and data in the read / write cache, to the target end); or the count value was 0 before this, or the counter was counting non-null fields before this (i.e., no unrecorded null values). For example, if this is the first field to be processed, the counter value is still 0; or if i non-null fields have already been counted, the counter value is a positive number. Since this embodiment preferably uses negative numbers to count null value fields, preferably, this embodiment determines whether there are unrecorded null values ​​by judging whether the current counter value i is negative. If it is negative, it means there are unrecorded null values; otherwise, it means there are no unrecorded null values.

[0161] If the judgment result is yes, then proceed to step S2011; if the judgment result is no, then proceed to step S2012.

[0162] S2011. Write the count value to the target terminal; reset the counter to zero i = 0.

[0163] It should be noted that if there are unrecorded null values, i.e., the counter value is negative, its absolute value represents the number of null values ​​to be recorded. This count value is written to the target terminal in byte format, and the counter is reset to zero. This completes the recording of unrecorded null values.

[0164] S2012, i = i + 1; Cache the serialized field into the read / write cache.

[0165] It should be noted that the counter has already been cleared in step S2011, and now it is necessary to count the non-empty fields (i.e., the non-empty fields in the aforementioned embodiments). Preferably, positive numbers are used to count the non-empty fields, i.e., i = i + 1, where i before the equals sign represents the current count value of the non-empty field, and i after the equals sign represents the previous count value of the non-empty field. Simultaneously, the non-empty fields are serialized and cached in the read / write cache.

[0166] It is understandable that if step S2012 continues from step S2010, that is, there are no unrecorded null values, for example, if the field is the first field in this data serialization process, then i after the equal sign is 0, and after counting the non-null value field, i = 0 + 1 = 1; if the field is not the first field in this data serialization process, then i after the equal sign is ≠ 0, and the calculated i should be an integer greater than 1. Here, the value of i represents the number of non-null value fields counted. A positive value of i indicates that the count value represents the number of non-null value fields.

[0167] It can also be understood that if step S2012 continues with step S2011, since the counter has been reset to zero in step S2011, i.e., the count value after the equal sign i = 0, the read and write cache is empty, and after counting the empty value field, i = 0 + 1 = 1.

[0168] S2013, Determine whether i has reached BYTE_MAX.

[0169] It should be noted that the counter is not infinitely countable, but has an upper limit value BYTE_MAX based on the actual settings. It is necessary to determine whether the counter has reached the upper limit. Since i is preferably defined as a byte in this embodiment, the upper limit value BYTE_MAX is preferably 127.

[0170] If yes, proceed to step S2014; otherwise, proceed to step S2016.

[0171] S2014. Write the count value and serialized data to the target end.

[0172] It should be noted that since the counter value has reached its upper limit, the counter value is written to the target end, and the data in the read / write buffer, i.e., the serialized data, is also written to the target end. It can be understood that the counter value written to the target end at this point is a positive number, representing the number of non-empty fields counted. The data written from the read / write buffer to the target end is the serialized data corresponding to the non-empty fields that represent the count value.

[0173] S2015, Reset the counter to zero (i = 0); Clear the read / write cache.

[0174] It should be noted that after writing the counter value and the data in the read / write cache to the target end, the counter is reset to zero and the read / write cache is cleared.

[0175] S2016. Determine whether the processing of this row of data has ended.

[0176] It should be noted that in this embodiment of the application, the serialization of data is performed line by line. Therefore, it is necessary to determine whether a line of data has been processed.

[0177] If the judgment result is yes, then proceed to step S2017; if the judgment result is no, then proceed to step S202, and continue to repeat this process.

[0178] It can be understood that when jumping back to step S202, the process involves matching the next field in the data to be serialized after the currently matched field and executing this procedure. Alternatively, it can be understood that since this embodiment serializes the data to be serialized, and a certain number of fields have already been processed, the data to be serialized again will naturally not contain fields that have already been serialized. In short, after jumping back to step S202, the process involves matching and processing the next field in the next data to be serialized.

[0179] S2017. Write the count value to the target terminal.

[0180] It should be noted that if it is determined that the data in a row has been processed, the count value is written to the target end. It can be understood that if the next step is S208 or S2013, the absolute value of the count at this point represents the number of null or non-null fields counted; if the next step is S209 or S2015, the counter value at this point is 0.

[0181] S2018. Determine if the read / write cache is empty.

[0182] It should be noted that since the data in this row has been processed, it is now necessary to determine whether there is still serialized data in the read / write cache that has not been written to the target end.

[0183] If the judgment result is "otherwise", proceed to step S2019; if the judgment result is "yes", end the processing of a row of data.

[0184] It should be noted that if step S2017 is followed by step S209 or S2015, the value written to the target end will be 0, which is the end marker in this embodiment. In this case, since the read / write cache has been cleared in the previous steps, the read / write cache will necessarily be empty. Therefore, in step S2017, if the count value written to the target end is 0, the processing of a row of data can be ended directly. Of course, the processing of a row of data can also be ended after the judgment in this step; this embodiment does not specifically limit this.

[0185] S2019. Write the cached data to the target end.

[0186] It should be noted that if the read / write cache is not empty, it means there is still serialized data that has not been stored on the target. In this case, the serialized data in the read / write cache is written to the target, and the read / write cache is cleared. Then, step S2020 is executed.

[0187] S2020: Write the end-of-line character 0 to the target.

[0188] It should be noted that after a line of data is processed, the terminator 0 is written to the target end, thus completing the serialization of a single line of data.

[0189] For example, the following shows a sample of the serialized data format obtained after serializing a row of data:

[0190]

[0191] It should also be noted that, summarizing the data serialization method of this application embodiment, there are five triggering conditions for writing data to the target end, namely:

[0192] (1) When the count value of an empty field reaches the lower limit, the count value needs to be written to the target end; (2) When the count value of a non-empty field reaches the upper limit, the count value and the data in the read / write cache need to be written to the target end; (3) When the matched field changes from an empty field to a non-empty field, the count value of the empty field needs to be written to the target end; (4) When the matched field changes from a non-empty field to an empty field, the count value of the non-empty field and the data in the read / write cache need to be written to the target end; (5) When a row of data serialization ends, if there is a count value of an empty field that has not yet been written to the target end, the count value of the empty field that has not yet been written to the target end and the end character need to be written to the target end; or, if there is data in the read / write cache, the count value of the non-empty field that has not yet been written to the target end, the data in the read / write cache and the end character need to be written to the target end; or, if the writing of the previous batch of data just ends, there is no count value or count value and cached data that has not yet been written to the target end, then only the end character needs to be written to the target end.

[0193] It should also be noted that the data serialization steps S201 to S2020 are steps to serialize one row of data in the data to be serialized. If there are multiple rows of data to be serialized, after the serialization of one row of data is completed, the remaining data to be serialized will continue to be serialized in this process until all data is serialized.

[0194] It should also be noted that, in this embodiment, positive numbers are used to count non-empty fields and negative numbers are used to count empty fields; of course, positive numbers can also be used to count empty fields and negative numbers to count non-empty fields, or other methods and numerical rules can be used for counting. This embodiment does not specifically limit this.

[0195] This application provides a data serialization method. The specific implementation of the aforementioned embodiments has been described in detail through the above embodiments. It can be seen that by counting empty fields and non-empty fields in the data to be serialized separately, for empty fields, only their count values ​​are written to the target end, and for non-empty fields, both their count values ​​and the serialized non-empty fields are written to the target end. In this way, through this custom serialization rule, data can be effectively compressed during data serialization, especially for sparse data, which greatly improves the data transmission efficiency during the data ETL process.

[0196] In another embodiment of this application, see [link to application]. Figure 3 This illustrates a flowchart of a data deserialization method provided in an embodiment of this application. Figure 3 As shown, the method may include:

[0197] S301. Obtain the serialization result.

[0198] It should be noted that the data deserialization method provided in this application embodiment can be applied to a data deserialization device or a device integrated with such a device. Here, the device can be such as a computer, smartphone, tablet computer, laptop computer, handheld computer, personal digital assistant (PDA), navigation device, server, etc., and this application embodiment does not specifically limit it in this regard.

[0199] It should also be noted that the serialization result obtained in the embodiments of this application is obtained by serializing the data to be serialized in the foregoing embodiments. That is, the data deserialization method provided in the embodiments of this application corresponds to the data serialization method provided in the foregoing embodiments.

[0200] S302. Read the current byte value from the serialization result.

[0201] It should be noted that when deserializing the data stream to be deserialized, a byte value is first read from the data stream to be deserialized, and then the sign of the byte value is determined.

[0202] S303. If the current byte value is greater than zero, then the field following the current byte value with the number of current byte values ​​is deserialized to obtain the deserialization result.

[0203] It should be noted that the data deserialization method provided in this application is corresponding to the data serialization method provided in the foregoing embodiments. Since the foregoing embodiments preferably use positive numbers to count non-empty fields, if the current byte value is greater than zero, it means that there is a serialized field with the current byte value after the current byte value in the serialization result, and then it is deserialized to obtain the deserialization result.

[0204] S304. If the current byte value is less than zero, determine the absolute value of the current byte value and deserialize an empty field containing the absolute value in the deserialization result.

[0205] It should be noted that the data deserialization method provided in this application is corresponding to the data serialization method provided in the foregoing embodiments. Since the foregoing embodiments preferably use negative numbers to count empty fields, if the current byte value is less than zero, it means that the absolute value of the number of empty fields should be deserialized at this time.

[0206] It should also be noted that deserialization of the serialization result is also performed line by line. Therefore, in some embodiments, the method may further include:

[0207] If the current byte value is an end-of-serializer, then the deserialization of the line to which the serialization result belongs is completed.

[0208] It should be noted that in the aforementioned data serialization embodiment, preferably, the end-of-line character is 0. Therefore, if the current byte value read is equal to zero, it indicates that the data in that line has been deserialized. At this point, the deserialization of the unfinished serialization results can continue according to this step until the deserialization of all serialization results is completed.

[0209] It should also be noted that when data needs to be computed, the serialized results must first be deserialized. However, in practice, such as in data ETL scenarios, the computation process may not involve all serialized fields, and after obtaining the computation result, the result still needs to be serialized for subsequent data storage. In this case, the traditional processing mode would deserialize all serialized results and then serialize them all after the computation is complete. This would slow down the execution efficiency of the entire computation process. Therefore, this application proposes a delayed deserialization mechanism.

[0210] In some embodiments, after obtaining the serialization result, the method may further include:

[0211] The serialization result is segmented to determine temporary objects;

[0212] Parse the preset expression to obtain target field information and target semantic information;

[0213] The temporary object is selectively deserialized based on the target field information to obtain the target data;

[0214] The target data is calculated based on the target semantic information to obtain the calculation result.

[0215] It should be noted that, since only a portion of the serialization results need to be deserialized for calculation, this embodiment first segments the serialization results according to certain rules, such as a custom schema, after obtaining the serialization results to determine temporary objects. These temporary objects can consist of multiple byte arrays (byte[]). Furthermore, the preset expression represents the calculation formula used in the calculation process. Parsing it yields target field information and target semantic information. Based on this target field information, the corresponding fields to be deserialized can be selected from the temporary objects. Then, the selected serialization fields are deserialized according to the aforementioned data deserialization method, and the resulting data is the target data. In other words, the temporary objects are selectively deserialized, and the target data is then calculated based on the target semantic information to obtain the calculation result.

[0216] After obtaining the calculation result, it may be necessary to store and transmit the data. Therefore, in some embodiments, after obtaining the calculation result, the method may further include:

[0217] After the calculation result is serialized, it is stored in the temporary object according to the target semantic information; the temporary object is then written into the serialized result.

[0218] It should be noted that, since the calculation results need to be stored and transmitted, the calculation results are serialized before being stored in a temporary object, and then the temporary object is written into the serialized result. Here, the method for serializing the calculation results can be the data serialization method described in the foregoing embodiments.

[0219] This application provides a data deserialization method. When deserializing the serialization result, if a positive number of bytes is read, only the field representing the number of subsequent bytes needs to be deserialized sequentially; if a negative number of bytes is read, an empty field representing the absolute number of bytes is directly deserialized. Furthermore, for application scenarios requiring multiple serialization and deserialization processes during data ETL, this application also proposes a delayed deserialization mechanism. By selectively deserializing the data stream to be deserialized, unnecessary serialization and deserialization operations are avoided during the ETL process, further improving the overall processing efficiency of the data ETL process.

[0220] In another embodiment of this application, see [reference needed]. Figure 4 This illustrates a detailed flowchart of a data deserialization method provided in an embodiment of this application. Figure 4 As shown, the method may include:

[0221] S401. Obtain the data stream to be deserialized.

[0222] It should be noted that during the data deserialization process, it is necessary to first obtain the data stream to be deserialized. Here, the data stream to be deserialized is the serialization result in the aforementioned embodiments. Since serialization can be the process of converting a data structure into a binary data stream or a text stream, the serialized data is convenient for transmission over a network and storage on a hard drive. Therefore, the serialization result can also be referred to as the data stream to be deserialized.

[0223] S402, Read bytes from the data stream, n = readByte.

[0224] It should be noted that after obtaining the data stream to be deserialized, bytes are read from the data stream, and the value read is n = readByte.

[0225] S403. Determine if n is greater than 0.

[0226] It should be noted that the data deserialization method provided in this application corresponds to the data serialization method provided in the foregoing embodiments. In the foregoing embodiments, preferably, positive numbers are used to count non-empty fields, and negative numbers are used to count empty fields. Therefore, in this application embodiment, preferably, n greater than 0 indicates that there are n non-empty fields after n in the serialized data; n less than 0 means that the absolute value of n empty fields should be deserialized.

[0227] In addition, a counting method can be selected based on the actual situation. For example, positive numbers can be used to count empty fields, and negative numbers can be used to count non-empty fields. In this case, n less than 0 indicates that there are n non-empty fields after n in the serialized data; n greater than 0 indicates that the absolute value of n empty fields should be deserialized. This application does not impose specific limitations on this.

[0228] If the judgment result is yes, then proceed to step S404; if the judgment result is no, then proceed to step S405.

[0229] S404. Deserialize the following n fields.

[0230] It should be noted that n > 0, meaning there are n fields that need to be deserialized after this byte. In this case, the byte stream is read forward, and the following n fields are deserialized sequentially according to the custom schema to obtain the deserialization result. Then, the process jumps back to step S402. The schema in the database is a collection of database objects. In this embodiment, the schema can be customized according to requirements.

[0231] S405. Determine if n is less than 0.

[0232] If yes, proceed to step S406; otherwise, proceed to step S407.

[0233] S406, deserialize to get abs(n) null values.

[0234] Since the aforementioned embodiment only recorded the count value when serializing the empty field, there is no need to read the byte stream afterward. It is only necessary to deserialize the subsequent abs(n) fields to null according to the custom schema, where abs(n) represents the absolute value of n. Then, jump back to step S402.

[0235] S407. Complete the deserialization of a single row of data.

[0236] It should be noted that, in the data serialization method provided in the foregoing embodiments, preferably, when a line of data serialization is completed, an end-of-line character 0 is written to the target end. Therefore, in this embodiment, correspondingly preferably, when n=0, it indicates that this line has been deserialized. At this time, the deserialization of a single line of data ends, and the next line of data continues to be deserialized according to steps S401 to S407 until the serialization of the data stream to be deserialized is completed.

[0237] It's important to note that in computational scenarios such as data ETL, the computation process may not involve all serialized fields. However, after the data computation, the result data still needs to be serialized for subsequent data storage. Traditional processing methods would deserialize all serialized fields that didn't participate in the computation before reserializing them. This is clearly an unnecessary and redundant operation that slows down the entire computation process.

[0238] Therefore, embodiments of this application propose a mechanism for delayed deserialization. See also Figure 5 This illustrates a flowchart of a data selective deserialization method provided in an embodiment of this application. Figure 5 As shown, the method may include:

[0239] S501. Obtain the data stream to be deserialized.

[0240] S502. Segment the data stream to be deserialized to obtain a temporary object.

[0241] It should be noted that, in this embodiment of the application, since only a portion of the serialized fields are involved in the calculation process, only this portion of the serialized fields needs to be deserialized for calculation. Therefore, in the calculation process, the unserialized data stream is first segmented according to the custom schema to obtain a temporary object composed of multiple byte[].

[0242] S503, Parse the calculation expression; obtain the information of the calculated column.

[0243] It should be noted that, since this embodiment of the application selectively serializes the data stream to be serialized, it is necessary to determine which serialization fields need to be deserialized. Therefore, based on the calculations required during the data ETL process, the calculation expressions are parsed to obtain the relevant field information and calculation semantics of the calculation expressions, and the calculated column information is obtained.

[0244] S504. Perform selective deserialization on temporary objects.

[0245] It should be noted that by combining the relevant field information and calculated column information obtained from parsing the calculation expression in step S503, selective deserialization of the temporary object can be performed. This means that only the serialized fields that need to be calculated are deserialized, while the serialized fields that do not need to be calculated remain serialized.

[0246] S505. Calculate the deserialization result to obtain the calculation result.

[0247] It should be noted that after selectively deserializing the serialized fields in the temporary object in step S504, the deserialized result data obtained in step S504 can be used to complete the calculation in combination with the computational semantics obtained in step S503, and the calculation result can be obtained.

[0248] S506. Serialize the calculation result and write it back to the temporary object.

[0249] It should be noted that after obtaining the calculation result, the result can be serialized to obtain the serialized fields, and then written back to the temporary object according to the calculation semantics.

[0250] S507. Write the temporary object back to the target.

[0251] Finally, the temporary object is written back to the target, thus completing the computation of the serialized data that needs to be computed.

[0252] In other words, in computational scenarios, this embodiment does not directly deserialize data to obtain a structure object according to a custom schema. Instead, it segments the data stream bytes according to the custom schema to obtain a temporary object composed of multiple byte arrays. During computation, the parser obtains relevant field information and computational semantics based on the computational expressions in the ETL process of parsing the data. Based on the obtained field information, the corresponding fields in the temporary object are deserialized, the data is calculated, and the calculation result is serialized and written back to the corresponding location. Fields not involved in the calculation, if still needed to be passed forward, can be directly concatenated from the byte stream.

[0253] This application provides a data deserialization method. As can be seen from the detailed description of the foregoing embodiments, when deserializing a data stream, this application performs deserialization according to custom semantics, and in scenarios where computation is required, only the data that needs to be computed is deserialized, forming a delayed deserialization mechanism. This avoids unnecessary serialization and deserialization operations and improves data processing efficiency.

[0254] In summary, this application provides a method for serialization and deserialization of sparse data during data ETL. During serialization, data is effectively compressed using custom semantics to improve the transmission efficiency of the serialized data. Furthermore, for scenarios involving multiple serializations and deserializations during data ETL computation, a delayed deserialization mechanism is provided, deserializing only necessary fields, reducing unnecessary serialization and deserialization operations, and significantly improving the overall processing efficiency of the data ETL process.

[0255] In this embodiment, the data serialization method is applied to the data serialization device, and the data deserialization method is applied to the data deserialization device. The data serialization device and the data deserialization device can be integrated into the same device or integrated into different devices; this embodiment does not specifically limit this.

[0256] Additionally, regarding the serialization and deserialization methods for sparse data during data ETL, the serialization process begins by initializing the counter i (Byte) to 0 and initializing the read / write buffer. Then, the data to be serialized is obtained, and data fields are matched. For each field: if the matched field is null, it is first checked whether serialized data exists in the read / write buffer. If it does, the counter value is written to the target, and then the data in the read / write buffer is written to the target. The counter i is reset to zero, the read / write buffer is cleared, and the operation on the buffered data is completed. Then, null values ​​are counted, i = i - 1 (because the newly matched field is null), completing this match. If subsequent matched fields are still null, since the data in the read / write buffer has already been processed, the buffer is empty, and the operation on the buffered data is skipped. The counter i continues to decrement by one, i = i - 1, until i reaches the lower limit BYTE_MIN. Then, the counter value is written to the target, and the count is completed. The counter i is reset to zero. If the matched field is not empty, first check if there are any empty values ​​that have not been recorded (when the counter is negative, its absolute value represents the number of empty values ​​to be recorded). If so, write the record to the target end, reset the counter i to zero, and complete the recording of empty values. Then, count the non-empty records, i = i + 1, and serialize the content of the non-empty fields to the read-write buffer. If the subsequent matched fields are not empty, continue to count the non-empty records, i = i + 1, and serialize the non-empty data to the read-write buffer until i reaches the upper limit BYTE_MAX. Then, write the counter value to the target end, write the data in the read-write buffer to the target end, reset the counter i to zero, and clear the read-write buffer. If the matching of the current line of data is finished, write 0 to the target end as the end bit, and complete the serialization of the line of data.

[0257] During the deserialization process, the data stream to be deserialized is first obtained, and a byte is read from the data stream. If the read data is negative, the corresponding absolute number of null values ​​is deserialized. If the read data is positive n, the deserializer parses the following n fields according to the custom schema to obtain the corresponding deserialized values. If the read data is 0, it means that the data in that row has been deserialized.

[0258] Specifically, for computational scenarios in the data ETL process, since the computation process may not involve all serialized fields, and after data computation, the computation result data still needs to be serialized for subsequent data storage. For fields that did not participate in the computation process, the traditional processing mode would deserialize them all before performing the serialization operation. This is an unnecessary redundant operation that slows down the execution efficiency of the entire computation process. Therefore, this application proposes a delayed deserialization mechanism. In the computation scenario, the data is not directly deserialized to obtain a structure object according to the custom schema. Instead, the data stream bytes are split according to the schema to obtain a temporary object composed of multiple byte arrays. During computation, the relevant field information and computational semantics are obtained by parsing the computation expression in the data ETL process according to the parser. Based on the obtained field information, the corresponding fields in the temporary object are deserialized to obtain data for computation, and the computation result is serialized and written back to the corresponding position. If fields that did not participate in the computation still need to be passed forward, the byte stream can be directly concatenated.

[0259] Thus, the serialization and deserialization methods for sparse data in the data ETL process proposed in this application have been specifically optimized for the characteristics of sparse data and the specific scenarios of ETL processing, greatly improving the overall processing efficiency of the ETL process. For example, during data serialization, custom serialization rules are defined to address the characteristics of sparse data, achieving effective compression of the data stream for higher transmission efficiency; during data deserialization, a delayed deserialization mechanism is provided to address the characteristics of multiple serializations and deserializations in the data ETL scenario, further improving the processing efficiency of the data ETL process.

[0260] In short, compared with the prior art, the technical solution of this application has the following advantages: (1) Compared with the serialization and deserialization rules for general data in the prior art, this application provides a custom serialization and deserialization rule for data ETL scenarios that can effectively compress sparse data, which can effectively compress the serialized data and greatly improve the transmission efficiency in the data transmission process. (2) For scenarios of multiple serialization and deserialization in the data ETL process, this application proposes a delayed serialization mechanism, which avoids unnecessary serialization and deserialization operations in the ETL process and greatly improves the processing efficiency of the entire data ETL process.

[0261] In another embodiment of this application, see [reference needed]. Figure 6 This illustrates a schematic diagram of the composition of a data serialization device 60 provided in an embodiment of this application. For example... Figure 6As shown, the data serialization device 60 may include: a first acquisition unit 601, a matching unit 602, and a first determination unit 603; wherein,

[0262] The first acquisition unit 601 is configured to acquire data to be serialized;

[0263] The matching unit 602 is configured to perform field matching on the data to be serialized to obtain a matching result;

[0264] The first determining unit 603 is configured to determine a first count value corresponding to the empty field when the matching result indicates that the field is an empty field; and to write the first count value into the serialization result when the first count value reaches a preset lower limit value.

[0265] In some embodiments, the first determining unit 603 is further configured to: determine a second count value corresponding to the non-empty field when the matching result indicates that the field is a non-empty field; and serialize the non-empty field and store it in a preset cache area; and when the second count value reaches a preset upper limit value, write the second count value and the cached data in the preset cache area into the serialization result.

[0266] In some embodiments, such as Figure 6 As shown, the data serialization device 60 may further include a judgment unit 604, configured to judge whether the preset buffer is empty;

[0267] The first determining unit 603 is specifically configured to: if the preset buffer is not empty, write the current count value of the counter and the cached data in the preset buffer into the serialization result; after clearing the counter and emptying the preset buffer, obtain the first count value by decrementing the counter by 1; and if the preset buffer is empty, obtain the first count value directly by decrementing the counter by 1.

[0268] In some embodiments, the determining unit 604 is further configured to determine whether there is an unstored first count value;

[0269] The first determining unit 603 is further configured to, if there is an unstored first count value, write the first count value into the serialization result, and after clearing the counter, obtain the second count value by incrementing the counter by 1; and if there is no unstored first count value, obtain the second count value directly by incrementing the counter by 1.

[0270] In some embodiments, the determining unit 604 is further configured to determine whether all rows of the data to be serialized have been serialized.

[0271] The first determining unit 603 is further configured to, if the determination result is negative, return to the step of obtaining the data to be serialized; if the determination result is positive, write the count value into the serialization result and determine whether the preset buffer is empty; and if the preset buffer is not empty, write an end character at the end of the serialization result after writing the cached data in the preset buffer into the serialization result; and if the preset buffer is empty, directly write an end character at the end of the serialization result.

[0272] This application provides a data serialization apparatus, which includes a first acquisition unit, a matching unit, and a first determination unit. Thus, when serializing data to be serialized using this apparatus, especially empty fields in sparse data, by using custom serialization rules, only the count values ​​of empty fields are written into the serialization result, thereby achieving effective compression of the data stream and improving transmission efficiency during the data ETL process.

[0273] In another embodiment of this application, see [reference needed]. Figure 7 This illustrates a schematic diagram of the composition of a data deserialization apparatus 70 provided in an embodiment of this application. Figure 7 As shown, the data deserialization device 70 may include: a second acquisition unit 701, a reading unit 702, and a deserialization unit 703; wherein,

[0274] The second acquisition unit 701 is configured to acquire the serialization result;

[0275] The reading unit 702 is configured to read the current byte value from the serialization result;

[0276] The deserialization unit 703 is configured to, if the current byte value is greater than zero, perform deserialization processing on the field following the current byte value with the number of current byte values ​​to obtain a deserialization result; and if the current byte value is less than zero, determine the absolute value of the current byte value and deserialize the empty field with the number of absolute values ​​in the deserialization result.

[0277] In some embodiments, such as Figure 7 As shown, the data deserialization device 70 may further include a second determining unit 704 and a calculation unit 705; wherein,

[0278] The second determining unit 704 is configured to segment the serialization result to determine a temporary object; and to parse a preset expression to determine target field information and target semantic information.

[0279] The deserialization unit 703 is further configured to selectively deserialize the temporary object according to the target field information to obtain target data;

[0280] The calculation unit 705 is configured to perform calculations on the target data based on the target semantic information to obtain a calculation result.

[0281] In some embodiments, the second determining unit 704 is further configured to serialize the calculation result and store it in the temporary object according to the target semantic information; and to write the temporary object into the serialization result.

[0282] This application provides a data deserialization apparatus, which includes a second acquisition unit, a reading unit, and a deserialization unit. Thus, when deserializing a serialization result using this apparatus, for positive byte values, only the field representing the number of subsequent byte values ​​needs to be deserialized sequentially; for negative byte values, an empty field representing the absolute number of byte values ​​is directly deserialized. Furthermore, by selectively deserializing the data stream to be deserialized, unnecessary serialization and deserialization operations are reduced, greatly improving the processing efficiency of the entire data ETL process.

[0283] Understandably, in the foregoing embodiments, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in the embodiments of this application can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated unit described above can be implemented in hardware or as a software functional module.

[0284] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this application embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0285] Therefore, embodiments of this application provide a computer storage medium storing a data serialization program or a data deserialization program; when the data serialization program is executed by a processor, it implements any of the data serialization methods described in the foregoing embodiments, or when the data deserialization program is executed by a processor, it implements any of the data deserialization methods described in the foregoing embodiments.

[0286] See Figure 8 This illustrates a schematic diagram of the specific hardware structure of an electronic device 80 provided in an embodiment of this application. For example... Figure 8 As shown, it may include: a communication interface 801, a memory 802, and a processor 803; the various components are coupled together through a bus system 804. It is understood that the bus system 804 is used to implement communication between these components. In addition to a data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The various buses are all labeled as bus system 804. Among them, the communication interface 801 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0287] Memory 802 is used to store computer programs that can run on processor 803;

[0288] Processor 803 is used to perform the following when running a data serialization program:

[0289] Obtain the data to be serialized;

[0290] The fields of the data to be serialized are matched to obtain the matching results;

[0291] If the matching result indicates that the field is an empty field, determine the first count value corresponding to the empty field;

[0292] When the first count value reaches the preset lower limit, the first count value is written into the serialization result.

[0293] Alternatively, in another implementation, the processor 803 is further configured to perform the following when running the data deserialization program:

[0294] Obtain the serialization result;

[0295] Read the current byte value from the serialization result;

[0296] If the current byte value is greater than zero, then the field following the current byte value with the number of current byte values ​​is deserialized to obtain the deserialization result;

[0297] If the current byte value is less than zero, then the absolute value of the current byte value is determined, and an empty field containing the absolute value is deserialized from the deserialization result.

[0298] It is understood that the memory 802 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0299] The processor 803 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 803 or by instructions in software form. The processor 803 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 802, and the processor 803 reads the information in memory 802 and, in conjunction with its hardware, completes the steps of the above method.

[0300] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0301] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0302] Alternatively, as another embodiment, the processor 803 is further configured to perform the steps of the method described in any of the foregoing embodiments when running the computer program.

[0303] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

[0304] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0305] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0306] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0307] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0308] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0309] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data serialization method, characterized in that, The method includes: Obtain the data to be serialized; The fields of the data to be serialized are matched to obtain the matching results; If the matching result indicates that the field is an empty field, determine the first count value corresponding to the empty field; When the first count value reaches the preset lower limit, the first count value is written into the serialization result; If the matching result indicates that the field is a non-empty field, a second count value corresponding to the non-empty field is determined, and the non-empty field is serialized and stored in a preset cache; the second count value is the count value of the non-empty field by a counter. When the second count value reaches the preset upper limit, the second count value and the cached data in the preset buffer are written into the serialization result; Wherein, when the matching result indicates that the field is an empty field, determining the first count value corresponding to the empty field includes: Determine whether the preset cache area is empty; If the preset buffer is not empty, the current count value of the counter and the cached data in the preset buffer are written into the serialization result. After clearing the counter and emptying the preset buffer, the first count value is obtained by decrementing the counter by 1. If the preset buffer is empty, the first count value is obtained by directly decrementing the counter by 1.

2. The method according to claim 1, characterized in that, When the matching result indicates that the field is a non-empty field, determining the second count value corresponding to the non-empty field includes: Determine if there is a first count value that is not stored; If there is a first count value that is not stored, the first count value is written into the serialization result, and after the counter is cleared, the second count value is obtained by incrementing the counter by 1. If there is no unstored first count value, the second count value is obtained by directly incrementing the counter by 1.

3. The method according to claim 1, characterized in that, The method further includes: Determine whether all rows of the data to be serialized have been serialized. If the judgment result is negative, then return to the step of obtaining the data to be serialized; If the determination result is yes, then the count value is written into the serialization result, and it is determined whether the preset buffer is empty; If the preset buffer is not empty, then after writing the cached data in the preset buffer into the serialization result, an end character is written at the end of the serialization result; If the preset buffer is empty, then an end character is written directly to the end of the serialization result.

4. A data deserialization method for deserializing a serialization result generated using the data serialization method according to any one of claims 1 to 3, characterized in that, The method includes: Obtain the serialization result; Read the current byte value from the serialization result; If the current byte value is greater than zero, then the non-empty fields following the current byte value for the same number of times the current byte value is deserialized to obtain the deserialization result; If the current byte value is less than zero, then the absolute value of the current byte value is determined, and an empty field containing the absolute value is deserialized from the deserialization result.

5. The method according to claim 4, characterized in that, After obtaining the serialization result, the method further includes: The serialization result is segmented to determine temporary objects; Parse the preset expression to obtain target field information and target semantic information; The temporary object is selectively deserialized based on the target field information to obtain the target data; The target data is calculated based on the target semantic information to obtain the calculation result.

6. The method according to claim 5, characterized in that, After obtaining the calculation result, the method further includes: After the calculation result is serialized, it is stored in the temporary object according to the target semantic information; Write the temporary object into the serialization result.

7. A data serialization device, characterized in that, The data serialization device includes: a first acquisition unit, a matching unit, and a first determination unit; wherein, The first acquisition unit is configured to acquire data to be serialized; The matching unit is configured to perform field matching on the data to be serialized to obtain a matching result; The first determining unit is configured to: determine a first count value corresponding to the empty field when the matching result indicates that the field is an empty field; and write the first count value into the serialization result when the first count value reaches a preset lower limit; determine a second count value corresponding to the non-empty field when the matching result indicates that the field is a non-empty field, and serialize the non-empty field and store it in a preset buffer; the second count value is a count value for counting the non-empty field by a counter; and write the second count value and the cached data in the preset buffer into the serialization result when the second count value reaches a preset upper limit. Wherein, when the matching result indicates that the field is an empty field, determining the first count value corresponding to the empty field includes: Determine whether the preset cache area is empty; If the preset buffer is not empty, the current count value of the counter and the cached data in the preset buffer are written into the serialization result. After clearing the counter and emptying the preset buffer, the first count value is obtained by decrementing the counter by 1. If the preset buffer is empty, the first count value is obtained by directly decrementing the counter by 1.

8. A data deserialization apparatus for deserializing a serialization result generated using the data serialization apparatus as described in claim 7, characterized in that, The data deserialization device includes: a second acquisition unit, a reading unit, and a deserialization unit; wherein, The second acquisition unit is configured to acquire the serialization result; The reading unit is configured to read the current byte value from the serialization result; The deserialization unit is configured to, if the current byte value is greater than zero, deserialize the number of non-empty fields following the current byte value to obtain a deserialization result; and if the current byte value is less than zero, determine the absolute value of the current byte value and deserialize the number of empty fields in the deserialization result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein, The memory is used to store computer programs that can run on the processor; The processor is configured to, when running the computer program, execute the data serialization method as described in any one of claims 1 to 3, or execute the data deserialization method as described in any one of claims 4 to 6.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the data serialization method as described in any one of claims 1 to 3, or the data deserialization method as described in any one of claims 4 to 6.

Images