Method, system and computer-readable storage medium for extracting features after data serialization

Abstract

The present invention discloses a method, system and storage medium for extracting features after data serialization. The method for extracting features after data serialization comprises the following steps: obtaining original data, and serializing the original data into character data; statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight, wherein the weight is the frequency of occurrence of the character; obtaining the number of layers of each character in the optimal binary tree data, and converting the binary tree data into array data according to the number of layers. The technical solution of the present invention can solve the technical problems in the related art that the data body is too large, resulting in excessive memory usage of computer devices, reduced computing speed and slow network transmission.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to a method and system for extracting features after data serialization, and a computer-readable storage medium. Background Art

[0002] In the related art, there are the following technologies:

[0003] 1. Data serialization technology

[0004] In the process of computer equipment storing and computing data, data serialization technology is an indispensable technology. Data serialization is to convert objects or data structures into a specific format so that they can be transmitted over the network or stored in memory or files. Deserialization is the opposite operation, which restores objects from serialized data.

[0005] The data format after data serialization can be binary, XML, JSON or any other format. The focus of data serialization is on data exchange and transmission.

[0006] When data persistence or network is required, the data needs to be serialized. For example:

[0007] (1) Data persistence: For example, when tens of thousands of users access an e-commerce platform concurrently, tens of thousands of session objects will be generated, which will put a lot of pressure on the memory. The session objects can be serialized (stored) to the hard disk and deserialized when needed to reduce memory pressure. The result of serialization is not binary data.

[0008] (2) Network transmission: After the system is split into multiple services, objects transmitted between services, regardless of the type of data, must be converted into binary streams for transmission and then converted into data objects after the recipient receives them.

[0009] Serialization concerns / metrics

[0010] (1) Data size after serialization

[0011] Because serialized data is usually transmitted over the network or stored in memory. Therefore, the less content after serialization, the shorter the transmission time and the less memory it occupies. Java's native serialization mechanism cannot reuse object structures, resulting in large serialized data.

[0012] (2) The time and CPU usage of serialization and deserialization.

[0013] (3) Whether it supports cross-language and cross-platform.

[0014] In the related technology, after data serialization, the data body is too large, resulting in a large memory occupancy of the computer device, reduced computing speed and slow network transmission.

[0015] Therefore, it is necessary to provide a new feature extraction method, system and computer-readable storage medium after data serialization to solve the above technical problems. Summary of the invention

[0016] The main purpose of the present invention is to provide a data serialization feature method, which aims to solve the technical problems in the related technology that the data body is too large, resulting in excessive memory usage of computer devices, reduced computing speed and slow network transmission.

[0017] To achieve the above object, the present invention provides a method for extracting features after data serialization, comprising the following steps:

[0018] Obtaining original data, and serializing the original data into character data;

[0019] Statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight, wherein the weight is the occurrence frequency of the character;

[0020] The layer number information of each character in the optimal binary tree data is obtained, and the binary tree data is converted into array-type data according to the layer number information.

[0021] Preferably, after the step of obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array data according to the layer number information, the following steps are also included:

[0022] The data group class data is serialized into binary data for external transmission.

[0023] Preferably, after the step of obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array data according to the layer number information, the following steps are also included:

[0024] The character is retrieved from the array data according to the memory address of the character.

[0025] Preferably, the step of statistically analyzing the weight of each character in the character data and converting the character data into optimal binary tree data according to the weight specifically comprises the following steps:

[0026] Statistically analyzing the weight of each character, and sorting the characters according to the weight;

[0027] Merge the two characters that are ranked later into a new character, and add the weights of the two characters as the weight of the new character;

[0028] Statistically analyzing the weights of each character again, and re-ranking the characters according to the weights;

[0029] Determine whether the number of characters is unique;

[0030] When the number of the characters is unique, performing a data layering operation on the unique characters, and setting the layer number information of each character to form the optimal binary tree data;

[0031] When the number of characters is not unique, the step of merging the two characters with the later weights into a new character and adding the weights of the two original characters is performed again as the weight of the new character.

[0032] Preferably, the step of performing data layering operation on the unique character and setting the layer number information of each character specifically includes the following steps:

[0033] In all the steps of merging the two characters that are ranked later into a new character and adding the weights of the two characters as the weight of the new character, obtaining a merging order and a merging form between all the characters;

[0034] According to the splitting order and splitting form, the splitting is performed in sequence starting from the unique character, and the layer number information of the split character is set to N, where N is the total number of splits of the current character. The splitting order is opposite to the merging order, and the splitting form corresponds to the merging form.

[0035] Preferably, the step of obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array data according to the layer number information specifically includes the following steps:

[0036] In the step of statistically analyzing the weight of each character and sorting the characters according to the weight; obtaining all the characters and the layer number information corresponding to the characters;

[0037] The characters are sorted according to the order of the layer number information, and a corresponding memory address is allocated to each character to form the array type data.

[0038] Preferably, the array-type data is an ordered linked list data structure.

[0039] In order to solve the above technical problems, the present invention also provides a feature extraction system after data serialization, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the feature extraction method after data serialization are implemented.

[0040] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the step of feature extraction after data serialization is implemented.

[0041] The present invention provides a method for extracting features after data serialization; obtaining original data and serializing the original data into character data; statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight, wherein the weight is the frequency of occurrence of the character; obtaining the number of layers of each character in the optimal binary tree data, and converting the binary tree data into array data according to the number of layers information.

[0042] Among them, statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight can be understood as simplified compression of data;

[0043] Obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array data according to the layer number information can be understood as extracting key features of the data to facilitate computer query and calculation.

[0044] The present invention achieves the purpose of key feature extraction and data compression in the data serialization process by serializing the data, constructing the original data into a Huffman binary tree, and sequentially connecting the leaf nodes of the binary tree to form a character code mapping table (array type data), thereby solving the technical problems of slow transmission speed, large capacity occupation, slow calculation and query in the network caused by the excessive size of the data body in the computer device. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of the first embodiment of the method for extracting features after data serialization provided by the present invention;

[0046] Figure 2 for Figure 1 The work flow diagram of step S20 shown;

[0047] Figure 3 for Figure 2 The work flow chart of step S205 shown;

[0048] Figure 4 for Figure 1 The work flow chart of step S30 shown;

[0049] Figure 5 A flowchart of a second embodiment of the end-to-end network data transmission method provided by the present invention;

[0050] Figure 6 A flowchart of a third embodiment of the method for extracting features after data serialization provided by the present invention;

[0051] Figure 7 The architecture diagram of the feature extraction system after data serialization provided by the present invention;

[0052] Figure 8 for Figure 2 A character sorting change diagram in step S201 is shown;

[0053] Fig. 9 for Figure 3 An evolution diagram of a hierarchical operation of characters in step S205 is shown;

[0054] Fig.10 A structural diagram for array-type data;

[0055] Fig.11 for Figure 4 A correspondence diagram of characters, character layers and memory addresses in step S30 is shown.

[0056] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0058] See also Figure 1 To achieve the above purpose, in a first embodiment of the present invention, a method for extracting features after data serialization includes the following steps:

[0059] S10, obtaining original data, and serializing the original data into character data;

[0060] In this embodiment, the original data may be picture data, audio data, video data, etc., and the original data may also be binary byte stream data. The user may serialize the original data in the computer device into character data according to actual needs. The character data may be data composed of multiple characters, for example, data composed of letters such as abcdef.

[0061] S20, statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight, wherein the weight is the occurrence frequency of the character;

[0062] In this embodiment, the binary tree data refers to data whose distribution structure is in the shape of a tree branch. The sorting distribution of each character data is similar to the shape of a tree branch. It can be defined that the node at the top of the tree branch is the root node, and the node at the forked end is the leaf node.

[0063] The optimal binary tree data refers to the data distribution in which the sum of the products of multiplying the path lengths between each leaf node and the root node by the weights of the characters corresponding to the leaf nodes is the smallest.

[0064] S30, obtaining the layer number information of each character in the optimal binary tree data, and converting the binary tree data into array type data according to the layer number information.

[0065] The data structure of array data is a classic linked list structure. Its structure is like a train. Each carriage of the train stores data content, and the bottom of the carriage is marked with a memory address (pointer address). In this way, all the data can be connected in series by connecting the data section by section and the pointer address.

[0066] Compared with the optimal binary tree data, the corresponding data retrieval and acquisition of array data is faster and more convenient. The corresponding data can be quickly obtained only through the internal address.

[0067] The present invention provides a method for extracting features after data serialization; obtaining original data and serializing the original data into character data; statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight, wherein the weight is the frequency of occurrence of the character; obtaining the number of layers of each character in the optimal binary tree data, and converting the binary tree data into array data according to the number of layers information.

[0068] Among them, statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight can be understood as simplified compression of data;

[0069] Obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array-type data according to the layer number information can be understood as extracting key features of the data to facilitate computer query and calculation.

[0070] The present invention achieves the purpose of key feature extraction and data compression in the data serialization process by serializing the data, constructing the original data into a Huffman binary tree, and sequentially connecting the leaf nodes of the binary tree to form a character code mapping table (array type data), thereby solving the technical problems of slow transmission speed, large capacity occupation, slow calculation and query in the network caused by the excessive size of the data body in the computer device.

[0071] See also Figure 2 , the step S20 specifically includes the following steps:

[0072] S201, statistically analyzing the weight of each character, and sorting the characters according to the weight;

[0073] In this embodiment, the characters are sorted according to the weights from high to low.

[0074] For example, see Figure 8 , sorted in descending order of weight, in the order of a, d, b, c, e and f.

[0075] S202, merging the two characters ranked later into a new character, and adding the weights of the two characters as the weight of the new character;

[0076] For example, see Figure 8-9 In adbcef, the weights of characters f and e are the lowest, so characters f and e are merged into a new character fe;

[0077] S203, statistically analyzing the weights of each character again, and re-ranking the characters according to the weights;

[0078] For example, see Figure 8-9 , sorted in descending order of weight, in the order of a, d, ef, b and c.

[0079] S204, determining whether the number of characters is unique;

[0080] S205, when the number of the characters is unique, performing a data layering operation on the unique characters, and setting the layer number information of each character to form the optimal binary tree data;

[0081] For example, Fig. 9 As shown in the lower right corner, the optimal binary tree data is formed with the root node being acbfed, and the two leaf nodes being a and cbfed, and so on.

[0082] When the number of characters is not unique, the process goes to step S202 again.

[0083] In this embodiment, it may be defined that the optimal binary tree data means that, among all possible binary tree data structures, the binary tree data is a binary tree with the shortest weighted path length.

[0084] We may define the node of character abcfde as the root node and other characters as leaf nodes; the layer number information of the root node is set to 0, and the difference between the layer number information of the leaf node and the layer number information of the root node is the path length.

[0085] The weighted path length is denoted as WPL; WPL = (W1*L1+W2*L2+W3*L3+...+Wn*Ln), where Wi (i=1,2,...n) is the weight and Li (i=1,2,...n) is the path length.

[0086] N weights Wi (i = 1, 2, ... n) form a binary tree with N leaf nodes, and the path length of the corresponding leaf node is Li (i = 1, 2, ... n). By comparing the calculations of this formula, it can be proved that the WPL of the optimal binary tree data is the smallest.

[0087] The step S205 specifically includes the following steps:

[0088] S2051, in all the steps S202, obtaining the merging order and merging form of all the characters;

[0089] S2052, according to the splitting order and splitting form, split in sequence starting from the unique character, and set the layer number information of the split character to N, where N is the total number of splits of the current character, the splitting order is opposite to the merging order, and the splitting form corresponds to the merging form.

[0090] For example, by Fig. 9 It can be learned that the layer information of characters acbfed is 0; the layer information of characters a and cbfed is 1; the layer information of characters cb and fed is 3; the layer information of characters c, b, d, and fe is 3; and the layer information of characters f and e is 4.

[0091] The step S30 specifically includes the following steps:

[0092] S301, obtaining all the characters and the layer number information corresponding to the characters in step S201;

[0093] S302, sorting the characters according to the order of the layer number information, and assigning a corresponding memory address to each character to form the array type data.

[0094] For example, see Fig.11 , the layer number information of character a is 1, and the corresponding memory address is 0x15(c); the layer number information of character c, character b, and character d is 3, and the corresponding memory addresses are 0x8h(b), 0x9e(d)0x34(f) respectively; the layer number information of character f and character e is 4; the corresponding memory addresses are 0x66(e)0x67(e) respectively.

[0095] The array data is an ordered linked list data structure. Fig.10 shown.

[0096] On the basis of the Huffman binary tree, all leaf nodes are connected in hierarchical order to form a separate sequential linked list, and the layer number information of the leaf node is marked:

[0097] 0x at the beginning indicates a memory address. The benefits of linked list data structure (mapping table) are:

[0098] The speed of node traversal is accelerated. For example, to find the d node in the binary tree, it needs to be compared three times in the original Huffman binary tree, but only once in the sequential word code mapping table.

[0099] Binary trees are used to store structural information of data. The construction of binary trees is completed through continuous data comparison and addition. However, as the height of the tree increases, the node search becomes slower and slower. When computing resources are limited, the time complexity will increase.

[0100] The array is accessed directly based on the array index (memory address). That is, no matter whether the data is at the first or last position in the array, the search time is the same. For the CPU, only one calculation is required to find the data, and the time complexity is O(1).

[0101] The way to search for binary tree data is to traverse the tree. There are currently three traversal methods: pre-order, in-order, and post-order. No matter which method is used, N comparisons are required when searching for a node. The value of N depends on the number of comparisons (smaller on the left and larger on the right), so the time complexity is O(N).

[0102] The algorithm provided by the present invention cleverly combines the advantages of two data structures, using a binary tree class to store and represent the frequency of occurrence of character strings, and using data to quickly search for leaf nodes.

[0103] Second embodiment

[0104] Based on the data serialization feature extraction method 100 provided in the first embodiment of the present invention, the second embodiment of the present invention provides another data serialization feature extraction method 200, wherein the steps S10 to S30 of the method are the same as those of the first embodiment and are not described one by one here. The difference is that the method further includes the following steps:

[0105] S40, serializing the data group class data into binary data for external transmission.

[0106] The purpose of serializing into binary data is to ensure the integrity and transferability of objects when they are transferred and saved.

[0107] Serialization can convert objects into an ordered byte stream for transmission over the network or storage in a local file.

[0108] The serialized byte stream saves the state of the Java object and related description information. The core function of the serialization mechanism is to save and rebuild the state of the object.

[0109] When two processes communicate remotely, they can send various types of data to each other, including text, pictures, audio, video, etc., and all of this data will be transmitted over the network in the form of binary sequences.

[0110] The serialization algorithm in the present invention is performed in the following steps:

[0111] Outputs class metadata associated with an object instance.

[0112] Recursively output descriptions of a class's superclasses until there are no more superclasses.

[0113] After the class metadata is completed, the actual data value of the object instance is output starting from the top-level superclass, and the instance data is output recursively from top to bottom.

[0114] Third embodiment

[0115] Based on the data serialization feature extraction method 100 provided in the first embodiment of the present invention, the second embodiment of the present invention provides another data serialization feature extraction method 300, wherein the steps S10 to S30 of the method are the same as those of the first embodiment and are not described one by one here. The difference is that the method further includes the following steps:

[0116] S41, searching for the character in the array type data according to the memory address of the character.

[0117] The present invention also provides a feature extraction system after data serialization.

[0118] A system for extracting features after data serialization includes: a memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the method for extracting features after data serialization are implemented.

[0119] The specific steps of the method for extracting features after data serialization refer to the above-mentioned embodiments. Since the system for extracting features after data serialization adopts all the technical solutions of all the above-mentioned embodiments, it at least has all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be described one by one here.

[0120] The present invention also provides a computer-readable storage medium.

[0121] A computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the step of extracting features after data serialization is implemented.

[0122] The specific steps of the feature extraction method after data serialization refer to the above-mentioned embodiments. Since the computer-readable storage medium adopts all the technical solutions of all the above-mentioned embodiments, it has at least all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be repeated here one by one.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to enter the method described in each embodiment of the present invention.

[0124] In the description of this specification, the description with reference to the terms "an embodiment", "another embodiment", "other embodiments", or "first embodiment to Xth embodiment" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, method steps or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0125] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0126] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0127] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for extracting features after data serialization, characterized in that: The steps include: Obtaining original data, and serializing the original data into character data; Statistically analyzing the weight of each character in the character data, and converting the character data into optimal binary tree data according to the weight, wherein the weight is the occurrence frequency of the character; Obtaining the layer number information of each character in the optimal binary tree data, and converting the binary tree data into array-type data according to the layer number information; After the step of obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array data according to the layer number information, the following steps are also included: Serializing the data group class data into binary data for external transmission; After the step of obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array data according to the layer number information, the following steps are also included: Retrieve the character from the array data according to the memory address of the character; The step of statistically analyzing the weight of each character in the character data and converting the character data into optimal binary tree data according to the weight specifically includes the following steps: Statistically analyzing the weight of each character, and sorting the characters according to the weight; Merge the two characters that are ranked later into a new character, and add the weights of the two characters as the weight of the new character; Statistically analyzing the weights of each character again, and re-ranking the characters according to the weights; Determine whether the number of characters is unique; When the number of the characters is unique, performing a data layering operation on the unique characters, and setting the layer number information of each character to form the optimal binary tree data; When the number of characters is not unique, the step of combining the two characters with the later weights into a new character is performed again, and the weights of the two original characters are added together as the weight of the new character; The optimal binary tree data means that among all possible binary tree data structures, the binary tree data is a binary tree with the shortest weighted path length; the weighted path length is recorded as WPL; WPL=(W1*L1+W2*L2+W3*L3+...+Wn*Ln), where Wi(i=1,2,...n)is the weight and Li(i=1,2,...n)is the path length; N weights Wi (i=1,2,...n) form a binary tree with N leaf nodes, and the path length of the corresponding leaf node is Li (i=1,2,...n); The step of performing data layering operation on the unique character and setting the layer number information of each character specifically includes the following steps: In all the steps of merging the two characters that are ranked later into a new character and adding the weights of the two characters as the weight of the new character, obtaining a merging order and a merging form between all the characters; According to the splitting order and the splitting form, the unique character is used as the starting point for sequential splitting, and the layer number information of the split character is set to M, where M is the total number of splits of the current character, the splitting order is opposite to the merging order, and the splitting form corresponds to the merging form; The step of obtaining the layer number information of each character in the optimal binary tree data and converting the binary tree data into array data according to the layer number information specifically includes the following steps: In the step of statistically analyzing the weight of each character and sorting the characters according to the weight; obtaining all the characters and the layer number information corresponding to the characters; Sort the characters according to the order of the layer number information, and assign a corresponding memory address to each character to form the array data; The array data is an ordered linked list data structure. On the basis of the Huffman binary tree, all leaf nodes are connected in hierarchical order to form a separate sequential linked list, and the layer number information of the leaf node is marked. The beginning of 0x indicates the memory address.

2. A data serialization feature extraction system, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the method for extracting features after data serialization as claimed in claim 1 is implemented.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for extracting features after data serialization according to claim 1 is implemented.

Images