Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and the data processing method comprises the steps: receiving a compression processing request of to-be-processed data, the to-be-processed data comprises a plurality of data blocks, and the plurality of data blocks comprise at least two data blocks of different data types; and in response to the compression processing request, for at least a first data block in the plurality of data blocks, performing coding rule query in the coding storage space based on the data type of the first data block, and performing response according to a coding rule query result to perform coding processing on the first data block. According to the data processing method, more effective compression can be provided for the to-be-processed data containing different data types in a self-adaptive mode, so that the compression processing response speed is increased, and better compression efficiency is achieved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method and apparatus, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of technology, data volumes are constantly increasing in fields such as e-commerce, the Internet of Things, artificial intelligence, and digital manufacturing. Efficiently managing and transmitting data has become a crucial task. Huffman coding, as a classic lossless data compression method, has broad applicability. However, Huffman coding often requires processing complex tree structures and frequency statistics, and the encoding consumes a large amount of storage space, resulting in low compression efficiency and, in some cases, low compression ratios. Summary of the Invention

[0003] At least one embodiment of the present disclosure provides a data processing method, comprising: receiving a compression processing request for data to be processed, the data to be processed including multiple data blocks, the multiple data blocks including at least two data blocks of different data types; in response to the compression processing request, for at least a first data block among the multiple data blocks, based on the data type of the first data block, performing a coding rule query in a coding storage space, and responding according to the coding rule query result to perform coding processing on the first data block; wherein responding according to the coding rule query result comprises: in response to the coding rule query result being that no result is found, generating a first coding rule corresponding to the first data block according to the data type of the first data block and at least part of the data elements in the first data block, and storing the first coding rule in the coding storage space; or in response to the coding query result being that the first coding rule corresponding to the first data block stored in the coding storage space is found, calling the first coding rule, the first coding rule being used to perform coding processing on the first data block.

[0004] For example, in the data processing method provided in at least some embodiments of the present disclosure, the first encoding rule in the encoding storage space includes: in response to the encoding storage space being full, replacing the replaced encoding rule determined by the calling situation in the encoding storage space to obtain storage space for storing the first encoding rule.

[0005] For example, in the data processing method provided in at least some embodiments of the present disclosure, the method for determining the replaced coding rule determined by the calling situation includes: in response to the statistical results of the calling frequency of each coding rule in the coding storage space, determining the coding rule with the lowest calling frequency in the coding storage space as the replaced coding rule.

[0006] For example, in the data processing method provided in at least some embodiments of the present disclosure, a first encoding rule corresponding to the first data block is generated based on the data type of the first data block and at least part of the data elements in the first data block, including: obtaining at least part of the data elements in the first data block that meet the corresponding encoding data amount requirement, the encoding data amount requirement being the minimum data amount required to obtain the symbol frequency distribution law of the first data block; and processing at least part of the data elements according to the compression encoding strategy corresponding to the data type of the first data block to generate the first encoding rule.

[0007] For example, in the data processing method provided in at least some embodiments of the present disclosure, the encoding data amount requirement corresponding to the first data block is determined based on the mapping relationship between the data type and the encoding data amount requirement and the data type of the first data block to determine at least some data elements in the first data block that meet the corresponding encoding data amount requirement.

[0008] For example, the data processing method provided in at least some embodiments of the present disclosure further includes: determining a compression coding strategy corresponding to the data type of the first data block based on data features corresponding to the data type of the first data block.

[0009] For example, in the data processing method provided in at least some embodiments of the present disclosure, data features include data patterns and / or data structures; according to the compression coding strategy corresponding to the data type of the first data block, at least part of the data elements are processed to generate a first coding rule corresponding to the first data block, including: determining the symbol frequency distribution of at least part of the data elements according to the data pattern, the data pattern including the symbol type corresponding to the data type and / or the probability of occurrence of a certain symbol type; according to the data type and symbol frequency distribution of the first data block, at least part of the data elements are processed to generate a first coding rule corresponding to the first data block.

[0010] For example, in the data processing method provided in at least some embodiments of the present disclosure, at least part of the data elements are processed according to the data type and symbol frequency distribution of the first data block to generate a first encoding rule corresponding to the first data block, including: determining a corresponding compression encoding method according to the data type of the first data block, and / or determining a corresponding data structure according to the symbol frequency distribution; processing at least part of the data elements according to the compression encoding method and / or data structure to generate a first encoding rule corresponding to the first data block.

[0011] For example, in the data processing method provided in at least some embodiments of the present disclosure, the compression coding method is Huffman coding, and the data structure is a tree; some data elements are processed to generate a first coding rule corresponding to the first data block, including: constructing a Huffman tree of at least some data elements based on the symbol frequency distribution; generating a Huffman coding table based on the Huffman tree, and the Huffman coding table is used as the first coding rule corresponding to the first data block.

[0012] For example, in the data processing method provided by at least some embodiments of the present disclosure, constructing a Huffman tree of at least part of the data elements includes: merging tree nodes corresponding to two symbols with closest symbol frequencies determined according to the symbol frequency distribution to construct a Huffman tree of at least part of the data elements.

[0013] For example, the data processing method provided in at least some embodiments of the present disclosure further includes: determining the data type of the first data block based on the metadata of the first data block.

[0014] At least one embodiment of the present disclosure further provides a data processing device, including a receiving module and a responding module, the receiving module being configured to receive a compression processing request for data to be processed, the data to be processed including multiple data blocks, the multiple data blocks including at least two data blocks of different data types; the responding module being configured to respond to the compression processing request, for at least a first data block among the multiple data blocks, perform a coding rule query in a coding storage space based on the data type of the first data block, and respond according to the coding rule query result to perform coding processing on the first data block; wherein responding according to the coding rule query result includes: in response to the coding rule query result being that no query is found, generating a first coding rule corresponding to the first data block according to the data type of the first data block and at least part of the data elements in the first data block, and storing the first coding rule in the coding storage space; or in response to the coding query result being that the first coding rule corresponding to the first data block stored in the coding storage space is found, calling the first coding rule, the first coding rule being used to perform coding processing on the first data block.

[0015] For example, in the data processing device provided in at least some embodiments of the present disclosure, it also includes an acquisition module and a coding rule generation module. The acquisition module is configured to acquire at least part of the data elements in the first data block that meet the corresponding coding data amount requirement, and the coding data amount requirement is the minimum data amount required to obtain the symbol frequency distribution law of the first data block; the coding rule generation module is configured to process at least part of the data elements according to the compression coding strategy corresponding to the data type of the first data block to generate a first coding rule.

[0016] For example, the data processing apparatus provided in at least some embodiments of the present disclosure further includes a data type identification module, which is configured to determine the data type of the first data block based on the metadata of the first data block.

[0017] At least some embodiments of the present disclosure also provide an electronic device comprising at least one memory or at least one processor, wherein the at least one memory stores computer-executable instructions; and at least one processor is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, implement the data processing method provided by any embodiment of the present disclosure.

[0018] At least some embodiments of the present disclosure further provide a non-transitory storage medium that non-transitory stores computer-executable instructions, wherein when the computer-executable instructions are executed by at least one processor, the data processing method provided by any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0020] Figure 1 An exemplary schematic diagram of Huffman encoding logic is shown;

[0021] Figure 2 A schematic flow chart of a data processing method provided by at least one embodiment of the present disclosure is shown;

[0022] Figure 3 A block diagram of a data processing device provided by at least one embodiment of the present disclosure is shown;

[0023] Figure 4 An exemplary logical diagram of a data processing device provided by at least one embodiment of the present disclosure is shown;

[0024] Figure 5 A schematic structural diagram of an electronic device provided by at least one embodiment of the present disclosure is shown;

[0025] Figure 6 A schematic structural diagram of an electronic device provided by at least one embodiment of the present disclosure is shown; and

[0026] Figure 7 A schematic diagram of a non-transitory storage medium provided by at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0029] The present disclosure is described below through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of known functions and known components. When any component of an embodiment of the present disclosure appears in more than one figure, the component is represented by the same or similar reference numeral in each figure.

[0030] Huffman coding is a lossless data compression technique based on the frequency (probability) of symbol occurrences. Proposed by David Huffman in 1952, Huffman coding is a variable-length coding (VLC) method whose core concept is to assign code lengths based on the frequency of symbol occurrence in the data. By assigning shorter codes to frequently occurring symbols and longer codes to less frequently occurring symbols, the overall code length is minimized. This coding method not only effectively reduces data volume and file size, thereby saving storage space and transmission bandwidth, but also ensures data integrity and reliability, thereby increasing data transmission speeds, reducing storage costs, and improving production efficiency. Therefore, Huffman coding plays a vital role in modern data management and transmission, and is widely used in file transfer, network communications, data storage, and processing, providing efficient solutions for production and daily life.

[0031] The compressed file generated by Huffman encoding consists of two parts: the compressed source file and the file header used for decoding (the Huffman table). The Huffman table is a mapping table between symbols in the source file and the corresponding codes in the compressed file. This table reflects the mapping relationship between symbols in the source file and codes in the compressed file.

[0032] Entropy is the theoretical limit of data compression. According to Shannon's First Theorem (also known as the noiseless coding theorem), for any compression coding method, the average code length after encoding cannot be less than the entropy. In other words, entropy is the theoretical lower limit for lossless compression; the average code length of any coding scheme is theoretically greater than or equal to the entropy. Average Code Length (ACL) refers to the average length of all symbols after encoding during the encoding process and reflects the efficiency of the coding scheme. Because Huffman coding is an optimal coding method based on symbol probabilities, it can minimize the average code length, thereby providing an average code length close to the entropy. Therefore, in theory, Huffman coding has a better code length and a better compression ratio.

[0033] Figure 1 An exemplary schematic diagram of Huffman encoding logic is shown.

[0034] like Figure 1 As shown, the Huffman encoding logic includes a frequency statistics module 10, a Huffman table generation module 20, an encoding module 30 and a decoding module 40.

[0035] If the input data is data to be encoded, it is first subjected to frequency statistics by the frequency statistics module 10, and then a Huffman table is generated for the input data according to the Huffman table generation module 20. Then, the input data can be encoded according to the generated Huffman table by the encoding module 30 to generate compressed data.

[0036] If the input data is compressed data to be decoded, the decoding module 40 can decode the input data according to the Huffman table used for encoding to obtain decompressed data.

[0037] The following describes the Huffman coding process. Huffman coding is usually based on a static coding table. That is, before data compression begins, the frequency of occurrence of all data symbols is pre-counted and a fixed coding table is generated. The implementation process of Huffman coding mainly includes the following steps:

[0038] 1. Frequency statistics: Identify the input data set, count the frequency of occurrence of each symbol in the data set, and calculate its probability of occurrence. The probability of occurrence is used to determine the weight of each symbol.

[0039] For example, the input source symbols and their weights are expressed as:

[0040] Source symbol set: I={i0, i1, i2, …, i n},

[0041] Weight set: W={w0, w1, w2, …, w n},

[0042] Among them, I is the source symbol set to be encoded, i j is the source symbol to be encoded (j=0,1,2……,n, j is the type identifier of the source symbol to be encoded, n is an integer), i n is the nth symbol; W is the weight set corresponding to the symbol, w j Is the symbol i j The corresponding weights (j=0,1,2…,n) are proportional to the frequency of the corresponding symbols in the data set.

[0043] 2. Huffman table generation: First, construct a binary tree. Second, use the constructed binary tree to map symbols to binary codes to generate codes.

[0044] 2.1. Construct a binary tree: Construct a special binary tree based on the probability of occurrence of symbols.

[0045] For example, you can first select the two symbols with the lowest probability of occurrence. For example, you can assign these two symbols with the lowest probability as code element 0 and code element 1 respectively. For example, you can assign the less frequent of the two symbols as code element 0 and the other as code element 1, or you can assign the less frequent of the two symbols as code element 1 and the other as code element 0. Then, add the probabilities of the two symbols with the lowest probability of occurrence to form a new symbol, and use the sum as the probability of occurrence of the new symbol. Then, reorder the probability of occurrence of this new symbol with the other symbols, and repeat the above process until the entire tree is constructed.

[0046] 2.2. Generate code: Generate a unique binary code for each symbol based on the constructed binary tree. The coding path goes from the root node to the leaf node of the tree, and then outputs the coding table (Huffman table), that is, the coding mapping table corresponding to the source symbol. For example, the Huffman table can be expressed as: H = {H0, H1, H2, …, H n}, where n is an integer.

[0047] For example, the source symbol set I = {i0, i1, i2, …, i7}, and the corresponding weight set W = {0.25, 0.21, 0.2, 0.15, 0.1, 0.05, 0.03, 0.01}. Table 1 illustrates the Huffman coding application process of the above source symbols and weight sets.

[0048] Table 1 Example process of Huffman coding to encode source symbols

[0049]

[0050] As can be seen from Table 1 above, the steps for constructing a binary tree for Huffman coding are:

[0051] (1) Sort all source symbols in order of probability from small to large.

[0052] (2) Select the two symbols with the smallest probabilities and merge them to form a new symbol whose probability is the sum of the probabilities of the two symbols.

[0053] (3) Add the new symbol to the source symbol set to replace the two original symbols with the lowest probability and reorder them.

[0054] (4) Repeat steps (2) and (3) until all symbols are merged into a binary tree (also called a Huffman tree).

[0055] Then, starting from the root node of the Huffman tree, the tree can be traversed downward to the leaf node corresponding to each source symbol. For example, the less frequent of the two symbols can be assigned as code 1, and the other symbol can be assigned as code 0. Alternatively, during the traversal process, the left branch codeword can be assigned as 1 and the right branch codeword as 0, or the left branch codeword can be assigned as 0 and the right branch codeword as 1. The codeword of each source symbol is formed based on the sequence of "0" and "1" encountered on the path from the root node to the symbol.

[0056] In the above-mentioned Huffman coding method, a Huffman tree is constructed to realize the mapping of symbols to binary codes. Therefore, the symbols with lower frequencies are located in the leaf nodes in the deep layer of the tree (i.e., farther away from the root node of the Huffman tree), while the symbols with higher frequencies are located in the leaf nodes in the shallow layer of the tree (i.e., closer to the root node of the Huffman tree). This can minimize the length of the overall coding, and Huffman coding can ensure that the compressed data can be completely restored to its original state, which is very important for application scenarios that require maintaining data integrity.

[0057] Although the Huffman coding shown above has shown many advantages in the field of data compression, the inventors of the present disclosure have noticed that the Huffman coding method still has limitations in certain scenarios, especially when facing dynamic data and large-scale data sets.

[0058] First, Huffman coding works well for static datasets with relatively stable distributions of probability. For example, assuming a static data probability distribution, the same compression standard is applied to all data types. However, in real-world datasets, the probability distribution of symbols is always uncertain. When the data distribution changes, a fixed Huffman table determined by the relatively stable distribution of probability will result in the overall encoding being suboptimal. Therefore, Huffman coding is less than ideal for dynamic datasets and suffers from poor compression when processing diverse data.

[0059] Secondly, Huffman coding's ability to achieve lossless compression (i.e., the original data can be fully recovered from the encoded data) relies on the completeness of the Huffman table. A complete Huffman table typically requires a lot of storage space. Therefore, the aforementioned Huffman encoding method doesn't offer optimal compression for all data types when dealing with data with varying data types. Furthermore, because the compression ratio is directly affected by the size of the Huffman table, storing Huffman tables for all data types significantly reduces the data's compression ratio. Especially when the source file is small, the space occupied by the Huffman table may be larger than that occupied by the compressed file.

[0060] Furthermore, Huffman coding also faces challenges in hardware implementation. The need to process the symbol frequency statistics of the dataset and generate a complex tree structure for the source file to be encoded increases the complexity of hardware design, especially when processing large datasets. This also affects the encoding and decoding speed of large datasets.

[0061] Furthermore, the Huffman tables generated for large, diverse data sets can become extremely large, requiring significant storage space to store them. This is particularly true when adapting to a wide variety of data types. This requires generating separate encoding tables for each type of data, while also requiring significant storage space to store different encoding tables to meet the encoding requirements of each data type. This increases the storage burden and, consequently, the cost and complexity of hardware implementation.

[0062] At least one embodiment of the present disclosure provides a data processing method, comprising: receiving a compression processing request for data to be processed, the data to be processed including multiple data blocks, the multiple data blocks including at least two data blocks of different data types; responding to the compression processing request, for at least a first data block among the multiple data blocks, based on the data type of the first data block, performing a coding rule query in a coding storage space, and responding according to the coding rule query result to perform coding processing on the first data block. Responding according to the coding rule query result includes: in response to the coding rule query result being no result found, generating a first coding rule corresponding to the first data block based on the data type of the first data block and at least some of the data elements in the first data block, and storing the first coding rule in the coding storage space; or, in response to the coding query result being a result of finding the first coding rule corresponding to the first data block stored in the coding storage space, calling the first coding rule, the first coding rule being used to perform coding processing on the first data block.

[0063] In the data processing method of the above-mentioned embodiment of the present disclosure, by performing a coding rule query in the coding storage space according to the data type of the data to be processed, and dynamically responding according to the coding rule query result, when the coding query result is a query found (i.e., the query hits), it is directly called, and when the coding query result is not a query found (i.e., the query misses), the coding rules are dynamically generated and stored according to the data type of the data block, so that there is no need to store the coding rules corresponding to data of all data types, and adaptive adjustments can be made to different data types, so that the coding storage space used to store the coding does not need to be too large, reducing the large-scale use of storage space, alleviating the storage burden, and reducing the cost and complexity of hardware implementation; and, in at least one example, data blocks of different types in the data to be processed can be responded to separately, and more effective compression can be provided for the data to be processed containing different data types in an intelligent and flexible manner, so that the compression processing response speed is accelerated to achieve better compression efficiency.

[0064] The various embodiments of the present disclosure will be described below with reference to specific examples.

[0065] like Figure 2 As shown, in some embodiments of the present disclosure, the data processing method includes steps S30 and S31.

[0066] Step S30: receiving a compression processing request for the data to be processed.

[0067] Step S31: In response to a compression processing request, for at least the first data block among multiple data blocks, a coding rule query is performed in the coding storage space based on the data type of the first data block, and a response is made according to the coding rule query result to perform coding processing on the first data block.

[0068] Furthermore, responding according to the encoding rule query result in the above step S31 includes step S311 or step S310.

[0069] Step S310: In response to the encoding rule query result being no result found, generate a first encoding rule corresponding to the first data block according to the data type of the first data block and at least some data elements in the first data block, and store the first encoding rule in the encoding storage space.

[0070] Step S311: In response to the encoding query result being that a first encoding rule corresponding to the first data block stored in the encoding storage space is found, the first encoding rule is called.

[0071] Here, the first encoding rule is used to refer to an encoding rule for encoding the first data block.

[0072] The data to be processed includes multiple data blocks, and the multiple data blocks include at least two data blocks of different data types. For example, each data block has a corresponding data type, and the data type may include one or more of text, image, audio, multimodality or other multimedia. For example, the data to be processed may include text-type data blocks and image-type data blocks, or the data to be processed may include audio-type data blocks and other multimedia-type data blocks, etc. The present disclosure does not limit the specific data types included in the data to be processed.

[0073] For example, the data to be processed may be data that has been error-checked and corrected using an ECC (Error-Correcting Code) algorithm, thereby ensuring that even if outliers or corruption occur in the data stream of the data to be processed, the encoding process of the data blocks in the data to be processed remains reliable.

[0074] The first data block is any data block among the multiple data blocks, and the partial data elements are partial data in the first data block. For example, the partial data elements may represent the symbol frequency distribution of the first data block to a certain extent, and the amount of data in the partial data elements is less than the total amount of data in the first data block. For example, the at least partial data elements may be partial data elements or may also be more data elements including the partial data elements, for example, may also be all data elements including the partial data elements in the first data block.

[0075] By generating a first encoding rule corresponding to the first data block based on the data type of the first data block and at least part of the data elements, in at least one example, since the data amount of at least part of the data elements is smaller than the data amount of the first data block, the first encoding rule is generated using at least part of the data elements. The generated first encoding rule can not only largely cover the data that needs to be encoded in the first data block, but also improve the generation efficiency of the first encoding rule and avoid the generated first encoding rule being too large.

[0076] It should be noted that the encoding rules (e.g., the first encoding rule) stored in the encoding storage space can autonomously learn and evolve based on the statistical information of the data stream during the encoding process of the corresponding data block (e.g., the first data block). The autonomous learning and evolution of the encoding rules not only occurs during the generation phase of the encoding rules, but also can be optimized in real time during the process as the encoding rules are used to encode the corresponding data block. For example, when the first encoding rule is used to compress and encode the first data block, if there are symbols to be encoded in the first data block that do not appear in the first encoding rule, the encoding rules for the symbols to be encoded that do not appear in the first encoding rule can be added to the original first encoding rule to obtain a new first encoding rule.

[0077] The coding storage space can be a storage space in a data processing device for storing coding rules corresponding to different data types. For example, the coding storage space can be provided by a reserved buffer (Buffer) (for example, through a static random access memory (SRAM)). For example, the coding storage space stores multiple coding rules, and the corresponding coding storage space can be provided by multiple buffers, or, for example, the coding storage space provided by the buffer can be divided into multiple storage spaces, each storage space corresponding to the storage of one coding rule. The storage form of the specific coding rules is not limited in this disclosure.

[0078] For example, the encoding rules stored in the encoding storage space are configured with corresponding data type identifiers to facilitate finding the encoding rule of the corresponding type based on the data type identifier. For example, the encoding rule of the corresponding data type (e.g., the first encoding rule) can be queried based on the data type of a data block (e.g., the first data block). If the encoding rule query result is "found", the corresponding encoding rule (e.g., the first encoding rule) can be directly called from the encoding storage space to perform encoding processing on the corresponding data block (e.g., the first data block).

[0079] For example, when the result of the encoding rule query is that no encoding rule is found, an encoding rule (e.g., a first encoding rule) for the corresponding data block (e.g., the first data block) can be generated immediately and stored in the encoding storage space. For example, when the encoding storage space is not full, the encoding rule can be stored in the free storage space of the encoding storage space. For example, after the first encoding rule is stored in the encoding storage space, the first encoding rule can be called to encode the first data block. For example, after the first encoding rule is generated, the first data block can be encoded using the generated first encoding rule, and the first encoding rule can be stored in the encoding storage space.

[0080] For example, after the encoding rules (such as the first encoding rules) corresponding to the data block (such as the first data block) are stored in the encoding storage space, if the encoding processing of another data block of the same data type as the data block is continued, the data processing method corresponding to the query result of the above encoding rules can continue to be applied.

[0081] For example, when the corresponding encoding rule is stored in the encoding storage space and the encoding rule has not been replaced in the encoding storage space, it means that the encoding rule exists in the encoding storage space and can be queried. Therefore, the corresponding processing steps when the encoding rule query result is queried can be applied, that is, the corresponding encoding rule (for example, the first encoding rule) can be directly called from the encoding storage space to encode the corresponding data block (for example, the first data block).

[0082] For example, when the corresponding encoding rule is stored in the encoding storage space, but the encoding rule has been replaced in the encoding storage space, it means that the encoding rule does not exist in the encoding storage space and the encoding rule cannot be queried. Therefore, the corresponding processing steps when the encoding rule query result is not found can be applied, that is, the encoding rule of the corresponding data block is generated and the encoding rule is stored in the encoding storage space.

[0083] It should be noted that the coding rules stored in the coding storage space can be replaced according to replacement rules such as the storage duration of the coding rules, the newness of the coding rules, etc. The present disclosure does not limit the specific replacement rules. It should be noted that for data blocks of different data types in the data to be processed, they can be responded to separately according to their respective coding query results, so that the data to be processed can be divided into multiple data blocks, and each data block is encoded separately using its own corresponding coding rules, which helps to deal with the difference in symbol frequency in different data blocks. If the hardware allows, two data blocks with different coding query results can be responded to in parallel to further improve the response speed; or, the above-mentioned embodiment of the present disclosure can be used on multiple processors or multiple processor cores to respond dynamically at the same time, so that different processors or processing cores can respond and process in parallel, thereby improving the processing speed of large-scale data sets.

[0084] The above-mentioned embodiments of the present disclosure can dynamically and effectively compress data blocks of various different data types, are flexibly applicable to various application scenarios, and can significantly reduce the storage space required for storing encoding rules without sacrificing compression efficiency, further reducing hardware costs.

[0085] In some embodiments of the present disclosure, step S310 of the above data processing method may further include step S32.

[0086] Step S32: In response to the encoding storage space being full, the replaced encoding rule determined by the calling situation in the encoding storage space is replaced to obtain storage space for storing the first encoding rule.

[0087] For example, when the encoding rule query result is not found, the encoding rule (such as the first encoding rule) generated. If the encoding storage space is full and there is no free storage space for storage, the replaced encoding rule can be determined and then replaced and stored. The replaced encoding rule is determined by the call situation.

[0088] Here, "call status" is used to reflect whether each coding rule stored in the coding storage space has been called or how frequently it has been called. For example, the call status may include the number of calls within a certain period of time, or it may be a call status classification determined by comprehensively combining the historical number of calls in different time periods, etc. The specific call status is not limited in this disclosure.

[0089] In addition, by replacing a selected replaced encoding rule (for example, at least the least recently used encoding rule) in the encoding storage space, the utilization of the encoding storage space can be made more efficient, and the required encoding storage space can be significantly reduced.

[0090] For example, in some embodiments of the present disclosure, the above data processing method may further include step S320.

[0091] Step S320: In response to the statistical result of the calling frequency of each coding rule in the coding storage space, the coding rule with the lowest calling frequency in the coding storage space is determined as the coding rule to be replaced.

[0092] For example, the calling frequency may be the number of calls within a preset time period. For example, each time a coding rule stored in the coding storage space is called, the calling frequency of the called coding rule is marked. For example, after a coding rule is called, a hardware counter or register may be used to record the number of calls, thereby determining the calling frequency.

[0093] For example, after the replaced coding rule is determined, the first coding rule may be written to the location where the replaced coding rule is stored, for example, the coding rule with the lowest current call frequency may be replaced with the first coding rule.

[0094] By replacing the least frequently called encoding rule in the encoding storage space with the first encoding rule, there is no need to expand the encoding storage space after a new encoding rule is generated, thereby achieving dynamic and efficient use of the encoding storage space and reducing the consumption of hardware storage space.

[0095] In some embodiments of the present disclosure, step S310 of the above data processing method may further include steps S33 and S34.

[0096] Step S33: obtaining at least some data elements in the first data block that meet the corresponding coding data amount requirement, where the coding data amount requirement is the minimum data amount required to obtain the symbol frequency distribution rule of the first data block.

[0097] Step S34: Process at least part of the data elements according to the compression coding strategy corresponding to the data type of the first data block to generate a first coding rule.

[0098] For the first data block, the coded data amount requirement may be set to the minimum data amount required to reflect the symbol frequency distribution pattern in the first data block. The data amount of some data elements determined according to the coded data amount requirement may be greater than or equal to the coded data amount requirement.

[0099] For example, part of the data in the first data block may be obtained according to a preset coding data amount requirement. For example, data blocks of different data types may correspond to respective coding data amount requirements.

[0100] For example, different data types may correspond to different compression encoding strategies.

[0101] In some embodiments of the present disclosure, step S33 of the above data processing method may further include step S35.

[0102] Step S35: Determine the encoding data amount requirement corresponding to the first data block based on the mapping relationship between the data type and the encoding data amount requirement and the data type of the first data block, so as to determine some data elements in the first data block that meet the corresponding encoding data amount requirement.

[0103] For example, the pre-set encoding data volume requirements corresponding to different data types can be stored so that after determining the data type of each data block, the corresponding encoding data volume requirements can be determined by a table lookup method; for example, it can be set that when the data type is text data, a minimum of 8KB of data volume is required; when the data type is an image, a minimum of 32KB of data volume is required.

[0104] It should be noted that the embodiments of the present disclosure do not limit the method for determining the mapping relationship between data types and encoded data volume requirements.

[0105] In some embodiments of the present disclosure, the above data processing method may further include step S40.

[0106] Step S40: Determine a compression coding strategy corresponding to the data type of the first data block according to data features corresponding to the data type of the first data block.

[0107] The data feature is information reflecting the data characteristics of the data type corresponding to the data block, and the data features corresponding to different data types may be different.

[0108] For example, data characteristics include data patterns and / or data structures.

[0109] For example, in the process of determining the data type of a data block (eg, the first data block), data features and data content may be analyzed to determine a data pattern and / or data structure corresponding to the data type.

[0110] For example, a data pattern may be whether the data content contains repeated character strings or character sequences. The data pattern is used to determine the patterns and regularities in the data, thereby enabling better prediction and compression of the data.

[0111] For example, the data structure may include an array, a linked list, a stack, a queue, a tree, or a graph, and the data structure may be used to efficiently manage and operate the data blocks to be compressed.

[0112] In some embodiments of the present disclosure, step S34 in the above data processing method may further include step S41 and step S42.

[0113] Step S41: Determine the symbol frequency distribution of at least part of the data elements according to the data pattern, where the data pattern includes the symbol type corresponding to the data type and / or the probability of occurrence of a certain symbol type.

[0114] Step S42: Process at least part of the data elements according to the data type and symbol frequency distribution of the first data block to generate a first encoding rule corresponding to the first data block.

[0115] For example, before determining the symbol probability distribution of some data elements according to the data pattern, some data elements may be preprocessed. For example, differential coding may be used for preprocessing to reduce redundancy in the data.

[0116] For example, character frequencies can be counted based on data patterns, the frequency of occurrence of each character in some data elements can be calculated, and the frequency of occurrence of each element in the input data can be counted to determine the symbol frequency distribution of some data elements.

[0117] The type of symbol that appears corresponding to a data type may be different for different data types. For example, for text data (i.e., natural language text), the corresponding symbols may be letters, numbers, special characters (such as ℃, $, etc.), spaces, or punctuation marks; for image data, the corresponding symbols may be pixel values, color components, or frequency domain components; for audio data, the corresponding symbols may be pitch, loudness, or spectrum; for multimodal data, the corresponding symbols may be the correspondence between different modalities, such as the synchronous correspondence between image and audio data; for other multimedia data such as web pages, the corresponding symbols may be links, codes, etc.

[0118] The probability of occurrence of a certain symbol type may be different in different data types. For example, in text data, the probability of letters appearing is greater than that of punctuation marks, and the probability of punctuation marks such as periods, commas, and spaces appearing is higher. For image data, the probability of occurrence of color components such as green, blue, and gray is higher, and the probability of occurrence of low-frequency components representing smoothness in the frequency domain is higher than that of high-frequency components representing details and edges. For audio data, the probability of occurrence of symbol types corresponding to tones of C and G is higher. For multimodal data, the probability of occurrence of key-value pairs is higher. For other multimedia, the probability of occurrence of symbols in data formats such as HTML, XML, and JSON is higher.

[0119] Since the symbol frequency distribution is determined according to the data pattern corresponding to the data type, the symbol frequency distribution of data blocks of different data types can be dynamically updated to adapt to the different data types of each data block, so that the encoding rules corresponding to each data block in the data to be processed can be dynamically and adaptively adjusted, and the encoding rules can be optimized for the current data type.

[0120] The above-mentioned embodiments of the present disclosure determine the symbol frequency distribution according to the data pattern corresponding to the data type, so that the statistics of the symbol frequency distribution of some data elements are more efficient and can more accurately reflect the symbol frequency distribution of the data block of the corresponding data type; generate a first encoding rule according to the symbol frequency distribution, so that the generated first encoding rule can better adapt to the characteristics of the corresponding data type; generate the first encoding rule by processing at least part of the data elements, so that compared with directly using the first data block itself to statistically generate the encoding rule, the first encoding rule occupies less space, so that the compression ratio is improved and the hardware cost is reduced.

[0121] In some embodiments of the present disclosure, step S42 in the above data processing method may further include step S420 and step S421.

[0122] Step S420: determining a corresponding compression encoding method according to the data type of the first data block, and / or determining a corresponding data structure according to the symbol frequency distribution.

[0123] Step S421: Process at least part of the data elements according to the compression encoding method and / or data structure to generate a first encoding rule corresponding to the first data block.

[0124] One or more compression encoding methods for a data type may be selected based on the data characteristics corresponding to the data type and the compression efficiency and performance.

[0125] For example, for text data, since some characters are repeated many times, Huffman coding can be used so that each character is finally encoded in binary, effectively reducing the encoding length of high-frequency characters, thereby realizing a character frequency-based encoding and achieving the purpose of compression.

[0126] For example, for image data, the image data can be read to determine the frequency distribution of pixel values ​​or color components in the image data. Since the pixel value distribution of image data is complex, if only Huffman coding is used, the coding table will be very large, and the compression efficiency and performance will be poor. Therefore, Huffman coding can be combined with other compression coding methods (for example, combined with discrete cosine transform in JPEG).

[0127] For example, for audio data, since audio has continuous waveform characteristics, its compression usually relies on two compression coding methods: frequency domain transformation and quantization processing. On the basis of the above two compression coding methods, Huffman coding can be further used to compress the quantized symbols of the audio data.

[0128] The choice of compression encoding method can be determined comprehensively based on hardware configuration and compression requirements, and this disclosure does not limit this.

[0129] The appropriate data structure for different types of data can be determined based on the symbol frequency distribution. For example, for text data, a tree data structure is more suitable due to the large differences in the frequency of character occurrences. For image data, a multidimensional array data structure can be used to more conveniently represent the image pixel matrix due to the large number of pixel matrices. For multimodal data, a graph data structure can be used to represent the relationship through the association between nodes and edges due to the large amount of data with corresponding relationships. The choice of data structure can be considered based on the comprehensive processing requirements and is not limited in this disclosure.

[0130] By determining the compression encoding method and / or data structure corresponding to the data type, data blocks of each data type can be compressed more dynamically and adaptively.

[0131] For example, in some embodiments of the present disclosure, the compression coding method is Huffman coding, and the data structure is a tree; step S34 in the above data processing method also includes step S50 and step S51.

[0132] Step S50: constructing a Huffman tree of at least part of the data elements according to the symbol frequency distribution.

[0133] Step S51: Generate a Huffman coding table according to the Huffman tree, and use the Huffman coding table as a first coding rule corresponding to the first data block.

[0134] The generated first encoding rule (eg, Huffman encoding table) can be used for compression or decompression. For example, during compression, the generated Huffman encoding table is used to encode the first data block to obtain compressed data of the first data block.

[0135] For example, during decompression, the compressed first data block is reversely parsed according to the generated Huffman coding table to obtain decompressed data of the first data block.

[0136] Since the data contents of data blocks of different data types are different, the corresponding symbol frequency distributions are also different. Therefore, for example, when Huffman coding is adopted by default and the default coding rule (such as the first coding rule) is the Huffman tree, the data processing method of the above embodiment of the present disclosure can dynamically generate Huffman trees of corresponding data types for data blocks belonging to different data types, thereby achieving the purpose of dynamically adjusting the Huffman table to match different data types as the data type changes.

[0137] The above-mentioned embodiments of the present disclosure match the Huffman table of the corresponding data type for data of different data types as encoding rules, thereby reducing the delay of compression and decompression and the consumption of computing resources, making the compression and decompression process faster, and improving the data processing speed, thereby improving the data throughput of the entire system and the system performance under big data processing and high-speed data transmission.

[0138] For example, when constructing a Huffman tree for at least some data elements, a priority queue can be used to manage each tree node. For example, the priority queue can sort the tree nodes corresponding to each symbol in ascending order of symbol frequency. For example, in the process of constructing the Huffman tree, the two tree nodes with the lowest symbol frequency can be taken out from the priority queue each time and merged to create a new tree node. The symbol frequency of the new tree node is the sum of the symbol frequencies of the above two tree nodes. Afterwards, the new tree node is reinserted into the priority queue to replace the two merged tree nodes, and the same sorting, merging, and creation process as described above is repeated until only one tree node remains in the priority queue, thereby completing the construction of the Huffman tree.

[0139] For example, in the process of constructing the Huffman tree, an optimized data structure (eg, an optimized tree structure) may be used to reduce the storage space required for the encoding rule (eg, the first encoding rule).

[0140] In some embodiments of the present disclosure, the data processing method step S50 may further include step S501.

[0141] Step S501: merging tree nodes corresponding to two symbols with the closest symbol frequencies determined according to the symbol frequency distribution to construct a Huffman tree of at least part of the data elements.

[0142] For example, during the construction of the Huffman tree, a dynamic merging strategy can be employed, that is, nodes with similar symbol frequencies are preferentially merged. For example, the two tree nodes with the closest symbol frequencies can be taken from the priority queue each time and merged, thereby creating a new tree node to replace the two merged tree nodes. The symbol frequency of the new tree node is the sum of the symbol frequencies of the two tree nodes with the closest symbol frequencies.

[0143] Through the above embodiments of the present disclosure, the height of the generated Huffman tree can be reduced, thereby obtaining a more balanced Huffman tree and reducing the storage space required for the encoding rules.

[0144] In some embodiments of the present disclosure, the data processing method may further include step S60.

[0145] Step S60: Determine the data type of the first data block according to the metadata of the first data block.

[0146] The metadata of the first data block is also the header information of the first data block. For example, information such as the file format of the first data block is included in the header information. Therefore, the data type of the first data block can be determined based on the file format included in the metadata of the first data block.

[0147] For example, at least one of the above embodiments of the present disclosure may include encrypting the encoded first data block to ensure data security and user privacy while compressing the data.

[0148] Figure 3 A block diagram of a data processing device provided by at least one embodiment of the present disclosure is shown.

[0149] At least one embodiment of the present disclosure further provides a data processing device 300 including a receiving module 310 and a responding module 320 .

[0150] The receiving module 310 is configured to receive a compression processing request for data to be processed, where the data to be processed includes a plurality of data blocks, and the plurality of data blocks include at least two data blocks of different data types.

[0151] The response module 320 is configured to respond to a compression processing request, perform a coding rule query in the coding storage space based on the data type of the first data block for at least the first data block among multiple data blocks, and respond according to the coding rule query result to perform coding processing on the first data block.

[0152] For example, the operation of responding according to the encoding rule query result includes: in response to the encoding rule query result being no result found, generating a first encoding rule corresponding to the first data block based on the data type of the first data block and at least part of the data elements in the first data block, and storing the first encoding rule in the encoding storage space; or, in response to the encoding query result being a first encoding rule corresponding to the first data block stored in the encoding storage space being found, calling the first encoding rule; the first encoding rule is used to encode the first data block.

[0153] In some embodiments of the present disclosure, the data processing device 300 further includes an acquisition module 330 and a coding rule generation module 340 .

[0154] The acquisition module 330 is configured to acquire at least some data elements in the first data block that meet a corresponding coding data amount requirement, where the coding data amount requirement is a minimum amount of data required to acquire a symbol frequency distribution rule of the first data block.

[0155] The encoding rule generation module 340 is configured to process at least part of the data elements according to the compression encoding strategy corresponding to the data type of the first data block to generate a first encoding rule.

[0156] In some embodiments of the present disclosure, the data processing device 300 further includes a data type identification module 350 .

[0157] The data type identification module 350 is configured to determine the data type of the first data block according to the metadata of the first data block.

[0158] For example, the response module 320 is further configured to replace the replaced encoding rule determined by the call situation in the encoding storage space in response to the encoding storage space being full, so as to obtain storage space for storing the first encoding rule.

[0159] For example, the response module 320 is further configured to determine the coding rule with the lowest calling frequency in the coding storage space as the coding rule to be replaced in response to the statistical result of the calling frequency of each coding rule in the coding storage space.

[0160] For example, the acquisition module 330 is also configured to determine the encoding data volume requirement corresponding to the first data block based on the mapping relationship between the data type and the encoding data volume requirement and the data type of the first data block, so as to determine at least some data elements in the first data block that meet the corresponding encoding data volume requirement.

[0161] For example, the encoding rule generation module 340 is further configured to determine a compression encoding strategy corresponding to the data type of the first data block according to a data pattern and / or data structure corresponding to the data type of the first data block.

[0162] For example, the encoding rule generation module 340 is also configured to determine the symbol frequency distribution of at least part of the data elements based on the data pattern, where the data pattern includes the symbol type corresponding to the data type and / or the probability of occurrence of a certain symbol type; and process at least part of the data elements based on the data type and symbol frequency distribution of the first data block to generate a first encoding rule corresponding to the first data block.

[0163] For example, the encoding rule generation module 340 is also configured to determine the corresponding compression encoding method based on the data type of the first data block, and / or determine the corresponding data structure based on the symbol frequency distribution; according to the compression encoding method and / or data structure, process at least part of the data elements to generate a first encoding rule corresponding to the first data block.

[0164] In some embodiments of the present disclosure, the compression coding method is Huffman coding, and the data structure is a tree; the coding rule generation module 340 is also configured to construct a Huffman tree of at least part of the data elements based on the symbol frequency distribution; and generate a Huffman coding table based on the Huffman tree, and the Huffman coding table serves as the first coding rule corresponding to the first data block.

[0165] For example, the encoding rule generation module 340 is further configured to merge tree nodes corresponding to two symbols with the closest symbol frequencies determined according to the symbol frequency distribution, so as to construct a Huffman tree of at least part of the data elements.

[0166] The technical effects of the data processing device of the above embodiment of the present disclosure are the same as the technical effects of the above data processing method, and therefore are not described in detail.

[0167] Figure 4 An exemplary logical diagram of a data processing device provided by at least one embodiment of the present disclosure is shown.

[0168] like Figure 4 As shown, when the data processing device 300 receives a compression processing request for the data to be processed, for example, the first data block among multiple data blocks in the data to be processed is first determined through the data type identification module 350, and then queries whether the corresponding first encoding rule is stored in the encoding storage space based on the data type of the first data block.

[0169] If the encoding query result is that the first encoding rule corresponding to the first data block stored in the encoding storage space is found, the first encoding rule is called.

[0170] If the encoding rule query result is that no encoding rule is found, the first data block enters the encoding rule generation module 340 to generate a first encoding rule.

[0171] The encoding rule generation module 340 includes a data preprocessing module 3401 , a frequency statistics module 3402 , a Huffman table generation module 3403 and a storage optimization module 3404 .

[0172] The first data block is first preprocessed by the data preprocessing module 3401. For example, the data preprocessing module 3401 preprocesses the first data block using differential encoding to reduce data redundancy in the first data block.

[0173] The preprocessed first data block can then enter the frequency statistics module 3402 to determine the symbol frequency distribution of various symbols included in the data elements of the first data block. For example, the symbol frequency distribution of at least some of the data elements in the obtained first data block that meet a corresponding coded data volume requirement can be determined based on the data pattern corresponding to the data type of the first data block. The coded data volume requirement is the minimum amount of data required to obtain the symbol frequency distribution pattern of the first data block.

[0174] Afterwards, the Huffman table generation module 3403 may generate a Huffman table according to the symbol frequency distribution of at least part of the data elements.

[0175] For example, before the encoding rule generation module 340 outputs the first encoding rule, the storage optimization module 3404 can perform storage optimization on the generated Huffman table, and the Huffman table optimized by the storage optimization module 3404 can be output as the first encoding rule. For example, storage optimization can be performed by dynamically adjusting the merging strategy. For example, the construction method of the Huffman tree can be optimized to preferentially merge nodes with similar symbol frequencies, thereby reducing the height of the Huffman tree and generating a more balanced Huffman tree, thereby reducing the storage space required for the Huffman table.

[0176] For example, the first encoding rule output by the encoding rule generation module 340 may be stored in the encoding storage space.

[0177] For example, the first data block may be data to be compressed. The encoding module 380 compresses the first data block and may use a first encoding rule to encode the first data block to generate compressed data of the first data block.

[0178] For example, the first data block may be data to be decompressed. The decoding module 390 may decompress the first data block and reversely parse the compressed first data block according to the generated Huffman coding table to obtain decompressed data of the first data block.

[0179] The technical effects of the data processing device of the above embodiment of the present disclosure are the same as the technical effects of the above data processing method, and therefore are not described in detail.

[0180] Figure 5A block diagram of an electronic device provided by at least one embodiment of the present disclosure is shown.

[0181] like Figure 5 As shown, the electronic device 600 includes at least one memory 610 and at least one processor 620 .

[0182] The memory 610 stores computer-executable instructions.

[0183] The processor 620 is configured to execute the computer-executable instructions. When the processor executes these computer-executable instructions, the data processing method provided by any embodiment of the present disclosure is implemented.

[0184] The memory 610 may include, for example, a semiconductor memory unit, such as a dynamic random access memory (DRAM), a random access memory (RAM), or a static random access memory (SRAM), or any other memory with a storage function.

[0185] The processor 620 may be, for example, any processing circuit with processing capabilities implemented through hardware or firmware, such as a central processing unit (CPU) or a coprocessor, a microcontroller unit (MCU), or a digital signal processor (DSP). For example, the coprocessor may be an accelerator (e.g., a graphics accelerator or a digital signal processing unit), a graphics processing unit (GPU), a programmable logic array, or any other processor with instruction execution capabilities. The embodiments of the present disclosure do not limit the implementation of the at least one memory 610 and the at least one processor 620.

[0186] The technical effects of the electronic device of the above embodiment of the present disclosure are the same as the technical effects of the above data processing method, and therefore are not described in detail.

[0187] Figure 6 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0188] The electronic devices in the embodiments of the present disclosure may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device 1000 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0189] For example, reference Figure 6 In some examples, electronic device 1000 includes a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003, such as executing the data processing method provided in any embodiment of the present disclosure. RAM 1003 also stores various programs and data required for the operation of the computer system. Processing device 1001, ROM 1002, and RAM 1003 are connected to each other via an internet network 1004. An input / output (I / O) interface 1005 is also connected to the internet network 1004.

[0190] For example, the following components can be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009 which can also include, for example, a network interface card such as a LAN card, a modem, etc. The communication device 1009 can allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data, and perform communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive 1010 as needed, so that the computer program read therefrom can be installed into the storage device 1008 as needed. Although Figure 6 The electronic device 1000 is shown as including various devices, but it should be understood that it is not required to implement or include all of the devices shown. More or fewer devices may be implemented or included instead.

[0191] For example, the electronic device 1000 may further include a peripheral interface (not shown in the figure), etc. The peripheral interface may be various types of interfaces, such as a USB interface, a lightning interface, etc. The communication device 1009 may communicate with a network and other devices through wireless communication, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communications may use any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0192] At least one embodiment of the present disclosure further provides a non-transitory storage medium that non-transitorily stores computer-executable instructions. For example, when the computer-executable instructions are executed by a processor, the data processing method provided by at least one embodiment of the present disclosure is implemented.

[0193] Figure 7 is a schematic diagram of a non-transitory storage medium provided by some embodiments of the present disclosure. Figure 7 As shown, the non-transitory storage medium 900 can non-transitory store computer-executable instructions 910 , which implement the data processing method provided by any embodiment of the present disclosure when executed by a computer.

[0194] Regarding this disclosure, the following points need to be explained:

[0195] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to conventional designs.

[0196] (2) Unless there is any conflict, the features of the same embodiment and different embodiments of the present disclosure may be combined with each other.

[0197] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A data processing method, comprising: receiving a compression processing request for data to be processed, wherein the data to be processed includes a plurality of data blocks, and the plurality of data blocks include at least two data blocks of different data types; In response to the compression processing request, for at least a first data block among the multiple data blocks, query an encoding rule in an encoding storage space based on a data type of the first data block, and respond according to a result of the encoding rule query to perform encoding processing on the first data block; Wherein, responding according to the encoding rule query result includes: In response to the encoding rule query result being no result found, generating a first encoding rule corresponding to the first data block based on the data type of the first data block and at least some data elements in the first data block, and storing the first encoding rule in the encoding storage space; or In response to the encoding query result being that the first encoding rule corresponding to the first data block stored in the encoding storage space is queried, the first encoding rule is called, The first encoding rule is used to encode the first data block.

2. The data processing method according to claim 1, wherein: The storing the first encoding rule in the encoding storage space includes: In response to the encoding storage space being full, the replaced encoding rule determined by the calling situation in the encoding storage space is replaced to obtain storage space for storing the first encoding rule.

3. The data processing method according to claim 2, wherein: The method for determining the replaced encoding rule determined by the calling situation includes: In response to a statistical result of the calling frequency of each coding rule in the coding storage space, the coding rule with the lowest calling frequency in the coding storage space is determined as the replaced coding rule.

4. The data processing method according to any one of claims 1 to 3, wherein: Generating a first encoding rule corresponding to the first data block according to the data type of the first data block and at least some data elements in the first data block includes: Obtaining the at least some data elements in the first data block that meet a corresponding coded data amount requirement, wherein the coded data amount requirement is a minimum amount of data required to obtain a symbol frequency distribution regularity of the first data block; The at least part of the data elements are processed according to a compression encoding strategy corresponding to the data type of the first data block to generate the first encoding rule.

5. The data processing method according to claim 4, wherein: Obtaining the at least some data elements in the first data block that meet the corresponding coded data amount requirement includes: Based on the mapping relationship between data type and coding data amount requirement and the data type of the first data block, the coding data amount requirement corresponding to the first data block is determined to determine the at least part of the data elements in the first data block that meet the corresponding coding data amount requirement.

6. The data processing method according to claim 4, further comprising: A compression coding strategy corresponding to the data type of the first data block is determined according to data features corresponding to the data type of the first data block.

7. The data processing method according to claim 6, wherein: The data characteristics include data patterns and / or data structures; Processing the at least part of the data elements according to a compression coding strategy corresponding to the data type of the first data block to generate a first coding rule corresponding to the first data block includes: Determining a symbol frequency distribution of at least part of the data elements according to the data pattern, wherein the data pattern includes a symbol type corresponding to the data type and / or an occurrence probability of a certain symbol type; The at least part of the data elements is processed according to the data type of the first data block and the symbol frequency distribution to generate a first encoding rule corresponding to the first data block.

8. The data processing method according to claim 7, wherein: The processing of the at least part of the data elements according to the data type of the first data block and the symbol frequency distribution to generate a first encoding rule corresponding to the first data block includes: determining a corresponding compression encoding method according to a data type of the first data block, and / or determining a corresponding data structure according to the symbol frequency distribution; According to the compression encoding mode and / or the data structure, the at least part of the data elements are processed to generate a first encoding rule corresponding to the first data block.

9. The data processing method according to claim 8, wherein: The compression coding method is Huffman coding, and the data structure is a tree; Processing the at least part of the data elements to generate a first encoding rule corresponding to the first data block includes: constructing a Huffman tree of at least part of the data elements according to the symbol frequency distribution; A Huffman coding table is generated according to the Huffman tree, and the Huffman coding table serves as a first coding rule corresponding to the first data block.

10. The data processing method according to claim 9, wherein: The constructing of the Huffman tree of at least part of the data elements comprises: The tree nodes corresponding to two symbols with the closest symbol frequencies determined according to the symbol frequency distribution are merged to construct a Huffman tree of the at least part of the data elements.

11. The data processing method according to any one of claims 1 to 3, further comprising: The data type of the first data block is determined according to the metadata of the first data block.

12. A data processing device comprising: A receiving module is configured to receive a compression processing request for data to be processed, wherein the data to be processed includes a plurality of data blocks, and the plurality of data blocks include at least two data blocks of different data types; a response module configured to, in response to the compression processing request, perform an encoding rule query in an encoding storage space based on a data type of at least a first data block among the multiple data blocks, and respond according to an encoding rule query result to perform encoding processing on the first data block; Wherein, responding according to the encoding rule query result includes: In response to the encoding rule query result being no result found, generating a first encoding rule corresponding to the first data block based on the data type of the first data block and at least some data elements in the first data block, and storing the first encoding rule in the encoding storage space; or In response to the encoding query result being that the first encoding rule corresponding to the first data block stored in the encoding storage space is queried, the first encoding rule is called, The first encoding rule is used to encode the first data block.

13. The data processing apparatus according to claim 12, further comprising: an acquisition module configured to acquire the at least some data elements in the first data block that meet a corresponding coded data amount requirement, wherein the coded data amount requirement is a minimum amount of data required to obtain a symbol frequency distribution regularity of the first data block; The encoding rule generating module is configured to process the at least part of the data elements according to the compression encoding strategy corresponding to the data type of the first data block to generate the first encoding rule.

14. The data processing apparatus according to claim 12 or 13, further comprising: The data type identification module is configured to determine the data type of the first data block according to the metadata of the first data block.

15. An electronic device comprising: at least one memory storing computer-executable instructions; as well as at least one processor configured to execute said computer-executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the data processing method according to any one of claims 1-11 is implemented.

16. A non-transitory storage medium that non-transitory stores computer-executable instructions, wherein: When the computer-executable instructions are executed by at least one processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images