A data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a data processing method and device, electronic equipment and storage medium, which can be applied to the technical field of data processing. The method comprises: obtaining a to-be-processed word from target data; determining a target coding dictionary to which the to-be-processed word belongs from different types of coding dictionaries according to the length of the to-be-processed word; determining a replacement code corresponding to the to-be-processed word according to the target coding dictionary; replacing the to-be-processed word with the replacement code in the target data to obtain coding data corresponding to the target data; and performing compression processing on the coding data to obtain a compression result for the target data. By using the embodiments of the present application, the data coding is more targeted, which helps to improve the compression rate of the data. The embodiments of the present application can also obtain the compression result of the target data through data calculation by a cloud server.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to the field of log data processing, and more particularly to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of science and technology, the amount of target data with high repetition has surged. To reduce the storage space occupied by target data, data compression has become a research hotspot. Taking log data as an example, log data typically has a huge volume and high repetition. In existing technologies, log data compression methods usually involve matching and compressing duplicate data within a certain data block using a window of a specific size. This method can only utilize the duplicate information within a single block for compression; the same data in different blocks will still be compressed repeatedly, resulting in a low compression ratio. Alternatively, the data can be encoded and converted before compression, but this method does not achieve a high compression ratio when facing high-concurrency random read / write big data log applications. Therefore, how to better process target data to improve the compression ratio has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, and medium that enables more targeted data encoding and helps improve data compression rates.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] Obtain the words to be segmented from the target data;

[0006] The target encoding dictionary to which the segment to be processed belongs is determined from one or more encoding dictionaries based on the length of the segment to be processed.

[0007] The replacement code corresponding to the word segment to be processed is determined according to the target encoding dictionary;

[0008] The word segment to be processed is replaced in the target data with the replacement encoding to obtain the encoded data corresponding to the target data;

[0009] The encoded data is compressed to obtain a compressed result for the target data.

[0010] On the other hand, embodiments of this application provide a data processing apparatus, which includes:

[0011] The acquisition module is used to obtain the words to be processed from the target data;

[0012] The processing module is used to determine the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries based on the length of the segment to be processed;

[0013] The processing module is further configured to determine the replacement code corresponding to the word segment to be processed based on the target encoding dictionary;

[0014] The processing module is further configured to replace the word segment to be processed in the target data with the replacement encoding to obtain the encoded data corresponding to the target data;

[0015] The processing module is further configured to compress the encoded data to obtain a compression result for the target data.

[0016] In another aspect, embodiments of this application provide an electronic device, characterized in that the electronic device includes a processor and a storage device, the processor and the storage device being interconnected, wherein the storage device is used to store computer program instructions, and the processor is configured to perform the following steps:

[0017] Obtain the words to be segmented from the target data;

[0018] The target encoding dictionary to which the segment to be processed belongs is determined from one or more encoding dictionaries based on the length of the segment to be processed.

[0019] The replacement code corresponding to the word segment to be processed is determined according to the target encoding dictionary;

[0020] The word segment to be processed is replaced in the target data with the replacement encoding to obtain the encoded data corresponding to the target data;

[0021] The encoded data is compressed to obtain a compressed result for the target data.

[0022] In another aspect, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the following steps:

[0023] Obtain the words to be segmented from the target data;

[0024] The target encoding dictionary to which the segment to be processed belongs is determined from one or more encoding dictionaries based on the length of the segment to be processed.

[0025] The replacement code corresponding to the word segment to be processed is determined according to the target encoding dictionary;

[0026] The word segment to be processed is replaced in the target data with the replacement encoding to obtain the encoded data corresponding to the target data;

[0027] The encoded data is compressed to obtain a compressed result for the target data.

[0028] In this embodiment, the target encoding dictionary to which the segment to be processed belongs can be determined from one or more encoding dictionaries based on the length of the segment to be processed. The replacement encoding corresponding to the segment to be processed is then determined based on the target encoding dictionary, allowing for the use of different encoding schemes to replace data from different dictionaries. The encoded data is then compressed to obtain a compression result specific to the target data, thereby improving the data compression rate. Therefore, by determining the replacement encoding corresponding to the data from different types of encoding dictionaries, data encoding becomes more targeted, contributing to a higher data compression rate. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of a dictionary data configuration item provided in an embodiment of this application;

[0032] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0033] Figure 4 This is a flowchart illustrating the principle of a compression algorithm provided in an embodiment of this application;

[0034] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0035] Figure 6a This is a schematic diagram of target data provided in an embodiment of this application;

[0036] Figure 6b This is a schematic diagram of a target data encoding process provided in an embodiment of this application;

[0037] Figure 6c This is a schematic diagram illustrating the correspondence between an encoding dictionary and marker characters provided in an embodiment of this application;

[0038] Figure 6dThis is a schematic diagram of encoded data provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0040] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0042] This application proposes a data processing scheme that can determine the replacement codes for each word segment to be processed in the target data from different types of encoding dictionaries, and match more suitable replacement codes for each word segment to be processed. Furthermore, the target data can be encoded according to each replacement code to obtain the encoded data corresponding to the target data, and compression can be performed based on the encoded data to obtain the compressed result of the target data. This makes the encoding of the target data more targeted and improves the compression rate of the target data.

[0043] The technical solution of this application can be applied to electronic devices, which can be terminals, servers, or other devices used for data processing; this application does not impose any limitations. Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication; this application does not impose any limitations on this connection.

[0044] Optionally, this solution can be applied to both the storage and transmission of target data; there are no restrictions. Taking log data as an example, when applied to log data storage, after obtaining the compression result (i.e., after the target data is compressed), the compression result can be stored, thereby reducing storage space usage. Optionally, the compression result can be stored on a local server or in the cloud; there are no restrictions. When applied to log data transmission, after obtaining the compression result (i.e., after the target data is compressed), the compression result can be transmitted to reduce the amount of data transmitted and improve transmission efficiency. Optionally, the compression result can be transmitted between computers, transmitted to the cloud, or retrieved from the cloud.

[0045] This solution significantly improves data compression rates compared to existing technologies when processing data with high repetition rates. Taking log data as an example, log data refers to various data generated during the operation of computer operating systems or application software. Compared to ordinary text data, log data has a higher degree of repetition, sometimes local and sometimes global. Simply using compression algorithms to compress log data can only utilize the repetitive data within a single block, resulting in a low compression rate. For example, a 64-byte UserID appears repeatedly in Blocks 1 through 10. Within each block, regardless of how the compression algorithm re-encodes the data, a certain amount of information is still needed to represent this 64-byte UserID. In the worst-case scenario, no duplicate strings may be found within the entire block. In Blocks 1 through 10, this means that at least 10 copies of this 64-byte UserID are stored. Therefore, directly using compression algorithms to compress data with high repetition rates cannot achieve good compression results and the compression ratio is low. However, the data processing scheme proposed in this application determines replacement codes from different types of encoding dictionaries and then compresses the encoded data, which can make the data encoding more targeted and improve the data compression ratio.

[0046] To help users better understand the application of this solution, the following example demonstrates how this data processing solution is applied in, for instance... Figure 1 The data processing scheme proposed in this embodiment of the invention will be described using the system shown as an example. See also... Figure 1The data processing system proposed in this application embodiment may include: a data processing server and multiple user terminals. The user terminals can be used to define parameters for the execution of this solution, and the data processing server can be used to execute the data processing operations of this solution. The data processing server can connect to multiple user terminals. Taking the processing of log data using the system proposed in this embodiment as an example, the data processing server can obtain the log data generated in the system and store the raw log data in a field of a source file based on the search service (Elasticsearch service), such as a field named "_source". In this solution, the default field type of the source file can be defined as the target type. This target type can be JSON, XML, etc., without limitation. If it is a user-defined field, its type can also be defined as the target type. When the data is refreshed into an index file (Segment) of the built-in index engine (Lucene) of the Elasticsearch service, the data processing server will store the data in a cache and determine the type of each field. If the data type is detected as JSON, the JSON data will be used as the target data in this solution for data processing operations. Some parameters used by the user to execute this solution on the data processing server can be defined by the user through the user terminal, such as... Figure 2 As shown, this is a configuration item data for processing the above-mentioned JSON type log data and its corresponding purpose. Obviously, the logical constraints in each configuration item shown in the figure are: the minimum length of high-frequency words is less than the minimum length of medium-frequency words, which is less than the minimum length of long words; the minimum frequency value of long words is less than the minimum frequency value of medium-frequency words, which is less than the minimum frequency value of high-frequency words.

[0047] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0048] Based on the above description, this application proposes a data processing method. Please refer to [link to relevant documentation]. Figure 3 The data processing method may include steps 301-305:

[0049] 301. Obtain the words to be processed from the target data.

[0050] The target data can be ordinary text data or system data; there are no restrictions. For example, the target data can be log data of type target, which can be JSON or XML type log data; there are no restrictions.

[0051] In one possible implementation, after acquiring the target data, each character of the target data is traversed, and delimiters are detected. The data between the two closest delimiters is used to obtain the segment to be processed. These delimiters can be quotation marks, parentheses, colons, commas, spaces, etc., and there are no restrictions here. For example, if a piece of JSON-type log data is "address" "Shenzhen,GuangDong", then by traversing this piece of JSON-type log data and detecting delimiters, the segment to be processed is obtained as "address", "Shenzhen", "GuangDong".

[0052] 302. Determine the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries based on the length of the segment to be processed.

[0053] The encoding dictionary includes one or more of the following: a high-frequency word dictionary, a medium-frequency word dictionary, and a long word dictionary. The encoding dictionary includes one or more dictionary-based word segments, each of which has a corresponding replacement code.

[0054] Optionally, the decision to enable an encoding dictionary, and which one or more encoding dictionaries to enable, can be made based on actual business conditions. For example, in actual business scenarios, log data rarely contains long words, so a long word dictionary can be disabled, and only the high-frequency word dictionary and the mid-frequency word dictionary can be enabled. Then, the target encoding dictionary to which the word to be processed belongs is determined from the high-frequency word dictionary and the mid-frequency word dictionary according to the length of the word to be processed. If only one encoding dictionary is enabled, it is only necessary to determine whether the word to be processed belongs to that encoding dictionary. If it does, then this single enabled encoding dictionary can be directly determined as the target encoding dictionary to which the word to be processed belongs.

[0055] In one possible implementation, assuming the enabled encoding dictionaries include a high-frequency word dictionary, a mid-frequency word dictionary, and a long word dictionary, the specific implementation of determining the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries based on the length of the segment to be processed can be as follows: If the length of the segment to be processed is greater than or equal to the minimum length of a long word, then the target encoding dictionary to which the segment to be processed belongs is the long word dictionary, and the minimum length of the long word can be defined by the user through the user terminal. If the length of the segment to be processed is less than the minimum length of a long word but greater than the minimum length of a high-frequency word, then it is determined whether the segment to be processed exists in the high-frequency word dictionary, and the minimum length of the high-frequency word can be defined by the user through the user terminal. If it exists in the high-frequency word dictionary, then the target encoding dictionary to which the segment to be processed belongs is the high-frequency word dictionary; if it does not exist in the high-frequency word dictionary, then it is determined whether the segment to be processed exists in the mid-frequency word dictionary. If it exists in the mid-frequency word dictionary, then the target encoding dictionary to which the segment to be processed belongs is the long word dictionary; if it does not exist in the mid-frequency word dictionary, then the segment to be processed does not belong to any target encoding dictionary. If the length of the word to be processed is less than the minimum length of the high-frequency word, then the word to be processed does not belong to the target encoding dictionary.

[0056] 303. Determine the replacement code corresponding to the word segment to be processed based on the target encoding dictionary.

[0057] In this context, a target encoding dictionary includes more than one dictionary segmentation. The replacement encoding refers to the replacement encoding corresponding to the dictionary segmentation in the encoding dictionary that replaces the segmentation to be processed when encoding the segmentation to be processed.

[0058] Specifically, when the segment to be processed exists in its corresponding target encoding dictionary, the target dictionary segment that is identical to the segment to be processed is queried, and the replacement code corresponding to the segment to be processed is determined according to the mapping relationship between the target dictionary segment and the replacement code. For example, if the target encoding dictionary is a high-frequency word dictionary, and the dictionary segment "userID" exists in this high-frequency word dictionary, and the replacement code corresponding to this dictionary segment is "1", when the segment to be processed "userID" is obtained from the target data, the dictionary segment that is identical to the segment to be processed is queried in the high-frequency word dictionary, and the replacement code corresponding to this segment to be processed is "1".

[0059] In one possible implementation, if the segment to be processed does not belong to a target encoding dictionary, then the segment to be processed does not have a corresponding replacement code. That is, during encoding conversion, the original segment to be processed is directly output. For example, if the segment to be processed is "AA", after step 302, it is detected that the segment to be processed does not belong to a target encoding dictionary, then the segment to be processed "AA" is directly output during encoding conversion.

[0060] 304. Replace the word segment to be processed in the target data with the replacement encoding to obtain the encoded data corresponding to the target data.

[0061] In one embodiment, when replacing the segment to be processed with replacement encoding, a marker character and the replacement encoding can be used to replace the segment. The marker character can be used to identify the target encoding dictionary when restoring data, and to identify the replacement encoding based on the encoding bit length corresponding to the target encoding dictionary. For the segment to be processed and the delimiter character that do not have a replacement encoding, the original string can be output directly.

[0062] In practice, the replacement method for any segmented word with a replacement code can be: marker character + replacement code, replacement code + marker character, or marker character + replacement code + marker character; there are no restrictions here. Within the same business scenario, different marker characters can be set for different types of encoding dictionaries: high-frequency word dictionaries correspond to high-frequency word marker characters, medium-frequency word dictionaries correspond to medium-frequency word marker characters, and long word dictionaries correspond to long word marker characters. Optionally, marker characters typically use invisible characters not used in the target data, such as #, &, $, etc. For example, in a data processing scenario, a high-frequency word dictionary and a mid-frequency word dictionary are enabled. The high-frequency word marker for the high-frequency word dictionary is "&", and the mid-frequency word marker for the mid-frequency word dictionary is "#". When a target data segment is obtained, which is "address" and "Shenzhen, GuangDong", the words to be processed, "address", "Shenzhen", and "GuangDong", can be detected. According to steps 303 and 304, it can be determined that "address" belongs to the high-frequency word dictionary and its corresponding replacement code is "2", "GuangDong" belongs to the mid-frequency word dictionary and its corresponding replacement code is "1234", and "Shenzhen" does not belong to any target encoding dictionary, so it has no corresponding replacement code. Finally, the corresponding encoded data can be obtained as "&2": "Shenzhen, #1234".

[0063] 305. Compress the encoded data to obtain a compression result for the target data.

[0064] The encoded target data can be generated into data blocks, which are then compressed using a common compression algorithm. Each data block can consist of multiple encoded data entries. This common compression algorithm can be Deflate, LZ4, ZSTD, etc., and is not limited here. Common compression algorithms typically compress small data blocks; for example, common block sizes are 16KB or 32KB. The principle of the compression algorithm is to match and find duplicate data within a window of a specific size within this small block, and then express the duplicate data in a specific form to achieve data compression. Please refer to [link to relevant documentation]. Figure 4 As shown, taking the compression of the string "AABCBBABC" as an example, the principle of the compression algorithm is explained. Each time a new character is input, the search starts to move backward. If a matching string is found, it is expressed by two numbers (Distance, Length). Distance indicates the position where the matching string was found, and Length indicates how many characters are matched.

[0065] In this embodiment, the target encoding dictionary to which the segment to be processed belongs can be determined from one or more encoding dictionaries based on the length of the segment to be processed. The replacement encoding corresponding to the segment to be processed is then determined based on the target encoding dictionary, allowing for the use of different encoding schemes to replace data from different dictionaries. The encoded data is then compressed to obtain a compression result specific to the target data, thereby improving the data compression rate. Therefore, by determining the replacement encoding corresponding to the data from different types of encoding dictionaries, data encoding becomes more targeted, contributing to a higher data compression rate.

[0066] Please see Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention. The data processing method may include the following steps 501-506:

[0067] 501. Obtain sample data and perform word segmentation on the sample data to obtain sample word segments corresponding to the sample data.

[0068] The sample data can be obtained from historical data. Tokenization of the sample data can be performed at the positions of delimiters such as spaces, commas, quotation marks, and colons. For example, taking log data as an example, consider historical log data of type JSON: "address": "Shenzhen, GuangDong". First, based on a simple tokenization algorithm, the word information constituting the JSON is extracted. This word information includes each property key and its corresponding property value. If spaces or commas are detected in the property values, further tokenization is required. After tokenization, the above JSON results in three sample tokens: "address", "Shenzhen", and "GuangDong".

[0069] 502. Determine the target candidate word list to which the sample word belongs based on the length of the sample word segmentation.

[0070] The target candidate word list includes either a mid-frequency word candidate word list or a long word candidate word list. The target candidate word list includes sample word segments and their corresponding word frequency information. The length of a sample word segment refers to the number of characters in the sample word segment; for example, the length of the sample word "address" is 6. In this application example, the corresponding encoding dictionaries generated based on the mid-frequency word candidate word list are a high-frequency word dictionary and a mid-frequency word dictionary, and the corresponding encoding dictionary generated based on the long word candidate word list is a long word dictionary.

[0071] In one possible implementation, the target candidate word list to which the sample word belongs is determined based on the length of the sample word segment. Specifically, when the length of the sample word segment is greater than the minimum length of a high-frequency word, the sample word segment is added to the mid-frequency word candidate word list, meaning the target candidate word list to which the sample word belongs is the mid-frequency word candidate word list. When the length of the sample word segment is greater than or equal to the minimum length of a high-frequency word, it is determined whether the sample word segment exists in the long word dictionary. If it does not exist, the sample word segment is added to the long word candidate word list, meaning the target candidate word list to which the sample word belongs is the long word candidate word list. If it exists, the sample word segment is directly ignored. When the length of the sample word segment is less than the minimum length of the mid-frequency word candidate word list, the sample word segment is directly ignored.

[0072] 503. Update the word frequency information of the sample word segmentation in the target candidate word list, and generate the corresponding encoding dictionary according to the target candidate word list.

[0073] The target candidate word list includes more than one dictionary-based word segmentation in its corresponding encoding dictionary. Updating the word frequency information of sample word segments in the target candidate word list involves: if a sample word segment is appearing for the first time, it is added to the target candidate word list, and its word frequency information is recorded as 1; if a sample word segment already exists in the target candidate word list, its word frequency information is incremented by 1. For example, when the sample word segment "AAA" is written for the first time, it is added to the target candidate word list, and its word frequency information is recorded as 1. When "AAA" is written again, its existence in the target candidate word list is detected, and its word frequency information is recorded as 2, and so on. This process will not be elaborated further here.

[0074] Optionally, the corresponding encoding dictionary generated based on the mid-frequency word candidate word list is a high-frequency word dictionary and a mid-frequency word dictionary. The high-frequency word dictionary includes one or more dictionaries that are segmented as high-frequency words, and the mid-frequency word dictionary includes one or more dictionaries that are segmented as mid-frequency words. The corresponding encoding dictionary generated based on the long word candidate word list is a long word dictionary, and the long word dictionary includes one or more dictionaries that are segmented as long words.

[0075] In one possible implementation, when the target candidate word list is a mid-frequency word candidate word list, the sample sequence number corresponding to the sample data is obtained. If the sample sequence number is detected to be less than the target threshold, the word frequency information of the sample word segmentation is updated in the target candidate word list. If the sample sequence number is detected to be equal to the target threshold, the update of the word frequency information of the sample word segmentation in the target candidate word list is stopped, and a high-frequency word dictionary and a mid-frequency word dictionary are generated according to the mid-frequency word candidate word list. Here, the sample sequence number refers to the document sequence number of the sample data. The documents of the sample data can be numbered beforehand, or the document number of the sample data can be marked with a sequence number when the sample data is written; there is no restriction here. The target threshold refers to the maximum value of the sample sequence number allowed to update the target candidate word list, set by the user through the user terminal. When the sample sequence number equals this target threshold, the update of the mid-frequency word candidate word list is stopped, and no further updates are performed even if sample data with a sample sequence number greater than the target threshold is written. Then, the word frequency information of each sample word segment in the mid-frequency word candidate list that has stopped updating is obtained. Sample word segments in the mid-frequency word candidate list whose word frequency information meets the high-frequency word condition are identified as high-frequency words, and a high-frequency word dictionary is generated based on these high-frequency words. Sample word segments in the mid-frequency word candidate list that, excluding high-frequency words, meet the mid-frequency word condition are identified as mid-frequency words, and a mid-frequency word dictionary is generated based on these mid-frequency words. The high-frequency word condition can be that the word frequency information is greater than the minimum frequency value of high-frequency words, and that the sample word segment has the largest target number of word frequency information. The minimum frequency value and target number of high-frequency words can be defined by the user through the user terminal. The mid-frequency word condition can be that the length of the sample word segment is greater than or equal to the minimum length of mid-frequency words, and that the word frequency information is greater than or equal to the minimum frequency value of mid-frequency words. The minimum length and minimum frequency value of mid-frequency words can be defined by the user through the user terminal.

[0076] For example, if the target threshold for the sample sequence number in the current scenario is 10000, when the sample sequence number of the detected sample data is 10000, the updating of the sample word segmentation and its word frequency information in the mid-frequency word candidate list is stopped. The sample word segmentation in the stopped mid-frequency word candidate list is then sorted according to its word frequency information, so that the sample word segmentation with higher frequency is placed at the top of the mid-frequency word candidate list. Therefore, the mid-frequency word candidate list contains sample word segmentation and its corresponding frequency information: “AAAAA”-5000, “BBB”-4500, “CCC”-4000, “DD”. -3500, "EEEE"-3000, "F"-2500, "MM"-2000. If the target number of the high-frequency word dictionary is 4, the minimum frequency value of the high-frequency word is 4500, the minimum length of the mid-frequency word is 2, and the minimum frequency value of the mid-frequency word is 1500, then we can obtain the high-frequency words: "AAAAA" and "BBB", and the mid-frequency words: "CCC", "DD", "EEEE", and "MM". Furthermore, based on the high-frequency and mid-frequency words, we can find that the high-frequency word dictionary contains the high-frequency words "AAAAA" and "BBB", and the mid-frequency word dictionary contains the mid-frequency words "CCC", "DD", and "EEEE".

[0077] In one possible implementation, when the target candidate word list is a long word candidate word list, the word frequency information of each sample word segment in the target candidate word list is obtained; based on the word frequency information, the sample word segments in the target candidate word list whose word frequency meets the long word condition are determined as long words; and a long word dictionary is generated based on the long words. This long word condition may refer to the word frequency information being greater than the minimum word frequency for long words, which can be defined by the user through the user terminal. Specifically, when the word frequency information of a sample word segment in the long word candidate word list is detected to be greater than or equal to the minimum frequency value, the sample word segment is determined as a long word, included in the long word dictionary, and deleted from the long word candidate word list. For example, in a data processing scenario, the minimum frequency of a long word is 4. When the sample data "NNNNNNNNNN" is written, it is detected that the sample word segment does not exist in the current long word dictionary, but it exists in the long word candidate list. Furthermore, the word frequency information of the sample word segment is 3. Therefore, the word frequency information of the sample word segment is incremented by 1, so that the word frequency information of "NNNNNNNNNN" in the long word candidate list is 4. Since 4 is greater than or equal to the minimum frequency of a long word, "NNNNNNNNNN" is a long word. "NNNNNNNNNN" is added to the long word dictionary, resulting in the long word "NNNNNNNNNN" existing in the long word dictionary.

[0078] Furthermore, since the generation of the long word dictionary is not affected by the sample sequence number, the long word dictionary is always in an updated state. If the encoding dictionary is always in an updated state, it will affect the encoding efficiency of the words to be processed. Therefore, a freeze operation can be performed on the long word dictionary to stop its updates. Specifically, the long word dictionary can be frozen by the user sending a freeze command based on the actual business situation. For example, based on the characteristics of the business data, system maintenance personnel can determine that the frequency of long words is very low and can actively send a dictionary freeze operation command to trigger the long word dictionary freeze operation. Alternatively, the long word dictionary can be frozen when the long word dictionary is generated. The data processing server can determine whether to freeze the long word dictionary based on the number of documents in the sample data and the number of newly added long words. For example, if the data processing server detects that the number of documents in the normally acquired sample data is 10,000, it can add 100 long words. If the number of newly added long words is only a few, it can be understood that the long word dictionary has stabilized. At this time, the data processing server can directly freeze the long word dictionary and stop its updates.

[0079] Furthermore, if the compression rate of newly written target data decreases recently, or the long word conversion rate decreases during encoding conversion, a long word dictionary recovery operation can be triggered. This involves updating the sample word segmentation of the long word candidate list and generating the long word dictionary based on the candidate list. The long word conversion rate can be determined based on the number of newly written target data documents and the number of encoding conversions performed on long words in actual business operations. For example, in 10,000 newly written target data documents, normally 20,000 encoding conversions can be performed on long words. If the number drops below 10,000, the long word dictionary is considered to need updating, triggering an update operation.

[0080] 504. Set the replacement codes for each dictionary word segmentation in the corresponding encoding dictionary according to the dictionary encoding rules.

[0081] This dictionary encoding rule is used to indicate: a first-digit replacement code is set for each high-frequency word in the high-frequency word dictionary; a second-digit replacement code is set for each medium-frequency word in the medium-frequency word dictionary; and a third-digit replacement code is set for each long word in the long word dictionary. The first-digit replacement code is less than the second-digit replacement code, and the second-digit replacement code is less than the third-digit replacement code.

[0082] In one possible implementation, a replacement code of a first encoding bit length is assigned to the high-frequency words in the high-frequency word dictionary. This replacement code can be randomly generated or generated sequentially according to the order of word frequency information. For example, based on the existing number of high-frequency words, the first encoding bit length can be determined to be 8 bits, i.e., 1 byte. Then, the replacement codes corresponding to the high-frequency words in the high-frequency word dictionary can be generated between 0x00 and 0xFF.

[0083] In one possible implementation, a replacement code with a second encoding bit length is assigned to the mid-frequency words in the mid-frequency word dictionary. This replacement code can be a hash value of the second encoding bit length, or it can be an integer (Int or VInt) value generated sequentially according to the order of word frequency information.

[0084] In one possible implementation, a replacement code with a third bit length is assigned to long words in the long word dictionary. This replacement code can be a hash value with a third bit length, or it can be a non-hash value generated sequentially according to the order of word frequency information, such as a long integer (Long or VLong) value.

[0085] Furthermore, the dictionary data can be stored, which may include an encoding dictionary and the replacement codes corresponding to each word segmentation in the encoding dictionary. Specifically, the dictionary data can be stored in a file containing the target data, or it can be stored in a separate key-value database or other databases; there are no restrictions here. Obviously, since the first encoding bit length is less than the second encoding bit length, and the second encoding bit length is less than the third encoding bit length, high-frequency word dictionary data generally requires less storage space than mid-frequency word dictionary data, and mid-frequency word dictionary data requires less storage space than long word dictionary data.

[0086] This example uses RocksDB as a key-value database to store the dictionary data. Since the high-frequency word data in the high-frequency word dictionary is limited and controllable, it only needs to be stored as a single key-value pair, such as: Key: _HIGH_RREQS_DICTS_, Value: {<high-frequency word 1, replacement code 1>, <high-frequency word 2, replacement code 2>, <high-frequency word 3, replacement code 3>}…}. Similarly, the mid-frequency word dictionary data is also limited and controllable, so it can be stored as a single key-value pair, such as: Key: _MEDI UM_RREQS_DICTS_, Value: {<Mid-frequency word 1, replacement code 4>, <Mid-frequency word 2, replacement code 5>, <Mid-frequency word 3, replacement code 6>…}; Since a long word often contains many characters, it requires more storage space than a mid-to-high frequency dictionary. Therefore, when using the hash value of the third encoding bit to generate the replacement code, the corresponding hash value replacement code can be obtained by hashing the long word. Each long word and its corresponding encoding result are stored as a separate KeyValue, such as: Key: Hash replacement code 1; Value: Long word 1. However, theoretically, there is a probability of collision when storing long word dictionary data using this method. Considering that log data is not sensitive business data, this extremely low collision probability is usually tolerable. When using non-hash values ​​to generate replacement codes, since it is impossible to deduce the corresponding replacement codes from long words, it is necessary to store the bidirectional mapping relationship between long words and replacement codes. Each long word and its corresponding encoding result are stored as two KeyValue pairs. For example: KeyValue1 is used to store the mapping relationship from replacement code to long word; Key: C_Replacement code 7; Value: Long word 2; KeyValue2 is used to store the mapping relationship from long word to encoding result; Key: T_Long word 3; Value: Replacement code 8.

[0087] Optionally, the high-frequency word dictionary and the mid-frequency word dictionary can be loaded from the stored disk into memory for storage, so as to facilitate faster calling of the encoding dictionary for encoding conversion.

[0088] 505. Obtain the word segment to be processed from the target data, and determine the target encoding dictionary to which the word segment to be processed belongs from one or more encoding dictionaries according to the length of the word segment to be processed.

[0089] This step is performed in accordance with steps 301 and 302, and will not be repeated here.

[0090] 506. Determine the replacement code corresponding to the word segment to be processed according to the target encoding dictionary, replace the word segment to be processed in the target data with the replacement code, obtain the encoded data corresponding to the target data, and compress the encoded data to obtain the compression result for the target data.

[0091] This step is performed in accordance with steps 303-305, and will not be repeated here.

[0092] In one possible implementation, the compressed result of the target data can be restored to obtain the original target data. The compressed result of the target data is then decompressed to obtain encoded data; the marker characters in the encoded data are obtained, along with the data corresponding to the number of bits in the encoding after the marker characters, to obtain the replacement code; the target encoding dictionary to which the replacement code belongs is determined based on the marker characters; the target word segmentation corresponding to the replacement code is determined based on the replacement code and the target encoding dictionary; and the marker characters and replacement codes in the target word segmentation encoding data are replaced to obtain the original target data. For example, after decompressing the compressed result, a piece of encoded data is obtained: "&2": "Shenzhen, #1234". The marker characters "&" and "#" in this encoded data can be obtained. Based on the pre-defined correspondence between the encoding dictionary and the marker characters, "&" represents a high-frequency word marker, and "#" represents a mid-frequency word marker. The encoding length for high-frequency words is 1 byte, and for mid-frequency words it is 4 bytes. Therefore, the replacement codes "2" and "1234" can be obtained. By querying the high-frequency and mid-frequency word dictionaries, the corresponding word segmentation can be found. The word segmentation corresponding to the replacement code "2" is "address", and the word segmentation corresponding to the replacement code "1234" is "GuangDong". Thus, the original target data is: "address": "Shenzhen, GuangDong". The pre-defined correspondence between the encoding dictionary and the marker characters can be pre-configured by the user through their terminal. For example, assuming this embodiment of the application is... Figure 1 The data processing server in the data processing system shown executes the configuration. Users can configure the correspondence between the encoding dictionary and the marked characters through their own user terminals, and after the configuration is completed, the correspondence between the encoding dictionary and the marked characters is stored in the data processing server.

[0093] In one possible implementation, when writing target data, if it is detected that the above three types of encoding dictionaries are already enabled, but no usable dictionary data is detected, then the target data is used as sample data to execute steps 501-504 to obtain a complete encoding dictionary. The newly written target data is then encoded and replaced to obtain encoded data, which is then compressed to obtain a compressed result for the newly written target data. This complete encoding dictionary refers to the mid-frequency word dictionary and high-frequency word dictionary obtained after the sample number of the sample data reaches the target threshold, as well as the long word dictionary after a freeze operation is performed on the long word dictionary.

[0094] This section uses a JSON log data example to illustrate in detail the data processing process performed by the server using this solution. Please refer to [link / reference]. Figure 6a The data processing server retrieves a log data document, which contains a JSON-type target data such as... Figure 6a As shown, then the unprocessed word segments in the target data are obtained, and the encoding dictionary and replacement encoding corresponding to each unprocessed word are obtained based on the unprocessed word segments. Figure 6b As shown, in this scenario, the user sets the first encoding bit length for high-frequency words to 8 bits (1 byte), the second encoding bit length for mid-frequency words to 32 bits (4 bytes), and the third encoding bit length for long words to 8 bits (64 bytes) via the user terminal, and sets the marker characters for different types of encoding dictionaries, such as... Figure 6c As shown, then according to... Figure 6a , 6b Encoding the target data yields encoded data such as... Figure 6d As shown.

[0095] In this embodiment, sample data can be segmented to obtain sample words; a list of target candidate words to which the sample words belong can be determined based on their length; the word frequency information of the sample words can be updated in the target candidate word list; a corresponding encoding dictionary can be generated based on the target candidate word list; and replacement codes corresponding to each dictionary word can be set in the corresponding encoding dictionary according to the dictionary encoding rules, so as to better process data of different lengths or word frequencies. Therefore, by constructing encoding dictionaries of different categories, data encoding can be made more targeted, which helps to improve the data compression rate.

[0096] Based on the description of the above data processing method embodiments, this application also discloses a data processing apparatus, which can be configured in the aforementioned electronic device. For example, the apparatus can be a computer program (including program code) running in the electronic device. This data processing apparatus can execute... Figure 3 , Figure 5 The method shown. Please refer to [link / reference]. Figure 7 The data processing device can operate the following modules:

[0097] The acquisition module 701 is used to acquire the words to be processed from the target data;

[0098] Processing module 702 is used to determine the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries according to the length of the segment to be processed;

[0099] The processing module 702 is further configured to determine the replacement code corresponding to the word segment to be processed based on the target encoding dictionary;

[0100] The processing module 702 is further configured to replace the word segment to be processed in the target data with the replacement encoding to obtain the encoded data corresponding to the target data;

[0101] The processing module 702 is further configured to compress the encoded data to obtain a compression result for the target data.

[0102] In one implementation, before determining the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries based on the length of the segment to be processed, the acquisition module 701 is further configured to acquire sample data and perform segmentation processing on the sample data to obtain sample segments corresponding to the sample data; the processing module 702 is further configured to determine the target candidate word list to which the sample segment belongs based on the length of the sample segment, the target candidate word list including: a mid-frequency word candidate word list or a long word candidate word list; update the word frequency information of the sample segment in the target candidate word list; generate a corresponding encoding dictionary based on the target candidate word list, the corresponding encoding dictionary including one or more dictionary segmentations; and set the replacement code corresponding to each dictionary segmentation in the corresponding encoding dictionary according to the dictionary encoding rules.

[0103] In one embodiment, the one or more encoding dictionaries include any one or more of the following: a high-frequency word dictionary, a mid-frequency word dictionary, and a long word dictionary. If the target candidate word list is the mid-frequency word candidate word list, then the corresponding encoding dictionary includes a high-frequency word dictionary and a mid-frequency word dictionary, wherein the high-frequency word dictionary includes one or more high-frequency words and the mid-frequency word dictionary includes one or more mid-frequency words. If the target candidate word list is the long word candidate word list, then the corresponding encoding dictionary includes a long word dictionary, wherein the long word dictionary includes one or more long words.

[0104] In one embodiment, the target candidate word list includes the mid-frequency candidate word list, and the acquisition module 701 is further configured to acquire the sample sequence number corresponding to the sample data; the processing module 702 is further configured to update the word frequency information of the sample word segment in the target candidate word list if the sample sequence number is detected to be less than the target threshold; if the sample sequence number is detected to be equal to the target threshold, stop updating the word frequency information of the sample word segment in the target candidate word list, and generate a high-frequency word dictionary and a mid-frequency word dictionary according to the target candidate word list.

[0105] In one embodiment, the acquisition module 701 is further configured to acquire word frequency information of each sample word segment in the target candidate word list; the processing module 702 is further configured to determine the sample word segments in the target candidate word list that meet the high-frequency word condition as high-frequency words according to the word frequency information, and generate a high-frequency word dictionary based on the high-frequency words; and to determine the sample word segments in the target candidate word list that meet the mid-frequency word condition, excluding the high-frequency words, as mid-frequency words according to the word frequency information, and generate a mid-frequency word dictionary based on the mid-frequency words.

[0106] In one embodiment, the target candidate word list includes a long word candidate word list, and the acquisition module 701 is further configured to acquire the word frequency information of each sample word segment in the target candidate word list; the processing module 702 is further configured to determine the sample word segments in the target candidate word list whose word frequency meets the long word condition as long words according to the word frequency information; and generate a long word dictionary according to the long words.

[0107] In one embodiment, the dictionary encoding rule is used to instruct: to set a replacement code of a first encoding bit length for each high-frequency word in the high-frequency word dictionary, to set a replacement code of a second encoding bit length for each medium-frequency word in the medium-frequency word dictionary, and to set a replacement code of a third encoding bit length for each long word in the long word dictionary, wherein the first encoding bit length is less than the second encoding bit length, and the second encoding bit length is less than the third encoding bit length.

[0108] In the various embodiments of this application, the functional modules can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules, and this application does not impose any limitations on this.

[0109] In this embodiment, the target encoding dictionary to which the segment to be processed belongs can be determined from one or more encoding dictionaries based on the length of the segment to be processed. The replacement encoding corresponding to the segment to be processed is then determined based on the target encoding dictionary, allowing for the use of different encoding schemes to replace data from different dictionaries. The encoded data is then compressed to obtain a compression result specific to the target data, thereby improving the data compression rate. Therefore, determining the replacement encoding corresponding to the data from different types of encoding dictionaries makes data encoding more targeted and helps improve the data compression rate.

[0110] Please see again Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. The electronic device of this embodiment includes a power supply module and other structures, and includes a processor 801 and a storage device 802. Optionally, the electronic device may also include a network interface 803. The processor 801, storage device 802, and network interface 803 can exchange data. The network interface 803 is controlled by the processor to send and receive information. The storage device 802 stores a computer program, which includes program instructions. The processor 801 executes the program instructions stored in the storage device 802. The processor 801 is configured to call the program instructions to execute the above-described method.

[0111] The storage device 802 may include volatile memory, such as random-access memory (RAM); the storage device 802 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the storage device 802 may also include a combination of the above types of memory.

[0112] The processor 801 may be a central processing unit (CPU). In one embodiment, the processor 801 may also be a graphics processing unit (GPU). The processor 801 may also be a combination of a CPU and a GPU.

[0113] In one embodiment, the storage device 802 is used to store program instructions. The processor 801 can invoke the program instructions to execute the following steps:

[0114] Obtain the words to be segmented from the target data;

[0115] The target encoding dictionary to which the segment to be processed belongs is determined from one or more encoding dictionaries based on the length of the segment to be processed.

[0116] The replacement code corresponding to the word segment to be processed is determined according to the target encoding dictionary;

[0117] The word segment to be processed is replaced in the target data with the replacement encoding to obtain the encoded data corresponding to the target data;

[0118] The encoded data is compressed to obtain a compressed result for the target data.

[0119] In one embodiment, before determining the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries based on the length of the segment to be processed, the processor 801 may further be configured to perform: acquiring sample data and performing segmentation processing on the sample data to obtain sample segments corresponding to the sample data; determining the target candidate word list to which the sample segment belongs based on the length of the sample segment, the target candidate word list including: a mid-frequency word candidate word list or a long word candidate word list; updating the word frequency information of the sample segment in the target candidate word list; generating a corresponding encoding dictionary based on the target candidate word list, the corresponding encoding dictionary including one or more dictionary segmentations; and setting the replacement code corresponding to each dictionary segmentation in the corresponding encoding dictionary according to the dictionary encoding rules.

[0120] In one embodiment, the one or more encoding dictionaries include any one or more of the following: a high-frequency word dictionary, a mid-frequency word dictionary, and a long word dictionary. If the target candidate word list is the mid-frequency word candidate word list, then the corresponding encoding dictionary includes a high-frequency word dictionary and a mid-frequency word dictionary, wherein the high-frequency word dictionary includes one or more high-frequency words and the mid-frequency word dictionary includes one or more mid-frequency words. If the target candidate word list is the long word candidate word list, then the corresponding encoding dictionary includes a long word dictionary, wherein the long word dictionary includes one or more long words.

[0121] In one embodiment, the target candidate word list includes the mid-frequency candidate word list, and the processor 801 can also be used to perform: obtaining the sample number corresponding to the sample data; if the sample number is detected to be less than a target threshold, updating the word frequency information of the sample word segment in the target candidate word list; if the sample number is detected to be equal to the target threshold, stopping the updating of the word frequency information of the sample word segment in the target candidate word list, and generating a high-frequency word dictionary and a mid-frequency word dictionary according to the target candidate word list.

[0122] In one embodiment, the processor 801 can also be used to perform: obtaining word frequency information of each sample word segment in the target candidate word list; determining the sample word segments in the target candidate word list that meet the high-frequency word condition according to the word frequency information as high-frequency words, and generating a high-frequency word dictionary based on the high-frequency words; determining the sample word segments in the target candidate word list that meet the mid-frequency word condition, excluding the high-frequency words, according to the word frequency information, as mid-frequency words, and generating a mid-frequency word dictionary based on the mid-frequency words.

[0123] In one embodiment, the target candidate word list includes a long word candidate word list, and the processor 801 can also be used to perform: obtaining word frequency information of each sample word segment in the target candidate word list; determining the sample word segments in the target candidate word list whose word frequency meets the long word condition as long words according to the word frequency information; and generating a long word dictionary according to the long words.

[0124] In one embodiment, the dictionary encoding rule is used to indicate the replacement encoding of each high-frequency word in the high-frequency word dictionary with a first encoding bit length, the replacement encoding of each medium-frequency word in the medium-frequency word dictionary with a second encoding bit length, and the replacement encoding of each long word in the long word dictionary with a third encoding bit length, wherein the first encoding bit length is less than the second encoding bit length, and the second encoding bit length is less than the third encoding bit length.

[0125] In specific implementations, the devices, processor 801, storage device 802, etc., described in the embodiments of this application can execute the implementation methods described in the above method embodiments, or they can execute the implementation methods described in the embodiments of this application, which will not be repeated here.

[0126] In this embodiment, the target encoding dictionary to which the segment to be processed belongs can be determined from one or more encoding dictionaries based on the length of the segment to be processed. The replacement encoding corresponding to the segment to be processed is then determined based on the target encoding dictionary, allowing for the use of different encoding schemes to replace data from different dictionaries. The encoded data is then compressed to obtain a compression result specific to the target data, thereby improving the data compression rate. Therefore, by determining the replacement encoding corresponding to the data from different types of encoding dictionaries, data encoding becomes more targeted, contributing to a higher data compression rate.

[0127] This application also provides a computer (readable) storage medium storing a computer program. The computer program includes program instructions, which, when executed by a processor, can perform some or all of the steps described in the above method embodiments. Optionally, the computer storage medium can be volatile or non-volatile.

[0128] This application also provides a computer program product or computer program, which includes program instructions that can be stored in a computer-readable storage medium. The processor of a computer device reads the program instructions from the computer-readable storage medium and executes the program instructions, causing the computer to perform some or all of the steps in the data display method described above, which will not be elaborated further here.

[0129] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer storage medium, which can be a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0130] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent changes made in accordance with the claims of this application, still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the words to be segmented from the target data; The target encoding dictionary to which the word to be processed belongs is determined from one or more encoding dictionaries based on the length of the word to be processed. These one or more encoding dictionaries include at least one of a high-frequency word dictionary, a mid-frequency word dictionary, and a long word dictionary. The high-frequency word dictionary includes high-frequency words, the mid-frequency word dictionary includes mid-frequency words, and the long word dictionary includes long words. The minimum length of a high-frequency word is less than the minimum length of a mid-frequency word, and the minimum length of a mid-frequency word is less than the minimum length of a long word. The minimum frequency value of a long word is less than the minimum frequency value of a mid-frequency word, and the minimum frequency value of a mid-frequency word is less than the minimum frequency value of a high-frequency word. Different types of encoding dictionaries correspond to different encoding schemes. The replacement code corresponding to the word segment to be processed is determined according to the target encoding dictionary; The word segment to be processed is replaced in the target data with the replacement encoding to obtain the encoded data corresponding to the target data; The encoded data is compressed to obtain a compressed result for the target data; The step of determining the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries based on the length of the segment to be processed includes: If the length of the word to be processed is greater than or equal to the minimum length of a long word, then the target encoding dictionary to which the word to be processed belongs is a long word dictionary; If the length of the word segment to be processed is less than the minimum length of a long word but greater than the minimum length of a high-frequency word, then it is determined whether the word segment to be processed exists in the high-frequency word dictionary. If it exists in the high-frequency word dictionary, then the target encoding dictionary to which the word segment to be processed belongs is the high-frequency word dictionary. If it does not exist in the high-frequency word dictionary, then it is determined whether the word segment to be processed exists in the mid-frequency word dictionary. If it exists in the mid-frequency word dictionary, then the target encoding dictionary to which the word segment to be processed belongs is the mid-frequency word dictionary. If it does not exist in the mid-frequency word dictionary, then the word segment to be processed does not belong to any target encoding dictionary. If the length of the word to be processed is less than the minimum length of a high-frequency word, then the word to be processed does not belong to a target encoding dictionary.

2. The method according to claim 1, characterized in that, Before determining the target encoding dictionary to which the segment to be processed belongs from one or more encoding dictionaries based on the length of the segment to be processed, the method further includes: Obtain sample data and perform word segmentation on the sample data to obtain sample word segments corresponding to the sample data; The target candidate word list to which the sample word belongs is determined based on the length of the sample word segmentation. The target candidate word list includes: a mid-frequency word candidate word list or a long word candidate word list. The word frequency information of the sample word segmentation is updated in the target candidate word list, and a corresponding encoding dictionary is generated according to the target candidate word list. The corresponding encoding dictionary includes more than one dictionary word segmentation. According to the dictionary encoding rules, set the replacement codes for each dictionary word segmentation in the corresponding encoding dictionary.

3. The method according to claim 2, characterized in that, The method further includes: If the target candidate word list is the mid-frequency word candidate word list, then the corresponding encoding dictionary includes a high-frequency word dictionary and a mid-frequency word dictionary, wherein the high-frequency word dictionary includes one or more high-frequency words and the mid-frequency word dictionary includes one or more mid-frequency words; If the target candidate word list is the long word candidate word list, then the corresponding encoding dictionary includes a long word dictionary, and the long word dictionary includes more than one long word.

4. The method according to claim 2, characterized in that, The target candidate word list includes the mid-frequency candidate word list. Updating the word frequency information of the sample word segmentation in the target candidate word list and generating a corresponding encoding dictionary based on the target candidate word list includes: Obtain the sample number corresponding to the sample data; If the sample number is detected to be less than the target threshold, the word frequency information of the sample word segmentation is updated in the target candidate word list; If the sample number is detected to be equal to the target threshold, then the word frequency information of the sample word segmentation in the target candidate word list is stopped from being updated, and a high-frequency word dictionary and a mid-frequency word dictionary are generated according to the target candidate word list.

5. The method according to claim 4, characterized in that, The process of generating a high-frequency word dictionary and a mid-frequency word dictionary based on the target candidate word list includes: Obtain the word frequency information of each sample word segmentation in the target candidate word list; Based on the word frequency information, the samples whose word frequencies meet the high-frequency word condition in the target candidate word list are segmented and determined as high-frequency words, and a high-frequency word dictionary is generated based on the high-frequency words; Based on the word frequency information, the sample words in the target candidate word list that meet the mid-frequency word condition, excluding the high-frequency words, are identified as mid-frequency words, and a mid-frequency word dictionary is generated based on the mid-frequency words.

6. The method according to claim 2, characterized in that, The target candidate word list includes a long word candidate word list, and the step of generating a corresponding encoding dictionary based on the target candidate word list includes: Obtain the word frequency information of each sample word segmentation in the target candidate word list; Based on the word frequency information, the word segments of the target candidate word list that meet the long word condition are determined as long words; Generate a dictionary of long words based on the given long words.

7. The method according to claim 2, characterized in that, The dictionary encoding rules are used to indicate: to set a replacement code of a first encoding bit length for each high-frequency word in the high-frequency word dictionary, to set a replacement code of a second encoding bit length for each medium-frequency word in the medium-frequency word dictionary, and to set a replacement code of a third encoding bit length for each long word in the long word dictionary, wherein the first encoding bit length is less than the second encoding bit length, and the second encoding bit length is less than the third encoding bit length.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to obtain the words to be processed from the target data; The processing module is used to determine the target encoding dictionary to which the word to be processed belongs from one or more encoding dictionaries according to the length of the word to be processed. The one or more encoding dictionaries include at least one of a high-frequency word dictionary, a mid-frequency word dictionary, and a long word dictionary. The high-frequency word dictionary includes high-frequency words, the mid-frequency word dictionary includes mid-frequency words, and the long word dictionary includes long words. The minimum length of the high-frequency words is less than the minimum length of the mid-frequency words, and the minimum length of the mid-frequency words is less than the minimum length of the long words. The minimum frequency value of the long words is less than the minimum frequency value of the mid-frequency words, and the minimum frequency value of the mid-frequency words is less than the minimum frequency value of the high-frequency words. Different types of encoding dictionaries correspond to different encoding schemes. The processing module is further configured to determine the replacement code corresponding to the word segment to be processed based on the target encoding dictionary; The processing module is further configured to replace the word segment to be processed in the target data with the replacement encoding to obtain the encoded data corresponding to the target data; The processing module is further configured to compress the encoded data to obtain a compression result for the target data; The processing module, when determining the target encoding dictionary to which the word to be processed belongs from one or more encoding dictionaries based on the length of the word to be processed, is specifically used for: If the length of the word to be processed is greater than or equal to the minimum length of a long word, then the target encoding dictionary to which the word to be processed belongs is a long word dictionary; If the length of the word segment to be processed is less than the minimum length of a long word but greater than the minimum length of a high-frequency word, then it is determined whether the word segment to be processed exists in the high-frequency word dictionary. If it exists in the high-frequency word dictionary, then the target encoding dictionary to which the word segment to be processed belongs is the high-frequency word dictionary. If it does not exist in the high-frequency word dictionary, then it is determined whether the word segment to be processed exists in the mid-frequency word dictionary. If it exists in the mid-frequency word dictionary, then the target encoding dictionary to which the word segment to be processed belongs is the mid-frequency word dictionary. If it does not exist in the mid-frequency word dictionary, then the word segment to be processed does not belong to any target encoding dictionary. If the length of the word to be processed is less than the minimum length of a high-frequency word, then the word to be processed does not belong to a target encoding dictionary.

9. An electronic device, characterized in that, The method includes a processor and a storage device, the processor being interconnected with the storage device, wherein the storage device is used to store computer program instructions, and the processor is configured to execute the program instructions to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, are used to perform the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes program instructions stored in a computer-readable storage medium, which, when read and executed by a processor of a computer device, cause the computer device to perform the method as described in any one of claims 1-7.

Images