Long text processing method, device, equipment and medium based on classification model

Through the long text processing method based on the classification model, the compression rate and decompression speed of long text are improved by using preprocessing and multi-stage compression algorithms, solving the problems of low compression efficiency and high resource consumption in the existing technology, and ensuring the completeness of data Sex and safety.

CN118568569BActive Publication Date: 2024-11-26HANGZHOU HUACE FILM & TELEVISION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411001770.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-26
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing long text compression technology has problems such as low compression efficiency, long compression time and large resource consumption, which cannot meet the requirements of fast storage and transmission, and the integrity and readability of the compressed text are difficult to guarantee.

Method used

Using a long text processing method based on the classification model, text features are obtained through pre-processing, and the target compression algorithm combination is used to predict the text with multiple levels of compression processing, and multi-stage decompression recovery is carried out when needed to ensure the compressed data Quick decompression and recovery.

Benefits of technology

It improves the compression rate of long text, reduces the compression cost, ensures the integrity and readability of compressed text, realizes rapid decompression and recovery, improves data transmission speed and storage efficiency, and enhances data security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118568569B_ABST
    Figure CN118568569B_ABST
Patent Text Reader

Abstract

The present application relates to a long text processing method, device, equipment and medium based on a classification model, wherein the method includes: obtaining long text data to be compressed, and preprocessing the long text data to be compressed to obtain text features and preprocessed text; performing algorithm prediction on text features through a pre-trained classification model to obtain a target compression algorithm combination; performing data encoding on the preprocessed text to obtain an encoded text; performing multi-level compression processing on the encoded text based on the target compression algorithm combination to obtain target compressed data; if a data recovery instruction is received, performing multi-level decompression and data recovery processing on the target compressed data to obtain target recovered data. The present application improves the compression rate of long texts, reduces compression costs, ensures the integrity and readability of compressed texts, and realizes rapid decompression and recovery of compressed data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a long text processing method, device, equipment and medium based on a classification model. Background Art

[0002] Text compression refers to using fewer bits or bytes to represent text, which can significantly reduce the space required to store text in a computer. In the current era of big data, the importance of long text compression technology has become increasingly prominent. Traditional text compression schemes have problems such as low compression efficiency, long compression time, and high resource consumption, and cannot meet the requirements of fast storage and transmission. How to improve the compression rate of long texts, reduce compression costs, ensure the integrity and readability of compressed texts, and achieve fast decompression and recovery of compressed data has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a long text processing method, device, equipment and medium based on a classification model to improve the compression rate of long texts, reduce compression costs, ensure the integrity and readability of the compressed text, and achieve rapid decompression and recovery of compressed data.

[0004] In order to solve the above technical problems, the embodiment of the present application provides a long text processing method based on a classification model, including:

[0005] Acquire long text data to be compressed, and preprocess the long text data to be compressed to obtain text features and preprocessed text;

[0006] Performing algorithm prediction on the text features through a pre-trained classification model to obtain a target compression algorithm combination;

[0007] Performing data encoding on the preprocessed text to obtain an encoded text;

[0008] Performing multi-level compression processing on the encoded text based on the target compression algorithm combination to obtain target compressed data;

[0009] If a data recovery instruction is received, multi-level decompression and data recovery processing are performed on the target compressed data to obtain target recovery data.

[0010] In order to solve the above technical problems, the embodiment of the present application provides a long text processing device based on a classification model, comprising:

[0011] A text preprocessing unit, used for acquiring long text data to be compressed, and preprocessing the long text data to be compressed to obtain text features and preprocessed text;

[0012] A compression algorithm prediction unit, used to perform algorithm prediction on the text features through a pre-trained classification model to obtain a target compression algorithm combination;

[0013] A text encoding unit, used for performing data encoding on the preprocessed text to obtain an encoded text;

[0014] A multi-stage compression unit, used for performing multi-stage compression processing on the encoded text based on the target compression algorithm combination to obtain target compressed data;

[0015] The multi-level decompression unit is used to perform multi-level decompression and data recovery processing on the target compressed data to obtain target recovery data if a data recovery instruction is received.

[0016] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the long text processing methods based on the classification model described above.

[0017] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the long text processing methods based on the classification model described above.

[0018] The embodiment of the present invention provides a long text processing method, device, equipment and medium based on a classification model. The method includes: obtaining long text data to be compressed, and preprocessing the long text data to be compressed to obtain text features and preprocessed text; performing algorithm prediction on the text features through a pre-trained classification model to obtain a target compression algorithm combination; performing data encoding on the preprocessed text to obtain an encoded text; performing multi-level compression processing on the encoded text based on the target compression algorithm combination to obtain target compressed data; if a data recovery instruction is received, performing multi-level decompression and data recovery processing on the target compressed data to obtain target recovered data. The embodiment of the present invention predicts the target compression algorithm combination through a classification model, and performs multi-level compression on the long text based on the target compression algorithm combination, so that a suitable compression algorithm can be selected for multi-level compression, thereby improving the compression rate of the long text, reducing the compression cost, and ensuring the integrity and readability of the compressed text; and when decoding is required, the compressed data is decompressed at multiple levels to achieve rapid decompression and recovery of the compressed data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a flow chart of the implementation of the long text processing method based on the classification model provided in the embodiment of the present application;

[0021] Figure 2 is a specific implementation flow chart of step S1 provided in an embodiment of the present application;

[0022] Figure 3 It is a flowchart of a specific implementation process before step S2 provided in an embodiment of the present application;

[0023] Figure 4 is a specific implementation flow chart of step S3 provided in the embodiment of the present application;

[0024] Figure 5 is a specific implementation flow chart of step S4 provided in an embodiment of the present application;

[0025] Figure 6 is a specific implementation flow chart of step S5 provided in an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of a long text processing device based on a classification model provided in an embodiment of the present application;

[0027] Figure 8 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0031] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the long text processing method based on the classification model provided in the embodiment of the present application is generally executed by a server, and accordingly, the long text processing device based on the classification model is generally configured in the server.

[0033] See also Figure 1 , Figure 1 A specific implementation of a long text processing method based on a classification model is shown.

[0034] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0035] S1: Obtain long text data to be compressed, and preprocess the long text data to be compressed to obtain text features and preprocessed text.

[0036] Specifically, when long text compression is required, the long text data to be compressed is obtained, and then the long text data to be compressed is preprocessed to obtain text features and preprocessed text. The preprocessing includes identifying the features of the long text data to be compressed and removing characters. The purpose of removing characters is to remove redundant information and provide more simplified data for subsequent long text data compression, thereby improving compression efficiency.

[0037] See also Figure 2 , Figure 2 A specific implementation of step S1 is shown, which is described in detail as follows:

[0038] S11: Identify features in the long text data to be compressed to obtain the text features, wherein the text features include text length, character type and repetition pattern;

[0039] S12: Removing invalid characters from the long text data to be compressed to obtain the preprocessed text.

[0040] In the embodiment of the present application, by traversing the long text data to be compressed, identifying the features in the long text data to be compressed, and obtaining text features, wherein the text features include text length, character type, and repetition pattern. The text features are used to subsequently select a suitable compression algorithm, and different compression algorithms are used for different data to improve the efficiency of long text compression. Invalid characters such as spaces, line breaks, and blanks exist in the long text data to be compressed. The embodiment of the present application removes these invalid characters to obtain preprocessed text.

[0041] S2: Perform algorithm prediction on the text features through a pre-trained classification model to obtain a target compression algorithm combination.

[0042] Specifically, the text features are predicted by an algorithm through a pre-trained classification model, so that the corresponding compression algorithm is selected for different long text data to be compressed, and a target compression algorithm combination is obtained. Since the optimal compression algorithm can be selected for different long text data, it is beneficial to improve the subsequent compression efficiency. For example, the target compression algorithm combination includes LZMA (Lempel-Ziv-Markov chain-Algorithm) and Snappy. LZMA is suitable for texts with more repeated patterns, and Snappy is suitable for texts with high real-time requirements.

[0043] See also Figure 3 , Figure 3 A specific implementation method before step S2 is shown, which is described in detail as follows:

[0044] S21: Collect long text samples with multiple features;

[0045] S22: extracting features of the long text sample to obtain target features;

[0046] S23: Obtaining a pre-annotated compression algorithm corresponding to the long text sample of each feature;

[0047] S24: Training a classification model based on the target features and the pre-labeled compression algorithm to obtain the pre-trained classification model.

[0048] Specifically, the classification model can be a random forest or a gradient boosting tree. In an embodiment of the present application, long text samples with multiple features are collected, and the feature refers to the length, character type and repetition pattern of the long text. The features of all long text samples are extracted to obtain the target features; before the classification model is trained, the compression algorithm will be pre-labeled for the long text samples of each feature, so when the model is trained, the pre-labeled compression algorithm corresponding to the long text sample of each feature is obtained, and then the classification model is trained based on the target feature and the pre-labeled compression algorithm, so that the pre-labeled compression algorithm gradually approaches the corresponding target feature. When the number of training iterations reaches the preset number, the model is completed and a trained classification model is obtained, that is, a pre-trained classification model is obtained. Among them, the pre-trained classification model is used to select the corresponding compression algorithm for the text feature.

[0049] S3: Performing data encoding on the preprocessed text to obtain encoded text.

[0050] See also Figure 4 , Figure 4 A specific implementation of step S3 is shown, which is described in detail as follows:

[0051] S31: Rearranging the preprocessed text so that the characters of the context in the preprocessed text are clustered to obtain a rearranged text;

[0052] S32: performing migration encoding on the rearranged text to convert repeated characters in the rearranged text into data codes to obtain the encoded text.

[0053] Specifically, the Burrows-Wheeler transform (BWT) is used to rearrange the text so that characters with similar contexts are clustered together, creating favorable conditions for subsequent encoding. Move-to-Front (MTF) encoding is used to convert the BWT rearranged data and encode frequently occurring characters into smaller numbers, which can further improve the compression efficiency.

[0054] Among them, the Burrows-Wheeler transform can rearrange the input data so that similar characters are clustered together. Move-to-Front (MTF) is a reversible transform, also known as forward coding.

[0055] S4: Perform multi-level compression processing on the encoded text based on the target compression algorithm combination to obtain target compressed data.

[0056] Specifically, the above steps have predicted the best compression algorithm combination, which includes multiple compression algorithms, wherein each compression algorithm compresses data with different text features.

[0057] See also Figure 5 , Figure 5 A specific implementation of step S4 is shown, which is described in detail as follows:

[0058] S41: compressing repeated character strings and long-distance repeated patterns in the encoded text using the LZ77 algorithm in the target compression algorithm combination to obtain preliminary compressed data;

[0059] S42: compressing the preliminary compressed data using Huffman coding in the target compression algorithm combination to obtain intermediate compressed data;

[0060] S43: compressing the intermediate compressed data using arithmetic coding in the target compression algorithm combination to obtain the target compressed data.

[0061] Specifically, the target compression algorithm combination includes LZ77 algorithm, Huffman coding and arithmetic coding. In the embodiment of the present application, the LZ77 algorithm in the target compression algorithm combination is used to compress the repeated character strings and long-distance repeated patterns in the encoded text to obtain preliminary compressed data; the Huffman coding in the target compression algorithm combination is used to compress the preliminary compressed data to obtain intermediate compressed data, where the compression is aimed at the unbalanced distribution of character frequencies. Finally, the arithmetic coding in the target compression algorithm combination is used to compress the intermediate compressed data to obtain the target compressed data.

[0062] Among them, the LZ77 algorithm is a dictionary-based lossless data compression algorithm proposed by Jacob Ziv and Abraham Lempel in 1977. It uses the repeated structural information of the data for compression and reduces the data size by identifying and replacing repeated strings in the file. The core idea of ​​LZ77 is to use sliding window technology to find and replace repeated string parts. Huffman Coding, also known as Huffman Coding, is a coding method. Huffman Coding is a type of variable word length coding (VLC). Huffman proposed a coding method in 1952. This method completely constructs codewords with the shortest average length of different letter heads based on the probability of character occurrence. It is sometimes called the best coding, and is generally called Huffman coding (sometimes also called Huffman coding). Arithmetic coding is one of the main algorithms for image compression. It is a lossless data compression method and also an entropy coding method. Furthermore, in each level of compression, text features are combined for optimization, such as parameter adjustment and adaptive symbol table establishment.

[0063] After step S4, the present application further provides a specific implementation method: using a preset encryption algorithm to encrypt the target compressed data.

[0064] Specifically, in order to ensure the security of the compressed text data, the target compressed data is encrypted. In a specific embodiment, a symmetric encryption algorithm is used to encrypt the target compressed data. The symmetric encryption algorithm is AES-256.

[0065] S5: If a data recovery instruction is received, multi-level decompression and data recovery processing are performed on the target compressed data to obtain target recovery data.

[0066] Specifically, the above steps have disclosed the multi-level compression of long text data, and in the embodiment of the present application, decompression is adopted in the opposite order of the multi-level compression of long text data, and then the decompressed data is restored to obtain the target restored data.

[0067] See also Figure 6 , Figure 6 A specific implementation of step S5 is shown, which is described in detail as follows:

[0068] S51: if the data recovery instruction is received, the target compressed data is divided into blocks by multi-thread parallel processing technology to obtain a plurality of block compressed data;

[0069] S52: constructing a target decompression algorithm combination based on algorithms in reverse order of the target compression algorithm combination;

[0070] S53: performing multi-level decompression processing on each of the block compressed data according to the target decompression algorithm combination to obtain target decompressed data;

[0071] S54: Perform data recovery processing on the target decompressed data to obtain the target recovered data.

[0072] Specifically, if the data recovery instruction is received, the target compressed data is divided into blocks through multi-threaded parallel processing technology to obtain multiple block compressed data. Due to the use of multi-threaded parallel processing technology, the compressed data can be divided into blocks, and each block is responsible for decompression by an independent thread, which greatly improves the decompression speed. In the above-mentioned target compression algorithm combination, the LZ77 algorithm, Huffman coding and arithmetic coding are used in sequence, and in the decompression process, the algorithms in the opposite order of compression are used, that is, arithmetic coding, Huffman coding and LZ77 algorithm are used in sequence, thereby constructing a target decompression algorithm combination. According to the target decompression algorithm combination, each of the block compressed data is subjected to multi-level decompression processing to obtain the target decompressed data. Finally, the target decompressed data is subjected to data recovery processing to obtain the target recovered data. Among them, data recovery includes restoring MTF coding and restoring BWT rearrangement to ensure that the target recovered data is consistent with the long text data to be compressed.

[0073] In an embodiment of the present application, long text data to be compressed is obtained, and the long text data to be compressed is preprocessed to obtain text features and preprocessed text; the text features are algorithmically predicted by a pre-trained classification model to obtain a target compression algorithm combination; the preprocessed text is data encoded to obtain an encoded text; the encoded text is subjected to multi-level compression processing based on the target compression algorithm combination to obtain target compressed data; if a data recovery instruction is received, the target compressed data is subjected to multi-level decompression and data recovery processing to obtain target recovered data. The embodiment of the present invention predicts a target compression algorithm combination through a classification model, and multi-level compression is performed on a long text based on the target compression algorithm combination, so that a suitable compression algorithm can be selected for multi-level compression, thereby improving the compression rate of the long text, reducing the compression cost, and ensuring the integrity and readability of the compressed text; and when decoding is required, the compressed data is decompressed at multiple levels to achieve rapid decompression and recovery of the compressed data.

[0074] Existing compression technologies often fail to achieve an ideal compression ratio when processing very long text data, resulting in a waste of storage space and transmission bandwidth. The embodiments of the present application introduce an adaptive compression algorithm selection mechanism and multi-level compression processing to significantly improve compression efficiency, effectively reduce data storage space and increase data transmission speed. Traditional compression technologies often take a long time to decompress, and may lose data integrity in the case of a high compression ratio.

[0075] The embodiment of the present application adopts an efficient decompression algorithm and combines it with enhanced data encoding technology to ensure that the compressed data can be quickly and accurately restored to its original state, achieving lossless compression and decompression. Existing compression technologies often ignore the security protection of data during compression, storage and transmission.

[0076] The embodiments of the present application improve data security and effectively prevent the risks of data leakage and illegal access by combining advanced encryption technology in the compression process.

[0077] Please refer to Figure 7 , as a response to the above Figure 1 The present application provides an embodiment of a long text processing device based on a classification model, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0078] like Figure 7 As shown, the long text processing device based on the classification model of this embodiment includes: a text preprocessing unit 61, a compression algorithm prediction unit 62, a text encoding unit 63, a multi-level compression unit 64 and a multi-level decompression unit 65, wherein:

[0079] The text preprocessing unit 61 is used to obtain long text data to be compressed, and preprocess the long text data to be compressed to obtain text features and preprocessed text;

[0080] A compression algorithm prediction unit 62, used to perform algorithm prediction on the text features through a pre-trained classification model to obtain a target compression algorithm combination;

[0081] A text encoding unit 63, used for performing data encoding on the preprocessed text to obtain an encoded text;

[0082] A multi-stage compression unit 64, configured to perform multi-stage compression processing on the encoded text based on the target compression algorithm combination to obtain target compressed data;

[0083] The multi-stage decompression unit 65 is used to perform multi-stage decompression and data recovery processing on the target compressed data to obtain target recovered data if a data recovery instruction is received.

[0084] Furthermore, the text preprocessing unit 61 includes:

[0085] A text feature recognition unit, used for recognizing features in the long text data to be compressed, and obtaining the text features, wherein the text features include text length, character type and repetition pattern;

[0086] The character removal unit is used to remove invalid characters in the long text data to be compressed to obtain the preprocessed text.

[0087] Furthermore, the text encoding unit 63 includes:

[0088] A rearrangement unit, used for rearranging the preprocessed text so that the characters of the context in the preprocessed text are clustered to obtain a rearranged text;

[0089] The encoding unit is used to perform migration encoding on the rearranged text to convert repeated characters in the rearranged text into data codes to obtain the encoded text.

[0090] Further, the multi-stage compression unit 64 comprises:

[0091] A primary compression unit, used to compress repeated character strings and long-distance repeated patterns in the encoded text using the LZ77 algorithm in the target compression algorithm combination to obtain preliminary compressed data;

[0092] A secondary compression unit, used for compressing the preliminary compressed data by adopting Huffman coding in the target compression algorithm combination to obtain intermediate compressed data;

[0093] The three-level compression unit is used to compress the intermediate compressed data by adopting arithmetic coding in the target compression algorithm combination to obtain the target compressed data.

[0094] Furthermore, before the compression algorithm prediction unit 62, the following steps are also included:

[0095] A sample collection unit, used to collect long text samples with multiple features;

[0096] A feature extraction unit, used to extract features of the long text sample to obtain target features;

[0097] An algorithm acquisition unit, used to acquire a pre-annotated compression algorithm corresponding to each feature of the long text sample;

[0098] A model training unit is used to train a classification model based on the target features and the pre-labeled compression algorithm to obtain the pre-trained classification model.

[0099] Furthermore, after the multi-stage compression unit 64 and before the multi-stage decompression unit 65, the following is further included:

[0100] The encryption unit is used to encrypt the target compressed data using a preset encryption algorithm.

[0101] Further, the multi-stage decompression unit 65 includes:

[0102] A block division unit is used for, upon receiving the data recovery instruction, dividing the target compressed data into blocks by a multi-threaded parallel processing technology to obtain a plurality of block compressed data;

[0103] A combination construction unit, used to construct a target decompression algorithm combination based on algorithms in the reverse order of the target compression algorithm combination;

[0104] A decompression unit, configured to perform multi-stage decompression processing on each of the block compressed data according to the target decompression algorithm combination to obtain target decompressed data;

[0105] The data recovery unit is used to perform data recovery processing on the target decompressed data to obtain the target recovered data.

[0106] In an embodiment of the present application, long text data to be compressed is obtained, and the long text data to be compressed is preprocessed to obtain text features and preprocessed text; the text features are algorithmically predicted by a pre-trained classification model to obtain a target compression algorithm combination; the preprocessed text is data encoded to obtain an encoded text; the encoded text is subjected to multi-level compression processing based on the target compression algorithm combination to obtain target compressed data; if a data recovery instruction is received, the target compressed data is subjected to multi-level decompression and data recovery processing to obtain target recovered data. The embodiment of the present invention predicts a target compression algorithm combination through a classification model, and multi-level compression is performed on a long text based on the target compression algorithm combination, so that a suitable compression algorithm can be selected for multi-level compression, thereby improving the compression rate of the long text, reducing the compression cost, and ensuring the integrity and readability of the compressed text; and when decoding is required, the compressed data is decompressed at multiple levels to achieve rapid decompression and recovery of the compressed data.

[0107] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0108] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 7 having three components: a memory 71, a processor 72, and a network interface 73, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0109] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0110] The memory 71 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 7. Of course, the memory 71 can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as the program code of the long text processing method based on the classification model, etc. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or are to be output.

[0111] The processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run the program code stored in the memory 71 or process data, such as running the program code of the long text processing method based on the classification model to implement various embodiments of the long text processing method based on the classification model.

[0112] The network interface 73 may include a wireless network interface or a wired network interface, and the network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0113] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the long text processing method based on the classification model as described above.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0115] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A long text processing method based on a classification model, characterized in that: include: Acquire long text data to be compressed, and preprocess the long text data to be compressed to obtain text features and preprocessed text; Collect long text samples with multiple features; Extracting features of the long text sample to obtain target features, wherein the target features include the length, character type, and repetition pattern of the long text; Obtaining a pre-annotated compression algorithm corresponding to the long text sample of each feature; Training a classification model based on the target features and the pre-labeled compression algorithm to obtain a pre-trained classification model; Performing algorithm prediction on the text features through the pre-trained classification model to obtain a target compression algorithm combination; Rearranging the preprocessed text by using Burrows-Wheeler transformation so that the characters of the context in the preprocessed text are clustered to obtain a rearranged text; Performing migration encoding on the rearranged text using Move-to-Front encoding to convert repeated characters in the rearranged text into data encoding to obtain an encoded text; Using the LZ77 algorithm in the target compression algorithm combination to compress repeated character strings and long-distance repeated patterns in the encoded text to obtain preliminary compressed data; Using Huffman coding in the target compression algorithm combination to compress the preliminary compressed data to obtain intermediate compressed data; Compressing the intermediate compressed data using arithmetic coding in the target compression algorithm combination to obtain target compressed data; If a data recovery instruction is received, multi-level decompression and data recovery processing are performed on the target compressed data to obtain target recovery data.

2. The long text processing method based on the classification model according to claim 1 is characterized in that: The step of obtaining the long text data to be compressed and preprocessing the long text data to be compressed to obtain text features and preprocessed text includes: Identify features in the long text data to be compressed to obtain the text features, wherein the text features include text length, character type and repetition pattern; Invalid characters in the long text data to be compressed are removed to obtain the preprocessed text.

3. The long text processing method based on the classification model according to claim 1 is characterized in that: After performing multi-stage compression processing on the encoded text based on the target compression algorithm combination to obtain target compressed data, if a data recovery instruction is received, before performing multi-stage decompression and data recovery processing on the target compressed data, the method further includes: The target compressed data is encrypted using a preset encryption algorithm.

4. The long text processing method based on the classification model according to any one of claims 1 to 3, characterized in that: If a data recovery instruction is received, the target compressed data is subjected to multi-level decompression and data recovery processing to obtain target recovery data, including: If the data recovery instruction is received, the target compressed data is divided into blocks by multi-threaded parallel processing technology to obtain a plurality of block compressed data; Constructing a target decompression algorithm combination based on an algorithm in reverse order of the target compression algorithm combination; Perform multi-level decompression processing on each of the block compressed data according to the target decompression algorithm combination to obtain target decompressed data; Performing data recovery processing on the target decompressed data to obtain the target recovered data.

5. A long text processing device based on a classification model, characterized in that: include: A text preprocessing unit, used for acquiring long text data to be compressed, and preprocessing the long text data to be compressed to obtain text features and preprocessed text; A sample collection unit, used to collect long text samples with multiple features; A feature extraction unit, used to extract features of the long text sample to obtain target features, wherein the target features include the length, character type and repetition pattern of the long text; An algorithm acquisition unit, used to acquire a pre-annotated compression algorithm corresponding to each feature of the long text sample; A model training unit, used to train a classification model based on the target feature and the pre-labeled compression algorithm to obtain a pre-trained classification model; A compression algorithm prediction unit, used to perform algorithm prediction on the text features through the pre-trained classification model to obtain a target compression algorithm combination; A rearrangement unit, configured to rearrange the preprocessed text by using Burrows-Wheeler transformation, so that the characters of the context in the preprocessed text are clustered to obtain a rearranged text; An encoding unit for performing migration encoding on the rearranged text using Move-to-Front encoding to convert repeated characters in the rearranged text into data encoding to obtain an encoded text; A primary compression unit, used to compress repeated character strings and long-distance repeated patterns in the encoded text using the LZ77 algorithm in the target compression algorithm combination to obtain preliminary compressed data; A secondary compression unit, used for compressing the preliminary compressed data by adopting Huffman coding in the target compression algorithm combination to obtain intermediate compressed data; A three-stage compression unit for compressing the intermediate compressed data using arithmetic coding in the target compression algorithm combination to obtain target compressed data; The multi-level decompression unit is used to perform multi-level decompression and data recovery processing on the target compressed data to obtain target recovery data if a data recovery instruction is received.

6. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the long text processing method based on the classification model as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the long text processing method based on a classification model as described in any one of claims 1 to 4 is implemented.