Data processing method and device, computer device, and storage medium

Abstract

The application discloses a data processing method and device, computer equipment and a storage medium. The data processing method comprises the following steps: obtaining target data characteristics corresponding to to-be-compressed data; inputting the target data characteristics into a pre-trained compression algorithm identification model to obtain a target algorithm identifier identified by the compression algorithm identification model, wherein the compression algorithm identification model is trained based on data characteristics corresponding to sample data and algorithm identifiers corresponding to target compression algorithms matched with the sample data, and a compression rate of the sample data compressed by the target compression algorithms meets a target compression condition; and compressing the to-be-compressed data based on a compression algorithm corresponding to the target algorithm identifier. The method can guarantee the compression effect of the to-be-compressed data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of computer and internet technologies, data compression technology has been widely applied. Data compression can reduce data storage space and increase the logical capacity of storage devices, thereby reducing data storage and transmission costs. However, when compressing data, a fixed compression algorithm is usually used, which can lead to some data not achieving the expected compression ratio. Summary of the Invention

[0003] This application proposes a data processing method, apparatus, computer equipment, and storage medium that can guarantee the compression effect of the data to be compressed.

[0004] In a first aspect, embodiments of this application provide a data processing method, the method comprising: acquiring target data features corresponding to data to be compressed; inputting the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier identified by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on data features corresponding to sample data and an algorithm identifier corresponding to a target compression algorithm matching the sample data, and the compression ratio of the sample data after being compressed by the target compression algorithm satisfies the target compression condition; and compressing the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0005] Secondly, embodiments of this application provide a data processing apparatus, comprising: a feature acquisition module, an algorithm selection module, and a data compression module. The feature acquisition module acquires target data features corresponding to the data to be compressed. The algorithm selection module inputs the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on data features corresponding to sample data and an algorithm identifier corresponding to a target compression algorithm matching the sample data, and the compression ratio of the sample data after compression by the target compression algorithm satisfies the target compression condition. The data compression module compresses the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0006] Thirdly, embodiments of this application provide a computer device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the data processing method provided in the first aspect above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the data processing method provided in the first aspect above.

[0008] The solution provided in this application obtains the target data features corresponding to the data to be compressed; inputs the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression ratio of the sample data after compression by the target compression algorithm meets the target compression conditions. Based on the compression algorithm corresponding to the target algorithm identifier, the data to be compressed is compressed. Therefore, for the data to be compressed, based on the data features of the data to be compressed and through the pre-trained compression algorithm recognition model, the compression algorithm matching the data to be compressed can be determined, ensuring that the compression ratio of the data to be compressed after compression by the determined compression algorithm meets the target compression conditions, improving the stability of the compression of the data to be compressed, and thus guaranteeing the compression effect of the data to be compressed. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of a data processing method according to an embodiment of this application is shown.

[0011] Figure 2 A flowchart of a data processing method according to another embodiment of this application is shown.

[0012] Figure 3 A flowchart of a data processing method according to yet another embodiment of this application is shown.

[0013] Figure 4 A flowchart of a data processing method according to another embodiment of this application is shown.

[0014] Figure 5 A flowchart of a data processing method according to another embodiment of this application is shown.

[0015] Figure 6 A flowchart of a data processing method according to yet another embodiment of this application is shown.

[0016] Figure 7 A block diagram of a data processing apparatus according to one embodiment of this application is shown.

[0017] Figure 8 This is a block diagram of a computer device for performing a data processing method according to an embodiment of this application.

[0018] Figure 9 This is a storage unit in this application embodiment for storing or carrying program code that implements the data processing method according to this application embodiment. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0020] In practical applications, such as when the amount of data to be stored is large, it is usually compressed first. Data compression can reduce the storage space required, increase the logical capacity of the storage device, and thus reduce the cost of data storage and transmission.

[0021] In related technologies, when data compression is required, a fixed compression algorithm is typically used to compress the data. For example, in the operating system of an electronic device, the virtual memory compression scheme (such as ZRAM) is configured with a lossless compression algorithm, which is used to compress all memory pages. This approach saves time and resources associated with algorithm selection. However, the actual data reduction effect achieved depends on the compressibility of the data. The compression effect usually varies when compressing different data in different scenarios, which may result in some data not achieving the expected compression ratio.

[0022] To address the aforementioned problems, the inventors have proposed a data processing method, apparatus, computer device, and storage medium as provided in the embodiments of this application. These methods can, based on the data characteristics of the data to be compressed and through a pre-trained compression algorithm identification model, determine a compression algorithm that matches the data to be compressed. This ensures that the compression ratio of the data to be compressed by the determined compression algorithm meets the target compression conditions, improving the stability of the compression process and thus guaranteeing the compression effect. The specific data processing method will be described in detail in subsequent embodiments.

[0023] The data processing method provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0024] Please see Figure 1 , Figure 1 A flowchart illustrating a data processing method according to an embodiment of this application is shown. In a specific embodiment, the data processing method is applied to a computer device, and will be discussed below. Figure 1 The process shown will be described in detail, and the data processing method may specifically include the following steps:

[0025] Step S110: Obtain the target data features corresponding to the data to be compressed.

[0026] The data to be compressed refers to the data that needs to be compressed. The data to be compressed can be memory page data in the memory of a computer device, data in a database, data in a file system, etc. The scenario in which the data to be compressed is located is not limited.

[0027] In this embodiment of the application, when a computer device compresses data to be compressed, it can obtain the data features corresponding to the data to be compressed as target data features, so as to determine the compression algorithm to be used to compress the data to be compressed based on the target data features corresponding to the data to be compressed.

[0028] In some implementations, the target data features corresponding to the data to be compressed may include at least one of the following features: data type, data dimension, data size, and data content features. Here, data dimension can refer to information such as the length, width, and height of the data; data type refers to the data type, such as integer, floating-point, or character types; and data content features refer to the characteristic information possessed by the data, which is related to the feature analysis method used. For example, the analysis methods for the data content features include at least one of the following: statistical analysis, principal component analysis (PCA), cluster analysis, hypothesis testing, etc.

[0029] The data content characteristics vary depending on the analytical method used. For example, in statistical analysis, the data content characteristics mainly refer to the fitted data distribution, mean, and variance; in covariance analysis, the data content characteristics mainly refer to the correlation between different dimensions; in PCA, the data content characteristics mainly refer to the important dimensions in the obtained data; others include hypothesis testing and cluster analysis.

[0030] In one possible implementation, the target data feature can be the data content feature of the data to be compressed. Furthermore, when determining the data content feature, corresponding bytes of data can be sampled from the data to be compressed, and the sampled data can be used as the data feature. Of course, the specific method for determining the data feature of the data to be compressed in this embodiment is not limited.

[0031] Step S120: Input the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm that matches the sample data. The compression ratio of the sample data after being compressed by the target compression algorithm meets the target compression condition.

[0032] After obtaining the target data features corresponding to the data to be compressed, a pre-trained compression algorithm recognition model can be used to determine the compression algorithm required for compressing the data based on the obtained target data features. Specifically, the target data features can be input into the compression algorithm recognition model to obtain the target algorithm identifier identified by the model. The algorithm identifier output by the compression algorithm recognition model corresponds to the target compression algorithm. Based on the algorithm identifier output by the compression algorithm recognition model, the corresponding compression algorithm can be determined.

[0033] In this embodiment, the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression ratio of the sample data after compression by the target compression algorithm meets the target compression condition. That is, the sample data used to train the compression algorithm recognition model includes the data features corresponding to different sample data and the algorithm labels labeled on the sample data. The compression algorithm recognition model can be a tree model, such as the LightGBM model, or a support vector machine (SVM), neural network, Naive Bayes classifier, etc., and the specific model type of the compression algorithm recognition model is not limited.

[0034] Compression ratio, referring to the ratio of data size before and after compression, is one of the main indicators used to evaluate the performance of compression algorithms. The target compression condition can be that after each of the candidate compression algorithms compresses the sample data, the target compression algorithm achieves the lowest compression ratio, or the compression ratio after compression is less than the target compression ratio. In other words, the algorithm identifier labeled for each sample data point can be understood as the algorithm identifier corresponding to the appropriate compression algorithm for that sample data, and the target compression algorithm guarantees a high compression ratio for the sample data. Therefore, the compression algorithm identification model trained using the above sample data and the corresponding algorithm identifiers of the target compression algorithms can determine the appropriate compression algorithm for the input data features, ensuring that the compression ratio of the data corresponding to those features meets the target compression condition.

[0035] Step S130: Compress the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0036] In this embodiment of the application, after obtaining the target algorithm identifier identified by the compression algorithm identification model, the data to be compressed can be compressed based on the compression algorithm corresponding to the target algorithm identifier. Since the compression algorithm identification model can determine the compression algorithm applicable to the data corresponding to the input data features, and the applicable compression algorithm ensures that the compression ratio of the data corresponding to the data features meets the target compression conditions, the compression ratio of the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier can also meet the target compression conditions, thereby ensuring the data compression effect.

[0037] The data processing method provided in this application involves obtaining target data features corresponding to the data to be compressed; inputting the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression ratio of the sample data after compression by the target compression algorithm meets the target compression conditions. Based on the compression algorithm corresponding to the target algorithm identifier, the data to be compressed is compressed. Therefore, for the data to be compressed, based on the data features of the data to be compressed and through a pre-trained compression algorithm recognition model, a compression algorithm matching the data to be compressed can be determined. This ensures that the compression ratio of the data to be compressed after compression by the determined compression algorithm meets the target compression conditions, improves the stability of the compression of the data to be compressed, and thus guarantees the compression effect of the data to be compressed.

[0038] Please see Figure 2 , Figure 2 A flowchart illustrating a data processing method according to another embodiment of this application is shown. This data processing method is applied to the aforementioned computer device, and will be discussed below. Figure 2 The process shown will be described in detail, and the data processing method may specifically include the following steps:

[0039] Step S210: Input the sample data into the initial recognition model to obtain the recognition result output by the initial recognition model.

[0040] Regarding the compression algorithm recognition model in this application embodiment, this application embodiment also includes a training method for the compression algorithm recognition model. It is worth noting that the training of the compression algorithm recognition model can be performed in advance based on the acquired sample data. Subsequently, whenever it is necessary to determine the compression algorithm, the trained compression algorithm recognition model can be used, without having to train the compression algorithm recognition model every time the compression algorithm is determined.

[0041] In this embodiment, the sample data set used to train the compression algorithm recognition model includes data features corresponding to the sample data to be compressed, and algorithm identifiers labeled on the data features. When training the compression algorithm recognition model based on the sample data in the sample data set, the data features corresponding to the sample data can be input into the initial recognition model to obtain the recognition result output by the initial recognition model for the data features of the input sample data.

[0042] In some implementations, sample data to be compressed can be obtained; the sample data can be compressed using each of a predetermined set of compression algorithms to obtain a compression ratio, which is used as the compression ratio corresponding to each compression algorithm; based on the compression ratio corresponding to each compression algorithm, a compression algorithm whose compression ratio satisfies the target compression condition can be determined as the target compression algorithm; and the data features corresponding to the sample data can be labeled with the algorithm identifier corresponding to the target compression algorithm. The initial identification model can be a Softmax logistic regression model, a support vector machine, etc., and the specific initial identification model is not limited.

[0043] This process involves identifying the compression scenarios used by the compression algorithm in the model, such as memory page compression, database data compression, and file system data compression. It involves acquiring sample data to be compressed corresponding to these application scenarios and pre-compressing the sample data using a selection of compression algorithms to obtain the compression ratios after each algorithm is applied. Based on these compression ratios, a target compression algorithm that meets the target compression condition is determined. The sample data is then labeled with the algorithm identifier corresponding to the target compression algorithm, thus obtaining the sample data set for training the model. The selected compression algorithms are those with a pre-determined overall score higher than the target score. This ensures that the candidate compression algorithms for different data characteristics will not have excessively high compression ratios or speeds, avoiding situations where the compression ratio is high but the compression speed is slow, or vice versa. The specific compression algorithm is not limited; for example, it could be RLE, Huffman algorithm, or Rice algorithm.

[0044] Step S220: Based on the recognition result and the algorithm identifier labeled in the sample data, determine the target loss value corresponding to the initial recognition model.

[0045] In this embodiment, after obtaining the recognition result output by the initial recognition model for the input data features, the target loss value corresponding to the initial recognition model can be determined based on the recognition result and the algorithm identifier labeled on the sample data corresponding to the data features. Specifically, the target loss value corresponding to the initial recognition model can be determined based on the difference between the recognition result and the algorithm identifier labeled on the sample data.

[0046] In some implementations, if the initial recognition model outputs the label probabilities corresponding to each different algorithm identifier for the input data features, then the target loss value corresponding to the initial recognition model is determined based on the algorithm identifier labeled in the sample data corresponding to the data features and the label probabilities corresponding to each different algorithm identifier output by the initial recognition model.

[0047] Step S230: Based on the target loss value, iteratively update the initial recognition model to obtain the compression algorithm recognition model.

[0048] In this embodiment of the application, after determining the target loss value corresponding to the initial recognition model, the initial recognition model can be iteratively trained based on the loss value to obtain the final compressed algorithm recognition model.

[0049] In some implementations, the model parameters of the initial recognition model can be adjusted according to the calculated target loss value; return to step S220, and repeat steps S220 to S240 until the training termination condition is met, and the trained compressed algorithm recognition model is obtained.

[0050] The process involves inputting the data features corresponding to each sample into the initial recognition model to obtain the detection result for each data feature. Then, based on the detection result and the algorithm identifier labeled on the sample data corresponding to each data feature, the total loss value is determined. Finally, the model parameters of the initial recognition model are adjusted according to the calculated target loss value, completing one epoch. Then, the process returns to step S220 to complete the next epoch, repeating this process to complete multiple epochs. Here, epoch refers to the number of times the data features corresponding to all sample data are used. In simpler terms, the value of epoch is the number of times the entire dataset is processed; one epoch is equivalent to training once using the data features corresponding to all sample data.

[0051] In one possible implementation, the initial recognition model can be iteratively trained using the Adam optimizer based on the total loss value until the loss value of the initial recognition model's output converges. The model at this point is then saved, resulting in the trained compressed algorithm recognition model. The Adam optimizer combines the advantages of both AdaGra (Adaptive Gradient) and RMSProp optimization algorithms, comprehensively considering the first moment estimation (mean of the gradient) and the second moment estimation (uncentered variance of the gradient) to calculate the update step size.

[0052] In some implementations, the training termination condition for iterative training may include: the number of iterations reaches the target number; or the total loss value of the output result of the initial recognition model meets the set condition.

[0053] Optionally, the convergence condition is to minimize the total loss value, using an initial learning rate of 1e-3, with the learning rate decaying cosinely with the number of steps, and a batch size of 8. After training for 16 epochs, convergence is considered complete. Here, batch size can be understood as a batch processing parameter, and its limit is the total number of samples in the training set.

[0054] Optionally, satisfying the set condition for the total loss value may include: the total loss value being less than a set threshold. Of course, the specific set condition is not a limitation.

[0055] Step S240: Obtain the target data features corresponding to the data to be compressed.

[0056] Step S250: Input the target data features into the compression algorithm recognition model to obtain the target algorithm identifier recognized by the compression algorithm recognition model.

[0057] Step S260: Compress the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0058] In the embodiments of this application, steps S240 to S260 can be referred to the contents of other embodiments, and will not be repeated here.

[0059] The data processing method provided in this application also provides a training process for a compression algorithm recognition model. An initial recognition model is trained using the data features corresponding to the sample data and the algorithm identifiers labeled on the sample data. This yields a compression algorithm recognition model that can identify compression algorithms based on the data features corresponding to the data to be compressed. When identifying compression algorithms for the data to be compressed, the appropriate compression algorithm can be determined based on this model. Therefore, for the data to be compressed, based on its data features and the pre-trained compression algorithm recognition model, a matching compression algorithm can be determined. This ensures that the compression ratio of the data compressed by the determined compression algorithm meets the target compression conditions, improving the stability of the compression and guaranteeing the compression effect.

[0060] Please see Figure 3 , Figure 3 A flowchart illustrating a data processing method according to another embodiment of this application is shown. This data processing method is applied to the aforementioned computer device, and will be discussed below. Figure 3 The process shown will be described in detail, and the data processing method may specifically include the following steps:

[0061] Step S310: Based on the target data selection rules, obtain the target number of bytes from the memory page data to be compressed, and obtain the data features corresponding to the memory page data as the target data features.

[0062] In this embodiment, the data compression scenario can be the compression of memory page data. The computer device can acquire the memory page data to be compressed (i.e., the data to be compressed), and based on target data selection rules, extract the target number of bytes from the memory page data to obtain the corresponding data features. These data features can then be used as target data features to extract data features from the memory page data through sampling. A memory page is a unit of memory management by the operating system. The size of a unit of memory occupied by a memory page is not limited; different mobile terminals may have different unit sizes for memory pages. Memory (also known as random access memory, RAM) represents the memory used when a program is running (i.e., running memory). It can only temporarily store data and is used to exchange cached data with the processor, but memory itself cannot be used for long-term data storage. Applications need to occupy memory space to store data during operation, and the amount of memory space occupied by different applications in different operating states may vary.

[0063] In one possible approach, the target number of bytes at a target location in the memory page data can be selected to obtain the corresponding data feature of the memory page data, which serves as the target data feature. The target location includes at least one of the head position and the tail position in the memory page data. That is, M bytes from the head or tail of the memory page data can be read as the data feature, where M is equal to the target number of bytes, for example, 3, 5, 8, etc., and the specific value is not limited.

[0064] In one possible approach, a target number of bytes can be randomly selected from the memory page data to obtain the corresponding data feature, which is then used as the target data feature. That is, M bytes can be randomly selected from the memory page data as the data feature, where M equals the target number of bytes, for example, 3, 5, 8, etc., and the specific value is not limited.

[0065] In one possible approach, the target number of bytes whose frequency of occurrence satisfies a target frequency condition can be obtained from the memory page data, thus obtaining the data characteristics corresponding to the memory page data, which are then used as the target data characteristics. The target frequency condition can be that the frequency of occurrence of the selected target number of bytes is greater than the frequency of occurrence of other data.

[0066] In one possible approach, the data type corresponding to the memory page data can be determined; then, based on the target data selection rules corresponding to the data type, the target number of bytes can be obtained from the memory page data to obtain the data characteristics corresponding to the memory page data. For example, for text data types, the first M characters of each word can be selected, where M equals the target number of bytes, such as 3, 5, 8, etc., and the specific value is not limited.

[0067] Step S320: Input the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm that matches the sample data. The compression ratio of the sample data after being compressed by the target compression algorithm meets the target compression condition.

[0068] Step S330: Compress the memory page data based on the compression algorithm corresponding to the target algorithm identifier.

[0069] In the embodiments of this application, steps S320 and S330 can be referred to the contents of other embodiments, and will not be repeated here.

[0070] The data processing method provided in this application embodiment can determine data features from memory page data by sampling data, and determine the compression algorithm that matches the memory page data by using a pre-trained compression algorithm identification model. This ensures that the compression ratio of the memory page data after compression by the determined compression algorithm meets the target compression conditions, improves the stability of memory page compression, increases memory compression efficiency, reduces memory usage, and lowers device costs.

[0071] Please see Figure 4 , Figure 4 A flowchart illustrating a data processing method provided in another embodiment of this application is shown. This data processing method is applied to the aforementioned computer device, and will be discussed below. Figure 4 The process shown will be described in detail, and the data processing method may specifically include the following steps:

[0072] Step S410: Obtain the target data features corresponding to the data to be compressed, wherein the data to be compressed is the memory page data of the virtual machine.

[0073] Step S420: From a variety of pre-trained compression algorithm identification models corresponding to different storage regions, determine the compression algorithm identification model corresponding to the storage region to which the memory page data belongs, and use it as the first identification model.

[0074] In this embodiment, the data to be compressed can be the memory page data of a virtual machine. In scenarios where compression is applied to the memory pages of a virtual machine, considering the different usage patterns of different storage regions (i.e., blocks) within the virtual machine, which can affect the compression ratio of the applied compression algorithm, a compression algorithm identification model corresponding to different storage regions of the virtual machine can be pre-trained. The method for training the compression algorithm identification model corresponding to different storage regions can be found in the aforementioned embodiments, except that the data features corresponding to the sample data used when training the corresponding compression algorithm identification model for each storage region are the data features corresponding to the sample data obtained in that storage region. When actually compressing the memory page data of the virtual machine, the compression algorithm identification model corresponding to the storage region to which the memory page data of the virtual machine to be compressed belongs can be determined from multiple pre-trained compression algorithm identification models corresponding to different storage regions. This first identification model is the compression algorithm identification model that needs to be used.

[0075] Step S430: Input the target data features into the first recognition model to obtain the target algorithm identifier recognized by the first recognition model.

[0076] In this embodiment of the application, after determining the first identification model, the target data features corresponding to the memory page data of the virtual machine can be input into the first identification model to obtain the target algorithm identifier identified by the first identification model.

[0077] Step S440: Compress the memory page data based on the compression algorithm corresponding to the target algorithm identifier.

[0078] In this embodiment, step S440 can be referred to the content of other embodiments, and will not be repeated here.

[0079] The data processing method provided in this application embodiment can, for memory page data in a virtual machine to be compressed, determine the compression algorithm identification model corresponding to the storage area corresponding to the memory page data from multiple pre-trained compression algorithm identification models, and then determine the compression algorithm matching the memory page data through the determined compression algorithm identification model. This ensures that the compression ratio of the memory page data after compression by the determined compression algorithm meets the target compression conditions, improves the stability of memory page compression, increases memory compression efficiency, reduces memory usage, and reduces device overhead. Furthermore, for memory page data in a virtual machine, the corresponding compression algorithm identification model is determined according to its storage area, and then the compression algorithm is determined according to the determined compression algorithm identification model, which can further improve the accuracy of the determined compression algorithm.

[0080] Please see Figure 5 , Figure 5 A flowchart illustrating a data processing method according to another embodiment of this application is shown. This data processing method is applied to the aforementioned computer device, and will be discussed below. Figure 5 The process shown will be described in detail, and the data processing method may specifically include the following steps:

[0081] Step S510: Obtain the target data features corresponding to the data to be compressed, wherein the data to be compressed is data in the database.

[0082] Step S520: From a variety of pre-trained compression algorithm recognition models corresponding to different data types, determine the compression algorithm recognition model corresponding to the data type to which the database belongs, and use it as the second recognition model.

[0083] In this embodiment, the data to be compressed can be data from a database. In scenarios where data in a database is compressed, considering that different data types in the database will have different usage scenarios, thus affecting the compression ratio of the compression algorithm used, a compression algorithm recognition model can be pre-trained for different data types. The method for training the compression algorithm recognition model for different data types can be found in the aforementioned embodiments, except that the data features corresponding to the sample data used when training the corresponding compression algorithm recognition model for each data type are the data features corresponding to the sample data of that data type. When actually compressing data in the database, the compression algorithm recognition model corresponding to the data type in the database can be determined from multiple pre-trained compression algorithm recognition models for different data types, and used as the second recognition model, which is the compression algorithm recognition model to be used.

[0084] Step S530: Input the target data features into the second recognition model to obtain the target algorithm identifier recognized by the second recognition model.

[0085] In this embodiment of the application, after determining the second recognition model, the target data features corresponding to the data in the database can be input into the second recognition model to obtain the target algorithm identifier recognized by the second recognition model.

[0086] Step S540: Compress the data in the database based on the compression algorithm corresponding to the target algorithm identifier.

[0087] In this embodiment, step S540 can be found in other embodiments and will not be repeated here.

[0088] The data processing method provided in this application embodiment can, for data in a database, determine a compression algorithm identification model corresponding to the data type of the database from multiple pre-trained compression algorithm identification models based on the storage area corresponding to the memory page data. Then, through the determined compression algorithm identification model, a compression algorithm matching the data to be compressed is determined, so that the compression ratio of the data in the database after compression by the determined compression algorithm can meet the target compression conditions, improving the stability of data compression in the database and ensuring the compression effect. Furthermore, for the data in the database, the corresponding compression algorithm identification model is determined according to the data type of the database, and then the compression algorithm is determined according to the determined compression algorithm identification model, which can further improve the accuracy of the determined compression algorithm.

[0089] Please see Figure 6 , Figure 6 A schematic flowchart of a data processing method according to yet another embodiment of this application is shown. This data processing method is applied to the aforementioned computer device, and will be discussed below. Figure 6 The process shown will be described in detail, and the data processing method may specifically include the following steps:

[0090] Step S610: Obtain the target data features corresponding to the data to be compressed, wherein the data to be compressed is file data in the file system.

[0091] Step S620: From a variety of pre-trained compression algorithm recognition models corresponding to different file attributes, determine the compression algorithm recognition model corresponding to the target file attribute to which the file data belongs, and use it as the third recognition model.

[0092] In this embodiment, the data to be compressed can be file data in a file system. In the scenario of compressing file data in a file system, compression algorithm recognition models corresponding to different file attributes can be pre-trained. The method of training compression algorithm recognition models corresponding to different file attributes can refer to the content of the foregoing embodiments, except that the data features corresponding to the sample data used when training the corresponding compression algorithm recognition model for each file attribute are the data features corresponding to the sample data of that file attribute. When actually compressing the file data, the compression algorithm recognition model corresponding to the target file attribute to which the file attribute to be compressed belongs can be determined from multiple pre-trained compression algorithm recognition models corresponding to different file attributes, and used as the third recognition model, which is the compression algorithm recognition model to be used.

[0093] Step S630: Input the target data features into the third recognition model to obtain the target algorithm identifier recognized by the third recognition model.

[0094] In this embodiment of the application, after determining the above third recognition model, the target data features corresponding to the file data can be input into the third recognition model to obtain the target algorithm identifier recognized by the third recognition model.

[0095] Step S640: Compress the memory page data based on the compression algorithm corresponding to the target algorithm identifier.

[0096] In this embodiment, step S640 can be found in other embodiments and will not be repeated here.

[0097] The data processing method provided in this application embodiment can, for file data in a file system to be compressed, determine the compression algorithm identification model corresponding to the file attribute from multiple pre-trained compression algorithm identification models based on the file attribute to which the file data belongs. Then, through the determined compression algorithm identification model, a compression algorithm matching the file data to be compressed is determined, so that the compression ratio of the file data after compression by the determined compression algorithm can meet the target compression conditions, improve the stability of compressing file data in the file system, and ensure the compression effect. Furthermore, for file data in the file system, the corresponding compression algorithm identification model is determined according to its file attribute, and then the compression algorithm is determined according to the determined compression algorithm identification model, which can further improve the accuracy of the determined compression algorithm.

[0098] Please see Figure 7 This diagram illustrates a structural block diagram of a data processing apparatus 700 provided in an embodiment of this application. The data processing apparatus 700 utilizes the aforementioned computer equipment and includes: a feature acquisition module 710, an algorithm selection module 720, and a data compression module 730. The feature acquisition module 710 acquires target data features corresponding to the data to be compressed. The algorithm selection module 720 inputs the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on data features corresponding to sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression ratio of the sample data after compression by the target compression algorithm meets the target compression condition. The data compression module 730 compresses the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0099] In some implementations, the feature acquisition module 710 can be specifically used to acquire data on the target number of bytes from the memory page data based on the target data selection rules, and obtain the data features corresponding to the memory page data as the target data features.

[0100] In one possible implementation, the feature acquisition module 710 can also be used to select the number of target bytes at a target location in the memory page data to obtain the data feature corresponding to the memory page data, which serves as the target data feature. The target location includes at least one of the head position and the tail position in the memory page data; or

[0101] Randomly select a target number of bytes from the memory page data to obtain the data features corresponding to the memory page data, and use these as the target data features; or

[0102] The number of target bytes whose occurrence frequency meets the target frequency condition is obtained from the memory page data, and the data feature corresponding to the memory page data is obtained as the target data feature.

[0103] In one possible implementation, the feature acquisition module 710 can also be used to determine the data type corresponding to the memory page data; based on the target data selection rule corresponding to the data type, to obtain the target number of bytes from the memory page data, and to obtain the data features corresponding to the memory page data.

[0104] In some implementations, the data to be compressed is memory page data of a virtual machine. The algorithm selection module 720 can be specifically used to determine the compression algorithm identification model corresponding to the storage area to which the memory page data belongs from a variety of pre-trained compression algorithm identification models corresponding to different storage areas, and use it as the first identification model; input the target data features into the first identification model to obtain the target algorithm identifier identified by the first identification model.

[0105] In some implementations, the data to be compressed is data in a database. The algorithm selection module 720 can be specifically used to determine the compression algorithm recognition model corresponding to the data type to which the database belongs from a variety of pre-trained compression algorithm recognition models corresponding to different data types, and use it as a second recognition model; input the target data features into the second recognition model to obtain the target algorithm identifier recognized by the second recognition model.

[0106] In some implementations, the data to be compressed is file data in a file system. The algorithm selection module 720 can be specifically used to determine the compression algorithm recognition model corresponding to the target file attribute to which the file data belongs from a variety of pre-trained compression algorithm recognition models corresponding to different file attributes, and use it as a third recognition model; input the target data features into the third recognition model to obtain the target algorithm identifier recognized by the third recognition model.

[0107] In some embodiments, the data processing apparatus 700 may further include a model training module. The model training module may be used to: input the sample data into an initial recognition model to obtain the recognition result output by the initial recognition model; determine the target loss value corresponding to the initial recognition model based on the recognition result and the algorithm identifier labeled on the sample data; and iteratively update the initial recognition model based on the target loss value to obtain the compressed algorithm recognition model.

[0108] In one possible implementation, the data processing device 700 may further include a sample acquisition module, a compression ratio determination module, an algorithm determination module, and a data annotation module. The sample acquisition module is used to acquire sample data to be compressed before inputting the sample data into an initial recognition model to obtain the recognition result output by the initial recognition model. The compression ratio determination module is used to compress the sample data using each of a pre-determined set of compression algorithms to obtain a compression ratio after compression using each algorithm, which serves as the compression ratio corresponding to each algorithm. The algorithm determination module is used to determine, based on the compression ratio corresponding to each algorithm, a compression algorithm whose compression ratio satisfies the target compression condition as the target compression algorithm. The data annotation module is used to annotate the sample data with the algorithm identifier corresponding to the target compression algorithm.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0110] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0111] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0112] In summary, the solution provided in this application obtains the target data features corresponding to the data to be compressed; inputs the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression ratio of the sample data after compression by the target compression algorithm meets the target compression conditions. Based on the compression algorithm corresponding to the target algorithm identifier, the data to be compressed is compressed. Therefore, for the data to be compressed, based on the data features of the data to be compressed and through a pre-trained compression algorithm recognition model, the compression algorithm matching the data to be compressed can be determined, ensuring that the compression ratio of the data to be compressed after compression by the determined compression algorithm meets the target compression conditions, improving the stability of the compression of the data to be compressed, and thus guaranteeing the compression effect of the data to be compressed.

[0113] Please refer to Figure 8 This diagram illustrates a structural block diagram of a computer device according to an embodiment of this application. The computer device 100 can be a smartphone, tablet, smartwatch, e-reader, or other computer device capable of running applications. The computer device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0114] Processor 110 may include one or more processing cores. Processor 110 connects to various parts within the computer device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0115] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the computer device 100 during use (such as phonebooks, audio and video data, chat log data, etc.).

[0116] Please refer to Figure 9 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0117] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the target data features corresponding to the data to be compressed; The target data features are input into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model. The compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm that matches the sample data. The compression ratio of the sample data after being compressed by the target compression algorithm meets the target compression condition. Based on the compression algorithm corresponding to the target algorithm identifier, the data to be compressed is compressed; When the data to be compressed is memory page data, the target data feature is obtained by acquiring the target number of bytes from the memory page data based on the target data selection rules. The acquisition of the target data feature corresponding to the data to be compressed includes: Select the target byte count data at the target location in the memory page data to obtain the data feature corresponding to the memory page data, which is used as the target data feature. The target location includes at least one of the head position and the tail position in the memory page data; or Randomly select a target number of bytes from the memory page data to obtain the data features corresponding to the memory page data, and use these as the target data features; or The number of target bytes whose occurrence frequency meets the target frequency condition is obtained from the memory page data, and the data feature corresponding to the memory page data is obtained as the target data feature.

2. The method according to claim 1, characterized in that, Based on the target data selection rules, the target number of bytes is obtained from the memory page data to obtain the data features corresponding to the memory page data, which are used as the target data features, including: Determine the data type corresponding to the memory page data; Based on the target data selection rules corresponding to the data type, the target number of bytes is obtained from the memory page data, and the data characteristics corresponding to the memory page data are obtained.

3. The method according to claim 1, characterized in that, The data to be compressed is memory page data of a virtual machine. The step of inputting the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model includes: From a variety of pre-trained compression algorithm identification models corresponding to different storage regions, determine the compression algorithm identification model corresponding to the storage region to which the memory page data belongs, and use it as the first identification model; The target data features are input into the first recognition model to obtain the target algorithm identifier recognized by the first recognition model.

4. The method according to claim 1, characterized in that, The data to be compressed is data from a database. The step of inputting the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model includes: From a variety of pre-trained compression algorithm recognition models corresponding to different data types, determine the compression algorithm recognition model corresponding to the data type to which the database belongs, and use it as the second recognition model; The target data features are input into the second recognition model to obtain the target algorithm identifier recognized by the second recognition model.

5. The method according to claim 1, characterized in that, The data to be compressed is file data in a file system. The step of inputting the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier identified by the compression algorithm recognition model includes: From a variety of pre-trained compression algorithm recognition models corresponding to different file attributes, determine the compression algorithm recognition model corresponding to the target file attribute to which the file data belongs, and use it as the third recognition model; The target data features are input into the third recognition model to obtain the target algorithm identifier recognized by the third recognition model.

6. The method according to any one of claims 1-5, characterized in that, The compression algorithm recognition model is trained in the following way: The data features corresponding to the sample data are input into the initial recognition model to obtain the recognition result output by the initial recognition model; Based on the recognition results and the algorithm identifiers labeled with the data features corresponding to the sample data, the target loss value corresponding to the initial recognition model is determined. Based on the target loss value, the initial recognition model is iteratively updated to obtain the compression algorithm recognition model.

7. The method according to claim 6, characterized in that, Before inputting the sample data into the initial recognition model and obtaining the recognition result output by the initial recognition model, the method further includes: Obtain the sample data to be compressed; The sample data is compressed using each of a predetermined compression algorithm to obtain the compression ratio of the sample data after compression using each compression algorithm, which is used as the compression ratio corresponding to each compression algorithm. Based on the compression ratio corresponding to each compression algorithm, the compression algorithm whose compression ratio satisfies the target compression condition is determined as the target compression algorithm; The data features corresponding to the sample data are labeled with the algorithm identifier corresponding to the target compression algorithm.

8. A data processing apparatus, characterized in that, The device includes: a feature acquisition module, an algorithm selection module, and a data compression module, wherein, The feature acquisition module is used to acquire the target data features corresponding to the data to be compressed; The algorithm selection module is used to input the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identifier recognized by the compression algorithm recognition model. The compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm that matches the sample data. The compression ratio of the sample data after being compressed by the target compression algorithm meets the target compression condition. The data compression module is used to compress the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier; When the data to be compressed is memory page data, the target data feature is obtained by acquiring the target number of bytes from the memory page data based on the target data selection rules. Specifically, the feature acquisition module is used to: select the target number of bytes at a target location in the memory page data to obtain the data feature corresponding to the memory page data, which serves as the target data feature. The target location includes at least one of the head position and the tail position in the memory page data; or Randomly select a target number of bytes from the memory page data to obtain the data features corresponding to the memory page data, and use these as the target data features; or The number of target bytes whose occurrence frequency meets the target frequency condition is obtained from the memory page data, and the data feature corresponding to the memory page data is obtained as the target data feature.

9. A computer device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-7.

Images