Data processing method and device, computer equipment and storage medium

By obtaining the characteristics of the data to be compressed and identifying the model using a pre-trained compression algorithm, the applicable compression algorithm is solved, and the problem that the fixed compression algorithm cannot achieve the expected compression rate is improved, and the stability and effect of data compression are improved.

CN118971892AActive Publication Date: 2024-11-15GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD

Patent Information

Application Number
CN202310549796.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-15
Estimated Expiration
2043-05-15

AI Technical Summary

Technical Problem

In the prior art, when data compression is compressed, the fixed compression algorithm cannot effectively achieve the expected compression rate, resulting in unstable compression effect.

Method used

By obtaining the target data characteristics of the data to be compressed, and identifying the model using a pre-trained compression algorithm, the matching compression algorithm is determined to achieve more efficient data compression.

Benefits of technology

The stability and effect of data compression are improved, so that the compression rate can meet the target conditions and reduce storage and transmission costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118971892A_ABST
    Figure CN118971892A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method and device, computer equipment and a storage medium. The data processing method comprises the steps that target data features corresponding to to-be-compressed data are acquired; inputting the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier recognized by the compression algorithm recognition model, the compression algorithm recognition model is obtained by training based on data features corresponding to sample data and an algorithm identifier corresponding to a target compression algorithm matched with the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets a target compression condition; and compressing the to-be-compressed data based on a compression algorithm corresponding to the target algorithm identifier. According to the method, the compression effect of compressing the to-be-compressed data can be ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of computer technology and Internet technology, data compression technology has been widely used. Data compression can reduce the storage space of data, increase the logical capacity of storage devices, and thus reduce the storage and transmission costs of data. However, when compressing data, a fixed compression algorithm is usually used to compress the data to be compressed, which makes the compression rate of some data after compression fail to achieve the expected effect. Summary of the invention

[0003] The present application proposes a data processing method, an apparatus, a computer device and a storage medium, which can ensure the compression effect of compressing the data to be compressed.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising: obtaining target data features corresponding to data to be compressed; inputting the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier recognized by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on data features corresponding to sample data and an algorithm identifier corresponding to a target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm satisfies a target compression condition; compressing the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0005] In a second aspect, an embodiment of the present application provides a data processing device, comprising: a feature acquisition module, an algorithm selection module and a data compression module, wherein the feature acquisition module is used to acquire target data features corresponding to the data to be compressed; the algorithm selection module is used to input the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier identified by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on data features corresponding to sample data and an algorithm identifier corresponding to a target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition; the data compression module is used to compress the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0006] In a third aspect, an embodiment of the present application provides a computer device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the data processing method provided in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the data processing method provided in the first aspect above.

[0008] The solution provided by the present application is to obtain the target data features corresponding to the data to be compressed; input the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identification identified by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identification corresponding to the target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition; based on the compression algorithm corresponding to the target algorithm identification, the data to be compressed is compressed. Thus, it is possible to determine the compression algorithm matching the data to be compressed based on the data features of the data to be compressed and through the pre-trained compression algorithm recognition model, so that the compression rate of the data to be compressed after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of the compression of the data to be compressed, and thereby ensuring the compression effect of the data to be compressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flow chart of a data processing method according to an embodiment of the present application is shown.

[0011] Figure 2 A flow chart of a data processing method according to another embodiment of the present application is shown.

[0012] Figure 3 A flow chart of a data processing method according to yet another embodiment of the present application is shown.

[0013] Figure 4 A flow chart of a data processing method according to another embodiment of the present application is shown.

[0014] Figure 5 A flow chart of a data processing method according to yet another embodiment of the present application is shown.

[0015] Figure 6 A flow chart of a data processing method according to yet another embodiment of the present application is shown.

[0016] Figure 7 A block diagram of a data processing device according to an embodiment of the present application is shown.

[0017] Figure 8 It is a block diagram of a computer device according to an embodiment of the present application for executing a data processing method according to an embodiment of the present application.

[0018] Fig. 9 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the data processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0020] In practical applications, when the amount of data to be stored is relatively large, the data to be stored is usually compressed first. Data compression can reduce the storage space of data, increase the logical capacity of the storage device, and thus reduce the storage and transmission costs of data.

[0021] In the related art, in scenarios where data compression is required, the data to be compressed is usually compressed using a fixed compression algorithm. For example, in the operating system of an electronic device, the virtual memory compression scheme (such as in ZRAM) is configured as a lossless compression algorithm, and all memory pages are compressed using this algorithm. This method saves time and resource consumption for algorithm selection, but the actual data reduction effect achieved by data compression depends on the compressibility of the data to be compressed. When compressing different data in different scenarios, the compression effect usually changes, so the compression rate of some data after compression cannot reach the expected effect.

[0022] In view of the above problems, the inventors have proposed the data processing method, device, computer equipment and storage medium provided in the embodiments of the present application, which can determine the compression algorithm matching the data to be compressed based on the data features of the data to be compressed and through a pre-trained compression algorithm recognition model, so that the compression rate of the data to be compressed after being compressed by the determined compression algorithm can meet the target compression condition, improve the stability of compressing the data to be compressed, and thus ensure the compression effect of compressing the data to be compressed. Among them, the specific data processing method is described in detail in the subsequent embodiments.

[0023] The data processing method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0024] See also Figure 1 , Figure 1 The flowchart of a data processing method provided by an embodiment of the present application is shown in FIG. Figure 1 The process shown is described in detail, and the data processing method may specifically include the following steps:

[0025] Step S110: Acquire target data features corresponding to the data to be compressed.

[0026] The data to be compressed refers to data that needs to be compressed. The data to be compressed may be memory page data in the memory of a computer device, data in a database, data in a file system, etc. The scenario in which the data to be compressed is located may not be limited.

[0027] In an embodiment of the present application, when compressing data to be compressed, a computer device can obtain data features corresponding to the data to be compressed as target data features, so as to determine the compression algorithm required to compress the data to be compressed based on the target data features corresponding to the data to be compressed.

[0028] In some embodiments, the target data feature corresponding to the data to be compressed may include at least one of the following features: data type, data dimension, data size, and data content feature. Among them, data dimension may refer to information such as length, width, and height of the data; data type refers to the type of data, such as integer type, floating point type, or character type; data content feature refers to feature information possessed by the data, and the data content feature is related to the feature analysis method used. Exemplarily, the analysis method for the data content feature of the data includes at least one of the following: statistical analysis, principal components analysis (PCA), cluster analysis, hypothesis testing, etc.

[0029] Among them, the data content characteristics of the data are different according to different analysis methods. For example, based on statistical analysis, the data content characteristics of the data mainly refer to the fitted data distribution, mean, variance, etc.; based on covariance analysis, the data content characteristics of the data mainly refer to the degree of correlation between different dimensions; based on PCA, the data content characteristics of the data mainly refer to the important dimensions of the obtained data; others include hypothesis testing, cluster analysis, etc.

[0030] In a possible implementation, the target data feature may be a data content feature of the data to be compressed, and when determining the data content feature, data of corresponding bytes may be sampled from the data to be compressed, and the sampled data may be used as the data feature. Of course, the specific method for determining the data feature of the data to be compressed in the embodiment of the present application may not be limited.

[0031] Step S120: Input the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier identified by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on data features corresponding to the sample data and an algorithm identifier corresponding to a target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition.

[0032] After obtaining the target data features corresponding to the data to be compressed, the compression algorithm required to compress the data to be compressed can be determined based on the obtained target data features through the pre-trained compression algorithm recognition model. The target data features can be input into the compression algorithm recognition model to obtain the target algorithm identification obtained by the compression algorithm recognition model. The algorithm identification output by the compression algorithm recognition model corresponds to the target compression algorithm. Based on the algorithm identification output by the compression algorithm recognition model, the corresponding compression algorithm can be determined.

[0033] In an embodiment of the present application, the compression algorithm identification model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition. In other words, the sample data used to train the compression algorithm identification model includes the data features corresponding to different sample data and the algorithm annotations of the sample data. Among them, the compression algorithm identification model can be a tree model, such as a lightGBM model, or a support vector machine (SVM), a neural network, a naive Bayes classifier, etc. The specific model type of the compression algorithm identification model may not be limited.

[0034] Among them, the compression rate refers to the ratio between the data size before and after compression, and is one of the main indicators for evaluating the quality of compression algorithms; the target compression condition can be that after the sample data is compressed by each of the multiple compression algorithms in the candidate, the compression rate of the sample data after the target compression algorithm is the smallest, or the compression rate of the sample data after the target compression algorithm is less than the target compression rate, that is, the algorithm identifier marked with each sample data can be understood as the algorithm identifier corresponding to the compression algorithm applicable to the sample data, and the target compression algorithm can guarantee the compression rate of the sample data. Therefore, the compression algorithm recognition model obtained by training the above sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data can determine the compression algorithm applicable to the data corresponding to the data feature according to the input data feature, and the applicable compression algorithm ensures that the compression rate of the data corresponding to the data feature after compression meets the target compression condition.

[0035] Step S130: compressing the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0036] In an embodiment of the present application, after obtaining the target algorithm identifier identified by the above compression algorithm identification model, the data to be compressed can be compressed based on the compression algorithm corresponding to the target algorithm identifier. Since the above compression algorithm identification model can determine the compression algorithm applicable to the data corresponding to the data feature based on the input data feature, the applicable compression algorithm ensures that the compression rate of the data corresponding to the data feature after compression meets the target compression condition. Therefore, based on the compression algorithm corresponding to the target algorithm identifier, the compression rate of the data to be compressed after compression can also meet the target compression condition, thereby ensuring the data compression effect.

[0037] The data processing method provided by the embodiment of the present application obtains the target data features corresponding to the data to be compressed; inputs the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identification identified by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identification corresponding to the target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition; based on the compression algorithm corresponding to the target algorithm identification, the data to be compressed is compressed. Thus, it is possible to determine the compression algorithm matching the data to be compressed based on the data features of the data to be compressed and through the pre-trained compression algorithm recognition model, so that the compression rate of the data to be compressed after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of compressing the data to be compressed, and thereby ensuring the compression effect of compressing the data to be compressed.

[0038] See also Figure 2 , Figure 2 A schematic diagram of a data processing method provided by another embodiment of the present application is shown. The data processing method is applied to the above-mentioned computer device. Figure 2 The process shown is described in detail, and the data processing method may specifically include the following steps:

[0039] Step S210: inputting sample data into an initial recognition model to obtain a recognition result output by the initial recognition model.

[0040] Regarding the compression algorithm identification model in the embodiment of the present application, the embodiment of the present application also includes a training method for the compression algorithm identification model. It is worth noting that the training of the compression algorithm identification model can be carried out in advance based on the acquired sample data. Subsequently, each time the compression algorithm needs to be determined, the trained compression algorithm identification model can be used, without the need to train the compression algorithm identification model each time the compression algorithm is determined.

[0041] In an embodiment of the present application, the sample data set used to train the compression algorithm recognition model includes data features corresponding to the sample data to be compressed, and an algorithm identifier annotated on the data features. When the compression algorithm recognition model is obtained based on the sample data in the sample data set, the data features corresponding to the sample data can be input into the initial recognition model to obtain the recognition result output by the initial recognition model for the data features of the input sample data.

[0042] In some implementations, sample data to be compressed may be obtained; the sample data may be compressed by each compression algorithm of a plurality of compression algorithms determined in advance, and a compression ratio after the sample data is compressed by each compression algorithm may be obtained as the compression ratio corresponding to each compression algorithm; based on the compression ratio corresponding to each compression algorithm, a compression algorithm whose compression ratio satisfies the target compression condition may be determined as the target compression algorithm; and the data features corresponding to the sample data may be labeled with an algorithm identifier corresponding to the target compression algorithm. The initial recognition model may be a Softmax logistic regression model, a support vector machine, etc., and the specific initial recognition model may not be limited.

[0043] Among them, the compression scenarios used by the compression algorithm identification model, such as memory page compression, data compression in the database, data compression in the file system, etc., can be targeted, and the sample data to be compressed corresponding to the application scenario can be obtained, and the sample data can be compressed in advance by each of the determined multiple compression algorithms to obtain the compression rate after the sample data is compressed by each of the above compression algorithms respectively, and according to the compression rate corresponding to each compression algorithm obtained, the target compression algorithm whose compression rate meets the target compression condition is determined, and then the algorithm identifier corresponding to the target compression algorithm is marked on the sample data, thereby obtaining a sample data set for training the model. Among them, the above multiple compression algorithms can predetermine the compression algorithm with a comprehensive score higher than the target score, that is, the compression rate and compression speed of the candidate compression algorithms for sample data corresponding to different data features will not be too high, so as to avoid the situation that the compression rate is very high but the compression speed is very slow, or the compression speed is very fast but the compression rate is very low when the data is compressed by the determined compression algorithm. The specific compression algorithm can be not limited, for example, it can be RLE algorithm, Huffman algorithm, Rice algorithm, etc.

[0044] Step S220: Based on the recognition result and the algorithm identifier marked by the sample data, determine the target loss value corresponding to the initial recognition model.

[0045] In the embodiment of the present application, after obtaining the recognition result output by the above initial recognition model for the input data feature, the target loss value corresponding to the initial recognition model can be determined based on the recognition result and the algorithm identifier marked by the sample data corresponding to the data feature. Among them, the target loss value corresponding to the initial recognition model can be determined based on the difference between the recognition result and the algorithm identifier marked by the sample data.

[0046] In some embodiments, if the initial recognition model outputs the label probabilities corresponding to the different algorithm identifiers corresponding to the input data features, the target loss value corresponding to the initial recognition model is determined based on the algorithm identifier marked by the sample data corresponding to the data features and the label probabilities corresponding to the different algorithm identifiers output by the initial recognition model.

[0047] Step S230: Based on the target loss value, iteratively update the initial recognition model to obtain the compression algorithm recognition model.

[0048] In an embodiment of the present application, after determining the target loss value corresponding to the initial recognition model, the initial recognition model can be iteratively trained according to the loss value to obtain the final compression algorithm recognition model.

[0049] In some implementations, the model parameters of the initial recognition model can be adjusted according to the calculated target loss value; return to step S220, and repeat steps S220 to S240 until the training end condition is met to obtain the trained compression algorithm recognition model.

[0050] Among them, after the data features corresponding to each sample data are input into the initial recognition model respectively, the detection results corresponding to each data feature are obtained, and then the total loss value is determined according to the detection results corresponding to each data feature and the algorithm identifier marked by the sample data corresponding to each data feature, and then the model parameters of the initial recognition model are adjusted according to the calculated target loss value, one epoch is completed; then return to step S220 to complete the next epoch, and repeat in this way to complete multiple epochs. Among them, epoch refers to the number of times the data features corresponding to all the sample data are used. In layman's terms, the value of epoch is the number of times the entire data set is rounded, and one epoch is equal to training once using the data features corresponding to all the sample data.

[0051] In a possible implementation, the initial recognition model can be iteratively trained using the Adam optimizer according to the total loss value until the loss value of the output result of the initial recognition model converges, and the model at this time is saved to obtain the trained compression algorithm recognition model. Among them, the Adam optimizer combines the advantages of the two optimization algorithms AdaGra (Adaptive Gradient) and RMSProp, comprehensively considers the first moment estimation (First Moment Estimation, i.e. the mean of the gradient) and the second moment estimation (Second Moment Estimation, i.e. the uncentered variance of the gradient), and calculates the update step size.

[0052] In some embodiments, the training end conditions of the iterative training may include: the number of iterative training reaches the target number; or the total loss value of the output result of the initial recognition model meets the set conditions.

[0053] Optionally, the convergence condition is to make the total loss value as small as possible, use the initial learning rate 1e-3, the learning rate decays with the cosine of the step number, batch_size = 8, and after training for 16 epochs, it can be considered that convergence is complete. Among them, batch_size can be understood as a batch processing parameter, and its limit value is the total number of samples in the training set.

[0054] Optionally, the total loss value meeting the set condition may include: the total loss value is less than a set threshold. Of course, the specific set condition may not be used as a limitation.

[0055] Step S240: Obtain target data features corresponding to the data to be compressed.

[0056] Step S250: inputting the target data feature into the compression algorithm recognition model to obtain a target algorithm identifier recognized by the compression algorithm recognition model.

[0057] Step S260: compressing the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

[0058] In the embodiment of the present application, steps S240 to S260 may refer to the contents of other embodiments and will not be described in detail here.

[0059] The data processing method provided in the embodiment of the present application also provides a training process for a compression algorithm recognition model. The initial recognition model is trained through data features corresponding to sample data and algorithm identifiers marked by the sample data, thereby obtaining a compression algorithm recognition model that can recognize the compression algorithm according to the data features corresponding to the data to be compressed that are actually obtained. Then, when the compression algorithm is recognized for the data to be compressed, the compression algorithm applicable to the data to be compressed can be determined based on the compression algorithm recognition model. Therefore, for the data to be compressed, based on the data features of the data to be compressed and through a pre-trained compression algorithm recognition model, a compression algorithm matching the data to be compressed can be determined, so that the compression rate of the data to be compressed after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of compressing the data to be compressed, and thereby ensuring the compression effect of compressing the data to be compressed.

[0060] See also Figure 3 , Figure 3 A flow chart of a data processing method provided by another embodiment of the present application is shown. The data processing method is applied to the above-mentioned computer device. Figure 3 The process shown is described in detail, and the data processing method may specifically include the following steps:

[0061] Step S310: based on the target data selection rule, obtaining data of a target number of bytes from the memory page data to be compressed, and obtaining data features corresponding to the memory page data as the target data features.

[0062] In an embodiment of the present application, the scenario of compressing data may be a scenario of compressing memory page data. The computer device may obtain the memory page data to be compressed (i.e., the data to be compressed), and for the memory page data, based on the target data selection rule, obtain the target number of bytes of data from the memory page data, and obtain the data features corresponding to the memory page data. As the target data features, the data features can be obtained from the memory page data by sampling data. The memory page is the unit of memory management by the operating system, wherein the unit memory size occupied by the memory page may not be limited, and the unit memory size occupied by the memory page set by different mobile terminals may be different; the memory (also known as random access memory, random access memory, RAM) refers to the memory used when running the program (i.e., running memory), which can only temporarily store data and is used to exchange cache data with the processor, but the memory itself cannot be used for long-term data storage. When the application is running, it needs to occupy memory space to store data during the running process. The size of the memory space occupied by different applications in different running states may not be the same.

[0063] In one possible manner, data of a target number of bytes at a target position in the memory page data may be selected to obtain a data feature corresponding to the memory page data as the target data feature, and the target position includes at least one of a head position and a tail position in the memory page data. In other words, M bytes at the head or tail may be read from the memory page data as a data feature, where M is equal to the above target number of bytes, for example, 3, 5, 8, etc., and the specific value may not be limited.

[0064] In one possible manner, data of a target number of bytes may be randomly selected from the memory page data to obtain data features corresponding to the memory page data as the target data features. In other words, M bytes may be randomly selected from the memory page data as data features, where M is equal to the above target number of bytes, for example, 3, 5, 8, etc., and the specific value may not be limited.

[0065] In one possible manner, data of a target number of bytes whose occurrence frequency satisfies a target frequency condition can be obtained from the memory page data to obtain data features corresponding to the memory page data as the target data features. The target frequency condition can be that the occurrence frequency of the selected target number of bytes is greater than the occurrence frequency of other data.

[0066] In one possible manner, the data type corresponding to the memory page data may also be determined; then based on the target data selection rule corresponding to the data type, data of a target number of bytes is obtained from the memory page data to obtain data features corresponding to the memory page data. For example, for a text data type, the first M characters of each word may be selected for the text data, where M is equal to the above target number of bytes, for example, 3, 5, 8, etc., and the specific value may not be limited.

[0067] Step S320: Input the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier identified by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition.

[0068] Step S330: compressing the memory page data based on the compression algorithm corresponding to the target algorithm identifier.

[0069] In the embodiment of the present application, step S320 and step S330 may refer to the contents of other embodiments and will not be described in detail here.

[0070] The data processing method provided in the embodiment of the present application can determine data features from memory page data to be compressed by sampling data, and determine a compression algorithm that matches the memory page data through a pre-trained compression algorithm recognition model, so that the compression rate of the memory page data after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of memory page compression, improving memory compression efficiency, reducing memory usage, and reducing device overhead.

[0071] See also Figure 4 , Figure 4 A flow chart of a data processing method provided by another embodiment of the present application is shown. The data processing method is applied to the above-mentioned computer device. Figure 4 The process shown is described in detail, and the data processing method may specifically include the following steps:

[0072] Step S410: obtaining target data features corresponding to the data to be compressed, wherein the data to be compressed is memory page data of the virtual machine.

[0073] Step S420: Determine, from among a plurality of pre-trained compression algorithm recognition models corresponding to different storage areas, a compression algorithm recognition model corresponding to the storage area to which the memory page data belongs as a first recognition model.

[0074] In an embodiment of the present application, the data to be compressed may be the memory page data of a virtual machine. In the scenario of compressing the memory page of a virtual machine, considering that the usage of different storage areas (i.e., blocks) in the virtual machine is different, it will affect the compression rate of the compression algorithm used. Therefore, the compression algorithm recognition models corresponding to the different storage areas of the virtual machine can be pre-trained. The method of training the compression algorithm recognition models corresponding to the different storage areas can refer to the content of the aforementioned embodiment, except that the data features corresponding to the sample data used when training the corresponding compression algorithm recognition model for each storage area are the data features corresponding to the sample data obtained in the storage area. When the memory page data of the virtual machine is actually compressed, the compression algorithm recognition model corresponding to the storage area to which the memory page data of the virtual machine to be compressed belongs can be determined from a plurality of pre-trained compression algorithm recognition models corresponding to different storage areas as the first recognition model, that is, the compression algorithm recognition model to be used.

[0075] Step S430: input the target data features into the first recognition model to obtain a target algorithm identifier identified by the first recognition model.

[0076] In an embodiment of the present application, after the above first recognition model is determined, the target data features corresponding to the memory page data of the virtual machine can be input into the first recognition model to obtain the target algorithm identifier recognized by the first recognition model.

[0077] Step S440: compressing the memory page data based on the compression algorithm corresponding to the target algorithm identifier.

[0078] In the embodiment of the present application, step S440 can refer to the contents of other embodiments and will not be repeated here.

[0079] The data processing method provided in the embodiment of the present application can be implemented for memory page data in a virtual machine to be compressed. Based on the storage area corresponding to the memory page data, a compression algorithm recognition model corresponding to the storage area is determined from multiple pre-trained compression algorithm recognition models, and then a compression algorithm matching the memory page data is determined through the determined compression algorithm recognition model, so that the compression rate of the memory page data after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of memory page compression, improving memory compression efficiency, reducing memory usage, and reducing device overhead; and, for the memory page data in the virtual machine, a corresponding compression algorithm recognition model is determined according to the storage area where it is located, and then the compression algorithm is determined according to the determined compression algorithm recognition model, which can further improve the accuracy of the determined compression algorithm.

[0080] See also Figure 5 , Figure 5 A schematic diagram of a data processing method provided by another embodiment of the present application is shown. The data processing method is applied to the above-mentioned computer device. Figure 5 The process shown is described in detail, and the data processing method may specifically include the following steps:

[0081] Step S510: Acquire target data features corresponding to the data to be compressed, where the data to be compressed is data in a database.

[0082] Step S520: Determine, from a plurality of pre-trained compression algorithm recognition models corresponding to different data types, a compression algorithm recognition model corresponding to the data type to which the database belongs as a second recognition model.

[0083] In an embodiment of the present application, the data to be compressed may be data in a database. In the scenario of compressing the data in the database, considering that the data types of the database are different, their usage conditions will also be different, which will affect the compression rate of the compression algorithm used. Therefore, compression algorithm recognition models corresponding to different data types can be pre-trained. The method of training compression algorithm recognition models corresponding to different data types can refer to the content of the aforementioned embodiment, except that the data features corresponding to the sample data used when training the corresponding compression algorithm recognition model for each data type are the data features corresponding to the sample data of the data type. When actually compressing the data in the database, the compression algorithm recognition model corresponding to the data type in the database can be determined from a plurality of pre-trained compression algorithm recognition models corresponding to different data types as the second recognition model, that is, the compression algorithm recognition model that needs to be used.

[0084] Step S530: input the target data feature into the second recognition model to obtain a target algorithm identifier recognized by the second recognition model.

[0085] In an embodiment of the present application, after the above second recognition model is determined, the target data features corresponding to the data in the database can be input into the second recognition model to obtain the target algorithm identifier recognized by the second recognition model.

[0086] Step S540: compressing the data in the database based on the compression algorithm corresponding to the target algorithm identifier.

[0087] In the embodiment of the present application, step S540 can refer to the contents of other embodiments and will not be repeated here.

[0088] The data processing method provided in the embodiment of the present application can be implemented to determine, for the data in the database, a compression algorithm recognition model corresponding to the data type of the database from multiple pre-trained compression algorithm recognition models based on the storage area corresponding to the memory page data, and then determine the compression algorithm that matches the data to be compressed through the determined compression algorithm recognition model, so that the compression rate of the data in the database after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of compressing the data in the database and ensuring the compression effect; and, for the data in the database, the corresponding compression algorithm recognition model is determined according to the data type of the database, and then the compression algorithm is determined according to the determined compression algorithm recognition model, which can further improve the accuracy of the determined compression algorithm.

[0089] See also Figure 6 , Figure 6 The flowchart of the data processing method provided by another embodiment of the present application is shown in FIG. Figure 6 The process shown is described in detail, and the data processing method may specifically include the following steps:

[0090] Step S610: Acquire target data features corresponding to the data to be compressed, where the data to be compressed is file data in the file system.

[0091] Step S620: Determine, from among a plurality of pre-trained compression algorithm recognition models corresponding to different file attributes, a compression algorithm recognition model corresponding to the target file attribute to which the file data belongs as a third recognition model.

[0092] In the embodiment of the present application, the data to be compressed may be file data in a file system. In the scenario of compressing file data in a file system, compression algorithm recognition models corresponding to different file attributes may be pre-trained. The method of training compression algorithm recognition models corresponding to different file attributes may refer to the content of the aforementioned embodiment, except that the data features corresponding to the sample data used when training the corresponding compression algorithm recognition model for each file attribute are the data features corresponding to the sample data of the file attribute. When the file data is actually compressed, the compression algorithm recognition model corresponding to the target file attribute to which the file attribute to be compressed belongs may be determined from a plurality of pre-trained compression algorithm recognition models corresponding to different file attributes as the third recognition model, that is, the compression algorithm recognition model to be used.

[0093] Step S630: input the target data feature into the third recognition model to obtain a target algorithm identifier identified by the third recognition model.

[0094] In an embodiment of the present application, after the above third recognition model is determined, the target data features corresponding to the file data can be input into the third recognition model to obtain a target algorithm identifier identified by the third recognition model.

[0095] Step S640: compressing the memory page data based on the compression algorithm corresponding to the target algorithm identifier.

[0096] In the embodiment of the present application, step S640 can refer to the contents of other embodiments and will not be repeated here.

[0097] The data processing method provided in the embodiment of the present application can be implemented to determine, for file data in a file system to be compressed, a compression algorithm recognition model corresponding to the file attribute from multiple pre-trained compression algorithm recognition models based on the file attribute to which the file data belongs, and then determine a compression algorithm matching the file data to be compressed through the determined compression algorithm recognition model, so that the compression rate of the file data after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of compressing the file data in the file system and ensuring the compression effect; and, for the file data in the file system, the corresponding compression algorithm recognition model is determined according to the file attribute to which it belongs, and then the compression algorithm is determined according to the determined compression algorithm recognition model, which can further improve the accuracy of the determined compression algorithm.

[0098] See also Figure 7 , which shows a structural block diagram of a data processing device 700 provided in an embodiment of the present application. The data processing device 700 applies the above-mentioned computer device, and the data processing device 700 includes: a feature acquisition module 710, an algorithm selection module 720 and a data compression module 730. Among them, the feature acquisition module 710 is used to obtain the target data features corresponding to the data to be compressed; the algorithm selection module 720 is used to input the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identification obtained by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identification corresponding to the target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition; the data compression module 730 is used to compress the data to be compressed based on the compression algorithm corresponding to the target algorithm identification.

[0099] In some implementations, the feature acquisition module 710 may be specifically configured to acquire a target number of bytes of data from the memory page data based on a target data selection rule, and obtain data features corresponding to the memory page data as the target data features.

[0100] In a possible implementation, the feature acquisition module 710 may also be used to select data of a target number of bytes at a target position in the memory page data to obtain a data feature corresponding to the memory page data as the target data feature, wherein the target position includes at least one of a head position and a tail position in the memory page data; or

[0101] Randomly select data of a target number of bytes from the memory page data, and obtain data features corresponding to the memory page data as the target data features; or

[0102] Data of a target number of bytes whose occurrence frequency satisfies a target frequency condition is obtained from the memory page data, and data features corresponding to the memory page data are obtained as the target data features.

[0103] In a possible implementation, the feature acquisition module 710 can also be used to determine the data type corresponding to the memory page data; based on the target data selection rule corresponding to the data type, obtain data of a target number of bytes from the memory page data to obtain data features corresponding to the memory page data.

[0104] In some embodiments, the data to be compressed is memory page data of a virtual machine, and the algorithm selection module 720 can be specifically used to determine the compression algorithm recognition model corresponding to the storage area to which the memory page data belongs from a plurality of pre-trained compression algorithm recognition models corresponding to different storage areas, as a first recognition model; input the target data features into the first recognition model to obtain a target algorithm identifier recognized by the first recognition model.

[0105] In some embodiments, the data to be compressed is data in a database, and the algorithm selection module 720 can be specifically used to determine the compression algorithm recognition model corresponding to the data type to which the database belongs from a plurality of pre-trained compression algorithm recognition models corresponding to different data types, as a second recognition model; input the target data features into the second recognition model to obtain a target algorithm identifier recognized by the second recognition model.

[0106] In some embodiments, the data to be compressed is file data in a file system, and the algorithm selection module 720 can be specifically used to determine, from a plurality of pre-trained compression algorithm recognition models corresponding to different file attributes, a compression algorithm recognition model corresponding to a target file attribute to which the file data belongs, as a third recognition model; input the target data feature into the third recognition model to obtain a target algorithm identifier recognized by the third recognition model.

[0107] In some embodiments, the data processing device 700 may further include a model training module. The model training module may be used to: input the sample data into the initial recognition model to obtain a recognition result output by the initial recognition model; determine a target loss value corresponding to the initial recognition model based on the recognition result and the algorithm identifier marked by the sample data; and iteratively update the initial recognition model based on the target loss value to obtain the compression algorithm recognition model.

[0108] In a possible implementation, the data processing device 700 may further include a sample acquisition module, a compression rate determination module, an algorithm determination module, and a data labeling module. The sample acquisition module may be used to obtain the sample data to be compressed before the sample data is input into the initial recognition model to obtain the recognition result output by the initial recognition model; the compression rate determination module is used to compress the sample data by each compression algorithm among a plurality of compression algorithms determined in advance, and obtain the compression rate after the sample data is compressed by each compression algorithm as the compression rate corresponding to each compression algorithm; the algorithm determination module is used to determine the compression algorithm whose compression rate meets the target compression condition as the target compression algorithm based on the compression rate corresponding to each compression algorithm; the data labeling module is used to label the sample data with the algorithm identifier corresponding to the target compression algorithm.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0110] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0111] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0112] In summary, the solution provided by the present application is to obtain the target data features corresponding to the data to be compressed; input the target data features into a pre-trained compression algorithm recognition model to obtain the target algorithm identification identified by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identification corresponding to the target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition; based on the compression algorithm corresponding to the target algorithm identification, the data to be compressed is compressed. Thus, it is possible to determine the compression algorithm matching the data to be compressed based on the data features of the data to be compressed and through a pre-trained compression algorithm recognition model, so that the compression rate of the data to be compressed after being compressed by the determined compression algorithm can meet the target compression condition, thereby improving the stability of compressing the data to be compressed, and thereby ensuring the compression effect of compressing the data to be compressed.

[0113] Please refer to Figure 8 , which shows a structural block diagram of a computer device provided in an embodiment of the present application. The computer device 100 may be a computer device such as a smart phone, a tablet computer, a smart watch, an e-book, etc. that can run applications. The computer device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to execute the method described in the aforementioned method embodiment.

[0114] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire computer device 100, and executes various functions and processes data of the computer device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0115] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the computer device 100 during use.

[0116] Please refer to Fig. 9 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0117] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain target data features corresponding to the data to be compressed; Inputting the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier recognized by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identifier corresponding to the target compression algorithm matching the sample data, and the compression ratio of the sample data after being compressed by the target compression algorithm meets the target compression condition; The data to be compressed is compressed based on the compression algorithm corresponding to the target algorithm identifier.

2. The method according to claim 1, characterized in that The data to be compressed is memory page data, and the step of obtaining target data features corresponding to the data to be compressed includes: Based on the target data selection rule, data of a target number of bytes is obtained from the memory page data, and data features corresponding to the memory page data are obtained as the target data features.

3. The method according to claim 2, characterized in that The target data selection rule is based on which the target number of bytes of data is obtained from the memory page data to obtain data features corresponding to the memory page data as the target data features, including: Selecting data of a target number of bytes at a target position in the memory page data to obtain a data feature corresponding to the memory page data as the target data feature, wherein the target position includes at least one of a head position and a tail position in the memory page data; or Randomly select data of a target number of bytes from the memory page data, and obtain data features corresponding to the memory page data as the target data features; or Data of a target number of bytes whose occurrence frequency satisfies a target frequency condition is obtained from the memory page data, and data features corresponding to the memory page data are obtained as the target data features.

4. The method according to claim 2, characterized in that: The target data selection rule is based on which the target number of bytes of data is obtained from the memory page data to obtain data features corresponding to the memory page data as the target data features, including: Determine the data type corresponding to the memory page data; Based on the target data selection rule corresponding to the data type, data of a target number of bytes is obtained from the memory page data to obtain data features corresponding to the memory page data.

5. The method according to claim 1, characterized in that The data to be compressed is memory page data of a virtual machine, and the target data feature is input into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier recognized by the compression algorithm recognition model, including: Determine, from a plurality of pre-trained compression algorithm recognition models corresponding to different storage areas, a compression algorithm recognition model corresponding to the storage area to which the memory page data belongs as a first recognition model; The target data feature is input into the first recognition model to obtain a target algorithm identifier recognized by the first recognition model.

6. The method according to claim 1, characterized in that The data to be compressed is data in a database, and the target data feature is input into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier recognized by the compression algorithm recognition model, including: Determine, from a plurality of pre-trained compression algorithm recognition models corresponding to different data types, a compression algorithm recognition model corresponding to the data type to which the database belongs as a second recognition model; The target data feature is input into the second recognition model to obtain a target algorithm identifier recognized by the second recognition model.

7. The method according to claim 1, characterized in that The data to be compressed is file data in a file system, and the target data feature is input into a pre-trained compression algorithm recognition model to obtain a target algorithm identifier recognized by the compression algorithm recognition model, including: Determining, from a plurality of pre-trained compression algorithm recognition models corresponding to different file attributes, a compression algorithm recognition model corresponding to the target file attribute to which the file data belongs as a third recognition model; The target data feature is input into the third recognition model to obtain a target algorithm identifier identified by the third recognition model.

8. The method according to any one of claims 1 to 7, characterized in that: The compression algorithm recognition model is trained in the following way: Inputting the data features corresponding to the sample data into an initial recognition model to obtain a recognition result output by the initial recognition model; Based on the recognition result and the algorithm identifier marked by the data feature corresponding to the sample data, determine the target loss value corresponding to the initial recognition model; Based on the target loss value, the initial recognition model is iteratively updated to obtain the compression algorithm recognition model.

9. The method according to claim 8, characterized in that Before inputting the sample data into the initial recognition model to obtain the recognition result output by the initial recognition model, the method further includes: Obtain sample data to be compressed; Compressing the sample data by using each compression algorithm among a plurality of compression algorithms determined in advance, and obtaining a compression ratio after compressing the sample data by using each compression algorithm as the compression ratio corresponding to each compression algorithm; Based on the compression ratio corresponding to each compression algorithm, determining a compression algorithm whose compression ratio satisfies the target compression condition as a target compression algorithm; The data features corresponding to the sample data are marked with an algorithm identifier corresponding to the target compression algorithm.

10. A data processing device, characterized in that: The device comprises: a feature acquisition module, an algorithm selection module and a data compression module, wherein: The feature acquisition module is used to acquire target data features corresponding to the data to be compressed; The algorithm selection module is used to input the target data features into a pre-trained compression algorithm recognition model to obtain a target algorithm identification obtained by the compression algorithm recognition model, wherein the compression algorithm recognition model is trained based on the data features corresponding to the sample data and the algorithm identification corresponding to the target compression algorithm matching the sample data, and the compression rate of the sample data after being compressed by the target compression algorithm meets the target compression condition; The data compression module is used to compress the data to be compressed based on the compression algorithm corresponding to the target algorithm identifier.

11. A computer device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Data compression method and equipment and computer readable storage medium

    CN108197168A

  • Data compression method and device, electronic equipment and computer readable medium

    CN112994701A

  • Model training method and device, computer equipment and storage medium

    CN114612728A

  • Virtual machine live migration method and device, electronic equipment and storage medium

    CN115048183A

  • Recognition model training method and apparatus, image recognition method and apparatus, and device and medium

    WO2021017261A1

Cited By

  • Meteorological data compression and transmission intelligent control system

    CN119420705A

  • An intelligent control system for meteorological data compression and transmission

    CN119420705B