Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a storage medium and a program product. The method comprises the steps of obtaining index information and a data type of to-be-compressed data, performing compression processing on the to-be-compressed data of the image data type according to the index information and a first compression algorithm under the condition that the data type is image data, and performing compression processing on the to-be-compressed data of the image data type according to the first compression algorithm under the condition that the data type is non-image data. Performing compression processing on the to-be-compressed data of the non-image data type according to the index information and a second compression algorithm; the first compression algorithm is realized based on a deep learning model; the data compression rate of the second compression algorithm is greater than that of the first compression algorithm; in the method, compression processing is performed on the image data by adopting the compression algorithm realized based on the deep learning model, and compression processing is performed on the non-image data by adopting other compression algorithms with high compression rate so as to adapt to data characteristics of different types of data, so that the data compression effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] With the advent of the big data era, the storage of massive data has become a research hotspot in this field.

[0003] Taking unstructured data (such as pictures, videos, audio, documents, etc.) as an example, object storage, as a non-structured and flat data storage method, is suitable for the storage of unstructured data and requires a correspondingly large storage space.

[0004] In order to save storage space, compression algorithms are usually used in related technologies to compress data, but the compression effect of related technologies on high-latitude data is poor. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, apparatus, computer equipment, storage medium and program product to address the above technical issues.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] Obtain index information and data type of the data to be compressed;

[0008] When the data type is image data, compressing the data to be compressed of the image data type according to the index information and the first compression algorithm; the first compression algorithm is implemented based on a deep learning model;

[0009] When the data type is non-image data, the data to be compressed of the non-image data type is compressed according to the index information and the second compression algorithm; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

[0010] In one embodiment, obtaining index information and data type of data to be compressed includes:

[0011] Scan the data storage space through the index query thread to obtain metadata information of the data to be compressed;

[0012] The index information and data type of the data to be compressed are extracted from the metadata information.

[0013] In one embodiment, compressing the to-be-compressed data of the image data type according to the index information and the first compression algorithm includes:

[0014] Acquire the to-be-compressed data of the image data type according to the index information through the first compression thread;

[0015] The first compression algorithm is used to compress the image data to be compressed.

[0016] In one embodiment, using a first compression algorithm to compress the image data to be compressed includes:

[0017] Adding a compression command to the command queue in the buffer storage area; the compression command is used to instruct the graphics processor to compress the data to be compressed of the image data type through the variational autoencoder model to obtain compressed data;

[0018] Get the compressed data and store it over the image data type's uncompressed data.

[0019] In one embodiment, the method further comprises:

[0020] In response to the decompression instruction, obtaining index information and data type of the data to be decompressed;

[0021] Obtain the data to be decompressed according to the index information of the data to be decompressed;

[0022] Decompress the data to be decompressed according to its data type.

[0023] In one embodiment, decompressing the compressed data according to the data type of the data to be decompressed includes:

[0024] When the data type of the data to be decompressed is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the data to be decompressed of the image data type through the variational autoencoder model.

[0025] In a second aspect, the present application further provides a data processing device, comprising:

[0026] An information acquisition module, used to obtain index information and data type of data to be compressed;

[0027] A first compression module, for compressing the data to be compressed of the image data type according to the index information and a first compression algorithm when the data type is image data; the first compression algorithm is implemented based on a deep learning model;

[0028] The second compression module is used to compress the data to be compressed of the non-image data type according to the index information and the second compression algorithm when the data type is non-image data; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

[0029] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in any of the above-mentioned data processing methods when executing the computer program.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above-mentioned data processing methods when the computer program is executed by a processor.

[0031] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data processing methods when executed by a processor.

[0032] In the above-mentioned data processing method, device, computer equipment, storage medium and program product, by obtaining the index information and data type of the data to be compressed, when the data type is image data, the data to be compressed of the image data type is compressed according to the index information and the first compression algorithm, and when the data type is non-image data, the data to be compressed of the non-image data type is compressed according to the index information and the second compression algorithm; the first compression algorithm is implemented based on a deep learning model; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm; in the above-mentioned method, a compression algorithm based on a deep learning model is used for compression processing of image data, and other compression algorithms with high compression rates are used for compression processing of non-image data, so that different types of data are compressed in different ways to adapt to the data characteristics of different types of data, thereby improving the data compression effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 is an internal structure diagram of a computer device in one embodiment;

[0035] Figure 2 is a flow chart of a data processing method in one embodiment;

[0036] Figure 3 A schematic diagram of a process for obtaining index information and data type of data to be compressed in one embodiment;

[0037] Figure 4A diagrammatic representation of a data processing method in one embodiment;

[0038] Figure 5 is a schematic diagram of a process of compressing image data in one embodiment;

[0039] Figure 6 is a schematic diagram of a flow chart of compressing image data in another embodiment;

[0040] Figure 7 A schematic diagram of the VAE model training process in one embodiment;

[0041] Figure 8 A schematic diagram of a decompression process in one embodiment;

[0042] Fig. 9 is a flow chart of a data processing method in another embodiment;

[0043] Fig.10 FIG. 4 is a structural block diagram of a data processing device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The data processing method provided in the embodiment of the present application can be applied to Figure 1 The computer device shown in the figure may be a terminal. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface.

[0046] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, a trackball or a touch pad set on the computer device housing, or an external keyboard, touch pad or mouse, etc.

[0047] Among them, the processor includes a central processing unit (CPU) and a graphics processing unit (GPU).

[0048] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0049] In an exemplary embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 The CPU of a computer device is used as an example to illustrate the process, including the following steps:

[0050] S210: Obtain index information and data type of the data to be compressed.

[0051] The index information is used to indicate the data storage address. The data type may include multiple types, such as image data such as pictures or videos, and non-image data such as text or audio.

[0052] Taking the case where the data to be compressed is unstructured data that is archived and stored in an object storage manner as an example, the index information and data type of the data to be compressed are stored in the data storage space in the object storage system, which may also be referred to as a bucket.

[0053] Optionally, in a distributed storage system (such as Ceph), a computer device communicates with the object storage system as a client, and a CPU in the computer device initiates a data compression request to the object storage system to obtain index information and data type of the data to be compressed from the storage bucket.

[0054] Exemplarily, the CPU may scan the storage bucket in response to a compression instruction triggered by a user, so as to obtain index information and data type of the data to be compressed indicated by the compression instruction in the storage bucket.

[0055] S220. When the data type is image data, compress the data to be compressed of the image data type according to the index information and a first compression algorithm; the first compression algorithm is implemented based on a deep learning model.

[0056] Among them, image data is usually high-dimensional data, which requires more storage space than low-dimensional data and has a higher demand for compression ratio. In particular, high-dimensional cold data (such as medical image data) that needs to be stored for a long time but has a low access frequency has a higher demand for compression ratio.

[0057] The deep learning model that implements the first compression algorithm can perform dimensionality reduction processing on the original high-dimensional data to obtain low-dimensional data that can effectively represent the high-dimensional data, which is used as compressed data after compression processing of the data to be compressed, so as to save more storage space and improve the compression effect on the image data.

[0058] It should also be noted that the deep learning model used for data compression can not only utilize the information of adjacent data samples, but also utilize the information of distant data samples, which can achieve a higher compression ratio, thereby saving more resources and costs, and at the same time improve the transmission efficiency of files in the object storage Ceph.

[0059] Optionally, when it is determined that the data type of the data to be compressed is image data, the CPU can determine the data storage address based on the index information of the data to be compressed, so as to obtain the data to be compressed of the image data type from the data storage address, and input the data to be compressed of the image data type into a pre-trained deep learning model, and run a first compression algorithm through the deep learning model to compress the data to be compressed of the image data type to obtain compressed data.

[0060] S230. When the data type is non-image data, compress the data to be compressed of the non-image data type according to the index information and the second compression algorithm; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

[0061] Among them, non-image data is usually low-dimensional data, which requires less storage space than high-dimensional data, and has a correspondingly lower compression ratio requirement. The first compression algorithm based on the deep learning model can achieve a higher compression ratio, but the compression rate is slow and time-consuming. Therefore, for non-image data with a lower compression ratio requirement, a second compression algorithm that is different from the first compression algorithm and has a higher data compression speed can be used for compression processing, thereby improving the compression efficiency of non-image data.

[0062] Exemplarily, the second compression algorithm may be a compression algorithm that uses a hash table and a dictionary to find and replace repeated data segments for compression processing, such as the Snappy compression algorithm, or a compression algorithm that uses dictionary encoding, such as the Zstd compression algorithm or the Lz4 compression algorithm.

[0063] Optionally, when it is determined that the data type of the data to be compressed is non-image data, the CPU can determine the data storage address based on the index information of the data to be compressed, so as to obtain the data to be compressed of the non-image data type from the data storage address, and use a second compression algorithm to compress the data to be compressed of the non-image data type to obtain compressed data.

[0064] In an embodiment of the present application, by obtaining index information and data type of data to be compressed, when the data type is image data, the data to be compressed of the image data type is compressed according to the index information and the first compression algorithm; when the data type is non-image data, the data to be compressed of the non-image data type is compressed according to the index information and the second compression algorithm; the first compression algorithm is implemented based on a deep learning model; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm; in the above method, a compression algorithm based on a deep learning model is used to compress image data, and other compression algorithms with high compression rates are used to compress non-image data, thereby realizing compression processing in different ways for different types of data to adapt to the data characteristics of different types of data, thereby improving the data compression effect.

[0065] The metadata information of the data to be compressed includes index information and data type. Based on this, in one embodiment, Figure 3 As shown, the above S210, obtaining the index information and data type of the data to be compressed, includes:

[0066] S310 . Scan the data storage space through the index query thread to obtain metadata information of the data to be compressed.

[0067] The index query thread is a thread used by the CPU to scan the data storage space. The metadata information of the data to be compressed is used to characterize the attribute information of the data to be compressed, and may include the data size, data type, index information, etc. of the data to be compressed.

[0068] Optionally, the CPU enables the index query thread to scan the data storage space, query and obtain metadata information of the data to be compressed. Figure 4 As shown, the CPU provides compression services and queries the bucket index through the index query thread. Each index corresponds to a set of metadata information.

[0069] S320: Extract index information and data type of the data to be compressed from the metadata information.

[0070] Optionally, after obtaining the metadata information of the data to be compressed, the CPU may directly extract the index information and the data type of the data to be compressed from the metadata information.

[0071] In an embodiment of the present application, the data storage space is scanned by an index query thread to obtain metadata information of the data to be compressed, so as to extract the index information and data type of the data to be compressed from the metadata information; in the above method, the index information and data type of the data to be compressed are accurately recorded in the metadata information, and the index information and data type are extracted based on the metadata information, thereby simultaneously improving the extraction efficiency and accuracy of the index information and data type.

[0072] To improve data processing efficiency, in one embodiment, Figure 5 As shown, the above S220 compresses the data to be compressed of the image data type according to the index information and the first compression algorithm, including:

[0073] S510: Obtain, through a first compression thread, data to be compressed of an image data type according to index information.

[0074] The first compression thread is a thread used by the CPU to process image data. In a distributed storage system, index information is used to indicate an object storage device (OSD) node storing data in the object storage system.

[0075] Optionally, the CPU enables the first compression thread to determine the OSD node storing the data to be compressed according to the index information of the data to be compressed, and obtains the data to be compressed from the OSD node. The first compression thread and the index query thread are independent of each other and can be executed concurrently.

[0076] For example, Figure 4As shown, the CPU provides compression service. When it is determined through the index query thread that the data to be compressed is image data, the data to be compressed of the image data type is allocated to the first compression thread. The first compression thread determines the placement group (Placement Group, PG) according to the index information of the data to be compressed, and determines the OSD node according to the mapping relationship between PG and OSD, so as to obtain the data to be compressed of the image data type from the OSD node.

[0077] S520: Use a first compression algorithm to compress the data to be compressed of the image data type.

[0078] Optionally, after obtaining the data to be compressed of the image data type, the CPU compresses the data to be compressed of the image data type using a deep learning model that implements a first compression algorithm through a first compression thread.

[0079] Among them, the deep learning model that implements the first compression algorithm can use a multi-layer neural network to model the probability density function of the original data, and then use the probability density function to compress and decompress the original data. The encoder in the deep learning model is responsible for converting the original high-dimensional data into low-dimensional latent variables, and the decoder is used to restore the latent variables to the original data, thereby realizing lossless data compression based on deep learning.

[0080] It should be noted that, in addition to the index query thread and the first compression thread, the CPU can also enable a second compression thread for processing non-image data, so as to compress the data to be compressed of the non-image data type using a second compression algorithm. The index query thread, the first compression thread and the second compression thread are independent of each other, and at least two of them can be executed in parallel.

[0081] For example, Figure 4 As shown, the CPU provides compression service. When it is determined through the index query thread that the data to be compressed is non-image data, the data to be compressed of the non-image data type is allocated to the second compression thread. The second compression thread determines the PG according to the index information of the data to be compressed, and determines the OSD node according to the mapping relationship between the PG and the OSD, so as to obtain the data to be compressed of the non-image data type from the OSD node, and uses the second compression algorithm to compress the data to be compressed of the non-image data type.

[0082] In an embodiment of the present application, a first compression thread obtains the data to be compressed of the image data type according to index information, and a first compression algorithm is used to compress the data to be compressed of the image data type; in the above method, by starting different threads to perform different tasks, the targeted execution of tasks is improved, and the execution efficiency of each task and the overall data processing efficiency are correspondingly improved.

[0083] The deep learning model implementing the first compression algorithm can be run using GPU resources. Based on this, in one embodiment, Figure 6 As shown, the above S520, using the first compression algorithm to compress the data to be compressed of the image data type, includes:

[0084] S610, adding a compression command to the command queue in the buffer storage area; the compression command is used to instruct the graphics processor to compress the to-be-compressed data of the image data type through the variational autoencoder model to obtain compressed data.

[0085] The buffer storage area is a memory area of ​​a computer device, which is used to store a command queue that instructs the GPU to execute tasks. The CPU and GPU in the computer device are connected together through a PCIe bus, providing a heterogeneous parallel architecture of CPU+GPU.

[0086] Exemplarily, the buffer storage area includes at least one command queue. When the buffer storage area includes one command queue, the command queue may include different types of task commands, such as a compression command instructing the GPU to perform compression processing and a decompression command instructing the GPU to perform decompression processing; when the buffer storage area includes two command queues, they are a command queue for storing compression commands and a command queue for storing decompression commands.

[0087] Optionally, when the CPU determines that the data to be compressed is image data, it generates a compression command for the data to be compressed, and adds the compression command to a command queue in the buffer storage area for the GPU to read and perform the corresponding compression processing task by running the variational autoencoder model. The process of the CPU adding commands to the command queue and the process of the GPU reading commands from the command queue are independent of each other.

[0088] It should be noted that, when the CPU determines that the data to be compressed is image data, it will send the image data type to be compressed to the GPU through the first compression thread, and the GPU will compress the image data type in response to the compression command read. After the first compression thread sends the data to be compressed to the GPU, it will actively give up the occupied CPU time slice for use by other threads (such as the index query thread and / or the second compression thread). These threads occupy relatively low CPU resources and can be executed concurrently, which increases the task processing efficiency while also improving the utilization of CPU computing resources.

[0089] The Variational Auto-Encoder (VAE) model is a deep learning model that stacks multiple layers of encoders and decoders. The VAE model can provide more hidden parameters for learning. The sampling process follows the Markov chain. Finally, the maximum likelihood method is used to estimate the stored data, and the obtained probability density function is then used for data decompression.

[0090] like Figure 7 As shown, the training process of the VAE model is as follows:

[0091] The encoder E encodes the original data x of the input model to obtain a learnable probability density function , further obtained The mean μ and variance σ of the latent variable z1 are calculated, and then the decoder D is used to decode the latent variable z1 to obtain the probability density function Finally, the loss value between the latent variable z1 and the original data x is calculated through the following loss function L, and the latent variable z1 obtained in this layer is used as the input of the encoder E in the next layer, and so on. Finally, the model is trained by optimizing the marginal likelihood so that the latent variable zn generated by it is infinitely close to the original data x.

[0092]

[0093] S620: Obtain compressed data, and store the compressed data over the data to be compressed of the image data type.

[0094] Optionally, the CPU starts the first compression thread to obtain the compressed data processed by the GPU, and stores the compressed data over the to-be-compressed data of the image data type corresponding to the compressed data. Exemplarily, after the GPU performs compression processing on the to-be-compressed data Data1 of the image data type, it obtains the compressed data Data2, and the CPU obtains Data2, and stores Data2 over Data1.

[0095] In an embodiment of the present application, a compression command is added to a command queue in a buffer storage area to obtain compressed data, and the compressed data is used to overwrite the data to be compressed of the image data type; the compression command is used to instruct the graphics processor to compress the data to be compressed of the image data type through a variational autoencoder model to obtain compressed data; in the above method, the VAE model is run with the help of GPU resources for compression processing, thereby realizing asynchronous computing of the GPU and the CPU, and the GPU has more computing cores than the CPU, which correspondingly improves the data compression efficiency and improves the utilization rate of device resources.

[0096] In addition to the compression process, the above data processing method also includes a decompression process. In one embodiment, Figure 8 As shown, the above method also includes:

[0097] S810. In response to the decompression instruction, obtain index information and data type of the data to be decompressed.

[0098] Optionally, in a distributed storage system (such as Ceph), a computer device communicates with the object storage system as a client, and a CPU in the computer device initiates a data decompression request to the object storage system to obtain index information and data type of the data to be decompressed from the storage bucket.

[0099] Exemplarily, the CPU may scan the storage bucket in response to the decompression instruction triggered by the user to obtain the index information and data type of the data to be compressed indicated by the compression instruction in the storage bucket. Similar to the compression process, the CPU may enable the index query thread to scan the storage bucket, query and obtain the metadata information of the data to be decompressed, and extract the index information and data type of the data to be decompressed therefrom.

[0100] S820. Obtain the data to be decompressed according to the index information of the data to be decompressed.

[0101] Optionally, the CPU may determine the data storage address according to the index information of the data to be decompressed, so as to obtain the data to be decompressed from the data storage address. Exemplarily, the CPU may enable the image decompression thread to determine the OSD node storing the data to be decompressed according to the index information of the data to be decompressed, and obtain the data to be decompressed from the OSD node. The image decompression thread is also an independent thread on the CPU and can be executed concurrently with other threads.

[0102] S830: Decompress the data to be decompressed according to the data type of the data to be decompressed.

[0103] Among them, different data types correspond to different decompression algorithms.

[0104] Optionally, after obtaining the data type of the data to be decompressed, the CPU may determine the decompression algorithm of the corresponding data type based on the correspondence between the data type and the decompression algorithm, so as to call the decompression algorithm to decompress the data to be decompressed of the corresponding data type.

[0105] Exemplarily, the correspondence between data types and decompression algorithms specifically includes the mapping relationship between data types, compression algorithms and decompression algorithms. After the CPU obtains the data type of the data to be decompressed, it can first determine the compression algorithm corresponding to the data type of the data to be decompressed based on the correspondence, and then determine the decompression algorithm corresponding to the compression algorithm based on the correspondence, so as to use the decompression algorithm to decompress the data to be decompressed.

[0106] In an embodiment of the present application, by responding to a decompression instruction, the index information and data type of the data to be decompressed are obtained, and the data to be decompressed is obtained according to the index information of the data to be decompressed, thereby decompressing the data to be decompressed according to the data type of the data to be decompressed; in the above method, decompression is performed based on the data type of the data to be decompressed, which not only improves the comprehensiveness of data processing, but also improves the adaptability between the decompression processing process and the data type, which helps to improve the restoration degree of the decompression processing.

[0107] Similar to the compression process of the image data, the decompression process of the image data can also be implemented using GPU resources. Therefore, in one embodiment, the above S830, decompressing the data to be decompressed according to the data type of the data to be decompressed, includes:

[0108] When the data type of the data to be decompressed is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the data to be decompressed of the image data type through the variational autoencoder model.

[0109] Optionally, when the CPU determines that the data to be decompressed is image data, it generates a decompression command for the data to be decompressed, and adds the decompression command to a command queue in a buffer storage area for the GPU to read and perform a corresponding decompression processing task by running the variational autoencoder model. The process of the CPU adding commands to the command queue and the process of the GPU reading commands from the command queue are independent of each other.

[0110] Exemplarily, the CPU can add the decompression command for the data to be decompressed of the image data type to the command queue storing decompression commands in the buffer storage area through the image decompression thread, and the GPU reads the decompression command from the command queue storing the decompression command to run the VAE model to decompress the data to be decompressed of the image data type.

[0111] In an embodiment of the present application, when the data type of the data to be decompressed is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the data to be decompressed of the image data type through a variational autoencoder model; in the above method, the VAE model is run with the help of GPU resources to perform decompression processing, thereby realizing asynchronous computing of the GPU and the CPU, and the GPU has more computing cores than the CPU, which correspondingly improves the data decompression efficiency and improves the utilization rate of device resources.

[0112] In order to facilitate the understanding of those skilled in the art, the data processing method provided by this application is described in detail below. Fig. 9 As shown, the method may include:

[0113] S901, in response to a compression instruction, scanning a data storage space through an index query thread to obtain metadata information of the data to be compressed;

[0114] S902, extracting index information and data type of the data to be compressed from the metadata information;

[0115] S903: When the data type is image data, obtain the to-be-compressed data of the image data type according to the index information through the first compression thread;

[0116] S904, adding a compression command to the command queue in the buffer storage area to instruct the graphics processor to compress the data to be compressed of the image data type through the variational autoencoder model to obtain compressed data; the variational autoencoder model is used to implement the first compression algorithm;

[0117] S905: When the data type is non-image data, obtain the to-be-compressed data of the non-image data type according to the index information through the second compression thread;

[0118] S906, compressing the data to be compressed of the non-image data type using a second compression algorithm to obtain compressed data; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm;

[0119] S907, obtaining compressed data, and storing the compressed data over the data to be compressed;

[0120] S908, in response to the decompression instruction, obtaining index information and data type of the data to be decompressed;

[0121] S909, obtaining the data to be decompressed according to the index information of the data to be decompressed;

[0122] S910: When the data type of the to-be-decompressed data is image data, add a decompression command to the command queue in the buffer storage area to instruct the graphics processor to decompress the to-be-decompressed data of the image data type through the variational autoencoder model.

[0123] It should be noted that for the description in the above S901-S910, reference may be made to the relevant description in the above embodiment, and the effects are similar, which will not be repeated in this embodiment.

[0124] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0125] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0126] In an exemplary embodiment, Fig.10 As shown, a data processing device is provided, including: an information acquisition module 1001, a first compression module 1002 and a second compression module 1003, wherein:

[0127] The information acquisition module 1001 is used to obtain the index information and data type of the data to be compressed;

[0128] The first compression module 1002 is used for compressing the data to be compressed of the image data type according to the index information and the first compression algorithm when the data type is image data; the first compression algorithm is implemented based on the deep learning model;

[0129] The second compression module 1003 is used to compress the data to be compressed of the non-image data type according to the index information and the second compression algorithm when the data type is non-image data; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

[0130] In one embodiment, obtaining index information and data type of data to be compressed includes:

[0131] Scan the data storage space through the index query thread to obtain metadata information of the data to be compressed;

[0132] The index information and data type of the data to be compressed are extracted from the metadata information.

[0133] In one embodiment, compressing the to-be-compressed data of the image data type according to the index information and the first compression algorithm includes:

[0134] Acquire the to-be-compressed data of the image data type according to the index information through the first compression thread;

[0135] The first compression algorithm is used to compress the image data to be compressed.

[0136] In one embodiment, using a first compression algorithm to compress the image data to be compressed includes:

[0137] Adding a compression command to the command queue in the buffer storage area; the compression command is used to instruct the graphics processor to compress the data to be compressed of the image data type through the variational autoencoder model to obtain compressed data;

[0138] Get the compressed data and store it over the image data type's uncompressed data.

[0139] In one embodiment, the method further comprises:

[0140] In response to the decompression instruction, obtaining index information and data type of the data to be decompressed;

[0141] Obtain the data to be decompressed according to the index information of the data to be decompressed;

[0142] Decompress the data to be decompressed according to its data type.

[0143] In one embodiment, decompressing the compressed data according to the data type of the data to be decompressed includes:

[0144] When the data type of the data to be decompressed is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the data to be decompressed of the image data type through the variational autoencoder model.

[0145] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0146] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0147] Obtain index information and data type of data to be compressed; when the data type is image data, compress the data to be compressed of the image data type according to the index information and a first compression algorithm; the first compression algorithm is implemented based on a deep learning model; when the data type is non-image data, compress the data to be compressed of the non-image data type according to the index information and a second compression algorithm; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

[0148] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0149] The data storage space is scanned by the index query thread to obtain metadata information of the data to be compressed; and the index information and data type of the data to be compressed are extracted from the metadata information.

[0150] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0151] The data to be compressed of the image data type is obtained according to the index information through the first compression thread; and the data to be compressed of the image data type is compressed using the first compression algorithm.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] A compression command is added to the command queue in the buffer storage area; the compression command is used to instruct the graphics processor to compress the data to be compressed of the image data type through the variational autoencoder model to obtain compressed data; the compressed data is obtained, and the compressed data is used to cover the data to be compressed of the image data type.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] In response to the decompression instruction, the index information and data type of the data to be decompressed are obtained; the data to be decompressed is obtained according to the index information of the data to be decompressed; and the data to be decompressed is decompressed according to the data type of the data to be decompressed.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0157] When the data type of the data to be decompressed is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the data to be decompressed of the image data type through the variational autoencoder model.

[0158] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0159] Obtain index information and data type of data to be compressed; when the data type is image data, compress the data to be compressed of the image data type according to the index information and a first compression algorithm; the first compression algorithm is implemented based on a deep learning model; when the data type is non-image data, compress the data to be compressed of the non-image data type according to the index information and a second compression algorithm; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

[0160] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0161] The data storage space is scanned by the index query thread to obtain metadata information of the data to be compressed; and the index information and data type of the data to be compressed are extracted from the metadata information.

[0162] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0163] The data to be compressed of the image data type is obtained according to the index information through the first compression thread; and the data to be compressed of the image data type is compressed using the first compression algorithm.

[0164] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0165] A compression command is added to the command queue in the buffer storage area; the compression command is used to instruct the graphics processor to compress the data to be compressed of the image data type through the variational autoencoder model to obtain compressed data; the compressed data is obtained, and the compressed data is used to cover the data to be compressed of the image data type.

[0166] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0167] In response to the decompression instruction, the index information and data type of the data to be decompressed are obtained; the data to be decompressed is obtained according to the index information of the data to be decompressed; and the data to be decompressed is decompressed according to the data type of the data to be decompressed.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] When the data type of the data to be decompressed is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the data to be decompressed of the image data type through the variational autoencoder model.

[0170] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0171] Obtain index information and data type of data to be compressed; when the data type is image data, compress the data to be compressed of the image data type according to the index information and a first compression algorithm; the first compression algorithm is implemented based on a deep learning model; when the data type is non-image data, compress the data to be compressed of the non-image data type according to the index information and a second compression algorithm; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

[0172] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0173] The data storage space is scanned by the index query thread to obtain metadata information of the data to be compressed; and the index information and data type of the data to be compressed are extracted from the metadata information.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0175] The data to be compressed of the image data type is obtained according to the index information through the first compression thread; and the data to be compressed of the image data type is compressed using the first compression algorithm.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0177] A compression command is added to the command queue in the buffer storage area; the compression command is used to instruct the graphics processor to compress the data to be compressed of the image data type through the variational autoencoder model to obtain compressed data; the compressed data is obtained, and the compressed data is used to cover the data to be compressed of the image data type.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0179] In response to the decompression instruction, the index information and data type of the data to be decompressed are obtained; the data to be decompressed is obtained according to the index information of the data to be decompressed; and the data to be decompressed is decompressed according to the data type of the data to be decompressed.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0181] When the data type of the data to be decompressed is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the data to be decompressed of the image data type through the variational autoencoder model.

[0182] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0183] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0184] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain index information and data type of the data to be compressed; In the case where the data type is image data, compressing the data to be compressed of the image data type according to the index information and a first compression algorithm; the first compression algorithm is implemented based on a deep learning model; When the data type is non-image data, the data to be compressed of the non-image data type is compressed according to the index information and a second compression algorithm; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

2. The method according to claim 1, characterized in that The obtaining of index information and data type of the data to be compressed includes: Scan the data storage space through the index query thread to obtain metadata information of the data to be compressed; The index information and data type of the data to be compressed are extracted from the metadata information.

3. The method according to claim 1 or 2, characterized in that: The compressing the to-be-compressed data of the image data type according to the index information and the first compression algorithm comprises: Acquire the to-be-compressed data of the image data type according to the index information through the first compression thread; The first compression algorithm is used to compress the data to be compressed of the image data type.

4. The method according to claim 3, characterized in that The using the first compression algorithm to compress the data to be compressed of the image data type includes: Adding a compression command to the command queue in the buffer storage area; the compression command is used to instruct the graphics processor to compress the data to be compressed of the image data type through the variational autoencoder model to obtain compressed data; The compressed data is acquired, and the compressed data is used to cover the data to be compressed of the image data type and stored.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to the decompression instruction, obtaining index information and data type of the data to be decompressed; Acquire the data to be decompressed according to the index information of the data to be decompressed; The data to be decompressed is decompressed according to the data type of the data to be decompressed.

6. The method according to claim 5, characterized in that The decompressing the data to be compressed according to the data type of the data to be decompressed includes: When the data type of the to-be-decompressed data is image data, a decompression command is added to the command queue in the buffer storage area; the decompression command is used to instruct the graphics processor to decompress the to-be-decompressed data of the image data type through a variational autoencoder model.

7. A data processing device, characterized in that: The device comprises: An information acquisition module, used to obtain index information and data type of data to be compressed; An image compression module, configured to compress the data to be compressed of the image data type according to the index information and a first compression algorithm when the data type is image data; the first compression algorithm is implemented based on a deep learning model; A text compression module is used to compress the data to be compressed of the text data type according to the index information and a second compression algorithm when the data type is text data; the data compression rate of the second compression algorithm is greater than the data compression rate of the first compression algorithm.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images