Data processing method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure relates to a data processing method and device, electronic equipment, storage medium and program product. The method comprises: in response to first data belonging to first type data, compressing the first data in the foreground to obtain second data, and storing the second data in a preset storage area; the first data is data to be written into the preset storage area, and the first type data is data whose influence on system performance is greater than a first influence degree threshold; in response to third data belonging to second type data, compressing the third data in the background to obtain fourth data; the third data is data associated with the first data stored in the preset storage area, and the second type data is data whose influence on system performance is greater than a second influence degree threshold. The present disclosure can reduce the write pressure of the electronic equipment, and reduce the occupation of the storage space of the electronic equipment by using the bidirectional compression mode of the foreground and the background, thereby improving the running speed of the electronic equipment.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Electronic devices are equipped with memory. During the use of electronic devices, data can be written to the memory and / or read from the memory. However, data reading and writing are limited by various factors, resulting in low efficiency of reading and / or writing. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a data processing method, apparatus, electronic device, storage medium, and program product.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0005] In response to the fact that the first data belongs to the first type of data, the first data is compressed in the foreground to obtain the second data, and the second data is stored in a preset storage area; the first data is the data to be written to the preset storage area, and the first type of data is the data whose impact on system performance is greater than a first impact threshold.

[0006] In response to the fact that the third data belongs to the second type of data, the third data is compressed in the background to obtain the fourth data; the third data is the data associated with the first data stored in the preset storage area, and the second type of data is the data whose impact on system performance is greater than the second impact threshold.

[0007] In this embodiment, the write pressure on the electronic device can be reduced, and the bidirectional compression method in the foreground and background can reduce the storage space occupied by the electronic device, thereby improving the operating speed of the electronic device. Furthermore, compressing the first data in the foreground and compressing the third data associated with the first data in the background allows the data associated with the first data to be compressed on both the foreground and background sides, directly or indirectly reducing the storage space occupied on both sides and saving storage costs.

[0008] In some embodiments, the method further includes:

[0009] In response to the fact that the data type of the first data is a preset data type, and / or that the access frequency of the first data is greater than a preset frequency threshold, it is determined that the first data belongs to the first type of data.

[0010] In this embodiment of the disclosure, data can be dynamically classified according to data type and access frequency, which can more accurately determine whether the first data meets the compression conditions, thereby enabling compression of data that meets the compression conditions, making the compression more targeted.

[0011] In some embodiments, the method further includes:

[0012] Obtain the content information of the first data; the content information of the first data is used to characterize the degree of repetition of the content of the first data;

[0013] In response to the content information of the first data satisfying a preset condition, it is determined that the first data belongs to the first type of data; the content information of the first data satisfying the preset condition is used to characterize that the repetition degree of the content of the first data is greater than a preset repetition degree threshold.

[0014] In this embodiment of the disclosure, by analyzing the content information of the data, data with a high degree of repetition can be identified, allowing for more efficient compression. Furthermore, data with a high degree of repetition typically has greater compression potential, thus significantly reducing the data volume.

[0015] In some embodiments, the method further includes:

[0016] In response to the content information including a preset content structure, and / or the content information including a redundancy level greater than a preset redundancy level threshold, it is determined that the content information of the first data satisfies the preset condition.

[0017] In this embodiment of the disclosure, data can be dynamically classified according to the content structure and redundancy of the content information, which can more accurately determine whether the first data meets the compression conditions, thereby enabling compression of data that meets the compression conditions, making the compression more targeted.

[0018] In some embodiments, the method further includes:

[0019] Obtain a preset cache page of the first data; the preset cache page includes at least a portion of the content of the first data;

[0020] Based on the preset cache page, the content information of the first data is determined.

[0021] In this embodiment of the disclosure, the content information of the first data can be quickly obtained by acquiring a preset cache page, thereby improving the efficiency of content information acquisition.

[0022] In some embodiments, the first data includes multiple data blocks, and the step of compressing the first data in the foreground to obtain the second data includes:

[0023] Based on the preset compression rate of each data block in the plurality of data blocks, each data block is compressed in the foreground to obtain the compressed data block corresponding to each data block; the second data includes the compressed data block corresponding to each data block, and the space occupied by the compressed data block corresponding to the data block is negatively correlated with the preset compression rate of the data block.

[0024] In this embodiment of the disclosure, the first data is divided into multiple data blocks. By setting a preset compression rate for each data block, targeted compression can be performed according to the characteristics and importance of the data blocks, thereby improving user satisfaction with data processing.

[0025] In some embodiments, compressing each data block in the foreground based on a preset compression rate for each of the plurality of data blocks to obtain a compressed data block corresponding to each data block includes:

[0026] Each data block is input into a preset model, and the compressed data block corresponding to each data block is output. The preset model is used to determine the compressed data block corresponding to each data block based on the frequency of occurrence of the input data block in the target data, and to determine the output data based on the compressed data block corresponding to each data block. The target data includes at least the first data. The space occupancy of the compressed data block corresponding to each data block is negatively correlated with the frequency of occurrence of the data block, and the frequency of occurrence of the data block is positively correlated with the preset compression ratio of the data block. The preset model includes the correspondence between the frequency of occurrence of the data block and the compressed data block corresponding to the data block.

[0027] In this embodiment, a preset model can be pre-trained, and during the actual compression process, the first data is input into the preset model to directly obtain the second data. This saves time spent on data compression and accelerates data processing and writing speeds.

[0028] In some embodiments, the third data includes fifth data, which is used to characterize data that has been compressed in the foreground. In response to the third data belonging to a second type of data, the third data is compressed in the background to obtain fourth data, including:

[0029] In response to the fact that the fifth data belongs to the second type of data and the compression degree of the fifth data is less than the preset compression degree threshold, the fifth data is compressed in the background to obtain the compressed fifth data.

[0030] In this embodiment, when the fifth data belongs to the second type of data and its compression level is less than a preset compression level threshold, further compression can be performed in the background on data that has already been compressed in the foreground. This effectively reduces the storage space occupied by the electronic device and improves its response speed. Furthermore, triggering compression when the fifth data belongs to the second type of data and its compression level is less than the preset compression level threshold provides a dynamic strategy that can be flexibly adjusted based on actual needs, avoiding unnecessary compression operations.

[0031] In some embodiments, the method further includes:

[0032] The fourth data is stored in a preset physical block within the preset storage area; the preset physical block is used to store the compressed data in the background.

[0033] In this embodiment of the disclosure, the compressed data in the background can be stored in a preset physical block, so that all the compressed data in the background is stored in the same physical block. This can effectively reduce the space fragmentation of the storage space and reduce the space occupation of the storage space. In addition, for frequently accessed data, the data can be read more efficiently, effectively solving the problem of read amplification and improving the system response speed.

[0034] In some embodiments, the method further includes:

[0035] In response to the decompression operation of the fourth data, the decompressed fourth data is obtained;

[0036] The decompressed fourth data is stored in the storage area corresponding to the first storage address; the first storage address is the storage address of the third data.

[0037] In this embodiment of the disclosure, storing the decompressed data at the address of the original data enables address reuse, thereby reducing unnecessary data movement and input / output operations. Furthermore, storing the decompressed data at the address of the original data avoids creating additional copies in the storage space, thus saving storage space.

[0038] In some embodiments, storing the decompressed fourth data in the storage area corresponding to the first storage address includes:

[0039] In response to the fact that no new data is stored in the storage area corresponding to the first storage address after obtaining the fourth data, the decompressed fourth data is stored in the storage area corresponding to the first storage address.

[0040] In this embodiment, it can be checked whether new data is stored in the storage area corresponding to the first storage address. This avoids overwriting existing new data with the decompressed fourth data, thereby ensuring data consistency and integrity and preventing data loss or corruption. Furthermore, storing the decompressed data in this area avoids unnecessary data redundancy, thus improving the utilization rate of storage resources.

[0041] In some embodiments, the step of responding to the decompression operation of the fourth data to obtain the decompressed fourth data includes:

[0042] In response to performing the decompression operation on the fourth data, the decompression result of the fourth data is obtained;

[0043] Based on the verification information corresponding to the fourth data, the decompression result of the fourth data is verified; the verification information is the information generated during the compression process of the third data.

[0044] In response to the successful verification of the decompression result of the fourth data, the decompression result of the fourth data is determined as the decompressed fourth data.

[0045] In this embodiment of the disclosure, by using verification information to verify the decompression result, it can be ensured that the decompressed data is consistent with the original data, avoiding data corruption caused by errors in the compression or decompression process. Furthermore, the decompression result is only determined as the final data after the verification passes, ensuring the integrity and reliability of the decompressed data.

[0046] In some embodiments, compressing the first data in the foreground to obtain the second data includes:

[0047] In response to the first resource utilization rate meeting the first resource utilization condition, the first data is compressed on the front end to obtain the second data.

[0048] In this embodiment of the disclosure, by monitoring the first resource utilization rate and performing the compression operation only when the first resource utilization rate meets the first resource utilization condition, excessive resource consumption can be avoided when system resources are scarce, thereby ensuring the stability and smoothness of the electronic device during operation.

[0049] In some embodiments, the method further includes:

[0050] In response to the load being greater than a first load threshold, it is determined that the first resource utilization rate meets the first resource utilization condition.

[0051] In this embodiment of the disclosure, when the electronic device is under heavy load, it can be determined that the first resource utilization rate meets the first resource utilization condition, so that compression can be performed in the foreground, thereby quickly reducing the amount of data and the occupation of storage resources, and alleviating the storage burden of the electronic device.

[0052] In some embodiments, the method further includes:

[0053] In response to the fact that the space occupancy of the first data is greater than the first space occupancy threshold, it is determined that the first resource utilization rate meets the first resource utilization condition.

[0054] In this embodiment of the disclosure, when the data occupies more than a threshold, it is determined that the first resource utilization rate meets the first resource utilization condition, and the data is compressed. This can compress data that occupies too much space in a timely manner, optimize the utilization efficiency of storage resources, and ensure that electronic devices can better support the storage needs of other data.

[0055] In some embodiments, compressing the first data in the foreground to obtain the second data includes:

[0056] The first data and its metadata are compressed on the front end to obtain the second data.

[0057] In this embodiment of the disclosure, the first data and its metadata can be compressed together to achieve deep compression of the data, while ensuring the consistency and integrity of the data and its metadata, and avoiding inconsistency problems caused by the separation of data and metadata.

[0058] In some embodiments, compressing the third data in the background to obtain the fourth data includes:

[0059] In response to the second resource utilization rate meeting the second resource utilization condition, the third data is compressed in the background to obtain the fourth data.

[0060] In this embodiment of the disclosure, when the second resource utilization rate meets the second resource utilization condition, the system's idle resources can be fully utilized to compress the third data in the background, thereby improving resource utilization efficiency. Furthermore, by compressing the third data, the storage space occupied by the data can be reduced, thus saving storage resources.

[0061] In some embodiments, the method further includes:

[0062] In response to a load less than or equal to a second load threshold, it is determined that the second resource utilization rate meets the second resource utilization condition.

[0063] In this embodiment of the disclosure, when the electronic device has a low load, that is, when the electronic device has idle resources, it can be determined that the second resource utilization rate meets the second resource utilization condition. In this way, the idle resources of the electronic device can be used to perform compression operations in the background, avoiding resource waste and improving the overall resource utilization rate.

[0064] In some embodiments, the method further includes:

[0065] In response to the fact that the space occupancy of the third data is greater than the second space occupancy threshold, it is determined that the second resource utilization rate meets the second resource utilization condition.

[0066] In this embodiment of the disclosure, when the data occupies more than a threshold, it is determined that the second resource utilization rate meets the second resource utilization condition, and the data is compressed. This can compress data that occupies too much space in a timely manner and optimize the use of storage space.

[0067] In some embodiments, compressing the third data in the background to obtain the fourth data includes:

[0068] The third data and its metadata are compressed in the background to obtain the fourth data.

[0069] In this embodiment of the disclosure, the third data and its metadata can be compressed together to achieve deep compression of the data, while ensuring the consistency and integrity of the data and its metadata, and avoiding inconsistency problems caused by the separation of data and metadata.

[0070] In some embodiments, for the same data, the compression rate when the data is compressed in the foreground is greater than the compression rate when the data is compressed in the background, and the compression degree when the data is compressed in the foreground is less than the compression degree when the data is compressed in the background.

[0071] In this embodiment of the disclosure, for the same data, the front-end compression rate is high but the compression degree is low, which can quickly complete the initial data compression task without occupying too much front-end resources, so that the data can be quickly written to the preset storage area; while the back-end compression rate is relatively low, but a higher compression effect can be achieved through compression with a higher compression degree, thereby reducing the space occupation of the preset storage area and saving storage costs.

[0072] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0073] The first processing module is configured to compress the first data in the foreground to obtain the second data in response to the first data belonging to the first type of data, and store the second data in a preset storage area; the first data is the data to be written to the preset storage area, and the first type of data is the data whose impact on system performance is greater than a first impact threshold.

[0074] The second processing module is configured to compress the third data in the background in response to the third data belonging to the second type of data, to obtain the fourth data; the third data is the data associated with the first data stored in the preset storage area, and the second type of data is the data whose impact on system performance is greater than the second impact threshold.

[0075] In this embodiment, the write pressure on the electronic device can be reduced, and the bidirectional compression method in the foreground and background can reduce the storage space occupied by the electronic device, thereby improving the operating speed of the electronic device. Furthermore, compressing the first data in the foreground and compressing the third data associated with the first data in the background allows the data associated with the first data to be compressed on both the foreground and background sides, directly or indirectly reducing the storage space occupied on both sides and saving storage costs.

[0076] In some embodiments, the first processing module is further configured to:

[0077] In response to the fact that the data type of the first data is a preset data type, and / or that the access frequency of the first data is greater than a preset frequency threshold, it is determined that the first data belongs to the first type of data.

[0078] In this embodiment of the disclosure, data can be dynamically classified according to data type and access frequency, which can more accurately determine whether the first data meets the compression conditions, thereby enabling compression of data that meets the compression conditions, making the compression more targeted.

[0079] In some embodiments, the first processing module is further configured to:

[0080] Obtain the content information of the first data; the content information of the first data is used to characterize the degree of repetition of the content of the first data;

[0081] In response to the content information of the first data satisfying a preset condition, it is determined that the first data belongs to the first type of data; the content information of the first data satisfying the preset condition is used to characterize that the repetition degree of the content of the first data is greater than a preset repetition degree threshold.

[0082] In this embodiment of the disclosure, by analyzing the content information of the data, data with a high degree of repetition can be identified, allowing for more efficient compression. Furthermore, data with a high degree of repetition typically has greater compression potential, thus significantly reducing the data volume.

[0083] In some embodiments, the first processing module is further configured to:

[0084] In response to the content information including a preset content structure, and / or the content information including a redundancy level greater than a preset redundancy level threshold, it is determined that the content information of the first data satisfies the preset condition.

[0085] In this embodiment of the disclosure, data can be dynamically classified according to the content structure and redundancy of the content information, which can more accurately determine whether the first data meets the compression conditions, thereby enabling compression of data that meets the compression conditions, making the compression more targeted.

[0086] In some embodiments, the second processing module is further configured to:

[0087] The fourth data is stored in a preset physical block within the preset storage area; the preset physical block is used to store the compressed data in the background.

[0088] In this embodiment of the disclosure, the compressed data in the background can be stored in a preset physical block, so that all the compressed data in the background is stored in the same physical block. This can effectively reduce the space fragmentation of the storage space and reduce the space occupation of the storage space. In addition, for frequently accessed data, the data can be read more efficiently, effectively solving the problem of read amplification and improving the system response speed.

[0089] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0090] processor;

[0091] Memory used to store processor-executable instructions;

[0092] The processor is configured to perform the data processing method as described in the first aspect of this disclosure.

[0093] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the data processing method as described in the first aspect of the present disclosure.

[0094] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method as described in the first aspect of the present disclosure.

[0095] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0096] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0097] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0098] Figure 2A This is a schematic diagram of a data reading method according to an exemplary embodiment. Figure 1 .

[0099] Figure 2B This is a schematic diagram of a data reading method according to an exemplary embodiment.

[0100] Figure 2C This is a schematic diagram of a data reading method according to an exemplary embodiment. Figure 3 .

[0101] Figure 3 This is a flowchart illustrating a compression method according to an exemplary embodiment.

[0102] Figure 4 This is a flowchart illustrating the training process of a preset model according to an exemplary embodiment.

[0103] Figure 5 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0104] Figure 6 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0105] Figure 7 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment.

[0106] Figure 8 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0107] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0108] The data processing method provided in this disclosure is executed by an electronic device, which may specifically be a mobile phone, tablet computer, laptop, vehicle system, intelligent robot, smart wearable device, or other smart device.

[0109] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 1 The method includes the following steps:

[0110] In step S101, in response to the fact that the first data belongs to the first type of data, the first data is compressed in the foreground to obtain the second data, and the second data is stored in the preset storage area.

[0111] The first data is the data to be written to the preset storage area. In some embodiments, the first data can be obtained in response to the write operation. The write operation can be triggered by the user or electronic device, such as the user requesting the write operation through a graphical interface or command line tool.

[0112] In a foreground running scenario, the first data can include relevant data from the application running in the foreground. For example, if the application is a game, the first data could be data stored in a preset storage area during the game application's update process. The first data can be any type of data stream; for example, it can be a text data stream, an image data stream, or an audio data stream.

[0113] In some embodiments, the first type of data is data whose impact on system performance is greater than a first impact threshold. The first impact threshold is a pre-set threshold, and all other thresholds mentioned herein are also pre-set thresholds and will not be elaborated further below.

[0114] In some embodiments, the first type of data can represent data suitable for compression. Optionally, the first type of data can represent write-hot data obtained from the foreground. Write-hot data refers to data that is frequently written. Write-hot data has high requirements for system performance, and whether write-hot data can be successfully written is related to the system's operating performance. When the first data belongs to the first type of data, compressing the first data in the foreground can effectively compress data related to system performance, such as write-hot data, thereby compressing frequently written data, optimizing the writing experience, and reducing writing pressure. Optionally, for read-cold data, which refers to data that is not frequently read, the foreground may not need to compress this data and can directly read the original data.

[0115] In some embodiments, compressing the first data may include performing normal compression and performing deep compression. Normal compression refers to compressing the data's content itself, while deep compression refers to compressing both the data's content and its metadata. Metadata refers to the data's attribute information, such as the data's storage path, timestamp, and physical storage address on the disk.

[0116] The preset storage area can refer to any storage area in the electronic device. Optionally, the preset storage area can refer to the storage area on the disk of the electronic device.

[0117] In some embodiments, the first data includes multiple data blocks. Each data block can be compressed at the foreground based on a preset compression ratio, resulting in a compressed data block for each data block. The second data includes the compressed data blocks corresponding to each data block. Here, a data block can refer to the basic unit being compressed, and a data block can be represented as a character, number, or any other smallest unit of data that can be recognized and processed. The preset compression ratio of each data block can be used to characterize the expected compression level of each data block. For example, if the preset compression ratio of data block 1 is 0.5, it means that the expected compression level of data block 1 is to be compressed to half the size of the original data block. The space occupied by the compressed data block corresponding to a data block is negatively correlated with the preset compression ratio of the data block; that is, the lower the preset compression ratio of the data block, the larger the space occupied by the compressed data block corresponding to the data block; the higher the preset compression ratio of the data block, the smaller the space occupied by the compressed data block corresponding to the data block.

[0118] In some embodiments, the first data can be compressed in the foreground using any compression method to obtain the second data. In one example, a first encoding algorithm can be used to compress each data block among the multiple data blocks included in the first data in the foreground. The type of the first encoding algorithm can be set and selected based on actual needs. For example, the first encoding algorithm can be the Shannon-Fanno encoding algorithm.

[0119] Additionally, it should be noted that the original format of the input data can be image streams, audio streams, and video streams, etc. Before compressing the input data, the non-block format input data can be preprocessed to convert it into block format. In this case, the input data includes multiple blocks. For example, in one example, the image stream and audio stream can be in binary format. Before compressing the input data, Basic64 encoding can be used to convert the input data from binary format to block format.

[0120] With electronic devices increasingly larger storage spaces, excessive data writing can significantly consume these spaces, leading to decreased operating speed and performance. However, in this embodiment, the first data can be compressed in the foreground and stored in a preset storage area, effectively reducing the storage space occupied by written data. Furthermore, compressing the first data to be written to the preset storage area also effectively reduces writing pressure and accelerates the writing speed.

[0121] Furthermore, in some embodiments, in response to the first data belonging to a first type of data, the first data can be compressed in real time at the foreground to obtain the second data. Real-time compression refers to compressing data immediately during the writing or reading process. This compression method is suitable for scenarios requiring fast response, such as real-time data transmission, video streams, and audio streams. Thus, in this embodiment of the disclosure, by compressing the first data in real time at the foreground, the efficiency of compression and decompression can be improved, and data transmission delays can be avoided.

[0122] It should be noted that after storing the second data in the preset storage area, the space usage and storage address of the second data can be saved to the index structure of the electronic device. The index structure is used to store the space usage and storage address of the data. Furthermore, after storing the second data in the preset storage area, the metadata of the second data can also be updated. The metadata records the attribute information of the second data, which may include information such as the space usage, storage address, and storage time.

[0123] In some embodiments, the electronic device is provided with a front-end compression module, which can perform the process shown in step S101.

[0124] In step S102, in response to the fact that the third data belongs to the second type of data, the third data is compressed in the background to obtain the fourth data. The third data is the data associated with the first data stored in the preset storage area.

[0125] In some embodiments, the third data may refer to the second data that is associated with the first data and obtained after compression in the foreground; or, in some embodiments, the third data may refer to the data associated with the first data that is not compressed in the foreground; or, in some embodiments, the third data may refer to other data that is associated with the first data and obtained after compression in the foreground, different from the second data. For example, file 1 is used as the first data, and files 2 and 3 are associated with file 1. The electronic device compresses files 1 and 3 but not file 2, and then stores the compressed file 1, compressed file 2, and compressed file 3 in a preset storage area. At this time, the third data can be any of the compressed files 1, compressed file 3, and compressed file 2.

[0126] In some embodiments, the third data can be any type of data stream, such as a text data stream, an image data stream, or an audio data stream. Furthermore, the third data can also be an executable file. By compressing the executable file, space usage can be effectively reduced, and the read latency during application startup can be lowered, thereby improving application startup efficiency.

[0127] In some embodiments, the second type of data refers to data whose impact on system performance is greater than a second impact threshold. The second impact threshold may be the same as or different from the first impact threshold. In some embodiments, the second type of data may represent data suitable for compression. Optionally, the second type of data may be hot data, which refers to data that is frequently read, written, and modified, such as the same resource files used by different applications at startup. Hot data has high requirements for system performance, and whether hot data can be read and written smoothly is related to the system's operating performance. Optionally, the second type of data may include write hot data and / or read hot data. Additionally, cold data can also be compressed in the background, and the compressed cold data can be stored in a preset physical block.

[0128] Similarly, compressing third-party data can include performing normal compression and performing deep compression.

[0129] In some embodiments, the third data can be compressed using any compression method to obtain the fourth data, and the fourth data is stored in a preset storage area. The fourth data includes the compressed third data. In one example, a second encoding algorithm can be used to compress the third data to obtain the fourth data. The second encoding algorithm can be the same as or different from the first encoding algorithm. For example, the second encoding algorithm can be the Huffman coding algorithm. The process of compressing the third data using the second encoding algorithm will not be described in detail here.

[0130] In related technologies, data fragmentation exists in the storage space of electronic devices. When reading data from the storage space, multiple data points need to be read from multiple storage addresses, leading to read amplification. Read amplification refers to the situation where, because data is scattered across multiple locations, more data blocks need to be read to obtain the required data. In some embodiments, a fourth data point can be stored in a preset physical block within a preset storage area. This preset physical block is used to store data compressed in the background.

[0131] In other words, multiple files compressed in the background can be stored in the same physical block (the default physical block). For example... Figure 2A-2C The diagram shows one method of data reading. Figure 2A This indicates a method for reading uncompressed data. In this method, uncompressed data 1-4 needs to be read multiple times to obtain data 1-4; while Figure 2B This indicates a method for reading compressed data in related technologies. In this method, the compressed data 1-4 needs to be read multiple times to obtain data 1-4; that is, Figure 2A and Figure 2B The diagram illustrating data reading shows a problem with read magnification. And... Figure 2C This indicates the reading method shown in this disclosure. In this method, the compressed data 1-data 4 are stored in the same physical block. Therefore, data 1-data 4 can be obtained through a single read. Based on the above content and... Figures 2A-2C The diagram shows that storing multiple data in the same physical block can reduce storage fragmentation and space usage. In addition, for frequently accessed data such as third-party data, data can be read more efficiently, effectively solving the read amplification problem and improving the system's response speed.

[0132] It should be noted that after storing the fourth data in the preset storage area, the space occupied by the fourth data and its storage address can be saved to the electronic device's index structure. The index structure is used to store the space occupied and storage address of the data. Furthermore, after storing the fourth data in the preset storage area, the metadata of the fourth data can be updated. The metadata records the attribute information of the fourth data, which may include information such as the data's space occupied, storage address, and storage time. Moreover, the electronic device can determine the address mapping relationship between the storage addresses of the original data and the storage addresses of the compressed data. For example, if the third data is stored in the storage space corresponding to the first storage address, and the fourth data is stored in the storage space corresponding to the second storage address, the mapping relationship between the first and second storage addresses can be determined for use in the subsequent decompression process.

[0133] In some embodiments, in response to decompressing the fourth data to obtain the decompressed fourth data, the decompressed fourth data is stored in the storage area corresponding to the first storage address; the first storage address is the storage address of the third data. Correspondingly, when decompressing the fourth data, the storage address of the third data can be reused, and the decompressed fourth data is stored in the storage area corresponding to the storage address of the third data.

[0134] In some embodiments, in response to the fact that the third data belongs to the second type of data, the third data can be non-continuously compressed in the background to obtain the fourth data.

[0135] In some embodiments, the electronic device is provided with a background compression module, which can perform the process shown in step S102.

[0136] Additionally, it should be noted that if the third data is the second data, then step S102 can be executed after step S101; if the third data is other data obtained after compression in the foreground, or if the third data is data that has not been compressed in the foreground, step S102 can be executed before, after, or simultaneously with step S101.

[0137] In some embodiments, for the same data, the compression rate when compressing data in the foreground is greater than the compression rate when compressing data in the background, and the compression degree when compressing data in the foreground is less than the compression degree when compressing data in the background. For example, for the same data with a space occupation of 1MB, compression is performed in both the foreground and the background. The time taken to compress the data in the foreground is 0.5s, while the time taken to compress the data in the background is 1s, that is, the compression rate of the foreground compression is greater than the compression rate of the background compression; the space occupation of the 1MB data after foreground compression is 500KB, while the space occupation of the 1MB data after background compression is 450KB.

[0138] Thus, for the same data, the front-end compression rate is high but the compression degree is low, which can quickly complete the initial data compression task without occupying too much front-end resources, so that the data can be quickly written to the preset storage area; while the back-end compression rate is relatively low, but it can achieve a higher compression effect through compression with a higher compression degree, thereby reducing the space occupied by the preset storage area and saving storage costs.

[0139] The method provided in this disclosure can compress the first data (which belongs to a first type of data) in the foreground and store the second data in a preset storage area, thereby reducing the write pressure on the electronic device and reducing the storage space occupied by the electronic device. When the third data belongs to a second type of data, it is compressed in the background to obtain fourth data, which is data associated with the first data stored in the preset storage area. This bidirectional compression method, using both foreground and background compression, reduces the storage space occupied by the electronic device and improves its operating speed. Furthermore, compressing the first data in the foreground and the associated third data in the background allows for compression processing of the data associated with the first data on both the foreground and background sides, directly or indirectly reducing storage space occupied on both sides and saving storage costs.

[0140] In some embodiments, the first data can be determined to belong to the first type of data in the following three ways. The three ways are described below.

[0141] Method 1: In response to the fact that the data type of the first data is a preset data type, and / or the access frequency of the first data is greater than a preset frequency threshold, determine that the first data belongs to the first type of data.

[0142] The preset data types can include application installation package data types, log data types, and database data types. The access frequency of the first data can be determined by the number of times the first data is read and written. If the first access frequency is greater than a preset frequency threshold, it indicates that the first data is data with a relatively high number of read and write operations.

[0143] Method 2: Obtain the content information of the first data; the content information of the first data is used to characterize the degree of repetition of the content of the first data; in response to the content information of the first data meeting the preset conditions, determine that the first data belongs to the first type of data; the content information of the first data meeting the preset conditions is used to characterize that the degree of repetition of the content of the first data is greater than the preset repetition threshold.

[0144] In some embodiments, the content information of the first data may refer to information related to the actual content of the first data, and the degree of repetition of the content may refer to the repetition rate of the content or other parameters related to the degree of repetition. In response to the fact that the degree of repetition of the content of the first data is greater than a preset repetition threshold, it can be determined that the first data belongs to a first type of data. For example, in response to the fact that the repetition rate of the content of the first data is greater than the preset repetition threshold, it can be determined that the first data belongs to a first type of data.

[0145] In some embodiments, in response to the content structure included in the content information being a preset content structure, and / or the redundancy included in the content information being greater than a preset redundancy threshold, it is determined that the content information of the first data meets the preset conditions.

[0146] In one example, the content structure of the first data can characterize the organization and logical relationships of the content represented by the first data. Optionally, the organization can include parameters such as hierarchical structure, relational structure, sequential structure, and network structure. For example, a hierarchical structure can be a tree structure, a relational structure can represent a table structure in a database, a sequential structure can represent the structure of an array or linked list, and a network structure can represent a social network or topology. For example, logical relationships can include the relationships between data, such as one-to-one, one-to-many, and many-to-many relationships. In some embodiments, the redundancy of the first data can refer to the proportion of duplicate or unnecessary information in the first data.

[0147] In one example, the preset content structure can represent any type of complex content result. Therefore, when the data content structure is too complex, and / or the data redundancy is too high, it will affect the data transmission efficiency and data processing efficiency. In this case, if it is determined that the first data belongs to the first type of drama, the first data can be compressed on the front end, thereby reducing the complexity and redundancy of the data content structure, improving the data transmission and processing efficiency, and accelerating the data writing speed.

[0148] In some embodiments, a preset cache page of first data is obtained; the preset cache page includes at least a portion of the content of the first data; based on the preset cache page, the content information of the first data is determined. The preset cache page includes at least a portion of the content of the first data, and the electronic device can determine the content information of the first data based on this at least portion. Optionally, the preset cache page can be the first cache page of the first data. In some embodiments, the at least portion of the content may include some header information or key metadata of the first data.

[0149] Method 3: If the data type of the first data is a preset data type, and / or the access frequency of the first data is greater than a preset frequency threshold, and in response to the content structure of the first data being a preset content structure, and / or the redundancy of the first data being greater than a preset redundancy threshold, determine that the first data belongs to the first type of data. For details, please refer to Methods 1 and 2 above, which will not be repeated here.

[0150] In some embodiments, the following situations can be understood as having a greater impact on system performance than the first impact threshold: the data type of the first data is a preset data type, the access frequency of the first data is greater than a preset frequency threshold, the content structure of the content information is a preset content structure, and / or the redundancy of the content information is greater than a preset redundancy threshold.

[0151] It should be noted that the process of determining that the third data belongs to the second type of data can also be carried out using any of the three methods mentioned above, and the specific process will not be elaborated here.

[0152] In this embodiment of the disclosure, data can be classified according to data type, access frequency, content structure and / or redundancy of content information. This allows for a more accurate determination of whether the first data meets the compression conditions, thereby enabling compression of data that meets the compression conditions, making the compression more targeted.

[0153] The following section uses the Shannon-Vanno coding algorithm as an example to briefly explain the compression process of input data.

[0154] like Figure 3 The flowchart shown is a compression method. Figure 3 The flowchart shown includes the following steps: acquiring input data; converting the input data using the Shannon-Fanno coding algorithm to obtain compressed data blocks corresponding to the data blocks in the input information; encoding the input data based on the compressed data blocks corresponding to the data blocks in the input information; and performing regularization processing on the encoding.

[0155] In this context, the compressed data block can refer to a codeword, which is a binary encoding sequence assigned to the data block.

[0156] The input data is transformed using the Shannon-Fanno coding algorithm to obtain compressed data blocks corresponding to the data blocks in the input information, including:

[0157] The process involves determining the frequency of each data block within the input data, sorting the data blocks by frequency from highest to lowest, resulting in a frequency set. This frequency set is then bisected (dividing it into two subsets with approximately equal frequencies). The first subset is assigned a value of 0, and the second subset a value of 1. This process is repeated until each subset contains only one element. The values ​​of the subsets to which each element belongs are then summarized to obtain the compressed data block corresponding to each data block. The space occupied by the compressed data block is negatively correlated with the frequency of the data block, and positively correlated with the preset compression ratio. In other words, a higher frequency of a data block results in a lower space occupied by the compressed data block and a higher preset compression ratio.

[0158] For example, the input data includes four data blocks A, D, and E, with data block A having a frequency of 0.3, data block B having a frequency of 0.1, data block C having a frequency of 0.4, and data block D having a frequency of 0.2. Sorting these in descending order yields the character sequence: data block C (0.4), data block A (0.3), data block D (0.2), and data block B (0.1). A binary search method is used for segmentation: In the first segmentation, data block C is grouped separately, while data blocks A, D, and B are grouped together. Data block C is assigned codeword 0, and data blocks A, D, and B are assigned codeword 1. In the second segmentation, data block A is grouped separately, while data blocks D and B are grouped together. Data block A is assigned codeword 0, and data blocks D and B are assigned codeword 1. In the third segmentation, data block D is assigned codeword 0, and data block B is assigned codeword 1. By summarizing the value of each character, we can obtain the codeword for data block C as 0, the codeword for data block A as 10, the codeword for data block D as 110, and the codeword for data block B as 111.

[0159] The following is a brief explanation of how to encode input data based on the compressed data blocks corresponding to the data blocks in the input information.

[0160] In some embodiments, the second data can be determined based on the compressed data block corresponding to each data block. For example, the final encoding can be determined based on the codeword corresponding to each data block. For example, if the first data is "ADCACCDBA C", then the encoding of the first data (the second data) can be 1011001000110111100.

[0161] The following is a brief explanation of the regularization process for encoding. After obtaining the encoding corresponding to the first data (i.e., the second data), the encoding corresponding to the first data can be regularized to align the codewords corresponding to each data block to a preset length, such as 16 bits for 8 bits, thus facilitating the storage and retrieval of the second data. For codewords whose length is less than the preset length, zeros can be added to the end of the codeword to make it reach the preset length. For example, if the preset length is 8 bits, and the codeword of a certain data block is 0111, zeros can be added to the end of the codeword to obtain the processed codeword 01110000, and the length of the processed codeword meets the preset length.

[0162] In some embodiments, the first data can be compressed using the Shannon-Fanno coding algorithm to obtain the second data. Accordingly, for data blocks that occur frequently in the first data, the compressed data block has a low space requirement and a high preset compression ratio; conversely, for data blocks that occur infrequently in the first data, the compressed data block has a high space requirement and a low preset compression ratio. Thus, by allocating compressed data blocks with different space requirements to data blocks of different frequencies, the amount of data in the compressed data block can be effectively reduced, achieving a higher compression ratio.

[0163] The method provided in this disclosure describes the process of determining second data based on the Shannon-Vanno coding algorithm and first data. By using the Shannon-Vanno coding algorithm, a corresponding compressed data block can be assigned to each data block among multiple data blocks in the first data, thereby compressing the first data and obtaining the second data.

[0164] In some embodiments, a preset model can be determined to save compression time. In some embodiments, if the preset model is a model based on the Shannon-Fanno coding algorithm, the preset model can be used to determine the compressed data block corresponding to each data block based on the frequency of occurrence of the input data block in the target data, and to determine the output data based on the compressed data block corresponding to each data block. The target data includes at least the first data, and may also include other historical input data besides the first data. The space occupancy of the compressed data block corresponding to the data block is negatively correlated with the frequency of occurrence of the data block, and the frequency of occurrence of the data block is positively correlated with the preset compression ratio of the data block.

[0165] like Figure 4The flowchart shown illustrates the training process of a preset model. During the training process, multiple historical input data can be acquired, and features of each historical input data can be extracted. The features of the historical input data include the frequency of occurrence of each data block in the multiple data blocks included in the historical input data. The preset model is constructed based on the features of the multiple historical input data, and Shannon-Fanno coding compression is performed based on the constructed preset model.

[0166] In some embodiments, constructing a pre-defined model based on features of multiple historical input data may include: determining the compressed data block corresponding to each data block based on the frequency of occurrence of each data block in the multiple historical data, thereby obtaining a pre-defined encoding table, and then constructing a pre-defined model based on the pre-defined encoding table. The pre-defined encoding table can characterize the correspondence between the frequency of occurrence of data blocks and the corresponding compressed data blocks. During training, the correspondence between data blocks and their corresponding compressed data blocks in the pre-defined encoding table can be continuously corrected based on the input historical input data, thereby reducing the parameter error of the model. In this way, the frequency of occurrence of data blocks and their corresponding compressed data blocks can be predicted more accurately, thereby constructing a more accurate encoding table.

[0167] In the actual encoding process, the first data can be directly input into the preset model to obtain the second data corresponding to the first data; or, the third data can be input into the preset model to obtain the fourth data. In one example, after the first data is input into the preset model, the preset model can determine the compressed data block corresponding to each data block of the first data based on the preset encoding table, and output the second data based on the compressed data block corresponding to each data block.

[0168] Additionally, it's important to note that as the input data changes, the compressed data blocks corresponding to data blocks may also change. Accordingly, the preset model can calculate the frequency of occurrence of each data block in real time based on the input data, and update the compressed data blocks and preset encoding table for each data block in real time based on the frequency of occurrence. In this way, when the compressed data block of a certain data block changes, the preset encoding table can be updated, thereby updating the preset model. For example, after inputting the current input data, if the frequency of occurrence of data block A increases, the compressed data block of data block A may change.

[0169] The method provided in this disclosure can quickly determine the compressed data block corresponding to each data block in multiple data blocks of input data based on a preset module, thereby achieving rapid compression of input data and accelerating compression efficiency. Furthermore, by monitoring the input data in real time and updating the preset encoding table and preset model, more accurate compression results can be obtained during the actual compression process, and the computational cost of compression can be saved, achieving rapid compression and improving compression efficiency.

[0170] In some embodiments, the third data includes fifth data, which characterizes data that has already been compressed in the foreground and can be further compressed. The process of determining to compress the fifth data is described below.

[0171] In some embodiments, in response to the fifth data belonging to the second type of data and the compression degree of the fifth data being less than a preset compression degree threshold, the fifth data is compressed in the background to obtain compressed fifth data; the fourth data includes the compressed fifth data.

[0172] In some embodiments, the compression level is used to characterize whether the fifth data is thoroughly / completely compressed. It should be noted that when the electronic device is under high load, the fifth data obtained by compression in the foreground may be a temporary compression result, still with room for further compression. In this case, when scanning data in the preset storage area in the background, if the fifth data is detected as data that has already been compressed in the foreground, and the compression level of the fifth data is less than the preset compression level threshold, further compression can be performed on the fifth data in the background, i.e., secondary compression, to ensure a higher compression level and reduce memory usage. For example, when an electronic device writes a 1MB file, a compression algorithm (such as the Shannon-Vanno coding algorithm) is used in the foreground to compress the file, reducing the file size to 500KB. The compressed file is stored in the preset storage area of ​​the disk. Then, the files on the disk are scanned in the background to find files with lower compression levels that can be further optimized, and files that have been compressed in the foreground (such as the 500KB file mentioned above) are found. At this point, the 500KB file can be compressed in the background. In one example, Huffman coding can be used to further compress a file that has already been compressed in the foreground, reducing the file size to 450KB after compression in the background. At this point, the compression level of the file after background compression is greater than or equal to a preset compression threshold.

[0173] In some embodiments, the third data further includes the sixth data. The fifth data is used to characterize data that is not compressed in the foreground. In this case, in response to the sixth data belonging to the second type of data and the compression degree of the sixth data being less than a preset compression degree threshold, the sixth data is compressed in the background to obtain the compressed sixth data; the fourth data includes the compressed sixth data.

[0174] The method provided in this disclosure can compress the data in the background again when the data compressed in the foreground is not fully compressed, thereby further reducing the space occupied by the electronic device and improving the operating speed of the electronic device.

[0175] In some embodiments, when compression is performed in the foreground and in the background, the decision to perform compression in the foreground or in the background can be based on resource utilization. The following describes the cases of compressing first data in the foreground and compressing third data in the background, as well as the compression process for compressing first data in the foreground and third data in the background.

[0176] For the front desk:

[0177] In some embodiments, in response to a first resource utilization rate satisfying a first resource utilization condition, the first data is compressed in the foreground to obtain the second data. The resource utilization rate can characterize the ratio of the actual amount of resources used to the total amount of available resources.

[0178] In some embodiments, in response to the load being greater than a first load threshold, and / or in response to the space occupancy of the first data being greater than a first space occupancy threshold, it is determined that the first resource utilization rate meets the first resource utilization condition.

[0179] In some embodiments, background compression requires significant computational resources; therefore, it needs to be performed when the electronic device's load is low. In response to a load exceeding a first load threshold, a first resource utilization rate can be determined to meet a first resource utilization condition, allowing for foreground compression of the first data to obtain the second data. In some embodiments, a load exceeding the first load threshold indicates high utilization of the electronic device's central controller. When the load is high, the background compression process cannot be executed, thus triggering the foreground compression process. By performing compression in the foreground, write speed can be improved and write pressure reduced.

[0180] In some embodiments, when the space usage of a certain data is high, it indicates that the data needs to be compressed. Therefore, in response to the space usage of the first data exceeding a first space usage threshold, the first data can be compressed in the foreground to obtain the second data. In this way, data with high space usage can be compressed, thereby saving space usage in the preset storage area.

[0181] In some embodiments, the decision to perform normal compression or deep compression on the data can be based on the data's space footprint.

[0182] In some embodiments, in response to the space occupancy of the first data being greater than a first space occupancy threshold, the first data and its metadata are compressed in the foreground to obtain second data; the second data includes the compressed first data and its metadata.

[0183] It should be noted that during the compression of the first data and its metadata, checksum information can be calculated for the first data. This checksum information is then used to verify the data against the checksum information used in the subsequent decompression process, ensuring the consistency between the decompressed and compressed data. Optionally, the checksum information can be a checksum, which is a fixed-size value calculated from the data using a specific algorithm.

[0184] In addition, in deep compression scenarios, a new index structure can be established to represent the location of the first compressed data and the location of the metadata of the first compressed data.

[0185] For the backend:

[0186] In some embodiments, in response to the second resource utilization rate meeting the second resource utilization condition, the third data is compressed in the background to obtain the fourth data.

[0187] In some embodiments, in response to a load less than or equal to a second load threshold, and / or in response to a load of an electronic device less than or equal to a second load threshold, the third data is compressed in the background to obtain the fourth data.

[0188] In some embodiments, compressing data in the background requires significant computing resources. Therefore, when the load on the electronic device is low, a second resource utilization rate is determined to meet a second resource utilization condition, thereby initiating the background compression process. It should be noted that the second load threshold may be the same as or different from the first load threshold.

[0189] In some embodiments, when the space usage of a certain data is high, it indicates that the data needs to be compressed. Therefore, in response to the space usage of the third data exceeding the second space usage threshold, the third data can be compressed in the background to obtain the fourth data. In this way, data with high space usage can be compressed, thereby saving space usage in the preset storage area.

[0190] In some embodiments, the appropriate compression method (normal or deep) for the data can be determined based on its space occupancy. In some embodiments, in response to the third data's space occupancy exceeding a second space occupancy threshold, the third data and its metadata are compressed in the background to obtain fourth data; the fourth data includes the compressed third data and its metadata. It should be noted that the second space occupancy threshold may be the same as or different from the first space occupancy threshold.

[0191] In addition, for other related content in the backend deep compression scenario, please refer to the content shown in the frontend deep compression scenario above, which will not be repeated here.

[0192] The method provided in this disclosure determines under what circumstances first data is compressed in the foreground and third data is compressed in the background. Thus, when certain conditions are met, both foreground and background compression processes can be executed, achieving bidirectional data compression. Furthermore, specific details regarding ordinary or deep compression of data are provided. In deep compression scenarios, data is compressed deeply to further reduce its space footprint, thereby saving memory resources of electronic devices.

[0193] In some embodiments, the fourth data is used as an example below, through Figure 5 The illustrated example provides a simple explanation of the decompression process for compressed data.

[0194] Figure 5 This is a flowchart illustrating a data processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 5 The method includes the following steps:

[0195] Step S501: In response to the decompression operation of the fourth data, the decompressed fourth data is obtained.

[0196] In some embodiments, in response to a read operation on the fourth data, the fourth data can be decompressed. It should be noted that when reading the fourth data, the storage address of the fourth data can be obtained through an index structure, and the fourth data can be read based on its storage address.

[0197] In some embodiments, based on the compression scenario of the third data, the corresponding decompression method can be determined. For example, in response to a normal compression scenario, a decompression operation is performed on the fourth data to obtain the decompressed fourth data, i.e., the third data.

[0198] In some embodiments, in response to the compression scenario of the third data being a deep compression scenario, verification information can be added during the compression process. This verification information is used to verify the integrity and accuracy of the information. Correspondingly, during decompression, in response to performing a decompression operation on the fourth data and obtaining the decompression result of the fourth data, the decompression result of the fourth data is verified based on the verification information corresponding to the fourth data. This verification information is generated during the compression process of the third data. In response to the decompression result of the fourth data passing the verification, the decompression result of the fourth data is determined as the decompressed fourth data.

[0199] In some embodiments, first verification information of the third data during the compression process of the third data can be obtained, and second verification information of the decompression result of the fourth data during the decompression process of the fourth data can be obtained. If the first verification information and the second verification information are consistent, the verification is determined to be successful; otherwise, if the first verification information and the second verification information are inconsistent, the verification is determined to be unsuccessful.

[0200] Step S502: The decompressed fourth data is stored in the storage area corresponding to the first storage address; the first storage address is the storage address of the third data.

[0201] In some embodiments, after obtaining the fourth data, the electronic device may delete the third data or not. In one example, in response to new data being stored in the storage area corresponding to the first storage address, the new data overwrites the third data.

[0202] In some embodiments, the first storage address can be determined by the address mapping relationship between the storage address of the original data and the storage address of the compressed data.

[0203] In one example, in response to the situation where no new data is stored in the storage area corresponding to the first storage address after the fourth data is obtained, the decompressed fourth data is stored in the storage area corresponding to the first storage address. That is, when no new data is stored in the storage area corresponding to the first storage address after the fourth data is obtained, the address of the first storage address can be reused during the decompression process of the fourth data, and the decompressed fourth data can be stored in the storage area corresponding to the first storage area.

[0204] In response to the fact that new data is stored in the storage area corresponding to the first storage address after the fourth data is obtained, the decompressed fourth data is stored in the storage area corresponding to other storage addresses other than the first storage address.

[0205] The method provided in this disclosure can verify the accuracy of the decompressed data using verification information during the decompression process of deeply compressed data, thereby ensuring the integrity and correctness of the data. Furthermore, the decompressed data can be stored at the address of the original data, achieving address reuse.

[0206] Optionally, the effectiveness of the data processing method provided in the embodiments of this disclosure can be evaluated. In one example, the same test data can be compressed using both the data processing method in related technologies and the data processing method provided in this disclosure in the same electronic device, and the compressed test data obtained by the two processing methods can be acquired. The space occupied by the compressed test data corresponding to the data processing method provided in this disclosure is less than the space occupied by the compressed test data corresponding to the data processing method in related technologies. In addition, the time required to read the same compressed test data can also be acquired. The reading time corresponding to the data processing method provided in this disclosure is less than the reading time corresponding to the data processing method in related technologies.

[0207] Figure 6 This is a flowchart illustrating a data processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 6 The method includes the following steps:

[0208] Step S601: In response to the fact that the data type of the first data is a preset data type, the access frequency of the first data is greater than a preset frequency threshold, the content structure of the first data is a preset content structure, and / or the redundancy of the first data is greater than a preset redundancy threshold, it is determined that the first data belongs to the first type of data.

[0209] In step S602, in response to the fact that the first data belongs to the first type of data, the first data is input into the preset model to obtain the second data, and the second data is stored in the preset storage area.

[0210] In step S603, in response to the electronic device's load being less than or equal to the second load threshold and the third data belonging to the second type of data, the third data is compressed in the background to obtain the fourth data, and the fourth data is stored in a preset physical block in a preset storage area.

[0211] Step S604: In response to the decompression operation of the fourth data, the decompressed fourth data is obtained.

[0212] In step S605, in response to the fact that no new data is stored in the storage area corresponding to the first storage address after obtaining the fourth data, the decompressed fourth data is stored in the storage area corresponding to the first storage address.

[0213] Figure 7This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment, configured in an electronic device, see [link to relevant documentation]. Figure 7 The device includes:

[0214] The first processing module 701 is configured to compress the first data in the foreground to obtain the second data in response to the first data belonging to the first type of data, and store the second data in a preset storage area; the first data is the data to be written to the preset storage area, and the first type of data is the data whose impact on system performance is greater than the first impact threshold.

[0215] The first processing module 701 is configured to compress the third data in the background in response to the third data belonging to the second type of data, to obtain the fourth data; the third data is the data associated with the first data stored in the preset storage area, and the second type of data is the data whose impact on system performance is greater than the second impact threshold.

[0216] In some embodiments, the first processing module 701 is further configured to:

[0217] In response to the fact that the data type of the first data is a preset data type, and / or the access frequency of the first data is greater than a preset frequency threshold, it is determined that the first data belongs to the first type of data.

[0218] In some embodiments, the first processing module 701 is further configured to:

[0219] Obtain the content information of the first data; the content information of the first data is used to characterize the degree of repetition of the content of the first data;

[0220] In response to the content information of the first data satisfying the preset conditions, it is determined that the first data belongs to the first type of data; the content information of the first data satisfying the preset conditions is used to characterize that the repetition degree of the content of the first data is greater than the preset repetition degree threshold.

[0221] In some embodiments, the first processing module 701 is further configured to:

[0222] In response to the content information including a preset content structure, and / or the redundancy of the content information being greater than a preset redundancy threshold, it is determined that the content information of the first data satisfies the preset conditions.

[0223] In some embodiments, the first processing module 701 is further configured to:

[0224] Retrieve a preset cache page of the first data; the preset cache page includes at least a portion of the first data.

[0225] Based on the preset cache page, determine the content information of the first data.

[0226] In some embodiments, the first data includes multiple data blocks, and the first processing module 701 is configured to:

[0227] Based on the preset compression rate of each data block in multiple data blocks, each data block is compressed in the foreground to obtain the compressed data block corresponding to each data block; the second data includes the compressed data block corresponding to each data block, and the space occupied by the compressed data block corresponding to the data block is negatively correlated with the preset compression rate of the data block.

[0228] In some embodiments, the first processing module 701 is configured to:

[0229] Each data block is input into a preset model, and the compressed data block corresponding to each data block is output. The preset model is used to determine the compressed data block corresponding to each data block based on the frequency of occurrence of the input data block in the target data, and to determine the output data based on the compressed data block corresponding to each data block. The target data includes at least the first data. The space occupied by the compressed data block corresponding to the data block is negatively correlated with the frequency of occurrence of the data block, and the frequency of occurrence of the data block is positively correlated with the preset compression ratio of the data block. The preset model includes the correspondence between the frequency of occurrence of the data block and the compressed data block corresponding to the data block.

[0230] In some embodiments, the third data includes fifth data, which is used to characterize data that has been compressed in the foreground, and the second processing module 702 is configured to:

[0231] Since the fifth data belongs to the second type of data and the compression level of the fifth data is less than the preset compression level threshold, the fifth data is compressed in the background to obtain the compressed fifth data.

[0232] In some embodiments, the second processing module 702 is further configured to:

[0233] The fourth data is stored in a preset physical block within a preset storage area; the preset physical block is used to store the compressed data in the background.

[0234] In some embodiments, the second processing module 702 is further configured to:

[0235] In response to the decompression operation on the fourth data, the decompressed fourth data is obtained;

[0236] The decompressed fourth data is stored in the storage area corresponding to the first storage address; the first storage address is the storage address of the third data.

[0237] In some embodiments, the second processing module 702 is configured to:

[0238] In response to the fact that no new data is stored in the storage area corresponding to the first storage address after the fourth data is obtained, the decompressed fourth data is stored in the storage area corresponding to the first storage address.

[0239] In some embodiments, the second processing module 702 is configured to:

[0240] In response to the decompression operation on the fourth data, the decompression result of the fourth data is obtained;

[0241] Based on the verification information corresponding to the fourth data, the decompression result of the fourth data is verified; the verification information is generated during the compression of the third data.

[0242] In response to the successful verification of the decompression result of the fourth data, the decompression result of the fourth data is determined as the decompressed fourth data.

[0243] In some embodiments, the first processing module 701 is configured to:

[0244] In response to the first resource utilization rate meeting the first resource utilization condition, the first data is compressed on the front end to obtain the second data.

[0245] In some embodiments, the first processing module 701 is further configured to:

[0246] In response to the load being greater than the first load threshold, it is determined that the first resource utilization rate meets the first resource utilization condition.

[0247] In some embodiments, the first processing module 701 is further configured to:

[0248] In response to the fact that the space occupied by the first data is greater than the first space occupied threshold, it is determined that the first resource utilization rate meets the first resource utilization condition.

[0249] In some embodiments, the first processing module 701 is configured to:

[0250] The first data and its metadata are compressed on the front end to obtain the second data.

[0251] In some embodiments, the second processing module 702 is configured to:

[0252] In response to the second resource utilization rate meeting the second resource utilization condition, the third data is compressed in the background to obtain the fourth data.

[0253] In some embodiments, the second processing module 702 is further configured to:

[0254] In response to a load less than or equal to a second load threshold, it is determined that the second resource utilization rate meets the second resource utilization condition.

[0255] In some embodiments, the second processing module 702 is further configured to:

[0256] In response to the fact that the space occupancy of the third data is greater than the second space occupancy threshold, it is determined that the second resource utilization rate meets the second resource utilization condition.

[0257] In some embodiments, the second processing module 702 is configured to:

[0258] The third data and its metadata are compressed in the background to obtain the fourth data.

[0259] In some embodiments, for the same data, the compression rate of data compression in the foreground is greater than the compression rate of data compression in the background, and the compression degree of data compression in the foreground is less than the compression degree of data compression in the background.

[0260] In some embodiments, for the same data, the compression rate of data compression in the foreground is greater than the compression rate of data compression in the background, and the compression degree of data compression in the foreground is less than the compression degree of data compression in the background.

[0261] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0262] This disclosure also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the data processing method described above.

[0263] Figure 8 This is a block diagram of an electronic device 800 according to an exemplary embodiment.

[0264] Reference Figure 8 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0265] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0266] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0267] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0268] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0269] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0270] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0271] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0272] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0273] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0274] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0275] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a data processing method provided by an exemplary embodiment of the present disclosure.

[0276] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the above embodiments.

[0277] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0278] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: In response to the fact that the first data belongs to the first type of data, the first data is compressed in the foreground to obtain the second data, and the second data is stored in a preset storage area; the first data is the data to be written to the preset storage area, and the first type of data is the data whose impact on system performance is greater than a first impact threshold. In response to the fact that the third data belongs to the second type of data, the third data is compressed in the background to obtain the fourth data; the third data is the data associated with the first data stored in the preset storage area, and the second type of data is the data whose impact on system performance is greater than the second impact threshold.

2. The data processing method according to claim 1, characterized in that, The method further includes: In response to the fact that the data type of the first data is a preset data type, and / or that the access frequency of the first data is greater than a preset frequency threshold, it is determined that the first data belongs to the first type of data.

3. The data processing method according to claim 2, characterized in that, The method further includes: Obtain the content information of the first data; the content information of the first data is used to characterize the degree of repetition of the content of the first data; In response to the content information of the first data satisfying a preset condition, it is determined that the first data belongs to the first type of data; the content information of the first data satisfying the preset condition is used to characterize that the repetition degree of the content of the first data is greater than a preset repetition degree threshold.

4. The data processing method according to claim 3, characterized in that, The method further includes: In response to the content information including a preset content structure, and / or the content information including a redundancy level greater than a preset redundancy level threshold, it is determined that the content information of the first data satisfies the preset condition.

5. The data processing method according to claim 3 or 4, characterized in that, The method further includes: Obtain a preset cache page of the first data; the preset cache page includes at least a portion of the content of the first data; Based on the preset cache page, the content information of the first data is determined.

6. The data processing method according to claim 1, characterized in that, The first data includes multiple data blocks, and the compression of the first data in the foreground to obtain the second data includes: Based on the preset compression rate of each data block in the plurality of data blocks, each data block is compressed in the foreground to obtain the compressed data block corresponding to each data block; the second data includes the compressed data block corresponding to each data block, and the space occupied by the compressed data block corresponding to the data block is negatively correlated with the preset compression rate of the data block.

7. The data processing method according to claim 6, characterized in that, The step of compressing each data block at a preset compression rate based on the plurality of data blocks in the foreground to obtain a compressed data block corresponding to each data block includes: Each data block is input into a preset model, and the compressed data block corresponding to each data block is output. The preset model is used to determine the compressed data block corresponding to each data block based on the frequency of occurrence of the input data block in the target data, and to determine the output data based on the compressed data block corresponding to each data block. The target data includes at least the first data. The space occupancy of the compressed data block corresponding to each data block is negatively correlated with the frequency of occurrence of the data block, and the frequency of occurrence of the data block is positively correlated with the preset compression ratio of the data block. The preset model includes the correspondence between the frequency of occurrence of the data block and the compressed data block corresponding to the data block.

8. The data processing method according to claim 1, characterized in that, The third data includes fifth data, which is used to characterize data that has already been compressed in the foreground. In response to the third data belonging to the second type of data, the third data is compressed in the background to obtain fourth data, including: In response to the fact that the fifth data belongs to the second type of data and the compression degree of the fifth data is less than the preset compression degree threshold, the fifth data is compressed in the background to obtain the compressed fifth data.

9. The data processing method according to claim 1, characterized in that, The method further includes: The fourth data is stored in a preset physical block within the preset storage area; the preset physical block is used to store the compressed data in the background.

10. The data processing method according to claim 1, characterized in that, The method further includes: In response to the decompression operation of the fourth data, the decompressed fourth data is obtained; The decompressed fourth data is stored in the storage area corresponding to the first storage address; the first storage address is the storage address of the third data.

11. The data processing method according to claim 10, characterized in that, The step of storing the decompressed fourth data into the storage area corresponding to the first storage address includes: In response to the fact that no new data is stored in the storage area corresponding to the first storage address after obtaining the fourth data, the decompressed fourth data is stored in the storage area corresponding to the first storage address.

12. The data processing method according to claim 10, characterized in that, The step of decompressing the fourth data to obtain the decompressed fourth data includes: In response to performing the decompression operation on the fourth data, the decompression result of the fourth data is obtained; Based on the verification information corresponding to the fourth data, the decompression result of the fourth data is verified; the verification information is the information generated during the compression process of the third data. In response to the successful verification of the decompression result of the fourth data, the decompression result of the fourth data is determined as the decompressed fourth data.

13. The data processing method according to claim 1, characterized in that, The step of compressing the first data in the foreground to obtain the second data includes: In response to the first resource utilization rate meeting the first resource utilization condition, the first data is compressed on the front end to obtain the second data.

14. The data processing method according to claim 13, characterized in that, The method further includes: In response to the load being greater than a first load threshold, it is determined that the first resource utilization rate meets the first resource utilization condition.

15. The data processing method according to claim 13, characterized in that, The method further includes: In response to the fact that the space occupancy of the first data is greater than the first space occupancy threshold, it is determined that the first resource utilization rate meets the first resource utilization condition.

16. The data processing method according to claim 1, characterized in that, The step of compressing the first data in the foreground to obtain the second data includes: The first data and its metadata are compressed on the front end to obtain the second data.

17. The data processing method according to claim 1, characterized in that, The process of compressing the third data in the background to obtain the fourth data includes: In response to the second resource utilization rate meeting the second resource utilization condition, the third data is compressed in the background to obtain the fourth data.

18. The data processing method according to claim 17, characterized in that, The method further includes: In response to a load less than or equal to a second load threshold, it is determined that the second resource utilization rate meets the second resource utilization condition.

19. The data processing method according to claim 17, characterized in that, The method further includes: In response to the fact that the space occupancy of the third data is greater than the second space occupancy threshold, it is determined that the second resource utilization rate meets the second resource utilization condition.

20. The data processing method according to claim 1, characterized in that, The process of compressing the third data in the background to obtain the fourth data includes: The third data and its metadata are compressed in the background to obtain the fourth data.

21. The data processing method according to claim 1, characterized in that, For the same data, the compression rate when the data is compressed in the foreground is greater than the compression rate when the data is compressed in the background, and the compression degree when the data is compressed in the foreground is less than the compression degree when the data is compressed in the background.

22. A data processing apparatus, characterized in that, include: The first processing module is configured to compress the first data in the foreground to obtain the second data in response to the first data belonging to the first type of data, and store the second data in a preset storage area; the first data is the data to be written to the preset storage area, and the first type of data is the data whose impact on system performance is greater than a first impact threshold. The second processing module is configured to compress the third data in the background in response to the third data belonging to the second type of data, to obtain the fourth data; the third data is the data associated with the first data stored in the preset storage area, and the second type of data is the data whose impact on system performance is greater than the second impact threshold.

23. The data processing apparatus according to claim 22, characterized in that, The first processing module is further configured to: In response to the fact that the data type of the first data is a preset data type, and / or that the access frequency of the first data is greater than a preset frequency threshold, it is determined that the first data belongs to the first type of data.

24. The data processing apparatus according to claim 23, characterized in that, The first processing module is further configured to: Obtain the content information of the first data; the content information of the first data is used to characterize the degree of repetition of the content of the first data; In response to the content information of the first data satisfying a preset condition, it is determined that the first data belongs to the first type of data; the content information of the first data satisfying the preset condition is used to characterize that the repetition degree of the content of the first data is greater than a preset repetition degree threshold.

25. The data processing apparatus according to claim 24, characterized in that, The first processing module is further configured to: In response to the content information including a preset content structure, and / or the content information including a redundancy level greater than a preset redundancy level threshold, it is determined that the content information of the first data satisfies the preset condition.

26. The data processing apparatus according to claim 22, characterized in that, The second processing module is further configured to: The fourth data is stored in a preset physical block within the preset storage area; the preset physical block is used to store the compressed data in the background.

27. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the data processing method as described in any one of claims 1-21.

28. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data processing method as described in any one of claims 1-21.

29. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-21.

Images