Data processing method and apparatus, and device, storage medium and program product

Abstract

The present application belongs to the technical field of storage. Disclosed are a data processing method and apparatus, and a device, a storage medium and a program product. The method comprises: acquiring an access statistics result of first-type data blocks; on the basis of the access statistics result corresponding to the first-type data blocks, determining at least one candidate data block from among the first-type data blocks; acquiring an access statistics result of second-type data blocks; and on the basis of the access statistics result of the second-type data blocks, processing data in the second-type data blocks. In the present application, instances of access to coarse-grained data blocks can be counted by acquiring the access statistics result of the first-type data blocks, such that the coarse-grained data blocks can be screened to obtain at least one candidate data block; and fine-grained statistics can also be performed on the coarse-grained data blocks obtained after the screening, so as to realize fine-grained data processing. The present application can also ensure that the storage space occupied by statistics results is relatively small on the basis of realizing fine-grained statistics.

Description

Data processing method, device, apparatus, storage medium and program product

[0001] The present application claims priority to Chinese Patent Application No. 202410551050.2, filed on April 29, 2024, entitled "Hotspot Data Migration Method, Device and System", and to Chinese Patent Application No. 202410721450.3, filed on June 4, 2024, entitled "Data Processing Method, Device, Apparatus, Storage Medium and Program Product", the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of storage, in particular to a data processing method, device, apparatus, storage medium and program product. BACKGROUND

[0003] In the field of cloud computing, memory is one of the key resources to support various applications and services. Whether it is database query, big data analysis or machine learning training, a large amount of memory resources are needed to support.

[0004] At present, the memory close to the central processing unit (CPU) (also known as high-speed memory) has the characteristics of high price and fast read-write speed; the memory far from the CPU has relatively low price and relatively poor read-write speed, so migrating the frequently accessed data to the high-speed memory can enhance the overall performance of the system; migrating the relatively infrequently accessed data from the high-speed memory to a storage medium with lower energy consumption and more economical cost can effectively reduce the cost and reduce unnecessary energy consumption and storage medium wear and tear.

[0005] As can be seen, processing different types of data can reduce resource waste and reduce costs. However, how to efficiently and accurately process data has become an important technical problem to be solved in the current technical development. SUMMARY

[0006] The present application provides a data processing method, device, apparatus, storage medium and computer program, which can efficiently and accurately process data. The technical solution is as follows:

[0007] In a first aspect, a data processing method is provided. The method comprises: obtaining access statistical results of first data blocks, the first data blocks being data blocks in a data storage area, sizes of the data blocks in the first data blocks being the same, the access statistical results of the first data blocks indicating respective access frequencies of the data blocks in the first data blocks, determining at least one candidate data block from the first data blocks based on the access statistical results of the first data blocks, obtaining access statistical results of second data blocks, each candidate data block comprising a plurality of data blocks in the second data blocks, sizes of the data blocks in the second data blocks being the same, the access statistical results of the second data blocks indicating respective access frequencies of the data blocks in the second data blocks, and processing data in the second data blocks based on the access statistical results of the second data blocks.

[0008] The present application realizes the processing of data through the statistics of twice access frequencies. The at least one candidate data block is determined from the first data blocks based on the access statistical results of the first data blocks, and then the access statistical results of the second data blocks are obtained, and the data in the second data blocks is processed based on the access statistical results of the second data blocks. Since the second data is the data blocks in the candidate data blocks, the size of the first data blocks is larger than the size of the second data blocks. As can be seen, the access statistical results of the first data blocks can realize the statistics of the access frequencies of the coarse-grained pages, so as to realize the screening of the coarse-grained data blocks to obtain the at least one candidate data block. Since the second data blocks are the data blocks in the candidate data blocks, the access statistical results of the second data blocks can realize the fine-grained statistics of the screened coarse-grained data blocks, so as to realize the fine-grained data processing. Since the fine-grained statistics is for the candidate data blocks in the first data blocks, compared with the related art, the present application can realize the fine-grained statistics while ensuring that the storage space occupied by the statistical results is small.

[0009] Optionally, the first data blocks are divided according to a first division size, the storage medium corresponding to the data storage area is a volatile storage medium or a non-volatile storage medium.

[0010] Before obtaining the access statistical results of the first data blocks, the processor can also determine the first division size.

[0011] Optionally, the first division size is determined based on a first statistical quantity and a size of the data storage area, the first statistical quantity indicating an upper limit of the number of the first data blocks.

[0012] Optionally, a first value is determined by dividing the size of the data storage region by the value after the first statistical number, and the first division size is determined based on the first value.

[0013] It should be noted that, since the access statistical result of the first type of data block indicates the number of times each data block in the first type of data block is accessed, it can be seen that each data block in the first type of data block corresponds to a counter, the counter is used to count the number of times the data block is accessed, and the bit width of the counter determines the maximum number of times the counter can record the number of times of access. In actual application, the size of the first storage region storing the access statistical result of the first type of data block is fixed, and the bit width of each counter in the first storage region is also fixed. In this case, the size of the first storage region can be divided by the bit width of the counter to obtain the upper limit of the number of data blocks that the first statistical region can count (i.e. the first statistical number). If the first division size is too small, it will exceed the upper limit of the number of data blocks that the first statistical region can count, therefore, the first division size needs to be determined according to the first statistical number and the size of the data storage region.

[0014] Optionally, the first value is directly taken as the first division size, or any value greater than the first value and smaller than the size of the data storage region is taken as the first division size.

[0015] Optionally, the first division size is determined based on a storage medium allocation granularity, and the data storage region is allocated to the process in the storage medium allocation granularity.

[0016] It should be noted that the data storage region can be a memory region corresponding to the process. In this case, before the access statistical result of the first type of data block is obtained, the process can apply for memory to the operating system when it needs to use memory, and the operating system can allocate the data storage region to the process in the storage medium allocation granularity. At this time, the size of the data storage region is a multiple of the storage medium allocation granularity.

[0017] Optionally, any value greater than the storage medium allocation granularity and smaller than the size of the data storage region is taken as the first division size.

[0018] The implementation of determining at least one candidate data block from the first type of data block based on the access statistical result corresponding to the first type of data block includes multiple ways, and two of the implementation ways are introduced as follows.

[0019] In a first implementation, the processor sorts each data block in the first type of data blocks based on the respective access frequency of each data block in the first type of data blocks to obtain a sorting result, and determines the at least one candidate data block from the first type of data blocks based on the sorting result. For ease of description, the sorting result of each data block in the first type of data blocks is referred to as a first sorting result.

[0020] The processor can sort each data block in the first type of data blocks in descending order of the access frequency to obtain the first sorting result. Alternatively, the processor can also sort each data block in the first type of data blocks in ascending order of the access frequency to obtain the first sorting result.

[0021] Alternatively, the at least one candidate data block includes a first type of candidate data block and / or a second type of candidate data block.

[0022] The first type of candidate data block corresponds to an access frequency greater than that of other data blocks in the first type of data blocks, and the second type of candidate data block corresponds to an access frequency less than that of other data blocks in the first type of data blocks.

[0023] In the case where the at least one candidate data block includes the first type of candidate data block, if the first sorting result is obtained in descending order of the access frequency, the processor can take the first M1 percent of data blocks in the first sorting result, or the first N1 data blocks in the first sorting result, as the at least one first type of candidate data block. If the first sorting result is obtained in ascending order of the access frequency, the processor can take the last M1 percent of data blocks in the first sorting result, or the last N1 data blocks in the first sorting result, as the at least one first type of candidate data block.

[0024] In the case where the at least one candidate data block includes the second type of candidate data block, if the first sorting result is obtained in descending order of the access frequency, the processor can take the last M2 percent of data blocks in the first sorting result, or the last N2 data blocks in the first sorting result, as the at least one second type of candidate data block. If the first sorting result is obtained in ascending order of the access frequency, the processor can take the first M1 percent of data blocks in the first sorting result, or the first N1 data blocks in the first sorting result, as the at least one second type of candidate data block.

[0025] In the second implementation, the at least one candidate data block includes a first type of candidate data block and / or a second type of candidate data block, the first type of candidate data block corresponds to a number of access times greater than that of other data blocks of the first type, and the second type of candidate data block corresponds to a number of access times less than that of other data blocks of the first type.

[0026] In the case where the at least one candidate data block includes the first type of candidate data block, the processor can determine, as the first type of candidate data block, at least one data block in the first type of data block corresponding to a number of access times greater than a first access time threshold.

[0027] In the case where the at least one candidate data block includes the second type of candidate data block, the processor can determine, as the second type of candidate data block, at least one data block in the first type of data block corresponding to a number of access times less than a second access time threshold.

[0028] Optionally, the second type of data block is divided according to a second division size, and the first division size is greater than the second division size.

[0029] Before obtaining the access statistical result of the second type of data block, the processor can also determine the second division size.

[0030] The processor can determine the second division size based on a second statistical quantity and a size of a memory region composed of the at least one candidate data block, the second statistical quantity indicating an upper limit of a number of the second type of data block.

[0031] Since the size of the second storage region in which the access statistical result of the second type of data block is stored is fixed, and the bit width of each counter in the second storage region is also fixed, in this case, the size of the second storage region can be divided by the bit width of the counter to obtain an upper limit of a number of data blocks that the second statistical region can count (i.e., the second statistical quantity). If the second division size is too small, it will exceed the upper limit of the number of data blocks that the second statistical region can count, and therefore, the second division size needs to be determined according to the second statistical quantity and the size of the memory region composed of the at least one candidate data block.

[0032] A value obtained by dividing the size of the data storage region by the second statistical quantity is determined as a second value, and the second division size is determined based on the second value.

[0033] Optionally, the second value is directly used as the second division size, or any value greater than the second value and less than the size of the data storage region is used as the second division size.

[0034] Optionally, the second statistical quantity and the first statistical quantity can be the same or different. In the case that the size of the second storage area and the first storage area is the same, and the bit width of the counter is the same, the second statistical quantity and the first statistical quantity are the same.

[0035] Optionally, the second storage area and the first storage area can be the same area.

[0036] The processor can determine the second division size based on the first division size.

[0037] Any one factor of the first division size, except the first division size, is taken as the second division size, or any one value smaller than the first division size is taken as the second division size.

[0038] Optionally, the second division size is determined based on the second value and the first division size.

[0039] Optionally, any one value greater than the second value and smaller than the first division size is taken as the second division size, or any one factor of the first division size, greater than the second value and smaller than the first division size, is taken as the second division size.

[0040] The processor determines the cold data and / or the hot data in the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks, and processes the cold data and / or the hot data in the second type of data blocks.

[0041] Optionally, the at least one candidate data block includes the first type of candidate data block, in which case, the processor can determine at least one hot data block from the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks, the access time corresponding to the at least one hot data block being greater than the access times corresponding to other second type of data blocks, and determine the data in the at least one hot data block as the hot data.

[0042] The processor determines the at least one hot data block from the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks, and the implementation manner includes multiple ways, two of which are introduced next.

[0043] The first implementation manner is to sort each data block in the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks to obtain a second sorting result, and determine the at least one hot data block from the second type of data blocks based on the second sorting result.

[0044] Optionally, the processor can sort the data blocks in the second type of data blocks in descending order of the number of times of being accessed to obtain a second sorting result. Of course, in actual applications, the data blocks in the second type of data blocks can also be sorted in ascending order of the number of times of being accessed to obtain the second sorting result.

[0045] If the second sorting result is obtained in descending order of the number of times of being accessed, the processor can take the first A1 percent of data blocks in the second sorting result or the first B1 data blocks in the second sorting result as the at least one hot data block. If the second sorting result is obtained in ascending order of the number of times of being accessed, the processor can take the last A1 percent of data blocks in the second sorting result or the last B1 data blocks in the second sorting result as the at least one hot data block.

[0046] In a second implementation, at least one data block corresponding to the number of times of being accessed greater than a third threshold of the number of times of being accessed in the second type of data blocks is determined as the at least one hot data block, and the third threshold of the number of times of being accessed is greater than the first threshold of the number of times of being accessed.

[0047] Optionally, the at least one candidate data block includes a second type of candidate data block, in which case, the processor can determine at least one cold data block from the second type of data blocks based on the number of times of being accessed corresponding to each data block in the second type of data blocks, the number of times of being accessed corresponding to the at least one cold data block is less than the number of times of being accessed corresponding to other data blocks in the second type of data blocks, and data in the at least one cold data block is determined as cold data.

[0048] The processor determines at least one cold data block from the second type of data blocks based on the number of times of being accessed corresponding to each data block in the second type of data blocks, and there are multiple implementation manners, two of which are introduced as follows.

[0049] In a first implementation, each data block in the second type of data blocks is sorted based on the number of times of being accessed corresponding to each data block to obtain a third sorting result, and the at least one cold data block is determined from the second type of data blocks based on the third sorting result.

[0050] If the third sorting result is obtained in descending order of the number of times of being accessed, the processor can take the last A2 percent of data blocks in the third sorting result or the last B2 data blocks in the third sorting result as the at least one cold data block. If the third sorting result is obtained in ascending order of the number of times of being accessed, the processor can take the first A2 percent of data blocks in the third sorting result or the first B2 data blocks in the third sorting result as the at least one cold data block.

[0051] In the second implementation, at least one data block in the second type of data blocks, whose corresponding access times are less than a fourth access time threshold, is determined as the at least one cold data block. The fourth access time threshold is less than the second access time threshold.

[0052] Optionally, the implementation process of processing the cold data and / or the hot data in the second type of data blocks includes: moving the hot data from the first storage medium to the second storage medium, or moving the cold data from the second storage medium to the first storage medium, wherein the performance of the second storage medium is superior to that of the first storage medium.

[0053] Optionally, the moving of the data can be implemented by means of migration, copying or swapping.

[0054] In a second aspect, a data processing apparatus is provided, which has a function of implementing the behaviors of the data processing method in the first aspect. The data processing apparatus includes at least one module for implementing the data processing method provided in the first aspect.

[0055] In a third aspect, a chip is provided, which includes a processor configured to execute the data processing method in the first aspect.

[0056] Optionally, the processor can further include an interface circuit for receiving instructions and transmitting the instructions to the processor.

[0057] In a fourth aspect, a data processing system is provided, which includes a processor, a first storage medium and a second storage medium. The processor is configured to process data in a second type of data blocks according to the data processing method in the first aspect. In the case that the second type of data blocks include cold data and / or hot data, the processor is configured to move the hot data from the first storage medium to the second storage medium, or move the cold data from the second storage medium to the first storage medium, wherein the performance of the second storage medium is superior to that of the first storage medium.

[0058] In a fifth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is run on a computer or a processor, the computer or the processor is caused to execute the steps of the data processing method in the first aspect.

[0059] In a sixth aspect, a computer program product is provided, which contains computer instructions, when the computer instructions are run on a computer or a processor, make the computer execute the steps of the data processing method of the first aspect. Alternatively, a computer program is provided, when the computer program is run on a computer or a processor, make the computer or the processor execute the steps of the data processing method of the first aspect.

[0060] The technical effects obtained by the second aspect to the sixth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0061] FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0062] FIG. 2 is a schematic diagram of a statistical unit provided by an embodiment of the present application;

[0063] FIG. 3 is a flowchart of a data processing method provided by an embodiment of the present application;

[0064] FIG. 4 is a schematic diagram of processing cold data and hot data provided by the present application;

[0065] FIG. 5 is a schematic diagram of access statistical results of a first type of data block provided by the present application;

[0066] FIG. 6 is a schematic diagram of access statistical results of a second type of data block provided by the present application;

[0067] FIG. 7 is a flowchart of another data processing method provided by an embodiment of the present application;

[0068] FIG. 8 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0069] To make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0070] To facilitate understanding, before the data processing method provided by the embodiments of the present application is explained in detail, the application scenarios and implementation environments involved in the embodiments of the present application are introduced.

[0071] Next, the application scenarios involved in the embodiments of the present application are introduced.

[0072] With the rapid development of big data and cloud computing technologies, the demand for data storage is growing exponentially, and the importance of storage media as a key link in data preservation and processing is self-evident. For example, in the field of cloud computing, memory is one of the key resources supporting various applications and services. Whether it is database queries, big data analysis, or machine learning training tasks, a large amount of memory resources are needed to support them. Currently, there are two main problems facing the use of server memory: low memory utilization and high memory cost proportion.

[0073] First, for low memory utilization, when configuring a server, the memory size is usually set based on the maximum business load. However, in actual operation, the server may only be in peak load state for part of the time, and during non-peak periods, a large amount of memory space is in an idle state and is not effectively utilized. Moreover, not all programs in the software running process will frequently read data in the memory, and only a part of the memory data in the memory space may be frequently accessed, while the rest of the memory data is accessed at a very low frequency.

[0074] Second, for high memory cost proportion, with the iteration of technology, for example, the peripheral component interconnect express (PCIe) technology has been upgraded from the fourth generation to the fifth generation, although it has brought higher data transmission rate and lower latency, but the need to purchase and upgrade more advanced storage media to adapt to the new technology, thus the cost also rises.

[0075] In related technologies, physical memory is usually divided into fixed-size pages, and if a process needs to use memory, it needs to apply for a page from the operating system. In this case, access to data in memory is actually access to the page where the data is located, so access requests can be counted to determine the number of accesses to each page, and then the pages can be divided into cold pages and hot pages according to the number of accesses. Hot pages are frequently accessed pages, while cold pages are relatively infrequently accessed and have a much lower access frequency than hot pages. Then, according to the type of page, the data in the memory is processed.

[0076] For example, considering that the memory (also referred to as high-speed memory) close to the central processing unit (CPU) is high in price but has high read-write speed, and the memory far from the CPU is relatively low in price and relatively poor in read-write speed, the data frequently accessed can be migrated to the high-speed memory, thereby enhancing the overall performance of the system, and the data relatively infrequently accessed can be removed from the high-speed memory and migrated to a storage medium with lower energy consumption and lower cost, thereby effectively reducing the cost and reducing unnecessary energy consumption and wear of the storage medium.

[0077] However, in order to implement fine processing of data, such as fine-grained migration of data, the physical memory needs to be divided into a plurality of smaller pages, thereby implementing fine-grained page statistics, but the smaller pages increase the number of pages that need to be counted, thereby occupying a large storage space to cache the statistical data and reducing the performance and efficiency of the system. If the physical memory is divided into a plurality of larger pages, fine processing of data cannot be implemented.

[0078] Based on this, an embodiment of the present application provides a data processing method, which can implement processing of data through counting of access times twice. At least one candidate data block is determined from the first type of data block according to the access statistical result of the first type of data block, and then the access statistical result of the second type of data block is obtained, and the data in the second type of data block is processed according to the access statistical result of the second type of data block. Since the second type of data is the data block in the candidate data block, the size of the first type of data block is greater than the size of the second type of data block. As can be seen, obtaining the access statistical result of the first type of data block can implement counting of access times of coarse-grained pages, thereby implementing screening of coarse-grained data blocks to obtain at least one candidate data block. Since the second type of data block is the data block in the candidate data block, obtaining the access statistical result of the second type of data block can implement fine-grained statistics of the screened coarse-grained data blocks, thereby implementing fine-grained data processing, and since the fine-grained statistics are for the candidate data block in the first type of data block, compared with related technologies, the embodiment of the present application can ensure that the storage space occupied by the statistical result is small on the basis of implementing fine-grained statistics.

[0079] Next, an implementation environment related to the embodiments of the present application is introduced. The implementation environment includes a processor 01. The processor 01 can acquire access statistical results of first type data blocks, each data block in the first type data blocks is a data block in a data storage area, each data block in the first type data blocks has a same size, the access statistical results of the first type data blocks indicate a number of times of access corresponding to each data block in the first type data blocks respectively, at least one candidate data block is determined from the first type data blocks based on the access statistical results corresponding to the first type data blocks, and then access statistical results of second type data blocks are acquired, each candidate data block includes a plurality of data blocks in the second type data blocks, each data block in the second type data blocks has a same size, the access statistical results of the second type data blocks indicate a number of times of access corresponding to each data block in the second type data blocks respectively, and data in the second type data blocks is processed based on the access statistical results of the second type data blocks.

[0080] In some embodiments, referring to FIG. 1, which is a schematic diagram of an implementation environment provided by the embodiments of the present application, the implementation environment further includes a statistical unit 02, the processor 01 is in communication connection with the statistical unit 02, and the communication connection can be wired connection or wireless connection, which is not limited in the embodiments of the present application.

[0081] Optionally, before acquiring the access statistical results of the first type data blocks, the processor 01 can send first statistical configuration information to the statistical unit 02, the statistical configuration information includes a first address range and a first division size, and the first address range is an address range corresponding to the data storage area. The statistical unit can be configured according to the first statistical configuration information, after the configuration is completed, the statistical unit 02 divides the data storage area according to the first division size to obtain the first type data blocks, and counts the number of times of access of each data block in the first type data blocks, stops the counting when a statistical stop condition is met, and obtains the access statistical results of the first type data blocks. The statistical unit 02 sends the access statistical results of the first type data blocks to the processor 01, and then the processor 01 can acquire the access statistical results of the first type data blocks.

[0082] Optionally, after determining the at least one candidate data block from the first type of data blocks, the processor 01 can send second statistical configuration information to the statistical unit 02, the second statistical configuration information comprising a second address range and a second division size, the second address range being an address range corresponding to the at least one candidate data block. The statistical unit 02 divides the storage area corresponding to the second address range according to the second block division size to obtain second type of data blocks, and counts the access times of each data block in the second type of data blocks. When the statistical stop condition is met, the counting is stopped, and the access statistical result of the second type of data blocks is obtained. The statistical unit 02 sends the access statistical result of the second type of data blocks to the processor 01, and the processor 01 can obtain the access statistical result of the second type of data blocks.

[0083] Optionally, during the counting by the statistical unit, the processor can receive an access request, the access request carrying a storage address of accessed data. The processor can perform a read operation or a write operation on the data in the storage address according to the storage address of the accessed data. The processor can also send the storage address of the accessed data to the statistical unit.

[0084] Optionally, referring to FIG. 2, FIG. 2 is a schematic diagram of a statistical unit according to an embodiment of the present application. The statistical unit comprises an address indexing subunit, a conversion subunit and a storage subunit. The indexing subunit can determine a data block corresponding to the storage address of the accessed data (i.e., the memory access information) according to the storage address of the accessed data. The conversion subunit can further obtain the access times corresponding to the data block from the storage subunit, add one to the access times, and store the added access times to the storage subunit.

[0085] Optionally, the statistical unit further comprises a compression subunit. Before sending the access statistical result to the processor 01, the compression subunit can compress the access statistical result and send the compressed access statistical result to the processor 01. The access statistical result can be the access statistical result of the first type of data blocks or the access statistical result of the second type of data blocks.

[0086] For example, the storage subunit can be a random access memory (RAM).

[0087] Optionally, before counting, the statistical unit 02 can determine whether the storage subunit stores data. When it is determined that the storage subunit does not store data, the counting can be started. When it is determined that the storage subunit stores data, the data in the storage subunit can be emptied, and then the counting can be started.

[0088] In some embodiments, the statistical stop condition comprises at least one of the following: a difference between a time when the statistics start and a current time (i.e., a statistical duration) is equal to or greater than a statistical duration threshold, and a number of times each data block is accessed is equal to or greater than an access number threshold.

[0089] Of course, in actual applications, the statistical stop condition can also include other contents, such as a total number of accesses of the first type of data block is greater than or equal to a total number threshold, and the like, which are not limited by the embodiments of the present application.

[0090] The statistical duration threshold and the access number threshold are both set in advance, and can also be adjusted according to different needs in different cases. For example, the statistical duration threshold can be set to 24 hours, and the access number threshold can be set to 16.77 million times.

[0091] In some embodiments, the interaction mode between the processor 01 and the statistical unit 02 can be an interrupt mode or a polling mode.

[0092] In the case of the interrupt mode, the statistical unit 02 can actively report the access statistics of the first type of data block or the access statistics of the second type of data block to the processor 01 when the statistical stop condition is met.

[0093] In the case of the polling mode, the processor 01 can periodically determine whether the statistical unit 02 meets the statistical stop condition, and when the statistical stop condition is met, the processor 01 can send a statistical stop instruction to the statistical unit 02. The statistical unit 02 stops statistics and reports the access statistics of the first type of data block or the access statistics of the second type of data block to the processor 01 when receiving the statistical stop instruction sent by the processor 01.

[0094] The processor 01 is configured to manage at least one storage medium, and the at least one storage medium comprises a target storage medium, and the target storage medium comprises a data storage area.

[0095] For example, the processor 01 can be a processor in a storage medium controller, and the storage medium controller is configured to manage the target storage medium. The processor 01 can be a processor in a storage medium cluster controller, and the storage medium cluster controller is configured to manage a plurality of storage media, and the plurality of storage media comprises the target storage medium.

[0096] Optionally, the storage medium corresponding to the data storage area can be one or more, which are not limited by the embodiments of the present application. When the storage medium corresponding to the data storage area is multiple, the data storage area is divided into multiple parts, and the multiple parts are distributed in the multiple storage media.

[0097] It should be noted that the processor 01 and the statistical unit 02 can be located in two independent physical devices respectively, of course, the processor 01 can also be integrated with the statistical unit 02. For example, the processor 01 can also be integrated with the statistical unit 02, which can be that the processor 01 and the statistical unit 02 are integrated into the same chip, or the processor 01 also has the function of the statistical unit 02, and the embodiments of the present application do not limit this. In the case that the processor 01 and the statistical unit 02 can be located in two independent physical devices respectively, the statistical unit 02 can be integrated with the storage medium managed by the processor 01.

[0098] Those skilled in the art should understand that the above processor and statistical unit are only examples, and other existing or future processor and statistical unit that can be applicable to the embodiments of the present application should also be included in the protection scope of the embodiments of the present application, and are included herein by reference.

[0099] It should be noted that the application scenarios and implementation environments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0100] FIG. 3 is a flowchart of a data processing method provided by an embodiment of the present application. Please refer to FIG. 3, the method includes the following steps.

[0101] Step 301: Obtain an access statistical result of a first type of data block, the first type of data block is a data block in a data storage area, sizes of each data block in the first type of data block are the same, and the access statistical result of the first type of data block indicates a respective access frequency corresponding to each data block in the first type of data block.

[0102] Optionally, the first type of data block is divided according to a first division size, the storage medium corresponding to the data storage area is a volatile storage medium or a non-volatile storage medium, and the embodiments of the present application do not limit this.

[0103] In actual application, before obtaining the access statistical result of the first type of data block, the processor can also determine the first division size.

[0104] In some embodiments, the processor can determine the first division size based on a first statistical quantity and a size of the data storage area, the first statistical quantity indicating an upper limit of the number of the first type of data block.

[0105] Optionally, a first value is determined by dividing the size of the data storage region by the value after the first statistical number, and the first division size is determined based on the first value.

[0106] It should be noted that, since the access statistical result of the first type of data block indicates the number of times each data block in the first type of data block is accessed, it can be seen that each data block in the first type of data block corresponds to a counter, the counter is used to count the number of times the data block is accessed, and the bit width of the counter determines the maximum number of times the counter can record the number of times of access. In actual application, the size of the first storage region storing the access statistical result of the first type of data block is fixed, and the bit width of each counter in the first storage region is also fixed. In this case, the size of the first storage region can be divided by the bit width of the counter to obtain the upper limit of the number of data blocks that the first statistical region can count (i.e. the first statistical number). If the first division size is too small, it will exceed the upper limit of the number of data blocks that the first statistical region can count, therefore, the first division size needs to be determined according to the first statistical number and the size of the data storage region.

[0107] For example, the first value can be directly used as the first division size. In actual application, any value greater than the first value and smaller than the size of the data storage region can also be used as the first division size, and the embodiments of the present application do not limit this.

[0108] In other embodiments, the first division size is determined based on the storage medium allocation granularity, and the data storage region is allocated to the process in the storage medium allocation granularity.

[0109] It should be noted that the data storage region can be a memory region corresponding to the process. In this case, before the access statistical result of the first type of data block is obtained, the process can apply for memory to the operating system when it needs to use memory, and the operating system can allocate data storage region to the process in the storage medium allocation granularity. At this time, the size of the data storage region is a multiple of the storage medium allocation granularity.

[0110] Based on this, the first division size can be determined according to the storage medium allocation granularity, for example, any value greater than the storage medium allocation granularity and smaller than the size of the data storage region can be used as the first division size.

[0111] For example, the storage medium allocation granularity is 4 kilobytes (KB), in this case, if process 1 applies for memory to the operating system, the operating system can allocate a data storage area for process 1 in units of 4 KB. The size of the data storage area allocated to this process 1 can be 64 gigabytes (GB), which is 16777216 times 4 KB, in this case, 2 megabytes (MB) can be used as the first division size.

[0112] Step 302: determining at least one candidate data block from the first type of data block based on the access statistical result corresponding to the first type of data block.

[0113] The implementation of determining at least one candidate data block from the first type of data block based on the access statistical result corresponding to the first type of data block includes multiple modes, and two implementation modes are introduced as follows.

[0114] In a first implementation mode, the processor sorts each data block in the first type of data block based on the access frequency corresponding to each data block, to obtain a sorting result, and determines the at least one candidate data block from the first type of data block based on the sorting result. For ease of description, the sorting result of each data block in the first type of data block is referred to as a first sorting result.

[0115] Optionally, the processor can sort each data block in the first type of data block in descending order of the access frequency, to obtain the first sorting result, of course, in actual application, each data block in the first type of data block can also be sorted in ascending order of the access frequency, to obtain the first sorting result.

[0116] Optionally, the at least one candidate data block includes a first type of candidate data block and / or a second type of candidate data block.

[0117] The access frequency corresponding to the first type of candidate data block is greater than the access frequency of other first type of data blocks, and the access frequency corresponding to the second type of candidate data block is less than the access frequency corresponding to other first type of data blocks.

[0118] In a case where the at least one candidate data block comprises the first type of candidate data block, if the first sorting result is obtained by sorting in descending order of the access times, the processor can take the first M1 percent of data blocks in the first sorting result, or the first N1 data blocks in the first sorting result, as the at least one first type of candidate data block. If the first sorting result is obtained by sorting in ascending order of the access times, the processor can take the last M1 percent of data blocks in the first sorting result, or the last N1 data blocks in the first sorting result, as the at least one first type of candidate data block.

[0119] In a case where the at least one candidate data block comprises the second type of candidate data block, if the first sorting result is obtained by sorting in descending order of the access times, the processor can take the last M2 percent of data blocks in the first sorting result, or the last N2 data blocks in the first sorting result, as the at least one second type of candidate data block. If the first sorting result is obtained by sorting in ascending order of the access times, the processor can take the first M1 percent of data blocks in the first sorting result, or the first N1 data blocks in the first sorting result, as the at least one second type of candidate data block.

[0120] wherein M1, N1, M2, and N2 are set in advance, and can be adjusted according to different requirements in different cases. For example, M1 can be set to 10, N1 can be set to 64, M2 can be set to 10, and N2 can be set to 64.

[0121] In a second implementation, the at least one candidate data block comprises the first type of candidate data block and / or the second type of candidate data block, the first type of candidate data block corresponds to an access time greater than that of other first type of data blocks, and the second type of candidate data block corresponds to an access time less than that of other first type of data blocks.

[0122] In a case where the at least one candidate data block comprises the first type of candidate data block, the processor can determine, as the first type of candidate data block, at least one data block in the first type of data blocks corresponding to an access time greater than a first access time threshold.

[0123] In a case where the at least one candidate data block comprises the second type of candidate data block, the processor can determine, as the second type of candidate data block, at least one data block in the first type of data blocks corresponding to an access time less than a second access time threshold.

[0124] The first access frequency threshold and the second access frequency threshold are set in advance, the first access frequency threshold is greater than or equal to the second access frequency threshold, and the first access frequency threshold and the second access frequency threshold can be adjusted according to different requirements in different cases.

[0125] Step 303: Obtain an access statistical result of the second type of data block. Each candidate data block includes a plurality of data blocks in the second type of data block. The sizes of the data blocks in the second type of data block are the same. The access statistical result of the second type of data block indicates the respective access frequencies of the data blocks in the second type of data block.

[0126] Optionally, the second type of data block is divided according to a second division size, and the first division size is greater than the second division size.

[0127] In actual application, before obtaining the access statistical result of the second type of data block, the processor can also determine the second division size.

[0128] In some embodiments, the processor can determine the second division size based on the second statistical quantity and the size of the memory region composed of the at least one candidate data block. The second statistical quantity indicates an upper limit of the number of the second type of data blocks.

[0129] As described above, the size of the second storage region in which the access statistical result of the second type of data block is stored is fixed, and the bit width of each counter in the second storage region is also fixed. In this case, the size of the second storage region can be divided by the bit width of the counter to obtain an upper limit of the number of data blocks that can be counted by the second statistical region (i.e., the second statistical quantity). If the second division size is too small, it will exceed the upper limit of the number of data blocks that can be counted by the second statistical region. Therefore, the second division size needs to be determined according to the second statistical quantity and the size of the memory region composed of the at least one candidate data block.

[0130] Optionally, the size of the data storage region can be divided by the value after the second statistical quantity to determine a second value. The second value is used to determine the second division size.

[0131] For example, the second value can be directly used as the second division size. In actual application, any value greater than the second value and smaller than the size of the data storage region can be used as the second division size, which is not limited in the embodiments of the present application.

[0132] It should be noted that the second statistical quantity and the first statistical quantity can be the same or different, which is not limited in the embodiments of the present application. In the case that the size of the second storage region and the first storage region is the same and the bit width of the counter is the same, the second statistical quantity and the first statistical quantity are the same.

[0133] Optionally, the second storage area and the first storage area can be the same area.

[0134] In some embodiments, the processor can determine the second division size based on the first division size.

[0135] For example, any one factor of the first division size other than the first division size can be used as the second division size. In actual applications, any one value smaller than the first division size can also be used as the second division size, which is not limited in the embodiments of the present application.

[0136] In some embodiments, the second division size can also be determined based on the second value and the first division size.

[0137] For example, any one value greater than the second value and smaller than the first division size can be used as the second division size, or any one factor of the first division size greater than the second value and smaller than the first division size can be used as the second division size, which is not limited in the embodiments of the present application.

[0138] Step 304: processing data in the second type of data blocks based on the access statistics of the second type of data blocks.

[0139] In some embodiments, the processor can determine cold data and / or hot data in the second type of data blocks based on the access frequencies of each data block in the second type of data blocks, and process the cold data and / or hot data in the second type of data blocks.

[0140] Optionally, the at least one candidate data block includes the first type of candidate data block, in which case, the processor can determine at least one hot data block from the second type of data blocks based on the access frequencies of each data block in the second type of data blocks, the access frequency of the at least one hot data block being greater than the access frequencies of other second type of data blocks, and determine data in the at least one hot data block as hot data.

[0141] The processor can determine at least one hot data block from the second type of data blocks based on the access frequencies of each data block in the second type of data blocks in multiple ways, two of which are introduced as follows.

[0142] In a first implementation, each data block in the second type of data blocks is sorted based on the access frequencies of each data block in the second type of data blocks to obtain a second sorting result, and the at least one hot data block is determined from the second type of data blocks based on the second sorting result.

[0143] Optionally, the processor can sort the data blocks in the second type of data blocks in descending order of the number of times of being accessed to obtain a second sorting result. Of course, in actual applications, the data blocks in the second type of data blocks can also be sorted in ascending order of the number of times of being accessed to obtain the second sorting result.

[0144] If the second sorting result is obtained in descending order of the number of times of being accessed, the processor can take the first A1 percent of data blocks in the second sorting result or the first B1 data blocks in the second sorting result as the at least one hot data block. If the second sorting result is obtained in ascending order of the number of times of being accessed, the processor can take the last A1 percent of data blocks in the second sorting result or the last B1 data blocks in the second sorting result as the at least one hot data block.

[0145] Wherein, A1, B1 and the third threshold of the number of times of being accessed are set in advance, and can be adjusted according to different requirements in different cases. For example, A1 can be set to 10, and B1 can be set to 64.

[0146] In the second implementation, at least one data block corresponding to the number of times of being accessed greater than the third threshold of the number of times of being accessed in the second type of data blocks is determined as the at least one hot data block, and the third threshold of the number of times of being accessed is greater than the first threshold of the number of times of being accessed.

[0147] Optionally, the at least one candidate data block includes a second type of candidate data block, in which case, the processor can determine at least one cold data block from the second type of data blocks based on the number of times of being accessed corresponding to each data block in the second type of data blocks, the number of times of being accessed corresponding to the at least one cold data block being less than the number of times of being accessed corresponding to other data blocks in the second type of data blocks, and determine the data in the at least one cold data block as cold data.

[0148] The processor determines at least one cold data block from the second type of data blocks based on the number of times of being accessed corresponding to each data block in the second type of data blocks, and the implementation includes multiple ways, two of which are introduced as follows.

[0149] In the first implementation, the data blocks in the second type of data blocks are sorted based on the number of times of being accessed corresponding to each data block in the second type of data blocks to obtain a third sorting result, and the at least one cold data block is determined from the second type of data blocks based on the third sorting result.

[0150] Optionally, the processor can sort the data blocks in the second type of data blocks in an order of the number of times of being accessed from large to small to obtain a third sorting result. Of course, in actual applications, the data blocks in the second type of data blocks can also be sorted in an order of the number of times of being accessed from small to large to obtain the third sorting result.

[0151] If the third sorting result is obtained in the order of the number of times of being accessed from large to small, the processor can take the last A2 percent of data blocks in the third sorting result, or the last B2 data blocks in the third sorting result, as the at least one cold data block. If the third sorting result is obtained in the order of the number of times of being accessed from small to large, the processor can take the first A2 percent of data blocks in the third sorting result, or the first B2 data blocks in the third sorting result, as the at least one cold data block.

[0152] Wherein, A2, B2 and the third threshold of the number of times of being accessed are set in advance, and can be adjusted according to different needs in different cases. For example, A2 can be set to 10, and B2 can be set to 64.

[0153] In the second implementation, the at least one data block in the second type of data blocks corresponding to the number of times of being accessed less than a fourth threshold of the number of times of being accessed is determined as the at least one cold data block. The fourth threshold of the number of times of being accessed is less than the second threshold of the number of times of being accessed.

[0154] The implementation process of processing the cold data and / or the hot data in the second type of data blocks includes: moving the hot data from the first storage medium to the second storage medium, or moving the cold data from the second storage medium to the first storage medium, wherein the performance of the second storage medium is superior to that of the first storage medium.

[0155] Optionally, the processor can move the data by means of migration, copying or swapping, which is not limited in the embodiments of the present application.

[0156] For example, referring to FIG. 4, FIG. 4 is a schematic diagram of processing cold data and hot data provided by the present application. The storage media corresponding to the second type of data blocks are storage medium 1, storage medium 2 and storage medium 3, the performance of the storage medium 1 is superior to that of the storage medium 2, and the performance of the storage medium 2 is superior to that of the storage medium 3. The processor can determine the cold data in the storage medium 1, the cold data and the hot data in the storage medium 2, and the hot data in the storage medium 3, and then the processor can migrate the cold data in the storage medium 1 to the storage medium 2, migrate the hot data in the storage medium 2 to the storage medium 1, migrate the cold data in the storage medium 2 to the storage medium 3, and migrate the hot data in the storage medium 3 to the storage medium 2.

[0157] Next, the data processing method provided by the embodiment of the present application will be introduced again by means of FIG. 5 and 6.

[0158] Please refer to FIG. 5, the size of the data storage area is 64GB, the first division size is 2MB, then the number of the first type of data blocks is 32768, in this case, the processor can obtain the access statistical result of the first type of data blocks, the access statistical result of the first type of data blocks indicates the access times corresponding to the 32768 data blocks in the first type of data blocks respectively. The processor sorts the 32768 data blocks according to the access times corresponding to the 32768 data blocks in the first type of data blocks respectively in descending order according to the access times, to obtain the first sorting result, and then takes the first 64 data blocks in the first sorting result as the at least one first type of candidate data block. Please refer to FIG. 6, the total size of the at least one first type of candidate data block is 128MB, the second division size is 4KB, then the number of the second type of data blocks is 32768, in this case, the processor can obtain the access statistical result of the second type of data blocks, the access statistical result of the second type of data blocks indicates the access times corresponding to the 32768 data blocks in the second type of data blocks respectively. The processor sorts the 32768 data blocks according to the access times corresponding to the 32768 data blocks in the second type of data blocks respectively in descending order according to the access times, to obtain the second sorting result, and then takes the first 10% of the 4KB data blocks in the second sorting result as the at least one hot data block, and then processes the at least one hot data block.

[0159] Next, the data processing method provided by the embodiment of the present application will be introduced again by means of the statistical unit.

[0160] Referring to FIG. 7, if the size of the data storage area is 64 GB and the first division size is 2 MB, the number of the first type of data blocks is 32768. In this case, the processor can send first statistical configuration information to the statistical unit, and the statistical unit can be configured according to the first statistical configuration information. After the configuration is completed, the statistical unit determines whether there is data stored in the RAM. If it is determined that there is no data in the storage subunit, the statistical unit can start to perform the statistics. If it is determined that there is data in the RAM, the statistical unit can clear the data in the RAM and then start to perform the statistics, and then determine whether the interaction mode of the processor and the statistical unit is the interrupt mode. If the interaction mode is the interrupt mode, the statistical unit can actively report the access statistical result of the first type of data blocks to the processor when the statistical stop condition is met. If the interaction mode is not the interrupt mode, it is determined whether the interaction mode of the processor and the statistical unit is the polling mode. If the interaction mode is the polling mode, the processor can periodically determine whether the statistical unit meets the statistical stop condition. When the statistical stop condition is met, the processor can send a statistical stop instruction to the statistical unit. The statistical unit stops the statistics and reports the access statistical result of the first type of data blocks to the processor when receiving the statistical stop instruction sent by the processor. After the processor obtains the access statistical result of the first type of data blocks, it can be determined whether the size of the first type of data blocks is 2 MB. If the size of the first type of data blocks is 2 MB, the processor can sort the 32768 data blocks in the first type of data blocks in descending order of the access times according to the access times of the 32768 data blocks respectively corresponding to the 32768 data blocks, to obtain a first sorting result, and then take the first 64 data blocks in the first sorting result as the at least one first type of candidate data blocks. Then, the processor can send second statistical configuration information to the statistical unit, and the statistical unit can be configured according to the second statistical configuration information. After the configuration is completed, the statistical unit determines whether there is data stored in the RAM. If it is determined that there is no data in the storage subunit, the statistical unit can start to perform the statistics. If it is determined that there is data in the RAM, the statistical unit can clear the data in the RAM and then start to perform the statistics, and then determine whether the interaction mode of the processor and the statistical unit is the interrupt mode. If the interaction mode is the interrupt mode, the statistical unit can actively report the access statistical result of the second type of data blocks to the processor when the statistical stop condition is met. If the interaction mode is not the interrupt mode, it is determined whether the interaction mode of the processor and the statistical unit is the polling mode. If the interaction mode is the polling mode, the processor can periodically determine whether the statistical unit meets the statistical stop condition. When the statistical stop condition is met, the processor can send a statistical stop instruction to the statistical unit. The statistical unit stops the statistics and reports the access statistical result of the second type of data blocks to the processor when receiving the statistical stop instruction sent by the processor.After the processor obtains the access statistical result of the second type data block, it can determine whether the size of the first type data block is 2MB. In the case that the size of the first type data block is not 2MB, the processor can sort the 32768 data blocks in the second type data block according to the access times from large to small based on the access times corresponding to the 32768 data blocks respectively, to obtain a second sorting result, and then take the top 10% of the 4KB data blocks in the second sorting result as at least one hot data block, and then process the at least one hot data block.

[0161] The embodiment of the present application realizes the processing of data through the statistics of the access times twice. The at least one candidate data block is determined from the first type data block according to the access statistical result of the first type data block, and then the access statistical result of the second type data block is obtained, and the data in the second type data block is processed according to the access statistical result of the second type data block. Since the second type data is the data block in the candidate data block, the size of the first type data block is greater than the size of the second type data block. As can be seen, the access statistical result of the first type data block can realize the statistics of the access times of the coarse-grained page, so as to realize the screening of the coarse-grained data block to obtain at least one candidate data block. Since the second type data block is the data block in the candidate data block, the access statistical result of the second type data block can realize the fine-grained statistics of the screened coarse-grained data block, so as to realize the fine-grained data processing. Since the fine-grained statistics is for the candidate data block in the first type data block, compared with the related art, the embodiment of the present application can ensure that the storage space occupied by the statistical result is small on the basis of realizing the fine-grained statistics.

[0162] FIG. 8 is a structural schematic diagram of a data processing apparatus provided by the embodiment of the present application. The data processing apparatus can be realized by software, hardware or a combination of both to become part or all of the processor described above. Referring to FIG. 8, the apparatus includes a first obtaining module 801, a first determining module 802, a second obtaining module 803 and a processing module 804.

[0163] The first obtaining module 801 is configured to obtain the access statistical result of the first type data block. The first type data block is a data block in a data storage area. The sizes of the data blocks in the first type data block are the same. The access statistical result of the first type data block indicates the access times corresponding to the data blocks in the first type data block respectively. The detailed implementation process is referred to the corresponding content in the above embodiments, which will not be described here.

[0164] The first determining module 802 is configured to determine at least one candidate data block from the first type of data blocks based on the access statistical result corresponding to the first type of data blocks. For details, refer to the corresponding content in the above embodiments, which will not be repeated here.

[0165] The second obtaining module 803 is configured to obtain an access statistical result of a second type of data blocks. Each candidate data block includes a plurality of data blocks in the second type of data blocks, each data block in the second type of data blocks has the same size, and the access statistical result of the second type of data blocks indicates the respective access times of each data block in the second type of data blocks. For details, refer to the corresponding content in the above embodiments, which will not be repeated here.

[0166] The processing module 804 is configured to process the data in the second type of data blocks based on the access statistical result of the second type of data blocks. For details, refer to the corresponding content in the above embodiments, which will not be repeated here.

[0167] Optionally, the first type of data blocks are divided according to a first division size, and the second type of data blocks are divided according to a second division size, the first division size being greater than the second division size.

[0168] Optionally, the at least one candidate data block includes a first type of candidate data block and / or a second type of candidate data block, wherein the first type of candidate data block corresponds to an access time greater than that of other first type of data blocks, and the second type of candidate data block corresponds to an access time less than that of other first type of data blocks.

[0169] Optionally, the at least one candidate data block includes a first type of candidate data block.

[0170] The first determining module 802 is specifically configured to:

[0171] At least one data block in the first type of data blocks corresponding to an access time greater than a first access time threshold is determined as a first type of candidate data block.

[0172] Optionally, the at least one candidate data block includes a second type of candidate data block.

[0173] The first determining module 802 is specifically configured to:

[0174] At least one data block in the first type of data blocks corresponding to an access time less than a second access time threshold is determined as a second type of candidate data block.

[0175] Optionally, the first determining module 802 is specifically configured to:

[0176] sort each data block in the first type of data blocks based on the number of times each data block in the first type of data blocks is accessed, to obtain a sorting result;

[0177] determine at least one candidate data block from the first type of data blocks based on the sorting result.

[0178] Optionally, the processing module 804 is specifically configured to:

[0179] determine cold data and / or hot data in the second type of data blocks based on the number of times each data block in the second type of data blocks is accessed;

[0180] process the cold data and / or the hot data in the second type of data blocks.

[0181] Optionally, the at least one candidate data block includes a first type of candidate data block;

[0182] The processing module 804 is specifically configured to:

[0183] determine at least one hot data block from the second type of data blocks based on the number of times each data block in the second type of data blocks is accessed, the at least one hot data block corresponding to a number of times accessed greater than the number of times other data blocks in the second type of data blocks are accessed;

[0184] determine data in the at least one hot data block as hot data.

[0185] Optionally, the at least one candidate data block includes a second type of candidate data block;

[0186] The processing module 804 is specifically configured to:

[0187] determine at least one cold data block from the second type of data blocks based on the number of times each data block in the second type of data blocks is accessed, the at least one cold data block corresponding to a number of times accessed less than the number of times other data blocks in the second type of data blocks are accessed;

[0188] determine data in the at least one cold data block as cold data.

[0189] Optionally, the processing module 804 is specifically configured to:

[0190] move the hot data from the first storage medium to the second storage medium, or

[0191] move the cold data from the second storage medium to the first storage medium, wherein the performance of the second storage medium is better than that of the first storage medium.

[0192] Optionally, the apparatus further includes:

[0193] The second determining module is configured to determine the first division size based on a first statistical quantity and a size of the data storage region, the first statistical quantity indicating an upper limit of a number of the first type of data blocks.

[0194] Optionally, the first division size is determined based on a storage medium allocation granularity, and the data storage region is allocated to the process according to the storage medium allocation granularity.

[0195] Optionally, the apparatus further comprises:

[0196] The third determining module is configured to determine the second division size based on a second statistical quantity and a size of a memory region composed of the at least one candidate data block, the second statistical quantity indicating an upper limit of a number of the second type of data blocks; or

[0197] The fourth determining module is configured to determine the second division size based on the first division size.

[0198] Optionally, the storage medium corresponding to the data storage region is a volatile storage medium or a non-volatile storage medium.

[0199] In the embodiments of the present application, the data is processed by twice statistics of access times. The first type of data blocks are determined from the first type of data blocks according to the access statistical result of the first type of data blocks, and then the access statistical result of the second type of data blocks is obtained, and the data in the second type of data blocks is processed according to the access statistical result of the second type of data blocks. Since the second type of data is the data block in the candidate data block, the size of the first type of data block is greater than the size of the second type of data block. As can be seen, the access statistical result of the first type of data block can realize the statistics of the access times of the coarse-grained page, so as to realize the screening of the coarse-grained data block to obtain at least one candidate data block. Since the second type of data block is the data block in the candidate data block, the access statistical result of the second type of data block can realize the fine-grained statistics of the screened coarse-grained data block, so as to realize the fine-grained data processing. Since the fine-grained statistics is for the candidate data block in the first type of data block, compared with the related art, the embodiments of the present application can ensure that the storage space occupied by the statistical result is small on the basis of realizing fine-grained statistics.

[0200] It should be noted that the data processing apparatus provided in the above embodiments is only exemplified by the division of the above functional modules during data processing. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus and the data processing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0201] The embodiment of the present application further provides a data processing system, which comprises a processor, a first storage medium and a second storage medium; the processor is used for processing data in the second type of data block according to the data processing method in the above embodiment; in the case that the second type of data block comprises cold data and / or hot data, the processor is used for moving the hot data from the first storage medium to the second storage medium, or moving the cold data from the second storage medium to the first storage medium, and the performance of the second storage medium is superior to that of the first storage medium.

[0202] The embodiment of the present application further provides a chip, which comprises a processor, and the processor can execute the data processing method provided by the embodiment of the present application.

[0203] Optionally, the chip further comprises an interface circuit, which is used for receiving an instruction and transmitting the instruction to the processor.

[0204] The embodiment of the present application further provides a computer readable storage medium, and the storage medium stores instructions; when the instructions are executed on a computer or a processor, the computer or the processor executes the steps of the data processing method in the above embodiment.

[0205] The embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer or a processor, cause the computer or the processor to perform the steps of the data processing method in the above embodiment. Alternatively, a computer program is provided, which, when executed on a computer or a processor, causes the computer or the processor to perform the steps of the data processing method in the above embodiment.

[0206] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example: floppy disk, hard disk, magnetic tape), optical media (for example: digital versatile disc (DVD)) or semiconductor media (for example: solid state disk (SSD)) and the like. It should be noted that the computer-readable storage medium mentioned in the embodiments of the present application can be a non-volatile storage medium, in other words, it can be a non-transitory storage medium.

[0207] It should be understood that "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" herein only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, "first", "second" and the like are used to distinguish the same items or similar items with basically the same function and role in the embodiments of the present application. Those skilled in the art can understand that "first", "second" and the like do not limit the quantity and execution order, and "first", "second" and the like do not necessarily mean different.

[0208] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the access statistics of the first type of data block and the access statistics of the second type of data block involved in the embodiments of the present application are obtained under sufficient authorization.

[0209] The above describes the embodiments provided by the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining access statistics of a first type of data block, the first type of data block being a data block in a data storage area, each data block in the first type of data block being of the same size, the access statistics of the first type of data block indicating the number of accesses corresponding to each data block in the first type of data block respectively; determining at least one candidate data block from the first type of data block based on the access statistics corresponding to the first type of data block; obtaining access statistics of a second type of data block, each candidate data block comprising a plurality of data blocks in the second type of data block, each data block in the second type of data block being of the same size, the access statistics of the second type of data block indicating the number of accesses corresponding to each data block in the second type of data block respectively; processing data in the second type of data block based on the access statistics of the second type of data block.

2. The method of claim 1, wherein, The first type of data block is divided according to a first division size, and the second type of data block is divided according to a second division size, the first division size being greater than the second division size.

3. The method of claim 1 or 2, wherein, The at least one candidate data block comprises a first type of candidate data block and / or a second type of candidate data block, wherein the number of accesses corresponding to the first type of candidate data block is greater than the number of accesses of other first type of data blocks, and the number of accesses corresponding to the second type of candidate data block is less than the number of accesses corresponding to other first type of data blocks.

4. The method of claim 3, wherein, The at least one candidate data block comprises a first type of candidate data block. The determining of the at least one candidate data block from the first type of data block based on the access statistics of the first type of data block comprises: determining at least one data block in the first type of data block corresponding to a number of accesses greater than a first access threshold as the first type of candidate data block.

5. The method of claim 3 or 4, wherein, The at least one candidate data block comprises a second type of candidate data block. The determining of the at least one candidate data block from the first type of data block based on the access statistics of the first type of data block comprises: determining at least one data block in the first type of data block corresponding to a number of accesses less than a second access threshold as the second type of candidate data block.

6. The method of claim 2 or 3, wherein, The determining of the at least one candidate data block from the first type of data block based on the access statistics of the first type of data block comprises: sorting each data block in the first type of data block based on the number of accesses corresponding to each data block in the first type of data block respectively to obtain a sorting result; determining the at least one candidate data block from the first type of data block based on the sorting result.

7. The method of claim 3, wherein, The processing of data in the second type of data block based on the access statistics of the second type of data block comprises: determining cold data and / or hot data in the second type of data block based on the number of accesses corresponding to each data block in the second type of data block respectively; processing the cold data and / or hot data in the second type of data block.

8. The method of claim 7, wherein, The at least one candidate data block comprises the first type of candidate data block. The determining the hot data in the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks comprises: determining at least one hot data block from the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks, the access time corresponding to the at least one hot data block being greater than the access times corresponding to other data blocks in the second type of data blocks; determining the data in the at least one hot data block as hot data.

9. The method of claim 7 or 8, wherein, The at least one candidate data block comprises the second type of candidate data block. The determining the cold data in the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks comprises: determining at least one cold data block from the second type of data blocks based on the access times corresponding to each data block in the second type of data blocks, the access time corresponding to the at least one cold data block being less than the access times corresponding to other data blocks in the second type of data blocks; determining the data in the at least one cold data block as cold data.

10. The method of claim 7, wherein, The processing the cold data and / or the hot data in the second type of data blocks comprises: moving the hot data from a first storage medium to a second storage medium, or moving the cold data from the second storage medium to the first storage medium, wherein the performance of the second storage medium is superior to that of the first storage medium.

11. The method of claim 2, wherein, Before the obtaining the access statistical result of the first type of data blocks, the method further comprises: determining the first division size based on a first statistical quantity and the size of the data storage area, the first statistical quantity indicating an upper limit of the number of the first type of data blocks.

12. The method of claim 2, wherein, The first division size is determined based on a storage medium allocation granularity, and the data storage area is allocated to a process according to the storage medium allocation granularity.

13. The method of claim 2, 11 or 12, wherein, Before the obtaining the access statistical result of the second type of data blocks, the method further comprises: determining the second division size based on a second statistical quantity and the size of a memory area composed of the at least one candidate data block, the second statistical quantity indicating an upper limit of the number of the second type of data blocks; or determining the second division size based on the first division size.

14. The method of any one of claims 1-13, wherein, The storage medium corresponding to the data storage area is a volatile storage medium or a non-volatile storage medium.

15. A data processing apparatus, characterized by The apparatus comprises: a first obtaining module configured to obtain an access statistical result of a first type of data blocks, each data block in the first type of data blocks being in a data storage area, each data block in the first type of data blocks having the same size, the access statistical result of the first type of data blocks indicating an access time corresponding to each data block in the first type of data blocks; a first determining module configured to determine at least one candidate data block from the first type of data blocks based on the access statistical result corresponding to the first type of data blocks; and a processing module configured to process cold data and / or hot data in a second type of data blocks, the second type of data blocks being composed of the at least one candidate data block. The second obtaining module is configured to obtain access statistical results of second type data blocks, each candidate data block comprising a plurality of data blocks in the second type data blocks, sizes of respective data blocks in the second type data blocks being the same, and the access statistical results of the second type data blocks indicating respective access frequencies of the respective data blocks in the second type data blocks; The processing module is configured to process data in the second type data blocks based on the access statistical results of the second type data blocks.

16. The apparatus of claim 15, wherein, The first type data blocks are divided according to a first division size, and the second type data blocks are divided according to a second division size, the first division size being greater than the second division size.

17. The apparatus of claim 15 or 16, wherein, The at least one candidate data block comprises a first type candidate data block and / or a second type candidate data block, wherein the first type candidate data block corresponds to an access frequency greater than that of other first type data blocks, and the second type candidate data block corresponds to an access frequency less than that of other first type data blocks.

18. The apparatus of claim 17, wherein, The at least one candidate data block comprises a first type candidate data block. The first determining module is specifically configured to: determine, as the first type candidate data block, at least one data block in the first type data blocks corresponding to an access frequency greater than a first access frequency threshold.

19. The apparatus of claim 17 or 18, wherein, The at least one candidate data block comprises a second type candidate data block. The first determining module is specifically configured to: determine, as the second type candidate data block, at least one data block in the first type data blocks corresponding to an access frequency less than a second access frequency threshold.

20. The apparatus of claim 16 or 17, wherein, The first determining module is specifically configured to: sort respective data blocks in the first type data blocks based on respective access frequencies of the respective data blocks, to obtain a sorting result; determine the at least one candidate data block from the first type data blocks based on the sorting result.

21. The apparatus of claim 17, wherein, The processing module is specifically configured to: determine cold data and / or hot data in the second type data blocks based on respective access frequencies of respective data blocks in the second type data blocks; process the cold data and / or the hot data in the second type data blocks.

22. The apparatus of claim 21, wherein, The at least one candidate data block comprises the first type candidate data block. The processing module is specifically configured to: determine at least one hot data block from the second type data blocks based on respective access frequencies of respective data blocks in the second type data blocks, the at least one hot data block corresponding to an access frequency greater than that of other second type data blocks; determine data in the at least one hot data block as hot data.

23. The apparatus of claim 21 or 22, wherein, The at least one candidate data block comprises the second type candidate data block. The processing module is specifically configured to: determine at least one cold data block from the second type data blocks based on respective access frequencies of respective data blocks in the second type data blocks, the at least one cold data block corresponding to an access frequency less than that of other second type data blocks; determine data in the at least one cold data block as cold data.

24. The apparatus of claim 21, wherein, The processing module is specifically configured to: moving the hot data from the first storage medium to the second storage medium, or moving the cold data from the second storage medium to the first storage medium, wherein the second storage medium has a performance superior to the first storage medium.

25. The apparatus of claim 16, wherein, The apparatus further comprises: a second determining module configured to determine the first division size based on a first statistical quantity and a size of the data storage region, the first statistical quantity indicating an upper limit of a number of the first type of data blocks.

26. The apparatus of claim 16, wherein, The first division size is determined based on a storage medium allocation granularity, and the data storage region is allocated to a process according to the storage medium allocation granularity.

27. The apparatus of claim 16, 25, or 26, wherein, The apparatus further comprises: a third determining module configured to determine the second division size based on a second statistical quantity and a size of a memory region composed of the at least one candidate data block, the second statistical quantity indicating an upper limit of a number of the second type of data blocks; or a fourth determining module configured to determine the second division size based on the first division size.

28. The apparatus of any one of claims 15-27, wherein, The storage medium corresponding to the data storage region is a volatile storage medium or a non-volatile storage medium.

29. A chip, characterized by The chip comprises a processor configured to implement the steps of the method of any one of claims 1-14.

30. A data processing system comprising: The data processing system comprises a processor, a first storage medium and a second storage medium; the processor is configured to process data in the second type of data block according to the method of any one of claims 1-14; In the case where the second type of data block comprises cold data and / or hot data, the processor is configured to move the hot data from the first storage medium to the second storage medium, or move the cold data from the second storage medium to the first storage medium, the second storage medium having a performance superior to the first storage medium.

Images