Data processing method, system, equipment and medium

Abstract

The invention discloses a data processing method, system and device and a medium, and relates to the technical field of storage, and the method comprises the steps: starting from a plurality of dimensions such as a time dimension, a space dimension and a semantic dimension, fusing a time popularity value, a space popularity value and a semantic popularity value, and carrying out the calculation to obtain a popularity score of each data block; the matching between the data in each storage area and the storage area is ensured, the storage area of the data block is adjusted in time when the state of the data block changes, and cold and hot data mixing is avoided; and furthermore, the to-be-migrated data block is determined through the popularity score, and data migration is performed on the to-be-migrated data block, so that the problem of unreasonable migration caused by data migration according to a fixed sequence of a logic space is avoided.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to data processing methods, systems, devices, and media. Background Art

[0002] The development of hard disk storage technology has brought new challenges to storage while improving storage performance, such as problems like the mixing of hot and cold data.

[0003] Existing storage technology solutions still remain at statically partitioning the storage space to store data. By pre-dividing hot and cold zones for partitioned data storage, the mixing of hot and cold data is avoided. However, static partition storage has poor adaptability and can lead to waste of storage resources. Therefore, there is an urgent need for a data processing method for dynamically adjusting hot and cold zone data to solve the above problems. Summary of the Invention

[0004] This application provides a data processing method, system, device, and medium to at least solve the problems in the related art.

[0005] This application provides a data processing method, including:

[0006] Determining the time heat value of data blocks in a storage partition according to a preset sliding window model;

[0007] Determining the space heat value of data blocks according to a preset space prediction model;

[0008] Determining the semantic heat value of data blocks according to a preset semantic heat mapping rule and the file attributes of the data blocks;

[0009] Determining the heat score of data blocks according to the time heat value, space heat value, and semantic heat value;

[0010] Determining the data blocks to be migrated in the storage partition according to the heat score of the data blocks and the type of the storage partition to which they belong, and migrating the data blocks.

[0011] This application also provides a data processing system, including:

[0012] A heat analysis module, configured to determine the time heat value of data blocks in a storage partition according to a preset sliding window model;

[0013] The heat analysis module is further configured to determine the space heat value of data blocks according to a preset space prediction model;

[0014] The heat analysis module is configured to determine the semantic heat value of data blocks according to a preset semantic heat mapping rule and the file attributes of the data blocks;

[0015] A heat analysis module, configured to determine a heat score of a data block according to a time heat value, a space heat value, and a semantic heat value;

[0016] A data migration module, configured to determine data blocks to be migrated within a storage partition and migrate the data blocks according to the heat score of the data blocks and the type of the storage partition to which the data blocks belong.

[0017] The present application further provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above data processing methods when executing the computer program.

[0018] The present application further provides a computer-readable storage medium, in which a computer program is stored, and the computer program implements the steps of any one of the above data processing methods when being executed by a processor.

[0019] The present application further provides a computer program product, including a computer program, and the computer program implements the steps of any one of the above data processing methods when being executed by a processor.

[0020] The present application starts from multiple dimensions such as the time dimension, the space dimension, and the semantic dimension, combines the time heat value, the space heat value, and the semantic heat value to calculate the heat score of each data block, ensures the matching between the data in each storage area and the storage area, and adjusts the storage area of the data block in a timely manner when the state of the data block changes, so as to avoid the mixing of hot and cold data; further determines the data blocks to be migrated through the heat score, and migrates the data blocks to be migrated, thus avoiding the problem of unreasonable migration caused by migrating data in a fixed order according to the logical space. Description of the Drawings

[0021] In order to more clearly illustrate the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0022] Figure 1 A schematic diagram of a data processing method provided by an embodiment of the present application;

[0023] Figure 2 A schematic diagram of an elastic partition reorganization engine architecture provided by an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a garbage collection scheduler architecture provided by an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a data processing system architecture provided by an embodiment of the present application;

[0026] Figure 5 This is a diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. According to the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0028] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variation thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0029] To enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0030] As mentioned in the background art, the ZNS SSD (zoned namespace solid state drive) reduces the mapping table overhead of the FTL (flash translation layer) in traditional SSDs by introducing the concept of zones (continuous block regions of the logical address space), and forces the host to write data in order. However, its unique architecture also brings new problems:

[0031] 1) Mixing of hot and cold data: Due to the fixed capacity of zones and sequential write restrictions, frequently updated hot data and long-term unchanged cold data may coexist in the same zone, resulting in a large amount of valid data being migrated during garbage collection, exacerbating the WAF (Write Amplification Factor), up to 3.8 at most.

[0032] The optimization solutions for data recovery provided by the prior art mainly include: Static partitioning strategy: Pre-divide hot zones and cold zones, store frequently updated data in hot zones, and store long-term unchanged data in cold zones. The advantages are simple implementation and low management overhead, and it can avoid complete mixing of hot and cold data.

[0033] However, existing optimization solutions have drawbacks. For the static partitioning strategy, there are problems such as low resource utilization. In the case of load fluctuation scenarios (such as sudden writes in cloud computing), static partitioning results in a capacity waste of 15%-30%. It cannot handle the dynamic change of data hot and cold attributes (such as log files changing from hot to cold), and requires manual intervention to adjust the partitioning ratio, resulting in poor adaptability. Fixed partitioning leads to frequent cross-Die migrations, increasing additional write amplification and having low cross-region migration efficiency.

[0034] Therefore, in order to provide better services and optimize the stored data, this application provides a dynamic data processing method, as Figure 1 shown, mainly including the following content:

[0035] S1. Determine the time heat value of the data blocks in the storage partition according to a preset sliding window model.

[0036] Specifically, the sliding window model calculates the time heat value of the data blocks according to the interaction request and a preset time heat scoring formula. The interaction request is specifically an IO operation request issued by the user, and the interaction request at least includes the operation type (such as write operation or read operation), the timestamp, and the LBA (Logical Block Address). Among them, the time heat scoring formula is: T(t) represents the time heat value, λ represents the decay coefficient, t i represents the access timestamp of the i-th interaction request, w i represents the access weight of the i-th interaction request. The decay coefficient is preferably set to 0.5, and λ∈[0.01,0.1] is used to control the forgetting rate of historical interaction requests. The optimal value can also be determined by the grid search method. The above access weight can preferably be set by distinguishing the interaction request type. For example, if the interaction request is a read operation, it can be set to 1, and if the interaction request is a write operation, it can be set to 1.2, so as to reflect the stronger impact of the write operation on the data heat. Of course, it can also be set by those skilled in the art according to the actual situation, and this application does not make any limitations in this regard. The above storage partition is divided into two types: a hot storage area and a cold storage area.

[0037] It can be understood that the above sliding window model is preferably an exponentially decaying sliding window model, which assigns higher weights to recent accesses to reflect the temporal locality of data. Among them, the dynamic window size automatically expands and contracts according to the load fluctuation. Specifically, in the case of low load, the window is a fixed window, such as 24 hours; in the case of high load, the dynamic window is activated, with a minimum of 1 hour and a maximum of 72 hours. Specifically, the adjustment is triggered by variance detection, that is, the adjustment is triggered when the variance is greater than 60%. In other implementation scenarios, it can also be set to trigger when the variance is greater than 50%, ensuring the effective utilization of resources while ensuring the correct operation of the model. Among them, the load can be understood as the size of the data block accessed by the interaction request; the setting criteria for high load and low load are set by those skilled in the art according to the actual storage situation, and this application will not expand on this.

[0038] In addition, for the data block of newly written data, its initial time heat value is set to 0.7 to avoid the probability that it is misjudged as cold data by the model due to lack of historical data. To improve the model calculation efficiency, only the time heat value at the current time point is updated, reducing the calculation complexity, and it can be performed on a field programmable gate array in terms of hardware.

[0039] This application calculates the time heat value of the corresponding data block based on the interaction request through the preset sliding window model, calculates the heat value of the data block from the time dimension, enriches the dimension of calculating the heat score of the data block subsequently, and improves the reliability.

[0040] S2. Determine the spatial heat value of the data block according to the preset spatial prediction model.

[0041] Specifically, by obtaining the historical access times, the standard deviation of access intervals, and the request size entropy value of the historical interaction requests of the data block within a preset time period; the preset time period is set according to the dynamic window. For example, if the dynamic window is set to 24 hours, the preset time period is set to 24 hours; the request size entropy value is used to represent the uncertainty or diversity of the request size distribution. The specific acquisition method can be to collect the data of all interaction requests within the preset time period, and then divide these requests into intervals according to the size, and count the number of requests in each interval, so as to obtain the probability p(x i ) of the request appearance corresponding to each interval, and further calculate the entropy value according to the Shannon entropy formula, x represents the random variable of the interaction request size, x i represents the request size range of the i-th interval, and n is the total number of intervals. Among them, the historical access times and the standard deviation of access intervals can be obtained from the recorded access times and times in the metadata of the data block, and the specific calculation method belongs to the conventional setting, and this application will not expand on this here.

[0042] Input the historical access times, the standard deviation of access intervals, and the request size entropy value into the trained spatial prediction model to output the spatial heat value of the data block. Among them, the spatial prediction model is a neural network model. Specifically, the spatial prediction model is a model designed using an LSTM (Long Short-Term Memory) neural network, and the output is a Sigmoid activation function to predict the access probability S(z) of the data block being accessed within the next 24 hours. The training method of the above model is a conventional technical means in the art, and this application will not elaborate on it; of course, it can also be set as other neural network models. Through the preset spatial prediction model, this application calculates the spatial heat value of the data block from the spatial dimension, further improving the authenticity and reliability of subsequent data block scoring.

[0043] In some implementation scenarios, those skilled in the art can also define high-probability regions and low-probability regions according to the size of the access probability S(z). For example, define S(z)>0.8 as the high-probability region and S(z)<0.3 as the low-probability region; it should be noted that in this application, the high-probability region is preferentially stored in the cache layer, and the low-probability region can be marked as a candidate migration data block and preferentially recycled during subsequent garbage collection.

[0044] S3. Determine the semantic heat value of the data block according to the preset semantic heat mapping rule and the file attribute of the data block.

[0045] Specifically, in this application, a semantic heat mapping rule is predefined, including the mapping relationship between file attributes and heat coefficients. Taking a typical storage scenario as an example, the semantic heat mapping rule specifically defines that when the file attribute is a transaction log, the heat coefficient F(f) is 1; when the file attribute is a virtual machine image, the heat coefficient F(f) is 0.8; when the file attribute is a compressed backup, the heat coefficient F(f) is 0.4; when the file attribute is an archived video, the heat coefficient F(f) is 0.2. It can be understood that file attributes include but are not limited to the above several types. In other implementation scenarios, there are also other types of files, and those skilled in the art set the file attributes and corresponding heat coefficients according to the actual scenario and experience. There are various methods for identifying file types. For example, database logs can be identified by file suffix names and modification modes; video streaming media can be determined by parsing the Magic Number; temporary files can be determined by querying the / tmp directory or having the "delete_on_close" flag; archived files are files with a last access time>30 days and marked as read-only. It can be understood that when a file is renamed or its permissions are changed, it is necessary to re-invoke the semantic heat mapping rule to determine the heat coefficient of the changed file.

[0046] It is understandable that the above steps S1 - S3 can be executed simultaneously, and there is no order restriction among the above steps S1 - S3. One can first execute step S1, then step S2, and finally step S3; of course, one can also first execute step S3, then step S2, and finally step S1.

[0047] S4. Determine the heat score of the data block according to the time heat value, the space heat value, and the semantic heat value.

[0048] Specifically, the heat score of the data block is calculated using a weighted fusion formula, and the formula is:

[0049] H = 0.5T(t) + 0.3S(z) + 0.2F(f);

[0050] Among them, as disclosed above, the time heat value T(t) is used to reflect the short - term access heat of the data block, the space heat value S(z) is used to predict the long - term access trend of the data, and the semantic heat value F(f) is used to provide prior knowledge guidance. Of course, if data clearly marked as "archived files" is accessed recently, it should also be determined as cold data. Among them, the weights of the respective heat values in the weighted fusion formula are adjusted according to the load of the interaction request.

[0051] S5. Determine the data blocks to be migrated within the storage partition according to the heat score of the data block and the type of the storage partition to which it belongs, and migrate the data blocks.

[0052] Specifically, if the heat score of the data block belongs to the first heat range and the storage partition to which the data block belongs is a hot storage area, determine the data block as the first data block; if the heat score of the data block belongs to the second heat range and the storage partition to which the data block belongs is a hot storage area, determine the data block as the second data block; if the heat score of the data block belongs to the third heat range and the storage partition to which the data block belongs is a hot storage area, determine the data block as the data block to be migrated, that is, to migrate the "cold data" in the hot storage area; among them, the recovery priority of the data block to be migrated is greater than the recovery priority of the second data block, and the recovery priority of the second data block is greater than the recovery priority of the first data block. That is, the first data block has a higher heat score and high data utilization, so its priority is low during garbage collection. On the contrary, the third data block has a low heat score and low data access degree, and has the highest priority during garbage collection.

[0053] In some implementation scenarios, the above first heat range can be set to [0.8, 1], the above second heat range can be set to [0.5, 0.8), and the above third heat range can be set to [0, 0.5); of course, the setting of the heat range can also be adaptively modified by those skilled in the art, and this application does not make any limitations in this regard.

[0054] This application dynamically evaluates the heat score of data blocks by integrating time heat value, space heat value, and semantic heat value, ensuring that the data stored in the hot storage area is hot data. When the status of the data block changes, the storage area of the data block is adjusted in a timely manner to avoid the mixed storage of hot and cold data, thus realizing the dynamic storage of data. In addition, determining the data blocks to be migrated based on the heat score and avoiding the unreasonable migration problem caused by migrating data in a fixed order according to the logical space when migrating the data blocks to be migrated.

[0055] It can be understood that in this application, the storage area is divided into a hot storage area and a cold storage area. Generally speaking, the hot storage area has the characteristics of elastic capacity. The initial proportion is usually 15% of the entire storage system, and the maximum can be expanded to 30% (dynamically calculated according to the total capacity of the solid-state drive); it allows non-strict sequential writing, supports cross-block updates, and enables DRAM (Dynamic Random Access Memory) cache to accelerate write operations; its metadata optimization updates data by means of storing file system logs, database WAL (Write-Ahead Logging), etc. The cold storage area has the following characteristics: strictly comply with the sequential writing requirements of the ZNS protocol, and use the LZ4 algorithm to compress cold data (the average compression rate reaches 50%), so as to reduce the physical space occupation; the data stored therein is marked as read-only after migration to avoid invalid updates.

[0056] On this basis, this application also proposes that after determining the data blocks to be migrated in the storage area and migrating the data blocks according to the heat score of the data blocks and the type of the storage partition to which they belong, the hot and cold storage areas of the data are managed, and it is applied to, for example, Figure 2 the dynamic partition controller in the elastic partition reorganization engine as shown, calculates the capacity ratio of the hot storage area and the cold storage area based on the heat distribution, and triggers the expansion / contraction operation, specifically including:

[0057] Calculate the utilization rate of the hot area space in the hot storage area and the utilization rate of the cold area space in the cold storage area. In response to detecting that the utilization rate of the hot area space is greater than the first threshold, allocate idle logical space to the hot storage area. The first threshold is preferably set to 85%. In response to detecting that the utilization rate of the hot area space is less than the second threshold and the continuous duration is greater than the preset time period, release the first specified logical space in the hot storage area. The second threshold is preferably set to 60%, and the above preset time period is preferably set to 30 minutes. In response to detecting that the utilization rate of the cold area space is less than the third threshold, force the migration of the data blocks in the second specified logical space in the cold storage area to the secondary storage space. The third threshold is preferably set to 5%. The secondary storage space is auxiliary storage or external storage, which is used to store data and programs for a long time, such as a hard disk drive (HDD).

[0058] This application dynamically adjusts the partition layout of the storage area according to data heat, realizes the intelligent division and data migration of the hot storage area and the cold storage area, reduces invalid data migration, optimizes the physical layout, reduces the write amplification factor, and thus improves the overall performance and lifespan of the solid-state drive.

[0059] In addition, the elastic partition reorganization engine further includes a monitoring module, a decision-making module, and an execution module. The monitoring module consists of a tracker and a heat feedback interface. The tracker collects in real time metrics such as the access frequency, request type, and data block size of each logical space; the heat feedback interface is used to receive the heat scores of each data block disclosed above, and further mark the high-heat data blocks and low-heat data blocks. The decision-making module consists of a dynamic partition controller and a migration scheduler. The migration scheduler generates an optimal migration sequence of data blocks to be migrated according to the existing migration cost model. The execution module consists of an asynchronous migration engine and a hardware acceleration module. The asynchronous migration engine is used to perform data transfer in the background and support breakpoint resume and priority scheduling; the hardware acceleration module realizes high-speed DMA (Direct Memory Access) transfer through a programmable logic sequence, thereby reducing CPU resource occupancy.

[0060] It can be understood that there are still problems with the inefficient GC (Garbage Collection) trigger mechanism in the existing technology for storing based on static partition strategies: The GC strategy based on a fixed threshold (such as the space utilization rate exceeding 80%) cannot adapt to dynamic loads, and is prone to causing I / O latency spikes in high-concurrency write scenarios, with the latency fluctuation amplitude reaching 47%. Therefore, passive GC triggering is mostly adopted, and there are serious problems with performance jitter for the passive GC triggering strategy. In high-concurrency write scenarios, sudden GC causes the tail latency to soar, for example, the 99% latency increases from 5 ms to 15 ms. In addition, there are also problems such as the lack of priority scheduling and the competition for bandwidth between GC tasks and user I / Os; and the inability to perform predictive optimization: The passive response mechanism cannot release space in advance, resulting in forced execution of high-cost GC during peak periods.

[0061] Therefore, on the basis of the above technical solutions, an active garbage collection trigger mechanism is further proposed, specifically applied to the multi-level trigger engine in the garbage collection scheduler as Figure 3 shown, including the following content:

[0062] Generate a garbage collection request according to the space utilization rate, the distribution of invalid pages, and the predicted space demand rate; generate a first garbage collection request in response to detecting that the space utilization rate of the cold area is less than a third threshold; generate a second garbage collection request in response to detecting that the proportion of invalid pages in the hot storage area is greater than a fourth threshold; generate a third garbage collection request in response to detecting that the predicted space demand rate of the hot storage area is greater than a fifth threshold; wherein, the priority of the first garbage collection request is greater than the priority of the second garbage collection request, and the priority of the second garbage collection request is greater than the priority of the third garbage collection request. The value of the above-mentioned third threshold is the same as that disclosed previously. The above-mentioned fourth threshold is preferably set to 40%, and the above-mentioned fifth threshold is preferably set to 90%. Among them, the space utilization rate and the distribution of invalid pages can be directly monitored. The predicted space demand rate is output by a trained space prediction model.

[0063] In some implementation scenarios, the present application also proposes to monitor the interaction operation delay; in response to detecting that the interaction operation delay is greater than a preset delay threshold, change the priority of the second garbage collection request, that is, reduce the priority of the second garbage collection request. The above-mentioned preset delay threshold can be 10 ms. At the same time, the garbage collection task corresponding to the third garbage collection request can also be paused.

[0064] It can be understood that after the first garbage collection request is triggered, all user interaction requests are paused, and the garbage collection is executed at full speed to prevent write blocking caused by space exhaustion; after the second garbage collection request is triggered, background asynchronous recycling is performed to limit the CPU occupancy rate, thereby balancing space recycling and performance loss; the third garbage collection request is only executed during idle periods, so as to realize the utilization of idle bandwidth to release space in advance. The first garbage collection request has the highest priority and belongs to emergency garbage collection, which can occupy all resources and can also preempt all other tasks; the second garbage collection request has the second highest priority and belongs to regular garbage collection, with the CPU resource occupancy remaining within 20% and the bandwidth occupancy remaining within 30%, and it can only be preempted by the first garbage collection request task; the third garbage collection request has the lowest priority and belongs to preventive garbage collection, with the CPU resource occupancy remaining within 10% and the bandwidth occupancy remaining within 15%, and it cannot preempt other tasks and is only executed during idle times. It can be understood that in the present application, weights can be assigned to the garbage collection requests of each priority, for example, emergency: regular: preventive = 6:3:1; thereby ensuring the rapid completion of high-priority tasks.

[0065] It can be understood that the garbage collection scheduler further includes a monitoring module, a decision-making module, and an execution module. The monitoring module includes a QoS (Quality of Service) metric collection module for detecting the latency, throughput, and queue depth of user interaction requests; the monitoring module also includes a resource status monitoring module for monitoring the solid-state drive space utilization rate, invalid page distribution, and flash temperature, etc. The decision-making module includes a multi-level trigger engine for generating garbage collection requests based on the space utilization rate, invalid page ratio, and predicted space demand rate; the decision-making module also includes a priority allocator for classifying garbage collection requests into priorities. The execution module includes a concurrent executor for processing garbage collection tasks in a multi-threaded manner, supporting task preemption and resource isolation. The execution module also includes a bandwidth controller for restricting the CPU resources and bandwidth occupied by garbage collection tasks.

[0066] This application determines whether to trigger a garbage collection request based on the space utilization rate, invalid page ratio, and predicted space demand rate, and classifies the triggered garbage collection requests into priorities, balancing the garbage collection efficiency and the quality of user interaction services, minimizing the interference of garbage collection on user interaction requests; while implementing dynamic priority scheduling, it releases space in advance through the third garbage collection request to avoid sudden GCs.

[0067] In the script settings of the specific implementation scenario, this application implements the above steps through the following content:

[0068] 1. Initialize the configuration and preset the parameters. The pseudocode is as follows:

[0069] hot_zone_ratio = 20% / / Initial ratio of the hot zone;

[0070] gc_cold_threshold = 5% / / Emergency threshold for cold zone space;

[0071] gc_hot_invalid = 40% / / Conventional threshold for invalid pages in the hot zone;

[0072] migration_unit = 4MB / / Align with NVMe MTU;

[0073] Model loading:

[0074] Deploy a pre-trained LSTM model (optimized by TensorRT, model size 8MB, inference latency < 1ms).

[0075] Load the semantic rule library (including more than 500 file type features and lifecycle labels).

[0076] 2. Real-time operation process.

[0077] First, data collection and heat calculation: The model inputs are I / O requests (LBA, operation type, timestamp), and file metadata (inode, extended attributes). The model processes in the time dimension: sliding window statistics (window size dynamically adjusted, range 1 - 72 hours). In the space dimension: LSTM is used to predict the access probability of zones in the next 24 hours. In the semantic dimension: the file type and lifecycle tags are parsed. The model output is the comprehensive heat score H∈[0,1] for each data block.

[0078] Then, dynamic partition adjustment is performed: Expansion of the hot zone: When the space utilization rate of the hot zone > 85%, it is expanded according to the following strategy. Pseudo-code example:

[0079] def expand_hot_zone(current_ratio):

[0080] if current_ratio < 30%:

[0081] new_ratio = min(current_ratio + 5%, 30%)

[0082] allocate_zones(new_ratio - current_ratio)

[0083] Compression of the cold zone: LZ4 compression (compression level 1, balancing speed and ratio) is performed on the data migrated to the cold zone. Finally, garbage collection task scheduling: Task generation: GC requests are generated according to multi-level trigger conditions, and priorities are assigned (0 - urgent, 1 - normal, 2 - preventive). Scheduling execution. Pseudo-code example:

[0084]

[0085] An embodiment of the present application also provides a data processing system, as Figure 4 shown, including:

[0086] A heat analysis module 410, configured to determine the time heat value of a data block in a storage partition according to a preset sliding window model;

[0087] The heat analysis module 410 is further configured to determine the space heat value of a data block according to a preset space prediction model;

[0088] The heat analysis module 410 is configured to determine the semantic heat value of a data block according to a preset semantic heat mapping rule and the file attributes of the data block;

[0089] The heat analysis module 410 is configured to determine the heat score of a data block according to the time heat value, the space heat value, and the semantic heat value;

[0090] A data migration module 420 is configured to determine data blocks to be migrated within a storage partition based on the heat score of the data blocks and the type of the storage partition to which they belong, and migrate the data blocks.

[0091] For the description of the features in the corresponding embodiments of the above data processing system, reference can be made to the relevant descriptions in the corresponding embodiments of the data processing method, which will not be elaborated here one by one.

[0092] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above embodiments of the data processing method:

[0093] Determine the time heat value of the data blocks within the storage partition according to a preset sliding window model;

[0094] Determine the space heat value of the data blocks according to a preset space prediction model;

[0095] Determine the semantic heat value of the data blocks according to a preset semantic heat mapping rule and the file attributes of the data blocks;

[0096] Determine the heat score of the data blocks according to the time heat value, the space heat value, and the semantic heat value;

[0097] Determine the data blocks to be migrated within the storage partition based on the heat score of the data blocks and the type of the storage partition to which they belong, and migrate the data blocks.

[0098] Among them, Figure 5 An exemplary architecture of the electronic device is shown, which may specifically include a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, and a memory 520. The above processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520 can be communicatively connected through a bus 530.

[0099] Among them, the processor 510 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solutions provided by the present application.

[0100] The memory 520 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 520 may store an operating system 521 for controlling the execution of the electronic device 500, and a basic input / output system (BIOS) 522 for controlling the low-level operations of the electronic device 500. Additionally, a web browser 523, a data storage management system 524, an icon font processing system 525, etc. may also be stored. The above-mentioned icon font processing system 525 may be the application program that specifically implements the operations of the foregoing steps in the embodiments of the present application. In summary, when implementing the technical solution provided by the present application through software or firmware, the relevant program codes are stored in the memory 520 and called and executed by the processor 510.

[0101] The input / output interface 513 is used to connect to the input / output module to implement information input and output. The input / output module may be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.

[0102] The network interface 514 is used to connect to a communication module (not shown in the figure) to implement communication interaction between this device and other devices. Among them, the communication module may implement communication in a wired manner (such as USB, network cable, etc.) or in a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0103] The bus 530 includes a path for transmitting information between the various components of the device (such as the processor 510, the video display adapter 511, the disk drive 512, the input / output interface 513, the network interface 514, and the memory 520).

[0104] In addition, the electronic device 500 may also obtain information on specific collection conditions from the virtual resource object collection condition information database for condition judgment, etc.

[0105] It should be noted that although the above device only shows the processor 510, the video display adapter 511, the disk drive 512, the input / output interface 513, the network interface 514, the memory 520, the bus 530, etc., in the specific implementation process, the device may also include other components necessary for normal execution. In addition, those skilled in the art can understand that the above device may also only include the components necessary for implementing the solution of the present application and does not necessarily include all the components shown in the figure.

[0106] Embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any of the above data processing method embodiments when running:

[0107] Determine the time heat value of the data block in the storage partition according to a preset sliding window model;

[0108] Determine the space heat value of the data block according to a preset space prediction model;

[0109] Determine the semantic heat value of the data block according to a preset semantic heat mapping rule and the file attributes of the data block;

[0110] Determine the heat score of the data block according to the time heat value, space heat value and semantic heat value;

[0111] Determine the data blocks to be migrated in the storage partition according to the heat score of the data block and the type of the storage partition to which it belongs, and migrate the data blocks.

[0112] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks or optical discs that can store computer programs.

[0113] Embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above data processing method embodiments:

[0114] Determine the time heat value of the data block in the storage partition according to a preset sliding window model;

[0115] Determine the space heat value of the data block according to a preset space prediction model;

[0116] Determine the semantic heat value of the data block according to a preset semantic heat mapping rule and the file attributes of the data block;

[0117] Determine the heat score of the data block according to the time heat value, space heat value and semantic heat value;

[0118] Determine the data blocks to be migrated in the storage partition according to the heat score of the data block and the type of the storage partition to which it belongs, and migrate the data blocks.

[0119] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above data processing method embodiments are implemented:

[0120] Determine the time heat value of the data block in the storage partition according to a preset sliding window model;

[0121] Determine the space heat value of the data block according to a preset space prediction model;

[0122] Determine the semantic heat value of the data block according to a preset semantic heat mapping rule and the file attribute of the data block;

[0123] Determine the heat score of the data block according to the time heat value, the space heat value and the semantic heat value;

[0124] Determine the data block to be migrated in the storage partition according to the heat score of the data block and the type of the storage partition to which it belongs, and migrate the data block.

[0125] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this application can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0126] The above has introduced in detail a data processing method provided by this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized in that, The method includes: Determining the time heat value of data blocks in a storage partition according to a preset sliding window model; Determining the spatial heat value of the data blocks according to a preset spatial prediction model; Determining the semantic heat value of the data blocks according to a preset semantic heat mapping rule and the file attributes of the data blocks; Determining the heat score of the data blocks according to the time heat value, spatial heat value, and semantic heat value; Determining the data blocks to be migrated in the storage partition according to the heat score of the data blocks and the type of the storage partition to which they belong, and migrating the data blocks; 2. The method according to claim 1, wherein The type of the storage partition includes a hot storage area and a cold storage area. After determining the data blocks to be migrated in the storage partition according to the heat score of the data blocks and the type of the storage partition to which they belong, and migrating the data blocks, the method includes: Calculating the utilization rate of the hot area space in the hot storage area and the utilization rate of the cold area space in the cold storage area; Allocating idle logical space to the hot storage area in response to detecting that the utilization rate of the hot area space is greater than a first threshold; Releasing a first specified logical space in the hot storage area in response to detecting that the utilization rate of the hot area space is less than a second threshold and the duration is greater than a preset time period; Forcibly migrating the data blocks in a second specified logical space in the cold storage area to a secondary storage space in response to detecting that the utilization rate of the cold area space is less than a third threshold; 3. The method according to claim 1, characterized in that, The method further includes: Generating a garbage collection request according to the space utilization rate, invalid page distribution, and predicted space demand rate; Generating a first garbage collection request in response to detecting that the utilization rate of the cold area space is less than the third threshold; Generating a second garbage collection request in response to detecting that the proportion of invalid pages in the hot storage area is greater than a fourth threshold; Generating a third garbage collection request in response to detecting that the predicted space demand rate of the hot storage area is greater than a fifth threshold; Wherein, the priority of the first garbage collection request is greater than the priority of the second garbage collection request, and the priority of the second garbage collection request is greater than the priority of the third garbage collection request.

4. The method according to claim 3, characterized in that, The method further includes: Monitoring the interaction operation delay; Changing the priority of the second garbage collection request in response to detecting that the interaction operation delay is greater than a preset delay threshold; 5. According to the method described in any one of claims 1-4, characterized in that, The determining the time heat value of data blocks in a storage partition according to a preset sliding window model includes: The sliding window model calculates the time heat value of the data blocks according to interaction requests and a preset time heat score formula; The time heat score formula is as follows: Among them, the T(t) represents the time heat value, λ represents the attenuation coefficient, and t i represents the access timestamp of the i-th interaction request, and w i represents the access weight of the i-th interaction request, and n represents the total number of interaction requests.

6. The method according to any one of claims 1-4, characterized in that, The determining the spatial heat value of the data blocks according to a preset spatial prediction model includes: Obtaining the historical access times, access interval standard deviation, and request size entropy value of historical interaction requests of the data blocks within a preset time period; Inputting the historical access times, access interval standard deviation, and request size entropy value into the trained spatial prediction model to output the spatial heat value of the data blocks, wherein the spatial prediction model is a neural network model.

7. According to the method described in any one of claims 1-4, characterized in that, The determining the data blocks to be migrated in the storage partition according to the heat score of the data blocks and the type of the storage partition to which they belong further includes: If the heat score of the data block belongs to the first heat range and the storage partition to which the data block belongs is a hot storage area, determine that the data block is a first data block; If the heat score of the data block belongs to the second heat range and the storage partition to which the data block belongs is a hot storage area, determine that the data block is a second data block; If the heat score of the data block belongs to the third heat range and the storage partition to which the data block belongs is a hot storage area, determine that the data block is a data block to be migrated; Wherein, the recycling priority of the data block to be migrated is greater than the recycling priority of the second data block, and the recycling priority of the second data block is greater than the recycling priority of the first data block.

8. A data processing system, characterized in that, The system includes: A heat analysis module, configured to determine the time heat value of the data block in the storage partition according to a preset sliding window model; The heat analysis module is further configured to determine the space heat value of the data block according to a preset space prediction model; The heat analysis module is configured to determine the semantic heat value of the data block according to a preset semantic heat mapping rule and the file attributes of the data block; The heat analysis module is configured to determine the heat score of the data block according to the time heat value, the space heat value, and the semantic heat value; A data migration module, configured to determine the data block to be migrated in the storage partition and migrate the data block according to the heat score of the data block and the type of the storage partition to which it belongs.

9. An electronic device, characterized in that, Including: A memory, configured to store a computer program; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of the data processing method according to any one of claims 1 to 7 when executed by a processor.

Images