Data storage methods and electronic devices

Abstract

This application discloses a data storage method and electronic device, particularly relating to the field of data storage technology. The method includes: after deduplication analysis of the data to be stored is completed, determining the number of duplicate data blocks in at least one storage device based on storage device information of at least one duplicate data block; performing priority analysis on at least one storage device based on at least one load data of the at least one storage device to determine candidate storage devices for a target non-duplicate data block based on the priority analysis results; performing storage balancing verification on the candidate storage devices based on the number of duplicate data blocks in the candidate storage devices to determine the target storage device for the target non-duplicate data block, and storing the target non-duplicate data block in the target storage device. This effectively ensures load balancing across storage devices during the storage of non-duplicate data blocks, reducing the adverse effects caused by data fragmentation.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage method and electronic device. Background Technology

[0002] Deduplication is a key technology in data storage used to eliminate redundant data and optimize storage space utilization. It is widely used in data management scenarios of various storage systems. Currently, deduplication technologies mainly involve physically distributing logically continuous data. After deduplication, duplicate data blocks tend to be concentrated on a few disks, causing an imbalance in data distribution across disks. Summary of the Invention

[0003] This application provides a data storage method and an electronic device to at least solve the problem of uneven data distribution among disks caused by the centralized storage of duplicate data in related technologies.

[0004] This application provides a data storage method, including: When the deduplication analysis of the data to be stored is completed, the number of deduplication blocks in the at least one storage device is determined based on the storage device information of at least one deduplication block. The at least one deduplication block is at least one data block in the data to be stored that has been stored in the storage device after the deduplication analysis. Based on at least one load data of at least one storage device, priority analysis is performed on at least one storage device to determine candidate storage devices for a target non-duplicate data block from at least one storage device according to the priority analysis results. The target non-duplicate data block is any data block among at least one non-duplicate data blocks in the data to be stored that has not been stored in the storage device after the duplicate data analysis. Based on the number of duplicate data blocks in the candidate storage devices, a storage balance verification is performed on the candidate storage devices to determine the target storage device for the target non-duplicate data block according to the verification result, and the target non-duplicate data block is stored in the target storage device.

[0005] This application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of any of the above data storage methods.

[0006] The data storage method and electronic device provided in this application analyze the distribution of the number of duplicate data blocks stored in the data to be stored on each storage device, and combine the real-time load data of each storage device to dynamically filter and determine the target storage device for the target non-duplicate data block to be stored. This can effectively ensure the load balance of each storage device during the storage of non-duplicate data blocks, reduce the adverse effects caused by data fragmentation, achieve balanced distribution of storage load, and thus significantly improve the concurrent reading efficiency during data recovery, and comprehensively optimize the operating performance of the entire storage system. Attached Figure Description

[0007] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A schematic diagram illustrating the deduplication process provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the first data storage method provided in an embodiment of this application. Figure 3 This is a flowchart illustrating the second data storage method provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the third data storage method provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating a specific data storage method provided in an embodiment of this application. Figure 6 This is a schematic diagram of disk data in a data storage system provided in an embodiment of this application; Figure 7 This is a schematic diagram of the data on each disk after data storage, provided in an embodiment of this application. Figure 8 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0010] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0011] Deduplication is a key technology in data storage used to eliminate redundant data and optimize storage space utilization. Its core principle is as follows: Before data is written to the storage system, the data to be stored is divided into several data blocks, and a unique hash fingerprint is generated for each data block using a preset algorithm. The system identifies duplicate data blocks in the data to be stored by comparing these hash fingerprints. For each identified duplicate data block, the system retains only one physical copy and replaces the original duplicate data block with a pointer to that physical copy. Specifically... Figure 1 The diagram shown illustrates the process of deduplication.

[0012] In scenarios involving duplicate data, only a unique data block needs to be written to the physical storage device, and the hash fingerprint of the new data block is updated to a pre-defined fingerprint index table to provide a basis for deduplication operations during subsequent data writing. While this approach can effectively reduce duplicate data and save storage space, it can also lead to data fragmentation. Logically contiguous data to be stored may be distributed across different physical storage locations, and physical copies of some duplicate data blocks may be concentrated on the same disk, resulting in uneven data distribution across storage disks and ultimately affecting the speed of subsequent data recovery.

[0013] To address the aforementioned shortcomings of existing deduplication technologies, this application proposes a novel deduplication management strategy aimed at improving storage balance after data deduplication, thereby enhancing data recovery speed and optimizing the overall performance of the storage system.

[0014] To address the issues of uneven data distribution across disks and slow data recovery speeds due to insufficient data cache resources after data deduplication in related technologies, this application dynamically determines candidate storage devices by statistically analyzing the distribution of duplicate data blocks stored in the data to be stored across various storage devices, combined with the load data of each storage device, and then filters the target storage device using storage balancing verification. This achieves load balancing across multiple storage devices when writing non-duplicate data blocks, thereby alleviating cache resource pressure and improving data recovery efficiency and overall system performance.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Figure 2 This is a flowchart illustrating the first data storage method provided in an embodiment of this application.

[0017] like Figure 2 As shown, the method includes the following steps: Step 101: After the deduplication analysis of the data to be stored is completed, determine the number of deduplicated data blocks in at least one storage device based on the storage device information of at least one deduplicated data block.

[0018] In some embodiments, a duplicate data block is a data block that has already been stored in a storage device from the data to be stored, which has been determined after duplicate data analysis.

[0019] Storage device information is used to identify the physical storage location of duplicate data blocks, such as disk ID, storage node number, etc.

[0020] The number of duplicate data blocks is the total number of duplicate data blocks belonging to the data to be stored on each storage device.

[0021] Specifically, this application first performs deduplication analysis on the data to be stored to identify at least one duplicate data block and at least one non-duplicate data block in the data to be stored. Then, based on a preset identifier index table, it determines at least one metadata of at least one stored data block corresponding to at least one duplicate data block; it extracts at least one storage device identifier of at least one stored data block from the at least one metadata, and determines the at least one storage device identifier as the storage device information of at least one duplicate data block; based on the storage device information, it determines the number of duplicate data blocks in the at least one storage device.

[0022] The specific process of performing duplicate data analysis on the data to be stored in this application is as follows: dividing the data to be stored into at least one data block to be processed; generating a first data block identifier for the first data block to be processed based on a preset identifier generation algorithm, wherein the first data block to be processed is any one of the at least one data block to be processed; comparing the first data block identifier with the stored identifiers of the stored data blocks in the preset identifier index table; if it is determined that there is a stored identifier in the preset identifier index table that matches the first data block identifier, determining that the first data block to be processed is a duplicate data block; if it is determined that there is no stored identifier in the preset identifier index table that matches the first data block identifier, determining that the first data block to be processed is a non-duplicate data block.

[0023] Step 102: Based on at least one load data of at least one storage device, perform priority analysis on at least one storage device to determine candidate storage devices for the target non-duplicate data block from at least one storage device according to the priority analysis results.

[0024] In this application, the candidate storage device is used to attempt to store a target non-repeating data block, which is any data block among at least one non-repeating data block in the data to be stored that has not been stored in the storage device.

[0025] In some embodiments, this application can prioritize storage devices based on their current load, prioritizing devices with lower loads as candidate storage devices to avoid data being concentrated on high-load devices and ensure overall system load balance.

[0026] Among them, load data reflects the current busy level of storage devices, such as the number of data blocks written or the utilization rate of each storage device.

[0027] Priority analysis refers to sorting storage devices according to their load from low to high, with lower loads indicating higher priority.

[0028] Candidate storage devices are storage devices initially selected based on priority and used for writing non-duplicate data blocks.

[0029] The target non-duplicate data block is a non-duplicate data block in the data to be stored that has not yet been written to any storage device and needs to be stored at present.

[0030] Specifically, this application can sort at least one storage device from low to high according to the load data of at least one storage device, and determine the storage device with the highest sorted load as a candidate storage device.

[0031] Step 103: Perform storage balancing verification on the candidate storage devices based on the number of duplicate data blocks in the candidate storage devices, so as to determine the target storage device for the target non-duplicate data block according to the verification result, and store the target non-duplicate data block in the target storage device.

[0032] In some embodiments, this application can perform storage balance verification on candidate storage devices based on the number of duplicate data blocks on the candidate storage devices to determine whether the device is suitable for writing. If suitable, it is selected as the target device; if unsuitable, candidate devices are replaced according to priority until a device that meets the balance condition is selected, thereby avoiding data concentration and achieving balanced distribution.

[0033] The target storage device is the storage device that is ultimately determined to be used to write the target non-duplicate data blocks.

[0034] After writing the target non-repeating data block to the target storage device, this application can further determine the target non-repeating data block identifier of the target non-repeating data block and the target storage device identifier of the target storage device as the target non-repeating data block metadata of the target non-repeating data block, and update the target non-repeating data block metadata to the preset identifier index table.

[0035] The identifier index table is an index structure used to record information such as data block identifiers, corresponding storage devices, and metadata, which is used for subsequent deduplication and location.

[0036] In summary, by first counting the number of duplicate data blocks in each storage device, then dynamically determining candidate storage devices based on the storage device load, and finally selecting the target storage device based on the storage balance verification results of the candidate storage devices, we can effectively avoid uneven data distribution on each disk in deduplication scenarios, alleviate cache resource shortages and data fragmentation problems, and significantly improve data recovery speed and overall storage system performance.

[0037] Figure 3 A flowchart illustrating the second data storage method proposed in this application is further shown. Based on Figure 2 The illustrated embodiment further explains step 103. Figure 3 This may include the following steps.

[0038] Step 201: Perform storage balancing verification on the candidate storage devices based on the number of duplicate data blocks in the candidate storage devices, and determine the verification result of the candidate storage devices.

[0039] In some embodiments, if too many duplicate data blocks related to the currently stored data already exist on the candidate storage device, the current target non-duplicate data block will no longer be written to the candidate storage device to avoid further data concentration.

[0040] Specifically, when the number of duplicate data blocks in a candidate storage device is less than a preset threshold, the device load and distribution status are determined to be reasonable and the storage balance condition is met; when the number of duplicate data blocks in a candidate storage device is greater than or equal to the preset threshold, the duplicate blocks in the device are determined to be too concentrated and the storage balance condition is not met.

[0041] Among them, candidate storage devices are storage devices initially selected based on load priority and used to attempt to write non-duplicate data blocks.

[0042] The number of duplicate data blocks is the statistical number of duplicate data blocks that belong to the data to be stored in the current candidate storage device and already exist.

[0043] Storage balancing conditions are rules used to determine whether data can be written to a device, in order to avoid uneven data distribution.

[0044] The preset threshold is a pre-defined upper limit on the number of duplicate data blocks, used to distinguish whether the device is "too concentrated".

[0045] Step 202: If the verification result indicates that the candidate storage device meets the storage balance conditions, the candidate storage device is determined as the target storage device.

[0046] In some embodiments, if a candidate storage device meets the load balancing requirements, it is directly used as the final target storage device for writing, ensuring that non-duplicate data blocks are written to devices with balanced distribution and reasonable load. The target storage device is the storage device ultimately used to actually write the target non-duplicate data blocks.

[0047] Step 203: If the verification result indicates that the candidate storage device does not meet the storage balance condition, determine the update candidate storage device among at least one storage device according to the priority analysis result, and perform storage balance verification on the update candidate storage device based on the number of duplicate data blocks in the update candidate storage device to determine the target storage device for non-duplicate data blocks.

[0048] In some embodiments, if the current candidate storage device does not meet the balancing conditions, the next candidate storage device is replaced according to the load priority order (i.e., the candidate storage device is updated), and the judgment continues until a device that meets the data balancing storage conditions is found, so as to achieve dynamic and balanced selection of the write location.

[0049] At the same time, this application also needs to adjust the number of duplicate data blocks for storage devices that do not meet the conditions, so that the updated count can be used in the next data write, avoiding the same device being repeatedly rejected, and further improving the leveling effect.

[0050] Among them, the updated candidate storage device is the next candidate storage device selected based on the priority analysis results.

[0051] Updating a non-duplicate data block involves storing other non-duplicate data blocks that will be stored later.

[0052] The preset step size is the unit length used to adjust the number of repeated data blocks, usually 1 (corresponding to repeat_chunk_disk[ID]--).

[0053] In summary, this application first determines whether the candidate storage device meets the storage balancing conditions, and writes directly when the conditions are met, and dynamically replaces the candidate storage device when the conditions are not met, and adjusts the number of duplicate data blocks. This can effectively avoid uneven data distribution among storage devices in deduplication scenarios, alleviate cache resource shortages and data fragmentation problems, achieve load balancing among multiple devices, and significantly improve data recovery speed and overall storage system performance.

[0054] Figure 4 A flowchart illustrating the third data storage method proposed in this application is further shown. Based on Figure 2 The illustrated embodiment will further explain the process after step 103. Figure 4 This may include the following steps.

[0055] Step 301: Determine at least one usage frequency information of at least one data block stored in at least one storage device.

[0056] In some embodiments, within data storage and management scenarios, duplicate data, due to its high reusability, is often cross-referenced by multiple different data files. Its read frequency is significantly higher than that of unique, non-duplicate data; in some applications, the access frequency of duplicate data can even be several times to tens of times higher than that of unique data. This high-frequency access characteristic makes duplicate data a key factor affecting the overall performance of the storage system. Prioritizing its loading into the cache is crucial for improving storage response speed, optimizing data access efficiency, and reducing the operational pressure on backend disk devices.

[0057] As a high-speed data exchange layer in a storage system, caching enables rapid reading of frequently accessed data, significantly shortening data access paths and reading time. It avoids direct access to the backend disk on every request, effectively reducing disk I / O load and improving storage system efficiency. However, due to limitations in hardware cost and system architecture, storage systems typically have limited cache resources. How to achieve efficient caching and scheduling of duplicate data within this limited cache space becomes a key issue in improving system performance. Improper cache resource allocation can easily lead to low cache hit rates and wasted cache resources, making it difficult to fully leverage the acceleration effect of caching. To address these issues, this application further proposes a cache optimization strategy for duplicate data. By accurately identifying high-value, high-frequency duplicate data, it maximizes the cache hit rate of duplicate data within the constraints of limited cache resources, effectively reducing the access pressure on the backend disk, achieving efficient data access acceleration, and significantly improving the overall operating efficiency and stability of the storage system.

[0058] This application can obtain the usage frequency information of each data block by statistically analyzing the historical read count, read interval, system access records, etc., which can be used for subsequent cache optimization strategies.

[0059] Frequency of use information indicates how frequently a data block is accessed, read, or hit within a certain period of time. It may include the number of accesses, the frequency of readings, the popularity value, etc. In this application, frequency of use information refers to the number of times the data block is referenced.

[0060] Step 302: Based on at least one frequency information used, determine the high-frequency data block that meets the preset high-frequency conditions.

[0061] In some embodiments, this application can identify data blocks that are accessed more frequently than other data blocks by dynamic calculation rather than by a fixed threshold, ensuring that high-frequency identification is more reasonable and better suited to the actual situation of the system.

[0062] Among them, high-frequency data blocks are data blocks that are used significantly more frequently than the average level and are accessed frequently.

[0063] Specifically, this application can obtain the average usage frequency and discrete usage frequency data of all data blocks stored in at least one storage device; determine a decision threshold based on the discrete usage frequency data and a preset coefficient; determine the difference between the usage frequency information and the average usage frequency in a first data block, wherein the first data block is any data block among all data blocks stored in at least one storage device; and determine that the first data block satisfies a preset high-frequency condition if the difference is greater than the decision threshold.

[0064] Average usage frequency is the average of the usage frequencies of all data blocks. Discrete usage frequency data reflects the degree of deviation of the usage frequency of each data block from the average, such as variance, standard deviation, etc. In this application, standard deviation is used as an example.

[0065] The preset coefficient is a coefficient that is set in advance to adjust the sensitivity.

[0066] The decision threshold is a judgment limit dynamically calculated based on the product of the degree of dispersion and a preset coefficient.

[0067] The preset high-frequency condition is a rule for determining whether a data block is high-frequency, that is, the frequency of use is significantly higher than the average level.

[0068] Step 303: Load the high-frequency data block into the cache.

[0069] In some embodiments, after determining that a high-frequency data block has been obtained, this application can place the frequently accessed high-frequency data block into a high-speed cache, which can significantly reduce the number of reads from a low-speed disk, reduce IO latency, improve data recovery and reading speed, and solve the problems of scarce cache resources and slow recovery.

[0070] A cache is a caching medium with read and write speeds far exceeding those of ordinary storage devices, used to accelerate access to frequently accessed data.

[0071] In summary, this application dynamically identifies high-frequency data blocks by statistically analyzing the usage frequency of each data block and loads them into the cache. This effectively improves the reading speed of hot data, reduces disk I / O pressure, alleviates the problem of data cache resource shortage, and further improves the system's data recovery efficiency and overall operating performance.

[0072] Furthermore, for ease of understanding, such as Figure 5As shown, this application provides a specific data storage schematic diagram.

[0073] Reference Figure 5 First, the input data stream is segmented into multiple data blocks. Then, a hash fingerprint value is calculated for each data block to generate a unique fingerprint identifier. Next, the generated fingerprint value is matched and compared with existing fingerprints in a preset index table to identify duplicate data blocks. After that, a deduplication operation is performed on the duplicate data blocks, retaining only one physical copy. Finally, based on the access frequency of the data blocks and the storage balancing strategy, the deduplicated data blocks are written to disk or loaded into the cache, achieving efficient storage and access optimization.

[0074] The specific process of data storage can be referred to Figures 2 to 4 The embodiments shown will not be described in detail here.

[0075] It should be noted that, in order to support the implementation of the above data storage process, this application makes targeted modifications to the core data structure, as follows: Two key pieces of information have been added to the hash fingerprint metadata structure: First, a field recording the number of times a data block is referenced, used to quantify the frequency with which the data block is referenced by other data as a duplicate; a higher value in this field indicates greater reusability and access value of the data block. Second, a field recording the disk identifier of the data block, used to accurately locate its physical storage location. Hash fingerprint metadata is the core basis for the storage system to identify and manage each data block. The addition of these two key fields enables accurate recording of both the reference frequency and physical location of data blocks. This provides quantitative indicators for identifying high-value data blocks in subsequent cache scheduling and provides real-time location references for concentrating duplicate data on disk during data write-to-disk processing.

[0076] The disk space management data structure has been supplemented with records related to disk space utilization, including the total number of non-duplicate data blocks actually written and the disk space utilization rate (recorded as a percentage, ranging from 0 to 100). This is used to quantify the load status of each storage device. The disk space management data structure is the foundation for the storage system to manage backend physical storage resources. The addition of the aforementioned records related to disk space utilization comprehensively and quantitatively records the load and occupancy status of each disk from two dimensions: the number of data blocks and space utilization. This solves the problems of vague disk status descriptions and the lack of unified quantitative indicators, providing core data basis for subsequent disk sorting and load balancing allocation when writing data to disk, ensuring the scientific and reasonable selection of disks for data writing.

[0077] The optimization of the two data structures mentioned above provides a reliable quantitative basis for the dynamic selection of data disk storage devices and the caching scheduling of high-frequency data, thereby ensuring the effective execution of storage balancing strategies and cache optimization strategies.

[0078] Optionally, this application provides a detailed description with reference to a specific embodiment to facilitate understanding. To ensure that data is evenly distributed across disks and to improve data recovery speed, the data storage process may employ the following steps: Calculate and record the number of duplicate data blocks on each disk in the data to be stored. For example, ... Figure 6 The diagram shows the disks in the data storage. There are a total of 8 backend storage disks (disk0~disk7). The duplicate data blocks in the new data to be stored are chunk3, chunk6, and chunk9. Chunk3 is stored on disk1, and chunk6 and chunk9 are stored on disk4. At this time, the number of duplicate data blocks in each disk can be recorded by the array repeat_chunk_disk, that is, repeat_chunk_disk[1]=1, repeat_chunk_disk[4]=2, and the rest are 0.

[0079] Disk priority sorting: Disks are sorted in ascending order based on the number of data blocks written to each disk or disk utilization. For example, an array `disk_chunk_num` records the number of data blocks written to each disk, and the sorting result could be 2, 5, 0, 3, 6, 7, 1, 4.

[0080] Data is written according to the rules: Based on the priority sorting results mentioned above, a round-robin method is used to write non-duplicate data blocks to each disk in turn. Each time a disk is selected for writing, the value of `repeat_chunk_disk[ID]` is checked first. If the value is greater than or equal to 1, it indicates that there are duplicate data blocks stored on that disk in the new data segment, so `repeat_chunk_disk[ID]` is decremented by 1, that disk is skipped, and the next disk in the sorting process is selected for further judgment; if the value is 0, the data block is written to that disk.

[0081] For example, when disk0 is selected, repeat_chunk_disk[0] is 0, so the data block is written to disk0; when disk4 is selected, repeat_chunk_disk[4] is not 0, so the data block is not written to disk4. After decrementing repeat_chunk_disk[4] by 1, disk2 is then checked. Since repeat_chunk_disk[2] is 0, the data block is written to disk2. During the writing of new data to each disk, the number of data blocks already written or the disk utilization rate are updated, but the sorting results during this data write-to-disk process are not recalculated.

[0082] Ultimately, as Figure 7The diagram shown illustrates the data stored on each disk, indicating that the data will be evenly distributed across all disks.

[0083] With the data storage scheme proposed in this application, the data storage balance across all storage member disks can be guaranteed, ensuring that the storage of new data to be stored remains relatively even across all disks. Storage systems without this algorithm may result in some disks storing multiple data blocks of the data segment, while others store no data blocks, leading to some disks being busy and others idle during data recovery. With this algorithm, when a user re-reads the data, all disks can participate in the data reading, achieving concurrent reading and significantly improving data reading efficiency.

[0084] In addition, to optimize cache utilization and reduce disk pressure, this application may also adopt the following caching strategies: In the hash fingerprint metadata of each data block, add a member variable to record the number of times the data block is referenced. Maintain a global floating-point variable `avg_ref` to record the global average number of data block references. The value of `avg_ref` is periodically counted and updated by the system backend. The statistical scope includes all deduplicated data blocks in the system whose reference count is greater than 0. The formula for calculating the value of `avg_ref` is: `avg_ref = Sum of reference counts of all deduplicated data blocks globally`. The total number of deduplicated data blocks. A global floating-point standard deviation variable, delta_avg, is maintained to record the standard deviation of the reference count of global data blocks (delta_avg and avg_ref are periodically counted and updated synchronously; the statistical range of delta_avg is consistent with avg_ref; its core function is to quantify the dispersion of the reference count of each deduplicated data block in the system, reflecting the difference in reference popularity between different data blocks). An empirical value k (positive floating-point number) is set as the threshold coefficient for cache filtering. A filtering formula is established by combining this coefficient with global statistical data. When the reference count of a data block meets the condition (refcount – avg_ref) > k * delta_avg, the data block is determined to be high-value deduplicated data and is extracted into the cache for subsequent I / O operations. The empirical value 'k' is not fixed and needs to be adjusted flexibly based on the actual cache resource size of the storage system. The core principle is that the value of k is inversely proportional to the cache space size. That is, the larger the system cache space, the more high-value data blocks it can accommodate, and the smaller the value of k, the lower the filtering threshold, allowing more deduplicated data with above-average citation frequency to be loaded into the cache. Conversely, if the system cache space is smaller, to ensure efficient utilization of cache resources, the larger the value of k, the higher the filtering threshold, loading only core high-value data blocks with significantly higher citation frequency than average into the cache. The initial empirical value of k can be set to 1.0-2.0 based on common industry scenarios, and can be dynamically optimized based on actual system operating data to ensure maximum utilization efficiency of cache resources.

[0085] Thus, during the data recovery process, when the data read by the customer contains deduplicated data, the frequently referenced high-frequency data blocks have been pre-extracted and stored in the cache, eliminating the need to request them from the backend disk and significantly reducing the access pressure on the backend disk.

[0086] In summary, by implementing the aforementioned caching algorithm, the storage system can accurately identify and prioritize the caching of high-value deduplicated data. This ensures that during the data recovery phase, when the user reads original data containing deduplicated data, the high-value deduplicated data with numerous references and high access frequency has already been extracted and stored in the cache. Front-end I / O operations can directly retrieve this data from the cache quickly, without needing to send access requests to the back-end physical disk, significantly shortening the data read response time. Simultaneously, since the cache handles the frequently accessed core data, the back-end physical disk only needs to process a small number of access requests for ordinary value data, significantly reducing its I / O load. This improves the overall speed of data recovery, reduces disk mechanical wear, and extends the lifespan of the storage device. Furthermore, the algorithm uses quantitative indicators for screening, avoiding the subjectivity of manual intervention. It can dynamically and adaptively adjust according to the actual operating status of the system, maximizing both cache hit rate and resource utilization efficiency under limited cache resource constraints, further improving the overall stability and performance of the storage system.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0088] Embodiments of this application also provide a data storage device 800. Figure 8 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this application, such as... Figure 8 As shown, it includes: The determining unit 810 is used to determine the number of duplicate data blocks in at least one storage device based on the storage device information of at least one duplicate data block after the duplicate data analysis of the data to be stored is completed. The at least one duplicate data block is at least one data block in the data to be stored that has been stored in the storage device after the duplicate data analysis. Analysis unit 820 is configured to perform priority analysis on at least one storage device based on at least one load data of at least one storage device, so as to determine candidate storage devices for a target non-duplicate data block from at least one storage device according to the priority analysis result. The target non-duplicate data block is any data block among at least one non-duplicate data blocks in the data to be stored that has not been stored in the storage device after the duplicate data analysis. Storage unit 830 is used to perform storage balancing verification on candidate storage devices based on the number of duplicate data blocks in the candidate storage devices, so as to determine the target storage device for the target non-duplicate data block according to the verification result, and store the target non-duplicate data block to the target storage device.

[0089] In one possible implementation of this application embodiment, the determining unit 810 is configured to: determine at least one metadata of at least one stored data block corresponding to at least one duplicate data block based on a preset identifier index table; extract at least one storage device identifier of at least one stored data block from the at least one metadata, and determine the at least one storage device identifier as storage device information of at least one duplicate data block; and determine the number of duplicate data blocks in at least one storage device based on the storage device information.

[0090] In one possible implementation of this application embodiment, the analysis unit 820 is configured to: sort at least one storage device based on the load data of at least one storage device to obtain priority analysis results of at least one storage device; and determine the storage device with the highest priority in the priority analysis results as a candidate storage device for the target non-duplicate data block.

[0091] In one possible implementation of this application embodiment, the storage unit 830 is configured to: perform storage balance verification on candidate storage devices based on the number of duplicate data blocks in the candidate storage devices, and determine the verification result of the candidate storage devices; if the verification result indicates that the candidate storage devices meet the storage balance conditions, determine the candidate storage devices as target storage devices; if the verification result indicates that the candidate storage devices do not meet the storage balance conditions, determine an updated candidate storage device among at least one storage device according to the priority analysis result, and perform storage balance verification on the updated candidate storage devices based on the number of duplicate data blocks in the updated candidate storage devices, and determine the target storage device for the target non-duplicate data blocks.

[0092] In one possible implementation of this application embodiment, the storage unit 830 is configured to: determine that the verification result is that the candidate storage device meets the storage balance condition when the number of duplicate data blocks is less than a preset threshold; and determine that the verification result is that the candidate storage device does not meet the storage balance condition when the number of duplicate data blocks is greater than or equal to the preset threshold.

[0093] In one possible implementation of this application embodiment, the storage unit 830 is used to: adjust the number of duplicate data blocks in the candidate storage device according to a preset step size when the candidate storage device does not meet the storage balance condition, so as to determine the update target storage device for updating the non-duplicate data blocks based on the adjusted number of duplicate data blocks in the candidate storage device when storing the updated non-duplicate data blocks in the data to be stored.

[0094] In one possible implementation of this application embodiment, the data storage device further includes: a cache unit, configured to: after storing the target non-repeating data block to the target storage device, determine at least one usage frequency information of at least one data block stored in at least one storage device; based on the at least one usage frequency information, determine a high-frequency data block that satisfies a preset high-frequency condition; and load the high-frequency data block into the cache.

[0095] In one possible implementation of this application embodiment, the cache unit is configured to: obtain the average usage frequency and discrete usage frequency data of all data blocks stored in at least one storage device; determine a decision threshold based on the discrete usage frequency data and a preset coefficient; determine the difference between the usage frequency information and the average usage frequency in a first data block, wherein the first data block is any data block among all data blocks stored in at least one storage device; and determine that the first data block satisfies a preset high-frequency condition if the difference is greater than the decision threshold.

[0096] In one possible implementation of this application, the storage unit 830 is configured to: after storing the target non-repeating data block to the target storage device, determine the target non-repeating data block identifier of the target non-repeating data block and the target storage device identifier of the target storage device as the target non-repeating data block metadata of the target non-repeating data block, and update the target non-repeating data block metadata to a preset identifier index table.

[0097] For a description of the features in the embodiment corresponding to the data storage device, please refer to the relevant description in the embodiment corresponding to the data storage method, which will not be repeated here.

[0098] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data storage method embodiments.

[0099] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data storage method embodiments when it is run.

[0100] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0101] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0102] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0103] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The data storage method, electronic device, storage medium, and product provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data storage method, characterized by, include: When the deduplication analysis of the data to be stored is completed, the number of deduplication blocks in at least one storage device is determined based on the storage device information of at least one deduplication block. The at least one deduplication block is at least one data block in the data to be stored that has been stored in the storage device after the deduplication analysis. Based on at least one load data of the at least one storage device, priority analysis is performed on the at least one storage device to determine candidate storage devices for a target non-duplicate data block from the at least one storage device according to the priority analysis results. The target non-duplicate data block is any data block among at least one non-duplicate data blocks in the data to be stored that has not been stored in the storage device after the duplicate data analysis. Based on the number of duplicate data blocks in the candidate storage devices, a storage balancing verification is performed on the candidate storage devices to determine the target storage device for the target non-duplicate data block according to the verification result, and the target non-duplicate data block is stored in the target storage device.

2. The method of claim 1, wherein, Determining the number of duplicate data blocks in at least one storage device based on storage device information of at least one duplicate data block includes: Based on a preset identifier index table, at least one metadata of at least one stored data block corresponding to the at least one duplicate data block is determined; Extract at least one storage device identifier of the at least one stored data block from the at least one metadata, and determine the at least one storage device identifier as the storage device information of the at least one duplicate data block; Based on the storage device information, the number of duplicate data blocks in the at least one storage device is determined.

3. The method of claim 2, wherein, The step of performing priority analysis on the at least one storage device based on at least one load data of the at least one storage device, and determining candidate storage devices for the target non-duplicate data block from the at least one storage device according to the priority analysis results, includes: Based on the load data of the at least one storage device, the at least one storage device is sorted to obtain the priority analysis results of the at least one storage device; The storage device with the highest priority in the priority analysis results is determined as the candidate storage device for the target non-duplicate data block.

4. The method of claim 1, wherein, The step of performing storage balancing verification on the candidate storage devices based on the number of duplicate data blocks in the candidate storage devices, and determining the target storage device for the target non-duplicate data block based on the verification result, includes: Based on the number of duplicate data blocks in the candidate storage devices, a storage balancing verification is performed on the candidate storage devices to determine the verification result of the candidate storage devices. If the verification result indicates that the candidate storage device meets the storage balancing conditions, the candidate storage device is determined as the target storage device. If the verification result indicates that the candidate storage device does not meet the storage balancing condition, an update candidate storage device is determined among the at least one storage device based on the priority analysis result, and a storage balancing verification is performed on the update candidate storage device based on the number of duplicate data blocks in the update candidate storage device to determine the target storage device for the target non-duplicate data block.

5. The method according to claim 4, characterized in that, The step of performing storage balancing verification on the candidate storage devices based on the number of duplicate data blocks in the candidate storage devices, and determining the verification result of the candidate storage devices, includes: If the number of duplicate data blocks is less than a preset threshold, the verification result is determined to indicate that the candidate storage device meets the storage balancing condition. If the number of duplicate data blocks is greater than or equal to the preset threshold, the verification result is determined to be that the candidate storage device does not meet the storage balancing condition.

6. The method according to claim 4, characterized in that, After determining the verification result of the candidate storage devices by performing storage balancing verification based on the number of duplicate data blocks in the candidate storage devices, the method includes: If the verification result indicates that the candidate storage device does not meet the storage balancing condition, the number of duplicate data blocks in the candidate storage device is adjusted according to a preset step size. In order to determine the update target storage device for the update non-duplicate data blocks based on the adjusted number of duplicate data blocks in the candidate storage device while storing the updated non-duplicate data blocks in the data to be stored.

7. The method according to claim 1, characterized in that, After storing the target non-duplicate data block to the target storage device, the method includes: Determine at least one usage frequency information for at least one data block stored in the at least one storage device; Based on the at least one frequency information used, determine high-frequency data blocks that meet preset high-frequency conditions; The high-frequency data block is loaded into the cache.

8. The method according to claim 7, characterized in that, The step of determining the high-frequency data block that meets the preset high-frequency conditions based on the at least one used frequency information includes: Obtain the average usage frequency and discrete usage frequency data of all data blocks stored in the at least one storage device; Based on the frequency discrete data and preset coefficients, a decision threshold is determined; Determine the difference between the usage frequency information in the first data block and the average usage frequency, wherein the first data block is any data block among all data blocks stored in the at least one storage device; If the difference is greater than the decision threshold, the first data block is determined to satisfy the preset high-frequency condition.

9. The method according to claim 3, characterized in that, After storing the target non-duplicate data block to the target storage device, the method further includes: The target non-repeating data block identifier of the target non-repeating data block and the target storage device identifier of the target storage device are determined as the target non-repeating data block metadata of the target non-repeating data block, and the target non-repeating data block metadata is updated to the preset identifier index table.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data storage method according to any one of claims 1-9.

Images