Intelligent scheduling method, device and equipment for cold data and medium

Abstract

The invention relates to the field of big data, the scheme can be applied to the fields of medical treatment and finance, and the invention provides an intelligent scheduling method, device and equipment for cold data and a medium, and the method comprises the steps: collecting the multi-dimensional features of a file in a distributed file system; inputting the multi-dimensional features into a pre-constructed cold data prediction model, and outputting a cold data prediction probability of the corresponding file; judging the input data of each file according to a cold and hot data dynamic judgment algorithm, and outputting a cold data file list; determining a migration opportunity and a target storage hierarchy of each file in the cold data file list according to a migration scheduling strategy; and recording the actual access behavior after the file is archived, and establishing a closed-loop feedback mechanism. According to the invention, the intelligence and dynamics of cold data management are enhanced, and the utilization efficiency of storage resources is improved.

Description

Technical Field

[0001] This invention relates to the field of big data, and the solution can be applied to the medical and financial fields, especially to an intelligent scheduling method, device, equipment and medium for cold data. Background Technology

[0002] In the era of big data, enterprises have accumulated massive amounts of structured and unstructured data, of which approximately 70% to 80% is cold data—historical data that has not been accessed for a long time and has an extremely low access frequency (such as logs, audit records, backup files, and historical medical records and medical image storage in the medical field, and historical transaction records and risk control audit data in the financial field). Traditional storage architectures store all data uniformly in high-performance, high-cost local disk clusters, resulting in: 1. High storage costs: Cold data consumes a large amount of resources, causing resource waste; 2. Performance bottlenecks: Hot and cold data are stored together, affecting the performance of high-frequency access (such as the response efficiency of real-time medical diagnosis and treatment data and high-frequency financial transaction data); 3. Inefficient management: Lack of automation mechanisms, relying on manual judgment of hot and cold data; 4. Large recovery latency: Cold data needs to be loaded from low-speed media, making it difficult to meet the needs of emergency access to historical medical records in the medical field and rapid data retrieval in financial audits.

[0003] Existing solutions such as HDFS Tiered Storage, HAR archiving, and DistCp batch migration have the following drawbacks: 1. Hot / cold storage determination relies on static rules, which cannot adapt to complex access patterns; 2. Lack of predictive ability: It cannot predict future access behavior, leading to incorrect or delayed archiving; 3. Rigid scheduling strategy: Migration tasks have no priority or resource awareness, which can easily cause cluster load fluctuations; 4. Lack of system closed loop: There is no feedback optimization mechanism, which cannot continuously improve the accuracy of decision-making.

[0004] Therefore, existing cold data storage and management solutions suffer from insufficient intelligence and rigid scheduling strategies. Summary of the Invention

[0005] This invention provides an intelligent scheduling method, apparatus, device, and medium for cold data, aiming to solve the problems of insufficient intelligence and rigid scheduling strategies in existing cold data storage and management solutions.

[0006] In a first aspect, embodiments of the present invention provide an intelligent scheduling method for cold data, the method comprising: Collect multidimensional features of files in a distributed file system; The multidimensional features are input into a pre-built cold data prediction model, and the cold data prediction probability corresponding to the file is output. The input data of each file is determined according to the preset hot and cold data dynamic determination algorithm, and a list of cold data files is output; wherein, the input data includes the multidimensional features, the cold data prediction probability and the current system load; The migration timing and target storage level of each file in the cold data file list are determined according to the preset migration scheduling strategy, so that the corresponding file is migrated to the corresponding target storage level at the migration timing; Record the actual access behavior after the file is archived, and establish a closed-loop feedback mechanism to dynamically optimize the cold data prediction model and the migration scheduling strategy.

[0007] Secondly, embodiments of the present invention also provide an intelligent scheduling device for cold data, the device comprising: The acquisition unit is used to collect multidimensional features of files in the distributed file system. The input / output unit is used to input the multidimensional features into a pre-built cold data prediction model and output the cold data prediction probability corresponding to the file. The determination unit is used to determine the input data of each file according to a preset dynamic determination algorithm for hot and cold data, and output a list of cold data files; wherein, the input data includes the multidimensional features, the cold data prediction probability, and the current system load; The migration unit is used to determine the migration timing and target storage level of each file in the cold data file list according to a preset migration scheduling strategy, so as to migrate the corresponding file to the corresponding target storage level at the migration timing. A unit is established to record the actual access behavior after the file is archived, and a closed-loop feedback mechanism is established to dynamically optimize the cold data prediction model and the migration scheduling strategy.

[0008] Thirdly, embodiments of the present invention also provide an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect above.

[0009] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the method described in the first aspect.

[0010] This invention provides an intelligent scheduling method, apparatus, device, and medium for cold data. The method includes: collecting multidimensional features of files in a distributed file system; inputting the multidimensional features into a pre-constructed cold data prediction model and outputting the cold data prediction probability corresponding to the file; judging the input data of each file according to a preset cold and hot data dynamic judgment algorithm and outputting a cold data file list; wherein the input data includes the multidimensional features, the cold data prediction probability, and the current system load; determining the migration timing and target storage level of each file in the cold data file list according to a preset migration scheduling strategy, so as to migrate the corresponding file to the corresponding target storage level at the migration timing; recording the actual access behavior after the file is archived, and establishing a closed-loop feedback mechanism to dynamically optimize the cold data prediction model and the migration scheduling strategy. This invention improves the accuracy of cold data identification by collecting multi-dimensional features of files and combining them with a cold data prediction model, breaking through the limitations of traditional static rules and reducing the risk of misjudgment. The dynamic cold and hot data identification algorithm integrates multi-dimensional features, cold data prediction probability, and current system load, making cold data identification more in line with actual scenarios and outputting a more accurate list of cold data files. The migration scheduling strategy specifically determines the migration timing and target level, achieving efficient archiving of cold data. The closed-loop feedback mechanism records actual access behavior and dynamically optimizes the model and strategy, enhancing the intelligence and dynamism of cold data management and improving storage resource utilization efficiency. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the intelligent scheduling method for cold data provided in an embodiment of the present invention; Figure 2 A schematic block diagram of an intelligent scheduling device for cold data provided in an embodiment of the present invention; Figure 3 A schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0015] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0016] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. Embodiments of this invention provide an intelligent scheduling method, apparatus, device, and medium for cold data. The intelligent scheduling method for cold data can be applied to terminal devices or servers. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, and wearable device; the server can be a single server or a server cluster composed of multiple servers. The invention will now be described in detail through specific embodiments.

[0017] Figure 1 This is a flowchart illustrating the intelligent scheduling method for cold data provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S150.

[0018] S110. Collect multidimensional features of files in the distributed file system.

[0019] In this embodiment, multidimensional features of files in HDFS (Hadoop Distributed File System) are collected. These multidimensional features include Last Access Time, Access Frequency, File Size, Creation Time, File Type (logs, reports, raw data, etc.), and upper-layer application access patterns (such as Spark Job execution records).

[0020] S120. Input the multidimensional features into the pre-built cold data prediction model and output the cold data prediction probability corresponding to the file.

[0021] In this embodiment, the multidimensional features are input into a pre-built cold data prediction model, which outputs the cold data prediction probability of the corresponding file. The cold data prediction model is constructed using the XGBoost or LightGBM algorithm, and its training data includes historical archive records and actual file access behavior data. It supports incremental learning and online updates, and can adaptively adjust model parameters over time to adapt to changes in data access patterns. During the online inference stage, the cold data prediction model predicts the access probability of each file in real time for the next 90 days. If the access probability is lower than a preset access threshold (e.g., 0.1), the file is marked as high-probability cold data, providing a core basis for subsequent hot and cold data determination.

[0022] S130. The input data of each file is determined according to the preset hot and cold data dynamic determination algorithm, and a list of cold data files is output; wherein, the input data includes the multidimensional features, the cold data prediction probability and the current system load.

[0023] In this embodiment, the dynamic hot and cold data determination algorithm comprehensively determines each file from multiple dimensions such as the multi-dimensional features, the cold data prediction probability, and the current system load, avoiding the one-sidedness of single-dimensional determination, and realizing the accurate identification of cold data files and outputting a list of cold data files.

[0024] Furthermore, the multidimensional features are input into a pre-built cold data prediction model, which outputs the cold data prediction probability and confidence score corresponding to the file; files with high confidence scores are preferred to reduce the false positive rate.

[0025] In one embodiment, step S130 includes: extracting the access frequency corresponding to the file from the input data; combining the current system load and the judgment conditions in the hot and cold data dynamic judgment algorithm, comprehensively judging the access frequency and cold data prediction probability of each file respectively; and summarizing all the files that meet the judgment conditions to form the cold data file list.

[0026] In this embodiment, the access frequency of the corresponding file is extracted from the input data; combined with the current system load, and according to the judgment conditions in the dynamic hot and cold data judgment algorithm, the access frequency and cold data prediction probability of each file are comprehensively judged; all files that meet the judgment conditions are summarized to form the cold data file list; the judgment conditions support dynamic adjustment, specifically, they can be automatically adjusted according to cluster load, storage cost and SLA (Service Level Agreement) requirements.

[0027] In one embodiment, the step of combining the current system load and the judgment conditions in the dynamic cold and hot data judgment algorithm to comprehensively judge the access frequency and cold data prediction probability of each file includes: if the current system load is less than a third preset threshold, then the file whose cold data prediction probability is greater than a first preset threshold and whose access frequency is less than a second preset threshold satisfies the judgment conditions; and the file that satisfies the judgment conditions is marked as cold data.

[0028] In this embodiment, if the current system load is less than the third preset threshold T3 in the determination condition, then the file whose cold data prediction probability is greater than the first preset threshold T1 in the determination condition and whose access frequency is less than the second preset threshold T2 in the determination condition satisfies the determination condition; the file that satisfies the determination condition is marked as cold data.

[0029] Preferably, the first preset threshold T1, the second preset threshold T2, and the third preset threshold T3 support dynamic adjustment. Taking medical and financial scenarios as examples: In the medical scenario: Because the SLA requires that historical medical records and medical images be accessible in a timely manner (e.g., recovery within ≤3 seconds), and core diagnostic and treatment data cannot be mistakenly archived, T1 is set to a relatively high value, and only files with a high probability of cold data prediction are archived; T2 is set to a very low value (e.g., accessed ≤1 time in the last 30 days), and only non-core files that have not been accessed for a long time are selected; T3 is set to a relatively low value (e.g., CPU utilization ≤25%), to avoid migration occupying diagnostic and treatment system resources and to ensure the stability of access to hot data (e.g., real-time diagnostic and treatment data).

[0030] In the financial scenario: To meet the requirements of compliant storage of transaction records (e.g., retention for 5 years) and efficient audit backtracking (e.g., recovery within ≤10 seconds), while ensuring the performance of high-frequency transaction data, T1 is set to a medium-high value to balance storage costs and data availability; T2 is set to a medium value (e.g., access ≤3 times in the last 30 days) to adapt to the low-frequency access characteristics of auditing; T3 is set to a medium value (e.g., CPU utilization ≤35%) to improve the efficiency of cold data archiving without affecting the operation of the transaction system.

[0031] In one embodiment, after marking the files that meet the determination conditions as cold data, the method further includes: marking the files whose cold data prediction probability is less than a fourth preset threshold or whose access frequency meets the preset access conditions as hot data among the files that are not marked as cold data; and adding the files that are not marked as cold data or hot data to the monitoring queue.

[0032] In this embodiment, files not marked as cold data are marked as hot data if the predicted probability of cold data is less than the fourth preset threshold T4 or the access frequency meets the preset access conditions (e.g., accessed ≥ 1 time in the last 7 days). Files not marked as cold data or hot data are added to a monitoring queue. The access behavior and feature changes of the files in the monitoring queue are continuously monitored to provide real-time data support for subsequent secondary judgment, effectively avoid judgment omissions and misjudgments, and ensure the accuracy of cold and hot data identification.

[0033] S140. Determine the migration timing and target storage level of each file in the cold data file list according to the preset migration scheduling strategy, so as to migrate the corresponding file to the corresponding target storage level at the migration timing.

[0034] In this embodiment, the migration scheduling strategy comprehensively determines the migration timing and target storage level for each file in the cold data file list based on the access priority of cold data files, SLA requirements (such as access recovery timeliness and compliance storage period), file size, and cost-performance characteristics of each storage tier. This ensures that the migration process balances storage cost optimization and business availability. Specifically: 1. Migration Timing Determination: Prioritize migration during off-peak hours (e.g., 0-4 AM) or when system load is below the preset low-load threshold to avoid consuming core resources and affecting the performance of hot data access; at the same time, support dynamic adjustment of migration order based on migration queue length and file priority, such as prioritizing the migration of large files.

[0035] 2. Target storage tier matching: Allocate storage tiers based on file SLA requirements. For example, for cold data that needs to be recovered quickly (such as medical records that need to be accessed urgently), migrate to hot storage tiers (such as SSDs) or warm storage tiers (such as HDDs); for cold data that is archived for a long time and accessed infrequently (such as financial transaction records older than 5 years), migrate to cold storage tiers (such as tape libraries or object storage) to achieve a balance between cost and performance.

[0036] In one embodiment, step S140 includes: sorting each file in the cold data file list according to its multidimensional features and cold data prediction probability to obtain a corresponding migration priority queue; determining the migration timing for each file according to the migration priority queue, the current system load, and the load status of each resource coordinator, so as to migrate the corresponding file to the corresponding target storage level at the migration timing.

[0037] In this embodiment, the sorting of the migration priority queue is based on the cold data prediction probability (e.g., the higher the cold data prediction probability, the higher the migration priority), combined with features such as file size and recent access time in multi-dimensional features for auxiliary optimization; the determination of the migration timing is based on the migration priority queue, combined with the current system load and the load status of each resource coordinator: migration is started only when both types of load are below the preset load threshold, in the order of the queue; if the load exceeds the threshold, the migration of the corresponding file is postponed to the system's off-peak period, or scheduled to be executed by a low-load coordinator, which ensures efficient migration without affecting system stability.

[0038] Preferably, if the migration of cold data files fails and the number of restarts exceeds three, an alarm message is issued to prompt manual intervention.

[0039] In one embodiment, after migrating the corresponding file to the corresponding target storage level at the migration time, the method further includes: if multiple cold data files trigger recovery requests simultaneously, prioritizing them according to their data value to obtain a corresponding recovery priority queue; and migrating the cold data files that trigger recovery requests to a hot storage level or a warm storage level based on the recovery priority queue.

[0040] In this embodiment, if multiple cold data files trigger recovery requests simultaneously, they are prioritized according to their data value (i.e., high-value data is recovered first) to obtain a corresponding recovery priority queue. Based on the recovery priority queue, the cold data files that trigger recovery requests are migrated to the hot storage level or the warm storage level to ensure the access stability after data recovery and the reasonable allocation of system resources.

[0041] Preferably, the system load is monitored in real time during the migration process. If the load of the hot storage layer exceeds the preset load limit, some high-value files are dynamically moved to the warm storage layer. The migration is then completed after the load of the hot storage layer drops, ensuring that the recovery process does not affect the stability of access to existing hot data, and balancing recovery efficiency and system operation security.

[0042] S150. Record the actual access behavior after the file is archived, and establish a closed-loop feedback mechanism to dynamically optimize the cold data prediction model and the migration scheduling strategy.

[0043] In this embodiment, the actual access behavior after the file is archived is recorded, a closed-loop feedback mechanism is established, and the cold data prediction model and the migration scheduling strategy are dynamically optimized through reinforcement learning (such as DQN (Deep Q-Network)) to enhance the intelligence and dynamism of cold data management. Specifically, for prediction deviations (such as predicting files with low access but actually accessing them frequently), the feature weights of the cold data prediction model are adjusted or scenario-based training samples are supplemented to improve the accuracy of cold data identification. The migration timing determination logic and storage level matching logic are optimized in combination with actual access behavior to ensure that the migration scheduling strategy continuously matches the actual business needs.

[0044] In one embodiment, recording the actual access behavior after the file is archived and establishing a closed-loop feedback mechanism to dynamically optimize the cold data prediction model includes: determining whether the archived file is accessed within a preset number of days based on the actual access behavior; if the archived file is accessed within the preset number of days, triggering a false archive event; and feeding the file corresponding to the false archive event back to the training set as a false archive sample, wherein the training set is used to retrain the cold data prediction model.

[0045] In this embodiment, if the archived file is accessed within a preset number of days, an erroneous archiving event is triggered; the file corresponding to the erroneous archiving event is fed back to the training set as an erroneous archiving sample, and the training set is used to retrain the cold data prediction model to improve the accuracy of cold data identification.

[0046] In summary, this invention improves the accuracy of cold data identification by collecting multi-dimensional features of files and combining them with a cold data prediction model, breaking through the limitations of traditional static rules and reducing the risk of misjudgment. The dynamic cold and hot data identification algorithm integrates multi-dimensional features, cold data prediction probability, and current system load, making cold data identification more relevant to real-world scenarios and producing a more accurate list of cold data files. The migration scheduling strategy specifically determines the migration timing and target level, achieving efficient archiving of cold data. The closed-loop feedback mechanism records actual access behavior and dynamically optimizes the model and strategy, enhancing the intelligence and dynamism of cold data management and improving storage resource utilization efficiency.

[0047] Figure 2 This is a schematic block diagram of an intelligent scheduling device for cold data provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above-mentioned intelligent scheduling method for cold data, this invention also provides an intelligent scheduling device for cold data. The device is configured in a terminal device or a server. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device; the server can be a single server or a server cluster composed of multiple servers. For details, please refer to... Figure 2The intelligent scheduling device 700 for cold data includes: The acquisition unit 701 is used to acquire multidimensional features of files in the distributed file system. The input / output unit 702 is used to input the multidimensional features into a pre-built cold data prediction model and output the cold data prediction probability corresponding to the file. The determination unit 703 is used to determine the input data of each file according to a preset dynamic determination algorithm for hot and cold data, and output a list of cold data files; wherein, the input data includes the multidimensional features, the cold data prediction probability, and the current system load; Migration unit 704 is used to determine the migration timing and target storage level of each file in the cold data file list according to a preset migration scheduling strategy, so as to migrate the corresponding file to the corresponding target storage level at the migration timing; Unit 705 is established to record the actual access behavior after the file is archived, and to establish a closed-loop feedback mechanism to dynamically optimize the cold data prediction model and the migration scheduling strategy.

[0048] In some embodiments, when the determination unit 703 performs the step of determining the input data of each file according to a preset dynamic cold and hot data determination algorithm and outputting a list of cold data files, it is specifically used for: Extract the access frequency of the corresponding file from the input data; combine the current system load with the judgment conditions in the hot and cold data dynamic judgment algorithm to comprehensively judge the access frequency and cold data prediction probability of each file; summarize all the files that meet the judgment conditions to form the cold data file list.

[0049] In some embodiments, when the determination unit 703 performs a comprehensive determination step based on the current system load and the determination conditions in the dynamic hot and cold data determination algorithm, it specifically performs the following steps: If the current system load is less than the third preset threshold, then files whose cold data prediction probability is greater than the first preset threshold and whose access frequency is less than the second preset threshold are determined to meet the determination conditions; files that meet the determination conditions are marked as cold data.

[0050] In some embodiments, after performing the step of marking files that meet the determination conditions as cold data, the determination unit 703 is further configured to: Files not marked as cold data are marked as hot data if the predicted probability of cold data is less than a fourth preset threshold or the access frequency meets the preset access conditions; files that are not marked as cold data or hot data are added to the monitoring queue.

[0051] In some embodiments, when the migration unit 704 performs the step of determining the migration timing and target storage level of each file in the cold data file list according to a preset migration scheduling strategy, it is specifically used for: The files in the cold data file list are sorted according to their multidimensional features and cold data prediction probabilities to obtain a corresponding migration priority queue. The migration timing for each file is determined based on the migration priority queue, the current system load, and the load status of each resource coordinator, so that the corresponding file is migrated to the corresponding target storage level at the migration timing.

[0052] In some embodiments, after performing the step of migrating the corresponding file to the corresponding target storage level at the migration time, the migration unit 704 is further configured to: If multiple cold data files trigger recovery requests simultaneously, they are prioritized according to their data value to obtain a corresponding recovery priority queue. Based on the recovery priority queue, the cold data files that trigger recovery requests are migrated to the hot storage level or the warm storage level.

[0053] In some embodiments, when the establishment unit 705 performs the step of recording the actual access behavior after the file is archived and establishing a closed-loop feedback mechanism to dynamically optimize the cold data prediction model, it is specifically used for: Based on the actual access behavior, determine whether the archived file has been accessed within a preset number of days; if the archived file has been accessed within the preset number of days, trigger an erroneous archiving event; the file corresponding to the erroneous archiving event is returned to the training set as an erroneous archiving sample, and the training set is used to retrain the cold data prediction model.

[0054] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned intelligent scheduling device for cold data and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0055] The aforementioned intelligent scheduling device for cold data can be implemented as a computer program, which can, for example... Figure 3 It runs on the electronic device shown.

[0056] Please see Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 800 can be a terminal or a server, wherein the terminal can be an electronic device with communication functions. The server can be a standalone server or a server cluster composed of multiple servers.

[0057] See Figure 3 The electronic device 800 includes a processor 802, a memory, and a network interface 805 connected via a system bus 801. The memory may include a non-volatile storage medium 803 and internal memory 804.

[0058] The non-volatile storage medium 803 may store an operating system 8031 ​​and a computer program 8032. The computer program 8032 includes program instructions that, when executed, cause the processor 802 to perform an intelligent scheduling method for cold data.

[0059] The processor 802 provides computing and control capabilities to support the operation of the entire electronic device 800.

[0060] The internal memory 804 provides an environment for the execution of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can execute a smart scheduling method for cold data.

[0061] This network interface 805 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device 800 to which the present invention is applied. The specific electronic device 800 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0062] The processor 802 is used to run a computer program 8032 stored in the memory to perform the following steps: The system collects multidimensional features of files in a distributed file system; inputs these multidimensional features into a pre-built cold data prediction model, and outputs the cold data prediction probability for each file; it then determines the input data for each file according to a preset dynamic cold / hot data determination algorithm, and outputs a list of cold data files; wherein the input data includes the multidimensional features, the cold data prediction probability, and the current system load; it determines the migration timing and target storage level for each file in the cold data file list according to a preset migration scheduling strategy, so that the corresponding file is migrated to the corresponding target storage level at the migration timing; it records the actual access behavior after the file is archived, and establishes a closed-loop feedback mechanism to dynamically optimize the cold data prediction model and the migration scheduling strategy.

[0063] In some embodiments, when the processor 802 determines the input data of each file according to a preset dynamic cold / hot data determination algorithm and outputs a list of cold data files, the specific steps are as follows: Extract the access frequency of the corresponding file from the input data; combine the current system load with the judgment conditions in the hot and cold data dynamic judgment algorithm to comprehensively judge the access frequency and cold data prediction probability of each file; summarize all the files that meet the judgment conditions to form the cold data file list.

[0064] In some embodiments, when the processor 802 performs a comprehensive determination step on the access frequency and cold data prediction probability of each file based on the determination conditions in the dynamic hot and cold data determination algorithm, combined with the current system load, the processor 802 specifically implements the following steps: If the current system load is less than the third preset threshold, then files whose cold data prediction probability is greater than the first preset threshold and whose access frequency is less than the second preset threshold are determined to meet the determination conditions; files that meet the determination conditions are marked as cold data.

[0065] In some embodiments, after marking files that meet the determination criteria as cold data, the processor 802 further implements the following steps: Files not marked as cold data are marked as hot data if the predicted probability of cold data is less than a fourth preset threshold or the access frequency meets the preset access conditions; files that are not marked as cold data or hot data are added to the monitoring queue.

[0066] In some embodiments, when the processor 802 determines the migration timing and target storage level of each file in the cold data file list according to a preset migration scheduling strategy, the specific steps are as follows: The files in the cold data file list are sorted according to their multidimensional features and cold data prediction probabilities to obtain a corresponding migration priority queue. The migration timing for each file is determined based on the migration priority queue, the current system load, and the load status of each resource coordinator, so that the corresponding file is migrated to the corresponding target storage level at the migration timing.

[0067] In some embodiments, after implementing the step of migrating the corresponding file to the corresponding target storage level at the migration time, the processor 802 further implements the following steps: If multiple cold data files trigger recovery requests simultaneously, they are prioritized according to their data value to obtain a corresponding recovery priority queue. Based on the recovery priority queue, the cold data files that trigger recovery requests are migrated to the hot storage level or the warm storage level.

[0068] In some embodiments, when the processor 802 implements the step of recording the actual access behavior after the file is archived and establishing a closed-loop feedback mechanism to dynamically optimize the cold data prediction model, the following steps are specifically implemented: Based on the actual access behavior, determine whether the archived file has been accessed within a preset number of days; if the archived file has been accessed within the preset number of days, trigger an erroneous archiving event; the file corresponding to the erroneous archiving event is returned to the training set as an erroneous archiving sample, and the training set is used to retrain the cold data prediction model.

[0069] It should be understood that, in this embodiment of the invention, the processor 802 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0070] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0071] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps: The system collects multidimensional features of files in a distributed file system; inputs these multidimensional features into a pre-built cold data prediction model, and outputs the cold data prediction probability for each file; it then determines the input data for each file according to a preset dynamic cold / hot data determination algorithm, and outputs a list of cold data files; wherein the input data includes the multidimensional features, the cold data prediction probability, and the current system load; it determines the migration timing and target storage level for each file in the cold data file list according to a preset migration scheduling strategy, so that the corresponding file is migrated to the corresponding target storage level at the migration timing; it records the actual access behavior after the file is archived, and establishes a closed-loop feedback mechanism to dynamically optimize the cold data prediction model and the migration scheduling strategy.

[0072] In one embodiment, when the processor executes the program instructions to determine the input data of each file according to a preset dynamic cold / hot data determination algorithm and outputs a list of cold data files, the specific steps are as follows: Extract the access frequency of the corresponding file from the input data; combine the current system load with the judgment conditions in the hot and cold data dynamic judgment algorithm to comprehensively judge the access frequency and cold data prediction probability of each file; summarize all the files that meet the judgment conditions to form the cold data file list.

[0073] In one embodiment, when the processor executes the program instructions to perform a comprehensive determination step on the access frequency and cold data prediction probability of each file based on the current system load and the determination conditions in the dynamic cold data determination algorithm, the specific steps are as follows: If the current system load is less than the third preset threshold, then files whose cold data prediction probability is greater than the first preset threshold and whose access frequency is less than the second preset threshold are determined to meet the determination conditions; files that meet the determination conditions are marked as cold data.

[0074] In one embodiment, after executing the program instructions to mark files that meet the determination conditions as cold data, the processor further performs the following steps: Files not marked as cold data are marked as hot data if the predicted probability of cold data is less than a fourth preset threshold or the access frequency meets the preset access conditions; files that are not marked as cold data or hot data are added to the monitoring queue.

[0075] In one embodiment, when the processor executes the program instructions to determine the migration timing and target storage level of each file in the cold data file list according to a preset migration scheduling strategy, the specific steps are as follows: The files in the cold data file list are sorted according to their multidimensional features and cold data prediction probabilities to obtain a corresponding migration priority queue. The migration timing for each file is determined based on the migration priority queue, the current system load, and the load status of each resource coordinator, so that the corresponding file is migrated to the corresponding target storage level at the migration timing.

[0076] In one embodiment, after executing the program instructions to migrate the corresponding file to the corresponding target storage level at the migration time, the processor further implements the following steps: If multiple cold data files trigger recovery requests simultaneously, they are prioritized according to their data value to obtain a corresponding recovery priority queue. Based on the recovery priority queue, the cold data files that trigger recovery requests are migrated to the hot storage level or the warm storage level.

[0077] In one embodiment, when the processor executes the program instructions to record the actual access behavior after the file is archived and establishes a closed-loop feedback mechanism to dynamically optimize the cold data prediction model, the following steps are specifically implemented: Based on the actual access behavior, determine whether the archived file has been accessed within a preset number of days; if the archived file has been accessed within the preset number of days, trigger an erroneous archiving event; the file corresponding to the erroneous archiving event is returned to the training set as an erroneous archiving sample, and the training set is used to retrain the cold data prediction model.

[0078] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0080] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0081] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent scheduling of cold data, characterized in that, The method includes: Collect multidimensional features of files in a distributed file system; The multidimensional features are input into a pre-built cold data prediction model, and the cold data prediction probability corresponding to the file is output. The input data of each file is determined according to the preset dynamic cold and hot data determination algorithm, and a list of cold data files is output; wherein, the input data includes the multidimensional features, the cold data prediction probability, and the current system load; The migration timing and target storage level of each file in the cold data file list are determined according to the preset migration scheduling strategy, so that the corresponding file is migrated to the corresponding target storage level at the migration timing; Record the actual access behavior after the file is archived, and establish a closed-loop feedback mechanism to dynamically optimize the cold data prediction model and the migration scheduling strategy.

2. The intelligent scheduling method for cold data according to claim 1, characterized in that, The step of determining the input data of each file according to a preset dynamic cold and hot data determination algorithm and outputting a list of cold data files includes: Extract the access frequency of the corresponding file from the input data; Based on the current system load and the judgment conditions in the dynamic hot and cold data judgment algorithm, the access frequency and cold data prediction probability of each file are comprehensively judged. All files that meet the aforementioned criteria are compiled into the cold data file list.

3. The intelligent scheduling method for cold data according to claim 2, characterized in that, The process involves combining the current system load with the judgment conditions in the dynamic hot / cold data determination algorithm to comprehensively determine the access frequency and cold data prediction probability of each file, including: If the current system load is less than the third preset threshold, then files whose cold data prediction probability is greater than the first preset threshold and whose access frequency is less than the second preset threshold are determined to meet the determination conditions; files that meet the determination conditions are marked as cold data.

4. The intelligent scheduling method for cold data according to claim 3, characterized in that, After marking files that meet the determination criteria as cold data, the process further includes: In files not marked as cold data, files whose cold data prediction probability is less than a fourth preset threshold or whose access frequency meets preset access conditions are marked as hot data. Add files that are not marked as cold or hot data to the monitoring queue.

5. The intelligent scheduling method for cold data according to claim 1, characterized in that, The step of determining the migration timing and target storage level for each file in the cold data file list according to a preset migration scheduling strategy includes: The migration priority queue is obtained by sorting each file in the cold data file list according to its multidimensional features and cold data prediction probability. The migration timing for each file is determined based on the migration priority queue, the current system load, and the load status of each resource coordinator, so that the corresponding file is migrated to the corresponding target storage level at the migration timing.

6. The intelligent scheduling method for cold data according to claim 5, characterized in that, After migrating the corresponding file to the corresponding target storage level at the migration time, the process further includes: If multiple cold data files trigger recovery requests simultaneously, they are prioritized based on their data value to obtain a corresponding recovery priority queue. Based on the recovery priority queue, cold data files that trigger recovery requests will be migrated to a hot storage level or a warm storage level.

7. The intelligent scheduling method for cold data according to claim 1, characterized in that, The method of recording actual access behavior after the file is archived and establishing a closed-loop feedback mechanism to dynamically optimize the cold data prediction model includes: Determine whether the archived file has been accessed within a preset number of days based on the actual access behavior; If the archived file is accessed within a preset number of days, an erroneous archiving event will be triggered. The files corresponding to the erroneous archiving events are fed back into the training set as erroneous archiving samples, and the training set is used to retrain the cold data prediction model.

8. An intelligent scheduling device for cold data, characterized in that, The device includes: The acquisition unit is used to collect multidimensional features of files in the distributed file system. The input / output unit is used to input the multidimensional features into a pre-built cold data prediction model and output the cold data prediction probability corresponding to the file. The determination unit is used to determine the input data of each file according to a preset dynamic determination algorithm for hot and cold data, and output a list of cold data files; wherein, the input data includes the multidimensional features, the cold data prediction probability, and the current system load; The migration unit is used to determine the migration timing and target storage level of each file in the cold data file list according to a preset migration scheduling strategy, so as to migrate the corresponding file to the corresponding target storage level at the migration timing. A unit is established to record the actual access behavior after the file is archived, and a closed-loop feedback mechanism is established to dynamically optimize the cold data prediction model and the migration scheduling strategy.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the intelligent scheduling method for cold data as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the intelligent scheduling method for cold data as described in any one of claims 1-7.

Images