Data storage expansion method, device, equipment and storage medium

Abstract

The application discloses a data storage expansion method and device, equipment and a storage medium, and relates to the technical field of server storage expansion. The method comprises the following steps: firstly, intelligently analyzing the generation time and historical access behavior of data to identify and layer different value data. Then, combining the real-time disk usage rate of the core server, a corresponding data migration strategy is formulated. Then, according to the strategy and the running state information of the edge gradient resource pool, a specific data transmission instruction is generated. After that, based on the instruction, the data is transmitted to a remote storage node through a multi-thread concurrent technology. Finally, the transparent access request of the user is processed by using the file path mapping relationship, and the local access experience of the remote data is realized. Through the above method, the automatic and intelligent expansion and hierarchical management of storage resources are realized, and the storage resource utilization rate, system energy efficiency and data access convenience are effectively improved.

Description

Technical Field

[0001] This application relates to the field of server storage expansion technology, and in particular to data storage expansion methods, apparatus, equipment and storage media. Background Technology

[0002] The production servers in the core computer room of the laboratory need to handle real-time storage and high-frequency access tasks. Their data disks have limited capacity, and due to hardware limitations, they can only store short-term or near-expiration experimental data. When disk capacity is insufficient, manual migration of long-term data is often necessary. However, manual migration is inefficient, prone to path confusion or data loss, and cannot cope with the daily increase in experimental data. Summary of the Invention

[0003] The main purpose of this application is to provide a data storage expansion method, apparatus, device and storage medium, which aims to solve the technical problem of low data migration efficiency of servers in the prior art.

[0004] To achieve the above objectives, this application provides a data storage expansion method, the method comprising: Data hierarchical identification results are obtained based on data generation time and access behavior; A data migration strategy is derived based on the data hierarchical identification results and the core server disk usage. Data transmission instructions are obtained based on the data migration strategy and the status information of the edge tiered resource pool; Based on the data transmission instructions, control multiple threads to concurrently transmit data to the remote storage node to obtain the remote storage data; The localized data access result is obtained based on the file path mapping relationship and the transparent access request.

[0005] In one embodiment, the step of obtaining data hierarchical identification results based on data generation time and access behavior includes: The data generation time of the data file is scanned according to the preset time window to obtain the initial timeliness classification results; By analyzing access behavior in historical access logs, dynamic access behavior characteristics can be obtained. Based on the initial timeliness classification results and the dynamic access behavior characteristics, the data value prediction heat value is obtained; The data stratification identification result is obtained based on the predicted data value and the preset stratification threshold.

[0006] In one embodiment, the step of obtaining a data migration strategy based on the data layering identification results and the core server disk usage includes: Disk utilization is obtained based on real-time monitoring data of the core server disks; The real-time transfer trigger signal is determined based on the disk usage rate and the preset capacity threshold. An automatic migration task queue is obtained based on the data hierarchical identification results and the preset archiving strategy; A data migration strategy is generated based on the real-time transfer trigger signal and the automatic migration task queue.

[0007] In one embodiment, the step of obtaining the data transmission instruction based on the data migration strategy and the status information of the edge tiered resource pool includes: Extract edge node registration information, hardware parameters, node energy consumption monitoring data, and load status from the status information of the edge tiered resource pool; Resource hierarchical configuration information is obtained based on the edge node registration information and the hardware parameters; Based on the node energy consumption monitoring data and the load status, a dynamic energy efficiency assessment result is obtained; A node candidate sequence is obtained based on the resource classification and configuration information and the dynamic energy efficiency assessment results; Based on the data migration strategy and the node candidate sequence, hierarchical data transmission instructions are obtained.

[0008] In one embodiment, the step of controlling multi-threaded concurrent transmission to a remote storage node based on the data transmission instruction to obtain remote storage data includes: The data type and target node attributes are determined according to the data transmission instructions; Generate a transmission path optimization strategy based on the transmitted data type and the target node attributes; Concurrency control parameters are obtained based on network bandwidth characteristics and breakpoint transmission mechanisms; Based on the transmission path optimization strategy and the concurrency control parameters, a verification data transmission stream is obtained; The verification data transmission stream is updated based on hash operations and audit logs to obtain tamper-proof storage data. The tamper-proof storage data is then transmitted concurrently to a remote storage node using multiple threads to obtain remote storage data.

[0009] In one embodiment, the data storage expansion method further includes: Collect hardware status indicators of remote storage nodes, and obtain real-time health data based on the hardware status indicators; Based on the real-time health data, a prediction and assessment are performed to obtain node risk warning information; Determine the data distribution map, and generate a data migration strategy based on the data distribution map and the node risk warning information; Data migration is performed on the remote storage node based on the data migration strategy.

[0010] In one embodiment, the data storage expansion method further includes: Determine the data heatmap correction parameters based on user access behavior logs; Energy efficiency management instructions are generated based on node sleep thresholds and wake-up prediction results; The data migration strategy is updated based on the thermal value correction parameters and the energy efficiency management instructions.

[0011] Furthermore, to achieve the above objectives, this application also proposes a data storage expansion device, which includes: The layered identification module is used to obtain data layered identification results based on data generation time and access behavior; The strategy formulation module is used to obtain a data migration strategy based on the data hierarchical identification results and the core server disk usage rate. The instruction generation module is used to obtain data transmission instructions based on the data migration strategy and the status information of the edge tiered resource pool; The concurrent transmission module is used to control multi-threaded concurrent transmission to the remote storage node based on the data transmission instructions, so as to obtain the remote storage data; The transparent access module is used to obtain localized data access results based on file path mapping relationships and transparent access requests.

[0012] In addition, to achieve the above objectives, this application also proposes a data storage expansion device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data storage expansion method described above.

[0013] In addition, to achieve the above objectives, the present invention also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data storage expansion method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data storage expansion method described above.

[0015] This application provides a data storage expansion method. First, it intelligently analyzes the data generation time and historical access behavior to identify and stratify data of different values. Next, it formulates corresponding data migration strategies based on the real-time disk utilization of the core server. Then, based on this strategy and the operational status information of the edge tiered resource pool, it generates specific data transmission instructions. Subsequently, based on these instructions, it transmits data to remote storage nodes using multi-threaded concurrency technology. Finally, it utilizes file path mapping relationships to handle transparent user access requests, achieving a localized access experience for remote data. Through these methods, this solution achieves automated, intelligent expansion and hierarchical management of storage resources, effectively improving storage resource utilization, system energy efficiency, and data access convenience. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating Embodiment 1 of the data storage expansion method of this application; Figure 2 This is a hierarchical data migration scheduling diagram of one embodiment of the data storage expansion method of this application; Figure 3 This is a schematic diagram of the data storage system device connection topology according to an embodiment of the data storage expansion method of this application; Figure 4 This is a schematic diagram of the module structure of the data storage expansion device in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware operating environment involved in the data storage expansion method in this application embodiment.

[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is: to obtain data layering identification results based on data generation time and access behavior; A data migration strategy is derived based on the data hierarchical identification results and the core server disk usage. Data transmission instructions are obtained based on the data migration strategy and the status information of the edge tiered resource pool; Based on the data transmission instructions, control multiple threads to concurrently transmit data to the remote storage node to obtain the remote storage data; The localized data access result is obtained based on the file path mapping relationship and the transparent access request.

[0023] Currently, the production servers in the core computer room of the laboratory need to undertake real-time storage and high-frequency access tasks. Their data disk capacity is limited, and due to hardware constraints, they can only store short-term or near-expiration experimental data. When disk capacity is insufficient, manual migration of long-term data is often necessary. However, manual migration is inefficient, prone to path confusion or data loss, and cannot cope with the daily increase in experimental data.

[0024] This application provides a solution that, firstly, intelligently analyzes the generation time and historical access behavior of comprehensive data to identify and stratify data of different values. Next, it formulates corresponding data migration strategies based on the real-time disk utilization of the core server. Then, based on this strategy and the operational status information of the edge tiered resource pool, it generates specific data transmission instructions. Subsequently, based on these instructions, it transmits data to remote storage nodes using multi-threaded concurrency technology. Finally, it utilizes file path mapping relationships to handle transparent user access requests, achieving a localized access experience for remote data. Through these methods, this solution achieves automated, intelligent expansion and hierarchical management of storage resources, effectively improving storage resource utilization, system energy efficiency, and data access convenience.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or data storage expansion device capable of performing the above functions. This embodiment does not specifically limit it in this regard. The following uses a data storage expansion device as an example to describe this embodiment and the following embodiments.

[0026] All actions involving the acquisition of signals, information, or data in this application are carried out in accordance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the owner of the relevant device.

[0027] This application provides a data storage expansion method, referring to... Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the data storage expansion method of this application.

[0028] In this embodiment, the data storage expansion method includes steps S10 to S50: Step S10: Obtain data layering identification results based on data generation time and access behavior.

[0029] It should be noted that this embodiment provides an overall architecture, employing a tiered storage architecture with core server local retention, edge tiered resource hierarchical archiving, and high-speed transmission. The core logic is as follows: the core server stores only recent (within 10 days) or near-expiration latency data, ensuring speed and stability for high-frequency access. Then, edge resource devices (such as a 10TB NAS) are designated as primary edge storage nodes (storing important long-term data), and older computers are designated as secondary edge storage / transmission nodes (storing ordinary long-term data and acting as data transmission relays), together forming an edge tiered resource pool. Through preset rules, long-term data exceeding 10 days is automatically synchronized to the edge tiered resource pool every morning, or new data is transferred to edge nodes in real time when the core server's disk utilization reaches a threshold. Transparent access to long-term data is achieved using gigabit LAN and path mapping technology. Simultaneously, an older UPS is configured for the 10TB NAS to ensure power supply stability.

[0030] It is understandable that data generation time refers to the write time to disk, access behavior refers to the frequency of data access and access latency sensitivity, and data stratification identification result refers to the classification level of the data.

[0031] In the specific implementation, the write time of data recorded on the core server's disk is queried to determine the data generation time. Then, the current time is determined and compared with the data generation time to determine the data's lifespan on the core server. Simultaneously, the core server's access logs are retrieved to determine the frequency of access to various data stored on the core server's disk and their sensitivity to storage latency, thus identifying data access behavior. Based on the various possible permutations and combinations of data generation time and access records, multiple data classification strategies can be formed. Based on these strategies, the core data is classified to obtain data stratification and identification results.

[0032] In one feasible implementation, the step of obtaining data hierarchical identification results based on data generation time and access behavior includes: The data generation time of the data file is scanned according to the preset time window to obtain the initial timeliness classification results; By analyzing access behavior in historical access logs, dynamic access behavior characteristics can be obtained. Based on the initial timeliness classification results and the dynamic access behavior characteristics, the data value prediction heat value is obtained; The data stratification identification result is obtained based on the predicted data value and the preset stratification threshold.

[0033] It should be noted that the preset time window refers to the generation time threshold for each data level, and the specific scale is set according to the actual situation. Dynamic access behavior characteristics refer to the frequency of data access and the characteristics related to access sensitivity. The data value prediction heat value is used to describe the data classification level; the higher the data value prediction heat value, the more important it is to the core server.

[0034] In the specific implementation, the data generation time of the data file is first scanned according to a preset time window to determine which data classification level the data file generation time belongs to. During the time window scan, access behavior in historical access logs is analyzed simultaneously to determine the data file's read frequency and access latency sensitivity. For example, data can be divided according to a three-level classification system as the core basis for storage hierarchy, while preset capacity threshold rules are implemented, and all rule parameters can be adjusted as needed. 1) Three-level data classification rule: Automatically scan and identify data generation time and access logs, classifying the data into three categories: a) Recent data: Data generation time ≤ 0.5 months, the core characteristics are high access frequency and sensitivity to access latency; b) Long-term data: Data generated more than 0.5 months and less than or equal to 3 months, with no download records in the past 3 months. The core characteristic is that it needs to be retained for a long time but has a low access frequency. c) Long-term data: Data generated more than 3 months ago, or has been downloaded within the last 3 months (the loss of this type of data has little impact on scientific research). Its core characteristics are low retention requirements and extremely low access frequency.

[0035] 2) Capacity threshold rule: Real-time monitoring of the utilization rate of the core server's 2TB disk (default threshold ≥85%). When the utilization rate reaches the threshold, a dynamic conversion mechanism is immediately triggered—converting the relatively distant generation time portion of recent data into distant data and generating a real-time transfer trigger signal. 3) Category Update Rules: Data access logs and generation times are synchronized daily, and data category tags are automatically updated to ensure dynamic adaptability of storage allocation. The module only performs migration marking on distant and very distant data, and does not interfere with the normal access to recent data.

[0036] When dynamically predicting the value of data, multidimensional features of the data can be collected, including file type, size, creator, project (obtainable from the LIMS system), historical access frequency sequence, and recent access time. A lightweight machine learning model (such as XGBoost or lightweight LSTM) is trained and learned online to calculate a dynamic "heat value" (scaled from 0-1) for each data object, reflecting its probability of being accessed within a specific future time period (e.g., the next two weeks). This heat value will serve as an important or core basis for data stratification. For example, data with a heat value > 0.6 is prioritized for retention on the core server (recent data), data with a heat value ≤ 0.6 (0.2 < heat value ≤ 0.6) is migrated to first-level edge nodes (long-term data), and data with a heat value ≤ 0.2 is migrated to second-level edge nodes (ultra-long-term data). This rule can override or complement existing time rules to achieve more accurate classification.

[0037] Step S20: Obtain a data migration strategy based on the data layering identification results and the core server disk usage rate.

[0038] It should be noted that the data migration strategy refers to the strategy for migrating a certain data file based on the data stratification results and the disk utilization of the core server. The data to be migrated is received by a primary edge storage node or a secondary edge storage node.

[0039] It should be understood that, in addition to the core server disks, the experimental data storage also includes primary and secondary edge storage nodes. The primary edge storage nodes consist of idle 10TB NAS devices and large-capacity older workstations (no bad sectors, supports NFS / iSCSI protocols, read / write speeds ≥100MB / s). Their core hardware advantages include TB-level capacity, gigabit-level network speeds, and some devices have built-in UPS or can be connected to idle UPS systems, ensuring high stability. They are specifically designed for storing long-term data, relying on high stability to ensure the long-term security of this type of data, while also utilizing gigabit network ports to meet occasional access needs for long-term data. Secondary edge storage nodes: These are older computers in the lab that are constantly running (hard drive capacity 200GB-800GB, bad sector rate <0.5%, network port speed of 100Mbps). Their hardware performance is relatively weak, but their quantity is sufficient. They are specifically designed to store very long-term data, which is in line with the characteristics of this type of data: "low retention requirements and extremely low access frequency". At the same time, they can undertake data migration relay tasks during off-peak business periods, maximizing the use of idle resources.

[0040] Understandably, a data migration strategy can be derived based on the data stratification identification results and the core server disk utilization rate. Specifically: if the core server disk utilization rate does not reach a threshold, data migration may not be performed. When the core server disk utilization rate reaches the threshold, the data stratification identification result is determined as recent data, long-term data, or very long-term data. If the data is recent, it is retained on the core server disk; if the data is long-term, it is migrated to the first-level edge storage node; if the data is very long-term, it is migrated to the first-level edge storage node, and then from the first-level edge storage node to the second-level edge storage node.

[0041] In one feasible implementation, the step of obtaining the data migration strategy based on the data layering identification results and the core server disk usage includes: Disk utilization is obtained based on real-time monitoring data of the core server disks; The real-time transfer trigger signal is determined based on the disk usage rate and the preset capacity threshold. An automatic migration task queue is obtained based on the data hierarchical identification results and the preset archiving strategy; A data migration strategy is generated based on the real-time transfer trigger signal and the automatic migration task queue.

[0042] It should be noted that the real-time transfer trigger signal refers to the instruction information that triggers data migration. Upon receiving the real-time transfer trigger signal, data migration begins. The automatic migration task queue holds the data files that need to be migrated, maintaining a consistent correspondence with the archiving strategy.

[0043] In the specific implementation, disk utilization is first determined based on real-time monitoring data of the core server disks. When the disk utilization reaches a preset capacity threshold, a real-time transfer trigger signal is generated. The preset capacity threshold can be set according to the actual situation, such as 80% or 85%. After obtaining the real-time transfer trigger signal, a corresponding preset archiving strategy is matched based on the recent data, distant data, or very distant data in the data layer identification results. Recent data does not need to be migrated, while distant or very distant data needs to be migrated to a primary edge storage node or, respectively, to a primary edge storage node first, and then from the primary edge storage node to a secondary edge storage node. An automatic migration task queue is generated according to the type. The real-time transfer trigger signal includes the automatic execution time. Based on the real-time transfer trigger signal and the automatic migration task queue, a data migration strategy is generated.

[0044] Step S30: Obtain data transmission instructions based on the data migration strategy and the status information of the edge tiered resource pool.

[0045] It should be noted that data transmission instructions are information used to instruct when, where, and how data should be transmitted to a specific location. The status information of the edge tiered resource pool is a dynamic set of data describing the current operating status and availability of each edge storage node in the pool. It mainly includes node registration information (such as IP address and node ID), hardware configuration parameters (such as storage capacity and hard drive type), real-time load conditions (such as CPU / memory / disk I / O utilization and network traffic), and energy consumption indicators (such as instantaneous power and energy efficiency ratio).

[0046] Understandably, the first step is to obtain a snapshot of the real-time status information of all registered nodes from the edge tiered resource management module. The scheduler then parses the data migration strategy, clarifying the target tier (e.g., storage in a first- or second-tier edge node) and the amount of data to be migrated. Next, it performs multi-objective optimization decisions based on the status information: filtering candidate node clusters that meet storage requirements based on hardware parameters (e.g., remaining capacity, read / write speed); assessing the current processing capacity of nodes based on real-time load (e.g., CPU / memory utilization, network bandwidth usage) to avoid distributing tasks to overloaded nodes; and prioritizing nodes with high energy efficiency or those already in an active state, referencing energy consumption monitoring data to reduce overall system power consumption. Finally, the scheduler generates tiered data transmission instructions containing details such as the specific source-destination path, transmission priority, number of threads used, and a list of selectable relay nodes, and issues these instructions to the transmission execution module. For example, an important but large data packet might be specified to be transmitted directly via a gigabit network to a lightly loaded and stable first-tier NAS node; while multiple non-urgent small files might be distributed to multiple available second-tier nodes for parallel transmission to improve overall throughput.

[0047] In one feasible implementation, the step of obtaining the data transmission instruction based on the data migration strategy and the status information of the edge tiered resource pool includes: Extract edge node registration information, hardware parameters, node energy consumption monitoring data, and load status from the status information of the edge tiered resource pool; Resource hierarchical configuration information is obtained based on the edge node registration information and the hardware parameters; Based on the node energy consumption monitoring data and the load status, a dynamic energy efficiency assessment result is obtained; A node candidate sequence is obtained based on the resource classification and configuration information and the dynamic energy efficiency assessment results; Based on the data migration strategy and the node candidate sequence, hierarchical data transmission instructions are obtained.

[0048] It should be noted that the resource classification and configuration information is a structured data that classifies and classifies the capabilities of all nodes in the pool based on the registration details and inherent hardware attributes of the edge nodes, according to preset performance and reliability standards (for example, classifying NAS with RAID function and gigabit network card as first-level nodes, and classifying old computers with ordinary hard drives and 100 Mbps network card as second-level nodes).

[0049] In the specific implementation, refer to Figure 2 Based on a gigabit LAN environment in the laboratory, this system integrates multi-threaded concurrent transmission and breakpoint resume technology, optimizing transmission strategies for different data types: when transmitting important data, a high-speed link is directly established between the core server and the 10TB NAS via gigabit network; when transmitting ordinary data, the concurrent capabilities of secondary edge transmission nodes are utilized for traffic distribution, with a maximum transmission rate of up to 110MB / s. A built-in verification mechanism ensures no data loss or corruption during transmission. First, the system obtains the original status information of the edge nodes from the registry center and data collector. Then, the resource grading module performs hard screening and grading of nodes based on predefined hardware thresholds (such as storage capacity exceeding the amount of data to be migrated and network speed meeting minimum requirements), forming resource grading configuration information to ensure that data can only be written to nodes with compliant hardware and matching data types. In parallel, the energy efficiency assessment module analyzes the real-time energy consumption and load data of the nodes, using models (such as calculating the ratio of current power to IOPS) to derive dynamic energy efficiency assessment results, identifying green nodes and high-power, inefficient nodes that should be avoided. Then, the decision engine merges these two pieces of information. For example, it prioritizes nodes with excellent energy efficiency assessment results and current loads below a threshold within the same level, generating an ordered sequence of node candidates. Finally, the scheduler combines the data migration strategy with this sequence to generate specific, executable hierarchical data transmission instructions for each batch of data to be migrated. The instructions explicitly specify the target node and transmission priority.

[0050] Step S40: Based on the data transmission instruction, control multiple threads to concurrently transmit data to the remote storage node to obtain the remote storage data.

[0051] It should be noted that remote storage data refers to data that has been migrated from the core server disk and stored in the edge storage node.

[0052] Understandably, the engine first parses the instruction content to clarify the goal and constraints of the transmission task. Then, based on the number of threads specified in the instruction, the engine creates multiple independent transmission threads, each responsible for transmitting a large data file or a batch of small files. To optimize transmission efficiency, the engine dynamically adjusts the load on each thread based on network bandwidth and target node attributes (such as I / O capabilities), and uses a breakpoint resumption mechanism to record transmission progress, ensuring that transmission can resume from the breakpoint after network interruption or system failure recovery, avoiding redundant work. Before or after the actual transmission of data blocks, a checksum (such as MD5 or SHA-256 hash value) comparison mechanism is used to ensure data integrity during transmission. All transmission activities are recorded in the audit log. Finally, when all threads report that the data blocks they are responsible for have been successfully written to the target node and passed verification, it signifies that the remote storage data has been reliably created, and the system updates the metadata index and path mapping table to reflect the new location of the data.

[0053] In one feasible implementation, the step of controlling multi-threaded concurrent transmission to a remote storage node based on the data transmission instruction to obtain remote storage data includes: The data type and target node attributes are determined according to the data transmission instructions; Generate a transmission path optimization strategy based on the transmitted data type and the target node attributes; Concurrency control parameters are obtained based on network bandwidth characteristics and breakpoint transmission mechanisms; Based on the transmission path optimization strategy and the concurrency control parameters, a verification data transmission stream is obtained; The verification data transmission stream is updated based on hash operations and audit logs to obtain tamper-proof storage data. The tamper-proof storage data is then transmitted concurrently to a remote storage node using multiple threads to obtain remote storage data.

[0054] In practical implementation, when controlling multi-threaded concurrent transmission to remote storage nodes based on data transmission instructions to obtain remote storage data, various data synchronization methods exist depending on the actual situation. These may include at least a daily automatic synchronization mode and a real-time transfer mode. Daily automatic synchronization mode: For example, at 3 AM every day (off-peak period for scientific research), precise migration is performed based on the classification list generated by the core data hierarchical identification module: First, the portion of data that has been converted to long-term data in the core server (triggered by the core disk threshold) and the original long-term data are synchronized to the first-level edge storage node (10TB NAS / large-capacity old workstation); then, the ultra-long-term data (including data generated more than 3 months ago and long-term data with download records in the last 3 months) is gradually migrated from the first-level edge storage node to the second-level edge storage node (old computers); after synchronization is completed, redundant source files that have been migrated in the core server and the first-level node are automatically deleted, and the file access path mapping information of different categories of data is synchronized to the file transparent access module.

[0055] Furthermore, at 3 AM daily (the off-peak period for research operations), the dynamic synchronization and transfer module first obtains the dynamic heatmap values ​​of all data to be migrated from the data value dynamic prediction module. Then, it combines the heatmap values ​​with the original timeliness rules to generate an optimized migration list. For example, even if the data was generated more than 0.5 months ago, if its heatmap value increases due to the approaching project completion date, the migration can be postponed or the target level adjusted. During migration, the energy efficiency management module provides the current energy consumption and status information of each edge node, prioritizing the writing of data to active and energy-efficient nodes.

[0056] Real-time transfer mode: When the core server's 2TB disk utilization is ≥85%, the transfer mechanism is immediately triggered, and the newly added experimental data is directly written to the corresponding edge node to avoid core disk overflow; after the core server's disk utilization drops below the threshold, some data can be migrated back as needed.

[0057] Furthermore, when the core server disk utilization rate is ≥85%, a data rollback is immediately triggered. At this time, for newly added data, the system can quickly estimate an initial heat value based on its file attributes (such as type and creator), and combine this with the current optimal node recommended by the edge node energy efficiency management module to perform real-time writing.

[0058] Step S50: Obtain the localized data access result based on the file path mapping relationship and the transparent access request.

[0059] In the specific implementation, a file path mapping table between the core server and edge tiered devices is maintained. When researchers access the archived long-term data path on the core server, the module automatically identifies the storage location of the edge node corresponding to the path and temporarily mounts the target directory of the edge node to the local core server via gigabit network. After the access is completed, the mounted directory is automatically unmounted. The whole process is completely transparent to researchers, and the operation experience is consistent with accessing local data, with an access response time of ≤5 seconds.

[0060] Furthermore, when transparent access is triggered, this access event is sent in real time as an important feedback signal to the data value dynamic prediction module to update the access pattern of the data and dynamically adjust its heat value. For frequently accessed edge data, the system can automatically evaluate and migrate it back to a higher-speed storage layer (such as migrating it from a secondary node to a primary node or core server).

[0061] In one feasible implementation, the data storage expansion method further includes: Collect hardware status indicators of remote storage nodes, and obtain real-time health data based on the hardware status indicators; Based on the real-time health data, a prediction and assessment are performed to obtain node risk warning information; Determine the data distribution map, and generate a data migration strategy based on the data distribution map and the node risk warning information; Data migration is performed on the remote storage node based on the data migration strategy.

[0062] In the specific implementation, a lightweight Python agent is used to complete node registration and status monitoring, and edge nodes are divided into two levels according to storage performance, capacity, and reliability. First, a predictive assessment is performed based on real-time health data to obtain node risk warning information. The predictive assessment method can include proactive health monitoring and predictive maintenance triggering. Proactive health monitoring: Periodically (e.g., hourly), the agent collects deep hardware status information of edge nodes, including disk SMART attributes (such as reallocation sector count, seek error rate), memory ECC error count, network interface packet loss rate and bit error rate, and performs lightweight read and write benchmark tests to monitor performance degradation. Based on these indicators, a pre-trained fault prediction model or a set threshold alarm is used to calculate the node's "reliability score"; when the score is lower than the set threshold or a single indicator deteriorates sharply, the node is proactively marked as "suspicious" or "fault warning" and an early warning signal is immediately sent to the dynamic synchronization and transfer module. Predictive maintenance trigger: For nodes marked as "fault warning", this module will coordinate with the dynamic synchronization and transfer module to proactively migrate all data stored on them to other healthy edge nodes within a set safe time window, thereby achieving data risk avoidance before a fault occurs.

[0063] While meeting data storage reliability requirements, data writing prioritizes available nodes with high current energy efficiency. For secondary nodes storing very long-term data and without access or transmission tasks for extended periods (e.g., 24 consecutive hours), the module uses technologies such as Wake-on-LAN to put them into sleep or low-power states. When it is predicted that access to their data or their transmission capabilities will be needed, they will be woken up in advance. This mechanism can significantly reduce system standby power consumption.

[0064] In one feasible implementation, the data storage expansion method further includes: Determine the data heatmap correction parameters based on user access behavior logs; Energy efficiency management instructions are generated based on node sleep thresholds and wake-up prediction results; The data migration strategy is updated based on the thermal value correction parameters and the energy efficiency management instructions.

[0065] It should be noted that the data heat value adjustment parameter is calculated based on the user's recent actual access behavior. When data is accessed frequently, this parameter will increase its heat value, and vice versa, thus making the data stratification decision more closely reflect actual usage.

[0066] In its implementation, by continuously collecting and analyzing user access behavior logs, an algorithm generates a dynamic heat value correction parameter for each piece of data. This parameter overrides or corrects the initial static importance classification of the data. Simultaneously, the energy efficiency management module monitors the activity status of all nodes and the total system load. When a node's idle time exceeds a preset hibernation threshold and the prediction model indicates no high load risk in the short term, the module generates an energy efficiency management instruction to enter hibernation. Conversely, when a business peak is predicted or a scheduled task notification is received, a pre-wake-up instruction is generated. These real-time generated heat value correction parameters and energy efficiency management instructions are used as key inputs and fed back to the strategy engine. The strategy engine then dynamically adjusts the data migration strategy, for example, increasing the redundancy of high-heat data replicas and locking them to consistently online high-performance nodes, or guiding low-heat data to concentrate on energy-saving nodes that can hibernate, thereby maximizing the number of hibernating nodes when allowed to hibernate.

[0067] This embodiment also includes an edge power supply protection module, which is connected to an old UPS device that is not in use in the laboratory, and is dedicated to providing uninterrupted power supply to the primary edge storage node (10TB NAS). When the mains power is interrupted, the UPS automatically switches the power supply mode and sends a power outage warning signal to the administrator via email to ensure the storage security of important long-term data in the 10TB NAS and avoid data damage or loss due to power outage.

[0068] Let's take an example to illustrate this. This embodiment is based on the experimental data storage expansion scenario of a university laboratory (such as an optical imaging laboratory, molecular biology laboratory, material structure analysis laboratory, ultrastructure laboratory, cell biology laboratory, pathology laboratory, small animal imaging laboratory, and testing technology innovation laboratory, etc.). The core server's 2TB disk only stores 10 days of data, and the edge tiered resources include one 10TB NAS and 10 old computers. The laboratory's local area network has gigabit bandwidth.

[0069] First, the experimental environment hardware configuration includes: Core data center production server: Equipped with a 2TB SATA data disk, with the file system ext4, storing only 10 days of material test data (daily increment of approximately 200GB, 10-day cumulative total of 2TB, disk utilization rate of 50%), connected to the core data center gigabit switch, with read and write speeds ≥200MB / s.

[0070] Edge tier devices: 1 idle 10TB NAS (4-bay RAID 5, supports NFS protocol, no bad sectors, read / write speed of 110MB / s) as a primary edge storage node; 10 long-term powered-on old computers (Core i5-6th generation, 8GB memory, 2TB SATA hard drive, gigabit network port, hard drive bad sector rate of 0.3%) as secondary edge storage / transmission nodes.

[0071] Power supply equipment: 1 old laboratory UPS (rated power 2000VA, battery life ≥30 minutes), dedicated to powering the 10TB NAS; Software configuration: lightweight Python agent, NFS distributed storage protocol, multi-threaded resume download tool, file path mapping tool.

[0072] AI and energy management components: Python Scikit-learn or TensorFlow Lite libraries (for lightweight models), power monitoring software (such as powertop) adaptation, and Wake-on-LAN configuration tools.

[0073] In the specific implementation, a Python agent program is deployed on a 10TB NAS and 10 old computers to complete the edge node registration. After detecting the hardware parameters, the edge tiered resource management module sets the 10TB NAS as a primary edge storage node and the 10 old computers as secondary edge storage / transmission nodes, thus constructing an edge tiered resource pool. At the same time, the energy efficiency management module measures and records the basic power consumption (e.g., 35W-50W) of each old computer in idle state and enables its WOL function.

[0074] In the core data stratification and identification module of the core server, a time stratification threshold of 10 days and a core disk capacity threshold of 85% were set. Data matching rules were configured: material composition analysis data (important data) was stored on a 10TB NAS, and experimental process logs (ordinary data) were stored on secondary edge storage nodes. Simultaneously, the data value dynamic prediction module began collecting two weeks of data access history for model initialization training. Initially, data classification was jointly determined by a 70% weighted time rule and a 30% weighted heat value rule; as data accumulated, it gradually transitioned to a heat value rule as the primary factor.

[0075] Every day at 3 AM, the core data layering identification module scans for data older than 10 days (approximately 200GB), including 100GB of important data and 100GB of general data. The dynamic synchronization and transfer module utilizes five secondary edge transmission nodes to synchronize the important data to the 10TB NAS and the general data to the secondary edge storage nodes via gigabit network. The transfer rate is stable at 105MB / s, and the migration takes approximately 32 minutes. After synchronization, the path mapping table is updated and the source files on the core server are deleted, causing the core disk utilization to drop back to 50%. Alternatively, every day at 3 AM, the data value dynamic prediction module calculates the heat value of all data to be migrated. It finds that a batch of "material characterization maps" (belonging to long-term data) generated 15 days ago has a heat value of 0.75 because the associated project is scheduled for a mid-term presentation next week. The system decides not to migrate this batch of data and to keep it on the core server. At the same time, the energy efficiency management module shows that three secondary nodes have excellent energy efficiency ratios and low loads over the past 24 hours, so they are prioritized for this transfer. The final data migration volume became 150GB, taking approximately 24 minutes.

[0076] In a simulated scenario of sudden data surge, the core server's disk utilization rose to 86% within two hours, triggering a real-time data transfer mechanism. The dynamic synchronous transfer module immediately wrote the newly added experimental data directly to the secondary edge transmission node, which then distributed it to the corresponding storage node. The core disk utilization did not rise further, and data writing was uninterrupted. During real-time transfer, the newly added "Experimental Environment Parameter Record" file was quickly estimated to have a low thermal value (0.15), and the system detected that a dormant node had just been awakened (its energy efficiency was temporarily low). Therefore, it was routed to the 7th old computer node, which had been active and had the highest energy efficiency, for temporary storage.

[0077] When researchers needed to retrieve material composition analysis data from 15 days prior, they accessed the path / data / material / 20260101 on the core server. The transparent file access module automatically recognized this path as corresponding to the / archive / material / 20260101 directory on the 10TB NAS, completed mounting and data retrieval within 3 seconds, providing an experience consistent with accessing local data. This access event was recorded, and the overall heat map value of the data in that directory was slightly increased.

[0078] In a simulated power outage scenario in the laboratory, the old UPS immediately powered the 10TB NAS and sent an alert email to the administrator. Within 30 minutes of the backup power, the administrator completed the emergency power connection, and the important data stored in the 10TB NAS was not damaged or lost.

[0079] Over the weekend, with no tasks in the lab, the edge node energy efficiency management module detected that, excluding node number 7 which was responsible for temporary task storage, the other nine out of ten older computers had not performed any data access or transmission tasks for 12 consecutive hours. The system then sequentially sent hibernation commands to these nine nodes, putting them into a low-power state. On Monday morning at 8:00 AM, based on historical access patterns, the system predicted an increase in access volume after 9:00 AM, and therefore, at 8:30 AM, it batch-wake up these nine nodes for scheduling purposes. This process saved approximately 36 hours of standby power consumption over the weekend.

[0080] In one feasible implementation, when the available capacity of the 10TB NAS is insufficient, the edge tiered resource management module automatically migrates subsequent important data to a less loaded secondary edge storage node and marks it as a "temporary important data storage node." Once the 10TB NAS has available capacity, it automatically migrates back, achieving load balancing among edge nodes. When deciding on migration targets, the node energy efficiency ratio provided by the edge node energy efficiency management module serves as an important reference indicator for load balancing, prioritizing nodes with moderate load and high energy efficiency. Simultaneously, it can also integrate edge tiered equipment from other laboratories on the same floor to construct a cross-laboratory tiered resource pool; VLAN isolation technology can be used to divide data storage areas in different laboratories, achieving resource sharing and data isolation, suitable for the storage expansion needs of large-scale research platforms.

[0081] In one feasible implementation, the data storage expansion method further includes: constructing a topology that includes a core server and at least one edge tiered resource pool device connection, wherein the edge tiered resource pool includes a primary edge storage node and a secondary edge storage node connected to the core server via a local area network, wherein a storage device that supports a first preset hardware standard and is connected to an uninterruptible power supply is registered as the primary edge storage node, and the primary edge storage node is used to store long-term data and undertake data transmission relay tasks; A computing device that supports the second preset hardware standard is registered as the secondary edge storage node, which is used to store long-term data.

[0082] It should be noted that device connection topology refers to the physical and logical network structure that constitutes the data storage system. Its core is to connect the core server and the edge tiered resource pool composed of heterogeneous devices in a hierarchical manner through a local area network. The first-level edge storage node refers to a dedicated storage device that meets high hardware specifications and is equipped with uninterruptible power supply. It is responsible for storing important data and providing stable services. The second-level edge storage node refers to a resource pool composed of a large number of idle computing devices with slightly lower performance. It is used to handle data storage of lower value.

[0083] In the specific implementation, refer to Figure 3 By deploying lightweight agents across the network, the system automatically scans and identifies eligible devices. For NAS or servers with large capacity (e.g., ≥4TB), gigabit network cards, and RAID capabilities, and connected to a UPS, these devices are registered as primary nodes and marked as high-reliability storage targets. For older computers and other devices with stable online status, after verifying their hard drive health (bad sector rate below a threshold) and network connectivity, they are registered as secondary nodes. Subsequently, the core server's resource management module maintains a dynamic node list, recording each node's IP address, hardware capabilities, current load, and power status. Based on this, a star or tree-like logical topology is established. The core server, acting as the scheduling center, communicates with all edge nodes through a heartbeat mechanism, monitoring their availability. This provides a solid physical network foundation and real-time status awareness for the execution of data migration strategies.

[0084] This embodiment provides a data storage expansion method. First, intelligent analysis is performed on the data generation time and historical access behavior to identify and stratify data of different values. Next, a corresponding data migration strategy is formulated based on the real-time disk utilization of the core server. Then, specific data transmission instructions are generated according to this strategy and the operational status information of the edge tiered resource pool. Subsequently, data is transmitted to remote storage nodes using multi-threaded concurrency technology based on these instructions. Finally, transparent access requests from users are handled using file path mapping relationships, achieving a localized access experience for remote data. Through the above methods, this solution achieves automated and intelligent expansion and hierarchical management of storage resources, effectively improving storage resource utilization, system energy efficiency, and data access convenience.

[0085] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data storage expansion method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0086] This application also provides a data storage expansion device; please refer to... Figure 3 The data storage expansion device includes: The hierarchical identification module 10 is used to obtain data hierarchical identification results based on data generation time and access behavior; The strategy formulation module 20 is used to obtain a data migration strategy based on the data layering identification results and the core server disk usage rate; Instruction generation module 30 is used to obtain data transmission instructions based on the data migration strategy and the status information of the edge tiered resource pool; The concurrent transmission module 40 is used to control multi-threaded concurrent transmission to the remote storage node based on the data transmission instruction to obtain the remote storage data; The transparent access module 50 is used to obtain localized data access results based on the file path mapping relationship and the transparent access request.

[0087] In one feasible implementation, the hierarchical identification module 10 is further used to scan the data generation time of the data file according to a preset time window to obtain an initial timeliness classification result; By analyzing access behavior in historical access logs, dynamic access behavior characteristics can be obtained. Based on the initial timeliness classification results and the dynamic access behavior characteristics, the data value prediction heat value is obtained; The data stratification identification result is obtained based on the predicted data value and the preset stratification threshold.

[0088] In one feasible implementation, the strategy formulation module 20 is further configured to obtain disk utilization based on real-time monitoring data of the core server disk; The real-time transfer trigger signal is determined based on the disk utilization rate and the preset capacity threshold. An automatic migration task queue is obtained based on the data hierarchical identification results and the preset archiving strategy; A data migration strategy is generated based on the real-time transfer trigger signal and the automatic migration task queue.

[0089] In one feasible implementation, the instruction generation module 30 is further used to extract edge node registration information, hardware parameters, node energy consumption monitoring data and load status from the status information of the edge tiered resource pool; Resource hierarchical configuration information is obtained based on the edge node registration information and the hardware parameters; Based on the node energy consumption monitoring data and the load status, a dynamic energy efficiency assessment result is obtained; A node candidate sequence is obtained based on the resource classification and configuration information and the dynamic energy efficiency assessment results; Based on the data migration strategy and the node candidate sequence, hierarchical data transmission instructions are obtained.

[0090] In one feasible implementation, the concurrent transmission module 40 is further configured to determine the transmission data type and target node attributes according to the data transmission instruction; Generate a transmission path optimization strategy based on the transmitted data type and the target node attributes; Concurrency control parameters are obtained based on network bandwidth characteristics and breakpoint transmission mechanisms; Based on the transmission path optimization strategy and the concurrency control parameters, a verification data transmission stream is obtained; The verification data transmission stream is updated based on hash operations and audit logs to obtain tamper-proof storage data. The tamper-proof storage data is then transmitted concurrently to a remote storage node using multiple threads to obtain remote storage data.

[0091] In one feasible implementation, the data migration module 60 is further used to collect hardware status indicators of remote storage nodes and obtain real-time health data based on the hardware status indicators. Based on the real-time health data, a prediction and assessment are performed to obtain node risk warning information; Determine the data distribution map, and generate a data migration strategy based on the data distribution map and the node risk warning information; Data migration is performed on the remote storage node based on the data migration strategy.

[0092] In one feasible implementation, the energy efficiency management module 70 is further configured to determine data thermal value correction parameters based on user access behavior logs; Energy efficiency management instructions are generated based on node sleep thresholds and wake-up prediction results; The data migration strategy is updated based on the thermal value correction parameters and the energy efficiency management instructions.

[0093] The data storage expansion device provided in this application, employing the data storage expansion method in the above embodiments, can solve the technical problem of low server data migration efficiency. Compared with the prior art, the beneficial effects of the data storage expansion device provided in this application are the same as those of the data storage expansion method provided in the above embodiments, and other technical features in the data storage expansion device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0094] This application provides a data storage expansion device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data storage expansion method in the first embodiment described above.

[0095] The following is for reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing the data storage expansion device in the embodiments of this application. The data storage expansion device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The data storage expansion device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0096] like Figure 4As shown, the data storage expansion device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the data storage expansion device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the data storage expansion device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data storage expansion devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0097] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0098] The data storage expansion device provided in this application, employing the data storage expansion method described in the above embodiments, can solve the technical problem of data storage expansion. Compared with the prior art, the beneficial effects of the data storage expansion device provided in this application are the same as those of the data storage expansion method provided in the above embodiments, and other technical features of this data storage expansion device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0099] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0100] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0101] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the data storage expansion method in the above embodiments.

[0102] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0103] The aforementioned computer-readable storage medium may be included in the data storage expansion device; or it may exist independently and not assembled into the data storage expansion device.

[0104] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the data storage expansion device, enable the data storage expansion device to: obtain data layering identification results based on data generation time and access behavior; A data migration strategy is derived based on the data hierarchical identification results and the core server disk usage. Data transmission instructions are obtained based on the data migration strategy and the status information of the edge tiered resource pool; Based on the data transmission instructions, control multiple threads to concurrently transmit data to the remote storage node to obtain the remote storage data; The localized data access result is obtained based on the file path mapping relationship and the transparent access request.

[0105] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0108] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data storage expansion method, thereby solving the technical problem of data storage expansion. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data storage expansion method provided in the above embodiments, and will not be repeated here.

[0109] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data storage expansion method described above.

[0110] The computer program product provided in this application can solve the technical problem of low data migration efficiency on servers. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data storage expansion method provided in the above embodiments, and will not be repeated here.

[0111] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for expanding data storage capacity, characterized in that, The data storage expansion method includes: Data hierarchical identification results are obtained based on data generation time and access behavior; A data migration strategy is derived based on the data hierarchical identification results and the core server disk usage. Data transmission instructions are obtained based on the data migration strategy and the status information of the edge tiered resource pool; Based on the data transmission instructions, control multiple threads to concurrently transmit data to the remote storage node to obtain the remote storage data; The localized data access result is obtained based on the file path mapping relationship and the transparent access request.

2. The method as described in claim 1, characterized in that, The steps for obtaining data hierarchical identification results based on data generation time and access behavior include: The data generation time of the data file is scanned according to the preset time window to obtain the initial timeliness classification results; By analyzing access behavior in historical access logs, dynamic access behavior characteristics can be obtained. Based on the initial timeliness classification results and the dynamic access behavior characteristics, the data value prediction heat value is obtained; The data stratification identification result is obtained based on the predicted data value and the preset stratification threshold.

3. The method as described in claim 1, characterized in that, The step of obtaining a data migration strategy based on the data hierarchical identification results and the core server disk usage includes: Disk utilization is obtained based on real-time monitoring data of the core server disks; The real-time transfer trigger signal is determined based on the disk usage rate and the preset capacity threshold. An automatic migration task queue is obtained based on the data hierarchical identification results and the preset archiving strategy; A data migration strategy is generated based on the real-time transfer trigger signal and the automatic migration task queue.

4. The method as described in claim 1, characterized in that, The step of obtaining the data transmission instruction based on the data migration strategy and the status information of the edge tiered resource pool includes: Extract edge node registration information, hardware parameters, node energy consumption monitoring data, and load status from the status information of the edge tiered resource pool; Resource hierarchical configuration information is obtained based on the edge node registration information and the hardware parameters; Based on the node energy consumption monitoring data and the load status, a dynamic energy efficiency assessment result is obtained; A node candidate sequence is obtained based on the resource classification and configuration information and the dynamic energy efficiency assessment results; Based on the data migration strategy and the node candidate sequence, hierarchical data transmission instructions are obtained.

5. The method as described in claim 1, characterized in that, The step of controlling multi-threaded concurrent transmission to a remote storage node based on the data transmission command to obtain remote storage data includes: The data type and target node attributes are determined according to the data transmission instructions; Generate a transmission path optimization strategy based on the transmitted data type and the target node attributes; Concurrency control parameters are obtained based on network bandwidth characteristics and breakpoint transmission mechanisms; Based on the transmission path optimization strategy and the concurrency control parameters, a verification data transmission stream is obtained; The verification data transmission stream is updated based on hash operations and audit logs to obtain tamper-proof storage data. The tamper-proof storage data is then transmitted concurrently to a remote storage node using multiple threads to obtain remote storage data.

6. The method as described in claim 1, characterized in that, The data storage expansion method further includes: Collect hardware status indicators of remote storage nodes, and obtain real-time health data based on the hardware status indicators; Based on the real-time health data, a prediction and assessment are performed to obtain node risk warning information; Determine the data distribution map, and generate a data migration strategy based on the data distribution map and the node risk warning information; Data migration is performed on the remote storage node based on the data migration strategy.

7. The method as described in claim 1, characterized in that, The data storage expansion method further includes: Determine the data heatmap correction parameters based on user access behavior logs; Energy efficiency management instructions are generated based on node sleep thresholds and wake-up prediction results; The data migration strategy is updated based on the thermal value correction parameters and the energy efficiency management instructions.

8. A data storage expansion device, characterized in that, The data storage expansion device includes: The hierarchical identification module is used to obtain data hierarchical identification results based on data generation time and access behavior. The strategy formulation module is used to obtain a data migration strategy based on the data hierarchical identification results and the core server disk usage rate. The instruction generation module is used to obtain data transmission instructions based on the data migration strategy and the status information of the edge tiered resource pool; The concurrent transmission module is used to control multi-threaded concurrent transmission to the remote storage node based on the data transmission instructions, so as to obtain the remote storage data; The transparent access module is used to obtain localized data access results based on file path mapping relationships and transparent access requests.

9. A data storage expansion device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data storage expansion method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data storage expansion method as described in any one of claims 1 to 7.

Images