Cold data migration method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides a cold data migration method, device, storage medium and electronic device, wherein the method comprises: when the first space occupancy rate of the first hard disk in the target server is greater than the first threshold, obtaining the attribute information of the target data stored in the first hard disk; determining the target cold data to be migrated included in the target data based on the attribute information; migrating the target cold data to the second hard disk in the target server, wherein the second space occupancy rate of the second hard disk is less than the second threshold. Through the present invention, the problem of unreasonable data query and large-scale storage data allocation existing in the related art is solved, and the effect of reasonable allocation of data query and large-scale storage data is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the communication field, and in particular, to a cold data migration method, device, storage medium and electronic device. Background Art

[0002] In big data scenarios, Elasticsearch is often used as an engine or database for big data query and search because of its near real-time query and distributed storage of large amounts of data. Elasticsearch's performance depends not only on its excellent system design, but also on the underlying storage medium. Usually, Elasticsearch data is stored on SSDs (Solid State Disks), but SSD disks have small capacity and high costs, and users want to store as much data as possible within a certain budget, which is a contradiction. In addition, most data has a certain timeliness. Query and write in large data scenarios often target data within a recent period of time (hot data), while data in other time periods are rarely queried or almost never queried, and the writing pressure is small (cold data). Users want to keep this part of data, but rarely query it. However, with limited storage resources, the real-time data storage space is getting smaller and smaller.

[0003] It can be seen from this that there are problems with unreasonable data query and large-scale storage data allocation in the relevant technology.

[0004] Currently, no effective solution has been proposed for the above-mentioned problems existing in the related technologies. Summary of the invention

[0005] The embodiments of the present invention provide a cold data migration method, device, storage medium and electronic device to at least solve the problems of unreasonable data query and large-scale storage data allocation existing in the related art.

[0006] According to one embodiment of the present invention, a cold data migration method is provided, comprising: when a first space occupancy rate of a first hard disk in a target server is greater than a first threshold value, acquiring attribute information of target data stored in the first hard disk; determining target cold data to be migrated included in the target data based on the attribute information; and migrating the target cold data to a second hard disk in the target server, wherein a second space occupancy rate of the second hard disk is less than a second threshold value.

[0007] According to another embodiment of the present invention, a cold data migration device is provided, comprising: an acquisition module, used to acquire attribute information of target data stored in a first hard disk in a target server when a first space occupancy rate of the first hard disk is greater than a first threshold value; a determination module, used to determine target cold data to be migrated included in the target data based on the attribute information; and a migration module, used to migrate the target cold data to a second hard disk in the target server, wherein a second space occupancy rate of the second hard disk is less than a second threshold value.

[0008] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0009] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0010] Through the present invention, when the first space occupancy rate of the first hard disk in the target server is greater than the first threshold, the attribute information of the target data existing in the first hard disk is obtained, the target cold data to be migrated included in the target data is determined according to the attribute information, and the target cold data is migrated to the second hard disk in the target server. Since when the first space occupancy rate of the first hard disk is greater than the first threshold, the target cold data can be migrated to the second hard disk in the target server whose second space occupancy rate is less than the second threshold, the space occupancy rate of the first hard disk is released, so that the first hard disk can store more other data and can meet the needs of fast data query, therefore, the problem of unreasonable data query and large-scale storage data allocation existing in the related art can be solved, and the effect of reasonable allocation of data query and large-scale storage of data can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a hardware structure block diagram of a mobile terminal of a cold data migration method according to an embodiment of the present invention;

[0012] Figure 2 is a flowchart of a cold data migration method according to an embodiment of the present invention;

[0013] Figure 3 is a diagram of Es node and service relationships according to an exemplary embodiment of the present invention;

[0014] Figure 4 is a flow chart of a cold data migration method according to a specific embodiment of the present invention;

[0015] Figure 5 is a flowchart of Es index migration according to a specific embodiment of the present invention;

[0016] Figure 6 4 is a structural block diagram of a cold data migration device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a cold data migration method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the cold data migration method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0021] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] In this embodiment, a cold data migration method is provided. Figure 2 is a flow chart of a cold data migration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0023] Step S202, when a first space occupancy rate of a first hard disk in the target server is greater than a first threshold, acquiring attribute information of target data stored in the first hard disk;

[0024] Step S204, determining target cold data to be migrated included in the target data based on the attribute information;

[0025] Step S206: Migrate the target cold data to a second hard disk in the target server, wherein a second space occupancy rate of the second hard disk is less than a second threshold.

[0026] In the above embodiment, the first hard disk can be an SSD solid state disk, the second hard disk can be a SATA hard disk, and the target server can adopt a mixed mode of SSD and SATA. SSD has high cost and fast read and write speed, and can store hot data; SATA has low cost and slow read and write speed, and can store cold data. Mixing SSD and SATA on the same server can realize data query and reasonable allocation of large-scale storage at a certain cost. In the target server, the capacity of SSD can be 1.5T, the capacity of SATA can be 2T, the memory can be 125G, and the CPU can be 32 cores. It should be noted that the capacity of SSD, SATA and memory can also be other capacities, and the CPU can also be 64 cores, etc., and the present invention does not limit this.

[0027] In the above embodiment, the second hard disk may be one or more, for example, multiple SATA hard disks may be configured in the target server. When there are multiple second hard disks, a target second hard disk may be selected from the second hard disks to store the target cold data to be migrated, and when determining the target second hard disk, the hard disk with the smallest space occupancy rate among the second hard disks may be determined as the target second hard disk, or any hard disk included in the second hard disk whose space occupancy rate is less than the second threshold hard disk may be determined as the target second hard disk.

[0028] In the above embodiment, the attribute information may be information created when storing data, and the index of the target data may be created according to the type of data. For example, the index is created based on whether the data is time-sensitive. An index without timeliness is only created, and an index with timeliness is created according to a fixed time period. The attribute information of the index may be suffixed with time (for example: xxxx-2021.07). The attributes in the index can be added with index.routing.allocation.include.type:ssd, so that the newly created indexes are placed on the SSD, and the imported time-sensitive data are initially on the SSD to ensure the performance of data queries.

[0029] In the above embodiment, the first threshold value can be 80%, and the second threshold value can be 85%. When the space of the second hard disk is insufficient, migration is not performed. When the space occupancy rate of the first hard disk exceeds the first threshold value, data migration is performed. During the data migration process, the first space occupancy rate of the first hard disk can be detected in real time. When the first space occupancy rate is less than the fourth threshold value, migration can be stopped. Among them, the fourth threshold value can be 75%. It should be noted that the above first threshold value, the second threshold value and the fourth threshold value are only exemplary descriptions. The above threshold values ​​all support configuration file modification, that is, users can customize and modify the above threshold values. The present invention does not limit the above threshold values.

[0030] Optionally, the execution subject of the above steps can be a background processor, or other devices with similar processing capabilities, or a machine that at least integrates a data processing device, wherein the data processing device may include terminals such as computers and mobile phones, but is not limited to this.

[0031] Through the present invention, when the first space occupancy rate of the first hard disk in the target server is greater than the first threshold, the attribute information of the target data existing in the first hard disk is obtained, the target cold data to be migrated included in the target data is determined according to the attribute information, and the target cold data is migrated to the second hard disk in the target server. Since when the first space occupancy rate of the first hard disk is greater than the first threshold, the target cold data can be migrated to the second hard disk in the target server whose second space occupancy rate is less than the second threshold, the space occupancy rate of the first hard disk is released, so that the first hard disk can store more other data and can meet the needs of fast data query, therefore, the problem of unreasonable data query and large-scale storage data allocation existing in the related art can be solved, and the effect of reasonable allocation of data query and large-scale storage of data can be achieved.

[0032] In an exemplary embodiment, migrating the target cold data to the second hard disk in the target server includes: when the cluster where the target cold data is located is in a normal working state and the cluster includes cold nodes, determining the target number of nodes included in the cluster; determining the target migration rule of the target cold data based on the target number; migrating the target cold data based on the target migration rule. In this embodiment, the characteristic of the optimal 31G heap memory of Elasticsearch can be utilized to deploy 2 Es nodes on the target server, one hot data node and one cold data node; the data directory is placed on SSD and SATA respectively; the hot and cold nodes are distinguished by the node name (nade.name) and the node attribute (node.attr.type); the cold node and the hot node belong to the same cluster. When the cluster where the target cold data is located is in a healthy state, that is, a normal working state, and the cluster includes cold nodes, the target number of nodes included in the cluster can be determined, the target migration rule can be determined according to the target number, and the target cold data can be migrated based on the target migration rule. Among them, the relationship diagram of Es nodes and services can be found in the attached. Figure 3 .

[0033] In an exemplary embodiment, determining the target migration rule of the target cold data based on the target quantity includes: determining the target migration rule corresponding to the target quantity from the correspondence between the pre-configured quantity and the migration rule; wherein the correspondence includes: when the quantity is greater than the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the first attribute, and setting the attribute of the target index of the cold data to the second attribute; when the quantity is less than or equal to the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the third attribute, and setting the attribute of the target index of the cold data to the second attribute. In this embodiment, the data will first be stored in the first hard disk. When the data scale in the first hard disk accumulates to a certain amount, the cluster SSD usage space is gradually filled, and the early data is rarely or not queried, and this batch of data becomes cold data. The hot and cold migration management program regularly detects the disk size. When it reaches a certain threshold, it starts to perform hot and cold migration, and the cold data is gradually migrated to the cold node, and the SSD disk space is released to ensure that the new time-sensitive data is normally imported and queried. The target migration rule of the target cold data can be determined according to the target number of nodes included in the cluster where the cold node is located. That is, the target migration rule corresponding to the target number can be determined according to the correspondence between the number of nodes and the migration rule. When the number of nodes is greater than the third threshold, the migration rule may be to update the attribute of the cold node to the first attribute, and to set the attribute of the target index of the cold data to the second attribute. Among them, the first attribute may be rack, and the second attribute may be sata. That is, update the cluster.routing.allocation.awareness.attributes attribute to rack, and update the index.routing.allocation.include.type attribute to sata. When the number of nodes is less than or equal to the third threshold, update the attribute of the cold node to the third attribute, and set the attribute of the target index of the cold data to the second attribute. That is, update the cluster.routing.allocation.same_shard.host attribute to false, and update the index.routing.allocation.include.type attribute to sata. Among them, the third threshold can be 4, that is, when determining the target migration rule, it can be first determined whether the number of nodes in the cluster exceeds 4. When it exceeds 4, the cluster.routing.allocation.awareness.attributes attribute is updated to rack; when it does not exceed 4, the cluster.routing.allocation.same_shard.host attribute is updated to false.It should be noted that the third threshold is only an exemplary description, and the third threshold can also be set to 2, 3, 5, 6, etc., which is not limited in the present invention. When the third threshold is 4, different migration methods are adopted according to the number of nodes in the cluster, and the methods executed by clusters with less than 4 nodes and more than 4 nodes are different; the migration rules can be shown in Table 1:.

[0034] Table 1

[0035]

[0036] In an exemplary embodiment, migrating the target cold data based on the target migration rule includes: reducing the index concurrent recovery number of the cluster to adjust; when the index concurrent recovery number is reduced, migrating the target index of the target cold data based on the target migration rule. In this embodiment, when migrating the target cold data, the cluster's index concurrent recovery number can be reduced, and after reducing the cluster's index concurrent recovery number, the target cold data is migrated. For example, the index concurrent recovery number (cluster.routing.allocation.node_concurrent_recoveries) is reduced to a predetermined number. Reducing the cluster's index concurrent recovery number can reduce SATA's disk io. Among them, the predetermined number can be 2 (this number is only an exemplary description, the predetermined number can also be 1, 3, etc., and the present invention is not limited to this). After the data migration is completed, the index concurrent recovery number can be restored to the default value, such as 24.

[0037] In an exemplary embodiment, determining the target cold data to be migrated included in the target data based on the attribute information includes: determining that the attribute information included in the target data is first data of a fourth attribute; determining the target cold data from the first data based on time information included in the fourth attribute, wherein the time information indicates the time when the first data is stored. In this embodiment, when determining the target cold data, it can be determined that the attribute information included in the target data is a first quantity of the fourth attribute, wherein the fourth attribute can be an attribute including ssd. The target cold data is determined from the first data based on the time information included in the fourth attribute. The time information can be time information for creating a data index.

[0038] In an exemplary embodiment, determining the target cold data from the first data based on the time information included in the fourth attribute includes: determining the creation time of the first data based on the time information included in the fourth attribute; and determining the data whose time difference between the creation time included in the first data and the current time is greater than a predetermined threshold as the target cold data. In this embodiment, when determining the target cold data, the creation time of the first data can be determined based on the time information included in the fourth attribute, and the data whose time difference between the creation time and the current time is greater than a predetermined threshold is determined as the target cold data. The predetermined threshold may be 1 week (or other thresholds, such as 5 days, 10 days, etc.). That is, the time stored first can be migrated to the second hard disk first in chronological order.

[0039] In an exemplary embodiment, the method further includes: determining the data type of the acquired data; when the data type indicates that the data is time-sensitive data, configuring the attribute information of the data to include time information and attribute information of a thermal data type; when the data type indicates that the data is non-time-sensitive data, configuring the attribute information of the data to include attribute information of a static data type and a thermal data type. In this embodiment, after the data is acquired, the data type of the acquired data can be determined, and when the data is time-sensitive data, the attribute information of the data can be configured to include attribute information of a time information and a thermal data type, and when the data is non-time-sensitive data, the attribute information of the data can be configured to include attribute information of a static data type and a thermal data type. Among them, the thermal data type can be represented as .ssd. When .ssd is included in the attribute information, it means that the data is stored in the SSD.

[0040] The following describes the cold data migration method in conjunction with a specific implementation method:

[0041] Figure 4 is a flow chart of a cold data migration method according to a specific embodiment of the present invention. Figure 4 As shown, the process includes:

[0042] Step S402, obtain cluster disk information. At the beginning of the process, the program will check the Es client connection status. If the connection fails, it will be checked every 10 minutes (configurable). If the connection fails after a timeout of 3 hours (configurable), the migration operation will be abandoned. If the connection is successful, continue to proceed.

[0043] Step S404, determine whether cold data nodes are included. If the result is yes, execute step S406, if not, execute step S412. Before each migration operation, check whether the cluster contains cold data nodes to prevent the failure of hot and cold migration due to changes in the on-site cluster environment or improper operation. When the historical version without cold disks is upgraded to the version with cold disks, after the console is started, the historical index will be added with the index.routing.allocation.include.type attribute, the value is: ssd, and the index shards will be distributed on the hot data nodes.

[0044] Step S406, obtaining all indexes.

[0045] Step S408, detecting the cold disk space.

[0046] Step S410, determine whether the cold disk space (corresponding to the second hard disk) exceeds the threshold. If the judgment result is yes, execute step S412, if the judgment result is no, execute step S414. The cold disk space threshold (corresponding to the second threshold) is 85%. When the cold disk space is insufficient, migration is not performed;

[0047] Step S412, end the task.

[0048] Step S414, detecting the hot disk space.

[0049] Step S416, determine whether the hot disk space (corresponding to the first hard disk) exceeds the threshold. If the judgment result is yes, execute step S418, if the judgment result is no, execute step S412. The hot disk space threshold (corresponding to the first threshold) is 80%. Data migration is performed only when the hot disk space exceeds the threshold. The hot disk space threshold supports configuration file modification.

[0050] Step S418, Es cold-hot migration.

[0051] Step S420, determine whether the execution is completed. If the judgment result is yes, execute step S422, if the judgment result is no, execute step S418.

[0052] Step S422, set a target threshold (corresponding to the fourth threshold mentioned above). The default value of the target threshold is 75%, and configuration file modification is supported. The hot disk space will continue to migrate below the target threshold.

[0053] Figure 5 is a flowchart of Es index migration according to a specific embodiment of the present invention. Figure 5As shown, the program can select the earliest index in the time series index each time, and can judge according to the time suffix in the index name (xxxx-2021.01). When the cluster contains multiple indexes with the same suffix, the index to be migrated can be determined based on the prefix. Determine whether the cluster status is healthy. If it is healthy, reduce the number of concurrent recoveries of the index (cluster.routing.allocation.node_concurrent_recoveries) to 2 to reduce the disk io of SATA. Different migration methods are adopted according to the number of nodes in the cluster. The methods for clusters with less than 4 nodes and more than 4 nodes are different. For specific methods, please refer to Table 1. After the method is executed, the index is in the migration state, and the program needs to monitor the migration status regularly. After the index migration is completed, the number of concurrent recoveries will be restored.

[0054] In the above embodiment, by mounting SSD and SATA disk on the same server, the hot and cold nodes are deployed on different storage media of the same server during cluster deployment, so as to achieve the separation of hot and cold data on the storage media; and meet the customer's demand for storing as much data as possible within a certain budget cost. By creating the time-sensitive data of the same time range in the same index, the management of hot and cold data is achieved, so that most cold data can be separated from hot data.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0056] In this embodiment, a cold data migration device is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0057] Figure 6 is a structural block diagram of a cold data migration device according to an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0058] An acquisition module 62, configured to acquire attribute information of target data stored in a first hard disk in a target server when a first space occupancy rate of the first hard disk is greater than a first threshold;

[0059] A determination module 64, configured to determine target cold data to be migrated included in the target data based on the attribute information;

[0060] The migration module 66 is configured to migrate the target cold data to a second hard disk in the target server, wherein a second space occupancy rate of the second hard disk is less than a second threshold.

[0061] In an exemplary embodiment, the migration module 66 can migrate the target cold data to the second hard disk in the target server in the following manner: when the cluster where the target cold data is located is in normal working state and the cluster includes cold nodes, determine the target number of nodes included in the cluster; determine the target migration rule of the target cold data based on the target number; and migrate the target cold data based on the target migration rule.

[0062] In an exemplary embodiment, the migration module 66 can implement the target migration rule of the target cold data based on the target quantity in the following manner: determine the target migration rule corresponding to the target quantity from the correspondence between the pre-configured quantity and the migration rule; wherein the correspondence includes: when the quantity is greater than a third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the first attribute, and setting the attribute of the target index of the cold data to the second attribute; when the quantity is less than or equal to the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the third attribute, and setting the attribute of the target index of the cold data to the second attribute.

[0063] In an exemplary embodiment, the migration module 66 can migrate the target cold data based on the target migration rule in the following manner: reducing the number of concurrent recovery of the index of the cluster; and migrating the target index of the target cold data based on the target migration rule when the number of concurrent recovery of the index is reduced.

[0064] In an exemplary embodiment, the determination module 64 can determine the target cold data to be migrated included in the target data based on the attribute information in the following manner: determine that the attribute information included in the target data is first data of a fourth attribute; determine the target cold data from the first data based on the time information included in the fourth attribute, wherein the time information indicates the time when the first data is stored.

[0065] In an exemplary embodiment, the determination module 64 can determine the target cold data from the first data based on the time information included in the fourth attribute in the following manner: determine the creation time of the first data based on the time information included in the fourth attribute; and determine the data whose time difference between the creation time included in the first data and the current time is greater than a predetermined threshold as the target cold data.

[0066] In an exemplary embodiment, the device can be used to determine the data type of the acquired data; when the data type indicates that the data is time-sensitive data, the attribute information of the data is configured to include time information and attribute information of a thermal data type; when the data type indicates that the data is non-time-sensitive data, the attribute information of the data is configured to include attribute information of a static data type and a thermal data type.

[0067] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0068] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0069] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0070] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0071] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0072] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0073] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cold data migration method, characterized in that: include: When a first space occupancy rate of a first hard disk in the target server is greater than a first threshold, acquiring attribute information of target data stored in the first hard disk; Determining target cold data to be migrated included in the target data based on the attribute information; Migrating the target cold data to a second hard disk in the target server, wherein a second space occupancy rate of the second hard disk is less than a second threshold; Migrating the target cold data to the second hard disk in the target server includes: determining a target number of nodes included in the cluster when the cluster where the target cold data is located is in a normal working state and the cluster includes cold nodes; determining a target migration rule for the target cold data based on the target number; and migrating the target cold data based on the target migration rule; Determining the target migration rule of the target cold data based on the target quantity includes: determining the target migration rule corresponding to the target quantity from the correspondence between the pre-configured quantity and the migration rule; wherein the correspondence includes: when the quantity is greater than a third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the first attribute, and setting the attribute of the target index of the cold data to the second attribute; when the quantity is less than or equal to the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the third attribute, and setting the attribute of the target index of the cold data to the second attribute; The first attribute is rack; and the third attribute is false.

2. The method according to claim 1, characterized in that Migrating the target cold data based on the target migration rule includes: Reduce the number of concurrent index restores for the cluster; When the number of concurrent recovery of the index is reduced, the target index of the target cold data is migrated based on the target migration rule.

3. The method according to claim 1, characterized in that Determining the target cold data to be migrated included in the target data based on the attribute information includes: Determining that the attribute information included in the target data is first data of a fourth attribute; The target cold data is determined from the first data based on time information included in the fourth attribute, wherein the time information indicates a time when the first data is stored.

4. The method according to claim 3, characterized in that Determining the target cold data from the first data based on the time information included in the fourth attribute includes: determining a creation time of the first data based on time information included in the fourth attribute; The data in the first data, the time difference between the creation time and the current time being greater than a predetermined threshold, is determined as the target cold data.

5. The method according to claim 1, characterized in that The method further comprises: Determine the data type of the acquired data; In the case where the data type indicates that the data is time-sensitive data, configuring the attribute information of the data to include time information and attribute information of a hot data type; In a case where the data type indicates that the data is non-time-sensitive data, the attribute information of the data is configured to include attribute information of a static data type and a hot data type.

6. A cold data migration device, characterized in that: include: an acquisition module, configured to acquire attribute information of target data stored in a first hard disk in a target server when a first space occupancy rate of the first hard disk is greater than a first threshold; A determination module, configured to determine target cold data to be migrated included in the target data based on the attribute information; A migration module, configured to migrate the target cold data to a second hard disk in the target server, wherein a second space occupancy rate of the second hard disk is less than a second threshold; The migration module implements the migration of the target cold data to the second hard disk in the target server in the following manner: when the cluster where the target cold data is located is in a normal working state and the cluster includes cold nodes, determining the target number of nodes included in the cluster; determining the target migration rule of the target cold data based on the target number; and migrating the target cold data based on the target migration rule; The device determines the target migration rule of the target cold data based on the target quantity in the following manner: determining the target migration rule corresponding to the target quantity from the correspondence between the pre-configured quantity and the migration rule; wherein the correspondence includes: when the quantity is greater than a third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the first attribute, and setting the attribute of the target index of the cold data to the second attribute; when the quantity is less than or equal to the third threshold, the corresponding migration rule includes: updating the attribute of the cold node to the third attribute, and setting the attribute of the target index of the cold data to the second attribute; The first attribute is rack; and the third attribute is false.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 5 when executed by a processor.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.

Images