Multi-node Load Balancing Cold and Hot Data Migration Device, Method, Terminal and Medium

Abstract

The present invention relates to the field of data migration, and specifically discloses a multi-node load balancing hot and cold data migration device, method, terminal and medium. The data collection module collects the IO request access information of each data block in each node; the data value calculation module calculates the data value of the data block according to the IO request access information; the hot and cold data identification module divides the data value range, and determines the nodes to be migrated out and the nodes to be migrated in corresponding to each data value range according to the distribution of the data blocks on each node in the data value range; the migration module migrates the data blocks on the nodes to be migrated out to the nodes to be migrated in. The present invention determines the data value of the data block, divides the data value range, appropriately migrates the stored file according to the distribution range of the data value, and by finding the storage nodes with relatively small load pressure, migrates some data to these storage nodes with relatively small load pressure, so as to achieve the load balance of the system, thereby improving the stability of the system.

Description

Technical Field

[0001] The present invention relates to the field of data migration, and particularly to a multi-node load balancing cold and hot data migration device, method, terminal and medium. Background Art

[0002] In the big data era, the storage and transmission rates of data play an important role in the normal and fast operation of storage servers. For a multi-node storage system, the resource usage of data storage nodes will seriously affect its performance. Usually, the sudden access and concurrent access of users will make the load of the nodes storing these data heavier. Therefore, it is difficult for the nodes with heavy load to bear the sudden access pressure, which brings serious access delay and network bandwidth problems to the whole system. Summary of the Invention

[0003] To solve the above problems, the present invention provides a multi-node load balancing cold and hot data migration device, method, terminal and medium, which determine the data value of data blocks, divide the data value intervals, appropriately migrate the stored files according to the distribution intervals of the data values, and find storage nodes with less load pressure, and migrate some data to these storage nodes with less load pressure to achieve load balancing of the system, thereby improving the stability of the system.

[0004] In a first aspect, the technical solution of the present invention provides a multi-node load balancing cold and hot data migration device, including:

[0005] A data collection module: collecting IO request access information of each data block in each node;

[0006] A data value calculation module: calculating the data value of the data block according to the IO request access information;

[0007] A cold and hot data identification module: dividing the data value intervals, and determining the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of the data blocks on each node in the data value intervals;

[0008] A migration module: migrating the data blocks on the nodes to be migrated out to the nodes to be migrated in.

[0009] Further, the data block IO access request information collected by the data collection module includes the total access frequency, random read / write access frequency, sequential read / write access frequency, the last access time of the file, the last access type, and the end address of the last access;

[0010] The data value calculation module calculates the data value of the data block according to the IO request access information, specifically as follows:

[0011] Assign weights to each access information factor and perform weighted summation to obtain the data value of the data block.

[0012] Further, the hot and cold data identification module is specifically used for:

[0013] Divide into N data value intervals;

[0014] For each node, calculate the percentage of data blocks in each data value interval;

[0015] For each data value interval, compare the node with the highest percentage of data blocks with the node with the lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the highest percentage of data blocks is recorded as the node to be migrated out, and the node with the lowest percentage of data blocks is recorded as the node to be migrated in; otherwise, no migration is performed. Then compare the node with the second-highest percentage of data blocks with the node with the second-lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the second-highest percentage of data blocks is recorded as the node to be migrated out, and the node with the second-lowest percentage of data blocks is recorded as the node to be migrated in; otherwise, no migration is performed. And so on.

[0016] Further, the device further includes,

[0017] Metadata module: Record the physical storage location of each data block file, including the storage node, storage device type, and specific storage location in the storage device;

[0018] Migration plan generation module: Set data migration rules, including migration method, migration period, migration time, and wear leveling mechanism;

[0019] The migration module is specifically used for:

[0020] Based on the nodes to be migrated out and the nodes to be migrated in determined by the hot and cold data identification module, and based on the data migration rules set by the migration calculation module, migrate the data blocks; and after the migration is completed, notify the metadata module to update the metadata information.

[0021] In a second aspect, the technical solution of the present invention provides a multi-node load balancing hot and cold data migration method, including the following steps:

[0022] S1, Collect the IO request access information of each data block in each node;

[0023] S2, Calculate the data value of the data block according to the IO request access information;

[0024] S3, Divide the data value intervals, and determine the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of the data blocks on each node in the data value intervals;

[0025] S4. Migrate the data blocks on the node to be migrated out to the node to be migrated in.

[0026] Further, the data block I / O access request information collected in step S1 includes the total access frequency, random read / write access frequency, sequential read / write access frequency, the last access time of the file, the last access type, and the end address of the last access.

[0027] Step S2 is specifically as follows:

[0028] Assign weights to each access information factor and perform weighted summation to obtain the data value of the data block.

[0029] Further, step S3 specifically includes:

[0030] Divide into N data value intervals;

[0031] For each node, calculate the percentage of data blocks in each data value interval;

[0032] For each data value interval, compare the node with the highest percentage of data blocks with the node with the lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the highest percentage of data blocks is recorded as the node to be migrated out, and the node with the lowest percentage of data blocks is recorded as the node to be migrated in; otherwise, no migration is performed. Then compare the node with the second-highest percentage of data blocks with the node with the second-lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the second-highest percentage of data blocks is recorded as the node to be migrated out, and the node with the second-lowest percentage of data blocks is recorded as the node to be migrated in; otherwise, no migration is performed. And so on.

[0033] Further, the method further includes,

[0034] S0. Record the physical storage location of each data block, including the storage node, the storage device type, and the specific storage location in the storage device; set the data migration rules, including the migration method, migration period, migration time, and wear leveling mechanism;

[0035] Step S4 specifically includes:

[0036] According to the determined node to be migrated out and the node to be migrated in, based on the set data migration rules, migrate the data blocks; and after the migration is completed, notify the metadata module to update the metadata information.

[0037] In a third aspect, the technical solution of the present invention provides a terminal, including:

[0038] A memory for storing a multi-node load balancing hot and cold data migration program;

[0039] A processor for implementing the steps of the multi-node load balancing hot and cold data migration method as described in any one of the above when executing the multi-node load balancing hot and cold data migration program.

[0040] In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium, on which a multi-node load balancing hot and cold data migration program is stored. When the multi-node load balancing hot and cold data migration program is executed by a processor, the steps of the multi-node load balancing hot and cold data migration method as described in any one of the above are implemented.

[0041] The multi-node load balancing hot and cold data migration device, method, terminal and medium provided by the present invention have the following beneficial effects compared with the prior art: determining the data value of data blocks, dividing the data value intervals, appropriately migrating the stored files according to the distribution intervals of the data values, finding storage nodes with less load pressure, and migrating some data to these storage nodes with less load pressure, which can greatly balance the access pressure between nodes, reduce the latency problem caused by uneven load of node resources, and save the user's search time; at the same time, due to the balanced use of resources between nodes, the latency and slow response problems caused by sudden access or concurrent access of a single node within a short period of time are reduced, the hot data is separated to each working node, the bandwidth between nodes is balanced, the IOPS and bandwidth of the storage system are improved, and finally the user experience and the effect of IOPS are improved. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is a schematic block diagram of a multi-node load balancing hot and cold data migration device provided in Embodiment 1 of the present invention.

[0044] Figure 2 It is a schematic block diagram of a multi-node load balancing hot and cold data migration device provided in Embodiment 2 of the present invention.

[0045] Figure 3 It is a schematic flow chart of a multi-node load balancing hot and cold data migration method provided in Embodiment 3 of the present invention.

[0046] Figure 4 It is a schematic flow chart of a multi-node load balancing hot and cold data migration method provided in Embodiment 4 of the present invention.

[0047] Figure 5 It is a schematic structural diagram of a terminal provided in Embodiment 5 of the present invention. Detailed implementation manners

[0048] In order to enable those skilled in the art to better understand the solution of this application, the following further elaborates on this application in conjunction with the accompanying drawings and specific implementation manners. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope protected by this application.

[0049] Embodiment 1

[0050] Figure 1 It is a schematic block diagram of a multi-node load balancing hot and cold data migration device provided in Embodiment 1 of the present invention, including a data collection module, a data value calculation module, a hot and cold data identification module, and a migration module.

[0051] Data collection module: Collect the IO request access information of each data block in each node.

[0052] It should be noted that before the user issues an IO to write to the disk, the data in the disk is divided or aggregated into data blocks of the same granularity size. The data collection module collects the IO request access information of the data blocks to provide information for calculating the data value of the data blocks subsequently.

[0053] Data value calculation module: Calculate the data value of the data block according to the IO request access information.

[0054] The data value is the hot and cold degree of the data block. The data value module calculates the data value of the data block according to the IO request access information of the data block, that is, evaluates the hot and cold value of the data block.

[0055] Hot and cold data identification module: Divide the data value interval, and determine the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of the data blocks on each node in the data value interval.

[0056] Divide the data value interval according to actual needs. For example, divide it into 10 intervals. Then each data block will be distributed in the corresponding data value interval. For each node, the distribution of the data blocks in the data value interval will be obtained. Then, for each data value interval, determine the nodes to be migrated out and the nodes to be migrated in, so that the data in each data value interval is distributed as evenly as possible among the nodes.

[0057] Migration module: Migrate the data blocks on the nodes to be migrated out to the nodes to be migrated in.

[0058] The migration module is used to perform the migration of data blocks and execute the migration according to the determined nodes to be migrated out and the nodes to be migrated in.

[0059] A multi-node load balancing hot and cold data migration device provided in Embodiment 1 of the present invention determines the data value of data blocks, divides the data value intervals, and appropriately migrates the storage files according to the distribution intervals of the data values. By finding storage nodes with less load pressure and migrating some data to these storage nodes with less load pressure, it can greatly balance the access pressure between nodes, reduce the latency problems caused by uneven load of node resources, and save the user's search time; at the same time, because the resource usage between nodes is balanced, it reduces the latency and slow response problems caused by sudden access or concurrent access of a single node within a short period of time, separates the hot data to each working node, balances the bandwidth between nodes, improves the IOPS and bandwidth of the storage system, and ultimately improves the user experience and the effect of IOPS.

[0060] On the basis of the above embodiment, as a preferred implementation, the data block IO access request information collected by the data collection module includes the total access frequency, random read / write access frequency, sequential read / write access frequency, the last access time of the file, the last access type, and the end address of the last access.

[0061] It should be noted that the user can also set other access information according to needs, and the specific selection of which accesses does not affect the implementation of this embodiment.

[0062] Correspondingly, the data value calculation module calculates the data value of the data block according to the IO request access information, specifically: assigns weights to each access information factor and performs weighted summation to obtain the data value of the data block.

[0063] It should be noted that weights can be assigned to each access new factor according to user experience, and finally the data value of the file is calculated by weighted summation.

[0064] On the basis of the above embodiment, as a preferred implementation, the hot and cold data identification module determines the nodes to be migrated out and the nodes to be migrated in for each data value interval through the following steps.

[0065] Step 1, divide N data value intervals;

[0066] Step 2, for each node, calculate the percentage of data blocks in each data value interval;

[0067] Step 3: For each data value range, compare the node with the highest data block percentage with the node with the lowest data block percentage. If the difference between the two data block percentages exceeds the threshold percentage, the node with the highest data block percentage is marked as the node to be migrated out, and the node with the lowest data block percentage is marked as the node to be migrated in; otherwise, no migration is performed. Then, compare the node with the second-highest data block percentage with the node with the second-lowest data block percentage. If the difference between the two data block percentages exceeds the threshold percentage, the node with the second-highest data block percentage is marked as the node to be migrated out, and the node with the second-lowest data block percentage is marked as the node to be migrated in; otherwise, no migration is performed. And so on.

[0068] It can be understood that a total of N data value ranges are divided, and it is necessary to determine whether migration is required for each data value range. If migration is required, the node to be migrated out and the node to be migrated in are determined. For example, if 10 data value ranges are divided and there are 10 nodes in total, for the tenth data value range (the range with a data value of 9 - 10), the first node has the highest data block distribution percentage in this data value range, which is 10%, and the fourth node has the lowest data block distribution percentage in this data value range, which is 1%. Then the difference between the two exceeds 5%, so it is necessary to migrate the data of the first node in this data value range to the fourth node. For this data value range, the first node is marked as the node to be migrated out, and the fourth node is marked as the node to be migrated in.

[0069] Embodiment 2

[0070] Figure 2 FIG. is a schematic block diagram of a multi-node load balancing hot and cold data migration device provided by Embodiment 2 of the present invention, including a metadata module, a migration plan generation module, a data collection module, a data value calculation module, a hot and cold data identification module, and a migration module.

[0071] Metadata module: Records the physical storage location of each data block, including the storage node, the type of storage device, and the specific storage location in the storage device.

[0072] The metadata module records the physical storage location of a data block's file, determines on which node, which type of storage device (HDD / SSD), and the specific storage location in the storage device the file is stored. When a new I / O request arrives, the I / O request and the file access information in the mapping table are used to update or find the data information of the storage block on the disk.

[0073] Migration plan generation module: Sets data migration rules, including the migration method, migration period, migration time, and wear leveling mechanism.

[0074] It includes setting requirements for user setting migration, whether it is automatic or manual migration. If it is automatic migration, set the migration period and migration time of the system, usually at 0:00 every day. Set relevant requirements for migration, such as the requirement to achieve load migration balance on which nodes. At the same time, a wear leveling mechanism is added. When selecting migration blocks, try to select data blocks with fewer recent migration times within the same heat range. Send the metadata information of the final migrated data blocks to the data migration module, and notify the metadata module to update the block physical address information after migration is completed.

[0075] Data collection module: Collect the IO request access information of each data block in each node.

[0076] It should be noted that before the user issues an IO to the disk, the data in the disk is divided or aggregated into data blocks of the same granularity size. The data collection module collects the IO request access information of the data blocks, providing information for calculating the data value of the data blocks subsequently.

[0077] Data value calculation module: Calculate the data value of the data blocks according to the IO request access information.

[0078] The data value is the hot and cold degree of the data block. The data value module calculates the data value of the data block according to the IO request access information of the data block, that is, evaluates the hot and cold value of the data block.

[0079] Hot and cold data identification module: Divide the data value range, and determine the nodes to be migrated out and the nodes to be migrated in corresponding to each data value range according to the distribution of data blocks in the data value range on each node.

[0080] Divide the data value range according to actual needs, for example, divide it into 10 divisions. Then each data block will be distributed in the corresponding data value range. For each node, the distribution of data blocks in the data value range will be obtained. Then, for each data value range, determine the nodes to be migrated out and the nodes to be migrated in, so that the data in each data value range is distributed as evenly as possible among the nodes.

[0081] Migration module: Migrate the data blocks on the nodes to be migrated out to the nodes to be migrated in.

[0082] The migration module is used to execute the migration of data blocks and perform the migration according to the determined nodes to be migrated out and the nodes to be migrated in. At the same time, the migration module migrates the data blocks based on the data migration rules set by the generated migration calculation module; and after the migration is completed, it notifies the metadata module to update the metadata information.

[0083] A multi-node load balancing hot and cold data migration device provided in the second embodiment of the present invention determines the data value of data blocks, divides the data value intervals, and appropriately migrates the stored files according to the distribution intervals of the data values. By finding storage nodes with less load pressure, some data is migrated to these storage nodes with less load pressure, which can balance the access pressure between nodes to a large extent, reduce the latency problems caused by uneven load of node resources, and save the user's search time. At the same time, due to the balanced use of resources between nodes, the latency and slow response problems caused by sudden access or concurrent access of a single node within a short period of time are reduced. The hot data is separated to each working node, achieving balanced bandwidth between nodes, improving the IOPS and bandwidth of the storage system, and ultimately improving the user experience and the effect of IOPS.

[0084] Embodiment Three

[0085] In the above Embodiment One, the embodiment of a multi-node load balancing hot and cold data migration device is described in detail. Based on the multi-node load balancing hot and cold data migration device described in the above embodiment, the present invention embodiment also provides a multi-node load balancing hot and cold data migration method corresponding to this device.

[0086] Figure 3 It is a schematic flow chart of a multi-node load balancing hot and cold data migration method provided in the third embodiment of the present invention, including the following steps.

[0087] S1. Collect the IO request access information of each data block in each node.

[0088] The collected data block IO access request information includes the total access frequency, random read / write access frequency, sequential read / write access frequency, the last access time of the file, the last access type, and the end address of the last access.

[0089] S2. Calculate the data value of the data block according to the IO request access information.

[0090] Assign weights to each access information factor and perform weighted summation to obtain the data value of the data block.

[0091] S3. Divide the data value intervals, and determine the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of data blocks in each data value interval on each node, specifically including:

[0092] S31. Divide N data value intervals;

[0093] S32. For each node, calculate the percentage of data blocks in each data value interval;

[0094] S33. For each data value range, compare the node with the highest data block percentage with the node with the lowest data block percentage. If the difference between the two data block percentages exceeds the threshold percentage, the node with the highest data block percentage is marked as the node to be migrated out, and the node with the lowest data block percentage is marked as the node to be migrated in; otherwise, no migration is performed. Then, compare the node with the second-highest data block percentage with the node with the second-lowest data block percentage. If the difference between the two data block percentages exceeds the threshold percentage, the node with the second-highest data block percentage is marked as the node to be migrated out, and the node with the second-lowest data block percentage is marked as the node to be migrated in; otherwise, no migration is performed. And so on.

[0095] S4. Migrate the data blocks on the node to be migrated out to the node to be migrated in.

[0096] The multi-node load balancing cold and hot data migration method of this embodiment is implemented based on the aforementioned multi-node load balancing cold and hot data migration device. Therefore, the specific implementation in this method can be seen in the embodiment part of the multi-node load balancing cold and hot data migration device in the previous text. Therefore, its specific implementation can refer to the descriptions of the corresponding various part embodiments and will not be elaborated here.

[0097] In addition, since the multi-node load balancing cold and hot data migration method of this embodiment is implemented based on the aforementioned multi-node load balancing cold and hot data migration device, its function corresponds to that of the above device and will not be repeated here.

[0098] Embodiment Four

[0099] In the above Embodiment One, the embodiment of a multi-node load balancing cold and hot data migration device is described in detail. Based on the multi-node load balancing cold and hot data migration device described in the above embodiment, the embodiment of the present invention also provides a multi-node load balancing cold and hot data migration method corresponding to this device.

[0100] Figure 4 It is a schematic flowchart of a multi-node load balancing cold and hot data migration method provided by Embodiment Four of the present invention, including the following steps.

[0101] S0. Record the physical storage location of each data block file, including the storage node, storage device type, and specific storage location in the storage device; set the data migration rules, including the migration method, migration period, migration time, and wear leveling mechanism.

[0102] S1. Collect the IO request access information of each data block in each node.

[0103] The collected data block IO access request information includes the total access frequency, random read / write access frequency, sequential read / write access frequency, the time of the last access to the file, the type of the last access, and the end address of the last access.

[0104] S2. Calculate the data value of the data block according to the access information of the IO request.

[0105] Assign weights to each access information factor and perform weighted summation to obtain the data value of the data block.

[0106] S3. Divide the data value intervals, and determine the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of the data blocks on each node in the data value intervals, specifically including:

[0107] S31. Divide N data value intervals;

[0108] S32. For each node, calculate the percentage of data blocks in each data value interval;

[0109] S33. For each data value interval, compare the node with the highest percentage of data blocks with the node with the lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the highest percentage of data blocks is recorded as the node to be migrated out, and the node with the lowest percentage of data blocks is recorded as the node to be migrated in, otherwise do not migrate; then compare the node with the second highest percentage of data blocks with the node with the second lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the second highest percentage of data blocks is recorded as the node to be migrated out, and the node with the second lowest percentage of data blocks is recorded as the node to be migrated in, otherwise do not migrate; and so on.

[0110] S4. Migrate the data blocks on the nodes to be migrated out to the nodes to be migrated in, specifically:

[0111] According to the determined nodes to be migrated out and the nodes to be migrated in, migrate the data blocks based on the set data migration rules; and after the migration is completed, notify the metadata module to update the metadata information.

[0112] The multi-node load balancing hot and cold data migration method of this embodiment is implemented based on the foregoing multi-node load balancing hot and cold data migration device. Therefore, the specific implementation manners in this method can be seen in the embodiment part of the multi-node load balancing hot and cold data migration device in the foregoing text. Therefore, its specific implementation manners can be referred to the descriptions of the corresponding parts of each embodiment, and will not be elaborated here.

[0113] In addition, since the multi-node load balancing hot and cold data migration method of this embodiment is implemented based on the foregoing multi-node load balancing hot and cold data migration device, its functions correspond to those of the above device, and will not be repeated here.

[0114] Embodiment 5

[0115] Figure 5Schematic diagram of a terminal device 500 provided by an embodiment of the present invention, including: a processor 510, a memory 520, and a communication unit 530. When the processor 510 is used to implement the multi-node load balancing hot and cold data migration program stored in the memory 520, the following steps are implemented:

[0116] S1. Collect the IO request access information of each data block in each node;

[0117] S2. Calculate the data value of the data block according to the IO request access information;

[0118] S3. Divide the data value range, and determine the nodes to be migrated out and the nodes to be migrated in corresponding to each data value range according to the distribution of data blocks on each node in the data value range;

[0119] S4. Migrate the data blocks on the nodes to be migrated out to the nodes to be migrated in.

[0120] The present invention determines the data value of the data block, divides the data value range, and appropriately migrates the stored file according to the distribution range of the data value. By finding the storage nodes with less load pressure and migrating some data to these storage nodes with less load pressure, the access pressure between nodes can be balanced to a large extent, reducing the delay problem caused by uneven load of node resources and saving the user's search time; at the same time, due to the balanced use of resources between nodes, the delay and slow response problems caused by sudden access or concurrent access of a single node within a short period of time are reduced. The hot data is separated to each working node to balance the bandwidth between nodes, improve the IOPS and bandwidth of the storage system, and ultimately improve the user experience and the effect of IOPS.

[0121] The terminal device 500 includes a processor 510, a memory 520, and a communication unit 530. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation to the present invention. It can be a bus structure, a star structure, and can also include more or fewer components than shown in the figure, or combine some components, or arrange different components.

[0122] Among them, the memory 520 can be used to store the execution instructions of the processor 510. The memory 520 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disc. When the execution instructions in the memory 520 are executed by the processor 510, the terminal 500 can execute some or all of the steps in the above method embodiments.

[0123] The processor 510 is the control center of the storage terminal, connecting various parts of the entire electronic terminal through various interfaces and lines. By running or executing the software programs and / or modules stored in the memory 520, and by calling the data stored in the memory, it executes various functions of the electronic terminal and / or processes data. The processor can be composed of an integrated circuit (IC). For example, it can be composed of a single packaged IC, or can be composed of multiple packaged ICs with the same or different functions connected. For example, the processor 510 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single operation core or can include multiple operation cores.

[0124] The communication unit 530 is used to establish a communication channel so that the storage terminal can communicate with other terminals. It receives user data sent by other terminals or sends user data to other terminals.

[0125] Embodiment Six

[0126] The present invention also provides a computer storage medium. The storage medium here can be a magnetic disk, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0127] The computer storage medium stores a multi-node load balancing hot and cold data migration program. When the multi-node load balancing hot and cold data migration program is executed by a processor, the following steps are implemented:

[0128] S1, collect the IO request access information of each data block in each node;

[0129] S2, calculate the data value of the data block according to the IO request access information;

[0130] S3. Divide the data value intervals, and determine the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of data blocks on each node in the data value intervals;

[0131] S4. Migrate the data blocks on the nodes to be migrated out to the nodes to be migrated in.

[0132] The present invention determines the data value of data blocks, divides the data value intervals, and appropriately migrates the stored files according to the distribution intervals of the data values. By finding the storage nodes with less load pressure and migrating some data to these storage nodes with less load pressure, the access pressure between nodes can be balanced to a large extent, reducing the latency problem caused by uneven load of node resources and saving the user's search time; at the same time, because the resource usage between nodes is balanced, the latency and slow response problems caused by sudden access or concurrent access of a single node within a short period of time are reduced. The hot data is separated to each working node, achieving the balance of the bandwidth between nodes, improving the IOPS and bandwidth of the storage system, and ultimately improving the user experience and the effect of IOPS.

[0133] Those skilled in the art can clearly understand that the technologies in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disc, etc., which can store program codes, including several instructions for causing a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in the embodiments of the present invention.

[0134] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical, or other form.

[0135] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit.

[0137] The above-disclosed are only the preferred embodiments of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be conceived by those skilled in the art, as well as several improvements and refinements made without departing from the principle of the present invention, should fall within the protection scope of the present invention.

Claims

1. A multi-node load balancing cold and hot data migration device, characterized in that, including Data collection module: Collecting the IO request access information of each data block in each node; Data value calculation module: Calculating the data value of the data block according to the IO request access information; The data value refers to the hotness and coldness degree of the data block, and calculating the data value of the data block means evaluating the hotness and coldness value of the data block; Hot and cold data identification module: Dividing the data value interval, and determining the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of the data blocks on each node in the data value interval; Migration module: Migrating the data blocks on the nodes to be migrated out to the nodes to be migrated in; Among them, the hot and cold data identification module is specifically used for: Dividing N data value intervals; For each node, calculating the percentage of data blocks in each data value interval; For each data value interval, comparing the node with the highest data block percentage with the node with the lowest data block percentage. If the difference between the two data block percentages exceeds the threshold percentage, the node with the highest data block percentage is recorded as the node to be migrated out, and the node with the lowest data block percentage is recorded as the node to be migrated in, otherwise no migration is performed; Then comparing the node with the second highest data block percentage with the node with the second lowest data block percentage. If the difference between the two data block percentages exceeds the threshold percentage, the node with the second highest data block percentage is recorded as the node to be migrated out, and the node with the second lowest data block percentage is recorded as the node to be migrated in, otherwise no migration is performed; And so on; The device further includes Metadata module: Recording the physical storage location of each data block, including the storage node, the storage device type, and the specific storage location in the storage device; Migration plan generation module: Setting data migration rules, including the migration method, the migration period, the migration time, and the wear leveling mechanism; The wear leveling mechanism includes preferentially selecting the data blocks with fewer migration times within a certain recent time within the same heat range when selecting the migration blocks; The migration module is specifically used for: According to the nodes to be migrated out and the nodes to be migrated in determined by the hot and cold data identification module, and based on the data migration rules set by the migration calculation module, migrating the data blocks to separate the hot data to each working node to balance the bandwidth between the nodes; And after the migration is completed, notifying the metadata module to update the metadata information; The data block IO access request information collected by the data collection module includes the total access frequency, the random read / write access frequency, the sequential read / write access frequency, the last access time of the file, the last access type, and the end address of the last access; The data value calculation module calculates the data value of the data block according to the IO request access information, specifically: Assigning weights to each access information factor and performing weighted summation to obtain the data value of the data block.

2. A multi-node load balancing hot and cold data migration method, characterized in that Including the following steps: S1. Collecting the IO request access information of each data block in each node; S2. Calculating the data value of the data block according to the IO request access information; The data value refers to the hotness and coldness degree of the data block, and calculating the data value of the data block means evaluating the hotness and coldness value of the data block; S3. Dividing the data value interval, and determining the nodes to be migrated out and the nodes to be migrated in corresponding to each data value interval according to the distribution of the data blocks on each node in the data value interval; S4. Migrate the data blocks on the node to be migrated out to the node to be migrated in. Step S3 specifically includes: Divide into N data value intervals. For each node, calculate the percentage of data blocks in each data value interval. For each data value interval, compare the node with the highest percentage of data blocks with the node with the lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the highest percentage of data blocks is recorded as the node to be migrated out, and the node with the lowest percentage of data blocks is recorded as the node to be migrated in; otherwise, do not migrate. Then compare the node with the second-highest percentage of data blocks with the node with the second-lowest percentage of data blocks. If the difference between the two percentages of data blocks exceeds the threshold percentage, the node with the second-highest percentage of data blocks is recorded as the node to be migrated out, and the node with the second-lowest percentage of data blocks is recorded as the node to be migrated in; otherwise, do not migrate. And so on. This method further includes S0. Record the physical storage location of each data block, including the storage node, the type of storage device, and the specific storage location in the storage device; set the data migration rules, including the migration method, migration period, migration time, and wear leveling mechanism; the wear leveling mechanism includes preferentially selecting the data blocks with fewer migration times within a certain period recently in the same heat range when selecting the migration blocks. Step S4 specifically includes: Based on the determined node to be migrated out and the node to be migrated in, and based on the set data migration rules, migrate the data blocks to separate the hot data to each working node to balance the bandwidth between the nodes; and after the migration is completed, notify the metadata module to update the metadata information. The data block IO access request information collected in Step S1 includes the total access frequency, random read / write access frequency, sequential read / write access frequency, the last access time of the file, the last access type, and the end address of the last access. Step S2 is specifically: Assign weights to each access information factor and perform weighted summation to obtain the data value of the data block.

3. A terminal, characterized in that, It includes: A memory for storing the multi-node load balancing hot and cold data migration program. A processor for implementing the steps of the multi-node load balancing hot and cold data migration method as described in claim 2 when executing the multi-node load balancing hot and cold data migration program.

4. A computer-readable storage medium, characterized in that, The multi-node load balancing hot and cold data migration program is stored on the readable storage medium, and when the multi-node load balancing hot and cold data migration program is executed by the processor, the steps of the multi-node load balancing hot and cold data migration method as described in any one of claim 2 are implemented.

Images