A disk configuration method, device, server, electronic device, and storage medium

Abstract

The present application relates to a disk configuration method, device, server, electronic device, and storage medium. The method is applicable to the configuration end of a disk array. The disk array includes at least one first type of disk and at least one second type of disk, and the data transmission speed of the first type of disk is greater than that of the second type of disk. The method includes: determining the available disk space of the first type of disk; configuring the available disk space of the first type of disk into a buffer area for at least one of the second type of disks. When data is transmitted, the data is first cached in the buffer area of at least one second type of disk, and then the second type of disk performs data transmission, so that the caching speed of the second type of disk becomes faster, thereby improving the data writing speed of the second type of disk, and further avoiding the problem that the hardware of the second type of disk limits the data transmission speed and the data transmission efficiency of the second type of disk is low when performing a data transmission task, which affects the data transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a disk configuration method, device, server, electronic device, and storage medium. Background Art

[0002] With the development of the Internet, the number of servers in the transcoding clusters of each Internet company has also increased. Each server has multiple disks, and each server manages the hard disks in the way of Logical Volume Manager (LVM). Specifically, multiple second-class disks are combined into one large disk for use as a data disk, and one first-class disk is used alone as a system disk. Using this method will result in insufficient utilization of the first-class disks, whether in terms of space (up to 30%) or performance (less than 10%); for the second-class disks in this usage mode, there is no need to perform data backup and consider disk reliability issues, but due to hardware limitations, there is an upper limit on the disk speed, resulting in performance problems with the second-class disks, and they run slowly when performing data transfer tasks; moreover, when the second-class disks are under high load, data transfer runs even more slowly, affecting the efficiency of data transfer. Therefore, how to improve the data transfer speed of the second-class disks has become an urgent problem to be solved; Summary of the Invention

[0003] This application provides a disk configuration method, device, server, electronic device, and storage medium to solve the problem in the related art that when performing data transfer tasks, the data transfer efficiency of the second-class disks is low, affecting the data transfer efficiency.

[0004] In a first aspect, this application provides a disk configuration method, which is applicable to the configuration end of a disk array. The disk array includes at least one first-class disk and at least one second-class disk. The data transfer speed of the first-class disks is greater than that of the second-class disks, and the method includes: determining the available disk space of the first-class disks; configuring the available disk space of the first-class disks into at least one buffer area for the second-class disks.

[0005] Optionally, configuring the available disk space of the first-class disks into at least one buffer area for the second-class disks includes: configuring the available disk space of the first-class disks into at least one read buffer area for the second-class disks.

[0006] Optionally, configuring the available disk space of the first-class disks into at least one buffer area for the second-class disks includes: configuring the available disk space of the first-class disks into at least one write buffer area for the second-class disks.

[0007] Optionally, configuring the available disk space of the first type of disk as a buffer area for at least one of the second type of disks includes: configuring the available disk space of the first type of disk as a read buffer area and a write buffer area for at least one of the second type of disks.

[0008] Optionally, configuring the available disk space of the first type of disk as a read buffer area and a write buffer area for at least one of the second type of disks includes: determining the read speed and write speed of the first type of disk; determining the size ratio of the read buffer area and the write buffer area according to the read speed and the write speed; and configuring the available disk space of the first type of disk as a read buffer area and a write buffer area for at least one of the second type of disks according to the determined size ratio.

[0009] Optionally, before configuring the available disk space of the first type of disk as a buffer area for at least one of the second type of disks, the method further includes: determining a disk allocation policy for the first type of disk according to the number of the second type of disks; and partitioning the available disk space of the first type of disk according to the disk allocation policy to configure the available disk space of the first type of disk as buffer areas corresponding to the number of the second type of disks.

[0010] Optionally, after configuring the available disk space of the first type of disk as a buffer area for at least one of the second type of disks, the method further includes: associating the buffer areas configured from the available disk space of the first type of disk with the corresponding second type of disks one by one to form at least one storage unit.

[0011] Optionally, when the available disk space of the first type of disk is configured as a read buffer area and a write buffer area for at least one of the second type of disks, after associating the buffer areas configured from the available disk space of the first type of disk with the corresponding second type of disks one by one to form at least one storage unit, the method further includes: creating a first physical volume according to at least one of the storage units, and creating a second physical volume according to the write buffer area; adding the first physical volume to a first volume group, and adding the second physical volume to a second volume group; creating a striped volume according to the first volume group, creating a solid-state disk volume according to the second volume group, and associating the striped volume and the solid-state disk volume to form a cache volume.

[0012] Second aspect, the present application provides a disk configuration device, which is connected to a disk array. The disk array includes at least one first type of disk and at least one second type of disk. The data transfer speed of the first type of disk is greater than that of the second type of disk. The disk configuration device includes: a first determination module, which is used to determine the available disk space of the first type of disk; a first configuration module, which is used to configure the available disk space of the first type of disk into a buffer area for at least one of the second type of disks.

[0013] Third aspect, the present application provides a server, which includes a disk array and a configuration terminal. The disk array includes at least one first type of disk and at least one second type of disk. The data transfer speed of the first type of disk is greater than that of the second type of disk. The configuration terminal includes: a second determination module, which is used to determine the available disk space of the first type of disk; a second configuration module, which is used to configure the available disk space of the first type of disk into a buffer area for at least one of the second type of disks.

[0014] Fourth aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0015] The memory is used to store a computer program;

[0016] The processor is used to implement the steps of the disk configuration method described in any embodiment of the first aspect when executing the program stored on the memory.

[0017] Fifth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the disk configuration method described in any embodiment of the first aspect.

[0018] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art:

[0019] The method provided by the embodiment of the present application is applicable to the configuration end of a disk array. The disk array includes at least one first type of disk and at least one second type of disk, and the data transfer speed of the first type of disk is greater than that of the second type of disk. The method includes: determining the available disk space of the first type of disk; configuring the available disk space of the first type of disk into a buffer area for at least one of the second type of disks. When data is transferred, the data is first cached in the buffer area of at least one second type of disk, and then the second type of disk performs data transfer, so that the caching speed of the second type of disk becomes faster, thereby improving the data writing speed of the second type of disk, and further avoiding the problem that the hardware of the second type of disk limits the data transfer speed and the data transfer efficiency of the second type of disk is low when performing data transfer tasks, which affects the data transfer efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention and used together with the specification to explain the principles of the present invention.

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a schematic flowchart of a disk configuration method provided by an embodiment of the present application;

[0023] Figure 2 It is a schematic diagram of the basic structure of a cache volume set up provided by an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the basic structure of a cache controller provided by an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the basic structure of a disk configuration device provided by an embodiment of the present application;

[0026] Figure 5 It is a schematic diagram of the basic structure of a server provided by an embodiment of the present application;

[0027] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are only a part rather than all of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0029] Figure 1 A disk configuration method provided for an embodiment of this application. The method is applicable to a configuration end of a disk array. The disk array includes at least one first-type disk and at least one second-type disk. The data transfer speed of the first-type disk is greater than that of the second-type disk. The method includes:

[0030] S101. Determine the available disk space of the first-type disk;

[0031] S102. Configure the available disk space of the first-type disk into at least one buffer area of the second-type disk.

[0032] It can be understood that this embodiment does not limit the types of the first-type disk or the second-type disk. It is only required that the data transfer speed of the first-type disk is greater than that of the second-type disk. Moreover, this embodiment does not limit the storage space sizes of the first-type disk and the second-type disk, which can be flexibly set by relevant users.

[0033] Among them, the disk array includes at least one first-type disk. When there is only one first-type disk in the disk array, the available disk space of the first-type disk is directly obtained as the available disk space of this first-type disk. When there are multiple first-type disks in the disk array, the available disk space of the first-type disk is determined as the sum of the available spaces of all first-type disks;

[0034] In some examples of this embodiment, determining the available disk space of the first-type disk includes: determining the total disk space of the first-type disk; determining whether there is reserved space in the first-type disk; when there is reserved space in the first-type disk, obtaining the reserved space, and determining the available disk space of the first-type disk according to the reserved space and the total disk space; when there is no reserved space in the first-type disk, directly taking the total disk space of the first-type disk as the available disk space of the first-type disk. Among them, the reserved space is the space for storing file data of a specified type; for example, the reserved space is the space for storing an operating system.

[0035] Continuing with the above example, specifically, for instance, if the total disk space capacity of the first type of disk is M and the reserved space capacity of the first type of disk is N, then the available disk space capacity of the first type of disk is M - N. When the reserved space capacity N of the first type of disk is zero, the available disk space of the first type of disk is directly M - 0 = M.

[0036] It can be understood that the disk configuration method provided in this embodiment is applicable to the configuration end of a disk array, which includes at least one second type of disk. When there are multiple second type of disks in the disk array, the data transfer rate of each second type of disk is lower than that of the first type of disk. And in some examples, when there are multiple second type of disks in the disk array, the data transfer rates of at least two second type of disks are different. In some examples, when there are multiple second type of disks in the disk array, the data transfer rates of at least two second type of disks are the same.

[0037] In some examples of this embodiment, before configuring the available disk space of the first type of disk into the buffer area of at least one second type of disk, the method further includes: determining the disk allocation strategy of the first type of disk according to the number of the second type of disks; partitioning the available disk space of the first type of disk according to the disk allocation strategy, so as to configure the available disk space of the first type of disk into buffer areas corresponding to the number of the second type of disks. Wherein the disk allocation strategy includes but is not limited to: the first way: configuring all the available disk space of the first type of disk into the buffer area of the second type of disk according to the number of the second type of disks; the second way: configuring a part of the available disk space of the first type of disk into the buffer area of the second type of disk according to the number of the second type of disks.

[0038] Continuing with the above example, when the disk allocation strategy is the first way, after obtaining the available disk space of the first type of disk, directly configure all the available disk space as the buffer area of the second type of disk; for example, when there is one second type of disk in the disk array, configure all the available disk space of the obtained first type of disk as the buffer area of this second type of disk; when there are multiple second type of disks in the disk array and the available disk space of the first type of disk is configured into the buffer areas of N second type of disks, obtain the available disk space capacity M of the first type of disk, and then divide the available disk space capacity M of the first type of disk into N buffer areas, so as to configure the available disk space of the first type of disk into the buffer areas of N second type of disks.

[0039] Continuing with the above example, dividing the available disk space of the first type of disk into N buffer areas includes but is not limited to one of the following ways:

[0040] The first method: evenly divide the available disk space of the first type of disk into N buffer areas, that is, the capacity of each buffer area is the same, and the capacity of each buffer area is M / N of the capacity of the available disk space;

[0041] The second method: randomly divide the available disk space of the first type of disk into N buffer areas, that is, the capacity of each buffer area is random, and the sum of the capacities of all buffer areas is equal to the capacity M of the available disk space of the first type of disk;

[0042] The third method: divide the available disk space of the first type of disk according to the parameters of each second type of disk, where the parameters of the second type of disk include but are not limited to: the total disk space of the second type of disk, the data transfer speed, the capacity of the data to be retrieved, etc. For example, when there are multiple second type of disks, obtain the total disk space of each second type of disk, and allocate a disk space as a buffer area for each second type of disk.

[0043] It should be understood that configuring the available disk space of the first type of disk into at least one buffer area of the second type of disk further includes: configuring a part of the available disk space of the first type of disk into at least one buffer area of the second type of disk. In some examples, the available disk space of the first type of disk is relatively large. If all the available disk space of the first type of disk is used as the buffer area of the second type of disk, it will cause waste of the available disk space of some first type of disks. Therefore, only a part of the available disk space of the first type of disk can be configured into at least one buffer area of the second type of disk. For example, the capacity of the available disk space of the first type of disk is 1T, and the capacity of the second type of disk is 500G. If all the available disk space of the first type of disk is used as the buffer area of the second type of disk, it will inevitably cause waste of the available disk space of the first type of disk. Therefore, based on the capacity of the second type of disk, a part of the available disk space of the first type of disk can be configured into at least one buffer area of the second type of disk; among them, the method of configuring a part of the available disk space of the first type of disk into at least one buffer area of the second type of disk is the same as the method of configuring all the available disk space of the first type of disk into at least one buffer area of the second type of disk, which will not be elaborated here.

[0044] In some examples of this embodiment, configuring the available disk space of the first type of disk into at least one buffer area of the second type of disk includes: configuring the available disk space of the first type of disk into at least one read buffer area of the second type of disk.

[0045] It should be understood that configuring the available disk space of the first type of disk as the read buffer of at least one of the second type of disks includes: configuring all the available disk space of the first type of disk as the read buffer of at least one of the second type of disks; or configuring a part of the available disk space of the first type of disk as the read buffer of at least one of the second type of disks; for the specific configuration method, refer to the above disk allocation policy and will not be elaborated here.

[0046] In some examples, configuring the available disk space of the first type of disk as the read buffer of at least one of the second type of disks can also be configuring the available disk space of the first type of disk as a fixed number of read buffers, and all the second type of hard disks share all the read buffers. For example, configuring the available disk space of the first type of disk as one read buffer, and all the second type of disks share this read buffer.

[0047] In some examples of this embodiment, configuring the available disk space of the first type of disk as the buffer of at least one of the second type of disks includes: configuring the available disk space of the first type of disk as the write buffer of at least one of the second type of disks.

[0048] It should be understood that configuring the available disk space of the first type of disk as the write buffer of at least one of the second type of disks includes: configuring all the available disk space of the first type of disk as the write buffer of at least one of the second type of disks; or configuring a part of the available disk space of the first type of disk as the write buffer of at least one of the second type of disks; for the specific configuration method, refer to the above disk allocation policy and will not be elaborated here.

[0049] In some examples, configuring the available disk space of the first type of disk as the write buffer of at least one of the second type of disks can also be configuring the available disk space of the first type of disk as a fixed number of write buffers, and all the second type of hard disks share all the write buffers. For example, configuring the available disk space of the first type of disk as one write buffer, and all the second type of disks share this write buffer.

[0050] In some examples of this embodiment, configuring the available disk space of the first type of disk as a buffer for at least one of the second type of disks includes: configuring the available disk space of the first type of disk as a read buffer and a write buffer for at least one of the second type of disks. Wherein, the sum of the capacities of the read buffer and the write buffer is the available disk space of the first type of disk; in some examples, configuring the available disk space of the first type of disk as a read buffer and a write buffer for at least one of the second type of disks includes: configuring the available disk space of the first type of disk as one write buffer and N read buffers, that is, the number of read buffers is equal to the number of the second type of disks, and all the second type of disks share this write buffer, and each second type of disk corresponds to one read buffer respectively; or, configuring the available disk space of the first type of disk as N write buffers and N read buffers, and each second type of disk corresponds to one read buffer and one write buffer respectively.

[0051] It can be understood that when configuring the available disk space of the first type of disk as a read buffer and a write buffer for at least one of the second type of disks, it can be to configure all the available disk space of the first type of disk as a read buffer and a write buffer for at least one of the second type of disks, or it can be to configure a part of the available disk space of the first type of disk as a read buffer and a write buffer for at least one of the second type of disks. This embodiment does not limit this.

[0052] In some examples of this embodiment, configuring the available disk space of the first type of disk as a read buffer and a write buffer for at least one of the second type of disks includes: determining the read speed and write speed of the first type of disk; determining the size ratio of the read buffer and the write buffer according to the read speed and the write speed; and configuring the available disk space of the first type of disk as a read buffer and a write buffer for at least one of the second type of disks according to the determined size ratio. It should be understood that through the determined size ratio, the capacities of the read buffer and the write buffer configured from the available disk space of the first type of disk can be determined, and then the available disk space of the first type of disk can be configured as a read buffer and a write buffer for at least one of the second type of disks;

[0053] Continuing from the above example, for example, if the ratio of the read speed to the write speed of the first type of disk is determined to be 1:4, then at this time, the ratio of the capacity of the read buffer and the capacity of the write buffer configured from the available disk space of the first type of disk is 1:4. When the capacity of the available disk space of the first type of disk is 500G and all the available disk space of the first type of disk is configured as a read buffer and a write buffer for at least one of the second type of disks, then the total capacity of the read buffer configured from the available disk space of the first type of disk is 100G, and the total capacity of the write buffer is 400G.

[0054] It should be understood that after determining the total capacity of the read cache area and the total capacity of the write cache area configured from the available disk space of the first type of disk, according to the determined disk allocation policy, the available disk space of the first type of disk is partitioned to configure the available disk space of the first type of disk into cache areas corresponding to the number of the second type of disks; specifically, for example, the total capacity of the read cache area configured from the available disk space of the first type of disk is 100G, the total capacity of the write cache area is 400G, there are N second type of disks, and at this time the disk allocation policy is: evenly allocate the read cache area, and all the second type of disks share a write cache area. Finally, the available disk space of the first type of disk is configured into a 400G write cache area and N read cache areas, and the capacity of each read cache area is 100G / N G.

[0055] In some examples of this embodiment, after configuring the available disk space of the first type of disk into cache areas for at least one of the second type of disks, the method further includes: associating the cache areas configured from the available disk space of the first type of disk with the corresponding second type of disks one by one to form at least one storage unit.

[0056] Continuing with the above example, for example, there are 4 second type of disks, and the 4 disk drive letters are / dev / sda, / dev / sdb, / dev / sdc, / dev / sdd respectively. The 4 read cache areas obtained by partitioning the first type of disk are / dev / nvmenp1, / dev / nvmenp2, / dev / nvmenp3, / dev / nvmenp4 respectively. After association binding and adding aliases, 4 new storage units are formed, and the four new storage units are / dev / sda - bcache, / dev / sdb - bcache, / dev / sdc - bcache, / dev / sdd - bcache respectively.

[0057] In some examples of this embodiment, configuring the storage unit and the write cache area to form a cache volume includes: creating a first physical volume according to at least one of the storage units, and creating a second physical volume according to the write cache area; adding the first physical volume to a first volume group, and adding the second physical volume to a second volume group; creating a striped volume according to the first volume group, creating a solid - state disk volume according to the second volume group, and associating the striped volume and the solid - state disk volume to form a cache volume.

[0058] Specifically, physical volumes (PVs) are created according to the storage units and the write cache areas respectively, and then logical volumes (LVs) are created based on the created PVs. Finally, cache volumes are created based on the logical volumes. Among them, the physical volume is at the bottom layer of the logical volume management system and can be the entire physical hard disk or a partition on the actual physical hard disk. The logical volume is built on the physical volume, and at least one physical volume should be included in one logical volume. After the logical volume is created, volumes can be dynamically added to the logical volume. There can be multiple logical volumes in a logical volume management system project. Specifically, for example, there are four second-class disks, and the four second-class disks are associated and bound to four read cache areas to obtain four storage units: / dev / sda-bcache, / dev / sdb-bcache, / dev / sdc-bcache, / dev / sdd-bcache. First, configure the Logical Volume Manager (LVM) PV, that is, create multiple first PVs according to the four storage units, and create a second PV according to the write cache area obtained from the first-class disk partition. Then configure the LVM VG, that is, add multiple first PVs to the first logical volume group, and then add the second PV to the second logical volume group. Finally, combine the LVM cache volume, that is, create an LVM stripe volume according to the first logical volume group, and then create an LVM solid-state disk volume (LVM ssd volume) according to the second logical volume group. Associate the LVM stripe volume and the LVM solid-state disk volume to form an LVM cache volume and set it as the write cache.

[0059] In some examples of this embodiment, after configuring the storage unit and the write cache area to form a cache volume, the method further includes: setting a cache controller for controlling the cache volume, and the cache controller is used to dynamically adjust the write cache disk dropping operation in the cache volume. Specifically, the operations mainly involved are the start write-back upper limit, the stop write-back lower limit, the write-back block size, the number of write-back tasks, and the immediate forced write-back. The cache controller will trigger control operations according to the current write cache area size and the current task disk space requirement situation.

[0060] In some examples of this embodiment, dynamically adjusting the write cache disk dropping operation in the cache volume includes: obtaining the current running transcoding task information, and the transcoding task information is used to obtain the target storage space required within the target time; determining the execution state of the write cache area according to the target storage space and the size of the write cache area; adjusting the write speed of the write cache area according to the execution state to adjust the write cache disk dropping operation in the cache volume.

[0061] Continuing with the above example, by obtaining the currently running transcoding task information, the transcoding task information is used to obtain the target storage space required to be used within the target time, and then the size of the target storage space accounts for the size of the write cache area to determine the execution state of the write cache area, and finally the write speed of the write cache area is adjusted according to the execution state to adjust the write cache disk operation in the cache volume. Specifically, first, the basic judgment is based on T task Within the target time, the following conditions are met based on the storage space size of the write cache and the write speed of the write cache:

[0062]

[0063]

[0064] Disk WriteBend =V WriteBend *T2

[0065] Among them, T task represents the median duration of the target task for business data, Indicates the maximum capacity of the LVMSSD write cache area, Disk release Indicates reserved capacity (reserved capacity is flexibly set by related users). Indicates T task Required write capacity within the target time, Disk WriteBend Indicates T task The time required to place the order within the time, V WriteBend Indicates the speed of transferring the LVM stripe volume to disk, V LVMSSD Indicates the LVMSSD write speed, T1 indicates the LVMSSD write cache write time, T2 indicates the time to write to the LVM stripe volume, and has the following expectations:

[0066] V Max_WriteBend ≥S block *N job

[0067] S block Indicates the write-back block size; N job Indicates the number of write-back tasks. This formula can ensure that the LVM stripe volume is not overloaded when writing.

[0068] According to the above formula, the target time T can be obtained task The required storage space size is then used to determine the execution state of the write cache area based on the required storage space size occupying the size of the write cache area, as follows:

[0069] State 1, no-load state, T task The storage space size required for writing the cache within a certain period of time Less than 50% At this time, T 2＝ T task , it is necessary to ensure that That is No additional trigger for adjustment is required;

[0070] State 2, full load state: T task The storage space size usage requirement in the buffer within the time Is between 50% - 75% At this time Then there is Then there is Therefore, when it is judged that the value exceeds 50%, an adjustment operation will be triggered, N job Value At this time, N job Has reached the expected maximum value;

[0071] State 3, overload state: T task The storage space size usage requirement in the buffer within the time Is between 75% - 100% At this time And there is When it exceeds 75%, an immediate forced write-back operation will be triggered, and the ssd write cache disk will be throttled to 50% of the original speed.

[0072] The cache controller provided in this example controls the cache volume through the above three states, can adjust the cache volume according to different states, and then precisely control the cache volume, continuously keeping the cache volume write in an available state, and overall improving the IO read and write rate of the cache volume.

[0073] The disk configuration method provided in this embodiment is applicable to the configuration end of a disk array. The disk array includes at least one first type of disk and at least one second type of disk, and the data transfer speed of the first type of disk is greater than that of the second type of disk. The method includes: determining the available disk space of the first type of disk; configuring the available disk space of the first type of disk into the cache area of at least one of the second type of disks. When performing data transfer, first cache the data into the cache area of at least one second type of disk, and then the second type of disk performs data transfer, so that the cache speed of the second type of disk becomes faster, and further improves the data write speed of the second type of disk, thereby avoiding the problem that the hardware of the second type of disk limits the data transfer speed and the data transfer efficiency of the second type of disk is low when performing data transfer tasks, which affects the data transfer efficiency.

[0074] To better understand the present invention, this example presents a more specific example to illustrate the present invention;

[0075] First, a disk array is provided, which includes: the first type of disk: a fast disk (SSD system disk); the second type of disk: a slow disk (HDD data disk). The data transfer speed of the first type of disk is greater than that of the second type of disk, and the fast disk and the slow disk are independent of each other, and their respective IOs are separated, and multiple partitions of the fast read disk do not affect the IO rate.

[0076] First, according to the available disk space of the fast disk, the data transfer speed of the fast disk, and the number of the second type of disks, determine the sizes of the read buffer and the write buffer configured by the fast disk to allocate the fast disk. According to the transcoding scenario, judge the IO bottleneck of the fast disk. In this example, in the transcoding scenario, the performance bottleneck caused by writing is 80%, so the size ratio of the write buffer to the read buffer can be controlled at 4:1; specifically, the following formula can be referred to:

[0077] Disk_ssd total =Number Disk_ HDD*Disk_ssd readcache +Disk_ssd writecache +Disk_ssd sys

[0078] Number Disk_HDD *Disk_ssd readcache :Disk_ssd writecache =1:4

[0079] Where Disk_ssd total is the entire disk size of the SSD disk, which is 800G; Disk_ssd sys is the space occupied by the root partition, which is 300G. According to Disk_ssd total and Disk_ssd sys it can be obtained that the available disk space of the SSD disk is 500G; Number Disk_HDD represents the number of HDD disks, which is, for example, 4 4T HDDs; Disk_ssd readcache represents the size allocated for the read buffer for each HDD disk, Disk_ssd writecache is the size of the LVM cache disk for the write buffer. Combining the above exemplary transcoding environment, the overall size ratio of the write buffer to the read buffer is 4:1. Therefore, it is obtained that Disk_ssd readcache is 25G, and Disk_ssd writecache is 400G;

[0080] Continuing with the above example, according to the read and write cache ratio confirmed in the previous step, partition the remaining two partitions of the SSD disk in a ratio of 4:1. In an exemplary transcoding environment, it is a separate 400G write cache area (write cache) and four 25G read cache areas (read cache);

[0081] Then bind the HDD disk to the above read cache area, configure the read cache area for the HDD disk, use bcache to associate and bind the pre-partitioned areas for the read cache area with each HDD disk, and set the cache as the read cache, while adding aliases; in an exemplary transcoding environment, the HDD disk drive letters are / dev / sda, / dev / sdb, / dev / sdc, / dev / sdd respectively, and the drive letters of the read cache areas are / dev / nvmenp1, / dev / nvmenp2, / dev / nvmenp3, / dev / nvmenp4. After the association, binding, and adding of aliases, new storage units / dev / sda-bcache, / dev / sdb-bcache, / dev / sdc-bcache, / dev / sdd-bcache are formed;

[0082] It should be understood that after forming the new storage units, execute the configuration of LVM PV, and create PVs for Disk_HDD+Disk_ssd readcache and Disk_ssd writecache respectively; after creating the PVs, configure LVM VG, add all Disk_HDD+Disk_ssd readcache to one VG, and Disk_ssd writecache is separately used as a VG_fast; combine the LVM cache volumes, create an LVM stripe volume using VG, create an LVMssd volume for the VG-fast single disk, associate the LVMssd and the LVM stripe volume to form an LVM cache volume, and set it as the write cache;

[0083] After completing the cache volume settings, deploy and configure the LVM cache controller. As Figure 2 shown, the LVM cache controller is used to dynamically adjust the LVM write cache disk write operation. The main operations involved are (start write-back upper limit, stop write-back lower limit, write-back block size, write-back task count, immediate forced write-back). The controller will trigger controller operations according to the current write cache disk size and the current task disk space requirements;

[0084] As Figure 3 shown, the structure of the cache controller is divided into three modules:

[0085] Task information collection module: This module is used to collect the information of the running transcoding tasks and calculate the future T taskDisk space to be used within a certain time

[0086] Cache disk information collection module: Collect the usage status of the cache disk in real time;

[0087] Instruction call module: Pre-store the configuration parameters for write cache operations. When receiving an instruction from the control module, directly call the cache volume for operation;

[0088] Status control module: Determine the status of the write cache disk according to the write requirements of the disk for the task and the real-time status of the write cache disk, and then call the corresponding instructions to regulate the write cache disk for different statuses;

[0089] When the status control module makes a status judgment, the basic judgment basis is at T task Within a certain time, there are the following conditions based on the disk capacity and write speed:

[0090]

[0091]

[0092] Disk WriteBend =V WriteBend *T2

[0093] Where, T task Is the median of the task duration represented by the service data, Represents the maximum capacity of the LVMSSD disk, Disk release Represents the reserved capacity, Represents T task The required write capacity within a certain time, Disk WriteBend Represents T task The time required for disk writing within a certain time, V WriteBend Represents the disk writing speed to the LVM stripe volume, V LVMSSD Represents the LVMSSD write speed, T1 represents the LVMSSD disk write time, T2 represents the write time to the LVM stripe volume, and there are the following expectations:

[0094] V Max_WriteBend ≥S block *N job

[0095] S block Represents the write-back block size; N job Represents the number of write-back tasks; This formula can ensure that the LVM stripe volume writing is not overloaded;

[0096] Status one, idle state, disk usage demand within T task Is less than 50% At this time, T 2＝ 2＝T task , it is necessary to ensure that that is no additional trigger adjustment is required;

[0097] State two, full load state: T task Disk usage demand within the time is between 50% - 75% At this time then there is so there is Therefore, when it is judged that the value exceeds 50%, an adjustment operation will be triggered, N job Value At this time, N job has reached the expected maximum value;

[0098] State three, overload state: T task Disk usage demand within the time is between 75% - 100% At this time and there is When it exceeds 75%, an immediate forced write-back operation will be triggered, and the ssd cache disk speed will be reduced to 50% of the original;

[0099] The cache volume setting method provided in this embodiment improves the one-layer cache of the LVM cache volume to a two-layer cache and realizes read-write separation; a part of the space of the fast disk is used as the cache of the slow disk and is encapsulated as a conventional disk to support the use of LVM; the LVM cache disk is only used as a write cache, and the HDD disk cache is only used as a read cache; at the same time, an LVM cache controller is added to accurately control the write cache to be written to the disk; overall, the problem of the reduction in the IO rate of the transcoding cluster is eliminated, the cache volume read-write separation is achieved, and the cache volume available state is accurately controlled and maintained continuously, and the IO read-write rate of the data disk is improved overall.

[0100] Based on the same concept, this embodiment provides a disk configuration device, such as Figure 4 shown, the disk configuration device is connected to a disk array, the disk array includes at least one first type of disk and at least one second type of disk, the data transmission speed of the first type of disk is greater than that of the second type of disk, and the disk configuration device includes:

[0101] A first determination module, the first determination module is used to determine the available disk space of the first type of disk;

[0102] A first configuration module, the first configuration module is used to configure the available disk space of the first type of disk into at least one buffer area of the second type of disk.

[0103] It should be understood that the disk configuration device provided in this embodiment can combine to implement each step of the above disk configuration method, achieving the same effects as each step of the above disk configuration method. Therefore, it will not be elaborated here.

[0104] Based on the same concept, this embodiment provides a server. As Figure 5 shown, the server includes a disk array and a configuration terminal. The disk array includes at least one first type of disk and at least one second type of disk. The data transfer speed of the first type of disk is greater than that of the second type of disk. The configuration terminal includes:

[0105] A second determination module, which is used to determine the available disk space of the first type of disk;

[0106] A second configuration module, which is used to configure the available disk space of the first type of disk into a buffer area for at least one of the second type of disks.

[0107] It should be understood that the configuration terminal in the server provided in this embodiment can combine to implement each step of the above disk configuration method, achieving the same effects as each step of the above disk configuration method. Therefore, it will not be elaborated here.

[0108] As Figure 6 shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. Among them, the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0109] The memory 113 is used to store a computer program;

[0110] In an embodiment of the present application, when the processor 111 executes the program stored on the memory 113, it implements the disk configuration method provided in any of the foregoing method embodiments, including: determining the available disk space of the first type of disk; configuring the available disk space of the first type of disk into a buffer area for at least one of the second type of disks.

[0111] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the disk configuration method provided in any of the foregoing method embodiments.

[0112] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0113] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will conform to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A disk configuration method, characterized in that, The method is applicable to the configuration end of a disk array. The disk array includes at least one first - type disk and at least one second - type disk, and the data transfer speed of the first - type disk is greater than that of the second - type disk. The method includes: Determine the available disk space of the first - type disk; Configure the available disk space of the first - type disk into a buffer for at least one of the second - type disks; Among them, configuring the available disk space of the first - type disk into a buffer for at least one of the second - type disks includes: configuring the available disk space of the first - type disk into a read buffer and a write buffer for at least one of the second - type disks; The method further includes: associating the read buffer configured from the available disk space of the first - type disk with the corresponding second - type disk one by one to form at least one storage unit; configuring the storage unit and the write buffer to form a cache volume, and setting a cache controller for controlling the cache volume, where the cache controller is used to dynamically adjust the write - cache disk - dropping operation in the cache volume.

2. The method according to claim 1, characterized in that, Configuring the available disk space of the first - type disk into a buffer for at least one of the second - type disks includes: Configuring the available disk space of the first - type disk into a read buffer for at least one of the second - type disks.

3. The method according to claim 1, characterized in that, Configuring the available disk space of the first - type disk into a buffer for at least one of the second - type disks includes: Configuring the available disk space of the first - type disk into a write buffer for at least one of the second - type disks.

4. The method according to claim 1, characterized in that, Configuring the available disk space of the first - type disk into a read buffer and a write buffer for at least one of the second - type disks includes: Determine the read speed and write speed of the first - type disk; Determine the size ratio of the read buffer and the write buffer according to the read speed and the write speed; According to the determined size ratio, configure the available disk space of the first - type disk into a read buffer and a write buffer for at least one of the second - type disks.

5. The method according to any one of claims 1 - 3, characterized in that, Before configuring the available disk space of the first - type disk into a buffer for at least one of the second - type disks, the method further includes: Determine the disk allocation strategy of the first - type disk according to the number of the second - type disks; Partition the available disk space of the first - type disk according to the disk allocation strategy, so as to configure the available disk space of the first - type disk into a buffer corresponding to the number of the second - type disks.

6. The method according to claim 1, characterized in that, When the available disk space of the first - type disk is configured into a read buffer and a write buffer for at least one of the second - type disks, after associating the buffer configured from the available disk space of the first - type disk with the corresponding second - type disk one by one to form at least one storage unit, the method further includes: Create a first physical volume according to at least one of the storage units, and create a second physical volume according to the write buffer; Add the first physical volume to a first volume group, and add the second physical volume to a second volume group; Create a striped volume according to the first volume group, create a solid - state disk volume according to the second volume group, and associate the striped volume and the solid - state disk volume to form a cache volume.

7. A disk configuration device, characterized in that, The disk configuration device is connected to a disk array. The disk array includes at least one first - type disk and at least one second - type disk. The data transfer speed of the first - type disk is greater than that of the second - type disk. The disk configuration device includes: A first determination module, which is used to determine the available disk space of the first - type disk; A first configuration module, which is used to configure the available disk space of the first - type disk into a buffer area for at least one of the second - type disks; Among them, the first configuration module is specifically used to: configure the available disk space of the first - type disk into a read buffer area and a write buffer area for at least one of the second - type disks; The disk configuration device is further used to: associate the read buffer area configured from the available disk space of the first - type disk with the corresponding second - type disk one by one to form at least one storage unit; configure the storage unit and the write buffer area to form a cache volume, and set a cache controller for controlling the cache volume. The cache controller is used to dynamically adjust the write - cache disk - writing operation in the cache volume.

8. A server, characterized in that, The server includes a disk array and a configuration terminal. The disk array includes at least one first - type disk and at least one second - type disk. The data transfer speed of the first - type disk is greater than that of the second - type disk. The configuration terminal includes: A second determination module, which is used to determine the available disk space of the first - type disk; A second configuration module, which is used to configure the available disk space of the first - type disk into a buffer area for at least one of the second - type disks; Among them, the second configuration module is specifically used to: configure the available disk space of the first - type disk into a read buffer area and a write buffer area for at least one of the second - type disks; The configuration terminal is further used to: associate the read buffer area configured from the available disk space of the first - type disk with the corresponding second - type disk one by one to form at least one storage unit; configure the storage unit and the write buffer area to form a cache volume, and set a cache controller for controlling the cache volume. The cache controller is used to dynamically adjust the write - cache disk - writing operation in the cache volume.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus; The memory is used to store a computer program; The processor is used to implement the steps of the disk configuration method according to any one of claims 1 - 6 when executing the program stored on the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by the processor, implements the steps of the disk configuration method according to any one of claims 1 - 6.

Images