Data writing method and device based on load balancing and storage medium

Abstract

The invention discloses a data writing method and device based on load balancing and a storage medium, and the method comprises the steps: carrying out the adjustment of an initial writing weight of to-be-written data according to a preset writing weight strategy corresponding to the data type of the to-be-written data, obtaining a current writing weight of the to-be-written data; determining the current performance weight of each candidate storage medium according to the initial performance weight of each candidate storage medium corresponding to the to-be-written data and a preset performance weight strategy; and selecting a target storage medium from the candidate storage media according to the current storage medium and the current writing weight, and writing the to-be-written data into the target storage medium. According to the scheme, the writing weight of the to-be-written data and the performance weight of each candidate storage medium can be comprehensively considered, and then the appropriate target storage medium is selected from each candidate storage medium for writing the data, so that the data writing effect is optimized.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and particularly to a data writing method, device, and storage medium based on load balancing. Background Art

[0002] With the continuous expansion of the construction scale of data cloud storage projects, the requirements for writing various data also show diversification.

[0003] For example, common data writing methods include preferentially writing hot data into high-performance storage media (such as SSD solid-state drives) and writing non-hot data into ordinary storage media (such as HDD mechanical hard drives), thereby balancing the performance and cost in the data writing process.

[0004] However, selecting the corresponding storage medium for writing and storing based on the access heat of data has a relatively single judgment dimension and a relatively single applicable scenario, and is not suitable for storing complex and diverse data. Summary of the Invention

[0005] This application provides at least a data writing method, device, equipment, and computer-readable storage medium based on load balancing.

[0006] In the first aspect of this application, a data writing method based on load balancing is provided, including: adjusting the initial writing weight of the data to be written according to a preset writing weight policy corresponding to the data type of the data to be written to obtain the current writing weight of the data to be written; determining the current performance weights of the candidate storage media corresponding to the data to be written according to the initial performance weights and the preset performance weight policy of the candidate storage media; selecting a target storage medium from the candidate storage media according to the current performance weight and the current writing weight, and writing the data to be written into the target storage medium.

[0007] In an embodiment, the adjusting the initial writing weight of the data to be written according to a preset writing weight policy corresponding to the data type of the data to be written to obtain the current writing weight of the data to be written includes: obtaining the initial writing weight corresponding to the data type and the writing weight adjustment step corresponding to the data type in the preset writing weight policy; adjusting the initial writing weight according to the waiting writing time of the data to be written and the writing weight adjustment step to obtain the current writing weight.

[0008] In one embodiment, determining the current performance weights of the candidate storage media according to the initial performance weights of the candidate storage media corresponding to the data to be written and a preset performance weight policy includes: obtaining the initial performance weights corresponding to the candidate storage media and the performance weight adjustment steps corresponding to the candidate storage media in the preset performance weight policy; respectively adjusting the initial performance weights corresponding to the candidate storage media according to the current performance of the candidate storage media and the corresponding performance weight adjustment steps to obtain the current performance weights of the candidate storage media.

[0009] In one embodiment, obtaining the initial performance weights corresponding to the candidate storage media and the performance weight adjustment steps corresponding to the candidate storage media in the preset performance weight policy includes: obtaining the remaining storage space, utilization rate, and storage media health status of the candidate storage media; respectively determining the performance weight adjustment steps of the candidate storage media from the preset performance weight policy according to the remaining storage space, the utilization rate, and the storage media health status.

[0010] In one embodiment, selecting a target storage media from the candidate storage media according to the current performance weight and the current write weight includes: determining a first storage media and a second storage media among the candidate storage media according to the current performance weight, where the performance of the first storage media is better than that of the second storage media; determining the data priority of the data to be written according to the current write weight; in response to the data priority of the data to be written being the first priority, determining the first storage media as the target storage media and writing the data to be written into the first storage media; in response to the data priority of the data to be written being the second priority, determining the second storage media as the target storage media and writing the data to be written into the second storage media, where the first priority is higher than the second priority.

[0011] In one embodiment, determining the first storage media and the second storage media among the candidate storage media according to the current performance weight includes: respectively performing a priority sorting process on the first storage media according to the current performance weights of the first storage media and performing a priority sorting process on the second storage media according to the current performance weights of the second storage media; after determining the first storage media and the second storage media among the candidate storage media according to the current performance weight, the method includes: in response to the data priority of the data to be written being the first priority, determining the target storage media from the sorted first storage media; in response to the data priority of the data to be written being the second priority, determining the target storage media from the sorted second storage media.

[0012] In one embodiment, after selecting a target storage medium from each candidate storage medium according to the current performance weight and the current write weight, and writing the data to be written into the target storage medium, the method further includes: obtaining the remaining storage space, utilization rate, and storage medium health status of each candidate storage medium; and periodically updating the current performance weight according to the remaining storage space, the utilization rate, the storage medium health status, and a preset performance weight policy.

[0013] In one embodiment, before adjusting the initial write weight of the data to be written according to a preset write weight policy corresponding to the data type of the data to be written to obtain the current write weight of the data to be written, the method further includes: determining whether the data to be written is streaming data; if so, and if a data file of the data to be written has been created, writing the data to be written into the data file.

[0014] A second aspect of the present application provides a data writing device based on load balancing, including: a write weight determination module, configured to adjust the initial write weight of the data to be written according to a preset write weight policy corresponding to the data type of the data to be written to obtain the current write weight of the data to be written; a performance weight determination module, configured to determine the current performance weight of each candidate storage medium according to the initial performance weight of each candidate storage medium corresponding to the data to be written and a preset performance weight policy; and a target determination module, configured to select a target storage medium from each candidate storage medium according to the current performance weight and the current write weight, and write the data to be written into the target storage medium.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, where the processor is configured to execute program instructions stored in the memory to implement the above-mentioned data writing method based on load balancing.

[0016] A fourth aspect of the present application provides a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the above-mentioned data writing method based on load balancing is implemented.

[0017] In the above solution, by adjusting the initial writing weight of the data to be written according to the preset writing weight policy corresponding to the data type of the data to be written, the current writing weight of the data to be written is obtained, so that the data importance of the data to be written can be characterized according to the current writing weight; the current performance weights of the candidate storage media corresponding to the data to be written are determined according to the initial performance weights and the preset performance weight policy of the candidate storage media, so that the current performance information of the candidate storage media available for storing the data to be written can be determined, facilitating the selection of a target storage media with better performance from the candidate storage media to write the data; then, the target storage media can be selected from the candidate storage media according to the current performance weight and the current writing weight, and the data to be written is written into the target storage media, realizing the comprehensive selection of a suitable target storage media for data writing according to the data type of the data to be written and the performance of each writable candidate storage media. It can achieve load balancing during data writing, optimize the data writing effect, and can also be widely compatible with various data type data writing application scenarios, facilitating subsequent data operation and maintenance.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings here are incorporated into the specification and constitute a part of this specification. These drawings show embodiments consistent with this application and are used together with the specification to illustrate the technical solutions of this application.

[0020] Figure 1 It is a schematic flowchart of an exemplary embodiment of the data writing method based on load balancing of this application;

[0021] Figure 2 It is a schematic flowchart of an exemplary storage media selection process in the data writing method based on load balancing of this application;

[0022] Figure 3 It is a schematic flowchart of an exemplary current performance weight update process in the data writing method based on load balancing of this application;

[0023] Figure 4 It is a schematic overall flowchart of the data writing method based on load balancing of this application;

[0024] Figure 5 It is a block diagram of a data writing device based on load balancing shown in an exemplary embodiment of this application;

[0025] Figure 6 It is a schematic structural diagram of an embodiment of an electronic device of this application;

[0026] Figure 7 It is a schematic structural diagram of an embodiment of the computer-readable storage medium of the present application. Detailed implementation manners

[0027] The solutions of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings of the specification.

[0028] In the following description, specific details such as specific system structures, interfaces, and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the present application.

[0029] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this article means two or more than two. In addition, the term "at least one" in this article represents any one of multiple types or any combination of at least two of multiple types. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C.

[0030] For ease of understanding, the existing application scenarios involved in the present application are described exemplarily. With the continuous expansion of the project construction scale and the continuous refinement of data types, the requirements for the data writing priority of different types of business data also show diversification. In the existing application scenarios, generally, hot data is written into the high-performance storage medium first, and non-hot data is written into the ordinary mechanical hard disk, thereby taking into account both performance and cost. Common scenarios and methods include: dividing data into different levels according to the data access heat, and then allocating corresponding storage media for storage. A storage system usually includes multiple data storage nodes, and each node is configured with a type of storage medium. For example, there are a total of 5 storage nodes, among which 3 storage nodes can be configured with SSDs, and 2 storage nodes can be configured with HDDs. In addition, there is also a method of dividing data into metadata and file system data. Different storage resource pools are constructed using different storage media, and then different storage resource pools are selected for data writing.

[0031] Please refer to Figure 1 , Figure 1 It is a schematic flowchart of an exemplary embodiment of the data writing method based on load balancing of the present application. Specifically, it can include the following steps:

[0032] Step S110, adjust the initial writing weight of the data to be written according to the preset writing weight policy corresponding to the data type of the data to be written, so as to obtain the current writing weight of the data to be written.

[0033] Among them, the data to be written can be video data (which may include audio), image data, audio data, text data, etc., and is not limited here. For the convenience of explanation, video data and image data will be mainly used as examples in the following text.

[0034] The data type of the data to be written can be divided according to the service type of the data to be written. The specific service type needs to be adaptively determined according to the actual application scenario and is not limited here.

[0035] For example, in the application scenario of video image detection, for video image data, video service data can include but is not limited to dividing video data into ordinary point video recordings, important point video recordings, intelligent point video recordings, and linkage video recordings, etc., according to the point priority of the video recording location. Among them, the linkage video recording refers to the video recorded due to the triggering of the linkage strategy because the video acquisition device has a communication linkage relationship with other electronic devices, such as infrared detectors, speakers, lighting lamps, etc., which is not limited here and can refer to various existing video recording linkage devices.

[0036] Similarly, image service data can also include but is not limited to dividing image data into ordinary point images, important point images, intelligent point images, and linkage images, etc., according to the point priority of the image acquisition location. Among them, the linkage image refers to the image acquired due to the triggering of the linkage strategy because the image acquisition device has a communication linkage relationship with other electronic devices, such as infrared detectors, speakers, lighting lamps, etc., which is not limited here and can refer to various existing photo linkage devices.

[0037] In addition, when detecting and analyzing video images, it can also include real-time analysis data, non-real-time analysis data, etc. In most cases, the priority of real-time analysis data can be set to the highest among all data types.

[0038] The data to be written of different data types can be respectively preset with the same or different initial writing weights, and the initial writing weight is used to represent the initial priority of the data to be written. The data to be written of different data types can also be respectively preset with the same or different preset writing weight strategies, and the preset writing weight strategy is used to adjust the initial writing weight of the data to be written of the corresponding data type. Thus, the corresponding initial writing weight and preset writing weight strategy can be determined according to the data type of the data to be written obtained currently, and the initial writing weight is adjusted through the preset writing weight strategy to obtain the current writing weight of the data to be written, thereby representing the current priority of the data to be written.

[0039] Step S120, determine the current performance weight of each candidate storage medium according to the initial performance weight and the preset performance weight strategy of each candidate storage medium corresponding to the data to be written.

[0040] In combination with the foregoing steps, one or more storage media may be provided in the storage system of the present application, which may include storage media capable of receiving data to be written and / or storage media incapable of receiving data to be written. Therefore, there may be one or more storage media capable of receiving data to be written. If there is one storage media, the data to be written is directly written into the storage media. If there are multiple storage media, the multiple storage media are candidate storage media, and a target storage media needs to be determined from each candidate storage media for data writing. Therefore, it is necessary to first determine the current performance weights of each candidate storage media, and the current performance weights can characterize the current ability of the storage media to receive data writing.

[0041] It should also be noted that each candidate storage media in the present application may include, but is not limited to, solid state drives and / or mechanical hard drives, etc. Usually, in the storage system, the scenario of mixing solid state drives and mechanical hard drives is relatively extensive. Therefore, in the subsequent example description, each candidate storage media is mainly taken as an example of a solid state drive and a mechanical hard drive for description.

[0042] Exemplarily, each storage media is the same as each data type in the foregoing embodiment, and the initial performance weight corresponding to each storage media and the preset performance weight policy may be preset. Among them, the initial performance weight is used to characterize the initial priority of the storage media; the preset performance weight policy is used to adjust the initial performance weight according to the current performance of the storage media. Thus, the current performance weight corresponding to each storage media can be determined according to the initial performance weight of each storage media and its preset performance weight policy. Among them, for solid state drives and mechanical hard drives (and other types of storage media can also be inferred), the present application does not strictly limit the initial performance weights of solid state drives and mechanical hard drives. In fact, the initial performance weights of the two types (or multiple types) of storage media can be set correspondingly according to the actual application scenario. For example, starting from the advantageous characteristics of solid state drives and mechanical hard drives respectively, if in an application scenario where data read / write speed needs to be focused on, the initial performance weight of the solid state drive can be set to be greater than that of the mechanical hard drive. If in an application scenario where data long-term storage reliability needs to be focused on (or in an application scenario where data storage cost needs to be focused on), the initial performance weight of the solid state drive can be set to be less than that of the mechanical hard drive, which will not be elaborated here.

[0043] It should be noted that the process of determining the current write weight of the data to be written and the process of determining the current performance weights of each storage media (or the process of determining the current performance weights of each candidate storage media) do not limit the execution order.

[0044] Step S130: Select a target storage medium from each candidate storage medium according to the current performance weight and the current write weight, and write the data to be written into the target storage medium.

[0045] Combined with the foregoing embodiments, the current performance weight can represent the priority information of each candidate storage medium, and the current write weight can represent the priority information of the data to be written. Therefore, through the matching relationship between the current performance weight and the current write weight, which is equivalent to the priority relationship between each candidate storage medium and the data to be written, a target storage medium suitable for writing the data to be written can be selected from each candidate storage medium. Then, write the data to be written into the target storage medium.

[0046] Exemplarily, the matching relationship between the current performance weight and the current write weight is positively correlated, that is, the data to be written with a higher current write weight should be written to the storage medium with a higher current performance weight. Thus, a comprehensive analysis can be performed based on the current write weight of the data to be written and the current performance weight of each storage medium, and the data to be written can be written into a suitable target storage medium, rather than blindly writing high-heat data into solid-state drives and low-heat data into mechanical hard drives as in the prior art, only considering data reading and writing while ignoring that different types of data to be written in actual application scenarios may be adapted to different storage media.

[0047] It can be seen that in this application, the initial write weight of the data to be written is adjusted according to the preset write weight policy corresponding to the data type of the data to be written to obtain the current write weight of the data to be written, so that the data importance of the data to be written can be represented according to the current write weight; the current performance weight of each candidate storage medium corresponding to the data to be written is determined according to the initial performance weight of each candidate storage medium and the preset performance weight policy, so that the current performance information of each candidate storage medium that can be used to store the data to be written can be determined, facilitating the selection of a target storage medium with better performance from each candidate storage medium to write data; then, the target storage medium can be selected from each candidate storage medium according to the current performance weight and the current write weight, and the data to be written is written into the target storage medium, realizing the comprehensive selection of a suitable target storage medium for data writing according to the data type of the data to be written and the performance of each writable candidate storage medium. It can achieve load balancing during data writing, optimize the data writing effect, and can also widely be compatible with various data writing application scenarios of different data types, facilitating subsequent data operation and maintenance.

[0048] Based on the above embodiments, the embodiments of the present application will describe the step of adjusting the initial write weight of the data to be written according to the preset write weight policy corresponding to the data type of the data to be written to obtain the current write weight of the data to be written. Specifically, the method of this embodiment includes the following steps:

[0049] Obtain the initial write weight corresponding to the data type and the write weight adjustment step corresponding to the data type in the preset write weight policy; adjust the initial write weight according to the waiting write time of the data to be written and the write weight adjustment step to obtain the current write weight.

[0050] Combined with the foregoing embodiments, in the method of the present application, the initial write weight corresponding thereto and the preset write weight policy can be obtained according to the data type of the data to be written. The preset write weight policy further includes a write weight adjustment step, which is used to adjust the initial write weight according to the relevant information of the data to be written. Among them, the relevant information of the data to be written may include, but is not limited to, the time that the data to be written has waited to be written (abbreviated as the waiting write time), the data volume of the data to be written, etc.

[0051] For example, as the waiting write time of the data to be written is longer (the waiting is longer), its initial write weight can be adjusted positively according to the corresponding write weight adjustment step to obtain the current write weight. In addition, the initial write weight can also be adjusted first in combination with the size of the data volume of the data to be written, or after adjusting the initial write weight according to the waiting write time, the current write weight can be adjusted and updated to obtain the latest current write weight, which will not be elaborated here.

[0052] Exemplarily, for ease of understanding, reference can be made to Table 1 shown below. Table 1 is an exemplary weight adjustment table for the data to be written in the data write method based on load balancing of the present application.

[0053]

[0054] Among them, if the data type of the data to be written is real-time analysis data, the initial write weight of the data to be written is 1.0. According to the corresponding weight adjustment policy in the table, the weight of real-time analysis data has nothing to do with the waiting write time. Therefore, for real-time analysis data, its current write weight can always be the initial write weight (1.0).

[0055] Furthermore, if the data type of the data to be written is a linked image, its initial writing weight is 0.95. According to the corresponding weight adjustment strategy in the table, the weight of the linked image is related to the waiting writing time, and the writing weight adjustment step is: if the waiting writing time of the linked image waits for every 100 ms, the initial writing weight can be increased by 0.05 until the weight increase reaches the preset increase threshold (0.05). According to this example strategy, it is equivalent to that even if the waiting writing time of the linked image reaches 200 ms, 300 ms or more, the initial writing weight of the linked image can only be increased by at most 0.05. If the waiting writing time of the linked image is less than 100 ms, the actual adjusted weight can be determined according to the ratio of the waiting writing time to 100 ms. In summary, the mathematical expression of the current writing weight w1 of the linked image can be: w1 = 0.95 + min(0.05, 0.05 * (x / 100)).

[0056] For the initial writing weights and predicted writing weight strategies corresponding to other data types, it can be exemplarily described in combination with Table 1 and the above examples, and will not be elaborated here.

[0057] Based on the above embodiments, the embodiments of the present application will describe the steps of determining the current performance weights of each candidate storage medium according to the initial performance weights and preset performance weight strategies of each candidate storage medium corresponding to the data to be written. Specifically, the method of this embodiment includes the following steps:

[0058] Obtain the initial performance weights corresponding to each candidate storage medium and the performance weight adjustment steps corresponding to each candidate storage medium in the preset performance weight strategy; respectively adjust the initial performance weights corresponding to each candidate storage medium according to the current performance of each candidate storage medium and the corresponding performance weight adjustment steps to obtain the current performance weights of each candidate storage medium.

[0059] Combined with the foregoing embodiments for description, the method for determining the initial performance weight and preset performance weight strategy of the storage medium in the present application can be similarly referred to the method for determining the initial writing weight and preset writing weight strategy of the data to be written according to the data type in the foregoing embodiments for description.

[0060] In the method of the present application, the corresponding initial performance weight and preset performance weight strategy can be obtained according to the type of the storage medium. The preset performance weight strategy also includes a performance weight adjustment step, and the performance weight adjustment step is used to adjust the initial performance weight according to the relevant information of the storage medium. Among them, the relevant information of the storage medium can include but is not limited to the remaining space of the storage medium, the hard disk read and write (IO) utilization rate, the hard disk health status, etc.

[0061] For example, if there is a large amount of remaining space in the storage medium, the initial performance weight can be adjusted according to the corresponding performance weight adjustment step to obtain the current performance weight. In addition, it can also be adjusted in combination with relevant information such as the hard disk I / O utilization rate and health status of the storage medium to obtain the current write weight, which will not be elaborated here.

[0062] Exemplarily, for ease of understanding, reference can be made to Table 2 below. Table 2 is a weight adjustment table of an exemplary storage medium in the data writing method based on load balancing of the present application.

[0063]

[0064] Among them, NVME, SATA, and SAS are all storage interface protocols. For a storage medium of NVME solid-state drive type, the initial write weight of the storage medium is 3.5. According to the corresponding weight adjustment strategy in the table, if there is 100 GB remaining in the storage medium, its initial write weight can be increased by 0.1, with a maximum increase of the preset increase threshold of 0.5 (equivalent to that even if there is 1 TB remaining in the storage medium, the weight value of this item can only be increased by a maximum of 0.5). In addition, if the hard disk I / O utilization rate is considered, after the I / O utilization rate is greater than 50%, for every 10% increase, the initial performance weight is increased by 0.1, and the maximum increase of this item is 0.5 weight value. Similarly, if the hard disk health status is considered, the corresponding weight value can be selected for multiplication according to the health status of the hard disk. If the health status is poor, it means that the hard disk cannot be used for current data writing. For the above hard disk-related information, one or more can be selected to adjust the initial performance weight of the hard disk, which is not limited here. If the above three examples are combined for judgment, the weight adjustment strategies corresponding to the three examples need to be integrated. The mathematical expression of the current performance weight (w2) of the NVME hard disk can be:

[0065] w2 = (3.5 + min(0.5, 0.1*(x / 100)) + max(0, (100 - y - 50) / 10)*0.1)*z.

[0066] It should be noted that in some implementation manners of scenarios, the initial performance weight of the mechanical hard disk can also be set to be greater than that of the solid-state hard disk, which will not be elaborated here.

[0067] Based on the above embodiments, the embodiments of the present application will describe the steps of obtaining the initial performance weights corresponding to each candidate storage medium and the performance weight adjustment steps corresponding to each candidate storage medium in the preset performance weight strategy. Specifically, the method of this embodiment includes the following steps:

[0068] Obtain the remaining storage space, utilization rate, and storage medium health status of each candidate storage medium; respectively determine the performance weight adjustment step size of each candidate storage medium from a preset performance weight policy according to the remaining storage space, utilization rate, and storage medium health status.

[0069] Combined with the foregoing embodiments for illustration, according to the foregoing examples, the relevant information of the storage medium may include the storage medium type (solid state, mechanical, etc.), storage interface protocol (NVME, SATA, etc.), remaining storage space, utilization rate (IO utilization rate), and storage medium health status, etc. Among them, the storage medium type and storage interface protocol can characterize their priorities through the values of the initial write weights. Therefore, in the specific implementation process, one or more relevant information can be selected from the relevant information such as the remaining storage space, utilization rate (IO utilization rate), and storage medium health status, and the initial performance weight can be adjusted in combination with the corresponding performance weight adjustment step size.

[0070] Among them, obtaining the remaining storage space, utilization rate, and storage medium health status of each candidate storage medium can refer to a variety of existing methods, which are not limited here. For example, the hard disk status can be obtained through the smartctl command, and the health status level can be divided according to the obtained hard disk status result and other methods.

[0071] Exemplarily, this embodiment mainly combines three hard disk-related information, namely the remaining storage space, utilization rate, and storage medium health status, to determine the performance weight adjustment step size of the hard disk. For example, combined with the foregoing example table, it can be seen that the minimum performance weight of the SSD type hard disk available for data writing is 3.0 * 0.6 = 1.8, and the maximum performance weight is (3.5 + 0.5 + 0.5) * 1 = 4.5; the minimum performance weight of the HDD type hard disk is 0.3 * 0.6 = 0.18, and the maximum performance weight is (0.5 + 0.5 + 0.5) * 1 = 1.5. Among them, the current performance weight of the hard disk can be refreshed regularly or irregularly, which is not limited here.

[0072] Based on the above embodiments, the embodiments of the present application illustrate the steps of selecting a target storage medium from each candidate storage medium according to the current performance weight and the current write weight. Specifically, the method of this embodiment includes the following steps:

[0073] Determine the first storage medium and the second storage medium among the candidate storage media according to the current performance weight, where the performance of the first storage medium is better than that of the second storage medium; determine the data priority of the data to be written according to the current write weight; in response to the data priority of the data to be written being the first priority, determine the first storage medium as the target storage medium and write the data to be written into the first storage medium; in response to the data priority of the data to be written being the second priority, determine the second storage medium as the target storage medium and write the data to be written into the second storage medium, where the first priority is higher than the second priority.

[0074] In combination with the foregoing embodiments, after calculating the current performance weights of the candidate storage media according to the foregoing embodiments, the target storage medium can be selected from the candidate storage media according to the current performance weight. The first priority may correspond to the first storage medium, and the second priority may correspond to the second storage medium.

[0075] On the one hand, a logic for selecting a storage medium is exemplarily provided here, and the method may include but is not limited to: overall combining the load balancing principle, and according to the current performance weight, the candidate storage media can be divided into a first storage medium (current performance weight in [1.8, 4.5]) and a second storage medium (current performance weight in [0.18, 1.5]); and determining the data priority of the data to be written as the first priority (current write weight in (0.5 - 1.0]) or the second priority (current write weight in [0.2 - 0.5]) according to the current write weight. If the data to be written is of the first priority, the first storage medium is used as the target storage medium; if the data to be written is of the second priority, the second storage medium is used as the target storage medium. That is to say, for the data to be written with the current write weight in the range of (0.5 - 1.0], a hard disk with the current performance weight in [1.8, 4.5] can be selected, and a hard disk with a higher current performance weight among [1.8, 4.5] can be preferentially selected; for the data to be written with the current write weight in the range of [0.2 - 0.5], a hard disk with the performance weight in [0.18, 1.5] can be selected, and a hard disk with a higher performance weight is preferentially selected.

[0076] On the other hand, the candidate storage media can also be divided into a first storage medium and a second storage medium according to the type of the storage medium. For example, the first storage medium may be a solid - state drive, while the second storage medium may be a mechanical hard drive. Therefore, it can be referred to as Figure 2 as shown Figure 2It is a schematic diagram of an exemplary storage medium selection process in the data writing method based on load balancing of the present application. Among them, the candidate storage media are divided into a solid-state drive resource pool (including one or more solid-state drives) and a mechanical hard drive resource pool (including one or more mechanical hard drive pools) according to the storage medium type. When receiving the data to be written, calculate the current writing weight of the data to be written according to the method provided in the foregoing embodiment. A writing weight threshold can be preset to distinguish between data of the first priority and data of the second priority, or the writing weight threshold can be determined according to the initial writing weights corresponding to each data type in the foregoing embodiment, which is not limited herein. For example, referring to the foregoing embodiment, the writing weight threshold can be set to 0.5. If the current writing weight of the data to be written is greater than 0.5, the data priority of the data to be written is the first priority, and the target storage medium can be selected from the solid-state drive resource pool. If the current writing weight of the data to be written is less than or equal to 0.5, the data priority of the data to be written is the second priority, and the target storage medium can be selected from the mechanical hard drive resource pool. Among them, when selecting the target storage medium from the solid-state drive resource pool and the mechanical hard drive resource pool, the storage media can be sequentially selected as the target storage medium from largest to smallest according to the current performance weights of the storage media in each resource pool, so that the data is preferentially written into the storage medium with better performance.

[0077] On the basis of the foregoing embodiment, the embodiment of the present application describes the steps of determining the first storage medium and the second storage medium in each candidate storage medium according to the current performance weight and the steps after determining the first storage medium and the second storage medium in each candidate storage medium according to the current performance weight. Specifically, the method of this embodiment includes the following steps:

[0078] Perform priority sorting processing on each first storage medium according to the current performance weight of each first storage medium, and perform priority sorting processing on each second storage medium according to the current performance weight of each second storage medium; in response to the data priority of the data to be written being the first priority, determine the target storage medium from the sorted first storage media; in response to the data priority of the data to be written being the second priority, determine the target storage medium from the sorted second storage media.

[0079] It will be described in combination with the foregoing embodiments. In order to enable data to be preferentially written into a storage medium with better performance. After determining the first storage medium and the second storage medium, the first storage media can be sorted according to the current performance weights of the respective first storage media. For example, the first storage medium with a higher current performance weight (such as an NVME solid-state drive) is sorted before the first storage medium with a lower current performance weight (such as a SATA solid-state drive). Similarly, the second storage media can also be sorted according to the current performance weights of the respective second storage media. For example, the second storage medium with a higher current performance weight (such as a SAS mechanical hard drive) is sorted before the second storage medium with a lower current performance weight (such as a SATA mechanical hard drive).

[0080] If the priority of the data to be written is the first priority, the target storage medium can be sequentially determined from the sorted first storage media. If the priority of the data to be written is the second priority, the target storage medium can be sequentially determined from the sorted second storage media, which will not be elaborated here.

[0081] It should also be noted that if there is only one type of storage medium in the current storage system, such as only solid-state drives or only mechanical hard drives, the step of selecting the storage medium type can be skipped. After sorting the storage media of the same type according to the current performance weights, the target storage medium is selected for data writing, which will not be elaborated here.

[0082] Based on the foregoing embodiments, the embodiments of the present application will describe the steps after selecting the target storage medium from the candidate storage media according to the current performance weight and the current write weight and writing the data to be written into the target storage medium. Specifically, the method of this embodiment includes the following steps:

[0083] Obtain the remaining storage space, utilization rate, and storage medium health status of each candidate storage medium; periodically update the current performance weight according to the remaining storage space, utilization rate, storage medium health status, and a preset performance weight policy.

[0084] It will be described in combination with the foregoing embodiments. In the present application, the current performance weight of each storage medium (or candidate storage medium) can be updated regularly or irregularly. For example, a weight update period can be preset to be 1 minute, that is, every 1 minute, the current performance weights of each storage medium (or candidate storage medium) will be recalculated. The specific calculation method can still refer to the description of the method for calculating the current performance weight in the foregoing embodiments.

[0085] Exemplarily, it can be referred to as Figure 3 as shown Figure 3It is a schematic diagram of an exemplary current performance weight update process in the data writing method based on load balancing of the present application. The current performance weight of all storage media in the storage system or candidate storage media for data to be written can be updated every weight update cycle. One implementable update process can be to obtain the remaining storage space, utilization rate, and storage medium health status of each candidate storage medium according to the drive letters of the storage media; calculate the current performance weight of the storage medium according to the remaining storage space, utilization rate, storage medium health status, and the preset performance weight policy, replace the previous weight value, and then sort the storage media.

[0086] Based on the above embodiments, the embodiments of the present application describe the steps before adjusting the initial write weight of the data to be written according to the preset write weight policy corresponding to the data type of the data to be written to obtain the current write weight of the data to be written. Specifically, the method of this embodiment includes the following steps:

[0087] Determine whether the data to be written is streaming data; if so, and if a data file for the data to be written has been created, write the data to be written into the data file.

[0088] It should be noted that for different types of data to be written, the data writing and data storage methods may be different. Therefore, different methods need to be selected according to the type of data to be written to perform data writing and storage.

[0089] Streaming data refers to a form of data that is dynamically and real-time generated and transmitted. Different from batch data, it is generated in a continuous manner, has a wide range of sources, can include sensors, clickstreams, event logs, etc., and usually has characteristics such as continuity, dynamics, low latency, and high throughput. Common streaming data includes, for example, video streams, audio streams, etc., and common non-streaming data includes, for example, images, texts, etc., which are not limited here. It should be noted that whether the data is streaming data or non-streaming data needs to be determined according to the data transmission method, such as continuous data transmission or one-time data transmission, etc. In some application scenarios, video and audio are not necessarily streaming data, and images and texts are not necessarily non-streaming data, which will not be elaborated here. Specifically, it can be understood through existing related technologies.

[0090] Exemplarily, it can be referred to as Figure 4 as shown Figure 4It is a schematic diagram of the overall process in the data writing method based on load balancing of this application. After receiving the data to be written, it is judged whether the data to be written is streaming data; if so, it is further judged whether the data file corresponding to the data to be written has been created; if it has been created, the data to be written is directly written into the corresponding data file; if it has not been created, the current writing weight of the data to be written is calculated, the target storage medium is selected by referring to the method of the foregoing embodiment, and then a data file is created in the target storage medium to store the data to be written. If the data to be written is not streaming data, its current writing weight can be directly calculated, and the target storage medium is selected by referring to the method of the foregoing embodiment for data writing.

[0091] Specifically, taking the video stream recording data as an example for illustration. The recording data is continuous. After creating a corresponding data file, the recording data needs to be continuously written into the data file. Therefore, the storage medium needs to be selected when starting to request to create a file; before switching the currently written data file, there is no need to select the storage medium again. For some images and real-time analysis data, etc., the data volume is generally small, usually a one-time data transmission, so the storage medium is selected each time a write request is made.

[0092] Based on the above embodiments, it can also be illustrated in the embodiments of this application that the storage system of this application can be a cloud storage system, and one cloud storage node can support multiple storage media (such as the 4 storage media mentioned in the foregoing embodiments). Each storage node can also mount multiple hard disks (such as 36 hard disks). Specifically, it can be default that 9 hard disks of each type are mounted, and the capacity sizes of different hard disks are allowed to be different, thereby forming a resource pool of different types of storage media, which is friendly to small-scale business scenarios. In addition, it also supports that each cloud storage node only mounts 1 type of storage media, and multiple cloud storage nodes are deployed in a cluster to meet large-scale business scenarios.

[0093] Based on the above embodiments, it can also be illustrated in the embodiments of this application that the information obtained and the data calculated during the implementation process of this application can all be displayed through a visual interface. For example, this application can provide a global view of the storage medium performance situation and performance weights. Through the global view, data such as the remaining capacity, hard disk IO utilization rate, and current performance weights of each storage medium can be viewed. When there are problems such as abnormalities or insufficient storage capacity of a certain hard disk, abnormal information can be generated for prompt, which is conducive to timely discovering problems and handling them, and facilitating the daily operation and maintenance of the storage system.

[0094] Further, it should be noted that the execution subject of the data writing method based on load balancing can be a data writing device based on load balancing. For example, the data writing method based on load balancing can be executed by a terminal device, a server, or other processing devices. Among them, the terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the data writing method based on load balancing can be implemented by a processor calling computer-readable instructions stored in a memory.

[0095] Figure 5 is a block diagram of a data writing device based on load balancing shown in an exemplary embodiment of the present application. As Figure 5 shown, the exemplary data writing device 500 based on load balancing includes: a write weight determination module 510, a performance weight determination module 520, and a target determination module 530. Specifically:

[0096] The write weight determination module 510 is configured to adjust the initial write weight of the data to be written according to a preset write weight policy corresponding to the data type of the data to be written, so as to obtain the current write weight of the data to be written.

[0097] The performance weight determination module 520 is configured to determine the current performance weight of each candidate storage medium according to the initial performance weight of each candidate storage medium corresponding to the data to be written and a preset performance weight policy.

[0098] The target determination module 530 is configured to select a target storage medium from each candidate storage medium according to the current performance weight and the current write weight, and write the data to be written into the target storage medium.

[0099] In the exemplary load - balanced data writing device, by adjusting the initial writing weight of the data to be written according to the preset writing weight policy corresponding to the data type of the data to be written, the current writing weight of the data to be written is obtained, thereby being able to characterize the data importance of the data to be written according to the current writing weight; determining the current performance weight of each candidate storage medium according to the initial performance weight of each candidate storage medium corresponding to the data to be written and the preset performance weight policy, thereby being able to determine the current performance information of each candidate storage medium available for storing the data to be written, so as to select a target storage medium with better performance from each candidate storage medium to write the data; then, the target storage medium can be selected from each candidate storage medium according to the current performance weight and the current writing weight, and the data to be written is written into the target storage medium, realizing the comprehensive selection of a suitable target storage medium for data writing according to the data type of the data to be written and the performance of each writable candidate storage medium. It can achieve load balancing during data writing, optimize the data writing effect, and can also be widely compatible with various data - type data writing application scenarios, facilitating subsequent data operation and maintenance.

[0100] It should be noted that the device provided in the above - mentioned embodiment and the method provided in the above - mentioned embodiment belong to the same concept. The specific ways in which each module and unit perform operations have been described in detail in the method embodiment, and will not be repeated here. In actual application, the device provided in the above - mentioned embodiment can, according to needs, allocate the above - mentioned functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not limited here.

[0101] Among them, the functions of each module can be referred to in the embodiment of the load - balanced data writing method, and will not be repeated here.

[0102] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute the program instructions stored in the memory 101 to implement the steps in any of the above - mentioned embodiments of the load - balanced data writing method. In a specific implementation scenario, the electronic device 100 may include, but is not limited to: a microcomputer, a server. In addition, the electronic device 100 may also include mobile devices such as a laptop computer, a tablet computer, etc., which are not limited here.

[0103] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above embodiments of the data writing method based on load balancing. The processor 102 can also be referred to as a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. Additionally, the processor 102 can be implemented jointly by integrated circuit chips.

[0104] In this exemplary electronic device, by adjusting the initial writing weight of the data to be written according to the preset writing weight policy corresponding to the data type of the data to be written, the current writing weight of the data to be written is obtained, so that the data importance of the data to be written can be characterized according to the current writing weight; the current performance weights of the candidate storage media corresponding to the data to be written are determined according to the initial performance weights of the candidate storage media and the preset performance weight policy, so that the current performance information of the candidate storage media that can be used to store the data to be written can be determined, facilitating the selection of a target storage media with better performance from the candidate storage media to write the data; then, the target storage media can be selected from the candidate storage media according to the current performance weight and the current writing weight, and the data to be written is written into the target storage media, realizing the comprehensive selection of a suitable target storage media for data writing according to the data type of the data to be written and the performance of each writable candidate storage media. It can achieve load balancing during data writing, optimize the data writing effect, and can also widely be compatible with various data type data writing application scenarios, facilitating subsequent data operation and maintenance.

[0105] Please refer to Figure 7 , Figure 7 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be run by a processor, and the program instructions 111 are used to implement the steps in any of the above embodiments of the data writing method based on load balancing.

[0106] In this exemplary storage medium, by running the program instructions in the storage medium, the initial write weight of the data to be written is adjusted according to the preset write weight policy corresponding to the data type of the data to be written, and the current write weight of the data to be written is obtained, so that the data importance of the data to be written can be characterized according to the current write weight; the current performance weights of the candidate storage media corresponding to the data to be written are determined according to the initial performance weights of the candidate storage media and the preset performance weight policy, so that the current performance information of the candidate storage media available for storing the data to be written can be determined, facilitating the selection of a target storage medium with better performance from the candidate storage media to write the data; then, the target storage medium can be selected from the candidate storage media according to the current performance weight and the current write weight, and the data to be written is written into the target storage medium, realizing the comprehensive selection of a suitable target storage medium for data writing according to the data type of the data to be written and the performance of each writable candidate storage medium. It can achieve load balancing during data writing, optimize the data writing effect, and can also be widely compatible with various data type data writing application scenarios, facilitating subsequent data operation and maintenance.

[0107] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0108] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments. Their similarities or similarities can be referred to each other. For the sake of brevity, they will not be repeated in this article.

[0109] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0110] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

Claims

1. A data writing method based on load balancing, characterized in that The method includes: Adjusting the initial write weight of the data to be written according to a preset write weight policy corresponding to the data type of the data to be written, to obtain the current write weight of the data to be written; Determining the current performance weights of the candidate storage media corresponding to the data to be written according to the initial performance weights of the candidate storage media corresponding to the data to be written and a preset performance weight policy; Selecting a target storage media from the candidate storage media according to the current performance weight and the current write weight, and writing the data to be written into the target storage media.

2. The method according to claim 1, wherein The adjusting the initial write weight of the data to be written according to a preset write weight policy corresponding to the data type of the data to be written, to obtain the current write weight of the data to be written, includes: Obtaining the initial write weight corresponding to the data type and the write weight adjustment step corresponding to the data type in the preset write weight policy; Adjusting the initial write weight according to the waiting write time of the data to be written and the write weight adjustment step, to obtain the current write weight.

3. The method according to claim 1, characterized in that The determining the current performance weights of the candidate storage media corresponding to the data to be written according to the initial performance weights of the candidate storage media corresponding to the data to be written and a preset performance weight policy, includes: Obtaining the initial performance weights corresponding to the candidate storage media and the performance weight adjustment steps corresponding to the candidate storage media in the preset performance weight policy; Respectively adjusting the initial performance weights corresponding to the candidate storage media according to the current performance of the candidate storage media and the corresponding performance weight adjustment steps, to obtain the current performance weights of the candidate storage media.

4. The method according to claim 3, characterized in that, The obtaining the initial performance weights corresponding to the candidate storage media and the performance weight adjustment steps corresponding to the candidate storage media in the preset performance weight policy, includes: Obtaining the remaining storage space, utilization rate, and storage media health status of each candidate storage media; Respectively determining the performance weight adjustment steps of the candidate storage media from the preset performance weight policy according to the remaining storage space, the utilization rate, and the storage media health status.

5. The method according to claim 1, characterized in that, The selecting a target storage media from the candidate storage media according to the current performance weight and the current write weight, includes: Determining a first storage media and a second storage media among the candidate storage media according to the current performance weight, where the performance of the first storage media is better than that of the second storage media; Determining the data priority of the data to be written according to the current write weight; In response to the data priority of the data to be written being the first priority, determining the first storage media as the target storage media, and writing the data to be written into the first storage media; In response to the data priority of the data to be written being the second priority, determining the second storage media as the target storage media, and writing the data to be written into the second storage media, where the first priority is higher than the second priority.

6. The method according to claim 5, wherein The determining a first storage media and a second storage media among the candidate storage media according to the current performance weight, includes: Perform priority sorting processing on each first storage medium according to the current performance weight of each first storage medium, and perform priority sorting processing on each second storage medium according to the current performance weight of each second storage medium; After determining the first storage medium and the second storage medium in each candidate storage medium according to the current performance weight, the method includes: In response to the data priority of the data to be written being the first priority, determine the target storage medium from the sorted first storage medium; In response to the data priority of the data to be written being the second priority, determine the target storage medium from the sorted second storage medium.

7. The method according to claim 1, characterized in that After selecting the target storage medium from each candidate storage medium according to the current performance weight and the current write weight, and writing the data to be written into the target storage medium, the method further includes: Obtain the remaining storage space, utilization rate, and storage medium health status of each candidate storage medium; Periodically update the current performance weight according to the remaining storage space, the utilization rate, the storage medium health status, and a preset performance weight policy.

8. The method according to claim 1, wherein Before adjusting the initial write weight of the data to be written according to the preset write weight policy corresponding to the data type of the data to be written to obtain the current write weight of the data to be written, the method further includes: Determine whether the data to be written is streaming data; If so, and if a data file of the data to be written has been created, write the data to be written into the data file.

9. An electronic device, characterized in that, Comprising a memory and a processor, the processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images