Business resource allocation method and computing device

Abstract

The embodiment of the invention provides a service resource allocation method and computing equipment, and the method comprises the steps: determining a target resource demand of a target service; wherein the target resource demand is used for indicating a storage resource demand and a computing resource demand of the target service for a distributed storage system; based on the target resource demand, determining an initial resource node used for executing the target service in the distributed storage system; in the execution process of the target service, determining real-time resource occupation information of the target service and real-time resource information of each resource node in the distributed storage system; wherein the real-time resource information comprises real-time storage information and real-time calculation information; and determining a target resource node for executing the target service in the distributed storage system based on the real-time resource information under the condition of determining that an initial resource node does not meet an execution requirement based on the real-time resource occupation information.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and more specifically, to a method for allocating business resources and a computing device. Background Technology

[0002] A distributed storage system is a storage architecture that distributes data across multiple storage nodes to achieve goals such as high reliability, scalability, and high performance, providing data storage services for various businesses.

[0003] Currently, most distributed storage systems adopt a storage-compute separation networking model. With this existing networking model, the high-concurrency access to distributed storage systems from large-scale computing systems with massive amounts of data can easily lead to unreasonable resource allocation for business operations, resulting in increased storage costs. Summary of the Invention

[0004] This application provides at least one method for allocating business resources and a computing device.

[0005] In a first aspect, embodiments of this application provide a method for allocating business resources, including:

[0006] Determine the target resource requirements of the target business; whereby the target resource requirements are used to indicate the target business's storage resource requirements and computing resource requirements for the distributed storage system;

[0007] Based on the target resource requirements, the initial resource nodes for executing the target business are determined in the distributed storage system;

[0008] During the execution of the target service, the real-time resource usage information of the target service and the real-time resource information of each resource node in the distributed storage system are determined; among which, the real-time resource information includes: real-time storage information and real-time computing information;

[0009] If the initial resource node does not meet the execution requirements based on real-time resource occupancy information, the target resource node for executing the target business is determined in the distributed storage system based on real-time resource information.

[0010] This technical solution allows for targeted dynamic adjustment of resource nodes based on real-time resource usage information and the real-time status of the target service, enabling resource nodes executing the target service to dynamically adapt to the target service. This not only improves the resource utilization of the distributed storage system but also enhances the execution efficiency of the target service, ensuring the stability of business operations.

[0011] In one optional implementation, determining the target resource requirements of the target service includes:

[0012] If a first service with a similarity greater than a preset similarity threshold to the target service is identified among the services already executed in the distributed storage system, the target resource requirements of the target service are determined based on the resource value consumed by the first service.

[0013] This technical solution, by determining resource requirements based on the resource consumption of a first service similar to the target service, effectively utilizes historical data to accurately predict the resources needed for the target service. By referencing the resource consumption of similar first services, it avoids blindly allocating resources and reduces resource waste and over-configuration.

[0014] In one optional implementation, the method further includes:

[0015] If no first service similar to the target service is identified among the services already executed in the distributed storage system, the input / output information matching the target service is determined from the historical input / output information; wherein, the historical input / output information is used to indicate the actual read / write performance information generated during the execution of the historically executed services in the distributed storage system;

[0016] Once the matching input and output information is determined, it is fed into the neural network model for processing to obtain the target resource requirements of the target business.

[0017] In this technical solution, matching information is determined from historical input / output data, and the target resource requirements of the target service are derived using a neural network model. Historical input / output data provides read / write performance data generated during actual service execution. The neural network model can predict the target resource requirements of the target service when executed on a distributed storage system based on this data. This ensures that resource allocation fully meets the service's operational needs without causing resource redundancy.

[0018] In one optional implementation, the method further includes:

[0019] In the absence of matching input / output information, determine the service configuration parameters of the target service;

[0020] Based on the business configuration parameters, determine the target resource requirements of the target business.

[0021] This technical solution determines the target resource requirements by identifying the business configuration parameters of the target service. This solves the problem of difficulty in predicting resource requirements when facing entirely new business types or lacking historical data for reference. Consequently, it improves the rationality of resource allocation in distributed storage systems under complex and ever-changing business scenarios and enhances the system's adaptability to unknown business needs.

[0022] In an optional implementation, after determining the target resource node for executing the target service in the distributed storage system, the method further includes:

[0023] Once the target business is completed, obtain the actual resource consumption value of the target business;

[0024] The model parameters of the neural network model are updated based on the actual resource consumption values ​​to obtain the updated neural network model; the updated neural network model is used to predict the resource requirements of the next target business.

[0025] This technical solution addresses the problem of traditional resource demand forecasting's difficulty in dynamically optimizing model parameters based on actual business conditions by updating the neural network model with actual resource consumption values. Actual resource consumption values ​​accurately reflect resource usage during business execution. Updating the neural network model parameters based on these values ​​allows the model to continuously learn and adapt to resource demand characteristics in new business scenarios, thereby more accurately predicting the resource needs of the target business.

[0026] In one optional implementation, if it is determined based on real-time resource occupancy information that the initial resource node does not meet the execution requirements, a target resource node for executing the target service is determined in the distributed storage system based on the real-time resource information, including:

[0027] If the initial resource node does not meet the execution requirements based on real-time resource occupancy information, the specified resource requirements of the target service are determined based on the real-time resource occupancy information; wherein, the specified resource requirements are used to instruct the distributed storage system to complete the storage resource requirements and computing resource requirements of the target service.

[0028] Based on real-time resource information, target resource nodes in the distributed storage system with remaining storage space and computing power greater than the specified resource requirements are identified.

[0029] This technical solution ensures that target services obtain the necessary resource nodes that meet their storage capacity and computing performance requirements by filtering storage nodes with remaining storage space greater than their storage resource needs based on real-time resource information, and then locking onto computing nodes with computing power greater than their computing resource needs based on real-time read / write performance information. This avoids service interruptions due to insufficient resources or waste caused by resource excess, thus improving the resource utilization efficiency of the distributed storage system.

[0030] In an optional implementation, after determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the method further includes:

[0031] Based on real-time resource information, determine the storage space occupancy value of each storage node in the distributed storage system;

[0032] Based on the storage space occupancy of each storage node in the distributed storage system, the storage priority of each storage node is determined.

[0033] After detecting a data storage request, the target storage node is determined based on the storage priority of each storage node, and the data to be stored is written to the target storage node.

[0034] This technical solution addresses the problem of uneven storage node space allocation. When the storage space occupancy of a storage node is too high, it leads to a decline in input / output performance, affecting normal business operations. By transferring data from storage nodes with occupancy exceeding a first threshold (the node to be transferred) to storage nodes with occupancy below a second threshold (the destination node), the space pressure on the node to be transferred can be reduced, allowing it to return to normal operating conditions and ensuring data read / write performance. This improves the stability and reliability of the distributed storage system.

[0035] In an optional implementation, after determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the method further includes:

[0036] Based on real-time resource information and storage priority, the storage node to be transferred and the destination storage node of the storage node to be transferred are determined in the distributed storage system; wherein, the storage node to be transferred is the storage node whose storage space occupancy value is greater than a preset occupancy threshold, and the destination storage node is the storage node with high storage priority.

[0037] Transfer at least a portion of the storage data in the storage node to be transferred to the destination storage node.

[0038] In this technical solution, after acquiring real-time resource information for each resource node, storage priority is determined based on the storage space occupancy value of each storage node. Then, by allocating storage nodes based on storage priority, the resource management level of the distributed storage system is improved. By associating storage space occupancy value with storage priority, nodes with lower occupancy values ​​are prioritized for data storage, preventing excessive data concentration on a few nodes and resulting in a more balanced distribution of storage resources across nodes.

[0039] In an optional implementation, after determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the method further includes:

[0040] If, based on real-time resource information, it is determined that the storage space occupancy of each storage node in the distributed storage system is greater than the preset occupancy threshold, a new storage node is added to the distributed storage system.

[0041] In this technical solution, after acquiring real-time resource information for each resource node, a new storage node is added promptly when the storage space occupancy of all storage nodes exceeds a first occupancy threshold. Excessive storage space occupancy on any storage node reduces input / output performance, leading to slow data read / write operations and impacting business efficiency. Adding new storage nodes rapidly expands the system's storage capacity, providing ample space for the target business's data to be stored, avoiding business delays or data backlogs caused by insufficient storage space, and ensuring efficient data storage and processing.

[0042] In a second aspect, embodiments of this application also provide a computing device, including: a processor and a memory, wherein the memory is used to store program instructions; and the processor is used to execute the program instructions to cause the computing device to perform the steps described in the first aspect, or any possible implementation of the first aspect.

[0043] The resource allocation method provided in this application allocates initial resource nodes to a target service based on its resource requirements. Then, during the execution of the target service, if the initial resource nodes do not meet the execution requirements, dynamic adjustments are made using real-time resource occupancy information of the target service and the implementation resource information of each storage node to determine the target resource node for executing the target service. This allows for targeted dynamic adjustments to resource nodes based on real-time resource occupancy information and the real-time status of resource nodes, ensuring that the resource nodes executing the target service dynamically adapt to the target service. This not only improves the resource utilization of the distributed storage system but also enhances the execution efficiency of the target service, ensuring the stability of business operations.

[0044] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this application and, together with the specification, serve to explain the technical solutions of this application. It should be understood that the following drawings only show some embodiments of this application and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This paper illustrates the architecture of a distributed storage system for a business resource allocation method provided in an embodiment of this application.

[0047] Figure 2 A flowchart illustrating a method for allocating business resources according to an embodiment of this application is shown;

[0048] Figure 3 A flowchart illustrating the workflow of a business resource allocation tool provided in an embodiment of this application is shown.

[0049] Figure 4 A schematic diagram of a business resource allocation device provided in an embodiment of this application is shown;

[0050] Figure 5 A schematic diagram of a computing device provided in an embodiment of this application is shown. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0053] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0054] A distributed storage system is a storage architecture that distributes data across multiple storage nodes to achieve high reliability, scalability, and high performance, providing data storage services for businesses. A distributed storage system includes storage nodes and compute nodes, where:

[0055] Storage nodes are the fundamental components of a distributed storage system, responsible for the actual data storage. These nodes are typically composed of storage devices such as hard drives and solid-state drives. They work together to store data in a distributed manner, ensuring data security and reliability through data redundancy and fault tolerance mechanisms.

[0056] Compute nodes primarily handle data processing and computation tasks. They work in conjunction with storage nodes. When user / task applications require data processing, the compute node retrieves data from the storage node and then utilizes its own computing resources, such as a central processing unit (CPU) and a graphics processing unit (GPU), to perform analysis and computation on the data. Compute nodes can be flexibly expanded or contracted according to business needs to adapt to computational tasks of varying scale and complexity.

[0057] Distributed storage systems generate input / output data during business operations, referring to the data transferred between storage nodes and compute nodes, and between storage nodes and business applications. When a business needs to store data, data is written from the user or compute node to the storage node in the form of storage input / output. When data needs to be retrieved, it is transmitted from the storage node to the compute node or user terminal in the form of storage input / output. The efficiency and stability of storage input / output data transmission directly affect the performance and user experience of the distributed storage system. Therefore, it is necessary to optimize the allocation of storage nodes and compute nodes to optimize the input / output paths, thereby improving the speed and reliability of data transmission.

[0058] Based on the above research, this application provides a method for allocating business resources. Initial resource nodes are allocated to the target business based on its resource requirements. Then, during the execution of the target business, if the initial resource nodes do not meet the execution requirements, adjustments are made dynamically based on the real-time resource occupancy information of the target business and the implementation resource information of each storage node to determine the target resource node for executing the target business. This allows for targeted dynamic adjustments to resource nodes based on the real-time resource occupancy information and the real-time status of the resource nodes, ensuring that the resource nodes executing the target business dynamically adapt to the target business. This not only improves the resource utilization of the distributed storage system but also enhances the execution efficiency of the target business, ensuring the stability of business operations.

[0059] See Figure 1 The diagram shown is an architecture diagram of a distributed storage system for a business resource allocation method provided in this application embodiment, including: a storage management unit 11, a monitoring unit 12, a data management unit 13, and a data analysis unit 14; wherein:

[0060] The storage management unit 11 is connected to the monitoring unit 12 and is used to store the configuration parameters and resource requirement information of the business that has been completed in the distributed storage system.

[0061] The monitoring unit 12 is connected to the data management unit 13 and is used to collect real-time resource usage information of the services being executed by the distributed storage system and monitor the usage of each resource node. Specifically, the monitoring unit can monitor the usage of each resource node corresponding to the computing cluster and the storage cluster. In the computing cluster, resource nodes are computing nodes, and in the storage cluster, resource nodes are storage nodes.

[0062] The data management unit 13 is connected to the data analysis unit 14 and is used to dynamically allocate resource nodes based on the real-time resource usage information of the business being executed and the usage status of each resource node obtained by the monitoring unit.

[0063] The data analysis unit 14 is used to analyze the real-time resource usage information of the currently executing business and to issue alarm information for business processes that are abnormal during execution.

[0064] To facilitate understanding of this embodiment, a method for allocating business resources disclosed in this application will first be described in detail. The execution subject of the business resource allocation method provided in this application is generally a computing device with certain computing capabilities. In some possible implementations, the business resource allocation method can be implemented by the processor calling computer-readable instructions stored in memory.

[0065] See Figure 2 The diagram shows a flowchart of a method for allocating business resources according to an embodiment of this application. The method includes steps S201 to S204, wherein:

[0066] S201. Determine the target resource requirements of the target business; wherein, the target resource requirements are used to indicate the storage resource requirements and computing resource requirements of the target business for the distributed storage system.

[0067] In the embodiments of this application, the target service can be analyzed based on the service configuration parameters of the target service to determine the target resource requirements of the target service.

[0068] Here, the business configuration parameters can be analyzed to determine whether the target business has been completed in the distributed storage system, so as to obtain the analysis results. The analysis results are used to indicate whether the target business has been completed at a historical time.

[0069] Then, based on the analysis results, the target resource requirements of the target business can be determined.

[0070] Storage resource requirements can be understood as the amount of storage space needed for the target business, i.e., the amount of data required to execute the target business. Computing resource requirements can be understood as the computing resources needed during the execution of the target business.

[0071] S202. Based on the target resource requirements, determine the initial resource nodes in the distributed storage system to execute the target business.

[0072] In the embodiments of this application, the target resource requirements can be analyzed by the data analysis unit, and the resource nodes that meet the target resource requirements can be determined in the distributed storage system.

[0073] Then, based on the resource information of the resource nodes, the initial resource nodes for executing the target business can be determined.

[0074] Here, the initial storage node for executing the target service can be determined based on the remaining storage space of the storage node; the initial computing node for executing the target service can be determined based on the current computing performance of the computing node. Then, the initial storage node and the initial computing node are designated as the initial resource nodes.

[0075] S203. During the execution of the target service, determine the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system; wherein, the real-time resource information includes: real-time storage information and real-time computing information.

[0076] In the embodiments of this application, resource information of each resource node can be collected by a monitoring unit to obtain real-time resource information. The resource nodes include storage nodes and computing nodes.

[0077] Here, the hardware resource information of each resource node includes at least one of the following: the running status information of the processor, the memory usage, the running status information of the hard disk, and the running status information of the network card.

[0078] The read / write performance information of each resource node includes at least one of the following: business information of the services performed by each resource node, bandwidth, number of input / output operations per second (IOPS), response time, access mode, access popularity and lifespan, and file / directory information.

[0079] The file / directory information is used to record the read / write performance information of the services executed by each resource node. For example, if the services executed by node 1 are recorded as tasks A, B, and C, when a user needs to find the read / write performance information of task B, they can determine the node 1 that executed the task based on the file / directory information, and then determine the read / write performance information of task B by searching the running log of node 1.

[0080] Here, the monitoring unit can also collect resource usage information of the target service to obtain real-time resource usage information.

[0081] The real-time resource usage information includes: the execution progress of the target service at the time of collection, the storage space occupied by the target service on the storage node, and the computing power usage of the target service.

[0082] S204. If the initial resource node does not meet the execution requirements based on the real-time resource occupancy information, the target resource node for executing the target business is determined in the distributed storage system based on the real-time resource information.

[0083] In the embodiments of this application, firstly, the target business prediction demand information can be predicted based on real-time resource usage information.

[0084] Then, based on the predicted resource information, it is determined whether the initial resource node meets the requirements. Finally, if it is determined that the initial resource node does not meet the requirements, resource nodes that meet the predicted requirements can be identified in the distributed storage system executing the target business based on real-time resource information, and these resource nodes that meet the predicted requirements are designated as the target resource nodes.

[0085] In the embodiments of this application, firstly, the target resource requirements of the target service are determined; wherein, the target resource requirements are used to indicate the storage resource requirements and computing resource requirements of the target service on the distributed storage system; secondly, based on the target resource requirements, the initial resource nodes for executing the target service are determined in the distributed storage system; thirdly, during the execution of the target service, the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system are determined; wherein, the real-time resource information includes: real-time storage information and real-time computing information; finally, if it is determined based on the real-time resource occupancy information that the initial resource nodes do not meet the execution requirements, the target resource nodes for executing the target service are determined in the distributed storage system based on the real-time resource information.

[0086] In the above embodiments, this application allocates initial resource nodes to the target service based on its resource requirements. Then, during the execution of the target service, if the initial resource nodes do not meet the execution requirements of the target service, dynamic adjustments are made based on the real-time resource occupancy information of the target service and the implementation resource information of each storage node to determine the target resource node for executing the target service. This allows for targeted dynamic adjustments to resource nodes based on the real-time resource occupancy information of the target service and the real-time status of the resource nodes, ensuring that the resource nodes executing the target service dynamically adapt to the target service. This not only improves the resource utilization of the distributed storage system but also enhances the execution efficiency of the target service, ensuring the stability of service operations.

[0087] In one optional implementation, the target resource requirements of the target service are determined, specifically including the following steps:

[0088] If a first service with a similarity greater than a preset similarity threshold to the target service is identified among the services already executed in the distributed storage system, the target resource requirements of the target service are determined based on the resource value consumed by the first service.

[0089] In the embodiments of this application, it is possible to determine the services that the distributed storage system executing the target service has already completed. Then, among the services that have already been completed, a first service whose similarity to the target service is greater than a preset similarity threshold can be identified.

[0090] Here, firstly, a first service of the same type as the target service can be identified from the already completed services. Then, the similarity between the operating parameters of the first service and the operating parameters of the target service is determined. Finally, the first service whose operating parameters are more similar to the target service than a preset similarity threshold is identified as the first service.

[0091] Here, the preset similarity threshold can be set to 1, that is, among the services that have been completed in the distributed storage system, the service that is exactly the same as the target service (service parameters) is identified as the first service.

[0092] Then, the target resource requirements of the first service can be determined based on the actual computing and storage resources consumed during the execution of the first service.

[0093] In the above embodiments, the method of determining the resource requirements based on the resource consumption of a first service similar to the target service can effectively utilize historical data and accurately predict the resources needed for the target service. By referring to the resource consumption of similar first services, blind resource allocation can be avoided, reducing resource waste and over-configuration.

[0094] In one optional implementation, the target resource requirements of the target service are determined, specifically including the following steps:

[0095] First, if no first service with a similarity greater than a preset similarity threshold to the target service is identified among the services already executed in the distributed storage system, the input / output information matching the target service is determined from the historical input / output information; wherein, the historical input / output information is used to indicate the actual read / write performance information generated during the execution of historically executed services in the distributed storage system;

[0096] Then, once the matching input and output information is determined, the matching input and output information is input into the neural network model for processing to obtain the target resource requirements of the target business.

[0097] In the embodiments of this application, it can be determined that the distributed storage system executing the target service has already completed the service. Then, if no first service with a similarity greater than a preset similarity threshold to the target service is identified among the already completed services (i.e., no service similar to the target service is identified), the input / output information matching the target service is determined from the historical input / output information.

[0098] First, we can identify the historical input / output data of the distributed storage system that does not execute the target service. This historical input / output data refers to the input / output information generated by the distributed storage system executing services other than those executing the target service.

[0099] Secondly, a second service with the same service type as the target service can be identified from the services corresponding to the historical input and output data.

[0100] Secondly, determine the similarity between the operating parameters of the second service and the operating parameters of the target service. Finally, the second service whose operating parameters are more similar to those of the target service than a preset second similarity threshold can be identified as the second service.

[0101] Here, the preset second similarity threshold can be set to 1, that is, among the services that have been completed by the distributed storage system that does not execute the target service, the service that is exactly the same as the target service (service parameters) is identified as the second service.

[0102] After identifying the second service, its input / output information can be determined from historical input / output data. Then, this input / output information can be matched with that of the target service.

[0103] Here, input and output information that matches the target business can be input into the neural network model for processing to obtain the predicted resource requirements of the neural network model, and the predicted resource requirements can be determined as the target resource requirements of the target business.

[0104] In the above implementation, matching information is determined from historical input / output information, and the target resource requirements of the target service are derived using a neural network model. Historical input / output information provides read / write performance data generated during actual service execution, and the neural network model can predict the target resource requirements of the target service when executed on the distributed storage system based on this data. This ensures that resource allocation fully meets the service's operational needs without causing resource redundancy.

[0105] In an optional implementation, the following steps are also included:

[0106] First, in the absence of matching input and output information, determine the service configuration parameters of the target service;

[0107] Then, based on the business configuration parameters, determine the target resource requirements of the target business.

[0108] In the embodiments of this application, no matching input / output information was determined, that is, none of the distributed storage systems have executed a business similar to the target business.

[0109] Here, the target resource requirements of the target business can be determined in the following way:

[0110] Method 1: First, the operating parameters of historical services can be determined. Second, based on the historical operating parameters of the historical services and the operating parameters of the target service, the historical resource requirements corresponding to the historical operating parameters whose similarity to the target service's operating parameters is greater than a preset third similarity threshold can be determined. Finally, the target resource requirements can be determined based on these historical resource requirements. Here, the historical resource requirements refer to the actual resource requirements determined after the historical services were executed.

[0111] Here, the differences between historical operating parameters and the operating parameters of the target service can be identified. Then, based on historical resource requirements and the aforementioned differences, the target resource requirements are determined.

[0112] Method 2: The target resource requirements of the target business can be estimated based on the operating parameters of the target business.

[0113] In the above implementation, the target service's configuration parameters are determined, and the target resource requirements are determined based on these parameters. This solves the problem of difficulty in predicting resource requirements when facing entirely new service types or lacking historical data for reference. Consequently, it improves the rationality of resource allocation in distributed storage systems under complex and ever-changing business scenarios and enhances the system's adaptability to unknown services.

[0114] In an optional implementation, after determining the target resource node for executing the target service in the distributed storage system, the following steps are also included:

[0115] First, after the target business is completed, obtain the actual resource consumption value of the target business;

[0116] Then, the model parameters of the neural network model are updated based on the actual resource consumption values ​​to obtain the updated neural network model; the updated neural network model is used to predict the resource requirements of the next target business.

[0117] In the embodiments of this application, when the target service is a service that has not been executed by the distributed storage system, the actual resource consumption value of the target service can be determined.

[0118] Next, the input and output information of the target service during execution can be determined. This input and output information can then be used as input features, and the actual resource consumption value of the target service can be used as labels to obtain training samples.

[0119] After obtaining the training samples, the neural network model can be updated using these training samples to obtain the updated neural network model.

[0120] The above implementation addresses the problem of traditional resource demand forecasting's difficulty in dynamically optimizing model parameters based on actual business conditions by updating the neural network model with actual resource consumption values. Actual resource consumption values ​​accurately reflect resource usage during business execution. Updating the neural network model parameters based on these values ​​allows the model to continuously learn and adapt to resource demand characteristics in new business scenarios, thereby more accurately predicting the resource demands of the target business.

[0121] In an optional implementation, if it is determined based on real-time resource occupancy information that the initial resource node does not meet the execution requirements, a target resource node for executing the target service is determined in the distributed storage system based on the real-time resource information. This specifically includes the following steps:

[0122] First, if the initial resource node does not meet the execution requirements based on real-time resource occupancy information, the specified resource requirements of the target service are determined based on the real-time resource occupancy information; wherein, the specified resource requirements are used to instruct the distributed storage system to complete the storage resource requirements and computing resource requirements of the target service.

[0123] Then, based on real-time resource information, target resource nodes in the distributed storage system with remaining storage space and computing power greater than the specified resource requirements are determined.

[0124] In the embodiments of this application, the resource nodes of the distributed storage system include storage nodes and computing nodes. The space occupancy (i.e., space occupancy value) of each storage node can be determined based on real-time resource information.

[0125] Next, the remaining storage space of each storage node can be determined based on the space occupancy. After determining the remaining storage space of each storage node, storage nodes whose remaining storage space exceeds the predicted storage demand from the target business's forecasted demand information can be identified.

[0126] If there are multiple storage nodes that predict storage demand based on the remaining storage space exceeding the predicted demand of the target business, the storage node with the largest remaining storage space can be selected as the storage node for the target business.

[0127] If there are no storage nodes that predict storage demand from the forecast demand information that are greater than the target business's remaining storage space, a new storage node can be added to the distributed storage space, and the new storage node can be designated as the target storage node for the target business.

[0128] Here, the computing power of each computing node can be determined based on real-time read / write performance information. Then, among the computing nodes, those whose computing power exceeds the predicted computing power requirement from the predicted demand information of the target business can be identified.

[0129] If there are multiple computing nodes that predict storage requirements from the predicted demand information that exceed the target business's computing power, the computing node with the strongest computing power can be selected as the target computing node for the target business.

[0130] If there are no computing nodes that can predict storage requirements based on the predicted demand information that exceeds the target business's computing capacity, new computing nodes can be added to the distributed storage space, and these new computing nodes can be designated as the target computing nodes for the target business.

[0131] Next, the target storage node and target computing node of the target business determined in the above steps are identified as the target resource nodes of the target business.

[0132] Here, real-time resource information can also be analyzed to determine whether there are any anomalies in the services executed by each resource node. If any service is found to be in an abnormal state, an alarm message will be generated for that service.

[0133] Here, while generating alarm information, it can also be determined whether the distributed storage system has executed any business operations similar to those currently in an abnormal state. If the distributed storage system has executed any business operations similar to those currently in an abnormal state, a repair report for the abnormal business is generated based on the operation logs of that business.

[0134] In the above implementation, by filtering storage nodes with remaining storage space greater than the storage resource requirement based on real-time resource information, and then locking computing nodes with computing power greater than the computing resource requirement based on real-time read / write performance information, the system ensures that the target service obtains the target resource node that meets its storage capacity and computing performance requirements. This avoids service interruptions due to insufficient resources or waste caused by resource excess, thus improving the resource utilization efficiency of the distributed storage system.

[0135] In an optional implementation, after determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the following steps are also included:

[0136] First, based on the storage space occupancy of each storage node in the distributed storage system, the storage priority of each storage node is determined;

[0137] Secondly, after detecting a data storage request, the target storage node is determined based on the storage priority of each storage node, and the data to be stored is written to the target storage node.

[0138] In the embodiments of this application, the storage space occupancy value of each storage node in the distributed storage system can be determined based on the real-time resource information in the real-time resource information.

[0139] Storage nodes are prioritized based on their storage space usage, with smaller storage space usage values ​​indicating higher storage priority.

[0140] After a data storage request is detected (i.e., a target service exists), the storage node with the highest storage priority can be identified as the target storage node for the target service.

[0141] Then, the data to be stored for the target business can be determined and written to the target storage node.

[0142] In the above implementation, after obtaining real-time resource information for each resource node, storage priority is determined based on the storage space occupancy value of the storage node. Then, by allocating storage nodes based on storage priority, the resource management level of the distributed storage system is improved. By associating storage space occupancy value with storage priority, nodes with smaller occupancy values ​​are preferentially selected for data storage, preventing excessive data concentration on a few nodes and resulting in a more balanced distribution of storage resources across nodes.

[0143] In an optional implementation, after determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the following steps are also included:

[0144] First, based on real-time resource information and storage priority, the storage node to be transferred and the destination storage node of the storage node to be transferred are determined in the distributed storage system; wherein, the storage node to be transferred is the storage node whose storage space occupancy value is greater than a preset occupancy threshold, and the destination storage node is the storage node with high storage priority.

[0145] Then, at least a portion of the storage data in the storage node to be transferred is transferred to the destination storage node.

[0146] In the embodiments of this application, storage nodes with storage space values ​​greater than a preset occupancy threshold can be determined based on real-time resource information, and these storage nodes can be identified as storage nodes to be transferred.

[0147] The preset occupancy threshold can be set based on the performance of the storage node. For example, when the storage space occupancy rate is greater than 90%, the performance of input / output operations degrades significantly, so the preset occupancy threshold can be set to 90%.

[0148] After determining the storage node to be transferred and the destination storage node, the amount of data to be transferred from the storage node to be transferred can be determined based on the remaining storage space of the destination storage node.

[0149] After determining the amount of data to be transferred, data of the same size as the amount to be transferred can be transferred from the storage node to the destination storage node.

[0150] The above implementation addresses the problem of uneven storage node space allocation. When the storage space occupancy of a storage node is too high, it leads to a decline in input / output performance, affecting the normal operation of services. By transferring data from storage nodes with occupancy exceeding a preset threshold (the storage nodes to be transferred) to the storage node with the highest storage priority (the destination storage node), the space pressure on the storage nodes to be transferred can be reduced, allowing them to return to normal operating conditions and ensuring data read / write performance. This improves the stability and reliability of the distributed storage system.

[0151] In an optional implementation, after determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the following steps are also included:

[0152] If, based on real-time resource information, it is determined that the storage space occupancy of each storage node in the distributed storage system is greater than the preset occupancy threshold, a new storage node is added to the distributed storage system.

[0153] In the embodiments of this application, the storage space occupancy value of each storage node in the distributed storage system can be determined based on the real-time resource information in the real-time resource information.

[0154] Here, the storage space occupied by each storage node is greater than the preset threshold, which means that the operation performance of the storage node's input and output operations will be reduced after new data is stored on all storage nodes.

[0155] At this point, new storage nodes can be added to the distributed storage system to store the data to be stored for the target business.

[0156] In the above implementation, firstly, the storage priority is determined based on the storage space occupancy value of each storage node, and the target storage node is selected to write data according to the priority. This allows data to be stored preferentially in nodes with more space, avoiding the situation where some nodes over-occupy resources while other nodes are idle, thus improving the overall utilization rate of storage resources.

[0157] Secondly, for storage nodes whose storage space occupancy exceeds a preset threshold, transferring some of their data to a destination storage node with higher storage priority can balance the storage pressure of each node in a timely manner and ensure the stable operation of the distributed storage system.

[0158] Finally, when the storage space occupancy of all storage nodes exceeds the preset occupancy threshold, a new storage node is added, enabling the distributed storage system to expand according to actual storage needs. This avoids data storage failures due to insufficient storage space and further enhances the applicability and reliability of the distributed storage system.

[0159] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0160] See Figure 3 The diagram shown is a flowchart of the workflow of the business resource allocation tool provided in this application embodiment, wherein:

[0161] The storage management unit communicates with the distributed storage system to unify the storage nodes, computing nodes, and business information collection interfaces of the distributed storage system.

[0162] The monitoring unit communicates with the interfaces of the computing cluster (i.e., computing nodes in the distributed storage system), the storage cluster (i.e., storage nodes in the distributed storage system), and the services. It is used to collect real-time resource information of resource nodes and real-time read / write performance information of services through the above interfaces and send them to the data management unit.

[0163] Real-time resource information includes hardware information and read / write performance information (i.e., input / output information).

[0164] The data management unit communicates with the data analysis unit to send real-time resource information from resource nodes and real-time read / write performance information of services to the data analysis unit, and receives analysis reports from the data analysis unit. It then performs the following operations: based on the real-time read / write performance information of services, it schedules the data needed by the services in advance at the resource nodes; it adjusts the data on the storage nodes of the distributed storage system; it alerts services in abnormal states; and it allocates target resource nodes to target services.

[0165] The data analysis unit communicates with the neural network model to determine the target resource requirements of the target business based on the neural network model. Then, based on the received real-time resource information of the resource nodes, the real-time read and write performance information of the business, and the target resource requirements, it generates an analysis report and sends it to the data management unit.

[0166] In actual operation, the following effects can be achieved:

[0167] (1) Provides a unified distributed storage interface, which can be connected to different distributed storage systems to realize real-time resource information analysis of different distributed storage systems.

[0168] (2) By collecting real-time resource information of computing nodes and storage nodes, and read and write performance information of services, a neural network model is used to conduct end-to-end detailed analysis and statistics on the storage needs of services, rationally allocate the storage and computing resources required by services, improve storage performance, and reduce storage costs.

[0169] Based on the same inventive concept, this application also provides a business resource allocation device corresponding to the business resource allocation method. Since the principle of the device in this application is similar to the business resource allocation method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0170] Reference Figure 4 The diagram shown is a schematic representation of a service resource allocation device according to an embodiment of this application. The device includes: a first determining module 41, a second determining module 42, a third determining module 43, and a fourth determining module 44; wherein,

[0171] The first determining module 41 is used to determine the target resource requirements of the target service; wherein, the target resource requirements are used to indicate the storage resource requirements and computing resource requirements of the target service for the distributed storage system;

[0172] The second determining module 42 is used to determine the initial resource node for executing the target service in the distributed storage system based on the target resource requirements.

[0173] The third determining module 43 is used to determine the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system during the execution of the target service; wherein, the real-time resource information includes: real-time storage information and real-time computing information;

[0174] The fourth determining module 44 is used to determine, based on the real-time resource information, a target resource node for executing the target service in the distributed storage system when the initial resource node is determined to be non-compliant with the execution requirements based on the real-time resource occupancy information.

[0175] This application's embodiments allocate initial resource nodes to the target service based on its resource requirements. Then, during the execution of the target service, if the initial resource nodes do not meet the execution requirements, adjustments are made dynamically based on the target service's real-time resource occupancy information and the implementation resource information of each storage node to determine the target resource node for executing the target service. This allows for targeted dynamic adjustments to resource nodes based on the target service's real-time resource occupancy information and the real-time status of resource nodes, ensuring that the resource nodes executing the target service dynamically adapt to the target service. This not only improves the resource utilization of the distributed storage system but also enhances the execution efficiency of the target service, ensuring the stability of service operations.

[0176] In one possible implementation, the first determining module is further configured to: determine the target resource requirement of the target service based on the resource value consumed by the first service when a first service with a similarity greater than a preset similarity threshold is identified among the services already executed in the distributed storage system.

[0177] In one possible implementation, the first determining module is specifically configured to: if no first service with a similarity greater than a preset similarity threshold to the target service is determined among the services already executed in the distributed storage system, determine input / output information matching the target service from historical input / output information; wherein, the historical input / output information is used to indicate the actual read / write performance information generated during the execution of historically executed services in the distributed storage system;

[0178] Once the matching input-output information is determined, it is input into a neural network model for processing to obtain the target resource requirements of the target service.

[0179] In one possible implementation, the first determining module is specifically used to: determine the service configuration parameters of the target service when the matching input / output information has not been determined;

[0180] Based on the aforementioned service configuration parameters, the target resource requirements of the target service are determined.

[0181] In one possible implementation, the first determining module is specifically used to: obtain the actual resource consumption value of the target service when the target service is completed;

[0182] The model parameters of the neural network model are updated based on the actual resource consumption value to obtain the updated neural network model; wherein, the updated neural network model is used to predict the resource requirements of the next target business.

[0183] In one possible implementation, the fourth determining module is further configured to: determine the specified resource requirements of the target service based on the real-time resource occupancy information if the initial resource node does not meet the execution requirements based on the real-time resource occupancy information; wherein the specified resource requirements are used to instruct the distributed storage system to complete the storage resource requirements and computing resource requirements of the target service.

[0184] Based on the real-time resource information, target resource nodes in the distributed storage system with remaining storage space and computing power greater than the specified resource requirements are determined.

[0185] In one possible implementation, the third determining module is specifically used to: determine the storage space occupancy value of each storage node in the distributed storage system based on the real-time resource information;

[0186] Based on the storage space occupancy of each storage node in the distributed storage system, the storage priority of each storage node is determined.

[0187] After a data storage request is detected, a target storage node is determined based on the storage priority of each storage node, and the data to be stored is written to the target storage node.

[0188] In one possible implementation, the third determining module is specifically used to: determine the storage node to be transferred in the distributed storage system based on the real-time resource information and the storage priority, and determine the destination storage node of the storage node to be transferred; wherein, the storage node to be transferred is a storage node whose storage space occupancy value is greater than a preset occupancy threshold, and the destination storage node is a storage node with a high storage priority;

[0189] At least a portion of the stored data in the storage node to be transferred is transferred to the destination storage node.

[0190] In one possible implementation, the third determining module is specifically used to: add a new storage node to the distributed storage system when it is determined, based on the real-time resource information, that the storage space occupancy value of each storage node in the distributed storage system is greater than the preset occupancy threshold.

[0191] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0192] Corresponding to Figure 2 In addition to the method for allocating business resources, this application also provides a computing device 500, such as... Figure 5 The diagram shown is a structural schematic of a computing device 500 provided in an embodiment of this application, including:

[0193] The system includes a processor 51, a memory 52, and a bus 53. The memory 52 stores execution instructions and includes main memory 521 and external memory 522. The main memory 521, also called internal memory, temporarily stores the computational data in the processor 51, as well as data exchanged with external memory such as a hard disk. The processor 51 exchanges data with the external memory 522 through the main memory 521. When the computing device 500 is running, the processor 51 communicates with the memory 52 through the bus 53, causing the processor 51 to execute the following instructions:

[0194] Determine the target resource requirements of the target business; wherein, the target resource requirements are used to indicate the storage resource requirements and computing resource requirements of the target business for the distributed storage system;

[0195] Based on the target resource requirements, an initial resource node for executing the target service is determined in the distributed storage system;

[0196] During the execution of the target service, the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system are determined; wherein, the real-time resource information includes: real-time storage information and real-time computing information;

[0197] If the initial resource node does not meet the execution requirements based on the real-time resource occupancy information, a target resource node for executing the target service is determined in the distributed storage system based on the real-time resource information.

[0198] This application also provides a computer-readable storage medium storing a computer program. When a processor runs the computer program, it executes the steps of the business resource allocation method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0199] This application also provides a computer program product that carries program code. The instructions included in the program code can be used to execute the steps of the business resource allocation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0200] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0201] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0202] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0203] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0204] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for allocating business resources, characterized in that, include: Determine the target resource requirements of the target business; wherein, the target resource requirements are used to indicate the storage resource requirements and computing resource requirements of the target business for the distributed storage system; Based on the target resource requirements, an initial resource node for executing the target service is determined in the distributed storage system; During the execution of the target service, the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system are determined; wherein, the real-time resource information includes: real-time storage information and real-time computing information; If the initial resource node does not meet the execution requirements based on the real-time resource occupancy information, a target resource node for executing the target service is determined in the distributed storage system based on the real-time resource information.

2. The method according to claim 1, characterized in that, Determining the target resource requirements of the target service includes: If a first service with a similarity greater than a preset similarity threshold to the target service is identified among the services already executed in the distributed storage system, the target resource requirement of the target service is determined based on the resource value consumed by the first service.

3. The method according to claim 2, characterized in that, The method further includes: If no first service with a similarity greater than a preset similarity threshold to the target service is identified among the services already executed in the distributed storage system, input / output information matching the target service is determined from historical input / output information; wherein, the historical input / output information is used to indicate the actual read / write performance information generated during the execution of historically executed services in the distributed storage system; Once the matching input-output information is determined, it is input into a neural network model for processing to obtain the target resource requirements of the target service.

4. The method according to claim 3, characterized in that, The method further includes: In the absence of matching input / output information, determine the service configuration parameters of the target service; Based on the aforementioned service configuration parameters, the target resource requirements of the target service are determined.

5. The method according to claim 3, characterized in that, After determining the target resource node for executing the target service in the distributed storage system, the method further includes: Upon completion of the target service, obtain the actual resource consumption value of the target service; The model parameters of the neural network model are updated based on the actual resource consumption value to obtain the updated neural network model; wherein, the updated neural network model is used to predict the resource requirements of the next target business.

6. The method according to claim 1, characterized in that, The step of determining a target resource node for executing the target service in the distributed storage system based on the real-time resource occupancy information, when it is determined that the initial resource node does not meet the execution requirements, includes: If, based on the real-time resource occupancy information, it is determined that the initial resource node does not meet the execution requirements, the specified resource requirements of the target service are determined based on the real-time resource occupancy information; wherein, the specified resource requirements are used to instruct the distributed storage system to complete the storage and computing resource requirements of the target service. Based on the real-time resource information, target resource nodes in the distributed storage system with remaining storage space and computing power greater than the specified resource requirements are determined.

7. The method according to claim 1, characterized in that, After determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the method further includes: Based on the real-time resource information, the storage space occupancy value of each storage node in the distributed storage system is determined; Based on the storage space occupancy of each storage node in the distributed storage system, the storage priority of each storage node is determined. After a data storage request is detected, a target storage node is determined based on the storage priority of each storage node, and the data to be stored is written to the target storage node.

8. The method according to claim 7, characterized in that, After determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the method further includes: Based on the real-time resource information and the storage priority, the system determines the storage node to be transferred and the destination storage node of the storage node to be transferred; wherein, the storage node to be transferred is a storage node whose storage space occupancy value is greater than a preset occupancy threshold, and the destination storage node is a storage node with high storage priority. At least a portion of the stored data in the storage node to be transferred is transferred to the destination storage node.

9. The method according to claim 8, characterized in that, After determining the real-time resource occupancy information of the target service and the real-time resource information of each resource node in the distributed storage system, the method further includes: If, based on the real-time resource information, it is determined that the storage space occupancy value of each storage node in the distributed storage system is greater than the preset occupancy threshold, a new storage node is added to the distributed storage system.

10. A computing device, characterized in that, include: Processor and memory; The memory is used to store program instructions; The processor is configured to execute the program instructions to cause the computing device to perform the service resource allocation method as described in any one of claims 1 to 9.

Images