A management and scheduling system and method based on distributed storage

Abstract

The present invention discloses a management and scheduling system and method based on distributed storage, including: based on a distributed storage management platform, statistics on the service opening of each storage area and analyze it to obtain analysis results; dynamically change the business model according to the analysis results, comprehensively match the storage performance and storage available capacity based on the analysis of customer needs in each area, dynamically adjust the business group model of each storage area, open the optimal business group model, and maximize the use of storage performance and capacity resources. The present invention schedules and opens the user business space to the storage node with the best comprehensive ratio of storage performance and storage available capacity, thereby improving the utilization rate of storage resources; in addition, not only the storage resources in this area are modeled, scheduled and allocated, but also the storage nodes in the associated area can be dynamically modeled, managed and scheduled, thereby maximizing the flexibility of the platform dynamic modeling and the utilization rate of storage resources.

Description

Technical Field

[0001] The present invention belongs to the field of storage scheduling management, and in particular relates to a management scheduling system and method based on distributed storage. Background Art

[0002] In actual use, distributed storage often encounters an imbalance between the number of storage accesses and the use of capacity resources. This is mainly because the concurrent performance of the old storage nodes in the early days was insufficient. During use, the business model pre-opened by the platform could not be dynamically adjusted according to the business development, storage performance, and available storage capacity. As a result, a large number of video spaces with short storage time were opened to low-performance storage points, causing the concurrent performance of the storage nodes to reach the bottleneck too early and no more space could be opened, resulting in a large amount of storage space resources being wasted. Summary of the invention

[0003] The purpose of the present invention is to dynamically change the business group model based on the video surveillance cloud platform and distributed storage technology, analyze the customer needs of each region within a certain period of time, comprehensively match the storage performance and available storage capacity, dynamically adjust the business group model of the original region, and form the optimal business group model to improve the platform's space response speed and maximize the use of storage performance and capacity resources.

[0004] To achieve the above object, the present invention provides the following solution: a management and scheduling method based on distributed storage, comprising:

[0005] During the initialization period, the system will form a business model based on manually configured basic data or imported basic business data. Each piece of business data will correspond to a business model. The business model records the types of services opened and the expected or actual opening ratio of different services. Multiple business models form a business group model.

[0006] Based on the distributed storage management platform, the service provisioning status of each storage area is counted and analyzed to obtain the analysis results;

[0007] Dynamic business group model changes are made according to the analysis results. Based on the analysis of customer needs in each region, storage performance and available storage capacity are comprehensively matched, and the business group model of each storage area is dynamically adjusted to form an optimal business group model. The optimal business group is the resource demand that best meets the actual business activation situation, thereby maximizing the use of storage performance and capacity resources.

[0008] Preferably, the process of collecting statistics on the service provisioning status of each storage area and analyzing the same based on the distributed storage management platform includes:

[0009] Based on the distributed storage management platform, database data is read to obtain the customer's historical business activation data, the number of activations of each business model in each storage area is counted, and the data is sorted according to the area and business group type according to the number to obtain the sorting results; according to the sorting results, each area is dynamically modeled to find the storage information and the activated business group models under each area.

[0010] Preferably, the dynamic business group model is changed according to the analysis result, and based on the analysis of customer needs in each region, storage performance and storage available capacity are comprehensively matched, and the process of dynamically adjusting the business group model of each storage area includes:

[0011] Based on the storage information in the target area and the service group models that have been activated, determine whether the service group model with the largest number of activations in the previous period exists. If it does exist, count the remaining number of service group types.

[0012] If it does not exist, check whether there is suitable storage in the target area to open this type of business group model; if so, after dynamically modifying the business group model of the storage area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

[0013] Preferably, the process of counting the remaining number of service group types also includes, when the remaining number of the service group type is less than the number of historical type service groups activated, deducting resources from other service group models in the target area and supplementing them to the service group model.

[0014] Preferably, in the process of searching whether there is suitable storage in the target area to open this type of business group model, if not, search whether there is suitable storage in the shared area; if so, after dynamically modifying the business group model of the shared area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

[0015] Preferably, in the process of searching whether there is suitable storage under the shared area to open this type of business group model, if not, search whether there is suitable storage in the upper-level area; if so, after dynamically modifying the business group model of the shared area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

[0016] A management and scheduling system based on distributed storage, comprising:

[0017] External platform and external interface corresponding to the external platform, internal system and external interface corresponding to the internal system, application layer, service gateway, capability layer, data layer, resource layer;

[0018] The internal system and external platform respectively realize information interaction between the management and scheduling system and the outside through the external interface and the external interface;

[0019] The application layer is used to implement storage, management and scheduling of data information by dividing areas.

[0020] Preferably, the external platform includes at least a storage vendor platform, and the external interface includes at least a streaming media storage management interface;

[0021] The internal system at least includes a streaming media storage system, a video access cloud platform, a unified operation and maintenance, and a business management platform;

[0022] The external interface at least includes a streaming media storage management interface, a streaming media storage service interface, a storage fault reporting interface, and a storage service acceptance interface;

[0023] Wherein, the storage vendor platform is correspondingly connected to the streaming media storage management interface;

[0024] The streaming media storage system, video access cloud platform, unified operation and maintenance, and business management platform are respectively connected to the streaming media storage management interface, streaming media storage business interface, storage fault reporting interface, and storage business acceptance interface.

[0025] Preferably, the application layer includes a user management unit, a storage management unit, a log management unit, a report statistics unit, a resource scheduling unit, and a monitoring alarm unit;

[0026] The user management unit is used to implement customer management, administrator management and operator management;

[0027] The storage management unit is used to implement storage alarm, local storage management, data node management, local node management, disk management, local space management, space management, streaming media storage management, whitelist management, and information source management;

[0028] The log management unit is used to manage system logs, administrator operation logs, and customer operation logs;

[0029] The report statistics unit is used to realize storage data statistics, customer data statistics, business data statistics and pre-order management;

[0030] The resource scheduling unit is used to realize shared configuration, nearby storage, priority configuration, business resource acceptance and storage modeling;

[0031] The monitoring and alarm unit is used for data node monitoring, streaming media storage monitoring, storage device alarms, streaming media storage alarms, smart operation and maintenance, smart storage, and business fault acceptance.

[0032] Preferably, the resource layer is used for edge storage, cloud storage, cloud storage, software-defined storage, Tianyi cloud storage and block data docking.

[0033] The present invention discloses the following technical effects:

[0034] The present invention provides a management and scheduling system and method based on distributed storage. Dynamic business group modeling is performed based on a distributed storage management platform to realize dynamic allocation of storage resources and improve storage resource utilization. First, the performance, capacity resources and activated business group models of subordinate storage are managed based on the distributed storage management platform, and the business activation status of each area in the corresponding time period is counted and analyzed. The business group model of the original area is dynamically adjusted according to the analysis result, and the commonly used business group type is activated to avoid the fact that the business group type pre-activated by the platform is not suitable for the actual space business activation status, resulting in reduced platform response efficiency. The temporary splitting of the program to generate a business group cannot fully consider the total ratio of storage performance and storage available capacity, which is easy to cause storage resource waste. Dynamic modeling fully realizes the maximization of storage performance and capacity resource utilization.

[0035] The present invention schedules user business space to storage nodes with the best comprehensive ratio of storage performance and available storage capacity, thereby improving the utilization rate of storage resources. In addition, not only the storage resources in the local area are modeled, scheduled and allocated, but also the storage nodes in the associated areas can be dynamically modeled, managed and scheduled, thereby maximizing the flexibility of the platform's dynamic modeling and the utilization rate of storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1 As shown, the present invention provides a management and scheduling method based on distributed storage, comprising:

[0042] During the initialization period, the system will form a business model based on manually configured basic data or imported basic business data. Each piece of business data will correspond to a business model. The business model records the types of services opened and the expected or actual opening ratio of different services. Multiple business models form a business group model.

[0043] Based on the distributed storage management platform, the service provisioning status of each storage area is counted and analyzed to obtain the analysis results;

[0044] Dynamic business group model changes are made according to the analysis results. Based on the analysis of customer needs in each region, storage performance and available storage capacity are comprehensively matched, and the business group model of each storage area is dynamically adjusted to form an optimal business group model. The optimal business group is the resource demand that best meets the actual business activation situation, thereby maximizing the use of storage performance and capacity resources.

[0045] To further optimize the solution, the process of collecting statistics on the service activation status of each storage area and analyzing it based on the distributed storage management platform includes:

[0046] Based on the distributed storage management platform, database data is read to obtain the customer's historical business activation data, the number of activations of each business model in each storage area is counted, and the data is sorted according to the area and business group type according to the number to obtain the sorting results; according to the sorting results, each area is dynamically modeled to find the storage information and the activated business group models under each area.

[0047] Further optimizing the solution, dynamically changing the business group model according to the analysis results, comprehensively allocating storage performance and storage available capacity based on the analysis of customer needs in each region, and dynamically adjusting the business group model of each storage region includes:

[0048] Based on the storage information in the target area and the business group models that have been activated, determine whether the business group model with the largest number of activations in the previous period exists. If it does exist, count the remaining number of business group types. In the process of counting the remaining number of business group types, when the remaining number of the business group type is less than the number of historical type business groups activated, deduct resources from other business group models in the target area and supplement them to the business group model.

[0049] If it does not exist, check whether there is suitable storage in the target area to open this type of business group model; if so, dynamically modify the business group model of the storage area and pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

[0050] In the process of searching whether there is suitable storage in the target area to open this type of business group model, if not, search whether there is suitable storage in the shared area; if so, after dynamically modifying the business group model of the shared area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

[0051] In the process of searching whether there is suitable storage under the shared area to open this type of business group model, if not, search whether there is suitable storage in the upper-level area; if so, after dynamically modifying the business group model of the shared area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

[0052] The process of counting the remaining number of service group types also includes, when the remaining number of the service group type is less than the number of historical type service groups activated, deducting resources from other service group models in the target area to supplement the service group model.

[0053] To further optimize the solution, the following is a specific discussion based on the above process:

[0054] 1. The system starts a scheduled task, reads database data, and obtains customer service activation status within a week;

[0055] 2. Count the number of models activated in each target area, and sort them from most to least according to the area and business group type;

[0056] 3. Dynamically model each area one by one, find the storage IP information under the area and the business group model that has been opened for storage;

[0057] 4. If the business group model with the largest number of activations in the previous cycle already exists in this area, the remaining number of this business group type will be counted. If the remaining number is less than the number of business groups of this type activated last week, resources will be deducted from other models in this storage to supplement this model.

[0058] 5. If the business group model with the largest number of activations in the previous cycle does not exist in this area, search for suitable storage in this area to activate this type of business model (comprehensively allocate storage performance and available storage capacity; low-performance storage is not suitable for activating business group models with low bit rates and few days).

[0059] 6. If there is suitable storage in the region to open this type of business group model, dynamically modify the original storage model and open this type of model in advance. The number of openings is the same as the statistical result;

[0060] 7. If there is no suitable storage in the area to open this type of business group model, search the shared area. If there is suitable storage in the shared area, dynamically modify the original model stored in the shared area and open this type of model in advance. The number of openings is the same as the statistical result.

[0061] 8. If there is no suitable storage in the shared area to open this type of business model, search the parent area. If there is suitable storage in the parent area, dynamically modify the original model stored in the parent area and open this type of model in advance. The number of openings is the same as the statistical result.

[0062] like Figure 2 As shown, the present invention also provides a management and scheduling system based on distributed storage, comprising:

[0063] External platform and external interface corresponding to the external platform, internal system and external interface corresponding to the internal system, application layer, service gateway, capability layer, data layer, resource layer;

[0064] The internal system and external platform respectively realize information interaction between the management and scheduling system and the outside through the external interface and the external interface;

[0065] The application layer is used to implement storage, management and scheduling of data information by dividing areas.

[0066] In a further optimization scheme, the external platform includes at least a storage vendor platform, and the external interface includes at least a streaming media storage management interface;

[0067] The internal system at least includes a streaming media storage system, a video access cloud platform, a unified operation and maintenance, and a business management platform;

[0068] The external interface at least includes a streaming media storage management interface, a streaming media storage service interface, a storage fault reporting interface, and a storage service acceptance interface;

[0069] Wherein, the storage vendor platform is correspondingly connected to the streaming media storage management interface;

[0070] The streaming media storage system, video access cloud platform, unified operation and maintenance, and business management platform are respectively connected to the streaming media storage management interface, streaming media storage business interface, storage fault reporting interface, and storage business acceptance interface.

[0071] Further optimizing the scheme, the application layer includes a user management unit, a storage management unit, a log management unit, a report statistics unit, a resource scheduling unit, and a monitoring alarm unit;

[0072] The user management unit is used to implement customer management, administrator management and operator management;

[0073] The storage management unit is used to implement storage alarm, local storage management, data node management, local node management, disk management, local space management, space management, streaming media storage management, whitelist management, and information source management;

[0074] The log management unit is used to manage system logs, administrator operation logs, and customer operation logs;

[0075] The report statistics unit is used to realize storage data statistics, customer data statistics, business data statistics and pre-order management;

[0076] The resource scheduling unit is used to realize shared configuration, nearby storage, priority configuration, business resource acceptance and storage modeling;

[0077] The monitoring and alarm unit is used for data node monitoring, streaming media storage monitoring, storage device alarms, streaming media storage alarms, smart operation and maintenance, smart storage, and business fault acceptance.

[0078] To further optimize the solution, the resource layer is used for edge storage, cloud storage, cloud storage, software-defined storage, Tianyi cloud storage and block data docking.

[0079] The embodiments described above are only descriptions of the preferred modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A management and scheduling method based on distributed storage, characterized in that: include: During the initialization period, the system will form a business model based on manually configured basic data or imported basic business data. Each piece of business data will correspond to a business model. The business model records the types of services opened and the expected or actual opening ratio of different services. Multiple business models form a business group model. Based on the distributed storage management platform, the service provisioning status of each storage area is counted and analyzed to obtain the analysis results; Dynamically change the business group model according to the analysis results, analyze the customer needs of each region, comprehensively match the storage performance and available storage capacity, dynamically adjust the business group model of each storage area, and form an optimal business group model; the optimal business group is the resource demand that best meets the actual business activation situation, and maximizes the use of storage performance and capacity resources; The process of counting and analyzing the service activation status of each storage area based on the distributed storage management platform includes: Based on the distributed storage management platform, the database data is read to obtain the historical service activation data of the customer, the activation quantity of each service model in each storage area is counted, and the order is obtained according to the area and service group type and the quantity; according to the order result, each area is dynamically modeled, and the storage information and the activated service group model under each area are searched; The process of dynamically changing the business group model according to the analysis results, comprehensively allocating storage performance and storage available capacity based on the analysis of customer needs in each region, and dynamically adjusting the business group model of each storage area includes: Based on the storage information in the target area and the service group models that have been activated, determine whether the service group model with the largest number of activations in the previous period exists. If it does exist, count the remaining number of service group types. If it does not exist, find out whether there is suitable storage in the target area to open this type of business group model; if so, after dynamically modifying the business group model of the storage area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of opened optimal business group models is the same as the statistical result; In the process of searching whether there is suitable storage in the target area to open this type of business group model, if not, search whether there is suitable storage in the shared area; if so, after dynamically modifying the business group model of the shared area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

2. The management and scheduling method based on distributed storage according to claim 1 is characterized in that: The process of counting the remaining number of service group types also includes, when the remaining number of the service group type is less than the number of historical type service groups activated, deducting resources from other service group models in the target area to supplement the service group model.

3. The management and scheduling method based on distributed storage according to claim 1 is characterized in that: In the process of searching whether there is suitable storage under the shared area to open this type of business group model, if not, search whether there is suitable storage in the upper-level area; if so, after dynamically modifying the business group model of the shared area, pre-open the optimal business group model to maximize storage performance and capacity resource utilization; wherein the number of activated optimal business group models is the same as the statistical result.

4. A management and scheduling system based on distributed storage, used to execute the management and scheduling method in claim 1, characterized in that: include, External platform and external interface corresponding to the external platform, internal system and external interface corresponding to the internal system, application layer, service gateway, capability layer, data layer, resource layer; The internal system and external platform respectively realize information interaction between the management and scheduling system and the outside through the external interface and the external interface; The application layer is used to implement storage, management and scheduling of data information by dividing areas.

5. The management and scheduling system based on distributed storage according to claim 4 is characterized in that: The external platform at least includes a storage vendor platform, and the external interface at least includes a streaming media storage management interface; The internal system at least includes a streaming media storage system, a video access cloud platform, a unified operation and maintenance, and a business management platform; The external interface at least includes a streaming media storage management interface, a streaming media storage service interface, a storage fault reporting interface, and a storage service acceptance interface; Wherein, the storage vendor platform is correspondingly connected to the streaming media storage management interface; The streaming media storage system, video access cloud platform, unified operation and maintenance, and business management platform are respectively connected to the streaming media storage management interface, streaming media storage business interface, storage fault reporting interface, and storage business acceptance interface.

6. The management and scheduling system based on distributed storage according to claim 4 is characterized in that: The application layer includes a user management unit, a storage management unit, a log management unit, a report statistics unit, a resource scheduling unit, and a monitoring alarm unit; The user management unit is used to implement customer management, administrator management and operator management; The storage management unit is used to implement storage alarm, local storage management, data node management, local node management, disk management, local space management, space management, streaming media storage management, whitelist management, and information source management; The log management unit is used to manage system logs, administrator operation logs, and customer operation logs; The report statistics unit is used to realize storage data statistics, customer data statistics, business data statistics and pre-order management; The resource scheduling unit is used to realize shared configuration, nearby storage, priority configuration, business resource acceptance and storage modeling; The monitoring and alarm unit is used for data node monitoring, streaming media storage monitoring, storage device alarms, streaming media storage alarms, smart operation and maintenance, smart storage, and business fault acceptance.

7. The management and scheduling system based on distributed storage according to claim 4 is characterized in that: The resource layer is used for edge storage, cloud storage, cloud storage, software-defined storage, Tianyi cloud storage and block data docking.

Images