End-to-end storage capacity management method and system

Abstract

The embodiments of the present application disclose a method and system for end-to-end storage capacity management. Aiming at the goal of end-to-end capacity management between storage and hosts, the system implements dynamic, automated, and real-time statistical collection and display of effectively used storage capacity for hosts of different platform types and different host-side storage usage patterns through customized deployment of host-side scripts. This avoids the one-sidedness of existing storage capacity management methods that only involve statistical storage capacity allocation and the lag of manually recording configuration information in files. Furthermore, the system integrates the end-to-end capacity management method with the operation and maintenance management platform to provide a comparative display of storage capacity management from multiple perspectives, improve the real-time visibility of end-to-end storage capacity management, enhance the level of operation and maintenance management, flexibly generate customized reports, and provide data reference for refined storage capacity management, configuration optimization, and other aspects, assisting in decision support. The system has outstanding technical advantages.

Description

Technical field

[0001] The present application relates to the field of storage capacity management technology in the field of Internet technology support, and in particular to a storage capacity end-to-end management method and system. [Background Technology]

[0002] Storage devices are widely used in the IT (Internet Technology) sector, and the effective use and management of storage capacity is becoming increasingly pressing. Existing storage capacity management methods are limited to capacity allocation on the storage side, failing to effectively monitor storage capacity utilization within the host platform. This lack of tracking and management of actual storage usage leads to waste of storage resources and prevents optimal storage capacity allocation. Furthermore, management methods are limited to static configuration data such as the number of storage devices and allocated capacity recorded in files.

[0003] Existing storage capacity management technologies are limited to recording and managing allocated storage capacity, lacking monitoring and tracking of effectively used storage capacity. Furthermore, the management of statistical information in the form of files cannot dynamically and in real time respond to usage changes. The main drawbacks are as follows:

[0004] 1. Unable to monitor effective storage usage

[0005] Existing storage capacity management methods are limited to capacity allocation management on the storage side, and are unable to monitor the utilization rate of storage capacity within the host platform. There is a lack of methods to obtain the effective utilization rate of storage capacity, resulting in a waste of storage resources and an inability to achieve optimal configuration of storage capacity.

[0006] 2. Storage capacity management information lags

[0007] Existing storage capacity management methods are relatively simplistic, often limited to using files to record information such as the number of storage devices and allocated capacity. However, this type of management statistics is static, often lagging behind actual usage and requiring manual updates, making it impossible to dynamically monitor and automatically track configuration information in real time.

[0008] Therefore, there is an urgent need to create an effective end-to-end capacity management solution between storage and hosts to realize dynamic and automated storage statistics collection and display and other functions. [Summary of the invention]

[0009] In view of this, embodiments of the present application provide a method and system for end-to-end storage capacity management to solve the above-mentioned technical problems existing in the prior art.

[0010] In a first aspect, an embodiment of the present application provides a method for end-to-end storage capacity management, the method comprising the following steps:

[0011] Customized collection of basic configuration information: Customize a list of basic information collection items, collect and organize configuration data related to storage and hosts, and provide basic data input for real-time statistical analysis of capacity management;

[0012] Real-time management and statistics of effective storage capacity: Deploy automated configuration scripts on the host side involved in storage to collect real-time information, and dynamically summarize the collected results by storage device;

[0013] Operation and maintenance management platform display and decision support: Integrate storage capacity management with the operation and maintenance management platform.

[0014] In the method disclosed in the embodiment of the present application, the step of customizing and collecting basic configuration information includes:

[0015] Customize the collection items of basic configuration information;

[0016] Build a data collection platform;

[0017] Collect storage and host configuration data to provide basic data for real-time statistical analysis of capacity management;

[0018] The content includes storage configuration data and host configuration data.

[0019] In the method disclosed in the embodiment of the present application, the step of storing real-time management statistics of effective capacity includes:

[0020] Capacity allocation statistics of storage-side access hosts;

[0021] Host-side automated scripts calculate and collect the actual effective usage of storage devices;

[0022] The effective usage summary statistics in storage units.

[0023] In the method disclosed in the embodiment of the present application, the step of calculating and collecting the actual effective usage of the storage device by the host-side automated script includes:

[0024] The management platform server sends data collection requests to each host;

[0025] Automated script execution;

[0026] Organize the output results.

[0027] In the method disclosed in the embodiment of the present application, the steps of running the automated script include:

[0028] Determine the host type and obtain the actual space usage of the file system of the storage hard disk based on the host type;

[0029] Determine whether the database is running. If the database is running, run the corresponding database effective usage capacity calculation script based on the database type and tablespace name to obtain the actual usage space of the database raw device storing the hard disk.

[0030] In the method disclosed in the embodiment of the present application, the steps of displaying and supporting decision-making on the operation and maintenance management platform include:

[0031] Develop an interface with the operation and maintenance management platform to display real-time statistics on the actual effective utilization rate of storage and host capacity;

[0032] Flexible generation of customized reports provides data support for refined storage capacity management, configuration optimization, and other aspects, creating effective decision-making models.

[0033] In a second aspect, an embodiment of the present application provides an end-to-end storage capacity management system, the system comprising:

[0034] The information collection module is used to customize a list of basic information collection items, collect and organize configuration data related to storage and hosts, and provide basic data input for real-time statistical analysis of capacity management;

[0035] The capacity statistics module is used to deploy automated configuration scripts on the host side involved in storage to collect real-time information and dynamically summarize the collected results by storage device;

[0036] The platform management module is used to integrate storage capacity management with the operation and maintenance management platform.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor:

[0038] The memory is used to store computer programs;

[0039] The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the method according to the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a program or instructions. When the program or instructions are run on a computer, the method described in the first aspect is executed.

[0041] Compared with the existing technology, this technical solution has at least the following beneficial effects:

[0042] The end-to-end storage capacity management method and system disclosed in the embodiments of this application are aimed at the goal of end-to-end capacity management between storage and hosts. Through customized deployment of host-side scripts, they achieve dynamic, automated, and real-time statistical collection and display of effectively used storage capacity for hosts of different platform types and different host-side storage usage patterns. This avoids the one-sidedness of existing storage capacity management methods that only involve statistical storage capacity allocation and the lag of manually recording configuration information in files. In addition, the end-to-end capacity management method is integrated with the operation and maintenance management platform to provide a comparative display of storage capacity management from multiple perspectives, improve the real-time visibility of end-to-end storage capacity management, enhance the level of operation and maintenance management, flexibly generate customized reports, provide data reference for refined storage capacity management and configuration optimization, and assist in decision support, with outstanding technical advantages.

Brief Description of the Drawings

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 This is a schematic diagram of the steps of the end-to-end storage capacity management method provided in Example 1 of the present application;

[0045] Figure 2 This is a schematic diagram of the steps for customizing and collecting basic configuration information in the end-to-end storage capacity management method provided in Example 1 of the present application;

[0046] Figure 3 This is a schematic diagram of the steps for real-time management and statistics of storage effective capacity in the end-to-end storage capacity management method provided in Example 1 of the present application;

[0047] Figure 4 This is a schematic diagram of the structure of the modules used in the end-to-end storage capacity management method provided in Example 1 of the present application;

[0048] Figure 5 This is a schematic diagram of the steps of calculating and collecting the actual effective usage of storage devices by a host-side automated script in the end-to-end storage capacity management method provided in Example 1 of the present application;

[0049] Figure 6 This is a schematic diagram showing the end-to-end management of storage capacity on the operation and maintenance management platform in the end-to-end storage capacity management method provided in Example 1 of the present application;

[0050] Figure 7 This is a structural diagram of the end-to-end storage capacity management system provided in Example 2 of the present application.

[0051] Reference numerals:

[0052] 1-Information collection module;

[0053] 2-Capacity statistics module;

[0054] 3-Platform management module. [Specific implementation method]

[0055] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0056] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0057] Example 1

[0058] Example 1 of the present application discloses an end-to-end storage capacity management method, which can effectively solve the following two technical problems existing in the prior art:

[0059] 1. Monitoring and management of effective storage utilization

[0060] Without affecting the existing operating configuration of storage and hosts, edit automation scripts and deploy them to relevant hosts to automatically count the effective utilization rate of host-side storage capacity. This includes automatic determination of the type of storage device used on the host side, statistics on the utilization rate of the file system and tablespace, and reasonable aggregation of data collected by the host platform on the storage side, thereby realizing monitoring and management of the effective utilization rate of storage.

[0061] 2. Dynamic real-time management of storage capacity

[0062] Collect host-side storage configuration information and improve basic data statistics from storage to hosts, avoiding the one-sided limitations of traditional management methods that only manage storage space allocation. Combined with automated script deployment, dynamic storage capacity statistics are performed to achieve real-time end-to-end capacity management from host to storage.

[0063] like Figure 1 As shown, the method disclosed in the embodiment of the present application includes the following steps:

[0064] Step 100: Customize and collect basic configuration information: Customize a list of basic information collection items, collect and organize configuration data related to storage and hosts, and provide basic data input for real-time statistical analysis of capacity management;

[0065] Step 200: Real-time management and statistics of storage effective capacity: Deploy automated configuration scripts on the hosts involved in storage to collect real-time information. The collected results are dynamically summarized by storage device, achieving real-time end-to-end capacity space management from the host to the storage.

[0066] Step 300: Operation and Maintenance Management Platform Presentation and Decision Support: Integrate storage capacity management with the operation and maintenance management platform. This provides a multi-dimensional comparative presentation of storage capacity management, improves real-time visibility of end-to-end storage capacity management, enhances operation and maintenance management capabilities, and flexibly generates customized reports, providing data support for decisions such as refined storage capacity management and configuration optimization.

[0067] like Figure 2 As shown, further, in the method disclosed in the embodiment of the present application, the step of customizing and collecting basic configuration information includes:

[0068] Step 101: Customize the collection items of basic configuration information;

[0069] Step 102: Build a data collection platform;

[0070] Step 103: Collect configuration data of storage and hosts to provide basic data for real-time statistical analysis of capacity management.

[0071] This includes storage configuration data and host configuration data. Storage configuration data includes storage brand and model, physical capacity, logical capacity, a list of connected hosts, and corresponding allocated capacity. Host configuration data includes the host operating system type, whether a database is used and the database software type, and the database tablespace name. This storage and host configuration data is entered into the relevant management system to generate storage capacity allocation ratio data and a host mapping table. The corresponding configuration files are then edited to serve as input parameters for running the automated script.

[0072] like Figure 3 As shown, further, in the method disclosed in the embodiment of the present application, the step of storing real-time management statistics of effective capacity includes:

[0073] Step 201: Capacity allocation statistics of the storage end connected to the host;

[0074] Step 202: The host-side automated script calculates and collects the actual effective usage of the storage device;

[0075] Step 203: Summarize the effective usage statistics in storage units.

[0076] Deploy automated configuration scripts on the host side connected to the storage, perform real-time statistics on the storage device capacity used by the host side, and dynamically summarize the output results by storage device unit, realizing real-time end-to-end capacity space management from the host to the storage, achieving the purpose of fine-grained management of storage capacity.

[0077] like Figure 4 What is shown is a structural diagram of the modules used in the method disclosed in the embodiment of the present application. The statistical collection method includes three modules: capacity allocation statistics of the storage end connected to the host, calculation and collection of the actual effective usage of the storage device by the host-side automated script, and summary statistics of the effective usage rate in units of storage.

[0078] In step 201, based on the basic configuration information customization collection platform built in Step 100, the basic configuration information of the storage and related hosts is obtained, the capacity allocation information of the storage end connected to the host is recorded, and processing is performed to generate a configuration file for use by subsequent computing modules.

[0079] Specifically, the existing network system has multiple storage devices of different brands. Each storage device is named according to the storage brand type, such as emc5363, emc2201, hds11287, etc., and the physical capacity, logical capacity, allocated capacity, allocated host and other information of each storage are sorted out for use;

[0080] The host platforms connected to each storage system vary, including AIX, Linux, HP-Unix, and other open platform systems. To facilitate data collection using automated scripts, enter the host IP address as the host name, and use "AIX," "Sol," and other identifiers for the host operating system. Also, confirm the application type running on the host. If the host does not have a database, mark it as "nodb." If it does, mark it as "sjjs" or "ivr1."

[0081] Based on the above collected information, compile a configuration file in the following format:

[0082] Storage name Host IP address Database instance name Database type Operating system type

[0083] Here are some examples:

[0084] emc2460 10.181.103.9nodb none aix

[0085] emc2460 10.175.104.11nodb none sol

[0086] emc5620 10.181.17.142nodb none aix

[0087] emc3751 10.175.129.54ivr1 oracle aix

[0088] emc3751 10.175.129.55nodb none linux

[0089] emc5669 10.175.129.52dhjl1 db2 aix

[0090] …

[0091] In step 202, the host-side automated script operation and collection module automatically selects the host platform type, storage operation mode and corresponding data capture method based on the configuration file input, so as to realize real-time dynamic collection of the effective utilization rate of storage devices in various modes (file system, database, etc.) on the host platform.

[0092] Among them, the real-time statistics of the storage capacity used by the host side are divided into two situations. The first is that the storage hard disk is mounted in file system mode and the actual effective usage space of each file system is directly summarized; the second is that the storage hard disk is mounted in raw device mode for database use, and the database script method is called to calculate the actual effective usage rate of the database, thereby realizing the statistical summary of the actual effective usage capacity of the host side storage under various situations.

[0093] The operation process of the storage capacity dynamic real-time statistics collection method module is as follows:

[0094] a. The management platform server sends a data collection request to each host;

[0095] b. The automated script runs, first determining the host type and, based on the host type, obtaining the actual space usage of the file system on the storage hard disk. It then determines whether a database is running. If a database is running, it runs the corresponding database effective capacity calculation script based on the database type and tablespace name, thereby determining the actual space usage of the database raw device on the storage hard disk.

[0096] c. Organize the output results and record the actual effective usage capacity of the file system / database raw device using the storage in each host, taking a single storage unit as the unit, to provide basic data for effective end-to-end storage capacity management.

[0097] Specifically, the automated collection script is deployed on each host platform. Using a secure protocol, the management platform periodically sends run requests to each host, performing real-time queries and summarizing the actual effective utilization of storage devices connected to each host. The file version of the automated collection script autocollect.sh deployed on the host platform is as follows:

[0098]

[0099]

[0100] In step 203, the output results of the collection module are automatically summarized in units of storage based on the automated script operation, and the effective utilization rate of storage capacity is displayed in real time.

[0101] The automated script provides an output result integration function, summarizing the output results by storage unit, including the actual effective utilization rate of the application [file system / tablespace] in each host, providing a data basis for subsequent statistics on the actual effective utilization rate of storage. The output result example is as follows:

[0102] emc1686|10.175.3.5|TS_GPRS_796_01|495320|375122|75.7%

[0103] emc1686|10.175.3.5|TS_GSM_793_IND|221166|167276|75.6%

[0104] emc1686|10.175.3.5|TS_GPRS_796_IND|148404|112055|75.5%

[0105] emc1686|10.175.12.8|TS_GSM_790|393184|295933|75.2%

[0106] emc1686|10.175.12.8|TS_GPRS_791_JY|2759526|2061438|74.7%

[0107] emc1686|10.175.105.11|TS_GPRS_792|540628|402879|74.5%

[0108] emc1686|10.175.105.11| / BOSSDB1|211759|156532|73.9%

[0109] emc1686|10.175.105.11| / BOSSDB2|520854|382434|73.4%

[0110] …

[0111] like Figure 5As shown, further, in the method disclosed in the embodiment of the present application, the step of calculating and collecting the actual effective usage of the storage device by the host-side automated script includes:

[0112] Step 301: The management platform server sends a data collection request to each host;

[0113] Step 302: Automated script execution;

[0114] Step 303: Arrange the output results.

[0115] The automated script execution process includes:

[0116] ① Determine the host type and obtain the actual space usage of the file system of the storage hard disk based on the host type;

[0117] ② Determine whether the database is running. If the database is running, run the corresponding database effective usage capacity calculation script based on the database type and table space name to obtain the actual usage space of the database raw device of the storage hard disk.

[0118] Furthermore, the process of display and decision support on the operation and maintenance management platform includes:

[0119] ① Develop an interface with the operation and maintenance management platform to display real-time statistics on the actual effective utilization rate of storage and host capacity;

[0120] ②Flexibly generate customized reports to provide data support for refined storage capacity management, configuration optimization, etc., and create effective decision-making models.

[0121] By using the host-side automated script in Step 300 to calculate and collect the actual effective usage of storage devices, it is possible to develop an interface with the operation and maintenance management platform, and to display the actual effective usage rate of storage and host-side capacity in real time. This allows for a comparative display of storage allocation capacity, host-side usage capacity, actual effective storage usage capacity, and other multi-dimensional perspectives, providing end-to-end real-time visibility of storage capacity and improving the level of operation and maintenance management. It can also flexibly generate customized reports to provide data support for refined management of storage capacity, configuration optimization, and other aspects, thereby creating an effective decision-making model.

[0122] like Figure 6 As shown, an operation and maintenance management platform interface is developed and an end-to-end capacity management module is deployed between storage and hosts to achieve multi-level and dynamic data display and analysis of the effective utilization rate of storage capacity, the effective utilization rate of host storage resources, and the actual storage utilization rate of each application in the host, providing data support for auxiliary decision-making.

[0123] Example 2

[0124] like Figure 7 As shown, Example 2 of the present application discloses an end-to-end storage capacity management system, the system including:

[0125] Information collection module 1 is used to customize a list of basic information collection items, collect and organize configuration data related to storage and hosts, and provide basic data input for real-time statistical analysis of capacity management;

[0126] Capacity statistics module 2 is used to deploy automated configuration scripts on the host side involved in storage to collect real-time information and dynamically summarize the collected results by storage device;

[0127] The platform management module 3 is used to integrate storage capacity management with the operation and maintenance management platform.

[0128] The end-to-end storage capacity management system disclosed in the embodiments of this application, through script customization, enables dynamic, automated, and real-time statistical collection and display of effective end-to-end capacity usage between storage and hosts. This avoids the one-sided nature of existing storage capacity management methods, which only focus on statistical storage end capacity allocation, and the lag associated with manual file recording of configuration statistics. This is a key core point of this proposal. Furthermore, it integrates with the operations and maintenance management platform, providing a comparative display of storage capacity management from multiple perspectives, improving real-time visibility of end-to-end storage capacity management, and assisting in decision support.

[0129] The present application also provides an electronic device including a memory and a processor.

[0130] Memory for storing computer programs;

[0131] The processor is used to execute the computer program stored in the memory so that the electronic device executes the text tracing detection method disclosed in Example 1 of the present application.

[0132] An embodiment of the present application also provides a computer-readable storage medium, including a program or instruction. When the program or instruction runs on a computer, the text tracing detection method disclosed in Example 1 of the present application is executed.

[0133] The end-to-end storage capacity management method and system disclosed in the embodiments of this application are aimed at the goal of end-to-end capacity management between storage and hosts. Through customized deployment of host-side scripts, they achieve dynamic, automated, and real-time statistical collection and display of effectively used storage capacity for hosts of different platform types and different host-side storage usage patterns. This avoids the one-sidedness of existing storage capacity management methods that only involve statistical storage capacity allocation and the lag of manually recording configuration information in files. In addition, the end-to-end capacity management method is integrated with the operation and maintenance management platform to provide a comparative display of storage capacity management from multiple perspectives, improve the real-time visibility of end-to-end storage capacity management, enhance the level of operation and maintenance management, flexibly generate customized reports, provide data reference for refined storage capacity management and configuration optimization, and assist in decision support, with outstanding technical advantages.

[0134] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for end-to-end storage capacity management, characterized in that: The method comprises the following steps: Customized collection of basic configuration information: Customize a list of basic information collection items, collect and organize configuration data for the relevant storage and hosts, and provide basic data input for real-time statistical analysis of capacity management; Real-time management and statistics of effective storage capacity: Deploy automated scripts on the host side involved in storage to collect real-time information, and dynamically summarize the collected results by storage device; Operation and maintenance management platform display and decision support: Integrate storage capacity management with the operation and maintenance management platform; The steps of collecting and customizing basic configuration information include: Customize the collection items of basic configuration information; Build a data collection platform; Collect storage and host configuration data to provide basic data for real-time statistical analysis of capacity management; The content includes storage configuration data and host configuration data. The storage configuration data includes storage brand and model, physical capacity, logical capacity, a list of connected hosts and corresponding allocated capacity. The host configuration data includes the host operating system type, whether a database is used and the database software type, and the database tablespace name. The automated script runs include: Determine the host type and obtain the actual space usage of the file system of the storage hard disk based on the host type; Determine whether the database is running. If the database is running, run the corresponding database effective usage capacity calculation script based on the database type and tablespace name to obtain the actual usage space of the database raw device storing the hard disk.

2. The end-to-end storage capacity management method according to claim 1, characterized in that: The step of real-time management and statistics of the storage effective capacity includes: Capacity allocation statistics of storage-side access hosts; Host-side automated scripts calculate and collect the actual effective usage of storage devices; Effective usage statistics summarized by storage device.

3. The end-to-end storage capacity management method according to claim 2, characterized in that: The steps of computing and collecting the actual effective usage of the storage device by the host-side automated script include: The management platform server sends data collection requests to each host; Automated script execution; Organize the output results.

4. The end-to-end storage capacity management method according to claim 1, characterized in that: The steps of displaying and supporting decision-making on the operation and maintenance management platform include: Develop an operation and maintenance management platform interface to display real-time statistics on the actual effective utilization rate of storage and host capacity; Generate customized reports to provide data support for refined storage capacity management and configuration optimization, creating effective decision-making models.

5. A storage capacity end-to-end management system, characterized in that: The system comprises: The information collection module is used to customize the basic information collection item list, collect and organize the configuration data of the storage and hosts involved, and provide basic data input for real-time statistical analysis of capacity management; The capacity statistics module is used to deploy automated scripts on the host side involved in storage to collect real-time information and dynamically summarize the collected results by storage device; Platform management module, used to integrate storage capacity management with the operation and maintenance management platform; The information collection module is specifically used to: Customize the collection items of basic configuration information; Build a data collection platform; Collect storage and host configuration data to provide basic data for real-time statistical analysis of capacity management; The content includes storage configuration data and host configuration data. The storage configuration data includes storage brand and model, physical capacity, logical capacity, a list of connected hosts and corresponding allocated capacity. The host configuration data includes the host operating system type, whether a database is used and the database software type, and the database tablespace name. The automated script runs include: Determine the host type and obtain the actual space usage of the file system of the storage hard disk based on the host type; Determine whether the database is running. If the database is running, run the corresponding database effective usage capacity calculation script based on the database type and tablespace name to obtain the actual usage space of the database raw device storing the hard disk.

6. An electronic device, characterized in that: include: Memory and processor: The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so as to enable the electronic device to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The method comprises a program or an instruction, and when the program or the instruction is run on a computer, the method according to any one of claims 1 to 4 is executed.

Images