Management system, management method by the management system, and computer program

Abstract

To manage a relationship between a database and a volume without installing an agent.SOLUTION: Disclosed is a management system 8000 for managing a relationship between a data catalog and a volume 4400 of a storage system 4000 in which the data used by the data catalog is stored. A processor 8300 of the management system 8000 performs access to the data including the authentication information of the volume 4400 possessed by the data catalog; detects the number of accesses of the volume 4400 in a time range including the timing of the access to the data; and manages the relationship between the data catalog and the volume 4400 based on the number of accesses.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a management system, a management method by the management system, and a computer program.

Background Art

[0002] More specifically, the present invention relates to discovering the relationship between a database application executed on a server or virtual machine and the volume where the data used by this database application exists.

[0003] With the development of hybrid cloud data services, it has become increasingly common for a database and the volume of a storage system as the entity where the data used by this database is stored to be arranged in different systems.

[0004] From the perspective of compliance, when deleting a database, the data used by this database must also be completely deleted. However, in a hybrid cloud data service where there are a large number of databases and a large number of storage systems, it is difficult to immediately grasp the correspondence between the database and the volume and appropriately delete the data.

[0005] As technologies for addressing such problems, there are the technologies disclosed in Patent Document 1 and Patent Document 2. In the technologies disclosed in these Patent Document 1 and Patent Document 2, the problem of finding the relationship between the server and the storage is solved by installing an agent on the server. The "Application Object Manager Engine" (agent on the host server machine) determines the mapping between the application object (database application) on the storage system and its storage object (logical volume). The agent is configured to execute in combination with other software on the server.

Prior Art Documents

Patent Document

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, installing an agent on a server, as in the technologies disclosed in Patent Document 1 and Patent Document 2, has the following problems.

[0008] First, due to security issues, the clients of the system may not want to share the authentication information (credentials) of Secure Shell (SSH). Second, the system may be running hundreds of database applications on virtual machines (VMs) or physical servers, and it may be cumbersome to manually collect such Secure Shell (SSH) authentication information. This is usually because there is no central repository for encryption keys and the configuration of virtual machines (VMs) is dynamic.

[0009] The present invention has been made in view of the above circumstances, and an object thereof is to provide a management system, a management method by the management system, and a computer program capable of managing the relationship between a database and a volume without installing an agent.

Means for Solving the Problems

[0010] To solve the above problems, a management system according to one aspect of the present invention is a management system that manages the relationship between a database and the volume of a storage system in which the data used by this database is stored. The management system has a processor, and the processor accesses the data including the authentication information of the volume that the database has, detects the number of accesses to the volume within a time range including the timing of access to the data, and manages the relationship between the database and the volume based on the number of accesses.

Advantages of the Invention

[0011] According to the present invention, it is possible to realize a management system, a management method by the management system, and a computer program that can manage the relationship between a database and a volume without installing an agent. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the following description and drawings are examples for explaining the present invention, and for the sake of clarity of explanation, appropriate omissions and simplifications are made, and they do not limit the technical scope of the present invention.

[0014] In the following description, various types of information will be described using expressions such as "table", "sheet", "list", "queue", etc., but the various types of information may be represented by data structures other than these. Therefore, in order to indicate that it does not depend on the data structure, it may simply be called "information". When explaining the content of various types of information, expressions such as "identification information", "identifier", "name", "name", "ID", "number", etc. are used, but these can be mutually replaced.

[0015] In the following description, the "program" may be used as the subject for explanation. However, a program performs the defined processing by being executed by a processor (such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit)), while appropriately using storage resources (such as memory) and interface devices (such as communication devices). Therefore, it may also be described with the processor as the subject. Similarly, the entity performing the processing by executing the program may be, for example, a controller, a device, a system, a computer, a node, a storage device, a server, a client, or a host having a processor. Also, part or all of the program may be processed using a hardware circuit.

[0016] Various programs may be installed on each computer by a program distribution server or a storage medium. Also, in the following description, two or more programs may be realized as one program, and conversely, one program may be realized as two or more programs.

[0017] In the drawings for explaining the embodiments, the same reference numerals are given to portions having the same function, and the repeated explanation thereof is omitted.

[0018] Also, in the following description, when explaining without distinguishing between the same type of elements, reference numerals (or common reference numerals among the reference numerals) are used, and when explaining by distinguishing between the same type of elements, the identification numbers (or reference numerals) of the elements may be used.

[0019] The positions, sizes, shapes, ranges, etc. of the respective components shown in the drawings may not represent the actual positions, sizes, shapes, ranges, etc. in order to facilitate the understanding of the invention. For this reason, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0020] The management system of this embodiment has the following configuration as an example.

[0021] That is, the management system of this embodiment relates to side-channel analysis technology. This technology based on Cloudscopy provides a novel approach for IP (Internet Protocol) address discovery, hop count measurement, and verification of co-existing virtual machines (VMs). To achieve this, hardware interrupts, cloud scans, and Time to Live (TTL) scans are utilized.

[0022] Similar to the side-channel analysis method, the management system of this embodiment proposes a new solution for discovering the relationship between servers and storage by probing the database with the help of a data catalog and analyzing the network to find mappings.

[0023] This embodiment proposes a method for finding the mapping of data from a database application running on a server or virtual machine to storage. This embodiment does not require installing an agent in the customer environment. The relationships discovered by this embodiment are useful in data management through end-to-end data flow visualization.

[0024] This embodiment eliminates the need to collect the secure shell (SSH) authentication information of customer servers or install any agents on them by utilizing existing information from management software such as data catalogs and storage performance.

[0025] The data catalog contains access details for all databases existing within the organization. This includes the table endpoints and the authentication information for the databases. Since this information is already prepared in the data catalog in an encrypted form, the solution of this embodiment does not require the collection of authentication information. If a customer can access the data catalog, they can directly use its application programming interface (API). These data catalogs also have the function of displaying the content of the tables via a data preview application programming interface (API).

[0026] Each time a table is read via the data preview application programming interface (API), it is predicted that the volume performance will increase somewhat rapidly in the same time period. The performance data includes parameters such as IOPS (Input Output per Seconds), transfer rate, and cache hit ratio. For tables with relatively little data of about gigabytes (GB), executing such read commands will impose a very large load on the volume performance. It should be significant enough to be detected.

[0027] According to the above considerations, a relationship identification program can be constructed. This program triggers read operations on the tables of the data catalog (one by one at a time) via the application programming interface (API). As a result, the database reads data from storage and sends the results back to the data catalog.

[0028] This relationship identification program also collects storage performance in parallel using a storage application programming interface (API). The collected performance data becomes time-series data for implementing an anomaly detection algorithm. Any anomaly detection algorithm can be considered. During the period of executing a data preview from a data catalog, if a high anomaly score is detected at any volume, it can be concluded that the database from which the preview command was read is using an appropriate volume for data storage.

[0029] If the database memory is larger than the table size, spikes cannot be observed during reading, but the proposal can be applied to almost all practical cases. The present embodiment can be applied whenever the table size is larger than the database memory size. This example is always applicable when there is write permission in the data catalog and the customer can specify and write to the sample space.

[0030] Automating the search for the relationships between hundreds of servers and storage also improves security because login authentication information is not required. This provides various advantages such as end-to-end data flow, root cause analysis, and policy compliance. Its usefulness has been discussed in detail through use cases of deleting data.

[0031] Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 1 to 10.

[0032] FIG. 1 is a block diagram showing an embodiment of an overall system infrastructure in a typical customer environment including the management system of the present embodiment. FIG. 1 shows each system existing in the customer environment, how they interact with each other, and our relationship identification program.

[0033] The customer environment is composed of a number of data catalog systems 6000, server systems 2000, storage systems 4000, and the management system 8000 of the present embodiment.

[0034] The data catalog system 6000 has a server processor 6100 that processes programs executed on the system. The server network adapter 6700 is responsible for the connectivity of the system. The server storage adapter 6900 helps connect the storage 6500 available on the network. Some data 6510 exists on the attached storage 6500. The server memory 6300 has the programs currently being executed on the system. Typical programs that can be executed on the server memory 6300 are the server OS 6310 and the data catalog application 6320. The data catalog application 6320 may be executed on a VM managed by the VM manager 6311 or may be executed bare-metal. A110 is a bus that connects all components of the data catalog system 6000.

[0035] A number of server systems 2000 can exist that are executed in the customer environment. This server system 2000 has a server processor 2100 that processes programs executed on the system. The server network adapter 2500 is responsible for the connectivity of the system. The server storage adapter 2700 helps connect the attached storage 2900 available on the network. The server memory 2300 is composed of the programs currently being executed on the system.

[0036] The server OS 2320 may be running the VM manager 2321. The VM 2310 may be running the application 2311 (which can be a database application or other application) along with its data 2312. The application 2301 can also run on bare metal with the data 2302. The data 2312 and 2302 are present on the disks 2910 or 2920 of the attached storage 2900. A120 is a bus that connects all components of the server system 2000.

[0037] The storage system 4000 has a server processor 4700 that processes programs running on the system. The network adapter 4500 is responsible for the connectivity of the system. The network target adapter 4900 helps connect the storage to other systems via the network. The server memory 4100 mostly consists of the storage operating system 4110. An array of disks and volumes 4400 is connected by the storage adapter 4300. A130 is a bus that connects all components of the storage system 4000.

[0038] The management system 8000 has a server processor 8300 that processes programs running on the system. The server network adapter 8700 is responsible for the connectivity of the system. The server storage adapter 8900 helps connect the attached storage 8500 available on the network. There may be some data 8510 within this attached storage 8500. The server memory 8100 is composed of the programs currently running on the system.

[0039] The data catalog management function 8110 retrieves relationship information regarding tables and databases from the data catalog system 6000 and stores it as settings 8111. The communication is shown as A520.

[0040] The application management function 8140 collects all inter-application communication information from the server system 2000 and stores it in the setting 8141. The communication is shown as A510.

[0041] The storage management function 8150 collects the performance data 8151 of all volumes and settings 8152, and the list thereof is related to volumes and disks and their relationships. The communication is shown as A500.

[0042] The relationship identification program 8120 is the main part of the management system 8000 of this embodiment. This communication, which will collect all settings and performance data from the data within the data catalog management function 8110, the application management function 8140, and the storage management function 8150, is shown as arrow A540. This calculates the end-to-end topology from the data catalog to the database application (as setting 8121) and stores it in the topology database 8130 as shown by arrow A550. The topology database 8130 is composed of two attribute topology objects 8131 as nodes and a topology link 8132 as an edge between the nodes.

[0043] The topology is sent to the Web client 9000 and visualized as the graphical user interface 9100. This connection is shown by arrow A560. A100 is a bus that connects all components of the management system 8000.

[0044] FIG. 2A and FIG. 2B are diagrams showing an example of the information tables possessed by the management system of this embodiment.

[0045] In FIG. 2, the information table T1000 has a performance data table T1100, a topology object T1200, a topology link T1300, a data catalog list T1400, a database list T1500, a table list T1600, and a volume list T1700.

[0046] T1100 is a performance data table, corresponding to the performance data 8151 in FIG. 1. The performance data table T1100 has, as entries, a time series T1110, IOPS (Input Output per seconds) T1120, a transfer rate T1130, and the like.

[0047] T1200 is a topology object table, corresponding to the topology object 8131 in FIG. 1. The topology object table T1200 has, as entries, an ID of a node T1210, a name of the node T1220, a type of the node T1230, and the like.

[0048] T1300 is a topology link table, corresponding to the topology link 8132 in FIG. 1. The topology link table T1300 has, as entries, an ID of a link T1310, a name of a source node (source) T1320, a name of a destination node (target) T1330, and the like.

[0049] T1400 is a data catalog list, corresponding to the setting 8111 of the data catalog management function 8110 in FIG. 1. The data catalog list T1400 has, as entries, an id of a data catalog T1410, a type of the data catalog T1420, an IP of the data catalog T1430, a user name T1440, and a password of the data catalog T1450, and the like.

[0050] T1500 is a database list obtained from a data catalog, corresponding to the setting 8121 of the relationship identification program 8120 in FIG. 1. The database list T1500 has, as entries, an ID of a database T1510, a name of the database T1520, a type of the database T1530, and the like.

[0051] T1600 is a list of tables obtained from the data catalog and corresponds to the setting 8121 of the relationship identification program 8120 in FIG. 1. The table list T1600 has the table id T1610, the table name T1620, the database to which the table belongs T1630, and so on.

[0052] T1700 is a volume list and is obtained from the storage system 4000. The volume list T1700 corresponds to the setting 8152 of the storage management function 8150 in FIG. 1. The volume list T1700 has the volume id T1710, the LDEV ID T1720 of the volume, and the label T1730 of the volume, and so on.

[0053] FIG. 3 shows an example of a method for connecting the server system 2000 to the storage.

[0054] The server system 2000a runs the database application 2303 in bare metal, while the server system 2000b runs the database application 2313 on the virtual machine 2310.

[0055] The tables 2304 and 2314 store their data in the directories 2305 and 2315. This connection is indicated by the arrows A200 and A210 respectively. These directories 2305 and 2315 are resident on the implementation devices 2307 and 2317 respectively. This relationship is managed by the LVM layers 2306 and 2316. The relationship between the directories is the LVM layer indicated by A201 and A211. The relationship between the LVM layer and the implemented device is indicated by A203 and A212.

[0056] The server system 2000 is connected to the storage component 5000 by the storage area network (SAN) 3000, and this connection is indicated by the arrows A220a, A220b, A230a, and A230b.

[0057] Ports 5400a and 5400b connect the server to the volume. Port 5400 may manage the connection between the group of server systems 2000 and the group of volumes represented by host groups 5100a and 5100b. Let the relationship between the host group and the port be A230a, A230b. Let the relationship between the host group and the volume be indicated by A240a, A240b.

[0058] The volume is an array of basic volumes 4400a split from an array of disks known as parity group 5200. This connection is indicated by arrow A250a. The volume can be a virtual volume 4401a split through a disk pool 5300. This connection is immediately made by arrow A250b.

[0059] Volumes 4400a and 4401a are mapped to implementation devices 2307 and 2317 with the help of protocols such as iSCSI / FC3100a and 3100b. The source and target unique names are managed by these protocols.

[0060] Figure 4 shows an example where data can be collected at each layer to find the end - to - end relationship topology of the data.

[0061] The information item T2000 can be collected from the data catalog application 6320. The data catalog application 6320 is connected to the server system 2000 by referring to the Internet Protocol (IP): port 3200. The information item T3000 can be retrieved from the server system 2000 by logging in to the server using Secure Shell (SSH) authentication information. The server system 2000 is connected to the volume 4400 by some protocol such as iSCSI / FC3100. The information item T4000 can be obtained from the storage software running in the storage system 4000.

[0062] Figure 5 shows the problems that occur when logging into the system to obtain relationship data.

[0063] The data catalog system 6000 has authentication information (Credential) C100. The storage system 4000 has authentication information C102. There may be hundreds of service systems with the authentication information C101 of the server system 2000. The data catalog system 6000 has access rights to all database applications 2301 within the server system 2000 indicated by the arrow A700. The storage system 4000 can have an array of volumes and disks 4400 indicated by the arrow A730. The application 2301 uses an implementation device 2307 for the storage volume as indicated by the arrow A710.

[0064] The problem is that the relationship identification program 8120 has to collect mount information A720 by logging into hundreds of server systems 2000 that require collecting hundreds of login / SSH authentication information, which is very difficult to collect and has security concerns. It is indicated by the arrow A300.

[0065] Figure 6 represents the core concept of the relationship identification program 8120. This is based on the type of side-channel analysis. Ideally, every time a load is generated on the data catalog for reading or writing to a table, a performance volume spike is expected to occur on the volume.

[0066] 7000 indicates the operations on the data in the data catalog. High means a read / write operation, and low means no operation. In this example, the read operation is performed from time 7010 to 7020.

[0067] 7500 indicates the volume performance. As indicators for measuring the volume performance, input / output operations per second (IOPS), transfer rate, or cache hit rate are possible. Abnormalities occur from time 7510 to 7520. Time 7510 is the same as 7010, and time 7520 is the same as 7020. The peak of the spike is at time 7530.

[0068] Figure 7 is a flowchart showing in detail how the management system 8000 of this embodiment, particularly the relationship identification program 8120, can find data relationships.

[0069] First, the relationship identification program 8120 collects the performance data of all volumes at P100. Next, the relationship identification program 8120 selects one database from the list of databases existing in the data catalog at P101. Next, the relationship identification program 8120 selects one table from the data catalog selected at P101 at P102. Next, the relationship identification program 8120 starts timing at P103. At this time, let the start time of timing be T1. Next, the relationship identification program 8120 uses the data catalog API to send a data preview command to the table selected at P102 and executes the process of reading the content of the selected table.

[0070] The relationship identification program 8120 determines at P105 whether the operations shown in P102 to P104 have been performed for all tables. If it is determined that the operations have been performed for all tables (YES at P105), it proceeds to P106. If it is determined that there are still tables for which the operations have not been performed (NO at P105), it returns to P102 to continue the operations.

[0071] In P106, the relationship identification program 8120 ends the timing. At this time, let the end time of timing be T2. Next, in P107, the relationship identification program 8120 calculates the anomaly score between time T1 and time T2 for all volumes. Then, in P108, if a certain volume gives a high anomaly score among the anomaly scores calculated in P107, the relationship identification program 8120 generates a mapping between the loaded database and this volume. And in P109, the relationship identification program 8120 collects the related information from the table to the database to generate the entire mapping (table → database → server → volume).

[0072] Then, in P110, the relationship identification program 8120 determines whether the operations shown in P102 to P109 have been performed for all databases. If it is determined that the operations have been performed for all databases (YES in P110), it proceeds to P111. If it is determined that there is still a database for which the operations have not been performed (NO in P110), it returns to P101 to continue the operations.

[0073] Then, in P111, the relationship identification program 8120 ends the collection of the performance data of the storage volume.

[0074] FIG. 8 is a flowchart showing an example of operations that are preferably executed prior to the operations shown in FIG. 7 when executing the operations of the management system of the present embodiment shown in the flowchart of FIG. 7.

[0075] First, in P200, the relationship identification program 8120 collects performance data for all volumes as a background task. Next, in P201, the relationship identification program 8120 identifies a period during which the load on the database (DB) and storage is low. Note that the period of low load is a period during which, even when processing in response to a data preview command is executed, the load on each component constituting the storage is below a value indicating overload. The identification of the period of low load may be determined, for example, by measuring in advance the degree of increase in load when processing in response to a data preview command is executed and determining whether the value obtained by adding the degree of increase to the current load is below a threshold indicating overload, or by setting a threshold indicating low load for the operating rate of each component and determining whether it is below the threshold. In P202, the relationship identification program 8120 determines the date and time to access the data with reference to the period identified in P201.

[0076] FIG. 9 shows a topology graph generated by the management system 8000 of the present embodiment.

[0077] All tables derived from the data catalog are displayed in N100. N110 is a database endpoint derived from the data catalog and includes important information such as an Internet Protocol (IP) address, a port, and database credential information. The database application N120 is composed of the names and metadata of the applications existing in the cluster. N130 is a volume mounted on N120. N140 is a disk on which the volume exists. In 5500, the data catalog view is split from the application view. In 5600, the application view and the storage view are separated.

[0078] FIG. 10 is a diagram showing an example of a GUI (Graphical User Interface) generated by the management system 8000 of the present embodiment. The GUI 9100 shown in FIG. 10 is displayed by a Web client 9000.

[0079] 5100 is a data catalog view, 5200 is an application view, and 5300 is a storage view. 5000 is the end-to-end highlighted path of the clicked table. N100 is a layer that displays all tables. N110 is a layer that displays all database endpoints. N120 is a layer that shows all applications. N130 is a layer that shows all volumes. N140 is a layer that displays all disks.

[0080] As described in detail above, according to the management system 8000 of the present embodiment, it is possible to manage the relationship between a database and a volume without installing an agent.

[0081] As described above, the main embodiments of the present invention have been described. However, this is an exemplification for the description of the present invention, and is not intended to limit the scope of the present invention only to this embodiment. It is not necessarily required to have all the configurations described, and it is also possible to replace or add a part of the configuration of one embodiment with the configuration of another embodiment. Similarly, it is also possible to change or delete a part of the configuration of each embodiment as needed.

[0082] In addition, each of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware by designing part or all of them, for example, by means of an integrated circuit. Further, the present invention can also be realized by a program code of software that realizes the functions of the embodiments. In this case, a storage medium storing the program code is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the same constitute the present invention. As a storage medium for supplying such a program code, for example, a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD (Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, etc. are used.

[0083] In addition, the program code for realizing the functions described in this embodiment can be implemented in a wide range of programs or script languages such as assembler, C / C++, perl, Shell, PHP, Java (registered trademark), Python, etc.

[0084] In the above-described embodiments, the control lines and information lines show those considered necessary for explanation, and not necessarily all the control lines and information lines on the product are shown. All the components may be interconnected.

Description of Reference Numerals

[0085] 2000, 2000a, 2000b... Server system 2900... Attached storage 2910... Disk 4000... Storage system 4400... Volume 4400a... Basic volume 4401a... Virtual volume 4700... Server processor 5200... Parity group 5300... Pool 5400... Port 6000... Data catalog system 6320... Data catalog application 8000... Management system 8100... Server memory 8110... Data catalog management functions 8111, 8121, 8141, 8152... Settings 8120... Relationship identification program 8130... Topology database 8131... Topology object 8132... Topology link 8140... Application management function 8150... Storage management function 8151... Performance data 8300... Server processor 9000... Web client 9100... Graphical user interface

Claims

1. A management system for managing the relationship between a database and a volume of a storage system in which data used by the database is stored, wherein the management system has a processor, and the processor, accesses data including authentication information of the volume, which the database has, detects the number of accesses to the volume within a time range including the timing of access to the data, and manages the relationship between the database and the volume based on the number of accesses. A management system characterized by this.

2. The management system according to claim 1, wherein the processor identifies the volume in which the data used by the database is stored based on the number of accesses.

3. The management system according to claim 2, wherein the processor identifies the volume in which the number of accesses has increased within the time range.

4. The management system according to claim 1, wherein the processor accesses the data by making a read request for the data including the authentication information of the volume, and / or issuing a data preview command to the data.

5. The management system according to claim 1, wherein the number of accesses is at least one of IOPS, transfer rate, and cache hit rate for the volume.

6. The management system according to claim 1, wherein the processor manages the relationship between the database and the volume based on an anomaly score of the volume based on the number of accesses.

7. The management system according to claim 1, wherein the processor accesses a data catalog of the database including the authentication information of the volume.

8. The management system according to claim 1, wherein the processor detects a period during which the database and the volume are under low load, and accesses data including the authentication information of the volume within the period.

9. The management system according to claim 1, wherein the processor generates data for displaying a GUI for visualizing the relationship between the database and the volume.

10. A management method by a management system for managing the relationship between a database and a volume of a storage system storing data used by this database, comprising: accessing data including authentication information of the volume, which the database has; detecting the number of accesses to the volume in a time range including the timing of access to the data; managing the relationship between the database and the volume based on the number of accesses. A management method by a management system characterized by the above.

11. A computer program executed by a computer for managing the relationship between a database and a volume of a storage system storing data used by this database, comprising: when the computer program is executed by the computer, causing the computer to: access data including authentication information of the volume, which the database has; detect the number of accesses to the volume in a time range including the timing of access to the data; manage the relationship between the database and the volume based on the number of accesses. A computer program characterized by the above.

Images