A Distributed Method and System for Proactive Discovery of Database Metadata

Abstract

This invention relates to the field of database metadata discovery technology, and provides a distributed proactive database metadata discovery method and system, comprising: receiving a database discovery task issued by a management terminal, parsing the control zone policy, and starting a scanning task; parsing the IP address of the target node from the scanning task, scanning the target node, and detecting target node information during the scanning process; parsing the target node information, writing it into an array, and sending it to the management terminal to obtain the metadata discovery task returned by the management terminal; parsing the database connection configuration from the metadata discovery task, establishing a database connection, and after successful connection, starting to read and parse metadata. For the management terminal, this eliminates the need for extensive business logic, making management more flexible and convenient.

Description

Technical Field

[0001] This invention belongs to the field of database metadata discovery technology, and in particular relates to a distributed proactive database metadata discovery method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, the traditional approach to proactive database metadata discovery is to use a single-node deployment and a single-line process model for database metadata discovery, such as... Figure 2 As shown, the general steps can be divided into: configuring data source discovery information, mainly configuring the IP address, port, database type, and driver attributes of the database to be discovered; starting a scan to check if the IP address and port number are pingable and open; checking connectivity, returning a connection failure message if a connection failure is detected, and prompting the user to enter their username and password if the connection is successful; verifying the correctness of the input, mainly verifying the database driver and the correctness of the target IP address, username, and password; and starting to obtain database source data information if the input is correct and the connection is successful.

[0004] The aforementioned implementation of proactive database metadata discovery can only meet the needs of small and medium-sized enterprises with small amounts of data and a small data distribution range.

[0005] Large enterprises typically establish subsidiaries across the country or even the world. These subsidiaries usually build various self-developed systems to support their production, operations, and office functions. These databases or data platforms are numerous, widely distributed, and have complex sources, making them more prone to data security issues.

[0006] Using traditional database metadata discovery technology to support related security products will increase the cost of managing and maintaining data assets, make data reporting difficult, and pose serious risks to enterprise data security. Summary of the Invention

[0007] To address the technical problems mentioned above, this invention provides a distributed database metadata proactive discovery method and system. It interacts with the management end through a message queue, receives scan and listen tasks, synchronizes status heartbeats, and uploads scan and listen logs. For the management end, there is no need to perform excessive business operations, making management more flexible and convenient, and reducing the control and maintenance costs of data assets.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides a distributed proactive database metadata discovery method, comprising:

[0010] Upon receiving the database discovery task from the management terminal, the control area policy is parsed and the scanning task is started.

[0011] After resolving the IP address of the target node from the scanning task, the target node is scanned, and target node information is detected during the scanning process;

[0012] After parsing the target node information, it is written into an array and sent to the management terminal to obtain the metadata discovery task returned by the management terminal;

[0013] The metadata discovery task parses the database connection configuration, establishes a database connection, and once the connection is successful, it begins reading and parsing the metadata.

[0014] Furthermore, the target node information includes: whether the target node is active, port open status, type of database being run, version information, and the type of operating system and device being detected.

[0015] Furthermore, the information written into the array includes: database IP address, port, and database type.

[0016] Furthermore, after obtaining the metadata discovery task returned by the management terminal, a JDBC driver is adapted, and the database connection parameters are encapsulated.

[0017] Furthermore, the metadata is parsed to extract instance information, library information, table information, and field information.

[0018] Furthermore, it also includes:

[0019] The data parsed from the metadata and the database connection configuration are encapsulated into data source information;

[0020] Parse the data source information and write the database connection configuration, database information, table information, and field information into the database for persistent storage.

[0021] Furthermore, it also includes: generating data source discovery logs and passing the data source discovery logs to the management end.

[0022] A second aspect of the present invention provides a distributed database metadata proactive discovery system, which includes a discovery subsystem main module and a discovery subsystem scanning component;

[0023] The main module of the discovery subsystem is configured to: receive a database discovery task issued by the management terminal, parse the control area policy, and then start a scanning task;

[0024] The detection subsystem scanning component is configured to: after resolving the IP address of the target node from the scanning task, scan the target node, and detect target node information during the scanning process;

[0025] The main module of the discovery subsystem is also configured to: parse the target node information and write it into an array and send it to the management terminal to obtain the metadata discovery task returned by the management terminal;

[0026] The discovery subsystem scanning component is also configured to: parse the database connection configuration from the metadata discovery task, establish a database connection, and after a successful connection, begin reading and parsing the metadata.

[0027] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a distributed database metadata proactive discovery method as described above.

[0028] A fourth aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a distributed database metadata proactive discovery method as described above.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] This invention provides a distributed proactive database metadata discovery method. It uniformly encapsulates the main module of the discovery subsystem, primarily used to receive scan and listener tasks, synchronize status heartbeats, upload scan and listener logs, and interact with the management terminal via a message queue. For the management terminal, when scanning the database, it only needs to distribute tasks to each discovery subsystem; when acquiring data, it only needs to grasp and collect the status and data information of each registered subsystem, without needing to perform excessive business logic, making management more flexible and convenient.

[0031] This invention provides a distributed proactive database metadata discovery method. Before implementing network scanning, user authorization is required. During implementation, the user can control the intensity of the scan, limit the scope of the scan, and provide a data collection list to prove to the customer that data outside the list will not be collected.

[0032] This invention provides a distributed proactive database metadata discovery method. Each discovery subsystem is responsible for its own corresponding scanning task, and the tasks are independent of each other, making it easy to maintain and control. Each discovery subsystem can be deployed and upgraded independently without affecting other discovery subsystems. Each discovery subsystem can select different servers according to actual business or data volume, can independently adjust hardware, and can be flexibly integrated into different technologies and frameworks. When a discovery subsystem task fails, it only affects its own or a small number of other discovery subsystems, without affecting the entire proactive database metadata discovery system on a large scale. Attached Figure Description

[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0034] Figure 1 This is a flowchart of a distributed database metadata proactive discovery method according to Embodiment 1 of the present invention;

[0035] Figure 2 This is a flowchart of the traditional implementation method for proactively discovering database metadata. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] Example 1

[0039] This embodiment provides a distributed method for proactive discovery of database metadata, such as... Figure 1 As shown, it is divided into two stages: the database discovery stage and the metadata discovery stage.

[0040] The database discovery phase can be roughly divided into the following steps:

[0041] Step 1: First, configure the discovery policy on the management side, mainly configuring the control area policy (the range of IP addresses to be scanned and the IP segments to be excluded). Simultaneously, enable heartbeat detection on the management side to check if the currently registered discovery subsystem modules are alive, facilitating status monitoring and the issuance of database discovery tasks. The discovery subsystem modules include a main module and a scanning component module.

[0042] Step 2: Issue a database discovery task through the management terminal. When the database discovery task is started, the management terminal will first check whether the main module of the discovery subsystem with the same IP address range is registered under the control area of ​​the task. If not, the task will be stopped. If so, the database discovery task will be issued to the corresponding main module of the discovery subsystem.

[0043] There are many ways to interact with data. You can replace pushing data via message queues with directly writing to a public server or database. Alternatively, you can replace message queues with API calls.

[0044] If the API interface call method is used, then step 2 is as follows: the database discovery task is issued through the management terminal. When the database discovery task is started, the management terminal will first check whether the discovery subsystem main module with the same IP address range is registered under the control area of ​​the task. If not, the task will be stopped. If so, the database discovery task API provided by the discovery subsystem main module will be called to issue the database discovery task to the corresponding discovery subsystem main module.

[0045] Step 3: After receiving the database discovery task from the management terminal, the main module of the discovery subsystem will first parse the content to obtain the control area policy, and start the scanning task according to the control area policy. During the startup process, the database discovery task will not be executed immediately, but the scanning task will be placed in the first message queue.

[0046] If the API interface call method is used, then step 3 is as follows: After the main module of the discovery subsystem receives the API request, it will first parse the content to obtain the control area policy, and start the scanning task according to the control area policy. During the startup process, the main module calls the API provided by the scanning component to start the scanning task and sends the database discovery task to the scanning component.

[0047] Step 4: The discovery subsystem scanning component begins executing the database discovery task:

[0048] Step 4.1: The discovery subsystem scanning component actively pulls the scanning tasks from the first message queue into its own memory and resolves the IP address of the target node (server), and then begins scanning the target node.

[0049] If the API interface call method is used, then step 4.1 is as follows: After the detection subsystem scanning component receives the API request for the scanning task, it calls the task function to execute, parses the API parameters to obtain the IP address of the target node (server), and then starts scanning the target node.

[0050] Step 4.2: During the scanning process, the target node information of the discovery subsystem scanning component is discovered, specifically including: detecting whether the target node is active and whether its ports are open, detecting the type and version information of the running database, detecting the operating system and device type, etc., and transmitting the content to the main module of the discovery subsystem through the second message queue.

[0051] If the API interface call method is used, then step 4.2 is as follows: During the scanning process, the target node information of the discovery subsystem scanning component is discovered, specifically including: detecting whether the target node is active and whether the port is open, detecting the type and version information of the running database, detecting the operating system and device type, etc., and calling the database list API provided by the main module to submit the content to the main module of the discovery subsystem.

[0052] Step 4.3: The main module receives the information sent by the scanning component, parses out the discovered database IP address, port, and database type, writes them into an array, and sends them to the management end. The database discovery phase ends.

[0053] If the API interface call method is used, then step 4.3 is as follows: The main module receives the content sent by the scanning component, parses out the discovered database IP address, port, and database type and writes them into an array, then calls the database list API provided by the management terminal to submit the array to the management terminal, and the database discovery phase ends.

[0054] The metadata discovery phase can be roughly divided into the following steps:

[0055] Step 5: Provide account and password, and issue metadata discovery task.

[0056] Step 5.1: Obtain the database list from the database discovery phase through the management terminal. At this point, you will need to provide the account and password corresponding to the database.

[0057] Step 5.2: Based on the provided database account and password, the management terminal issues another metadata discovery task.

[0058] If the API interface call method is used, then step 5.2 is as follows: Based on the provided database account and password, the management terminal calls the metadata discovery task API provided by the main module of the discovery subsystem to issue the metadata discovery task again.

[0059] Step 6: After the main module of the discovery subsystem receives the task issued by the management terminal, it adapts the received content to the corresponding JDBC (Java Database Connectivity) driver, then encapsulates the key database connection parameters (database connection address, port number, database username, database password, database type) and puts the metadata discovery task into the third message queue.

[0060] If the API interface call method is used, then step 6 is as follows: After the main module of the discovery subsystem receives the task issued by the management terminal, it adapts the received content to the corresponding JDBC (Java Database Connectivity) driver, then encapsulates the key parameters of the database connection, and calls the API provided by the scanning component to start the metadata discovery task, and sends the metadata discovery task to the scanning component.

[0061] It's important to note that the discovery subsystem module includes a main module and a scanning component module. The main module primarily receives tasks from the management end, parses them, and then sends them to the scanning component via a message queue. The scanning component is where the database connection actually occurs, so it requires the parameters encapsulated by the main module. The main module uses encapsulated data to send to the message queue to control the concurrency of the scanning component's threads. When multiple scanning tasks arrive, they execute simultaneously; the message queue controls the concurrency to prevent a large workload from overwhelming the scanning component. Furthermore, the main module and the scanning component can be deployed separately on different servers, and using a message queue to transmit data decouples the system.

[0062] Step 7: The discovery subsystem scanning component begins executing the metadata discovery task:

[0063] Step 7.1: The discovery subsystem scanning component actively pulls the metadata discovery task from the third message queue, puts it into its own memory, parses out the database connection configuration, and begins to establish a database connection.

[0064] If the API interface call method is used, then step 7.1 is as follows: After the detection subsystem scanning component receives the API request, it calls the task function to execute, parses the database connection configuration, and starts the database connection.

[0065] Step 7.2: After a successful connection, begin reading metadata, and parse key instance information, database information, table information, and field information according to the database DDL language.

[0066] Specifically, instance information includes: the instance name of the database; database information includes: database name and encoding format; table information includes: table name, encoding format, table size, creation time, and remarks; field information includes: field name, field type, field length, whether it is a primary key, whether it is a foreign key, encoding format, and remarks.

[0067] Step 7.3: Encapsulate the above information and connection information (database connection configuration) into data source information and place it in the fourth message queue.

[0068] If the API interface call method is used, then step 7.3 is: encapsulate the above information and connection information (database connection configuration) into data source information, call the data source information API provided by the main module, and submit the encapsulated data source information to the main module.

[0069] Step 8: The main module of the discovery subsystem begins parsing the data source information:

[0070] Step 8.1: Parse the data source information and write the connection information, database information, table information, and field information into the corresponding database for persistent storage;

[0071] Step 8.2: Generate data source discovery logs;

[0072] Step 8.3: Write the data source discovery log to the fifth message queue.

[0073] If the API interface call method is used, then step 8.3 is: the main module calls the data source discovery log API provided by the management terminal to submit the data source discovery log to the management terminal.

[0074] That is, after the main module of the discovery subsystem parses the database metadata information and writes it to the corresponding database, it generates a data source discovery log and writes it to the fifth message queue to be passed to the management end. The database metadata discovery phase ends, and the active discovery of database metadata is completed.

[0075] Step 9: After receiving the data source discovery log from the fifth message queue, the management terminal saves the data source discovery log to the server, and the database metadata discovery phase ends. At this point, the active discovery of database metadata is complete.

[0076] If an API call is used, step 9 is as follows: After receiving the API request, the management end saves the received data source discovery log to the server, ending the database metadata discovery phase. This completes the proactive database metadata discovery. This embodiment provides a distributed proactive database metadata discovery method. The management end monitors the connection and process monitoring capabilities of the discovery subsystem via remote execution of a shell script. Before network scanning, user authorization is required. During implementation, the user can control the scan intensity, limit the scan scope, and provide a data collection list to demonstrate to the client that data outside the list will not be collected. The discovery subsystem's main module is uniformly encapsulated, primarily used to receive scan listening tasks, synchronize status heartbeats, upload scan listening logs, and interact with the management end via a message queue.

[0077] This embodiment provides a distributed proactive database metadata discovery method. Each discovery subsystem is responsible for its own corresponding scanning task, and the tasks are independent of each other, making it easy to maintain and control. Each discovery subsystem can be deployed and upgraded independently without affecting other discovery subsystems. Each discovery subsystem can select different servers based on actual business needs or data volume, can independently adjust hardware, and can be flexibly integrated into different technologies and frameworks. When a discovery subsystem task fails, it only affects its own subsystem or a small number of other discovery subsystems, without significantly impacting the entire proactive database metadata discovery system. For the management end, scanning the database only requires issuing tasks to each discovery subsystem, and data acquisition only requires monitoring and collecting the status and data information of each registered subsystem, without requiring extensive business logic, making management more flexible and convenient.

[0078] This embodiment provides a distributed proactive database metadata discovery method. This method proactively discovers basic metadata such as deployed databases, tables, and fields within the user's network environment, collects it, and provides it to data security products. Additionally, as a supplement, it provides collection and parsing middleware and database audit logs to extract, analyze, and save the above data.

[0079] As a crucial component of the basic data infrastructure for data security products, the completeness and compliance of the basic data collected by proactive database metadata discovery technology directly affect the product's functionality and its widespread adoption.

[0080] Therefore, the core objective of proactive database metadata discovery is to collect and discover the aforementioned data in a compliant and complete manner, with user permission, to form a full dataset. Whether it's a single-node or distributed proactive database metadata discovery system, the product development goal is to support multiple methods, and the appropriate deployment method can be selected based on the customer's permission situation.

[0081] Example 2

[0082] This embodiment provides a distributed database metadata proactive discovery system, which adopts a plug-and-play system design to achieve the purpose of a distributed system architecture. Specifically, it includes two major modules: a management terminal and a discovery subsystem.

[0083] The management console primarily configures discovery policies, issues tasks, monitors the connection and process information of the host where the current discovery subsystem resides, and receives data source discovery information reported by the discovery subsystem.

[0084] The discovery subsystem is divided into a main module and a scanning component module. The main module is responsible for registering its own information with the management terminal, receiving tasks initiated by the management terminal and scheduling the execution of tasks, and reporting its survival status and scanning progress to the management terminal using a heartbeat mechanism. The scanning component module is responsible for actually executing the scanning tasks and returning the scanning results to the main module.

[0085] The main module of the discovery subsystem supports two deployment methods. The first is remote deployment via the management terminal, which requires providing the IP address of the deployment server, the SSH port number, and the account and password for connection to send packets remotely for installation. After successful installation, it will be directly registered to the management terminal. The second is manual deployment to the corresponding server. After successful installation, it will actively send the current server information, including IP address, port number, service name, and cluster name, to the management terminal, which will then register it in the list.

[0086] The main module of the discovery subsystem is configured to: receive a database discovery task issued by the management terminal, parse the control area policy, and then start a scanning task;

[0087] The detection subsystem scanning component is configured to: after resolving the IP address of the target node from the scanning task, scan the target node and detect target node information during the scanning process;

[0088] The main module of the discovery subsystem is also configured to: parse the target node information, write it into an array, and send it to the management terminal to obtain the metadata discovery task returned by the management terminal;

[0089] The discovery subsystem scanning component is also configured to: parse the database connection configuration from the metadata discovery task, establish a database connection, and, after a successful connection, begin reading and parsing the metadata.

[0090] This embodiment provides a distributed database metadata proactive discovery system. The overall architecture adopts a distributed master-slave architecture, that is, the architecture has two roles: Leader and Follower. The Leader is responsible for scheduling tasks and collecting data sent by the Followers, while the Followers are responsible for synchronizing data to the Leader for management.

[0091] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0092] Example 3

[0093] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a distributed database metadata proactive discovery method as described in Embodiment 1 above.

[0094] Example 4

[0095] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the distributed database metadata proactive discovery method described in Embodiment 1 above.

[0096] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed method for proactive discovery of database metadata, characterized in that, include: Upon receiving the database discovery task from the management terminal, the control area policy is parsed and the scanning task is started. The management system will first check whether the main module of the discovery subsystem with the same IP address range has been registered under the control area of ​​the task. If not, the task will be stopped. If it is, the database discovery task will be sent to the corresponding main module of the discovery subsystem. After resolving the IP address of the target node from the scanning task, the target node is scanned, and target node information is detected during the scanning process; The discovery subsystem scanning component actively pulls the scanning tasks from the first message queue into its own memory, resolves the IP address of the target node, and then begins scanning the target node; The discovery subsystem scans the target node information of the component and transmits the content to the main module of the discovery subsystem through the second message queue; The target node information includes: whether the target node is active, port open status, type of database being run, version information, and the type of operating system and device being detected; After parsing the target node information, it is written into an array and sent to the management terminal to obtain the metadata discovery task returned by the management terminal; After obtaining the metadata discovery task returned by the management terminal, adapt the JDBC driver, encapsulate the database connection parameters, and put the metadata discovery task into the third message queue; the discovery subsystem scanning component actively pulls the metadata discovery task from the third message queue. The database connection configuration is parsed from the metadata discovery task, a database connection is established, and after a successful connection, the metadata is read and parsed. Generate a data source discovery log and pass it to the management terminal; The main module and scanning components are deployed on different servers, and a message queue is used to pass data to decouple the system.

2. The distributed database metadata proactive discovery method as described in claim 1, characterized in that, The information written into the array includes: database IP address, port, and database type.

3. The distributed database metadata proactive discovery method as described in claim 1, characterized in that, The metadata is parsed to extract instance information, library information, table information, and field information.

4. The distributed database metadata proactive discovery method as described in claim 1, characterized in that, Also includes: The data parsed from the metadata and the database connection configuration are encapsulated into data source information; Parse the data source information and write the database connection configuration, database information, table information, and field information into the database for persistent storage.

5. A distributed database metadata proactive discovery system employing the method described in claim 1, characterized in that, This includes the main module of the discovery subsystem and the scanning component of the discovery subsystem; The main module of the discovery subsystem is configured to: receive a database discovery task issued by the management terminal, parse the control area policy, and then start a scanning task; The detection subsystem scanning component is configured to: after resolving the IP address of the target node from the scanning task, scan the target node, and detect target node information during the scanning process; The main module of the discovery subsystem is also configured to: parse the target node information and write it into an array and send it to the management terminal to obtain the metadata discovery task returned by the management terminal; The discovery subsystem scanning component is also configured to: parse the database connection configuration from the metadata discovery task, establish a database connection, and after a successful connection, begin reading and parsing the metadata.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in a distributed proactive database metadata discovery method as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the distributed database metadata proactive discovery method as described in any one of claims 1-4.

Images