A cloud-native-based data automatic acquisition method and device

Abstract

This invention discloses a cloud-native-based automatic data collection method and apparatus. The method comprises: 1) integrating a metadata probe module A1 into an application used to access a database, and then generating an image of the application; the database resides in a Docker environment; 2) setting the startup parameters of the image according to the actual usage scenario; 3) starting the image to capture user operation information on the database through the application; and then obtaining the database's metadata information from the captured operation information and reporting it to the enterprise metadata management platform. This invention is applicable to various cloud platforms and can achieve automatic, seamless, and accurate collection and reporting of metadata.

Description

Technical Field

[0001] This invention relates to the field of data governance, and specifically to a cloud-native method and apparatus for automatic metadata collection. Background Technology

[0002] In the digital economy era, data, as the most important new factor of production, is the foundation of digitalization, networking, and intelligence. It has been rapidly integrated into all aspects of production, distribution, circulation, consumption, and social service management, profoundly changing production methods and social governance methods.

[0003] Metadata is information about the organization, domains, and relationships of data; in other words, it describes the data itself. Metadata enables simple and efficient management of large amounts of networked data, facilitating the effective discovery, retrieval, integrated organization, and management of information resources. Only through unified management of metadata and the establishment of a clear data asset catalog can the problem of data silos be effectively solved, truly achieving data flow, aggregation, exchange, and sharing.

[0004] Currently, enterprises vary greatly in terms of industry, type, size, and products, resulting in diverse data. Their metadata templates and formats are inconsistent, leading to poor interoperability. Furthermore, metadata information is mainly collected and managed manually, which suffers from problems such as reliance on manual labor, untimely updates, and omissions or errors in statistical information. This makes it difficult for enterprises to fully and accurately grasp their internal metadata information, easily leading to the formation of data silos. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a cloud-native method and apparatus for automatic metadata collection, applicable to various cloud platforms, to achieve automatic, seamless, and accurate collection and reporting of metadata.

[0006] The technical solution adopted in this invention is:

[0007] A cloud-native automatic metadata collection device A, the collection device comprising:

[0008] Metadata probe module A1 uses JVM probe technology to read server web port traffic information from various heterogeneous data sources, uses bytecode technology to obtain database-related operations (such as CRUD operations on database instances, tables, fields, data, views, stored procedures, etc.) from the traffic information, and automatically forwards the database operation information to the following metadata analysis module A2 via HTTP.

[0009] The metadata analysis module A2 parses out the field information (i.e., metadata information) of the database table based on database operation-related information.

[0010] The metadata reporting module A3 reports the parsed metadata to the relevant enterprise metadata management system via HTTP or other means.

[0011] A cloud-native method for automatic metadata collection includes the following steps:

[0012] Step 1: The databases for which metadata is to be collected in this invention are all located in a Docker environment. Users generally access the database through an application. Therefore, the metadata probe module A1 is integrated into the application corresponding to the database to generate the corresponding image of the application.

[0013] Step 2: Manually set the boot parameters of the image according to the actual use case. The main parameters include:

[0014] P1: The type of database used by the application (Oracle, PostgreSQL, MySQL, and other mainstream databases);

[0015] P2: The connection string for the database used by the application (including IP, port, username, and password);

[0016] P3: The address of metadata analysis module A2;

[0017] P4-Pn: Parameters required for application business settings.

[0018] Step 3: The image starts normally. User operations on the database via the application can be captured by device A, and the current database metadata information can be analyzed and reported to the enterprise metadata management platform. The specific implementation process includes:

[0019] Step 3.1: The user performs a series of database operations through the application (including adding, deleting, modifying, and querying database instances, tables, fields, data, views, stored procedures, etc.). The metadata probe module A1 of device A obtains database-related operation information by reading the web port traffic information of the server where the database is located, and sends it to the metadata analysis module A2.

[0020] Step 3.2: The metadata analysis module A2 parses the field information (i.e., metadata information) of the database table from the above database-related operations. It then arranges and analyzes the database operation information according to time (for example, if a user queries field X at time t1, queries field Y at time t2, and modifies field Z to Z' at time t3, module A2 needs to comprehensively analyze these operations to obtain relatively complete metadata information {X, Y, Z'}, and send it to the metadata reporting module A3).

[0021] Step 3.3: The metadata reporting module A3 receives the metadata information for this period and converts it into a standardized metadata reporting message, which is then reported to the relevant enterprise metadata management system via HTTP or other means.

[0022] Step 3.4: In subsequent periods, repeat steps 3.1-3.3 above. By acquiring user operation information on the database tables, continuously improve and update metadata information and report it to the relevant enterprise metadata management system. Through the automatic metadata collection method provided by this invention, enterprises can promptly summarize and update metadata information of internal data, conduct unified management and format standardization, thereby solving the problems of independence and poor interoperability between various systems.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] This invention can accurately and timely collect metadata information from databases in cloud platforms and automatically report it, which helps enterprises to comprehensively, accurately and timely grasp internal metadata information, realize unified metadata management, and further lay the foundation for data aggregation, circulation and exchange sharing. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] Figure 1 This is a block diagram illustrating the structure of a cloud-native automatic metadata collection device A.

[0027] Figure 2 This is a flowchart illustrating a cloud-native method for automatic metadata collection.

[0028] Figure 3 This illustration shows an example of a data probe program A1 being integrated into an application system E according to an embodiment of the present disclosure.

[0029] Figure 4 This is a diagram illustrating the traditional manual statistical reporting of metadata.

[0030] Figure 5 This is a schematic diagram illustrating the automatic collection and reporting of metadata using this method. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0032] like Figure 1 The cloud-native automatic metadata collection device A shown mainly includes:

[0033] The metadata probe module A1 is used to read web port traffic information, obtain database-related operations, and automatically forward database operation information.

[0034] The metadata analysis module A2 parses out the database metadata information based on the database operation-related traffic information;

[0035] The metadata reporting module A3 reports to the deployed metadata management system via HTTP and other methods.

[0036] like Figure 2 The cloud-native-based automatic metadata collection method shown includes the following steps:

[0037] Step 1: E is an existing application deployed in a Docker environment, and the database used by E is W. Without modifying the source code of application E, create a base image of the metadata probe module A1 of device A and integrate it into application E. Adjust the Dockerfile and other cloud-native application configuration files according to the different requirements for deploying application E, and generate image E', as follows. Figure 3 As shown.

[0038] Step 2: Set the boot parameters for image E' according to the actual usage scenario. The main parameters include:

[0039] P1: The type of database W used by application E (Oracle, PostgreSQL, MySQL, and other mainstream databases);

[0040] P2: The connection string for the database used (including IP, port, username, and password);

[0041] P3: The address of metadata analysis module A2;

[0042] P4-Pn: Other parameters required for application business needs.

[0043] Step 3: Start image E normally. Metadata modifications made by the user to database W through application E can be automatically collected. The specific implementation process includes:

[0044] Step 3.1: The user performs operations on database W through application E (including storage operations on the database, and CRUD operations on instances, tables, fields, data, and views within the database). The metadata probe module A1 obtains relevant database operation information by mirroring the traffic information related to the above operations and sends it to the metadata analysis module A2. The following database is used as an example for illustration:

[0045] Table V of database W stores information about industrial and commercial enterprises in a certain region, as shown in Table 1. Initially, table V stores eight fields of information: "Company Name", "Registered Address", "Date of Establishment", "Legal Representative", "Registered Capital", "Size of Staff", "Contact Information", and "English Name".

[0046] Table 1 shows an example of database table V.

[0047] Company 1 Company 2 。。。

[0048] During time period T, the user performed the following database operations through application E:

[0049] At time t1, the user queries all field information corresponding to Company 1 in table V;

[0050] At time t2, the user changes the field name "Contact Information" in table V to "Contact Details".

[0051] At time t3, the user adds the field "Business Scope" to table V.

[0052] The metadata probe module A1 obtains the above database operation information through traffic mirroring and forwards it to the metadata analysis module A2.

[0053] Step 3.2: The metadata analysis module A2 sorts the database operation information by time and analyzes the key data of the relevant operations. The final metadata is summarized into 9 fields: {"Company Name", "Registered Address", "Date of Establishment", "Legal Representative", "Registered Capital", "Personnel Size", "Contact Information", "English Name", "Business Scope"}. This information is then sent to the metadata reporting module A3.

[0054] Step 3.3: The metadata reporting module A3 receives the metadata status information of table V within the above time period T, converts it into a standardized metadata reporting message, and reports it to the relevant system for enterprise metadata management via HTTP or other means.

[0055] Step 3.4: In the next time period T, repeat steps 3.1-3.3 above to obtain the update status of metadata and report it to the relevant enterprise metadata management system. This process is repeated. Through these steps, the enterprise aggregates all internal metadata information and updates it in a timely manner, facilitating unified metadata management and format standardization, thereby resolving the problems of data independence and poor interoperability between different systems.

[0056] like Figure 4 and Figure 5As shown in the comparison, compared with the existing manual statistical reporting method, the present invention can accurately and timely collect metadata information of the database in the cloud platform, and realize the automatic collection and reporting of metadata while ensuring metadata security.

[0057] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cloud-native-based automatic data collection method, comprising the following steps: 1) Integrate the metadata probe module A1 into the application used to access the database, and then generate an image of the application; The database is located in a Docker environment; 2) Set the startup parameters of the image according to the actual usage scenario; the startup parameters include the type of database used by the application, the connection string of the database used by the application, the address of the metadata analysis module A2, and the parameters required for the application business settings; 3) Start the image to capture user operation information on the database through the application; Then, the metadata information of the database is obtained from the captured operation information and reported to the enterprise metadata management platform; The method for obtaining the metadata information of the database from the captured operation information is as follows: 31) The metadata probe module A1 in the mirror obtains the operation information by reading the web port traffic information of the server where the database is located in the current time period, and sends it to the metadata analysis module A2; 32) The metadata analysis module A2 parses the metadata information of the database from the operation information; sorts the metadata corresponding to each operation information according to the operation time, and then sends the sorted metadata to the metadata reporting module A3; 33) The metadata reporting module A3 receives the metadata information for the current time period, converts it into a metadata reporting message, and reports it to the enterprise metadata management platform; 34) Repeat steps 31) to 33) above in each subsequent time period. The operation information includes storage operations on the database and add, delete, modify and query operations on instances, tables, fields, data and views in the database.

2. The method of claim 1, wherein, The metadata reporting module A3 reports the metadata to the enterprise metadata management platform via HTTP.

3. The method of claim 1, wherein, The metadata probe module A1 uses JVM probe technology to read the web port traffic information, and then uses bytecode technology to obtain the operation information from the web port traffic information.

4. The method of claim 1, wherein, The connection string includes IP address, port number, username, and password.

5. A cloud-native based data automatic acquisition device, characterized in that, include: Metadata probe module A1 is used to capture user operation information on the database through the application and forward it to metadata analysis module A2. Metadata probe module A1 is integrated into the application used to access the database, generates an image of the application, and sets the startup parameters of the image. The database resides in a Docker environment. The startup parameters include the type of database used by the application, the connection string of the database used by the application, the address of metadata analysis module A2, and parameters required for application business settings. The operation information is obtained by reading the web port traffic information of the server where the database resides during the current time period. Metadata analysis module A2 is used to obtain the metadata information of the database from the captured operation information and forward it to metadata reporting module A3. The method for obtaining the metadata information of the database from the captured operation information is as follows: 31) The metadata probe module A1 in the mirror obtains the operation information by reading the web port traffic information of the server where the database is located in the current time period, and sends it to metadata analysis module A2; 32) The metadata analysis module A2 parses the metadata information of the database from the operation information; sorts the metadata corresponding to each operation information according to the operation time, and then sends the sorted metadata to metadata reporting module A3; 33) Metadata reporting module A3 receives the metadata information in the current time period and converts it into a metadata reporting message and reports it to the enterprise metadata management platform; 34) In each subsequent time period, the above steps 31) to 33) are repeated. The operation information includes storage operations on the database and add, delete, modify and query operations on instances, tables, fields, data and views in the database.

Images