Metadata collection method, device and storage medium
By defining the connection class and collection class in the data source connection SDK and metadata collection SDK, the problem of low efficiency in collecting metadata from multiple microservices is solved, achieving more efficient metadata collection and a simplified management process.
Patent Information
- Application Number
- CN202411096408.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-12
AI Technical Summary
Existing technologies that collect metadata through multiple microservices are inefficient and have cumbersome and difficult-to-maintain thread concurrency management.
By determining the data source type field value corresponding to the collection task, the data source connection SDK and metadata collection SDK are passed to determine the connection class and collection class, obtain the connection parameters, connect to the target database, and collect the target metadata.
It improves the efficiency of metadata collection, simplifies the management process, and reduces the complexity caused by thread concurrency.
Smart Images

Figure CN118626550B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a metadata collection method, device and storage medium. Background Art
[0002] Metadata is data that describes data. It is a description of the organization, domain, and relationship of data. Currently, metadata collection usually involves determining the corresponding collection tasks based on the metadata collection strategy configured on the front end, and starting multiple services in the form of microservices to perform the collection tasks, so as to complete the parallel collection of metadata from multiple data sources.
[0003] However, in actual application, when collecting metadata through multiple microservices, it is necessary to consider the issue of thread concurrency. Multi-thread concurrency means that multiple threads switch to run, and the management and maintenance of thread switching is cumbersome and difficult to maintain, which leads to low efficiency of metadata collection in related solutions.
[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The present application provides a metadata collection method, device and storage medium, aiming to solve the problem of low efficiency in metadata collection by multiple microservices.
[0006] To achieve the above purpose, the present application provides a metadata collection method, which includes the following steps:
[0007] When the triggering condition of the collection task is met, determining the data source type field value corresponding to the collection task;
[0008] Passing the data source type field value as a parameter to the data source connection SDK, and determining the connection class in the data source connection SDK;
[0009] Acquire the connection parameters in the connection class, connect to the target database according to the connection parameters, and determine the collection class in the metadata collection SDK according to the data source type field value;
[0010] The collection parameters in the collection class are acquired, and the target metadata in the target data source is collected according to the collection parameters.
[0011] Optionally, before the step of determining the data source type corresponding to the collection task when the triggering condition of the collection task is met, the step further includes:
[0012] Determine configuration parameters of each data source, and build a data source management model according to the configuration parameters;
[0013] Creating the collection task according to a preset time plan, and saving the data source management model in association with the collection task;
[0014] When the preset time schedule arrives and / or when the start instruction is triggered, it is determined that the triggering condition of the acquisition task is met.
[0015] Optionally, after the step of constructing the data source management model according to the configuration parameters, the method further includes:
[0016] Get the preset data source connection method and determine the data source type enumeration value;
[0017] Using the data source type enumeration value as a prefix of a class name, determining the class name of each data source connection method, and generating a corresponding connection class;
[0018] The connection classes are integrated to generate the data source connection SDK, so that the management end calls the connection class in the data source connection SDK according to the class name.
[0019] Optionally, after the step of constructing the data source management model according to the configuration parameters, the method further includes:
[0020] Creating the metadata collection SDK, determining a preset metadata collection method, and obtaining data source connection information and data source type enumeration values;
[0021] Using the data source type enumeration value as the class name prefix of the metadata collection method, determining the class name of each metadata collection method, and generating a corresponding collection class;
[0022] Integrate the various acquisition classes and place them into the metadata acquisition SDK.
[0023] Optionally, the metadata collection method further includes:
[0024] Determine the data source connection SDK and the target class in the metadata collection SDK, wherein the target class is the collection class or the connection class;
[0025] When the target class is called, the class name of the target class and the corresponding class method are obtained;
[0026] The class name is used as a unique attribute, and the class method is used as a value, and is saved in a cache, so that when the class is subsequently called, the corresponding class function is directly called in the cache according to the class name.
[0027] Optionally, after the step of acquiring the collection parameters in the collection class and collecting the target metadata in the target data source according to the collection parameters, the step further includes:
[0028] Monitor the collection task, and after the collection task is completed, record the batch number corresponding to the collection task;
[0029] When the metadata changes, the version number of the target data source corresponding to the acquisition task is recorded;
[0030] A record log is generated and saved according to the batch number and the version number.
[0031] Optionally, after the step of generating and saving a version record log according to the batch number and the version number, the method further includes:
[0032] Obtaining the record log, and determining the version number according to the record log;
[0033] When the metadata change does not conform to the preset plan, the corresponding target metadata is determined according to the version number;
[0034] According to the target metadata, the changed data in the target data source is rolled back and saved.
[0035] Optionally, after the step of determining the data source type corresponding to the collection task when the triggering condition of the collection task is met, the method further includes:
[0036] Get the preset timeout threshold;
[0037] When the connection duration of the acquisition task is equal to the timeout threshold, and / or the acquisition duration is equal to the timeout threshold, determining the acquisition parameters corresponding to the acquisition task in the data source management model;
[0038] The acquisition parameters and corresponding prompt information are fed back to the client, so that the client displays the acquisition parameters and the prompt information.
[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a metadata collection device, which includes: a memory, a processor, and a metadata collection program stored in the memory and executable on the processor, wherein the metadata collection program is configured to implement the steps of the metadata collection method as described above.
[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, on which a metadata collection program is stored, and when the metadata collection program is executed by a processor, the steps of the metadata collection method described above are implemented.
[0041] The present application provides a metadata collection method, a metadata collection device and a storage medium. When a trigger condition of a collection task is met, the data source type field value corresponding to the collection task is determined, and then the data source type field value is passed as a parameter to a data source connection SDK and a metadata collection SDK, and a connection class and a collection class matching the parameters are determined in the data source connection SDK and the metadata collection SDK, and then a connection parameter is determined in the connection class, and a target data source is determined and connected according to the connection parameter, and, in addition, a collection parameter is determined in the collection class, and target metadata in the target data source is collected according to the collection parameter, so as to improve metadata collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a flowchart of the first embodiment of the metadata collection method of the present application;
[0043] Figure 2 This is a flow chart of the second embodiment of the metadata collection method of the present application;
[0044] Figure 3 This is a flowchart of the data source connection SDK involved in the embodiment of the present application;
[0045] Figure 4 This is a flowchart of a third embodiment of the metadata collection method of the present application;
[0046] Figure 5 This is a flowchart of the implementation of the metadata collection SDK involved in the embodiment of the present application;
[0047] Figure 6 This is a flowchart of a fourth embodiment of the metadata collection method of the present application;
[0048] Figure 7 This is a flowchart of the fifth embodiment of the metadata collection method of the present application;
[0049] Figure 8 This is a flowchart of a sixth embodiment of the metadata collection method of the present application;
[0050] Fig. 9 This is a schematic diagram of the architecture of the hardware operating environment of the metadata collection device involved in the embodiment of the present application.
[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0052] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0053] When the triggering condition of the collection task is met, the present application determines the data source type field value corresponding to the collection task, passes the data source type field value as a parameter to the data source connection SDK, and determines the connection class in the data source connection SDK; obtains the connection parameters in the connection class, connects to the target database according to the connection parameters, and determines the collection class in the metadata collection SDK according to the data source type field value; obtains the collection parameters in the collection class, and collects the target metadata in the target data source through the collection parameters, so as to solve the technical problem of low efficiency in collecting metadata by multiple microservices, thereby improving the efficiency of metadata collection.
[0054] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0055] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0056] In addition, the JDBC (Java Database Connectivity) in this application is an application program interface in the Java language that regulates how client programs access databases, and provides methods for querying and updating data in the database. By calling JDBC, Java programs can connect to the database and execute SQL (Structured Query Language) to query and operate the database. The SDK (Software Development Kit) in this application contains a series of tools and libraries required to develop a specific software or application, such as APIs, documents, sample codes, debuggers, etc.
[0057] Embodiment 1
[0058] Please refer to Figure 1 In a first embodiment, the metadata collection method includes the following steps:
[0059] Step S10: When the triggering condition of the collection task is met, determine the data source type field value corresponding to the collection task.
[0060] In this embodiment, the collection task includes but is not limited to data source type information and type information of metadata to be collected, wherein the data source type information includes links to various types of databases and fields corresponding to database type names. When the trigger condition is triggered, the metadata collection system automatically schedules the collection task and starts the corresponding collection process according to the collection task, thereby obtaining the data source type field value contained in the collection task according to the collection process.
[0061] Furthermore, in this embodiment, before step S10, the following steps are further included:
[0062] Determine the configuration parameters of each data source, and build a data source management model based on the configuration parameters; create the collection task according to the preset time plan, and save the data source management model in association with the collection task; when the preset time plan arrives and / or when the start instruction is triggered, determine whether the trigger condition of the collection task is met.
[0063] Specifically, the configuration parameters of the data source include the data source address (host), port number (port), and database name (database Name), which are key parameter information constituting the JDBC URL (Java Database Connectivity Uniform Resource Locator, a string used to identify and connect to the database). The configuration parameters can be uniformly configured and input through the page, or imported into the page in the form of a file, and the page identifies the data in the file and feeds it back to the background system. The background system configures the data source management model according to the configuration parameters to realize the visualization metadata collection data source connection configuration management. After configuring the configuration parameters of the data source, the background system will obtain a preset time plan, which is used to configure the execution cycle of the collection task, that is, to execute the collection task at a specified time and stop executing the collection task after the preset execution time. Finally, the data source management model is saved in the created collection task. When the above-mentioned specified time arrives, the execution instruction associated with the collection task will be automatically called, or the user selects the control corresponding to the start instruction in the interface corresponding to the background system based on the selection device, and triggers the control to trigger the start instruction, so that the background system automatically executes the collection task.
[0064] It is understandable that the data source management model is also configured with the user name, password, etc. of each data source, so that the database connection can be carried out through account and password authentication. By adopting the optional database authentication configuration provided by the data source management model, corresponding authentication parameters are provided for the data source of the enabled authentication information to improve the security of the database connection.
[0065] As an optional implementation method, the collection task can also be determined by configuring scheduling. First, the scheduling requirements are determined, including the frequency, start and end time, priority, etc. of the task. Then, according to the scheduling requirements, a suitable scheduling component is configured, wherein the scheduling component can be a special scheduling software, or a task management tool with a scheduling function. Then, the task information is entered in the scheduling parameter setting part of the scheduling tool or system, including the task name, description, priority, start and end time, etc., and according to the scheduling requirements, the corresponding scheduling rules are configured, that is, the execution frequency, priority rules, etc. of the task are set, and finally, the completed collection task is saved.
[0066] Furthermore, in this embodiment, after step S10, the following steps are further included:
[0067] Obtain a preset timeout threshold; when the connection duration of the acquisition task is equal to the timeout threshold, and / or the acquisition duration is equal to the timeout threshold, determine the acquisition parameters corresponding to the acquisition task in the data source management model; feed back the acquisition parameters and corresponding prompt information to the client, so that the client displays the acquisition parameters and the prompt information.
[0068] Specifically, the preset timeout threshold is configured in the data source management model in the form of a timeout parameter. The timeout parameter can be configured through a page, and the collection task is automatically timed when the collection task is executed. When the connection process of the collection task exceeds the timeout threshold, or the duration of the collection process exceeds the timeout threshold, it is determined that there is a problem with the class of the data source connection SDK or the metadata collection SDK in the data source management model. At this time, it is necessary to redetermine the target parameter according to the collection task, and the target parameter includes prompt information and corresponding parameters. The target parameter is fed back to the client so that the management personnel can adjust the class function or class method of the data source connection SDK and the metadata collection SDK according to the target parameter, thereby improving the management efficiency and application value of the metadata.
[0069] For example, when collecting metadata of a Hive data source, Kerbros authentication is usually enabled to ensure the security of the Hive data source. Therefore, it is necessary to dynamically select the connection authentication form in the configuration of the corresponding Hive data source to ensure smooth connection to the Hive data source.
[0070] Step S20: passing the data source type field value as a parameter to the data source connection SDK, and determining the connection class in the data source connection SDK.
[0071] In this embodiment, after the background system obtains the data source type field value, it will automatically pass the data source type field value as the parameter to the data source connection SDK through data input parameters, wherein the data source connection SDK stores connection classes for multiple data sources. After the data source connection SDK obtains the parameter through input parameters, it determines the class name that matches it based on the parameter, and then calls the connection class that matches the parameter. Similarly, the data source type field value is used as a parameter of the metadata collection SDK through input parameters and passed to the metadata collection SDK to obtain a collection class that matches the parameter.
[0072] Specifically, the input parameter refers to the parameter of the method or constructor, which can be used to receive the value or data passed from the caller for use inside the method or constructor body. That is, by receiving the data source type field value fed back by the client of the backend system, and using it in the data source connection SDK and the metadata collection SDK.
[0073] As an optional implementation method, the connection class and the collection class can also be obtained through a class reflection mechanism, that is, through the class reflection mechanism, when the data source type is obtained as a parameter, an object is dynamically created, and various properties and methods of the corresponding class are obtained through the object and according to the parameters.
[0074] Specifically, the class reflection mechanism is an important feature of the Java programming language, which can dynamically obtain class information at runtime and create and operate objects at runtime.
[0075] Step S30: obtaining connection parameters in the connection class, connecting to the target database according to the connection parameters, and determining the collection class in the metadata collection SDK according to the data source type field value.
[0076] In this embodiment, the connection class includes connection parameters of the target data source, and the connection parameters include the data source address, port number, database name, and user name and password for authentication of the specified database. The target data source to be connected is determined by the connection parameters, and authentication is performed according to the user name and password in the connection parameters, so as to connect to the target data source.
[0077] As an optional implementation method, the connection parameters can be determined and the target data source can be connected by a data source connection template, wherein the data source type, authentication connection method and connection parameters are defined in the data source connection template, and the database can be quickly connected through these parameters.
[0078] Specifically, users only need to follow these configuration items and fill in the corresponding configuration according to the characteristics of the data source to quickly develop a new data source connection SDK, thereby improving the management efficiency of the data source connection.
[0079] Step S40: Acquire the collection parameters in the collection class, and collect target metadata in the target data source through the collection parameters.
[0080] In this embodiment, the collection parameters in the collection class enable the system to collect metadata of the target data source, including which tables are included, the structure information of the tables, and the partition table, the field information corresponding to the table, etc., to describe the data in the target data source. In the above steps, the target data source has been connected through the connection parameters. At this time, the target metadata is obtained by using the metadata collection method provided by the database JDBC through the table name, field name, data type, etc. that need to be collected in the collection parameters.
[0081] In addition, to facilitate the statistics of target metadata, after executing the query statement, the metadata result set obtained can be parsed into a readable data structure, such as a dictionary or object. The target metadata collected from heterogeneous data sources at different stages are uniformly saved in the Json format to the business database corresponding to the system, such as the PostgreSQL database, so that the metadata corresponding to the collected data source can be quickly viewed on the system interface.
[0082] It should be understood that the steps of obtaining the metadata in the target data source through the metadata collection SDK are mainly divided into three steps to complete the metadata collection task: collecting database information, collecting table information corresponding to the database, and obtaining relevant information of the fields corresponding to the table, etc. The triggering timing can be used through MQ (Message Queue) to process different collection tasks for notification, thereby reducing the task operation blocking caused by obtaining too much metadata.
[0083] In the technical solution provided in this embodiment, by creating a collection task scheduling execution, and automatically executing the collection task when the triggering conditions of the collection task are met, the collection needs of users in various scenarios can be met. By passing the data source type field value as a parameter to the data source connection SDK and the metadata collection SDK, and determining the connection class and collection class matching the parameter in the data source connection SDK and the metadata collection SDK, the target data source is connected according to the determined connection class and collection class, and the target metadata is collected from the target data source, which can improve the efficiency of metadata collection.
[0084] Embodiment 2
[0085] Please refer to Figure 2In a second embodiment, the method for configuring the data source to connect to the SDK comprises the following steps:
[0086] Step S50: Obtain a preset data source connection method and determine a data source type enumeration value.
[0087] In this embodiment, each of the data source connection functions corresponds to a method of connecting to the target data source, which may be a function defined in advance and saved in the target SDK. The data source type enumeration value may be a field value of the data source type as the data source type enumeration value, or an arbitrary numerical value as the data source type enumeration value. It should be understood that if an arbitrary numerical value is used as the data source type enumeration value, the data source type enumeration value needs to be associated with and recorded with the corresponding data source type.
[0088] In addition, the number of the data source type enumeration values is consistent with the number of the data source connection functions, that is, the number of the generated data source type enumeration values and the number of the data source connection functions must be consistent.
[0089] In this embodiment, the acquired data source connection function or data source connection method can also be encapsulated so that corresponding connection class methods can be generated according to different data source types. Targeted personalized configuration parameters are performed for different data sources. For example, in addition to the common account password method, Hive and Mongodb database connection methods also have other methods such as no password and Kerberos authentication. At the same time, different connection parameter configurations can be set for different data sources, such as connection timeout settings. The connection configuration isolation of different data sources is achieved through the JDBC data source connection SDK, which is convenient for dynamic expansion of other data source connections.
[0090] As an optional implementation method, a processing class of a data source connection processor may be created first, and the data source connection processing class is a unified entry for realizing different database connections, so as to ensure that there is only one instance of the database connection processing class in the system and the processing class is easy to access from the outside, facilitate the control of the number of instances created by the system and save system resources, ensure that one data source corresponds to one data source connection processing, and thus avoid security issues caused by thread concurrency.
[0091] Step S60: using the data source type enumeration value as a prefix of a class name, determining the class name of each data source connection method, and generating a corresponding connection class.
[0092] Step S70: Integrate the connection classes and generate the data source connection SDK, so that the management end calls the connection class in the data source connection SDK according to the class name.
[0093] In this embodiment, after defining each data source type enumeration value, it is also necessary to define an interface to represent the data source connection method, create a class that implements this interface for each data source, then create a factory class to dynamically create the corresponding object according to the data source type enumeration value, and then write the data source connection method into the corresponding class to generate the connection class. Finally, the generated connection classes are integrated into the big class, and the data source connection SDK is generated. The data source connection SDK contains the generated connection class, and the corresponding connection class can be called by the class name.
[0094] In addition, based on the implementation of the data source connection SDK, users can complete the development of a data source connection SDK by adding a configuration creation connection template of a data source type. The data source type, authentication connection method and connection parameters are defined in the data source connection template. According to these configuration items, fill in the corresponding configuration according to the characteristics of the data source, and a new data source connection SDK can be quickly developed.
[0095] For example, Figure 3 As shown, Figure 3 The data source connection SDK flow chart involved in the embodiment of the present application first creates a connection class for the data source connection processor, wherein the connection class is created based on the singleton pattern in Java, which is used to ensure that a class has only one instance and provide a global access point to access the instance. Then the corresponding data source connection method is obtained through the class reflection mechanism, wherein obtaining the data source connection method is to obtain the data source connection implementation class method through the class reflection mechanism in the singleton pattern, and then the data source connection processing method is further obtained using the strategy method. The purpose of using the strategy pattern is to select different algorithm methods according to different conditions at runtime. The strategy pattern will encapsulate the behavior of the algorithm in independent strategy classes, and the client can use these strategy classes through the strategy interface, that is, call the corresponding data source connection method through the strategy interface. Then create a processing class corresponding to the data source connection method, and set the authentication method of the data source corresponding to each connection class, and finally place each connection class into the processing class to generate the data source connection SDK.
[0096] In the technical solution provided in this embodiment, the data source connection method is input as a parameter according to the data source type enumeration value to generate the connection class, and then a data source connection SDK is generated according to each connection class. When it is necessary to connect to the data source, the connection parameters can be quickly determined through the data source type information, thereby improving the connection efficiency of the data source.
[0097] Embodiment 3
[0098] Please refer to Figure 4In a third embodiment, the method for configuring the metadata collection SDK comprises the following steps:
[0099] Step S80: Create the metadata collection SDK, determine the preset metadata collection method, and obtain data source connection information and data source type enumeration value.
[0100] Step S90: using the data source type enumeration value as the class name prefix of the metadata collection method, determining the class name of each metadata collection method, and generating a corresponding collection class.
[0101] Step S100: Integrate the various acquisition classes and place them into the metadata acquisition SDK.
[0102] In this embodiment, each metadata collection function corresponds to a method for collecting target metadata, which can be a pre-created function or method. The data source type enumeration value can be defined as the data source type enumeration value with any numerical value. It can be understood that when the data source type enumeration value is defined as an arbitrary numerical value, it is necessary to associate the data source type enumeration value with the corresponding metadata collection method and record it. In addition, the data source type enumeration value can also be defined as the class name prefix of the metadata collection method to facilitate calling the corresponding metadata collection function. After each of the data source type enumeration values is associated with the metadata collection function, the data source type enumeration value is used as the class name of each of the metadata collection functions, and a corresponding collection class is generated. Finally, each of the collection classes is integrated and placed in the metadata collection SDK.
[0103] In addition, it is understandable that different data sources contain different metadata information. In addition to some metadata information, such as libraries, tables, fields, and other common metadata information unified acquisition methods, it is also possible to obtain additional metadata information that other databases do not have, such as obtaining the file format of hive, partition table type, etc., to meet the needs of personalized metadata information for heterogeneous data sources.
[0104] Specifically, metadata collection requires data source connection information, that is, database credentials. When collecting metadata based on the metadata collection SDK, it is necessary to obtain the corresponding data source connection information to have the ability to collect corresponding databases.
[0105] For example, Figure 5 As shown, Figure 5The implementation flow chart of the metadata collection SDK involved in the embodiment of the present application is as follows: first, a registration class of the metadata collection processor is created. This registration class is a unified entry for obtaining metadata from heterogeneous data sources. The registration class adopts a singleton pattern design method, which only allows the creation of a metadata collection processing object, so as to achieve the purpose of not starting multiple microservices, so as to ensure the global uniqueness of the method for obtaining database metadata collection, and ensure the consistency of the obtained metadata collection results. Then, the corresponding metadata collection method is obtained through the class reflection mechanism, and the strategy pattern is used to design a method that can select different algorithms according to different conditions at runtime. The strategy pattern will encapsulate the behavior of the algorithm in independent strategy classes, and the client can use these strategy classes through the strategy interface, that is, call the corresponding metadata collection method through the strategy interface. By adding enumeration values of different data source types, according to the data source type, the class reflection mechanism is used to dynamically obtain the corresponding metadata collection class method. The metadata collection class method adopts the strategy pattern design method, defines a series of collection methods, and encapsulates the heterogeneous data source metadata collection algorithm into a series of strategy classes with a public interface. The metadata collection algorithms of data source types with the same properties are combined into the same processing class, such as relational processing class, non-relational processing class, HDFS processing class, and other database processing classes. Different data source types can be replaced with each other, making the algorithm changes independent of the users who use it. Users only need to select the appropriate collection method according to different metadata collection scenarios to improve the flexibility and scalability of metadata collection.
[0106] In the technical solution provided in this embodiment, by creating the metadata collection SDK, determining the preset metadata collection method, and obtaining data source connection information and data source type enumeration value; using the data source type enumeration value as the class name prefix of the metadata collection method, and determining the class name of each metadata collection method, and generating a corresponding collection class; integrating each of the collection classes and placing them in the metadata collection SDK, when metadata needs to be collected, the collection parameters can be quickly determined through the data source type information, thereby improving the connection efficiency of the data source.
[0107] Embodiment 4
[0108] Please refer to Figure 6 In the fourth embodiment, the metadata collection method further includes the following steps:
[0109] Step S110: Determine the data source connection SDK and the target class in the metadata collection SDK, wherein the target class is the collection class or the connection class.
[0110] Step S120: When the target class is called, the class name of the target class and the corresponding class method are obtained.
[0111] Step S130: The class name is used as a unique attribute, and the class method is used as a value, and is saved in a cache, so that when the class is subsequently called, the corresponding class method is directly called in the cache according to the class name.
[0112] In this embodiment, by determining whether the connection class and the collection class are called in the data source connection SDK and the metadata collection SDK, when a collection class or connection class is called, the class name and class function corresponding to the target class are saved in the cache.
[0113] As an optional implementation method, a caching mechanism of the Map method is adopted. The Map method will be used for the processing class of the data source type method called for the first time, and it will be added to the Key of the Map method. The processing class of the corresponding data source type will be added to the Value of the Map method. After that, the connection processing class of the data source type will be directly returned to the user, thereby improving the efficiency of connecting to the database and the performance of the system.
[0114] Embodiment 5
[0115] Please refer to Figure 7 In the fifth embodiment, after step S40, the following steps are further included:
[0116] Step S140: monitoring the collection task, and after the collection task is completed, recording the batch number corresponding to the collection task.
[0117] Step S150: When the metadata changes, the version number of the target data source corresponding to the acquisition task is recorded.
[0118] Step S160: Generate and save a version record log according to the batch number and the version number.
[0119] In this embodiment, the monitoring and recording actions are performed by the metadata version management module, which can be an application or a background service system connected to a corresponding client. By monitoring the collection task and recording the batch number of the metadata collected by the collection task, the batch number is the sequence number of the metadata collected. When the metadata of the target data source changes, the version number of the target data source corresponding to the collection task is recorded, that is, when the target data source is updated, its version number will also be updated. Finally, a version record log is generated according to the batch number, version number and the data source name corresponding to the target data source.
[0120] Specifically, the metadata version management module is used to effectively control the changes of the collected metadata information. For the implementation of metadata version management, each time the metadata collection plan is executed to generate a collection instance, the batch number will be recorded. The batch number is generated by incrementing by one according to the number of collections. The version number is only recorded when there are records such as change tables in the collection task. The management and maintenance of the version number allows users to trace back to the historical collection version based on the saved version number, and also allows users to compare with the last metadata collection, and then find out which tables have been added or deleted in the database.
[0121] Furthermore, in this embodiment, after step S160, the following steps are further included:
[0122] The record log is obtained, and the version number is determined according to the record log; when the metadata change does not conform to the preset plan, the corresponding target metadata is determined according to the version number; according to the target metadata, the changed data in the target data source is rolled back and saved.
[0123] Specifically, the preset plan includes metadata modification rules and modification plans. When the metadata change does not comply with the modification rules or the modification plan, the modified metadata to be modified is first determined according to the modification action, and the modified metadata is fed back to the client to allow the client to confirm whether to modify. When the client confirms that it is an illegal change, the system automatically determines the target metadata according to the version number, and replaces the changed data in the target data source according to the target metadata and saves it.
[0124] In addition, when configuring metadata collection tasks, users can execute metadata collection plans by configuring manual immediate execution or starting periodic scheduling, such as using scheduled task components such as xxl-job to trigger database collection scheduling tasks.
[0125] In the technical solution provided in this embodiment, by recording the batch number corresponding to the collection task and the version number of the target data source when the metadata changes, and generating a record log, the metadata can be monitored and managed in real time, thereby improving the management efficiency of the metadata.
[0126] Embodiment 6
[0127] Please refer to Figure 8 , in the sixth embodiment, Figure 8 This is a flow chart of the sixth embodiment of the metadata collection method of the present application. Figure 8As shown, users can create and input data source connection information and configure metadata collection tasks through the configuration interface. The interface will feed back the data source connection information and metadata collection tasks created and configured by the user to the background system. The background system creates a metadata collection task instance according to the metadata collection task, obtains the corresponding connection class and collection class in the SDK, and then connects to the target data source through the connection class and collection class, collects metadata and feeds it back to the background system. The background system saves the collected metadata in the business database and feeds it back to the interface so that the user can view the metadata information.
[0128] Since the system introduced in the embodiment of the present application is a system used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the embodiment of the present application belong to the scope of protection of this application.
[0129] As an implementation scheme, Fig. 9 This is a schematic diagram of the architecture of the hardware operating environment of the metadata collection device involved in the embodiment of the present application.
[0130] like Fig. 9 As shown, the metadata collection device may include: a processor 1001, such as a core processor (Central Processing Unit, CPU), a communication bus 1002, a network interface 1003, and a memory 1004. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The network interface 1003 may optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1004 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1004 may also be a storage device independent of the aforementioned processor 1001.
[0131] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the metadata collection device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0132] like Fig. 9As shown, the memory 1004 as a storage medium may include an operating system, a data storage module, a network communication module, and a metadata collection program. The operating system is a program that manages and controls the hardware and software resources of the metadata collection device, the metadata collection program, and the operation of other software or programs.
[0133] exist Fig. 9 In the metadata collection device shown, the network interface 1003 is mainly used to connect to a terminal and perform data communication with the terminal; the processor 1001 can be used to call the metadata collection program stored in the memory 1004.
[0134] In this embodiment, the metadata collection device includes: a memory 1004, a processor 1001, and a metadata collection program stored in the memory and executable on the processor, wherein:
[0135] When the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0136] When the triggering condition of the collection task is met, determining the data source type field value corresponding to the collection task;
[0137] Passing the data source type field value as a parameter to the data source connection SDK, and determining the connection class in the data source connection SDK;
[0138] Acquire the connection parameters in the connection class, connect to the target database according to the connection parameters, and determine the collection class in the metadata collection SDK according to the data source type field value;
[0139] The collection parameters in the collection class are acquired, and the target metadata in the target data source is collected according to the collection parameters.
[0140] In one embodiment, when the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0141] Determine configuration parameters of each data source, and build a data source management model according to the configuration parameters;
[0142] Creating the collection task according to a preset time plan, and saving the data source management model in association with the collection task;
[0143] When the preset time schedule arrives and / or when the start instruction is triggered, it is determined that the triggering condition of the acquisition task is met.
[0144] In one embodiment, when the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0145] Get the preset data source connection method and determine the data source type enumeration value;
[0146] Using the data source type enumeration value as a prefix of a class name, determining the class name of each data source connection method, and generating a corresponding connection class;
[0147] The connection classes are integrated to generate the data source connection SDK, so that the management end calls the connection class in the data source connection SDK according to the class name.
[0148] In one embodiment, when the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0149] Creating the metadata collection SDK, determining a preset metadata collection method, and obtaining data source connection information and data source type enumeration values;
[0150] Using the data source type enumeration value as the class name prefix of the metadata collection method, determining the class name of each metadata collection method, and generating a corresponding collection class;
[0151] Integrate the various acquisition classes and place them into the metadata acquisition SDK.
[0152] In one embodiment, when the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0153] Determine the data source connection SDK and the target class in the metadata collection SDK, wherein the target class is the collection class or the connection class;
[0154] When the target class is called, the class name of the target class and the corresponding class method are obtained;
[0155] The class name is used as a unique attribute, and the class method is used as a value, and is saved in a cache, so that when the class is subsequently called, the corresponding class method is directly called in the cache according to the class name.
[0156] In one embodiment, when the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0157] Monitor the collection task, and after the collection task is completed, record the batch number corresponding to the collection task;
[0158] When the metadata changes, the version number of the target data source corresponding to the acquisition task is recorded;
[0159] A version record log is generated and saved according to the batch number and the version number.
[0160] In one embodiment, when the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0161] Obtaining the record log, and determining the version number according to the record log;
[0162] When the metadata change does not conform to the preset plan, the corresponding target metadata is determined according to the version number;
[0163] According to the target metadata, the changed data in the target data source is rolled back and saved.
[0164] In one embodiment, when the processor 1001 calls the metadata collection program stored in the memory 1004, the following operations are performed:
[0165] Get the preset timeout threshold;
[0166] When the connection duration of the acquisition task is equal to the timeout threshold, and / or the acquisition duration is equal to the timeout threshold, determining the acquisition parameters corresponding to the acquisition task in the data source management model;
[0167] The acquisition parameters and corresponding prompt information are fed back to the client, so that the client displays the acquisition parameters and the prompt information.
[0168] In addition, it can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the metadata collection device to implement the process steps of the embodiment of the above method.
[0169] Therefore, the present application also provides a storage medium, which stores a metadata collection program. When the metadata collection program is executed by a processor, the various steps of the metadata collection method described in the above embodiment are implemented.
[0170] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0171] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods described in each embodiment of the present invention.
[0173] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A metadata collection method, It is characterized in that The metadata collection method comprises the following steps: Determine configuration parameters of each data source, and build a data source management model according to the configuration parameters; Methods for configuring data source connection SDK include: Get the preset data source connection method and determine the data source type enumeration value; Using the data source type enumeration value as a prefix of a class name, determining the class name of each data source connection method, and generating a corresponding connection class; Integrate each of the connection classes and generate a data source connection SDK, so that the management end calls the connection class in the data source connection SDK according to the class name; Methods for configuring the metadata collection SDK include: Create a metadata collection SDK, determine the preset metadata collection method, and obtain data source connection information and data source type enumeration values; Using the data source type enumeration value as the class name prefix of the metadata collection method, determining the class name of each metadata collection method, and generating a corresponding collection class; Integrate each of the collection classes and place them into the metadata collection SDK; Determine the data source connection SDK and the target class in the metadata collection SDK, wherein the target class is the collection class or the connection class; When the target class is called, the class name of the target class and the corresponding class method are obtained; The class name is used as a unique attribute, and the class method is used as a value, and is saved in a cache, so that when the class is subsequently called, the corresponding class method is directly called in the cache according to the class name; Create a collection task according to a preset time plan, and associate and save the data source management model in the collection task; When the preset time plan arrives and / or when the start instruction is triggered, determining whether the triggering condition of the acquisition task is met; When the triggering condition of the collection task is met, determining the data source type field value corresponding to the collection task; Passing the data source type field value as a parameter to the data source connection SDK, and determining the connection class in the data source connection SDK; Acquire connection parameters in the connection class, connect to the target database according to the connection parameters, and determine the collection class in the metadata collection SDK according to the data source type field value; The collection parameters in the collection class are acquired, and the target metadata in the target data source is collected according to the collection parameters.
2. The metadata collection method according to claim 1, It is characterized in that After the step of obtaining the collection parameters in the collection class and collecting the target metadata in the target data source according to the collection parameters, the method further includes: Monitor the collection task, and after the collection task is completed, record the batch number corresponding to the collection task; When the metadata changes, the version number of the target data source corresponding to the acquisition task is recorded; A version record log is generated and saved according to the batch number and the version number.
3. The metadata collection method according to claim 2, It is characterized in that After the step of generating and saving a record log according to the batch number and the version number, the method further includes: Obtaining the record log, and determining the version number according to the record log; When the metadata change does not conform to the preset plan, the corresponding target metadata is determined according to the version number; According to the target metadata, the changed data in the target data source is rolled back and saved.
4. The metadata collection method according to claim 1, It is characterized in that After the step of determining the data source type corresponding to the acquisition task when the triggering condition of the acquisition task is met, the method further includes: Get the preset timeout threshold; When the connection duration of the acquisition task is equal to the timeout threshold, and / or the acquisition duration is equal to the timeout threshold, determining the acquisition parameters corresponding to the acquisition task in the data source management model; The acquisition parameters and corresponding prompt information are fed back to the client, so that the client displays the acquisition parameters and the prompt information.
5. A metadata collection device, It is characterized in that The metadata collection device comprises: a memory, a processor, and a metadata collection program stored in the memory and executable on the processor, wherein the metadata collection program is configured to implement the steps of the metadata collection method according to any one of claims 1 to 4.
6. A storage medium, It is characterized in that The storage medium stores a metadata collection program, and when the metadata collection program is executed by the processor, the steps of the metadata collection method according to any one of claims 1 to 4 are implemented.
