Metadata acquisition method and apparatus, electronic device, and computer-readable storage medium

Abstract

The application provides a metadata collection method and device, electronic equipment and computer readable storage medium, and relates to the technical field of big data. The method comprises the following steps: after receiving a metadata collection task issued by a task scheduling platform, performing data source type self-adaptation based on the data source type, and performing resource self-adaptation based on current device environment resource information; after the self-adaptation is completed, performing metadata collection, and pushing the collected metadata to a cloud message queue (MQ) for a preset consumption pipeline to consume the metadata through the cloud MQ and store the metadata in a preset metadata database. In this scheme, the data source type is self-adapted when the connection between the collector and the target data source is established, which can ensure the adaptability to different resource types in the public cloud environment. On the other hand, the current resource is self-adapted before data collection, which can adapt the resource to the amount of collected data to ensure the resource adaptability and the stability of metadata collection.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a metadata collection method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Metadata acquisition is the core of metadata application. Metadata, simply put, is data that describes data. It can be categorized by type, including technical metadata, business metadata, security metadata, and audit metadata. Different types of metadata have different uses. For example, technical metadata is used to provide the data needed by computing engines and storage engines to maintain system operation, and also in applications such as data governance and data asset management. Specifically, technical metadata for relational database data sources mainly includes information such as DB (database), TABLE (table), COLUMN (column), VIEW (view), and INDEX (index). In the era of big data, metadata is even more critical.

[0003] For big data platforms located within an internal network environment (such as enterprise big data platforms), which are maintained and provided with big data-related services by one or more teams, the closed-loop metadata within the enterprise environment simplifies metadata collection. In the cloud era, big data is no longer exclusive to any single enterprise; various cloud vendors provide unified big data platforms and services (public clouds), segmented according to different tenants. Public clouds are characterized by geographical isolation, VPC (Virtual Private Cloud) isolation, and resource isolation. In short, different enterprise users have different big data clusters on the cloud platform. Based on these characteristics, metadata collection in a public cloud environment needs to address issues such as data source adaptability and resource adaptability. However, existing metadata collection solutions for internal network environments do not consider these issues, thus failing to meet the metadata collection needs of public cloud environments. Therefore, it is necessary to provide a new metadata collection solution. Summary of the Invention

[0004] The purpose of this application is to at least solve one of the aforementioned technical defects. The technical solution provided by the embodiments of this application is as follows:

[0005] In a first aspect, embodiments of this application provide a metadata collection method, including:

[0006] Receive metadata collection tasks issued by the task scheduling platform;

[0007] The target data source and its type are determined based on the metadata collection task, and a connection is established between the collector and the target data source based on the data source type.

[0008] The system acquires metadata information and current device environment resource information from the target data source, and acquires at least one acquisition subtree corresponding to the target data source and adapted to the device environment resource information based on the data information. The data information is used to indicate the storage structure of the metadata in the target data source.

[0009] Metadata collection tasks are performed based on at least one collection subtree to obtain the metadata corresponding to each collection subtree. The metadata corresponding to each collection subtree is then pushed to the cloud message queue (MQ) for the preset consumption pipeline to consume the metadata through the cloud MQ and store it in the preset metadata database.

[0010] In one optional embodiment of this application, obtaining at least one acquisition subtree corresponding to the target data source and adapted to the device environmental resource information based on data information includes:

[0011] Based on the data information, obtain the parent node, child node and connection relationship between the target data source, and based on the parent node, child node and connection relationship between the nodes, obtain the corresponding acquisition tree and the cost-based optimization method (CBO) cost corresponding to the acquisition tree, and obtain the corresponding CBO cost threshold based on the equipment environment resource information.

[0012] If the cost-benefit (CBO) cost of the collection tree is not greater than the CBO cost threshold, then the collection tree is used as at least one collection subtree. If the CBO cost of the collection tree is greater than the CBO cost threshold, then the collection tree is split to obtain at least one collection subtree.

[0013] In one optional embodiment of this application, obtaining the parent node, child node, and connection relationships between nodes corresponding to the target data source based on data information includes:

[0014] The target data source is used as the parent node, and the database, file type or topic type in the target data source is used as the child node. The corresponding connection relationship is obtained based on the subordinate relationship between the nodes.

[0015] In one optional embodiment of this application, the acquisition tree is split to obtain at least one acquisition subtree, including:

[0016] The data collection tree is split from top to bottom at the splitting child nodes. The splitting child nodes at each split are determined from the child nodes of the corresponding level of the data collection tree or each subtree using the binary k-nearest neighbor query algorithm.

[0017] After each split, the subtrees in each split where the CBO cost is not greater than the CBO cost threshold are identified as the collection subtrees. The splitting of subtrees where the CBO cost is greater than the CBO cost threshold continues until the CBO cost of each split subtree is not greater than the CBO cost threshold, thus obtaining at least one collection subtree.

[0018] In one optional embodiment of this application, establishing a connection between the collector and the target data source based on the data source type includes:

[0019] Determine the collector parameters based on the data source type;

[0020] Establish a connection between the collector and the target data source based on the collector parameters.

[0021] In one optional embodiment of this application, performing a metadata collection task based on at least one collection subtree includes:

[0022] Collect metadata for portions of at least one collection subtree; or

[0023] The metadata corresponding to each collection subtree is collected according to the splitting order of each collection subtree.

[0024] In one optional embodiment of this application, the method further includes:

[0025] The metadata collection configuration information is obtained through the metadata management system, and the collector authorization verification is initiated to the cloud MQ. The metadata collection configuration information includes: data source information, scheduling information and encryption information.

[0026] If the collector authorization verification is successful, it will receive the authorization token sent by the cloud MQ through the metadata management system, and create a metadata collection task carrying data source information, scheduling information, authorization token and encryption information through the task scheduling platform.

[0027] In one optional embodiment of this application, receiving a metadata collection task issued by a task scheduling platform includes:

[0028] The task scheduling platform obtains the trigger time of the metadata collection task based on the scheduling information, and receives the metadata collection task issued by the task scheduling platform at the trigger time.

[0029] In one optional embodiment of this application, the metadata corresponding to each collection subtree is pushed to a cloud message queue (MQ), including:

[0030] Obtain encrypted information and authorization tokens based on metadata collection tasks;

[0031] Based on the encrypted information, the metadata corresponding to each collection subtree is encrypted to obtain encrypted metadata. Then, the encrypted metadata is transcoded according to a preset method to obtain transcoded metadata. Finally, the transcoded metadata is compressed to obtain encrypted and compressed metadata.

[0032] After completing the authorization verification with the cloud MQ based on the authorization token, the encrypted and compressed metadata is pushed to the cloud MQ.

[0033] Secondly, embodiments of this application provide a metadata collection device, including:

[0034] The metadata collection task receiving module is used to receive metadata collection tasks issued by the task scheduling platform.

[0035] The data source type adaptive module is used to determine the target data source to be collected and the data source type of the target data source based on the metadata collection task, and to establish a connection between the collector and the target data source based on the data source type.

[0036] The resource adaptation module is used to obtain the data information of the metadata in the target data source and the current device environment resource information, and to obtain at least one collection subtree corresponding to the target data source and adapted to the device environment resource information based on the data information. The data information is used to indicate the storage structure of the metadata in the target data source.

[0037] The metadata push module is used to perform metadata collection tasks based on at least one collection subtree to obtain the metadata corresponding to each collection subtree, and push the metadata corresponding to each collection subtree to the cloud message queue MQ, so that the preset consumption pipeline can consume the metadata through the cloud MQ and store it in the preset metadata database.

[0038] In one optional embodiment of this application, the resource adaptive module is specifically used for:

[0039] Based on the data information, obtain the parent node, child node and connection relationship between the target data source, and based on the parent node, child node and connection relationship between the nodes, obtain the corresponding acquisition tree and the cost-based optimization method (CBO) cost corresponding to the acquisition tree, and obtain the corresponding CBO cost threshold based on the equipment environment resource information.

[0040] If the cost-benefit (CBO) cost of the collection tree is not greater than the CBO cost threshold, then the collection tree is used as at least one collection subtree. If the CBO cost of the collection tree is greater than the CBO cost threshold, then the collection tree is split to obtain at least one collection subtree.

[0041] In an optional embodiment of this application, the resource adaptation module is further configured to:

[0042] The target data source is used as the parent node, and the database, file type or topic type in the target data source is used as the child node. The corresponding connection relationship is obtained based on the subordinate relationship between the nodes.

[0043] In an optional embodiment of this application, the resource adaptation module is further configured to:

[0044] The data collection tree is split from top to bottom at the splitting child nodes. The splitting child nodes at each split are determined from the child nodes of the corresponding level of the data collection tree or each subtree using the binary k-nearest neighbor query algorithm.

[0045] After each split, the subtrees in each split where the CBO cost is not greater than the CBO cost threshold are identified as the collection subtrees. The splitting of subtrees where the CBO cost is greater than the CBO cost threshold continues until the CBO cost of each split subtree is not greater than the CBO cost threshold, thus obtaining at least one collection subtree.

[0046] In one alternative embodiment of this application, the data source type adaptive module is specifically used for:

[0047] Determine the collector parameters based on the data source type;

[0048] Establish a connection between the collector and the target data source based on the collector parameters.

[0049] In one optional embodiment of this application, the metadata collection task receiving module is specifically used for:

[0050] Collect metadata for portions of at least one collection subtree; or

[0051] The metadata corresponding to each collection subtree is collected according to the splitting order of each collection subtree.

[0052] In an optional embodiment of this application, the device further includes a metadata collection task creation module, used for:

[0053] The metadata collection configuration information is obtained through the metadata management system, and the collector authorization verification is initiated to the cloud MQ. The metadata collection configuration information includes: data source information, scheduling information and encryption information.

[0054] If the collector authorization verification is successful, it will receive the authorization token sent by the cloud MQ through the metadata management system, and create a metadata collection task carrying data source information, scheduling information, authorization token and encryption information through the task scheduling platform.

[0055] In one optional embodiment of this application, the device further includes a metadata collection task creation module specifically used for:

[0056] The task scheduling platform obtains the trigger time of the metadata collection task based on the scheduling information, and receives the metadata collection task issued by the task scheduling platform at the trigger time.

[0057] In one optional embodiment of this application, the metadata push module is specifically used for:

[0058] Obtain encrypted information and authorization tokens based on metadata collection tasks;

[0059] Based on the encrypted information, the metadata corresponding to each collection subtree is encrypted to obtain encrypted metadata. Then, the encrypted metadata is transcoded according to a preset method to obtain transcoded metadata. Finally, the transcoded metadata is compressed to obtain encrypted and compressed metadata.

[0060] After completing the authorization verification with the cloud MQ based on the authorization token, the encrypted and compressed metadata is pushed to the cloud MQ.

[0061] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor;

[0062] The memory contains computer programs;

[0063] A processor for executing computer programs to implement the methods provided in the first aspect embodiment or any alternative embodiment of the first aspect.

[0064] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the first aspect embodiment or any optional embodiment of the first aspect.

[0065] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the first aspect embodiment or any optional embodiment of the first aspect.

[0066] The beneficial effects of the technical solution provided in this application are:

[0067] After receiving the metadata collection task from the task scheduling platform, the system determines the data source type of the target data source, adapts the collector to this data source type, and establishes a connection with the target data source. Then, based on the metadata information in the target data source, it divides the target data source into one or more collection subtrees with data volumes adapted to the current device environment resources. Finally, it collects metadata corresponding to the metadata collection task according to each collection subtree and pushes the metadata to the cloud MQ for consumption and storage. This solution, on the one hand, adapts to the data source type when establishing the connection between the collector and the target data source, ensuring adaptability to different resource types in a public cloud environment. On the other hand, it adapts to the current resources before data collection, ensuring resource adaptability by matching the resources with the amount of data to be collected, thereby guaranteeing the stability of metadata collection. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0069] Figure 1 A schematic diagram of the metadata collection system structure on which the metadata collection method provided in this application depends;

[0070] Figure 2 A flowchart illustrating a metadata collection method provided in an embodiment of this application;

[0071] Figure 3 This is a schematic diagram of the user interface for configuring a data source type in one example of an embodiment of this application;

[0072] Figure 4 A schematic diagram of the interface for configuring a target data source for a user in one example of an embodiment of this application;

[0073] Figure 5 This is a schematic diagram of the user scheduling information interface in one example of an embodiment of this application;

[0074] Figure 6 This is a schematic diagram of a data collection tree in one example of an embodiment of this application;

[0075] Figure 7 This is a schematic diagram illustrating the process of determining the acquisition subtree in an embodiment of this application;

[0076] Figure 8 This is a schematic diagram illustrating the top-down splitting of the data collection tree in one example of an embodiment of this application;

[0077] Figure 9 This is a schematic diagram illustrating the process of compressing collected metadata in one example of an embodiment of this application;

[0078] Figure 10 This is a schematic diagram of the architecture of a public cloud network environment and tenants in one example of an embodiment of this application;

[0079] Figure 11 This is a schematic diagram of a VPC tenant deployment method in one example of an embodiment of this application;

[0080] Figure 12 This is a schematic diagram illustrating a specific process of a metadata collection method provided in an embodiment of this application;

[0081] Figure 13 This is a schematic diagram of the overall process architecture of a metadata collection method in one example of an embodiment of this application;

[0082] Figure 14 A structural block diagram of a metadata acquisition device provided in an embodiment of this application;

[0083] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0084] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0085] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0087] Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0088] In this context, corresponding to the intranet environment discussed later, a private cloud is created within a firewall, housing cloud infrastructure and hardware / software resources for various departments within an organization or enterprise to share resources within a data center. Besides hardware resources, creating a private cloud typically involves cloud equipment (IaaS, Infrastructure as a Service) software.

[0089] Private cloud computing also comprises three layers: cloud hardware, cloud platform, and cloud services. The difference is that cloud hardware consists of the user's own personal computer or server, rather than the cloud computing provider's data center. Cloud computing providers build data centers to provide public cloud services to millions of users, thus requiring hundreds of thousands or even millions of servers. Private cloud computing, for individuals, serves only family and friends; for businesses, it serves only their own employees, customers, and suppliers. Therefore, an individual's or business's own personal computer or server is sufficient to provide cloud services.

[0090] In this context, corresponding to the public cloud environment discussed later, a public cloud typically refers to a cloud provided by a third-party provider to users. Public clouds are generally accessible via the Internet and may be free or inexpensive. The core attribute of a public cloud is shared resource services. Many instances of this type of cloud exist, providing services across today's open public networks.

[0091] Metadata collection for both public and private clouds involves the field of big data. Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0092] In an internal network environment, there are generally no issues such as resource limitations, low network transmission efficiency, or data security. Metadata collection typically employs three collection modes, as shown in Table 1:

[0093] Table 1

[0094]

[0095]

[0096] In an internal network environment (i.e., intranet environment), the traditional method of collecting metadata for big data platforms is as follows: configure corresponding SDKs (JDBC SDK, FILE SDK, API SDK, etc.) for different data sources (JDBC (Java Database Connectivity), FILE, API, etc.), then use these SDKs to detect the metadata in the corresponding data sources, and then actively report the metadata information to the metadata processing platform for unified processing and storage.

[0097] By comparing the data collection methods in Table 1 above with the data collection process of traditional big data platforms, it can be seen that traditional methods are not affected by factors such as location, network environment, and security. However, in a public cloud environment, these advantages are completely at odds with the cloud. The existing metadata collection solutions have the following problems:

[0098] (1) Resources: Intranet environments generally provide sufficient resources, but in public cloud environments, the types of user resources are unknown, which increases the failure rate of metadata collection.

[0099] (2) Transmission rate: In the intranet environment, the network card is generally 10 Gigabit or 1 Gigabit, and the network is not the bottleneck. However, in the public cloud environment, the network bandwidth is directly related to the amount of user purchases. Data compression and transmission are problems that must be solved.

[0100] (3) Security: The internal network environment is isolated from the public network, and the data generally does not come from threats from the public network. However, in the public cloud environment, data transmission is exposed to the public network, and data security becomes very important.

[0101] (4) Regarding the collection method: Since the big data environment in the intranet is maintained by a unified department, it is relatively easy to integrate the SDK. However, in the public cloud environment, the big data environment is isolated on the user side and different enterprise users are isolated. Continuing to use the SDK method becomes unfriendly and will also lead to users' data insecurity.

[0102] To address the aforementioned issues, this application provides a solution for cross-regional, high-efficiency, and secure metadata collection in a public cloud environment. The metadata collection method, apparatus, electronic device, and computer-readable storage medium provided in this application will be described in detail below.

[0103] Figure 1 This is a schematic diagram of the metadata collection system structure on which the metadata collection method provided in this application depends, as shown in the embodiment. Figure 1 As shown, the system mainly includes: a metadata management platform 101, a task scheduling platform 102, a collector 103, a data source 104, and a cloud-based MQ (Message Queue) 105. The metadata management platform 101 can be provided to users (enterprises or individual users, etc.) by a cloud service provider, or it can be provided by a third party other than the cloud service provider and the user. This metadata management platform 101 is located on a local server, and its front end includes an interactive interface for users to input and set metadata collection configuration information. The task scheduling platform 102 is located on both the local server and the cloud server. It can receive information sent by the metadata management platform 101 and schedule tasks for the corresponding collectors 103. Based on the metadata collection configuration information sent by the metadata management platform 101, it creates corresponding metadata collection tasks and schedules them (i.e., sends out metadata collection tasks and related task information). The tasks are then sent to the collectors 103 for execution. The collectors 103 execute the received metadata collection tasks, collecting the corresponding metadata from the corresponding data source 104 and pushing the collected metadata to the cloud-based MQ for consumption and storage. Both the collector 103 and the data source 104 are set up on the cloud server. The settings for the collector 103 and the data source 104 will be described in more detail later.

[0104] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not restricted herein.

[0105] Figure 2 This is a flowchart illustrating a metadata collection method provided in an embodiment of this application. The execution entity of this method can be... Figure 1 Collector 103 in the middle, such as Figure 2 As shown, the method may include:

[0106] Step S201: Receive the metadata collection task issued by the task scheduling platform.

[0107] The metadata collection task includes metadata collection configuration information, which serves as an instruction for the collector to perform metadata collection. Users can configure the metadata collection configuration information through the metadata management system according to their specific metadata collection needs, thereby ensuring that the subsequently collected metadata meets the requirements of metadata applications.

[0108] Specifically, when a task is triggered, the task scheduling platform schedules the metadata collection task to the corresponding collector. That is, the collector receives the metadata collection task issued by the task scheduling platform when the task is triggered.

[0109] Step S202: Determine the target data source and the data source type of the target data source based on the metadata collection task, and establish a connection between the collector and the target data source based on the data source type.

[0110] The metadata collection task carries configuration information including data source information, which includes the source of the metadata to be collected (i.e., the target data source) and the data source type of the target data source. For example, the target data source type could be JDBC, Hive (a data warehouse tool based on Hadoop), etc.

[0111] Specifically, the collector determines the target data source and its data source type based on the metadata collection configuration information carried by the received metadata collection task. After determining the data source type, the collector can configure its connection with the target data source based on that data source type. In other words, the collector will perform different connection configurations depending on the data source type to establish a connection with the target data source. For example, if the target data source type is JDBC, the collector will perform adaptive connection configuration, configuring itself to connect to JDBC data sources. After completing the adaptive connection configuration, the collector establishes a connection with the target data source for subsequent metadata retrieval.

[0112] It should be noted that the adaptive connection configuration in this step can be understood as adaptation to different data source types. Since different data source types correspond to different metadata formats, the adaptive process of the data source type of the collector may include not only the adaptive connection configuration mentioned above, but also the adaptive format processing when pulling metadata, that is, processing the different metadata formats pulled to obtain metadata of the specified format type.

[0113] Step S203: Obtain the metadata information and current device environment resource information from the target data source, and obtain at least one acquisition subtree corresponding to the target data source and adapted to the device environment resource information based on the data information. The data information is used to indicate the storage structure of the metadata in the target data source.

[0114] Metadata can be understood as data that describes data, and the metadata information obtained in this step can be understood as data that describes metadata. Specifically, this data information is used to indicate the storage structure of metadata in the target data source. For example, this data information may include the hierarchical relationship information between databases, tables, and columns in the target data source, and may further include information such as the number of data rows in the table. In other words, based on this data information, the hierarchical relationships at each level in the metadata of the target data source can be obtained, as well as information such as the amount of data corresponding to each level.

[0115] The equipment environmental resource information may include metrics such as the collector's CPU (Central Processing Unit), I / O (Input / Output), memory, and bandwidth.

[0116] Specifically, the collector acquires metadata information from the target data source, including its hierarchical structure and data volume. It also acquires current device environment resource information, such as CPU, I / O, memory, and bandwidth. To determine if the collector's current device environment resources meet the requirements for executing the received metadata collection task, a comparison is made between the current resource information and the resources required. If the collector's current device environment resources meet the requirements, collection is performed on the entire collection tree corresponding to the metadata in the target data source. If the collector's current device environment resources do not meet the requirements, the collection technique needs to be split into multiple collection subtrees, and collection is performed on each subtree.

[0117] Understandably, the amount of data corresponding to the acquisition tree and acquisition subtrees is different, and the amount of data corresponding to different acquisition subtrees is also different. Different amounts of data also require different device environment resources. Therefore, by splitting the metadata in the target data source into different acquisition subtrees according to the different device environment resources, and then collecting it, the amount of data collected each time is adjusted according to the different device environment resources, thus achieving resource adaptability.

[0118] Step S204: Perform a metadata collection task based on at least one collection subtree to obtain the metadata corresponding to each collection subtree, and push the metadata corresponding to each collection subtree to the cloud message queue MQ so that the preset consumption pipeline can consume the metadata through the cloud MQ and store it in the preset metadata database.

[0119] Specifically, after obtaining multiple collection subtrees through resource adaptation in the previous step, metadata is collected for each of the different collection subtrees, i.e., a metadata collection task is executed. Each collection operation retrieves the metadata corresponding to that subtree. All the metadata corresponding to these multiple collection subtrees constitutes all the metadata to be collected by the metadata collection task. The metadata corresponding to these collection subtrees is then pushed to a cloud-based message queue (MQ). A pre-defined consumption pipeline consumes this metadata and stores it in a pre-defined metadata database for subsequent use.

[0120] In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, capable of being shared by multiple users, with minimal redundancy, and independent of application programs.

[0121] A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters or mobile devices; or according to the query language used, such as SQL (Structured Query Language) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages.

[0122] The solution provided in this application, after receiving a metadata collection task from a task scheduling platform, determines the data source type of the target data source, adapts the collector to this data source type, and establishes a connection with the target data source. Then, based on the metadata information in the target data source, it divides the target data source into one or more data volume subtrees adapted to the current device environment resource collection. Finally, it collects metadata corresponding to the metadata collection task according to each collection subtree, and pushes the metadata to the cloud MQ for consumption and storage. This solution, on the one hand, adapts to the data source type when establishing the connection between the collector and the target data source, ensuring adaptability to different resource types in a public cloud environment; on the other hand, it adapts to the current resources before data collection, ensuring resource adaptability by matching the resources with the amount of data to be collected, thereby guaranteeing the stability of metadata collection.

[0123] In one optional embodiment of this application, the method may further include:

[0124] The metadata collection configuration information is obtained through the metadata management system, and the collector authorization verification is initiated to the cloud MQ. The metadata collection configuration information includes: data source information, scheduling information and encryption information.

[0125] If the collector authorization verification is successful, it will receive the authorization token sent by the cloud MQ through the metadata management system, and create a metadata collection task carrying data source information, scheduling information, authorization token and encryption information through the task scheduling platform.

[0126] As mentioned earlier, the user-configured collection configuration information can be obtained through the metadata management system. This metadata collection configuration information may include data source information, scheduling information, and encryption information. Specifically, users can configure the metadata collection configuration information through interactive methods such as device input and selection. The specific interaction method depends on the form of the front-end interactive interface of the metadata management system, and this application does not limit it.

[0127] Specifically, the data source information can include the target data source and its data type. A user may have multiple data source clusters on a public cloud; therefore, the target data source specifies which cluster to collect metadata from, and that cluster is the corresponding target data source. The data source type is the type corresponding to that cluster, including JDBC, Hive, etc. Scheduling information is used to configure the triggering information for the metadata collection task, which can include the trigger time or trigger period. Encryption information is used to encrypt the collected metadata before pushing it to the cloud message queue (MQ). Generally, encryption information is the default and does not require user configuration.

[0128] like Figure 3 As shown in the figure, in one example of this application, the configuration interface for the data source information of the metadata management system is as follows. On this interface, the data source type is first configured through the "Select Type" sub-interface. As shown, one can select the "Hive" data source corresponding to big data by clicking, or select the relational database "MySQL" by clicking. Further, as... Figure 4 As shown, after selecting "Hive" as the data source type, the user can further configure the target data source through the sub-interface shown in the figure, namely the "Data Source" selection interface. This interface pre-sets multiple clusters for the user (all of which are Hive data source types), and the user can select one of these clusters as the target data source. It is understandable that the user, based on their metadata collection needs, has clearly defined the target data source and its type, and only needs to configure it as shown in the example above. Furthermore, as... Figure 5 As shown in the figure, after configuring the data source information, the next step is to configure the scheduling information. The collection period can be configured as "every 10 minutes". The metadata collection task will be triggered once every 10 minutes. That is, every 10 minutes, the task scheduling platform will send the metadata collection task to the corresponding collector.

[0129] Furthermore, after obtaining metadata collection configuration information through the metadata management system, the metadata management system initiates a collector authorization verification with the cloud-based message queue (MQ). This verification verifies the collector's permissions for subsequent metadata collection. If the verification is successful, the cloud-based MQ will send an authorization token back to the metadata management system. The metadata management system will then send this authorization token, along with the aforementioned metadata collection configuration information, to the task scheduling platform. The task scheduling platform will then create a metadata collection task carrying the authorization token and the aforementioned metadata collection configuration information. If the verification fails, the corresponding metadata collection task will not be created; that is, task creation fails, and the user needs to reconfigure. This authorization verification process ensures the security of metadata collection.

[0130] In one optional embodiment of this application, receiving a metadata collection task issued by a task scheduling platform includes:

[0131] The task scheduling platform obtains the trigger time of the metadata collection task based on the scheduling information, and receives the metadata collection task issued by the task scheduling platform at the trigger time.

[0132] Specifically, the metadata collection configuration information includes scheduling information. After obtaining the metadata collection configuration information, the task scheduling platform establishes the corresponding metadata collection task. When to trigger (i.e., schedule) the metadata collection task depends on obtaining the trigger time of the metadata collection task based on the scheduling information, and then sending the metadata collection task to the collector at that trigger time.

[0133] In one optional embodiment of this application, obtaining at least one acquisition subtree corresponding to the target data source and adapted to the device environmental resource information based on data information includes:

[0134] Based on the data information, obtain the parent node, child node and connection relationship between the target data source, and based on the parent node, child node and connection relationship between the nodes, obtain the corresponding acquisition tree and the cost-based optimization method (CBO) cost corresponding to the acquisition tree, and obtain the corresponding CBO cost threshold based on the equipment environment resource information.

[0135] If the cost-benefit (CBO) cost of the collection tree is not greater than the CBO cost threshold, then the collection tree is used as at least one collection subtree. If the CBO cost of the collection tree is greater than the CBO cost threshold, then the collection tree is split to obtain at least one collection subtree.

[0136] The collection tree is a tree-like result composed of multiple nodes. Determining a collection tree mainly involves determining its parent node (or root node), multiple child nodes, and the connection relationships between the nodes.

[0137] Specifically, based on the metadata information in the target data source, multiple parent and child nodes are first identified, and then the connection relationships between these nodes are determined to obtain the corresponding data collection tree. To enable resource adaptation, it is determined whether the amount of data to be collected according to the current data collection tree matches the current device environment resources. If they match, the data collection tree is used as a subtree, meaning there is no need to split the tree; data collection proceeds directly according to the tree. If they do not match, the data collection tree needs to be split to obtain multiple subtrees before data collection proceeds according to each subtree.

[0138] Specifically, resource adaptation can be determined using CBO (Cost-Based Optimization). First, a CBO cost threshold is determined based on the current device environment resources. Then, the CBO cost corresponding to the acquisition tree is obtained and compared with the CBO cost threshold. If the CBO cost of the acquisition tree is not greater than the CBO cost threshold, meaning the amount of data to be acquired using the current acquisition tree matches the current device environment resources, then the acquisition tree is considered at least one acquisition subtree. If the CBO cost of the acquisition tree is greater than the CBO cost threshold, meaning the amount of data to be acquired using the current acquisition tree does not match the current device environment resources, then the acquisition tree is split to obtain at least one acquisition subtree.

[0139] In one optional embodiment of this application, obtaining the parent node, child node, and connection relationships between nodes corresponding to the target data source based on data information includes:

[0140] The target data source is used as the parent node, and the database, file type or topic type in the target data source is used as the child node. The corresponding connection relationship is obtained based on the subordinate relationship between the nodes.

[0141] For example, such as Figure 6As shown, the data source type of a target data source is JDBC, so the parent node of the collection tree corresponding to this target data source is denoted as "JDBC". This target data source contains three databases, denoted as "DB1", "DB2", and "DB3" respectively. Therefore, these three databases are treated as child nodes of the parent node JDBC, and each of these child nodes is connected to the parent node JDBC. Each of the three child nodes "DB1", "DB2", and "DB3" stores data in a table format, denoted as "table1" and "table2". Therefore, "table1" and "table2" can be treated as child nodes connected to "DB1", "DB2", and "DB3" respectively. Furthermore, the child nodes "table1" and "table2" of the child node "DB1" further include different topic types: "column", "partition", and "attribute". Therefore, these topic types can be treated as child nodes of the child nodes "table1" and "table2". Based on the parent node, child node, and the connection relationships between the nodes, the data collection tree shown in the figure can be determined.

[0142] In one optional embodiment of this application, the acquisition tree is split to obtain at least one acquisition subtree, including:

[0143] The data collection tree is split from top to bottom at the splitting child nodes. The splitting child nodes at each split are determined from the child nodes of the corresponding level of the data collection tree or each subtree using the binary k-nearest neighbor query algorithm.

[0144] After each split, the subtrees in each split where the CBO cost is not greater than the CBO cost threshold are identified as the collection subtrees. The splitting of subtrees where the CBO cost is greater than the CBO cost threshold continues until the CBO cost of each split subtree is not greater than the CBO cost threshold, thus obtaining at least one collection subtree.

[0145] Specifically, in the process of obtaining multiple acquisition subtrees through splitting, the most critical steps include determining the child nodes to be split and comparing the CBO cost of the resulting subtrees with the CBO cost threshold. The former determines from which child node to split, and the latter determines whether the resulting subtrees need to be further split. Top-down splitting refers to splitting from the parent node towards child nodes at different levels. If we define a child node as having fewer child nodes corresponding to its parent node, indicating a higher level, then top-down splitting means splitting from higher-level child nodes towards lower-level child nodes.

[0146] like Figure 7As shown, the process of obtaining split subtrees includes: First, calculating the cost-benefit (CBO) cost of the collection tree, and when the CBO cost is determined to be no greater than a CBO cost threshold, the collection tree is taken as the corresponding collection subtree. When the CBO cost is determined to be greater than the CBO cost threshold, a binary k-nearest neighbor (BNN) query algorithm is used to determine one or more split child nodes from the next level of child nodes adjacent to the parent node, and the collection tree is split into multiple subtrees at the split child nodes. The CBO cost of each subtree is calculated, and when the CBO cost is determined to be no greater than the CBO cost threshold, the corresponding subtree is determined as a collection subtree. When the CBO cost is determined to be greater than the CBO cost threshold, a binary k-nearest neighbor (BNN) query algorithm is used to determine one or more split child nodes from the next level of child nodes adjacent to the split level of child nodes, and the collection tree is split into multiple subtrees at the split child nodes. The above splitting and CBO cost comparison steps are repeated until all the obtained subtrees are called collection subtrees, or until the subtrees can no longer be split.

[0147] For example Figure 6 The collection tree shown is split into segments, such as... Figure 8 As shown, in the first split, the binary k-nearest neighbor (BNNL) algorithm determined a child node named "DB1". The resulting subtree is shown in the middle of the figure (other subtrees are not shown). Comparing the CBO cost of this subtree with the CBO cost threshold, it was determined that the CBO cost of this subtree is greater than the CBO cost threshold, therefore further splitting of this subtree is necessary. In the second split, the binary k-nearest neighbor (BNNL) algorithm determined a child node named "table1" (or "table2"). The two resulting subtrees are shown in the bottom of the figure. Since multiple subtrees are obtained, they can be figuratively called a "forest". Comparing the CBO costs of these two subtrees, they are not greater than the CBO cost threshold, therefore both subtrees are determined as harvesting subtrees and no further splitting is needed.

[0148] It should be noted that the splitting process is top-down. The splitting of child nodes uses the binary k-nearest neighbor (BNN) algorithm to obtain the split child nodes. The advantage of the BNN algorithm is that it can get infinitely close to the CBO cost threshold, which ensures the reliability of the split subtree to a certain extent under resource contention.

[0149] In one optional embodiment of this application, establishing a connection between the collector and the target data source based on the data source type includes:

[0150] Determine the collector parameters based on the data source type;

[0151] Establish a connection between the collector and the target data source based on the collector parameters.

[0152] Specifically, adaptive data source type processing involves two aspects: firstly, the adaptive connection configuration of the collector; and secondly, the adaptive format processing during metadata retrieval, which involves processing different metadata formats to obtain metadata of the specified format type. In detail, the adaptive connection configuration process can be understood as the collector determining its parameters based on the data source type, and then establishing a connection with the target data source based on these parameters, preparing for subsequent metadata collection.

[0153] In one optional embodiment of this application, performing a metadata collection task based on at least one collection subtree includes:

[0154] Collect metadata for portions of at least one collection subtree; or

[0155] The metadata corresponding to each collection subtree is collected according to the splitting order of each collection subtree.

[0156] Specifically, when performing a metadata collection task based on at least one collection subtree, it can be determined whether to collect metadata corresponding to all collection subtrees based on the collection content indicated by the metadata collection task. If it is determined that only metadata corresponding to some collection subtrees needs to be collected, then only the metadata corresponding to some collection subtrees is collected. If it is determined that metadata corresponding to all collection subtrees needs to be collected, then the metadata corresponding to each collection subtree can be collected separately according to the splitting order of each collection subtree. Through the above collection method, the metadata in the target data source can be filtered before the metadata is collected, improving the efficiency of subsequent metadata applications.

[0157] In one optional embodiment of this application, the metadata corresponding to each collection subtree is pushed to a cloud message queue (MQ), including:

[0158] Obtain encrypted information and authorization tokens based on metadata collection tasks;

[0159] Based on the encrypted information, the metadata corresponding to each collection subtree is encrypted to obtain encrypted metadata. Then, the encrypted metadata is transcoded according to a preset method to obtain transcoded metadata. Finally, the transcoded metadata is compressed to obtain encrypted and compressed metadata.

[0160] After completing the authorization verification with the cloud MQ based on the authorization token, the encrypted and compressed metadata is pushed to the cloud MQ.

[0161] Specifically, in this embodiment, in addition to authorizing and verifying the collector via cloud-based MQ before creating the metadata collection task, as mentioned above, to further enhance data security during transmission, the collected metadata can be encrypted after collection. Specifically, encrypted metadata is obtained by encrypting the collected metadata based on encrypted information. The encrypted information can indicate the specific encryption method; for example, it can indicate that asymmetric data encryption is used for the metadata. Therefore, the metadata collection scheme provided in this embodiment employs a two-layer security mechanism to ensure cloud security: authorization verification of the collector and encrypted transmission of the collected metadata.

[0162] Cloud security refers to the collective term for security software, hardware, users, organizations, and security cloud platforms based on cloud computing business models. It integrates emerging technologies and concepts such as parallel processing, grid computing, and the identification of unknown virus behavior. Through a large network of clients, it monitors anomalies in software behavior on the network, obtains the latest information on Trojans and malware on the internet, and sends this information to the server for automatic analysis and processing. Finally, it distributes solutions for viruses and Trojans to each client.

[0163] The main research directions in cloud security include: 1. Cloud computing security, which mainly studies how to ensure the security of the cloud itself and various applications on the cloud, including cloud computer system security, secure storage and isolation of user data, user access authentication, information transmission security, network attack protection, and compliance auditing; 2. Cloudification of security infrastructure, which mainly studies how to use cloud computing to build and integrate security infrastructure resources and optimize security protection mechanisms, including building a large-scale security event and information collection and processing platform through cloud computing technology to achieve the collection and correlation analysis of massive amounts of information and improve the ability to control network-wide security events and risks; 3. Cloud security services, which mainly studies various security services provided to users based on cloud computing platforms, such as antivirus services.

[0164] Additionally, the collected metadata can be compressed before transmission. Specifically, data collection and transmission are divided into intranet transmission and public network transmission. In a public cloud environment, cloud-based MQ products in different availability zones are isolated from each other. Intranet transmission is impossible, and data can only be transmitted via the public network. However, transmission bandwidth is limited by the user's purchased bandwidth capacity. How to achieve efficient metadata collection and transmission under limited bandwidth is a crucial issue that must be considered. The collector implementation process is as follows... Figure 9 As shown, after base64 encoding of the metadata, a compression algorithm is used to compress the data (the resulting compression format can be gzip). Then, multiple data entries from a subset of subtrees can be aggregated to generate batch metadata for batch sending. This aggregation can be achieved using the capabilities provided by cloud-based message queues (MQ).

[0165] In one optional embodiment of this application, since public cloud environments are generally used by enterprise users, traditional embedded SDKs or other code-based methods could pose risks to users. Therefore, an offline custom data collection method is used under this condition. The public cloud network environment and tenants are as follows: Figure 10 As shown, the metadata management system runs on the internal cloud platform and can include OPS (Orbeon Presentation Server, a web framework) components and microservices, WEB / API, gateways, and backend microservices. On the public cloud side, there are not only VPC tenants corresponding to different users, but also task execution resources and computing engines.

[0166] A public cloud environment consists of VPC tenants. Using offline custom data collection is the most user-friendly approach while avoiding impact on user services. Therefore, the deployment method is as follows: Figure 11 As shown, the data collection agent (i.e., the collector) is integrated and deployed within the provided scheduler, providing users with a Crawler runner type that poses no risk to user EMR, file sources, or other metadata. Simultaneously, the Crawler runner reports its heartbeat, status, and other information to the task scheduling platform. Here, DB is a conventional database format, while NEW DB is another special database format; the target data source is stored in either DB or NEW DB.

[0167] Understandable Figure 11 In a VPC tenant, there is a runner that integrates a collection agent. Essentially, it is still a single point of entry. However, it can be used to build a distributed collection capability group with multiple agents to further enhance the collection capability.

[0168] In summary, the metadata collection method provided in this application, such as Figure 12 As shown, the following steps can be further included:

[0169] (1) Users configure metadata collection configuration information in the metadata management system according to their needs;

[0170] (2) The metadata management system sends the corresponding collector authorization verification to the cloud MQ;

[0171] (3) If the authorization verification fails, the task creation will fail; if the authorization verification passes, the metadata collection configuration information will be sent to the task scheduling platform to create the metadata collection task.

[0172] (4) The task scheduler schedules the metadata collection task to the execution machine at the corresponding trigger time. If the scheduling fails, the execution task stops. If the scheduling is successful, the corresponding collector encrypts and compresses the collected metadata.

[0173] (5) After the collector completes the authorization verification with the cloud MQ based on the authorization token, it pushes the encrypted and compressed metadata to the cloud MQ.

[0174] (6) The message pipeline consumes this data and stores it in the metadata database of the unified data model.

[0175] The solution provided in this application addresses the limited execution resources under a user's VPC. To ensure the stability, efficiency, security, and risk-free nature of metadata collection, this application proposes a divide-and-conquer approach based on the Cost-Based Optimizer (CBO) mechanism, rationally splitting the collection tree to guarantee stability within limited resources. Simultaneously, this application proposes data transcoding and compression methods to reduce bandwidth consumption and improve transmission efficiency. Furthermore, this application employs asymmetric data encryption to ensure data transmission security, and a physical authorization mechanism ensures the collector's effectiveness. Finally, this application proposes a customizable metadata retrieval method, integrating the collector into the scheduling execution machine to ensure ease of use and user experience.

[0176] The following example further illustrates the solution of this application, such as... Figure 13 As shown, the three public cloud products—Data Lake Computing, Data Ingestion, and Data Assets—all utilize metadata management and collection capabilities. All three products integrate metadata services and expose user-interactive metadata configuration interfaces. The metadata itself relies on collection and scheduling capabilities to periodically acquire metadata information. To adapt to different scheduling engines and metadata systems, a relatively universal scheduling adapter is designed. This adapter not only adapts to the scheduling and collection configurations but also authorizes cloud MQ agents (i.e., collectors) with permissions at this step. Finally, the relevant configurations are distributed to the scheduling platform, triggering the collector to collect cloud data. The collector includes various JAR (Java Archive) types: ordinary JARs and Spark JARs. The ordinary JAR type is presented as a collector named Crawler, which primarily includes two types of metadata collection capabilities: relational JDBC pull and custom pull. The Spark JAR type is presented as a collector named InferSchema.

[0177] A standard JAR-type data collector can gather metadata from relational databases (such as MySQL, Oracle, and Postgres) and big data stores (such as Hive, HBase, and Kafka). Within a standard JAR-type collector, the JAR file includes capabilities such as adaptive data sources, unified data formats, data encryption, and data compression. After acquiring data, the collector pushes it to a cloud-based message queue (MQ). The diagram shows Pulsar (a distributed message publish / subscribe platform) as the MQ receiver. Subsequently, the metadata from the cloud MQ is consumed through the Databus (metadata consumption logic block), and various cloud data services (such as data adaptation, change discovery, change confirmation, and historical archiving) process the metadata before storing it in a persistent storage database (such as TD SQL or Elasticsearch).

[0178] The main difference between online and offline triggering methods in the collection type is whether the metadata collection is triggered immediately or at a set time. This part of the configuration is set by the user. After the metadata data is pushed to the message middleware cloud MQ through the collector, it will be processed by the message processing logic module of the metadata management and finally stored in the database for use by relevant applications.

[0179] In summary, the underlying metadata capabilities include metadata management, metadata collection, and scheduling adaptation. The productized platform's metadata source is a configured metadata collector, which is triggered by user-configured scheduling. The collector internally includes dynamic data source adaptation capabilities, dynamic resource adaptation capabilities, data encryption, and compression capabilities. Deployed on execution machines within the customer's VPC tenant network, the collector is naturally interconnected with the user's network, eliminating the need for additional network connectivity costs. Internally, the collector can adapt to various data sources and formats, converting different data formats into a universal metadata format. Finally, the collected data is pushed to a message middleware for consumption and processing by the built-in metadata service.

[0180] Figure 14 This is a structural block diagram of a metadata collection device provided in an embodiment of this application, such as... Figure 14 As shown, the device 1400 may include: a metadata acquisition task receiving module 1401, a data source type adaptive module 1402, a resource adaptive module 1403, and a metadata push module 1404, wherein:

[0181] The metadata collection task receiving module 1401 is used to receive metadata collection tasks issued by the task scheduling platform;

[0182] The data source type adaptive module 1402 is used to determine the target data source to be collected and the data source type of the target data source based on the metadata collection task, and to establish a connection between the collector and the target data source based on the data source type;

[0183] The resource adaptation module 1403 is used to obtain the data information of the metadata in the target data source and the current device environment resource information, and to obtain at least one acquisition subtree corresponding to the target data source and adapted to the device environment resource information based on the data information. The data information is used to indicate the storage structure of the metadata in the target data source.

[0184] The metadata push module 1404 is used to perform metadata collection tasks based on at least one collection subtree to obtain the metadata corresponding to each collection subtree, and push the metadata corresponding to each collection subtree to the cloud message queue MQ, so that the preset consumption pipeline can consume the metadata through the cloud MQ and store it in the preset metadata database.

[0185] The solution provided in this application, after receiving a metadata collection task from a task scheduling platform, determines the data source type of the target data source, adapts the collector to this data source type, and establishes a connection with the target data source. Then, based on the metadata information in the target data source, it divides the target data source into one or more collection subtrees with data volumes adapted to the current device environment resources. Finally, it collects metadata corresponding to the metadata collection task according to each collection subtree and pushes the metadata to the cloud MQ for consumption and storage. This solution, on the one hand, adapts to the data source type when establishing the connection between the collector and the target data source, ensuring adaptability to different resource types in a public cloud environment; on the other hand, it adapts to the current resources before data collection, ensuring resource adaptability by matching resources with the amount of data to be collected, thereby guaranteeing the stability of metadata collection.

[0186] In one optional embodiment of this application, the resource adaptive module is specifically used for:

[0187] Based on the data information, obtain the parent node, child node and connection relationship between the target data source, and based on the parent node, child node and connection relationship between the nodes, obtain the corresponding acquisition tree and the cost-based optimization method (CBO) cost corresponding to the acquisition tree, and obtain the corresponding CBO cost threshold based on the equipment environment resource information.

[0188] If the cost-benefit (CBO) cost of the collection tree is not greater than the CBO cost threshold, then the collection tree is used as at least one collection subtree. If the CBO cost of the collection tree is greater than the CBO cost threshold, then the collection tree is split to obtain at least one collection subtree.

[0189] In an optional embodiment of this application, the resource adaptation module is further configured to:

[0190] The target data source is used as the parent node, and the database, file type or topic type in the target data source is used as the child node. The corresponding connection relationship is obtained based on the subordinate relationship between the nodes.

[0191] In an optional embodiment of this application, the resource adaptation module is further configured to:

[0192] The data collection tree is split from top to bottom at the splitting child nodes. The splitting child nodes at each split are determined from the child nodes of the corresponding level of the data collection tree or each subtree using the binary k-nearest neighbor query algorithm.

[0193] After each split, the subtrees in each split where the CBO cost is not greater than the CBO cost threshold are identified as the collection subtrees. The splitting of subtrees where the CBO cost is greater than the CBO cost threshold continues until the CBO cost of each split subtree is not greater than the CBO cost threshold, thus obtaining at least one collection subtree.

[0194] In one alternative embodiment of this application, the data source type adaptive module is specifically used for:

[0195] Determine the collector parameters based on the data source type;

[0196] Establish a connection between the collector and the target data source based on the collector parameters.

[0197] In one optional embodiment of this application, the metadata collection task receiving module is specifically used for:

[0198] Collect metadata for portions of at least one collection subtree; or

[0199] The metadata corresponding to each collection subtree is collected according to the splitting order of each collection subtree.

[0200] In an optional embodiment of this application, the device further includes a metadata collection task creation module, used for:

[0201] The metadata collection configuration information is obtained through the metadata management system, and the collector authorization verification is initiated to the cloud MQ. The metadata collection configuration information includes: data source information, scheduling information and encryption information.

[0202] If the collector authorization verification is successful, it will receive the authorization token sent by the cloud MQ through the metadata management system, and create a metadata collection task carrying data source information, scheduling information, authorization token and encryption information through the task scheduling platform.

[0203] In one optional embodiment of this application, the device further includes a metadata collection task creation module specifically used for:

[0204] The task scheduling platform obtains the trigger time of the metadata collection task based on the scheduling information, and receives the metadata collection task issued by the task scheduling platform at the trigger time.

[0205] In one optional embodiment of this application, the metadata push module is specifically used for:

[0206] Obtain encrypted information and authorization tokens based on metadata collection tasks;

[0207] Based on the encrypted information, the metadata corresponding to each collection subtree is encrypted to obtain encrypted metadata. Then, the encrypted metadata is transcoded according to a preset method to obtain transcoded metadata. Finally, the transcoded metadata is compressed to obtain encrypted and compressed metadata.

[0208] After completing the authorization verification with the cloud MQ based on the authorization token, the encrypted and compressed metadata is pushed to the cloud MQ.

[0209] The following is for reference. Figure 15 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., performing...). Figure 2 The diagram illustrates the structure of the terminal device or server 1500 of the method shown. The electronic devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 15 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0210] The electronic device includes a memory and a processor. The memory stores a program for performing the methods described in the above-described method embodiments; the processor is configured to execute the program stored in the memory. The processor may be referred to as processing device 1501 as described below, and the memory may include at least one of read-only memory (ROM) 1502, random access memory (RAM) 1503, and storage device 1508 as described below, as specifically shown below:

[0211] like Figure 15As shown, the electronic device 1500 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage device 1508 into a random access memory (RAM) 1503. The RAM 1503 also stores various programs and data required for the operation of the electronic device 1500. The processing unit 1501, ROM 1502, and RAM 1503 are interconnected via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0212] Typically, the following devices can be connected to I / O interface 1505: input devices 1506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1509. Communication device 1509 allows electronic device 1500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 15 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0213] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1509, or installed from storage device 1508, or installed from ROM 1502. When the computer program is executed by processing device 1501, it performs the functions defined in the methods of embodiments of this application.

[0214] It should be noted that the computer-readable storage medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0215] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0216] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0217] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0218] The system receives metadata collection tasks from the task scheduling platform; determines the target data source and its type based on the task, and establishes a connection between the collector and the target data source based on the data source type; acquires the metadata information and current device environment resource information from the target data source, and obtains at least one collection subtree corresponding to the target data source and adapted to the device environment resource information based on the data information, which indicates the storage structure of the metadata in the target data source; executes metadata collection tasks based on at least one collection subtree to acquire the metadata corresponding to each collection subtree, and pushes the metadata corresponding to each collection subtree to the cloud message queue (MQ) for the preset consumption pipeline to consume the metadata through the cloud MQ and store it in the preset metadata database.

[0219] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0220] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0221] The modules or units described in the embodiments of this application can be implemented in software or hardware. The names of modules or units do not necessarily limit the specific unit; for example, a first constraint acquisition module can also be described as a "module for acquiring the first constraint".

[0222] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0223] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the following actions:

[0225] The system receives metadata collection tasks from the task scheduling platform; determines the target data source and its type based on the task, and establishes a connection between the collector and the target data source based on the data source type; acquires the metadata information and current device environment resource information from the target data source, and obtains at least one collection subtree corresponding to the target data source and adapted to the device environment resource information based on the data information, which indicates the storage structure of the metadata in the target data source; executes metadata collection tasks based on at least one collection subtree to acquire the metadata corresponding to each collection subtree, and pushes the metadata corresponding to each collection subtree to the cloud message queue (MQ) for the preset consumption pipeline to consume the metadata through the cloud MQ and store it in the preset metadata database.

[0226] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0227] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for collecting metadata, characterized in that, include: Receive metadata collection tasks issued by the task scheduling platform; Based on the metadata collection task, the target data source to be collected and the data source type of the target data source are determined, and a connection between the collector and the target data source is established based on the data source type; Obtain metadata information and current device environment resource information from the target data source, and obtain the parent node, child node and connection relationship between the nodes corresponding to the target data source based on the data information. Based on the parent node, child node and connection relationship between the nodes, obtain the corresponding acquisition tree and the cost-based optimization method (CBO) cost corresponding to the acquisition tree, and obtain the corresponding CBO cost threshold based on the device environment resource information. If the CBO cost of the acquisition tree is not greater than the CBO cost threshold, then the acquisition tree is used as at least one acquisition subtree; if the CBO cost of the acquisition tree is greater than the CBO cost threshold, then the acquisition tree is split to obtain the at least one acquisition subtree. The CBO cost of at least one acquisition subtree is not greater than the CBO cost threshold, and the data information is used to indicate the storage structure of metadata in the target data source; The metadata collection task is executed based on the at least one collection subtree to obtain the metadata corresponding to each collection subtree, and the metadata corresponding to each collection subtree is pushed to the cloud message queue MQ, so that the preset consumption pipeline can consume the metadata through the cloud MQ and store it in the preset metadata database.

2. The method according to claim 1, characterized in that, The step of obtaining the parent node, child node, and connection relationships between nodes corresponding to the target data source based on the data information includes: The target data source is taken as the parent node, and the database, file type or topic type in the target data source is taken as the child node. The corresponding connection relationship is obtained based on the subordinate relationship between the nodes.

3. The method according to claim 1, characterized in that, The step of splitting the collection tree to obtain the at least one collection subtree includes: The collection tree is split from top to bottom at the splitting child nodes. The splitting child nodes at each split are determined from the child nodes of the collection tree or each subtree at the corresponding level by the binary k-nearest neighbor query algorithm. After each split, the subtrees in each split where the CBO cost is not greater than the CBO cost threshold are identified as acquisition subtrees. The splitting of subtrees where the CBO cost is greater than the CBO cost threshold continues until the CBO cost of each split subtree is not greater than the CBO cost threshold, thus obtaining the at least one acquisition subtree.

4. The method according to claim 1, characterized in that, The process of establishing a connection between the collector and the target data source based on the data source type includes: Based on the data source type, determine the collector parameters of the collector; The collector is connected to the target data source based on the collector parameters.

5. The method according to claim 1, characterized in that, The execution of the metadata collection task based on the at least one collection subtree includes: Metadata corresponding to a portion of the collection subtrees in at least one collection subtree is collected; or The metadata corresponding to each collection subtree is collected according to the splitting order of each collection subtree.

6. The method according to claim 1, characterized in that, The method further includes: The metadata collection configuration information is obtained through the metadata management system, and the collector authorization verification is initiated to the cloud MQ. The metadata collection configuration information includes: data source information, scheduling information and encryption information. If the collector authorization verification is successful, it receives the authorization token sent by the cloud MQ through the metadata management system, and creates the metadata collection task carrying the data source information, the scheduling information, the authorization token and the encrypted information through the task scheduling platform.

7. The method according to claim 6, characterized in that, The metadata collection task issued by the receiving task scheduling platform includes: The task scheduling platform obtains the trigger time of the metadata collection task based on the scheduling information, and receives the metadata collection task issued by the task scheduling platform at the trigger time.

8. The method according to claim 6, characterized in that, The step of pushing the metadata corresponding to each collection subtree to the cloud message queue (MQ) includes: The encrypted information and the authorization token are obtained based on the metadata collection task; Based on the encrypted information, the metadata corresponding to each collection subtree is encrypted to obtain encrypted metadata. Then, the encrypted metadata is transcoded according to a preset method to obtain transcoded metadata. Finally, the transcoded metadata is compressed to obtain encrypted and compressed metadata. After completing the authorization verification with the cloud MQ based on the authorization token, the encrypted and compressed metadata is pushed to the cloud MQ.

9. A metadata acquisition device, characterized in that, include: The metadata collection task receiving module is used to receive metadata collection tasks issued by the task scheduling platform. The data source type adaptive module is used to determine the target data source to be collected and the data source type of the target data source based on the metadata collection task, and to establish a connection between the collector and the target data source based on the data source type; The resource adaptive module is used to acquire metadata information and current device environment resource information from the target data source, and based on the data information, acquire the parent node, child node, and connection relationship between the nodes corresponding to the target data source. Based on the parent node, child node, and connection relationship between the nodes, acquire the corresponding acquisition tree and the cost-based optimization (CBO) cost corresponding to the acquisition tree, and acquire the corresponding CBO cost threshold based on the device environment resource information. If the CBO cost of the acquisition tree is not greater than the CBO cost threshold, the acquisition tree is taken as at least one acquisition subtree. If the CBO cost of the acquisition tree is greater than the CBO cost threshold, the acquisition tree is split to obtain the at least one acquisition subtree. The CBO cost of at least one acquisition subtree is not greater than the CBO cost threshold, and the data information is used to indicate the storage structure of metadata in the target data source; The metadata push module is used to execute the metadata collection task based on the at least one collection subtree to obtain the metadata corresponding to each collection subtree, and push the metadata corresponding to each collection subtree to the cloud message queue MQ, so that the preset consumption pipeline can consume the metadata through the cloud MQ and store it in the preset metadata database.

10. An electronic device, characterized in that, Including memory and processor; The memory stores computer programs; The processor is configured to execute the computer program to implement the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images