Data lake metadata processing method and apparatus, electronic device, medium, and product

Abstract

The present disclosure provides a data lake metadata processing method and device, electronic equipment, medium and product, relates to the technical field of metadata processing, and can be applied to the technical field of financial technology. The method comprises the following steps: configuring attribute information of the data lake metadata, and defining the data lake metadata as business metadata, technical metadata and running metadata according to the attribute information; in response to a data lake entry request of the data lake metadata, collecting the business metadata and the technical metadata according to registered entry source table information and element information associated with the data lake metadata; and in response to the data lake entry of the data lake metadata, collecting actual running information when the data lake metadata enters the lake to obtain the running metadata.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of metadata processing, which can be applied to the technical field of financial technology, and particularly relates to a data lake metadata processing method and device, electronic equipment, medium and product. BACKGROUND

[0002] With the digital bank ecology, the economic society has entered the post-digital era, and data, as a new type of production factor, plays a crucial role. In this regard, the financial system builds a data lake, a data platform for centralized storage of data, to realize the integration of data from business source systems. At present, the data lake has formed thousands of data tables. In order to better manage the data assets of the source layer and clarify the data assets, metadata management of the data stored in the data lake platform is needed, and then effective support is provided to data users, so as to facilitate system developers and business personnel to understand and use data lake data from different angles.

[0003] In the process of implementing the present disclosure, the applicant found that the prior art lacks an effective metadata processing method to realize centralized processing and management of metadata, which makes it difficult for data users to effectively identify data information, and thus affects data use. SUMMARY

[0004] The main purpose of the present disclosure is to provide a data lake metadata processing method, device, electronic equipment, medium and product, which aims to solve the technical problem of how to effectively process and manage data lake metadata in the prior art.

[0005] To achieve the above-mentioned purpose, the first aspect of the present disclosure provides a data lake metadata processing method, comprising: configuring attribute information of the data lake metadata, and defining the data lake metadata as business metadata, technical metadata and running metadata according to the attribute information; in response to an entry request of the data lake metadata, collecting the business metadata and the technical metadata according to registered entry source table information and element information associated with the data lake metadata; and in response to entry of the data lake metadata, collecting actual running information when the data lake metadata enters, to obtain the running metadata.

[0006] According to the embodiments of the present disclosure, the collecting of the business metadata and the technical metadata according to the registered entry source table information and the element information associated with the data lake metadata specifically comprises: verifying whether the data lake metadata is allowed to enter according to the entry source table information and the element information; and in response to allowing the data lake metadata to enter, collecting the business metadata and the technical metadata.

[0007] According to an embodiment of the present disclosure, the method further comprises: in response to allowing the data lake metadata to be imported, generating import configuration information according to the registered import source table information and element information associated with the data lake metadata; and in response to the data lake metadata being imported, reading the data lake metadata transmitted by the business system according to the import configuration information and importing the data lake metadata.

[0008] According to an embodiment of the present disclosure, the data source application is invoked to register the import source table information and the element information associated with the data lake metadata.

[0009] According to an embodiment of the present disclosure, the processing method further comprises: transmitting the business metadata, the technical metadata and the running metadata to a big data asset management platform, and displaying the business metadata, the technical metadata and the running metadata.

[0010] According to an embodiment of the present disclosure, the business metadata, the technical metadata and the running metadata are transmitted to the big data asset management platform based on a timing transmission mechanism.

[0011] According to an embodiment of the present disclosure, the processing method further comprises: configuring a data asset number as a unique index identifier of the business metadata, the technical metadata and the running metadata; and retrieving and querying the business metadata, the technical metadata and the running metadata according to the unique index identifier.

[0012] According to an embodiment of the present disclosure, the business metadata at least includes business description, business field, business process, asset source system, data review department, system table name, data supplier, domestic and foreign sign and data sharing strategy; the technical metadata at least includes name of data table, primary key information, field name, field type, field length, field precision, field dictionary, data standard and data loading algorithm; and the running metadata at least includes latest data date, data loading frequency, data table capacity, existing earliest data time, most recent data loading time, storage period and data quality inspection problem.

[0013] The second aspect of the present disclosure provides a data lake metadata processing device, comprising: a configuration module configured to configure attribute information of the data lake metadata, and define the data lake metadata as business metadata, technical metadata and running metadata according to the attribute information; a first acquisition module configured to, in response to an import request of the data lake metadata, acquire the business metadata and the technical metadata according to registered import source table information and element information associated with the data lake metadata; and a second acquisition module configured to, in response to the data lake metadata being imported, acquire actual running information when the data lake metadata is imported, and obtain the running metadata.

[0014] The third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to the above description.

[0015] The fourth aspect of the present disclosure provides a computer-readable storage medium, the computer-readable storage medium stores executable instructions, the instructions are executed by a processor to make the processor perform the method according to the above description.

[0016] The fifth aspect of the present disclosure provides a computer program product, comprising a computer program, the computer program is executed by a processor to implement the method according to the above description.

[0017] The data lake metadata processing method, device, electronic device, medium and product provided by the embodiments of the present disclosure have at least the following beneficial effects:

[0018] By configuring the data lake metadata attribute and defining the data asset as business metadata, technical metadata and running metadata according to the attribute, since the business metadata and the technical metadata are static data and the running metadata is dynamic data, the business metadata and the technical metadata can be collected according to the registered lake source table information and the element information associated with the data lake metadata when responding to the lake request, and the actual running information of the lake can be automatically collected when the data is entered into the lake. Through this way of collecting and summarizing metadata at different times, the processing and management of the data lake metadata can be effectively realized.

[0019] Further, when the data is allowed to enter the lake, the lake configuration information is generated according to the registered lake source table information and the element information associated with the data lake metadata. Since the lake configuration information covers the business metadata and the technical metadata, the data entered into the lake stored by the business system can be efficiently and accurately read according to the lake configuration information when the data is entered into the house.

[0020] Further, the data asset number is configured as the unique index identifier of the business metadata, the technical metadata and the running metadata, so that the data asset map, retrieval and query service can be realized.

[0021] In addition, the summarized metadata is transmitted to the big data asset management platform for display at regular intervals, so that system developers and business personnel can intuitively understand and use the data lake metadata from different angles. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from the technical solutions shown in these drawings without creative effort.

[0023] Figure 1 The system architecture 100 of the data lake metadata processing method and system according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 2 The flowchart of the data lake metadata processing method according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 3 The flowchart of the operation S202 according to an embodiment of the present disclosure is schematically shown; Figure 2 The application scenario diagram of the data lake metadata processing method is schematically shown;

[0026] Figure 4 The flowchart of the operation S202 according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 5 The flowchart of the data lake metadata processing method according to another embodiment of the present disclosure is schematically shown;

[0028] Figure 6 The flowchart of the data lake metadata processing method according to another embodiment of the present disclosure is schematically shown;

[0029] Figure 7 The block diagram of the data lake metadata processing apparatus according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 8 The block diagram of the data lake metadata processing apparatus according to another embodiment of the present disclosure is schematically shown;

[0031] Figure 9 The block diagram of the data lake metadata processing apparatus according to another embodiment of the present disclosure is schematically shown;

[0032] Figure 10 The block diagram of the electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown.

[0033] The implementation, functional features and advantages of the present disclosure will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0034] In the following, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments, or one or more of the embodiments, can be practiced without these specific details. In other instances, well-known structures and

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "including" "comprising" and the like are meant to be inclusive, but not limiting to the components, steps, operations and / or features that were listed. The use of "including" "comprising" and "having" also modifies the term "comprising" to include additional feature.

[0036] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the use of certain terms or terminology in this document should not be taken to imply any particular intention or meaning beyond the normal or customary meaning of the term unless otherwise indicated.

[0037] In the event a statement similar to "at least one of A, B, and C, etc." is used, it is intended to mean that the scope of A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together, is also contemplated.

[0038] Some of the blocks and / or flowcharts can be implemented in computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus implement the functions / acts specified in the flowcharts and / or block diagrams.

[0039] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and the public order and good customs are not violated.

[0040] In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is acquired or collected.

[0041] To solve the technical problems in the related art, the present embodiment provides a data lake metadata processing method, including: configuring attribute information of data lake metadata, and defining the data lake metadata as business metadata, technical metadata and running metadata according to the attribute information. In response to a lake entry request of the data lake metadata, collecting the business metadata and the technical metadata according to registered lake entry source table information and element information associated with the data lake metadata; in response to the lake entry of the data lake metadata, collecting actual running information when the data lake metadata enters the lake to obtain the running metadata.

[0042] Figure 1 The system architecture 100 of the data lake metadata processing method and system according to the present embodiment is schematically shown. It should be noted that, Figure 1 The system architecture shown is only an example of the system architecture to which the present embodiment can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the present embodiment cannot be used in other devices, systems, environments or scenarios.

[0043] As Figure 1 shown, the system architecture 100 according to the embodiment can include a data source application 101, a network 102, a data lake metadata processing platform 103 and a big data asset management platform 104. The network 102 is used to provide a communication link between the data source application 101 and the data lake metadata processing platform 103 and between the data lake metadata processing platform 103 and the big data asset management platform 104.

[0044] The data source application 101 can be a source application for implementing data lake, which can be used to register the source table information of the data lake, and register the element information related to the data lake according to the requirements, and generate the lake configuration information after registration. The network 102 can include various connection types, such as wired, wireless communication link or optical cable, etc. The wired mode can be connected by using any one of the following various interfaces, such as fiber channel, infrared interface, D-type data interface, serial interface, USB interface, USB Type-C interface or Dock interface. The wireless mode can be connected by using wireless communication, which can use any one of the following wireless technology standards, such as Bluetooth, Wi-Fi, Infrared, ZigBee, etc. The data lake metadata processing platform 103 is used to configure the attribute information of the data lake metadata, define the data lake metadata as business metadata, technical metadata and running metadata according to the attribute information, and aggregate the business metadata, technical metadata and running metadata based on the data source application 101 to implement data lake through the network 102, and then send to the big data asset management platform 104 through the network for display.

[0045] It should be noted that the data lake metadata processing method provided by the embodiments of the present disclosure can be executed by the data lake metadata processing platform 103. Accordingly, the data lake metadata processing apparatus provided by the embodiments of the present disclosure can be arranged in the data lake metadata processing platform 103. Alternatively, the data lake metadata processing method provided by the embodiments of the present disclosure can also be executed by a data lake processing platform or a data lake processing platform cluster different from the data lake metadata processing platform 103 and capable of communicating with the data source application 101 and / or the data lake metadata processing platform 103. Accordingly, the data lake metadata processing apparatus provided by the embodiments of the present disclosure can also be arranged in a data lake processing platform or a data lake processing platform cluster different from the data lake metadata processing platform 103 and capable of communicating with the data source application 101 and / or the data lake metadata processing platform 103. Alternatively, the data lake metadata processing method provided by the embodiments of the present disclosure can also be partially executed by the data lake metadata processing platform 103 and partially executed by the data source application 101. Accordingly, the data lake metadata processing apparatus provided by the embodiments of the present disclosure can also be partially arranged in the data lake metadata processing platform 103 and partially arranged in the data source application 101.

[0046] It should be understood that Figure 1 The number of data source applications, networks and data lake processing platforms in the above-mentioned system is only illustrative. According to the implementation requirements, there can be any number of data source applications, networks and data lake processing platforms.

[0047] The data lake metadata processing method provided by the embodiments of the present disclosure can be applied to the field of financial technology. For example, for a bank, when the bank implements retail, public, financial market, risk management, internal management, and cross-field management businesses, a series of data assets will be generated, which play an important role in the technical research and development of R&D personnel and the business optimization of business personnel. Therefore, it is necessary to store and manage these data assets to clarify the understanding and use of data assets from different perspectives of system R&D personnel and business personnel. For example, at present, the Industrial and Commercial Bank of China data lake has formed 35,000 data tables of 200 business systems in the whole bank. It is necessary to present the data assets of more than 35,000 tables accumulated on the data lake to the whole bank, so that data users can intuitively and efficiently obtain data asset information. It is necessary to effectively process and manage the data assets of more than 35,000 accumulated tables.

[0048] There is a lack of effective metadata processing method in the prior art to implement centralized processing and management of data assets, which makes it difficult for data users to effectively identify data asset information. The data lake metadata processing method provided by the embodiments of the present disclosure can at least partially solve the technical problems existing in the prior art.

[0049] It should be understood that the data lake metadata processing method provided by the embodiments of the present disclosure is not limited to application in the field of financial technology. The above description is only exemplary. The data lake metadata processing method of the embodiments of the present disclosure can be applied to other technical fields related to data lake metadata processing.

[0050] Figure 2 A flowchart of a data lake metadata processing method according to an embodiment of the present disclosure is schematically shown.

[0051] As shown in Figure 2 The data lake metadata processing method may, for example, include operations S201-S203.

[0052] In operation S201, attribute information of data lake metadata is configured, and the data lake metadata is defined as business metadata, technical metadata, and running metadata according to the attribute information.

[0053] In operation S202, in response to a data lake metadata entry request, business metadata and technical metadata are collected according to registered entry source table information and element information associated with the data lake metadata.

[0054] In operation S203, in response to the data lake metadata entry, actual running information when the data lake metadata enters is collected to obtain running metadata.

[0055] In the embodiments of the present disclosure, metadata can be defined as "data about data", which is mainly information describing data attributes, and is used to support functions such as indicating storage location, historical data, resource lookup, file record, etc., therefore, data assets are converted into metadata, and storage management is performed based on a data lake to facilitate the clarification of data assets.

[0056] In the embodiments of the present disclosure, in combination with the current status of the existing financial business system and data architecture, lake table business metadata attributes, technical metadata attributes, and lake running metadata attributes can be configured for the data lake metadata corresponding to the data assets, and according to these attribute information, the data lake metadata corresponding to the data assets can be positioned as business metadata, technical metadata, and running metadata.

[0057] Further, the business metadata can at least include data business description, business field, business process, asset source system, data production department, system table Chinese name, system table English name, data supplier, domestic and foreign sign, security and permission. The business description may, for example, refer to data asset business definition description. The business field may, for example, include retail field, public field, financial market field, risk field, internal management field, and cross-field, etc. The business process may, for example, refer to the business process link where the data asset is generated. The asset source system may, for example, refer to the system name of the data resource source. The data production part may, for example, refer to the business department where the data is generated. The system table Chinese name may, for example, refer to the Chinese name of the system table where the data asset is located. The system table English name may, for example, refer to the English name of the system table where the data asset is located. The data supplier may, for example, refer to the data supplier name corresponding to the bank external data. The domestic and foreign sign may, for example, include domestic and foreign. The security and permission may, for example, refer to the data asset sharing strategy of each department and each level of the enterprise.

[0058] Further, the technical metadata can at least include data lake table Chinese name, data lake table English name, primary key information, field Chinese name, field English name, field type, field length, field precision, field dictionary, data standard, and data loading algorithm. The data lake table Chinese name may, for example, refer to the Chinese description name of the data table. The data lake table English name may, for example, refer to the English description name of the data table. The primary key information may, for example, refer to the primary key information of the data table. The field Chinese name may, for example, refer to the field Chinese description name of the data table. The field English name may, for example, refer to the field English description name of the data table. The field type may, for example, refer to the data structure type defined by the data table field. The field length may, for example, refer to the data structure length defined by the data table field. The field precision may, for example, refer to the data structure precision defined by the data table field. The field dictionary may, for example, refer to the data dictionary value and business meaning corresponding to the data table field. The data standard may, for example, refer to the data standard followed by the data field. The data loading algorithm may, for example, refer to the data table storage update mode.

[0059] Further, the running metadata at least includes a data lake table Chinese name, a data lake table English name, a latest data date, a data loading frequency, a data table capacity, an existing earliest data time, a last data loading time, a storage period, and a data quality check problem. The data lake table Chinese name may refer to a Chinese description name of the data table, for example. The data lake table English name may refer to an English description name of the data table, for example. The latest data date may refer to a current data date of the data table, for example. The data loading frequency may refer to a frequency of updating the data table, for example. The storage capacity of the data table may refer to a storage capacity of the data table, for example. The existing earliest data time may refer to an earliest data date stored by the data table, for example. The last loading time may refer to a latest update time of the data table, for example. The storage period may refer to a historical data retention period of the data table, for example. The data quality check problem may refer to a current data quality problem of the data table, for example.

[0060] Figure 3 A flowchart of operation S202 is schematically shown according to an embodiment of the present disclosure. Figure 2 An application scenario diagram of the data lake metadata processing method is shown.

[0061] As shown in Figure 3 In the data lake metadata, the business metadata and the technical metadata are mainly static information. The data lake metadata processing platform 103 can provide a unified registration function for the developers of various businesses to register when the data lake metadata is entered, and the running metadata is mainly actual running information of the data lake metadata entered, which is dynamic information and can be obtained from the running condition of the entry platform.

[0062] According to the data lake metadata processing method provided in the embodiments of the present disclosure, the data lake metadata attributes are configured, and the data assets are defined as the business metadata, the technical metadata, and the running metadata according to the attributes. Since the business metadata and the technical metadata are static data, and the running metadata is dynamic data, the business metadata and the technical metadata can be collected according to the registered entry source table information and the element information associated with the data lake metadata when responding to the entry request. The actual running information of the entry can be automatically collected when the data is entered. Through this way of collecting and summarizing the metadata at different times, the processing and management of the data lake metadata can be effectively realized, and the integration and management of the source data assets can be effectively clarified, and the unified inventory of the source data assets can be completed.

[0063] Figure 4 A flowchart of operation S202 is schematically shown according to an embodiment of the present disclosure.

[0064] As shown in Figure 4As shown, operation S202 may include operations S401 to S402, for example.

[0065] In operation S401, whether the data lake metadata is allowed to enter the lake is checked based on the source table information and feature information.

[0066] In the disclosed embodiment, the requirements for data entry into the lake are first determined. The data source application for entry is then called to register the source table information and element information associated with the data lake metadata. This element information can be, for example, lake entry research information. The registered information is then imported into the data lake R&D workstation of the data lake metadata processing platform for access verification. This determines whether the data lake metadata is allowed to enter the lake.

[0067] In operation S402 , in response to allowing data lake metadata to enter the lake, business metadata and technical metadata are collected.

[0068] In the embodiment of the present disclosure, when it is determined that data lake metadata is allowed to enter the lake, static business metadata and technical metadata can be collected.

[0069] It should be understood that when the verification fails, it indicates that the data lake metadata is not allowed to enter the lake, so there is no need to collect business metadata, technical metadata, and operational metadata.

[0070] Figure 5 The following schematically illustrates a flowchart of a data lake metadata processing method according to another embodiment of the present disclosure.

[0071] like Figure 5 As shown, the data lake metadata processing method may include operations S501 to S502.

[0072] In operation S501 , in response to allowing data lake metadata to enter the lake, lake entry configuration information is generated according to registered lake entry source table information and element information associated with the data lake metadata.

[0073] In the embodiment of the present disclosure, the configuration information covers business metadata and technical metadata. In operation S502, in response to the data lake metadata being entered into the lake, the data lake metadata downloaded from the business system is read according to the lake entry configuration information, and the data lake metadata is entered into the lake.

[0074] In the embodiment of the present disclosure, after the data lake metadata is successfully entered into the lake, the data lake metadata can be stored in the data table. During the process of entering the data lake metadata into the lake, operational metadata such as the date of the recorded data, the number of data records, and the amount of data can be collected.

[0075] Exemplarily, in the above business metadata, technical metadata and running metadata acquisition process, the collection of metadata can be started at a regular time (for example, metadata collection is performed daily), the data lake table list and the data lake job name are read, the business metadata configuration information is associated according to the data lake table list and the data lake job name, the metadata table is associated to obtain the table structure field information, the job scheduling table is associated to obtain the data lake information, the loading result table is associated to obtain the record number, finally, the associated information is used to generate the data lake table business metadata information result file, the data lake table technical metadata information result file, the data lake table running metadata information result file and the data lake table structure information result file.

[0076] According to the embodiments of the present disclosure, when data is allowed to enter the lake, the data lake configuration information is generated according to the registered data lake source table information and the element information associated with the data lake metadata. Since the data lake configuration information covers the business metadata and the technical metadata, when the data enters the lake, the data entered by the business system can be efficiently and accurately read according to the data lake configuration information.

[0077] Figure 6 A flowchart of a data lake metadata processing method according to still another embodiment of the present disclosure is schematically shown.

[0078] As shown in Figure 6 , the data lake metadata processing method may, for example, include operations S601-S602.

[0079] In operation S601, the data asset number is configured as a unique index identifier of the business metadata, the technical metadata and the running metadata.

[0080] In operation S602, the business metadata, the technical metadata and the running metadata are retrieved and queried according to the unique index identifier.

[0081] In the embodiments of the present disclosure, since the data asset number is unique, that is, the data asset has a unique identification code, the format of the code is "asset type abbreviation + sequence number", therefore, the data asset number can be used as the unique index identifier of the data asset. In other words, the business metadata, the technical metadata and the running metadata can also include the data asset number. Based on the data asset number as the unique index identifier of the data asset, the business metadata, the technical metadata and the running metadata can be quickly retrieved and queried.

[0082] In the embodiments of the present disclosure, the data lake metadata processing method may, for example, further include: transmitting the business metadata, the technical metadata and the running metadata to a big data asset management platform, and displaying the business metadata, the technical metadata and the running metadata.

[0083] Exemplarily, the generated business metadata information result file, the technical metadata information result file and the operation metadata information result file can be transmitted to the big data asset management platform.

[0084] Further, the business metadata, the technical metadata and the operation metadata can be transmitted to the big data asset management platform based on a timing transmission mechanism.

[0085] According to the embodiments of the present disclosure, the summarized metadata is transmitted to the big data asset management platform in a timing manner for display, so that system researchers and business personnel can intuitively understand and use the data lake metadata from different perspectives.

[0086] In summary, the data lake metadata processing method provided by the embodiments of the present disclosure configures the data lake metadata attributes, and defines the data assets as business metadata, technical metadata and operation metadata according to the attributes. Since the business metadata and the technical metadata are static data, and the operation metadata is dynamic data, the business metadata and the technical metadata can be collected according to the registered lake entry source table information and the element information associated with the data lake metadata in response to the lake entry request, and the actual operation information during the data lake entry can be automatically collected. Through this classified and timing collection of summarized metadata, the processing and management of the data lake metadata can be effectively realized, and the integration of the source data assets can be clarified, and the unified inventory of the source data assets can be effectively completed.

[0087] Figure 7 A block diagram of a data lake metadata processing apparatus according to an embodiment of the present disclosure is schematically shown.

[0088] As shown in Figure 7 The data lake metadata processing apparatus 700 can include a configuration module 710, a first collection module 720 and a second collection module 730.

[0089] The configuration module 710 is configured to configure attribute information of the data lake metadata, and define the data lake metadata as business metadata, technical metadata and operation metadata according to the attribute information.

[0090] The first collection module 720 is configured to collect the business metadata and the technical metadata according to the registered lake entry source table information and the element information associated with the data lake metadata in response to a lake entry request of the data lake metadata.

[0091] The second collection module 730 is configured to collect actual operation information during the lake entry of the data lake metadata in response to the lake entry of the data lake metadata, and obtain operation metadata.

[0092] Figure 8 A block diagram of a data lake metadata processing apparatus according to another embodiment of the present disclosure is schematically shown.

[0093] As Figure 8 shown, the data lake metadata processing apparatus 700 may, for example, further include a transmission module 740.

[0094] The transmission module 740 is configured to transmit the business metadata, the technical metadata and the operation metadata to a big data asset management platform, and display the business metadata, the technical metadata and the operation metadata.

[0095] Figure 9 A block diagram of a data lake metadata processing apparatus according to another embodiment of the present disclosure is schematically shown.

[0096] As Figure 9 shown, the data lake metadata processing apparatus 700 may, for example, further include an index module 750.

[0097] The index module 750 is configured to configure the data asset number as a unique index identifier of the business metadata, the technical metadata and the operation metadata, and retrieve and query the business metadata, the technical metadata and the operation metadata according to the unique index identifier.

[0098] Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure, or at least part of functions of any one or more of the modules, sub-modules, units, sub-units, can be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be at least partially implemented as a hardware circuit, for example, a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware through integration or packaging of circuits, or in any one of software, hardware and firmware or in an appropriate combination of any of the above. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be at least partially implemented as computer program modules, which can perform corresponding functions when the computer program modules are run.

[0099] For example, any of the configuration module 710, the first and second collection modules 720 and 730, the transmission module 740, and the index module 750 can be combined in one module / unit / sub-unit, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the function of one or more of the modules / units / sub-units can be combined with at least part of the function of other modules / units / sub-units, and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the configuration module 710, the first and second collection modules 720 and 730, the transmission module 740, and the index module 750 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware by integrating or packaging the circuit, or any one of software, hardware, and firmware or a proper combination of any of them. Alternatively, at least one of the configuration module 710, the first and second collection modules 720 and 730, the transmission module 740, and the index module 750 can be at least partially implemented as a computer program module that can perform the corresponding function when the computer program module is run.

[0100] It should be noted that the data lake metadata processing apparatus part in the embodiments of the present disclosure corresponds to the data lake metadata processing method part in the embodiments of the present disclosure, and the specific implementation details and the resulting technical effects are the same, which will not be repeated here.

[0101] Figure 10 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0102] As Figure 10As shown, the electronic device 1000 according to embodiments of the present disclosure includes a processor 1001 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage section 1008 into a random access memory (RAM) 1003. The processor 1001 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 1001 can also include an on-board memory for cache use. The processor 1001 can include a single processing unit or multiple processing units to perform the various actions of the method processes according to embodiments of the present disclosure.

[0103] In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The processor 1001 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 1002 and / or the RAM 1003. Note that the programs can also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0104] According to embodiments of the present disclosure, the electronic device 1000 can also include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 can also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1010 as necessary, so that a computer program read out therefrom is installed into the storage section 1008 as necessary.

[0105] According to an embodiment of the present disclosure, the method flow according to the embodiments of the present disclosure can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product comprising a computer program that is carried by a computer-readable storage medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 1009, and / or installed from the detachable medium 1011. When the computer program is executed by the processor 1001, the above-described functions defined in the system implementing the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0106] The present disclosure also provides a computer-readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, which when executed, implement the methods according to the embodiments of the present disclosure.

[0107] According to an embodiment of the present disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. For example, it can include but is not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0108] For example, according to an embodiment of the present disclosure, the computer-readable storage medium can include one or more memories other than the ROM 1002 and / or the RAM 1003 and / or the ROM 1002 and the RAM 1003 described above.

[0109] The computer program product of the second aspect of the disclosure can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to implement a method as described above; and instructions for causing a computer to operate based on a system as described above.

Claims

1. A data lake metadata processing method, characterized in that, The method comprises the following steps: configuring attribute information of the data lake metadata, and defining the data lake metadata as business metadata, technical metadata and running metadata according to the attribute information; in response to an entry request of the data lake metadata, checking whether the data lake metadata is allowed to enter the lake according to registered entry source table information and element information associated with the data lake metadata, and in response to allowing the data lake metadata to enter the lake, collecting the business metadata and the technical metadata; in response to the data lake metadata entering the lake, collecting actual running information when the data lake metadata enters the lake to obtain the running metadata; in response to allowing the data lake metadata to enter the lake, generating entry configuration information according to the registered entry source table information and the element information associated with the data lake metadata; and in response to the data lake metadata entering the lake, reading the data lake metadata transmitted by a business system according to the entry configuration information and entering the data lake metadata into the lake; configuring a data asset number as a unique index identifier of the business metadata, the technical metadata and the running metadata; retrieving and querying the business metadata, the technical metadata and the running metadata according to the unique index identifier; wherein the business metadata at least includes business description, business field, business process, asset source system, data review department, system table name, data supplier, domestic and foreign sign and data sharing strategy; the technical metadata at least includes data table name, primary key information, field name, field type, field length, field precision, field dictionary, data standard and data loading algorithm; and the running metadata at least includes latest data date, data loading frequency, data table capacity, existing earliest data time, recent data loading time, storage period and data quality inspection problem.

2. The data lake metadata processing method of claim 1, wherein, The data source application is called to register the entry source table information and the element information associated with the data lake metadata.

3. The data lake metadata processing method of claim 1, wherein, The processing method further comprises: transmitting the business metadata, the technical metadata and the running metadata to a big data asset management platform to display the business metadata, the technical metadata and the running metadata.

4. The data lake metadata processing method of claim 3, wherein, transmitting the business metadata, the technical metadata and the running metadata to the big data asset management platform based on a timing transmission mechanism.

5. A data lake metadata processing apparatus, characterized by, The method comprises the following steps: a configuration module configured to configure attribute information of the data lake metadata, and define the data lake metadata as business metadata, technical metadata and running metadata according to the attribute information; a first collection module configured to, in response to an entry request of the data lake metadata, check whether the data lake metadata is allowed to enter the lake according to registered entry source table information and element information associated with the data lake metadata, and in response to allowing the data lake metadata to enter the lake, collect the business metadata and the technical metadata; a second collection module configured to, in response to the data lake metadata entering the lake, collect actual running information when the data lake metadata enters the lake to obtain the running metadata; An indexing module is configured to number the data asset as a unique index identification of business metadata, technical metadata and operational metadata, and to retrieve and query the business metadata, the technical metadata and the operational metadata according to the unique index identification. The business metadata at least includes business description, business field, business process, asset source system, data review department, system table name, data provider, domestic and foreign sign and data sharing strategy; the technical metadata at least includes data table name, primary key information, field name, field type, field length, field precision, field dictionary, data standard and data loading algorithm; and the operational metadata at least includes latest data date, data loading frequency, data table capacity, existing earliest data time, latest data loading time, storage period and data quality check problem.

6. An electronic device, comprising: Comprise: one or more processors; a storage device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores executable instructions, which when executed by the processor, cause the processor to perform the method according to any one of claims 1-4.

8. A computer program product, characterised in that, Comprise a computer program, which when executed by the processor, implements the method according to any one of claims 1-4.

Images