Metadata processing method, electronic device, storage medium and program product

Abstract

The invention provides a metadata processing method, electronic equipment, a storage medium and a program product. The metadata processing method disclosed by the invention comprises the following steps: acquiring first metadata from a metadata generator; obtaining a conversion mode of the first metadata according to the requirement of the target rule; and converting the first metadata according to the conversion mode to obtain second metadata.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as data processing, and more particularly to a metadata processing method, an electronic device, a storage medium, and a program product. Background Art

[0002] Metadata is data about data, which provides detailed information about the structure, source, content and use of data. As the data scale of enterprises and institutions continues to expand and data sources and data types gradually diversify, the importance of metadata in data management and data governance is increasing day by day.

[0003] Modern enterprises have accumulated a large amount of structured, semi-structured and unstructured data in data warehouses, data lakes, big data platforms and other systems. Since data is distributed in multiple systems and each system has its own data specifications and metadata descriptions, metadata is scattered in various systems. Metadata in different systems often have significant differences. These differences may be reflected in metadata structure, naming specifications, field types, semantic understanding, and data storage formats.

[0004] Since metadata in different systems may differ significantly, the quality of metadata is poor, making it difficult to integrate and share metadata, affecting the efficiency and accuracy of metadata analysis, and in turn having a negative impact on the accuracy of decision-making. Summary of the invention

[0005] The present disclosure provides a metadata processing method, an electronic device, a storage medium, and a program product.

[0006] According to one aspect of the present disclosure, a metadata processing method is provided, comprising: Obtain first metadata from a metadata generator; Acquire a conversion method of the first metadata according to a requirement of a target rule; The first metadata is converted according to the conversion method to obtain second metadata.

[0007] According to the metadata processing method of at least one embodiment of the present disclosure, the step of obtaining first metadata from a metadata generator includes: In response to the first access of the metadata generator, obtaining a full amount of metadata from the metadata generator as the first metadata; or, In response to a metadata management operation in the metadata generator, incremental metadata is obtained from the metadata generator as the first metadata.

[0008] According to the metadata processing method of at least one embodiment of the present disclosure, the step of obtaining first metadata from a metadata generator includes: receiving first metadata pushed by the metadata generator; or, First metadata is pulled from the metadata generator.

[0009] According to the metadata processing method of at least one embodiment of the present disclosure, the receiving the first metadata pushed by the metadata generator includes: acquiring the first metadata from a first Kafka message queue, where the messages in the first Kafka message queue are sent by the metadata generator after being captured by a time processing tool; Pulling the first metadata from the metadata generator includes: attempting to pull the first metadata from the metadata generator; if the pulling attempt fails, trying again according to a retry rule until an end condition is reached, and obtaining the first metadata from the second Kafka message queue.

[0010] According to at least one embodiment of the metadata processing method of the present disclosure, after obtaining the second metadata, the method further includes: Determining whether the second metadata is missing according to the target rule; When the second metadata is missing, the missing data is obtained.

[0011] According to the metadata processing method of at least one embodiment of the present disclosure, the conversion method of obtaining the first metadata according to the requirements of the target rule includes: Acquire the data to be converted in the first metadata according to the requirement of the target rule; Acquire data specifications related to the data to be converted from the target rules; The conversion method is obtained according to the relationship between the data to be converted and the data specification.

[0012] According to the metadata processing method of at least one embodiment of the present disclosure, the step of obtaining the conversion mode of the first metadata according to the requirement of the target rule includes: obtaining the data type of the first metadata; if the data type is single, obtaining the conversion mode of the first metadata according to the requirement of the target rule; Optionally, if the data type is not single, the obtaining the conversion method of the first metadata according to the requirement of the target rule includes: splitting the first metadata according to the data type to obtain at least one metadata subset; for any metadata subset in the at least one metadata subset, obtaining the conversion method of the metadata subset according to the target rule related to the metadata subset; Optionally, after obtaining the second metadata, the method further includes: obtaining metadata to be compared from the metadata obtained by the current conversion; obtaining original metadata from the metadata generator; obtaining reference metadata related to the target rule obtained by converting the original metadata; comparing the metadata to be compared with the reference metadata to obtain a comparison result; and when the comparison result is inconsistent, updating the metadata to be compared according to the comparison result; Optionally, after obtaining the second metadata, the method further includes: storing the second metadata in a metadata database; and acquiring data related to the data query indication from the metadata database in response to a data query indication.

[0013] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the metadata processing method of any embodiment of the present disclosure.

[0014] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the metadata processing method of any embodiment of the present disclosure is implemented.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the metadata processing method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 It is a schematic diagram of an application scenario of the metadata processing method according to an embodiment of the present disclosure.

[0018] Figure 2 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 1 .

[0019] Figure 3 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 2 .

[0020] Figure 4 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 3 .

[0021] Figure 5The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 4 .

[0022] Figure 6 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 5 .

[0023] Figure 7 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 6 .

[0024] Figure 8 yes Figure 1 The process of the metadata acquisition method in the metadata processing method shown Figure 1 .

[0025] Fig. 9 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 7 .

[0026] Fig.10 yes Figure 1 The process of obtaining the method in the metadata processing method shown Figure 1 .

[0027] Fig.11 yes Figure 1 The process of obtaining the method in the metadata processing method shown Figure 2 .

[0028] Fig.12 yes Figure 1 The process of obtaining the method in the metadata processing method shown Figure 3 .

[0029] Fig.13 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 8 .

[0030] Fig.14 The process of the metadata processing method of one embodiment of the present disclosure is as follows Figure 9 .

[0031] Fig.15 is a schematic flowchart of a metadata processing method according to yet another embodiment of the present disclosure.

[0032] Fig.16 It is a schematic block diagram of the structure of a metadata processing device according to another embodiment of the present disclosure.

[0033] Fig.17 It is a schematic block diagram of the structure of an electronic device equipped with a metadata processing device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0035] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] Assume that a large retail company has multiple business departments, including online sales department, offline store department, supply chain management department, etc. Each department has its own business system to record and manage its own business data. For example, the online sales department uses JSON format to record product information, with fields including product_id, name, price, and category; the offline store department uses CSV format to record product information, with fields including prod_id, product_name, cost, and type; the supply chain management department uses XML format to record product information, with fields including item_id, item_name, unit_price, and classification. Since different business systems may use different field names to identify the same information, such as product_id, prod_id, and item_id may all represent the unique identifier of the product, this inconsistency increases the complexity of metadata integration; and in order to perform metadata analysis, a lot of time is often spent on metadata cleaning and preprocessing, which reduces the efficiency of metadata analysis, and these processes may also introduce errors, affecting the accuracy of the analysis results. In addition, if the integrated metadata is incomplete or erroneous, the metadata referenced during decision-making is not accurate enough, which leads to misjudgment of customer needs or market trends.

[0037] To this end, the present disclosure proposes a metadata processing method, electronic device, storage medium and program product. The present disclosure can be implemented by metadata processing software configured on electronic devices such as computers, tablet computers, mobile phones, servers, etc.

[0038] Figure 1The application scenario of the metadata processing method of an embodiment of the present disclosure is shown. In this application scenario, a business system 100 and a metadata processing system 200 may be included. The business system 100 is used to generate different types of metadata in daily operations. The business system 100 may include multiple types of metadata generators, such as: order management devices, customer relationship management devices, inventory management devices, financial management devices, production management devices, human resource management devices, etc. After the metadata generators of different types in the business system 100 generate metadata, they can be uniformly processed by the metadata processing system 200. In order to ensure that the metadata generated by the business system 100 can be safely and effectively used by the metadata processing system 200, while reducing the direct impact on the main library of the business system 100, a slave library and / or export component exp can also be set in the business system 100 or the metadata processing system 200; in particular, in order to further reduce the impact on the business system 100 and the metadata processing system 200 and improve security, the slave library and / or export component exp can also be designed as an independent system between the business system 100 and the metadata processing system 200.

[0039] For the convenience of description and to make the technical solutions of the specific implementation methods of the present disclosure easier to understand, before describing the metadata processing method implemented by the present disclosure, the technical terms involved in the specific implementation methods of the present disclosure are explained as follows: Metadata is data about data, that is, data that describes information such as data characteristics, content, structure, source, and usage.

[0040] A metadata generator is a system or module that generates metadata. It is usually a module or tool in a business system that can generate relevant metadata when data is created, updated, or processed.

[0041] Figure 2 FIG. 2 shows a schematic diagram of the overall process of a metadata processing method M100 according to an embodiment of the present disclosure. Figure 2 The method shown includes steps S110 to S130. The method can be executed by electronic devices such as mobile phones, tablet computers, and servers.

[0042] Specifically, Figure 2 The methods shown include: Step S110: Acquire first metadata from a metadata generator.

[0043] In some embodiments of the present disclosure, the first metadata obtained through step S110 may include system metadata (used to describe the structure, configuration, operation, performance and other information of the system itself, which may include system architecture, hardware configuration, software version, database table structure, configuration parameters, runtime logs, etc.), lineage metadata (used to describe the flow, source and conversion process of data, and track the data flow and transformation path from the original source to the final target, which may include data source, data target, data processing process, data conversion, data dependency, etc.), business metadata (used to describe the meaning and use of data in the business field and its relationship with business processes, including business definition of data, data dictionary, business rules, data classification, data relationship, business value, etc.) and other different categories of metadata. One or more.

[0044] The metadata generator in step S110 can be any device that can automatically generate and manage metadata, such as a business device, a data management device, etc.; wherein the business device can be an enterprise resource planning device, a customer relationship management device, an e-commerce device, a financial management device, a supply chain management device, a human resource management device, a content management device, a project management device, etc.

[0045] The first metadata obtained in step S110 may be full metadata at the time of initial access or synchronization, i.e., a complete set of all current metadata information, or may be incremental metadata when data changes, i.e., metadata information that has changed since the last synchronization.

[0046] Step S120: acquiring a conversion method of the first metadata according to a requirement of the target rule.

[0047] In some embodiments of the present disclosure, the target rules in step S120 may be pre-set, and the number of the pre-set target rules may be one or more. The target rules may be rules for defining, organizing and describing the structure of metadata; definition is a description of the basic attributes, characteristics and rules of metadata, clarifying the meaning and function of each element, and its purpose is to allow users or systems to understand the specific meaning of each metadata item; organization refers to how metadata is arranged and stored in a structured manner, and the organization process involves how to classify, group and hierarchize metadata elements for effective access and management; description is a detailed description of metadata, providing more information to help users understand the purpose and background of metadata. The description part is usually more detailed than the definition, involving context, usage scenarios, business needs and other content.

[0048] Among them, definition is the basis of the target rule and is mandatory, while organization and description are optional. In simple application scenarios, the target rule can only include definition. In complex application scenarios, in order to ensure the comprehensiveness and availability of metadata, the target rule usually includes definition, organization and description.

[0049] When different types of metadata focus on the same data object, table or field, step S120 can use the same target rule to obtain the conversion method, and each metadata type focuses on different parts or dimensions in the target rule. For example, a target rule can be designed to manage the data in the database table, and the target rule can include the following elements: table metadata (including table name, table description, creation time, modification time, owner, etc.), field metadata (including field name, data type, field constraint, default value, length, nullable, etc.), data source metadata (including data source name, data lineage, data update frequency, etc.). Technical metadata focuses on the technical details of data tables and fields, that is, technical metadata focuses on dimensions such as data types, indexes, constraints, etc. in the target rules; business metadata focuses on understanding the business meaning and purpose of the field, that is, business metadata focuses on dimensions such as the function and business classification of the field in the target rule; operational metadata focuses on dimensions such as life cycle management, change history, and access operations of the target rule, which are used for auditing and change tracking.

[0050] When there are large differences between metadata types, different target rules are usually used for metadata management; for example, when the content of technical metadata and business metadata is very different, different target rules can be used to describe the storage and business meaning of the data respectively.

[0051] Target rules are the basis and specification for metadata conversion operations. The conversion method is the specific processing and operation based on the target rules, which may include field mapping, data type conversion, format conversion, etc. The execution of conversion rules depends on the definition of target rules.

[0052] Step S120 can dynamically derive the conversion method of the first metadata by gradually parsing the first metadata and the target rule.

[0053] Step S130: convert the first metadata according to the conversion method to obtain second metadata.

[0054] In some embodiments of the present disclosure, the process of converting the first metadata according to the conversion method through step S130 may include: converting the field value of the relevant metadata in the first metadata from the source data type to the target data type according to the data type conversion rule in the conversion method; adjusting the data format of the relevant metadata in the first metadata, such as the date format, the time format, etc. according to the format conversion rule in the conversion method; merging the relevant metadata in the first metadata or calculating the derived fields according to the business conversion rule in the conversion method, etc.

[0055] The metadata processing method provided by the present disclosure converts the metadata generated by the metadata generator based on the target rule. Since the target rule can provide standardized definitions and structures, the converted metadata can be represented in a unified manner, thereby improving the consistency, interoperability and sharing of metadata, and also helping to manage the life cycle of metadata, improve metadata quality, and maintain complex relationships. This method solves the problem in the prior art that metadata in different systems may have significant differences, so the quality of metadata is poor, which makes it difficult to integrate and share metadata, affects the efficiency and accuracy of metadata analysis, and thus has a negative impact on the accuracy of decision-making.

[0056] Regarding step S110, in some embodiments of the present disclosure, Figure 3 As shown, it can be replaced by the following steps: In response to the first access of the metadata generator, a full amount of metadata is obtained from the metadata generator as first metadata.

[0057] In some embodiments of the present disclosure, the first time a metadata generator is connected to metadata processing software is the first access of the metadata generator. Since there may be a lot of stock metadata in the metadata generator before the first access, in order to improve the integrity and consistency of the metadata, the full amount of metadata can be obtained as the first metadata through this step, thereby ensuring that the metadata processing software has a complete understanding of the initial state of the metadata generator.

[0058] Regarding step S110, in some embodiments of the present disclosure, Figure 4 As shown, it can also be replaced by the following steps: In response to the metadata management operation in the metadata generator, incremental metadata is obtained from the metadata generator as first metadata.

[0059] In some embodiments of the present disclosure, this step can be used to respond to metadata management operations in real time, periodically, or according to the amount of data. The metadata management operations in this step may include one or more of creation, update, and deletion of metadata.

[0060] This step can achieve incremental metadata synchronization, reduce unnecessary data transmission and processing, improve synchronization efficiency, reduce resource consumption, and improve the real-time and consistency of metadata processing.

[0061] Regarding step S110, in some embodiments of the present disclosure, Figure 5 As shown, it can also be replaced by the following steps: The first metadata pushed by the metadata generator is received.

[0062] In some implementations of the present disclosure, the metadata generator may push the first metadata to the metadata processing software by directly importing or the like, so that the metadata processing software receives the first metadata through this step.

[0063] This step achieves data acquisition by pushing data, which can ensure the real-time and freshness of data and is suitable for one-time data migration or full data synchronization.

[0064] Regarding step S110, in some embodiments of the present disclosure, Figure 6 As shown, it can also be replaced by the following steps: Pull the first metadata from the metadata generator.

[0065] In some embodiments of the present disclosure, this step may pull the first metadata from the metadata generator by polling, API request, etc.; in particular, in order to reduce the pressure on the main library of the metadata generator and improve performance and availability, the metadata generator may also push the first metadata to the slave library, and the metadata processing software may pull the first metadata from the slave library.

[0066] This step achieves data acquisition by pulling data, so that the metadata processing software can independently decide when, where and how to pull metadata, which is more flexible and less intrusive to the metadata generator. The metadata processing software can also set a retry mechanism, so that even if the metadata generator is temporarily unavailable, the data can be pulled again later. In addition, the metadata processing software can also use secure authentication and encryption mechanisms to ensure the security of data transmission.

[0067] Specifically, the first access detection can be used to determine whether the metadata generator is accessing the metadata processing software for the first time. When a metadata generator accesses the metadata processing software for the first time, the metadata generator can actively push the current full metadata to the metadata processing software, and the metadata processing software can also actively pull the current full metadata from the library. The full metadata may include system metadata and / or lineage metadata.

[0068] When the metadata of a metadata generator changes after the first connection, such as addition, deletion, and modification, the metadata can be obtained through the Event Processing eXtension (EPX) tool. Specifically, the EPX tool can be pre-configured to connect to the detailed information of the source database in the metadata generator, including the database address, port, user name, password, etc.; the parameters for the EPX tool to connect to the Kafka message queue can also be configured, including the server address, port, topic name, partition configuration, etc. of Kafka; the offset of the EPX tool can also be specified to start listening to the log file binlog, usually continuing from the last listening point to avoid repeated processing of the changed data that has been sent. Listening through the EPX tool or importing from the library can reduce the intrusion into the metadata generator, thereby improving the maintainability and flexibility of the system.

[0069] At this time, if Figure 7 As shown, receiving the first metadata pushed by the metadata generator can specifically be obtaining the first metadata from the first Kafka message queue; wherein, the messages in the first Kafka message queue are captured and sent by the metadata generator through an event processing extension tool.

[0070] When the first metadata received through step S110 is the full first metadata, it can be directly converted through steps S120 to S130; when the first metadata received through step S110 is the incremental first metadata, the incremental first metadata can be first deserialized, filtered, converted, etc., and then the data that can be processed by steps S120 and S130 is generated and converted through steps S120 and S130.

[0071] In some embodiments of the present disclosure, Figure 8 As shown, step S110 may include steps S111 to S112.

[0072] Step S111 : Try to pull first metadata from a metadata generator.

[0073] In some embodiments of the present disclosure, when the attempt to pull data through step S111 is successful, step S120 is executed; when the attempt to pull data through step S111 fails, step S111 is executed again according to the retry rule until the end condition is met, and step S112 is executed.

[0074] The above retry process is generally a short retry for temporary errors such as network fluctuations, request timeouts, etc. In particular, in order to avoid resource consumption caused by frequent retries, a backoff strategy can be used when setting retry rules to gradually increase the retry interval after each attempt fails. The above end condition can be the maximum number of retries or the longest retry time, etc., to avoid excessive retries leading to resource waste.

[0075] The above attempt fails, which may be a failure to pull all the first metadata or a failure to pull part of the first metadata.

[0076] In particular, in order to improve the efficiency of problem diagnosis and / or resolution, when an attempt to pull data fails, the failed request parameters, error information, timestamp and other information can be recorded for subsequent analysis of the cause; and / or the failure-related information can be sent to an error tracking system for real-time monitoring and alerting.

[0077] Step S112: Acquire the first metadata from the second Kafka message queue.

[0078] In some embodiments of the present disclosure, when all the first metadata have not been obtained until the end condition is reached, the metadata processing software can notify the metadata generator to put the first metadata into the second Kafka message queue, so that the metadata processing software can obtain the first metadata from the second Kafka message queue. Figure 7 The first message queue in the message queue can be the same message queue or different message queues.

[0079] In particular, in order to improve efficiency and reduce redundancy, only the part of the data in the first metadata that failed to be pulled before can be pulled during the retry. At this time, the metadata processing software can notify the metadata generator to put this part of the data into the second Kafka message queue, so that the metadata processing software can obtain this part of the data from the second Kafka message queue.

[0080] Implementing fault tolerance through steps S111 to S112 can improve system stability and data consistency and integrity.

[0081] Furthermore, if Fig. 9 As shown, the metadata processing method provided by the present disclosure may further include steps S140 to S150 after step S130.

[0082] Step S140: determining whether the second metadata is missing according to the target rule.

[0083] In some embodiments of the present disclosure, the specific process of judging in step S140 may be to compare the format, type, length and other specifications defined by the target rule with the corresponding part of the second metadata to obtain a comparison result. The comparison result may include data consistency, missing second metadata, etc. When it is determined in step S140 that the second metadata is not missing, the metadata processing software may not perform subsequent operations; when it is determined in step S140 that the second metadata is missing, the metadata processing software executes step S150.

[0084] Step S150, obtaining missing data.

[0085] In some embodiments of the present disclosure, step S150 may specifically be to obtain the missing data from the relevant system. For example, the owner_ucid field is generally maintained in the User Center (UC) system. When the second metadata lacks the owner_ucid field and its data, the missing data can be obtained from the UC system through the owner_ucid field; the spaceId field is generally maintained in the permission system. When the second metadata lacks the spaceId field and its data, the missing data can be obtained from the permission system through the spaceId field.

[0086] Completing the missing data of the second metadata through steps S140 to S150 can improve data integrity, reduce deviations, improve analysis accuracy, and enhance the reliability of decision-making.

[0087] In some embodiments of the present disclosure, the conversion method in step S120 may include one or more of a format conversion method (such as converting a file in CSV format to a file in JSON format, converting a date format from MM / DD / YYYY to YYYY-MM-DD, etc.), a type conversion method (such as converting a small integer tinyint type to a large integer bigint type, converting a text type to a variable length string varchar type, etc., and the method can support multiple types such as integer int, large integer bigint, small integer tinyint, variable length string varchar, text text, date time datetime, etc.), a data aggregation method (such as summing a set of sales data to generate overall sales, summarizing a set of customer information by region to obtain the number of customers in each region, etc.) and a data segmentation method (such as segmenting sales data by year or month, segmenting customer data by geographic location, etc.).

[0088] Regarding step S120, Fig.10 As shown, the metadata processing method provided by the present disclosure may include steps S121 to S122.

[0089] Step S121: Acquire the data type of the first metadata.

[0090] In some embodiments of the present disclosure, the data type of the first metadata obtained through step S121 can be determined according to the function of the metadata, such as including system metadata, lineage metadata, etc.; it can also be determined according to the content of the metadata, such as including descriptive metadata, structural metadata, administrative metadata, etc.; it can also be determined according to the storage location of the metadata, such as including local metadata, global metadata, etc.

[0091] When the data type acquired in step S121 is single, step S122 is executed.

[0092] Step S122: acquiring a conversion method of the first metadata according to a requirement of the target rule.

[0093] The data type obtained through step S121 is single, and the conversion method is directly obtained through step S114, which can avoid complicated data processing operations, thereby simplifying the data processing flow and improving processing efficiency.

[0094] At this time, if Fig.11 As shown, when the data type obtained through step S121 is not single, step S120 in the metadata processing method provided by the present disclosure may also include steps S123 to S124.

[0095] Step S123: split the first metadata according to the data type to obtain at least one metadata subset.

[0096] In some embodiments of the present disclosure, step S123 may extract data from the first metadata into different data sets according to different data types to obtain at least one metadata subset; each metadata subset may include one or more metadata.

[0097] Step S124: for any metadata subset in the at least one metadata subset, a conversion method of the metadata subset is obtained according to a target rule related to the metadata subset.

[0098] Splitting the first metadata according to data types through steps S123 to S124 helps to simplify the processing process, apply appropriate conversion methods to different data types, and improve the efficiency and accuracy of metadata processing.

[0099] Regarding step S120, Fig.12 As shown, the metadata processing method provided by the present disclosure may also include steps S125 to S127.

[0100] Step S125 , obtaining the data to be converted in the first metadata according to the requirement of the target rule.

[0101] Step S126, obtaining data specifications related to the data to be converted from the target rules.

[0102] Step S127, obtaining a conversion method according to the relationship between the data to be converted and the data specification.

[0103] Steps S125 to S127 determine the conversion method based on the fields of the first metadata and the fields of the target rule, which can improve the accuracy, consistency and reliability of data conversion. This process has no restrictions on the fields of the first metadata and can also improve the flexibility and maintainability of the system.

[0104] Furthermore, if Fig.13 As shown, the metadata processing method provided by the present disclosure may further include steps S160 to S200 after step S130.

[0105] Step S160: obtaining metadata to be compared from the metadata currently converted.

[0106] In some embodiments of the present disclosure, the metadata to be compared obtained by step S160 may be all metadata currently converted; in particular, in order to improve data comparison efficiency, the metadata to be compared obtained by step S160 may also be metadata of changes between the previous comparison and the current comparison among all metadata currently converted.

[0107] Step S170: obtaining original metadata from the metadata generator.

[0108] In some embodiments of the present disclosure, the original metadata acquired through step S170 may be all metadata generated in the metadata generator; in particular, in order to improve acquisition efficiency, the original metadata may also be related to the metadata to be compared. For example, if the metadata to be compared is metadata changed between A year B month C day and A year D month E day, then step S170 may also be used to acquire the original metadata within this period from the metadata generator.

[0109] Step S180: obtaining reference metadata related to the target rule obtained by converting the original metadata.

[0110] In some implementations of the present disclosure, the original metadata may be converted to obtain reference metadata through step S180; the conversion process is similar to step S120 and step S130.

[0111] Step S190: Compare the metadata to be compared with the reference metadata to obtain a comparison result.

[0112] In some embodiments of the present disclosure, step S190 can compare the corresponding fields in the metadata to be compared and the benchmark metadata one by one, and compare the field values; the comparison result obtained by step S190 can be data consistency or data inconsistency, where data inconsistency includes newly added records (i.e., records that exist in the metadata to be compared but are missing in the benchmark metadata), missing records (i.e., records that exist in the benchmark metadata but are missing in the metadata to be compared), field value difference records (i.e., records with inconsistent field values ​​under the same field), structure difference records (i.e., records with inconsistent fields or structures), etc.

[0113] When the comparison result obtained in step S190 is that the data is consistent, it means that the quality of the metadata to be compared is high and no operation is required; when the comparison result obtained in step S190 is inconsistent, step S200 is executed.

[0114] Step S200: updating the metadata to be compared according to the comparison result.

[0115] In some embodiments of the present disclosure, for newly added difference records, the corresponding data can be directly deleted from the metadata to be compared through step S200; for missing difference records, the missing records or fields can be directly supplemented through step S200; for field value difference records or structure difference records, the corresponding parts in the metadata to be compared can be corrected based on the baseline metadata through step S200.

[0116] By performing metadata comparison and updating from step S160 to step S200, metadata consistency verification can be achieved, metadata loss or duplication can be avoided, and the accuracy and completeness of the metadata can be improved.

[0117] Furthermore, if Fig.14 As shown, the metadata processing method provided by the present disclosure may further include steps S210 to S220 after step S130.

[0118] Step S210: storing the second metadata into a metadata database.

[0119] In some embodiments of the present disclosure, step S210 can store the second metadata into the metadata database through SQL statements, and different types of metadata can be stored at the same time; in particular, in order to improve query efficiency, the metadata database can be divided into multiple areas, and different types of metadata can be classified and stored in different areas of the metadata database.

[0120] Furthermore, when the amount of data is large, in order to improve writing efficiency, step S210 may use multiple threads to execute the writing logic, and write to multiple areas in the metadata repository at the same time or write to multiple metadata repository at the same time.

[0121] Step S220: in response to the data query instruction, obtaining data related to the data query instruction from the metadata database.

[0122] In some implementations of the present disclosure, in order to efficiently and conveniently use the converted metadata, the metadata repository may perform metadata access control through a unified metadata access interface.

[0123] By storing the converted metadata in the metadata database and implementing query through steps S210 to S220, query efficiency can be significantly improved, data management capabilities can be enhanced, and flexible access control can be achieved.

[0124] The metadata processing method provided by the present disclosure can convert heterogeneous data generated by various metadata generators using corresponding conversion methods, thereby collecting metadata generated by various metadata generators together.

[0125] When a new metadata generator is connected, the existing metadata maintained by this metadata generator can be imported into the metadata processing software by full import; when the metadata in this metadata generator changes, it can be imported into the metadata processing software by incremental import, that is, this method is compatible with full import and incremental import modes. Both full import and incremental import use unified target rule conversion, which solves the problem of incompatibility of multiple types of metadata, thereby improving the consistency and accuracy of metadata. In addition, through asynchronous processing of metadata such as EPX collection, API collection, and Kafka message queues, the load can be automatically smoothed during peak processing periods, avoiding resource overload and performance bottlenecks, and ensuring stable system operation.

[0126] Fig.15 An exemplary flowchart of the metadata processing method according to the present disclosure is shown.

[0127] Fig.15 In the flowchart shown, taking the metadata generator as an order management device as an example, the metadata processing process may include: 1. When the order management device is connected for the first time, a full amount of metadata is obtained from the order management device as the first metadata.

[0128] In some embodiments of the present disclosure, the full metadata obtained through this step generally covers all basic information related to order management. The full metadata may include table structure definition (such as table name, field information, primary key and foreign key, etc.), field description and business meaning (such as field description, field unit and constraint, etc.), data dictionary (such as code and enumeration value, general business rules, etc.), data relationship and blood relationship, etc.

[0129] Take the first metadata in JSON format as an example, with the following content: { "name": "orders", "type": "transactional", "ownerucid": "12345", "owner": "John Doe", "spaceid": "1001", "data field": [ { "name": "order_id", "type": "int"}, { "name": "customer_id", "type": "int"}, { "name": "is_active", "type": "boolean"}, { "name": "order_date", "type": "datetime", "value": "2023-10-01T00:00:00Z"}, { "name": "total_amount", "type": "float", "value": 123.45} ] } 2. Obtain a conversion method for the first metadata according to requirements of the target rule.

[0130] In some embodiments of the present disclosure, taking the first metadata as an example, the target rule may include name, type, ownerucid, owner, spaceid, and property (data field, such as different types of metadata fields such as digital type metadata field and string type metadata field). The conversion method obtained through this step may include format conversion, type conversion, etc.

[0131] 3. Convert the first metadata according to the conversion method to obtain the second metadata.

[0132] In some embodiments of the present disclosure, the second metadata obtained through this step may be as follows: { "name": "orders", "type": "transactional", "ownerucid": "12345", "owner": "John Doe", "spaceid": "1001", "property": [ { "name": "order_id", "type": "int"}, { "name": "customer_id", "type": "int"}, { "name": "is_active", "type": "boolean"}, { "name": "order_date", "type": "datetime", "value": "2023-10-01T00:00:00Z"}, { "name": "total_amount", "type": "float", "value": 123.45} ] } 4. In response to the management operation of the metadata in the metadata generator, the incremental metadata is obtained from the order management device as the first metadata.

[0133] In some implementations of the present disclosure, after the incremental metadata is obtained as the first metadata through this step, the incremental metadata can be converted through steps two and three.

[0134] By converting the metadata generated by various metadata generators through the above steps, it can be compatible with the ingestion scenarios of batch metadata and real-time metadata, and improve the consistency and effectiveness of metadata processing.

[0135] Based on any of the above implementations, the present disclosure also provides a metadata processing device.

[0136] Fig.16 It is a schematic block diagram of the structure of a metadata processing device according to an embodiment of the present disclosure.

[0137] like Fig.16 As shown, the metadata processing device includes: The metadata acquisition module 110 is used to acquire first metadata from a metadata generator.

[0138] The conversion mode module 120 is used to obtain a conversion mode of the first metadata according to the requirement of the target rule.

[0139] The data conversion module 130 is used to convert the first metadata according to a conversion method to obtain second metadata.

[0140] Each of the above modules can be implemented by a computer program module. Fig.16 The metadata processing device shown can be implemented based on a computer program module architecture. The metadata processing device converts heterogeneous metadata generated by different metadata generators into universal metadata related to target rules through a data conversion module. The device can support multiple types of metadata access, realize universal metadata collection, and further realize multi-source fusion and flexible conversion.

[0141] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0142] The execution subject of the metadata processing method in the specific implementation of the present disclosure can be electronic devices such as mobile phones, tablet computers, servers, etc. Among them, the server is suitable for processing large-scale, complex or high-security metadata conversion tasks; mobile phones and tablet computers are suitable for real-time interaction, offline processing and lightweight metadata conversion tasks.

[0143] Therefore, based on any of the above embodiments, the present disclosure further provides an electronic device, which can execute the metadata processing method of any of the above embodiments of the present disclosure, and the metadata processing device of any of the above embodiments can be configured on the electronic device.

[0144] Fig.17 1 is a schematic block diagram of the structure of an electronic device 1000 equipped with a metadata processing device according to an embodiment of the present disclosure.

[0145] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0146] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0147] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0148] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0149] The computer program or instructions may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0150] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0154] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0155] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0156] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A metadata processing method, characterized in that: include: Obtain first metadata from a metadata generator; Acquire a conversion method of the first metadata according to a requirement of a target rule; as well as The first metadata is converted according to the conversion method to obtain second metadata.

2. The metadata processing method according to claim 1, characterized in that: The step of obtaining first metadata from a metadata generator includes: In response to the first access of the metadata generator, obtaining a full amount of metadata from the metadata generator as the first metadata; or, In response to a management operation on metadata in the metadata generator, incremental metadata is obtained from the metadata generator as the first metadata.

3. The metadata processing method according to claim 1, characterized in that: The step of obtaining first metadata from a metadata generator includes: receiving first metadata pushed by the metadata generator; or, First metadata is pulled from the metadata generator.

4. The metadata processing method according to claim 3, characterized in that: The receiving the first metadata pushed by the metadata generator includes: acquiring the first metadata from a first Kafka message queue, where the messages in the first Kafka message queue are sent after being captured by the metadata generator through a time processing tool; Pulling the first metadata from the metadata generator includes: attempting to pull the first metadata from the metadata generator; if the pulling attempt fails, trying again according to a retry rule until an end condition is reached, and obtaining the first metadata from the second Kafka message queue.

5. The metadata processing method according to any one of claims 1 to 4, characterized in that: After obtaining the second metadata, the method further includes: Determining whether the second metadata is missing according to the target rule; and When the second metadata is missing, the missing data is obtained.

6. The metadata processing method according to any one of claims 1 to 4, characterized in that: The conversion method of obtaining the first metadata according to the requirement of the target rule includes: Acquire the data to be converted in the first metadata according to the requirement of the target rule; Acquiring data specifications related to the data to be converted from the target rules; and The conversion method is obtained according to the relationship between the data to be converted and the data specification.

7. The metadata processing method according to any one of claims 1 to 4, characterized in that: The obtaining of the conversion method of the first metadata according to the requirement of the target rule includes: obtaining the data type of the first metadata; if the data type is single, obtaining the conversion method of the first metadata according to the requirement of the target rule; Optionally, if the data type is not single, the obtaining the conversion method of the first metadata according to the requirement of the target rule includes: splitting the first metadata according to the data type to obtain at least one metadata subset; for any metadata subset in the at least one metadata subset, obtaining the conversion method of the metadata subset according to the target rule related to the metadata subset; Optionally, after obtaining the second metadata, the method further includes: obtaining metadata to be compared from the metadata obtained by the current conversion; obtaining original metadata from the metadata generator; obtaining reference metadata related to the target rule obtained by converting the original metadata; comparing the metadata to be compared with the reference metadata to obtain a comparison result; and when the comparison result is inconsistent, updating the metadata to be compared according to the comparison result; Optionally, after obtaining the second metadata, the method further includes: storing the second metadata in a metadata database; and acquiring data related to the data query indication from the metadata database in response to a data query indication.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the metadata processing method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the metadata processing method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the metadata processing method according to any one of claims 1 to 7 is implemented.

Images