Master data automatic acquisition method based on metadata

Abstract

The invention provides an automatic master data collection method based on metadata, and relates to the technical field of data management, and the method comprises the following steps: based on metadata of various data sources, matching the metadata through a matching algorithm to obtain a corresponding relation of the metadata of different data sources; based on the corresponding relationship, establishing a standard model, and collecting the main data of the multiple data sources through the standard model in combination with a preset collection rule; according to the requirements of the business process, the operation specification and the data timeliness, the collection period and frequency of the timed task for collecting the main data are determined, and the collection period depends on the change speed of the data and the requirement of the business for the data freshness.

Description

Technical Field

[0001] The present application belongs to the technical field of data management, and specifically relates to a method for automatically collecting master data based on metadata. Background Art

[0002] With the rapid development of information technology, the scale of enterprise data is growing exponentially, and the management of master data (such as core business data such as customers, products, and suppliers) has gradually become a key support for enterprise operations and decision-making. The accuracy, consistency, and real-time nature of master data directly affect business efficiency and decision-making quality. In this context, the management of metadata (information that describes data attributes, structure, and relationships) has become the core link of master data governance. However, traditional master data management methods have significant defects in technical implementation and practical application, and it is difficult to meet the needs of modern enterprises for efficient and intelligent data management.

[0003] At present, master data management mainly relies on the following technical means: manually extract metadata from databases, API interfaces or files, and establish master data mapping rules. Predefine a unified data model to force each system to adapt to standard fields, but it requires a lot of manual intervention. Use scripts or tools to regularly extract data from data sources, but lack dynamic adjustment capabilities. Metadata matching based on simple rules (such as field names are exactly the same) is difficult to handle complex scenarios with heterogeneous data sources.

[0004] Although the above technologies support master data management to a certain extent, the following outstanding problems still exist: The collection, matching and integration of metadata rely heavily on manual operations, which are costly and error-prone, and it is difficult to cope with the processing needs of massive data. The field naming rules and data formats of different systems vary greatly, and there is a lack of intelligent matching mechanism, which makes data integration complex and inconsistent. Traditional scheduled tasks cannot dynamically respond to data changes, and master data updates lag behind, affecting the timeliness of business decisions. In addition, master data needs to be shared by multiple systems as a "golden record". If uncleaned data is directly extracted from the original system, data redundancy and conflict will occur. Existing technologies have limited adaptability to unstructured data (such as API, JSON / XML files), making it difficult to achieve cross-platform data integration. The system architecture is rigid, and new data sources or business rules need to be redeveloped, with high operation and maintenance costs and difficulty in adapting to the dynamic development needs of enterprises.

[0005] The core of the above defects is that the existing technology lacks automated and intelligent metadata analysis and collection capabilities, especially in key links such as fuzzy matching, dynamic scheduling and multi-source adaptation. In addition, master data needs to be shared by multiple systems as "golden records". If the uncleaned data is directly extracted from the original system, it will lead to data redundancy and conflict. This has led to the long-term bottleneck of low efficiency, high cost and poor quality in enterprise master data management, and an innovative solution that can fully cover data collection, intelligent matching, real-time update and flexible expansion is urgently needed. Summary of the invention

[0006] This application provides a metadata-based master data automatic collection method to solve the technical problems that the core data of an enterprise is scattered in different systems, the manual collection cost is high, the efficiency is low, the data format is chaotic, the update is not timely, and the master data is directly extracted from the original system without cleaning data, which will lead to data redundancy and conflict.

[0007] The technical solution adopted in this application is: The present application embodiment provides a method for automatically collecting master data based on metadata, comprising: Based on metadata of multiple data sources, the metadata are matched by a matching algorithm to obtain a corresponding relationship between the metadata of different data sources; Based on the corresponding relationship, a standard model is established, and master data from the multiple data sources are collected through the standard model in combination with preset collection rules; The collection cycle and frequency of the scheduled task of collecting master data are determined according to the business process, operating specifications and data timeliness requirements. The collection cycle depends on the speed of data change and the business's demand for data freshness.

[0008] According to an embodiment of the present application, the metadata based on multiple data sources is matched by a matching algorithm to obtain a corresponding relationship between the metadata of different data sources, specifically: Determine the multiple data sources from which the metadata needs to be collected, the data sources including: a relational database, a big data platform, and an API interface; Collecting the metadata and performing preprocessing, the metadata includes: data table structure, field information, data type, data length and constraint conditions; Matching the metadata based on a fuzzy matching algorithm to identify similar or identical metadata items; Based on the metadata items, common attributes of different data sources are obtained.

[0009] According to an embodiment of the present application, collecting the metadata specifically includes: Automatically executing a scheduled task according to a preset schedule to collect the metadata from the determined data source; The metadata is collected according to actual business needs by manually triggering the collection task.

[0010] According to an embodiment of the present application, the fuzzy matching algorithm performs matching based on string similarity of field numbers and semantic similarity of field names.

[0011] According to an embodiment of the present application, a standard model is established based on the corresponding relationship, and master data of multiple data sources are collected through the standard model in combination with preset collection rules, specifically: Based on the common attributes, establishing a field matching relationship with the data source to form the standard model; According to preset collection rules, the master data of the multiple data sources are collected, and the collection rules include: specifying data tables, fields and filtering conditions; The collected master data is cleaned, converted and integrated.

[0012] According to an embodiment of the present application, after determining the collection period and frequency of the scheduled task for collecting master data according to the business process, operation specifications and data timeliness requirements, the method further includes: The matching results of the metadata and the collection status of the master data are displayed through a graphical interface, which supports users to customize the collection rules and the scheduled tasks, and provides data quality detection reports and abnormal alarm functions.

[0013] According to an embodiment of the present application, the abnormal alarm includes: triggering email or SMS notification for acquisition failure and data format error events.

[0014] A computer program product comprising instructions, when running on a device, enables the device to execute the steps in a method for automatically collecting master data based on metadata.

[0015] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a method for automatically collecting master data based on metadata.

[0016] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for automatically collecting master data based on metadata are implemented.

[0017] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows: This application introduces fuzzy matching algorithms (such as string similarity calculation based on Levenshtein distance) to automatically identify metadata fields with similar semantics but inconsistent names in different data sources (such as "customer ID" and "user number"), significantly reducing the workload of manual verification and correction. It supports multi-dimensional matching rules (such as field type, data length, constraints, etc.) to further improve matching accuracy and reduce mapping errors caused by field ambiguity.

[0018] This application builds a unified standard model based on the matching results, maps the metadata of heterogeneous data sources into a standardized format, eliminates data silos, and ensures the consistency of master data across systems. Automatically adapt to multiple data sources (such as relational databases, API interfaces, JSON / XML files, etc.) through standard models to reduce development adaptation costs caused by differences in data sources. Preset collection rules (such as scheduled tasks, triggered updates) can dynamically adjust the data collection frequency (such as minutes or hours) according to business needs to ensure that master data updates are synchronized with business changes.

[0019] This application supports multiple collection methods (such as direct database connection, API call, file parsing), adapts to the access restrictions and interaction protocols of different data sources, and improves system compatibility. The automated collection process reduces the need for manual intervention. Operation and maintenance personnel only need to make final confirmation of the matching results generated by the system, compressing the traditional manual configuration time of several days to minutes. The modular architecture design supports the rapid expansion of new data sources or the update of matching rules, avoiding system reconstruction due to changes in business needs, and significantly reducing long-term maintenance costs. Cleaning and conversion rules (such as deduplication, format verification, and missing value filling) are embedded in the data collection stage to ensure the integrity, accuracy and availability of master data. Through real-time or near real-time data update mechanisms, the latest business views are provided to enterprises, supporting rapid response to market changes and accurate decision-making. Support hybrid cloud, local deployment and edge computing environments to adapt to the diverse IT infrastructure needs of enterprises. The use of microservice architecture and open API interfaces facilitates integration with existing data middle platforms, BI tools or AI platforms, and reserves expansion space for future technology upgrades. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a method for automatically collecting master data based on metadata provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features in each embodiment may be combined with each other without conflict.

[0023] In the present application, unless otherwise clearly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0024] Metadata: Information that describes the properties of data ("data about data"), used to explain the structure, source, format, and relationships of data.

[0025] Core role: Provide technical support for data governance, establish mapping relationships between data, and realize the integration of heterogeneous data sources. Typical content: data table structure, field name, data type, data length, constraints, data lineage relationship. Processing method: Establish cross-system field mapping relationships through fuzzy matching algorithms (such as Levenshtein distance) to form a standard model. Dynamic characteristics: relatively static (updated only when the data source structure changes). System positioning: data governance infrastructure.

[0026] Master Data: The core business entity data of an enterprise (such as customers / products / suppliers), which has high business value and cross-system sharing characteristics.

[0027] Core role: Provide technical support for data governance, establish mapping relationships between data, and realize the integration of heterogeneous data sources. Typical content: data table structure, field name, data type, data length, constraints, data lineage relationship. Processing method: Establish cross-system field mapping relationships through fuzzy matching algorithms (such as Levenshtein distance) to form a standard model. Dynamic characteristics: relatively static (updated only when the data source structure changes). System positioning: data governance infrastructure.

[0028] Example 1 like Figure 1As shown, a method for automatically collecting master data based on metadata includes: S100: Based on metadata from multiple data sources, the metadata are matched by a matching algorithm to obtain a corresponding relationship between the metadata from different data sources.

[0029] Specifically, first, the system will crawl metadata from various types of data sources. The data sources mentioned here include but are not limited to: Relational database: such as MySQL, Oracle, etc.

[0030] Big data platform: such as HBase and Hive in the Hadoop ecosystem.

[0031] API interface: It can be an API provided by the company's internal services or an API for obtaining data from the outside.

[0032] During the collection process, the system will record detailed information such as data table structure, field information (name, type), data length, constraints, etc. This step is mainly to obtain a complete overview of each data source.

[0033] Application of matching algorithm: Next, the collected metadata is processed using a fuzzy matching algorithm. This process aims to find common features or similarities between metadata from different data sources. Fuzzy matching algorithms can be customized according to actual needs. Common methods include matching based on string similarity of field numbers and semantic similarity of field names.

[0034] For example, if there is a field named "CustomerID" in both data sources, even though their specific implementation details (such as data types) may be different, through appropriate matching algorithms, the system can recognize that these two fields actually represent the same concept, that is, they are both unique numbers used to identify customers.

[0035] Establishing correspondence: Once the similarities or consistencies between metadata items are determined, field matching relationships between data sources can be established based on these shared attributes. This means creating a set of standard models that define how to map information from different data sources together. For example, all fields representing customer identity can be uniformly mapped to a standard "customer ID" field, thus laying the foundation for subsequent data integration.

[0036] For example, let’s say an enterprise uses three different systems to manage its business data: ERP system: used for financial management.

[0037] CRM system: used for customer relationship management.

[0038] SCM system: used for supply chain management.

[0039] Each system has its own database, and for the same business entity (such as customer information), they may store the same information using different field names and formats. For example: In the ERP system, the customer ID is named Cust_ID and the name is Name.

[0040] In the CRM system, the same information is called CustomerNumber and FullName respectively.

[0041] In the SCM system, it is ClientCode and ClientName.

[0042] Metadata collection: First, the system will capture metadata from these three different data sources. This includes but is not limited to data table structure, field information (such as name, type), data length, constraints, etc. In this example, we will focus on fields related to customer information.

[0043] Application of matching algorithms: Next, the collected metadata is analyzed using a fuzzy matching algorithm. This algorithm is designed to identify similar or identical metadata items in different data sources. In this case, the algorithm will try to find all fields that represent the customer's unique identifier and the field that represents the customer's name.

[0044] For a customer ID, the algorithm might discover that Cust_ID, CustomerNumber, and ClientCode are actually different representations of the same concept. Similarly, for a customer name, Name, FullName, and ClientName are considered equivalent. These matches are based on multiple factors, such as string similarity (edit distance) and semantic similarity (what the field actually means).

[0045] Establishing correspondence: Once the similarities or consistencies between metadata items are determined, field matching relationships between data sources can be established based on these shared attributes. This means creating a standard model that defines how to map information from different data sources together.

[0046] For example, in this standard model, all customer ID fields will be uniformly mapped to a new standardized field CustomerID. Similarly, all fields representing customer names will be mapped to CustomerName.

[0047] After metadata collection, in addition to matching analysis, a dedicated data cleaning and standardization module can be introduced. This module is responsible for cleaning errors and redundant information in the original data and unifying the data format. De-duplicate records: Identify and delete duplicate customer or product records from different data sources. Format unification: Ensure that all date fields use the same format (such as YYYY-MM-DD), all amount fields use the same currency unit, etc. Correct errors: Automatically correct spelling errors, incomplete address information, etc.

[0048] Furthermore, a comprehensive data quality monitoring and evaluation system can be established to monitor the quality of data in real time, including completeness, accuracy, consistency, etc. Quality scoring: Assign a quality score to each data item to help users quickly understand the overall health of the data. Anomaly detection: Automatically discover data points that do not conform to expected patterns, such as abnormally high sales or illogical inventory quantities. Trend analysis: Provide trend reports on data quality changes over time so that corrective measures can be taken in a timely manner.

[0049] Furthermore, an automated rules engine can be developed to allow administrators to define complex business rules to guide data processing. Conditional triggering: Automatically perform specific actions based on preset conditions, such as automatically generating a purchase order when inventory falls below a certain threshold. Decision support: Generate recommendations based on historical data analysis to assist management in making more informed decisions. Flexible configuration: Supports dynamic adjustment of rule settings to adapt to new business needs without modifying the underlying code.

[0050] Furthermore, an efficient and reliable real-time synchronization and update mechanism can be implemented to ensure that the data between various data sources is always up to date. Instant feedback: When any data changes, the associated data sources are immediately notified for synchronization and update. Reduced latency: Significantly reduce the risk of decision-making errors caused by data lag. Transaction consistency: Ensure that operations across multiple data sources are either all successful or all rolled back to maintain transaction consistency.

[0051] Furthermore, a strong security and privacy protection framework can be built to ensure the secure storage and legal access of sensitive data. Encrypted transmission: Use advanced encryption technology to protect the security of data during transmission. Access control: Implement fine-grained permission management strategies to limit access rights of unauthorized users. Audit tracking: Record all access activities for post-event review and accountability.

[0052] S200: Based on the corresponding relationship, a standard model is established, and master data from the multiple data sources are collected through the standard model in combination with preset collection rules.

[0053] Specifically, after determining the correspondence between metadata of different data sources, the next step is to establish a standard model. This standard model is an abstract data structure or framework that defines how to unify data from different data sources into a common format. Specifically, it includes the following aspects: Field matching: Based on previously identified metadata correspondences (such as Cust_ID, CustomerNumber, and ClientCode all represent unique customer identifiers), establish field matching relationships with various data sources.

[0054] Attribute mapping: Based on these matching relationships, a standardized model is formed. For example, all information representing customer ID will be mapped to a new standardized field CustomerID, and other important information such as name, address, etc. will be handled in the same way.

[0055] Once you have a standard model, you need to set specific collection rules. These rules guide the system on how to accurately extract the required data from different data sources. Collection rules can include but are not limited to the following: Specify data tables: clearly indicate which tables need to be collected. For example, in an ERP system, you may only need to extract data from the Customers table.

[0056] Select specific fields: Not limited to the entire table, you can also refine to the specific field level. For example, only collect information on the CustomerID and CustomerName fields.

[0057] Set filter conditions: You can filter data based on certain conditions. For example, only collect customer records that were newly added in the past year.

[0058] Using the above-established standard model and preset collection rules, the system can automatically collect master data from multiple data sources. This step usually involves the following implementation methods: Direct database connection: For relational databases or graph databases, after obtaining the necessary connection permissions, you can directly use SQL queries or graph query languages ​​to obtain data.

[0059] API interface access: Considering the independence between heterogeneous data sources, data can be obtained by calling the open API interface provided by the data source.

[0060] XML / JSON file parsing: When the data source regularly generates queryable data files (such as XML or JSON format), the system can receive data by parsing these files. This method is suitable for situations where the data is relatively stable and does not need to be updated frequently.

[0061] The collected data often needs further processing to ensure its quality and consistency: Data cleaning: Removing erroneous, duplicate or incomplete records.

[0062] Data conversion: Convert data in different formats into the form required by the standard data model.

[0063] Data integration: Merge data from multiple sources together to ensure there are no conflicts and that it is logically consistent.

[0064] For example, let’s say an enterprise uses three different systems to manage its business data: ERP system: used for financial management.

[0065] CRM system: used for customer relationship management.

[0066] SCM system: used for supply chain management.

[0067] Each system has its own database, and for the same business entity (such as customer information), they may store the same information using different field names and formats. For example: In the ERP system, the customer ID is named Cust_ID and the name is Name.

[0068] In the CRM system, the same information is called CustomerNumber and FullName respectively.

[0069] In the SCM system, it is ClientCode and ClientName.

[0070] First, the system uses a matching algorithm to identify similar or identical metadata items in different data sources. For example, the system finds the following correspondence: Cust_ID(ERP) = CustomerNumber(CRM) = ClientCode(SCM) Name(ERP) = FullName(CRM) = ClientName(SCM) These matches are based on multiple factors, such as string similarity (edit distance) and semantic similarity (the actual meaning of the field).

[0071] Based on the above correspondence, the system creates a set of standard models that define how to map information from different data sources together. In this example, the standard model may be as follows: Standardized field CustomerID, corresponding to all fields representing a unique identifier for a customer.

[0072] The standardized field CustomerName corresponds to all fields representing customer names.

[0073] The purpose of doing this is to make subsequent data processing more unified and standardized.

[0074] Next, set specific collection rules based on business needs. For example: Only records of new customers added in the past month are extracted from the CRM system.

[0075] For the ERP system, only those customer records with a credit score above 70 are extracted.

[0076] For SCM systems, extract all customer records related to a specific supplier.

[0077] Using the above standard model and collection rules, the system starts to automatically collect master data from various data sources. The following is a specific operation example: ERP system: Query conditions: Credit score>70 Collection fields: Cust_ID, Name Data conversion: map Cust_ID to CustomerID and Name to CustomerName CRM system: Query conditions: New customers added in the last month Collection fields: CustomerNumber, FullName Data conversion: Map CustomerNumber to CustomerID and FullName to CustomerName SCM system: Search criteria: Customers related to a specific supplier Collection fields: ClientCode, ClientName Data conversion: Map ClientCode to CustomerID and ClientName to CustomerName Based on the collected data, the system performs further processing to ensure the quality and consistency of the data: Data cleaning: remove duplicate records and correct erroneous data.

[0078] Data conversion: Format all data according to the requirements of the standard model. For example, the date format is unified into YYYY-MM-DD.

[0079] Data integration: Merge customer information from different data sources into a comprehensive view, ensuring there are no conflicts and it is logically consistent.

[0080] Furthermore, adaptive frequency adjustment can be introduced: the collection frequency is automatically adjusted according to the speed of data change. For example, when the rate of change of a certain type of data is detected to be accelerated, the system will automatically increase the collection frequency. Anomaly detection trigger mechanism: If abnormal fluctuations are found in the data of a specific data source, additional collection tasks can be immediately triggered for in-depth analysis. Real-time monitoring: The system can set up a real-time monitoring mechanism to detect data quality problems (such as duplication, missing values, etc.) and immediately notify relevant personnel when problems occur. Automatic repair: For some common data quality problems, the system can automatically correct them according to predefined rules. Introduce the concept of data life cycle to help manage the entire process from data creation to deletion: Version control: Keep historical versions for each data update to facilitate tracking of change history and rollback operations. Archiving strategy: Develop an archiving strategy based on the importance and frequency of use of data, and transfer data that is no longer frequently used to low-cost storage media for storage. Use the collected standardized data for more in-depth analysis and prediction to support decision making: Trend analysis: Discover trends by analyzing historical data and make future plans based on them. Predictive modeling: Apply machine learning algorithms to predict possible events in the future, such as customer churn rate prediction, market demand prediction, etc.

[0081] S300, determine the collection cycle and frequency of the scheduled task for collecting master data according to the business process, operation specifications and data timeliness requirements; The collection cycle depends on the speed of data change and the business demand for data freshness.

[0082] Specifically, business processes refer to the various workflows or operational steps within an enterprise that determine how data is generated, used, and stored. Different business processes have different data requirements. For example, the sales department may be more concerned with the latest customer order information, while the finance department is concerned with the monthly bill summary.

[0083] Operating specifications: include data processing guidelines and standards established within the company, such as data format, naming conventions, etc. These specifications ensure that all employees follow the same standards for data entry and management, thereby ensuring data consistency and accuracy.

[0084] Data timeliness: refers to the validity and relevance of data within a specific time period. For some application scenarios, real-time updated data is essential (such as stock trading), while for other scenarios, a certain delay is allowed (such as monthly reports).

[0085] After clarifying the above requirements, the next step is to determine the collection period and frequency of the scheduled task for collecting master data. This involves two key concepts: Collection cycle: refers to the time interval between each collection task execution. For example, a data collection task is executed once every day at 2 am.

[0086] Frequency: refers to the number of times a data collection task is executed within a certain period of time. For example, three data collection tasks are executed in one day.

[0087] The settings of these two mainly depend on the following factors: Data change speed: If the data in a data source is updated frequently (such as new transaction records every hour), the collection cycle should be shorter to ensure that the latest data is obtained.

[0088] Business requirements for data freshness: Different business departments have different requirements for data freshness. For example, the marketing department may need to understand customer feedback in real time, while the human resources department may only need to update employee information once a week.

[0089] Suppose an e-commerce company has the following requirements: Business process: Before 9 a.m. every day, the marketing team needs the latest user behavior data to make advertising decisions for the day; the finance department needs the total sales at the end of each month for report preparation.

[0090] Operational specifications: All data must be stored in a unified format, and any outliers need to be marked for subsequent review.

[0091] Data timeliness: For the marketing team, data timeliness is very important. They hope to obtain data within the last few minutes as much as possible. For the finance department, one month's data summary is sufficient to meet the needs.

[0092] Based on the above situation, the following collection strategy can be set: Data collection for marketing teams: Collection cycle: Execute data collection tasks every 5 minutes to ensure that the latest user behavior data is obtained.

[0093] Frequency: Run 24 / 7, especially during peak hours (e.g. 7pm to 10pm) to increase the frequency of data collection, as this is when user activity is most active.

[0094] Data collection for the finance department: Collection cycle: Data collection tasks start at 12:00 a.m. on the last day of each month to summarize sales data for the entire month.

[0095] Frequency: Only once a month, as monthly reports do not need to be updated too frequently.

[0096] In some embodiments of the present application, the metadata based on multiple data sources is matched by a matching algorithm to obtain the corresponding relationship between the metadata of different data sources, specifically: Determine the multiple data sources from which the metadata needs to be collected, the data sources including: a relational database, a big data platform, and an API interface; Collecting the metadata and performing preprocessing, the metadata includes: data table structure, field information, data type, data length and constraint conditions; Matching the metadata based on a fuzzy matching algorithm to identify similar or identical metadata items; Based on the metadata items, common attributes of different data sources are obtained.

[0097] Specifically, first, the system needs to determine which data sources will be used for metadata collection. The data sources mentioned here include but are not limited to: Relational database: such as MySQL, PostgreSQL, Oracle and other traditional relational databases.

[0098] Big data platforms: such as HBase, Hive, etc. in the Hadoop ecosystem, and NoSQL databases such as MongoDB.

[0099] API interface: It can be a RESTful API provided by internal enterprise services or an API for obtaining data from the outside.

[0100] Each data source has its own unique structure and storage method, so it is very important to clarify the data source type before collecting metadata.

[0101] Next, the system will collect metadata from the data sources identified above. Metadata refers to data that describes data and contains detailed information about the data itself. Specifically, metadata includes but is not limited to the following: Data table structure: describes how the tables in the database are organized, such as table name, column name, etc.

[0102] Field information: including field name, field type (such as integer, string), field length limit, etc.

[0103] Data type: Indicates the type of data stored in the field, such as date, number, Boolean, etc.

[0104] Data Length: For some field types (such as strings), you may also need to specify the maximum number of characters allowed.

[0105] Constraints: such as unique constraints, foreign key constraints, etc., to ensure the consistency and integrity of data.

[0106] After collecting these metadata, the system will preprocess them. The preprocessing step may include cleaning invalid data, standardizing the format, etc., so that subsequent analysis is more accurate and effective.

[0107] After the preprocessed metadata is obtained, the next step is to use a fuzzy matching algorithm to find similar or identical metadata items between different data sources. Fuzzy matching algorithms usually perform matching based on the following aspects: String Similarity of Field Numbers: Compares the similarity of field names, even if they are not exactly the same but very close they may be considered related. For example, "CustomerID" and "Cust_ID" may be considered similar.

[0108] Semantic similarity of field names: not only the literal similarity but also the actual meaning of the field is considered. For example, "FullName" and "ClientName" are different in literal meaning, but if they are both used to represent the name of a customer, they will be considered similar.

[0109] This fuzzy matching approach enables the system to identify fields that, despite having different names in different data sources, actually represent the same concept.

[0110] The last step is to extract the common attributes between different data sources based on the matching results. The goal of this step is to build a standard model across data sources so that all related data can be integrated according to this standard model. For example, if it is found that CustomerID, Cust_ID, and ClientCode are actually unique numbers used to identify customers in different data sources, then these three fields can be mapped to a new standardized field CustomerID.

[0111] This not only simplifies the subsequent data integration process, but also improves data consistency and availability, allowing enterprises to use their data resources more efficiently when processing multi-source data. In addition, this approach reduces the workload of manual configuration, reduces the risk of human error, and improves overall work efficiency.

[0112] In some embodiments of the present application, the collecting of the metadata is specifically: Automatically executing a scheduled task according to a preset schedule to collect the metadata from the determined data source; The metadata is collected according to actual business needs by manually triggering the collection task.

[0113] Specifically, the system can automatically execute scheduled tasks to collect metadata according to a preset schedule. This means that users can set a time plan (or scheduling strategy) in advance to determine when and how often to collect metadata from a specified data source.

[0114] Application scenario: For example, in an enterprise environment, it may be necessary to obtain the latest metadata information from multiple data sources such as the ERP system and CRM system at 2 a.m. every day. The advantage of this is that data collection can be performed during the off-peak period of business, reducing the impact on daily business operations.

[0115] Specific operation: Users set the collection cycle (such as daily, weekly, etc.) and specific time point (such as 2 a.m. every day) through the configuration interface. The system automatically generates and schedules scheduled tasks based on these settings. When the scheduled time arrives, the system automatically starts the collection process without manual intervention.

[0116] This automated approach greatly improves efficiency, reduces the possibility of human error, and ensures the timeliness and consistency of data. In addition, it allows companies to complete large-scale data collection without affecting normal business operations.

[0117] In addition to scheduled automatic collection, the system also supports users to manually trigger metadata collection tasks based on actual business needs. This method is more flexible and suitable for dealing with emergencies or specific project needs.

[0118] Application scenario: For example, one day you are suddenly notified that the structure of an important customer database has changed. At this time, you need to update the corresponding metadata information immediately instead of waiting for the next scheduled task to arrive.

[0119] Specific operation: The user logs in to the system backend and selects the target data source for which metadata is to be collected. Click the "Manual Trigger" button and the system immediately starts the collection task. After the collection is completed, the system usually generates a report showing the results of this collection, including the successful and failed parts and their reasons.

[0120] Manually triggered collection tasks provide greater flexibility, allowing companies to quickly respond to market changes or other emergencies. This is especially important for companies that need to obtain the latest data instantly to support decision making.

[0121] In some embodiments of the present application, the fuzzy matching algorithm performs matching based on string similarity of field numbers and semantic similarity of field names.

[0122] Specifically, string similarity of field numbers refers to comparing the string similarity between field numbers (usually field names or other unique identifiers) to determine whether they represent the same concept. This method mainly relies on string distance algorithms, such as Levenshtein Distance, to measure the difference between two strings.

[0123] Levenshtein Distance: This is a classic string similarity calculation method that defines the minimum number of editing operations (insertion, deletion, or replacement of characters) required to transform from one string to another. For example, the edit distance between the strings CustomerID and Cust_ID is small because only a small number of operations are required to transform one string into another.

[0124] Application scenario: When field names in different data sources are slightly different but represent the same concept (such as CustomerID vs Cust_ID), you can determine whether they should be considered equivalent by calculating the edit distance between these field names.

[0125] Semantic similarity of field names refers to matching based on the actual meaning of the field names. This usually involves natural language processing technology to understand the semantic information behind the field names, so as to more accurately identify those fields that are literally different but actually represent the same content.

[0126] Semantic similarity calculation: You can use word vector models (such as Word2Vec, GloVe), BERT and other deep learning models to calculate the semantic similarity between field names. These models can capture the semantic relationship between words and identify their similarity even if the words are not exactly the same. For example, although FullName and ClientName use different specific words, they both refer to the name of the customer, so they are semantically similar.

[0127] Application scenario: In an enterprise environment, different systems may use different terms to describe the same business entity. For example, Address is used in one system to represent an address, while Addr is used in another system. Although the two words have significant differences in spelling, they are very close in semantics, and semantic similarity analysis can help identify this relationship.

[0128] In practical applications, combining the string similarity of field numbers and the semantic similarity of field names can provide more comprehensive and accurate matching results. Relying solely on string similarity may miss some fields that are semantically equivalent but actually have large spelling differences; conversely, relying solely on semantic similarity may not identify those that are simply spelling errors. Therefore, considering these two factors together can help improve the accuracy of matching and ensure that related fields in different data sources can be correctly identified and integrated.

[0129] Assume we have two data sources: There are fields Cust_ID and Name in the ERP system There are fields CustomerNumber and FullName in the CRM system Through the fuzzy matching algorithm: For Cust_ID and CustomerNumber, although they are spelled differently, the edit distance calculation shows that their strings have high similarity and both semantically represent unique identifiers of customers.

[0130] Similarly, Name and FullName, although different in spelling, both semantically refer to the customer's name.

[0131] Finally, the system will map these fields together based on the above analysis to form a unified standard model for subsequent data collection and integration. The benefit of this is that it reduces the need for manual intervention and improves data management efficiency.

[0132] In some embodiments of the present application, a standard model is established based on the corresponding relationship, and master data of multiple data sources are collected through the standard model in combination with preset collection rules, specifically: Based on the common attributes, establishing a field matching relationship with the data source to form the standard model; According to preset collection rules, the master data of the multiple data sources are collected, and the collection rules include: specifying data tables, fields and filtering conditions; The collected master data is cleaned, converted and integrated.

[0133] Specifically, after identifying the metadata correspondence between different data sources (for example, Cust_ID, CustomerNumber, and ClientCode all represent customer unique identifiers), the next step is to use these common attributes to establish a unified standard model.

[0134] Common attributes: These are fields that represent the same business entity or concept in different data sources. For example, although they have different names, they are actually used to store unique identifiers for customers.

[0135] Field matching relationship: Based on the above common attributes, the system will create a mapping table or dictionary to match the fields in each data source with the fields in the standard model. For example, all fields representing customer ID will be mapped to the CustomerID field in the standard model.

[0136] The result of this step is a standardized data structure or framework that defines how to transform data from different data sources into a unified format for subsequent processing.

[0137] Once the standard model is established, the next step is to automatically collect master data from multiple data sources according to pre-set collection rules. The collection rules here mainly include the following aspects: Specify data tables: clearly indicate which tables need to be collected. For example, in an ERP system, you may only need to extract data from the Customers table.

[0138] Select specific fields: Not limited to the entire table, you can also refine to the specific field level. For example, only collect information on the CustomerID and CustomerName fields.

[0139] Set filter conditions: You can filter data based on certain conditions. For example, only collect customer records that were added in the past year, or customer records with a credit score higher than 70 points, etc.

[0140] These rules help the system accurately locate and obtain the required data, avoid unnecessary full data transmission, improve efficiency and reduce resource consumption.

[0141] The collected data usually needs further processing to ensure its quality and consistency: Data cleaning: Remove erroneous, duplicate or incomplete records. For example, delete records that contain invalid values ​​(such as null values ​​or outliers).

[0142] Data conversion: convert data in different formats into the form required by the standard data model. This includes operations such as date format unification and numerical unit standardization.

[0143] Data integration: Merge data from multiple sources together to ensure there are no conflicts and they are logically consistent. For example, integrate customer information from ERP systems, CRM systems, and SCM systems into one comprehensive view.

[0144] Through these steps, the final result is a high-quality, consistent and easy-to-analyze master data set, which can provide more accurate decision support for the enterprise.

[0145] Assume that a company has three different systems (ERP, CRM, SCM), each with its own way of storing customer information. By applying the above process: Establish a standard model: The system recognizes that Cust_ID in ERP, CustomerNumber in CRM, and ClientCode in SCM all represent unique customer identifiers, and maps them uniformly to the CustomerID field in the standard model.

[0146] Set collection rules: In the ERP system, extract data from only the CustomerID and CustomerName fields from the Customers table. Set filtering conditions to collect only customer records that were added in the past three months.

[0147] Data cleaning, transformation and consolidation: During the cleaning process, several customer records with invalid phone numbers were found and removed. All dates were formatted as YYYY-MM-DD.

[0148] Ultimately, customer information from ERP, CRM and SCM systems is integrated into a unified view for corporate management to review and analyze.

[0149] This approach greatly simplifies complex data integration tasks, reduces manual intervention, and improves data management efficiency. At the same time, it also provides a solid foundation for enterprise data analysis and decision-making.

[0150] In some embodiments of the present application, after determining the collection period and frequency of the scheduled task for collecting master data according to the business process, operation specifications and data timeliness requirements, the following is further included: The matching results of the metadata and the collection status of the master data are displayed through a graphical interface, which supports users to customize the collection rules and the scheduled tasks, and provides data quality detection reports and abnormal alarm functions.

[0151] Specifically, graphical interface: In order to enable non-technical personnel to easily understand and operate the complex metadata matching and master data collection process, the system provides an intuitive graphical interface.

[0152] Metadata matching result display: This interface shows how fields between different data sources are identified as similar or identical, and how these fields are mapped to the standard model. For example, it can show that Cust_ID in the ERP system, CustomerNumber in the CRM system, and ClientCode in the SCM system are all mapped to CustomerID in the standard model.

[0153] Master data collection status display: Real-time display of the collection progress and status of master data in each data source. Users can view which data has been successfully collected, which is being processed, whether there are any errors or failed tasks, etc. This visualization helps to quickly understand the operation status of the system and take timely actions.

[0154] Custom configuration: allows users to flexibly adjust collection rules and scheduled task settings according to actual business needs.

[0155] Customized collection rules: Users can select specific data tables, fields, and set filtering conditions through the interface. For example, you can choose to extract only the CustomerID and CustomerName fields from the Customers table in the ERP system, and only collect new customer records in the past three months.

[0156] Scheduled task customization: Users can set the specific execution time of scheduled tasks (such as 2 a.m. every day) or based on certain trigger conditions (such as when the new data reaches a certain amount). In addition, the collection cycle and frequency can be adjusted to meet the needs of different business scenarios.

[0157] Data quality detection report: The system will regularly generate quality reports on the collected data to help users evaluate the accuracy and completeness of the data.

[0158] Content example: The report may include the results of data integrity checks (such as the number of missing values), consistency verification (such as whether the field format meets expectations), detection of duplicate data, etc. This helps to identify potential problems and guide subsequent data cleaning work.

[0159] Abnormal alarm function: Once data quality problems or other abnormal situations are detected, the system will immediately issue a warning to notify relevant personnel.

[0160] Application scenarios: For example, if a key data source suddenly stops updating data, the system will send an alarm notification to the administrator; or when a large amount of duplicate data is detected entering the system, an alarm will be triggered for timely processing.

[0161] Implementation method: Users can be notified via email, SMS, or by pop-up messages directly on the graphical interface. This ensures that problems are responded to quickly and reduces the impact on business.

[0162] Assume that a business uses the above functionality to manage its customer data from multiple sources: Graphical interface display: After logging into the system, users can see all connected data sources and their corresponding metadata matching diagrams on the dashboard. There is a status indicator next to each match item to show whether the match has been completed or is in progress.

[0163] At the same time, users can also view the master data collection progress bar of each data source, as well as the specific collection status (completed, in progress, failed, etc.).

[0164] Custom configuration: The marketing team wants to obtain the latest customer interaction data before 9 a.m. every day for the day's advertising decisions. They can easily set a new scheduled task through the graphical interface, specify the collection time as 7 a.m. every day, and choose to extract only the interaction records within the last 24 hours from the CRM system.

[0165] Data quality detection report and abnormal alarm: The system generates a data quality report every night, listing problems with all the data collected that day (such as how many records are missing required contact fields).

[0166] In some embodiments of the present application, the abnormal alarm includes: triggering email or SMS notification for acquisition failure or data format error events.

[0167] Specifically, if abnormal fluctuations are found in the data from the ERP system one day (such as a sudden increase in a large number of invalid records), the system will automatically send an alarm email to the relevant person in charge, reminding them to check whether there is a fault in the data source.

[0168] A computer program product comprising instructions, when running on a device, enables the device to execute the steps in a method for automatically collecting master data based on metadata.

[0169] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a method for automatically collecting master data based on metadata.

[0170] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for automatically collecting master data based on metadata are implemented.

[0171] Example 2 Step 1: Metadata collection and preprocessing Determine the data source type The system supports a variety of heterogeneous data sources, including but not limited to: Relational databases (such as MySQL, Oracle) Big data platforms (such as Hadoop, Spark) API interface (such as RESTful API, SOAP) File-based data sources (such as JSON and XML files) Metadata collection method Scheduled task collection: Automatically trigger metadata collection tasks according to a preset schedule (such as 1 a.m. every day).

[0172] Manually triggered collection: Users manually start tasks through a graphical interface to meet temporary business needs.

[0173] Metadata content and preprocessing The metadata collected includes: Data table structure (such as table name, primary key, foreign key) Field information (such as field name, data type, data length) Constraints (such as non-null constraints, unique constraints) The preprocessing steps include: Deduplication: Remove duplicate metadata items.

[0174] Formatting: unify timestamp, character encoding and other formats.

[0175] Classified storage: Metadata is stored in the metadata warehouse by data source type.

[0176] Step 2: Metadata fuzzy matching and common attribute extraction Fuzzy matching algorithm implementation The string similarity calculation based on **Levenshtein Distance** is adopted, and the formula is as follows:

[0177] Application scenarios: Match semantically similar fields in different data sources (such as "Customer ID" and "User Number").

[0178] Combine field type (such as numeric type, character type) and data length (such as varchar(50)) to perform multi-dimensional similarity weighted calculation.

[0179] Common attribute extraction The system classifies metadata items with similarity exceeding a threshold (e.g., ≥80%) as “common attributes”.

[0180] Example: The field "OrderDate" of data source A and the field "TransactionTime" of data source B are identified as time-type common attributes.

[0181] Step 3: Standard model construction and master data collection Standard model generation Establish field mapping relationships based on common attributes. For example: Standard model field: OrderDate ↔ DataSourceA.OrderDate ↔ DataSourceB.TransactionTime Standard model field: OrderDate ↔ DataSourceA.OrderDate ↔ DataSourceB.TransactionTime The model supports dynamic expansion and automatically adapts mapping rules when new data sources are added.

[0182] Master data collection rule configuration Data table and field screening: specify the data tables and fields to be collected (such as only collecting the "Name" and "Phone" fields in the "Customer Table").

[0183] Filter conditions: Set business logic filters (such as "only collect order data after 2023").

[0184] Collection method selection: Direct database connection: Execute SQL queries through JDBC connection.

[0185] API call: Send HTTP request to obtain data in JSON format.

[0186] File parsing: Parse XML / JSON files and extract target fields.

[0187] Data cleaning and integration Cleaning rules: Deduplication (e.g. removing duplicate records based on primary key).

[0188] Format conversion (such as converting "2023-01-01" to "YYYYMMDD" format).

[0189] Missing value handling (such as filling empty fields with default values).

[0190] Integrated storage: The cleaned master data is stored in the central database according to the standard model.

[0191] Step 4: Scheduled tasks and dynamic scheduling Collection cycle setting Dynamically adjust the collection frequency according to business needs: High-frequency update: Order data is collected every 15 minutes.

[0192] Low-frequency update: Customer information is collected once every morning.

[0193] Supports abnormal retry mechanism (such as automatically retrying 3 times after network interruption).

[0194] Data freshness guarantee Monitor data source changes (such as changes in database table structure) and trigger real-time collection tasks.

[0195] Step 5: User Interaction and Monitoring Graphical interface function Visualization of matching results: Displays the field mapping relationship of different data sources in a tree diagram.

[0196] Acquisition status monitoring: real-time display of task progress, success rate and time consumption.

[0197] Rule configuration: Users can customize collection rules and filtering conditions by dragging and dropping.

[0198] Data quality reporting and alerting Quality testing: Automatically generate reports containing data completeness, consistency, and accuracy indicators.

[0199] Abnormal alarm: trigger email or SMS notification for events such as collection failure and data format error.

[0200] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0201] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0202] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for automatically collecting master data based on metadata, characterized in that: include: Based on metadata of multiple data sources, the metadata are matched by a matching algorithm to obtain a corresponding relationship between the metadata of different data sources; Based on the corresponding relationship, a standard model is established, and master data from the multiple data sources are collected through the standard model in combination with preset collection rules; The collection cycle and frequency of the scheduled task of collecting master data are determined according to the business process, operating specifications and data timeliness requirements. The collection cycle depends on the speed of data change and the business's demand for data freshness.

2. The method according to claim 1, characterized in that The metadata based on multiple data sources are matched by a matching algorithm; To obtain the corresponding relationship between the metadata of different data sources, specifically: Determine the multiple data sources from which the metadata needs to be collected, the data sources including: a relational database, a big data platform, and an API interface; Collecting the metadata and performing preprocessing, the metadata includes: data table structure, field information, data type, data length and constraint conditions; Matching the metadata based on a fuzzy matching algorithm to identify similar or identical metadata items; Based on the metadata items, common attributes of different data sources are obtained.

3. The method according to claim 2, characterized in that The collecting of the metadata specifically includes: Automatically executing a scheduled task according to a preset schedule to collect the metadata from the determined data source; The metadata is collected according to actual business needs by manually triggering the collection task.

4. The method according to claim 2, characterized in that: The fuzzy matching algorithm performs matching based on the string similarity of the field numbers and the semantic similarity of the field names.

5. The method according to claim 1, characterized in that Based on the corresponding relationship, a standard model is established, and the master data of the multiple data sources are collected through the standard model in combination with the preset collection rules, specifically: Based on the common attributes, establishing a field matching relationship with the data source to form the standard model; According to preset collection rules, the master data of the multiple data sources are collected, and the collection rules include: specifying data tables, fields and filtering conditions; The collected master data is cleaned, converted and integrated.

6. The method according to claim 1, characterized in that After determining the collection period and frequency of the scheduled task for collecting master data according to the business process, operation specifications and data timeliness requirements, the following is also included: The matching results of the metadata and the collection status of the master data are displayed through a graphical interface, which supports users to customize the collection rules and the scheduled tasks, and provides data quality detection reports and abnormal alarm functions.

7. The method according to claim 6, characterized in that The abnormal alarm includes: triggering email or SMS notification for acquisition failure and data format error events.

8. A computer program product comprising instructions, which, when executed on a device, is characterized in that: The device is enabled to execute the steps in the method for automatically collecting master data based on metadata as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for automatic master data collection based on metadata as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the method for automatic master data collection based on metadata as described in any one of claims 1 to 7 are implemented.

Images