A management method and system based on multi-source heterogeneous data

Abstract

This application provides a management method and system for multi-source heterogeneous data, relating to the field of data management, and solves the technical problem of low efficiency in existing multi-source heterogeneous data management methods. The method includes: receiving data streams from different systems via a message bus; standardizing the field structure of each data stream according to a unified business entity standard to obtain multi-source standardized data, and generating a relationship graph based on the multi-source standardized data; performing quality verification and correction on the multi-source standardized data based on a pre-built rule base and the relationship graph; and writing the verified data into an object storage layer and a real-time query layer, partitioned by business theme. This application is used for the entire process of centralized aggregation, standardization processing, hierarchical storage, and query services of multi-source heterogeneous data in enterprises, providing an efficient solution for breaking down data silos, improving data quality, and supporting business decision-making.

Description

Technical Field

[0001] This invention belongs to the field of data management, specifically a management method and system based on multi-source heterogeneous data. Background Technology

[0002] In the process of enterprise digital transformation, the efficient management and value mining of multi-source heterogeneous data has become a key link in supporting business decision-making. This data, which comes from different systems and has different formats, contains a wealth of business-related information. Only by achieving unified management and breaking down data silos can reliable support be provided for scenarios such as precision marketing and risk control.

[0003] However, existing technologies struggle to achieve efficient integration, verification, and service while ensuring data consistency and integrity when processing multi-source heterogeneous data. Specifically, data from different systems often follows independent field standards, and differences in field names, types, and formats lead to frequent semantic conflicts. Traditional methods relying on manual configuration of conversion rules are not only costly to adapt to but also unable to handle dynamic changes in business fields, resulting in delays in standardization. Storage and query mechanisms also fail to adapt to business scenarios. The use of a single storage model for all data fails to meet the low-latency query requirements of frequently accessed data and wastes storage resources for massive amounts of historical data. Ultimately, this makes it difficult to transform multi-source heterogeneous data into usable data assets, hindering the effective release of data value. Summary of the Invention

[0004] This application provides a management method and system based on multi-source heterogeneous data, which solves the technical problem of low efficiency in processing multi-source heterogeneous data in the prior art.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, a management method based on multi-source heterogeneous data is provided, including:

[0007] Receive data streams from different systems via the message bus;

[0008] The field structure of each data stream is standardized according to a unified business entity standard to obtain multi-source standardized data, and a relationship graph is generated based on the multi-source standardized data.

[0009] Based on a pre-built rule base and relationship graph, quality verification and correction are performed on multi-source standardized data;

[0010] The verified data is partitioned by business theme and written to the object storage layer and real-time query layer, and a data query interface is provided through the API gateway.

[0011] Based on the above technical solutions, the management method for multi-source heterogeneous data provided in this application unifies the reception of data streams from multiple systems through a message bus, breaking down data silos and eliminating data barriers between different systems, thus achieving centralized aggregation and linkage of cross-system data. Field standardization is performed based on unified business entity standards, standardizing data formats and definitions from the source, effectively solving the semantic conflict problem caused by field differences in multi-source data. A relationship graph is generated to clearly present the relationships between business entities. Multi-dimensional quality verification is performed based on a pre-set rule base and relationship graph, and abnormal data is actively intercepted and repaired through a correction mechanism, ensuring the accuracy and reliability of the data. Finally, data is partitioned and stored according to business themes, distinguishing between the object storage layer and the real-time query layer, and a unified query interface is provided by an API gateway, which not only meets the needs of diverse data application scenarios but also improves the convenience of data retrieval and use.

[0012] In conjunction with the first aspect above, in one possible implementation, the standardization of the field structure of each data stream according to a unified business entity standard includes:

[0013] The metadata registry defines a unified field standard for each business entity, which includes field name, data type, data length, data format, and business meaning.

[0014] When a data stream is accessed through the message bus, the monitoring mechanism of the metadata registry is triggered to extract the original fields of the data stream;

[0015] Compare the original fields with the unified field standard, and convert any mismatched fields according to the unified field standard;

[0016] The missing required fields in the original fields are marked as pending completion, and a field validation notification is triggered from the source system of the missing required fields.

[0017] In conjunction with the first aspect above, in one possible implementation, the comparison of the original field with the unified field standard includes:

[0018] The original fields and the unified field standard are input into a pre-trained field semantic embedding model, which outputs the original field vector and the standard field vector; wherein, the field semantic embedding model is obtained by fine-tuning a pre-trained Transformer model based on the historical pair metadata dataset and the unified business entity standard fields;

[0019] Calculate the cosine similarity score between the original field vector and the standard field vector, and set a first threshold based on the business scenario;

[0020] If the similarity score exceeds the first threshold, it is determined to be a field match, and the original field is converted according to the data type, data length, and data format in the unified field standard; otherwise, the original field is marked as pending review, and a clustering algorithm is used to group and identify the original fields pending review, and the grouping and identification results are used to assist manual verification.

[0021] In conjunction with the first aspect above, in one possible implementation, the use of a clustering algorithm to group and identify the original fields to be reviewed includes:

[0022] Extract the vector set of the original fields to be reviewed as the clustering input data;

[0023] The number of clusters K is determined using the elbow method or the profile coefficient method;

[0024] The vector set is grouped using the K-means clustering algorithm to obtain K cluster groups;

[0025] For each cluster, feature labels are generated. The feature labels include high-frequency words and average vector values ​​of the fields within the group. The high-frequency words are those whose frequency is higher than the average frequency of all words.

[0026] The clustering structure and feature labels are pushed to the human interface to help people identify the business meaning of the fields within the group, and to support human confirmation or adjustment of the grouping.

[0027] In conjunction with the first aspect above, in one possible implementation, the generation of the relationship graph based on multi-source standardized data includes:

[0028] The association rules between business entities are preset in the metadata registry center, and the hook mechanism is used to obtain data snapshots of multi-source standardized data at the flow nodes; the data snapshots include entity primary keys, attribute information, data source identifiers, timestamps and processing component identifiers, and the flow nodes represent preset key nodes used to trigger hook functions;

[0029] Identify the display association relationships of multi-source standardized data based on association rules;

[0030] Input data snapshots and flow nodes into a graph neural network model, and output potential related fields by analyzing the co-occurrence patterns and flow paths of the data;

[0031] Write the potential association fields as potential association rules into the metadata registry;

[0032] A graph database is used to construct a relationship graph, where the nodes of the relationship graph are business entities and the edges of the relationship graph are associations; wherein the associations include explicit associations and potential associations.

[0033] In conjunction with the first aspect above, in one possible implementation, the quality verification and correction of multi-source standardized data includes:

[0034] Quality verification of multi-source standardized data is performed based on relationship graphs and a pre-defined rule base; the pre-defined rule base includes data format rules, business logic rules, and cross-source consistency rules.

[0035] If data with verification errors is found, the abnormal fields are supplemented and the correction record is marked using preset correction rules; the correction rules include data format correction, business logic correction and cross-source consistency correction.

[0036] In conjunction with the first aspect above, in one possible implementation, the data format rules are used to verify whether the field-level syntax structure conforms to a unified standard; the unified standard includes, but is not limited to: checking whether the date field is in the format "YYYY-MM-DD", whether the mobile phone number matches an 11-digit regular expression, whether the numeric field is within a preset range, and whether the length of the character field is less than or equal to a preset length threshold, and the data format rules are implemented through regular expression matching and format validation functions;

[0037] The business logic rules are used to verify whether the data conforms to business process constraints. The business process constraints include, but are not limited to: "order creation time is earlier than payment time", "inventory quantity is a positive integer", and "member users must contain valid contact information". The rule set of business process constraints is dynamically loaded based on the business entity model of the metadata registry center and supports switching the rule set according to business scenarios.

[0038] The cross-source consistency rule is used to verify the attribute consistency of the same entity among multiple source data based on the relationship graph.

[0039] In conjunction with the first aspect above, in one possible implementation, the data format correction corrects the abnormal fields through a preset format conversion function;

[0040] The business logic correction is achieved by reasoning out values ​​that conform to business process constraints through rule sets, and by combining manual review to correct abnormal fields.

[0041] The cross-source consistency correction uses a relational graph to correct abnormal fields.

[0042] In conjunction with the first aspect above, in one possible implementation, the step of writing the verified data into the object storage layer and the real-time query layer by business theme partitioning includes:

[0043] Define multiple business theme partitions, with each theme associated with a corresponding set of business entities;

[0044] Based on the relationship graph query topic, the association relationship of each entity is obtained, and combined with the historical query frequency of the entity, the hot topic is predicted by machine learning model, and multiple hot topic partitions are obtained.

[0045] The verified data is divided into business subject partitions based on business entities, resulting in multi-partition data.

[0046] Multi-partition data is written to the object storage layer, and the partition data is serialized into columnar format. The columnar format data is stored using a hierarchical directory structure.

[0047] Data matching the hot topic partitions in the multi-partition data is written to the real-time query layer, and an asynchronous dual-write mechanism is used to ensure data consistency between the object storage layer and the real-time query layer. The asynchronous dual-write mechanism means that two concurrent write tasks are triggered asynchronously by the stream processing module. One task writes the data to the object storage layer, and the other task writes the data to the real-time query layer.

[0048] Secondly, a management device based on multi-source heterogeneous data is provided, including: a communication unit and a processing unit;

[0049] The communication unit receives data streams from different systems via a message bus and provides a data query interface to the outside world via an API gateway.

[0050] The processing unit is used to standardize the field structure of each data stream according to a unified business entity standard to obtain multi-source standardized data, and generate a relationship graph based on the multi-source standardized data.

[0051] Based on a pre-built rule base and relationship graph, quality verification and correction are performed on multi-source standardized data;

[0052] The validated data is partitioned by business theme and written to the object storage layer and the real-time query layer.

[0053] Thirdly, this application provides a management device based on multi-source heterogeneous data, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the method described in the first aspect and any possible implementation thereof. This management device based on multi-source heterogeneous data can be an electronic device or a chip within an electronic device.

[0054] Fourthly, this application provides a management system based on multi-source heterogeneous data, including: a data acquisition module, a metadata registration center, a stream processing module, and a data storage module; wherein,

[0055] The data acquisition module is used to receive data streams from different systems via a message bus;

[0056] The metadata registry center is used to standardize the field structure of each data stream according to a unified business entity standard, to obtain multi-dimensional standardized data, and to generate a relationship graph based on the standardized data.

[0057] The stream processing module is used to perform quality verification and correction on multi-source standardized data based on a threshold rule base and a relationship graph.

[0058] The data storage module is used to write the verified data into the object storage layer and the real-time query layer according to business themes, and to provide a data query interface through the API gateway.

[0059] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a management device based on multi-source heterogeneous data, cause the management device to perform the methods described in the first aspect and any possible implementation thereof.

[0060] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a management device based on multi-source heterogeneous data, cause the management device based on multi-source heterogeneous data to perform the methods described in the first aspect and any possible implementation thereof.

[0061] This application provides a management method and system based on multi-source heterogeneous data. It achieves unified reception and aggregation of data streams through a message bus, breaking down data silos between multiple systems. By combining a unified field standard from a metadata registry center and leveraging a pre-trained semantic embedding model and clustering algorithm to achieve intelligent standardization of fields, it can resolve semantic conflicts in multi-source data at the source, improving data consistency and interoperability. Simultaneously, by constructing a relationship graph containing explicit and potential associations, supplemented by a pre-built rule base and targeted correction mechanisms, it can achieve end-to-end data quality verification and optimization, ensuring data accuracy in format, business logic, and cross-source consistency, reducing manual intervention costs.

[0062] Furthermore, this invention, through a business theme partitioning strategy and a hierarchical storage architecture, combined with machine learning to predict hot topics, rationally allocates data to the object storage layer and the real-time query layer. This not only meets the needs of efficient storage and analysis of historical data, but also ensures rapid response to frequently accessed data. Furthermore, by providing a unified query interface through an API gateway, it can further improve the availability and access efficiency of data, providing strong support for the efficient management and value mining of multi-source heterogeneous data.

[0063] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A system architecture diagram of a management system based on multi-source heterogeneous data provided in this application embodiment;

[0066] Figure 2 A flowchart illustrating a management method based on multi-source heterogeneous data provided in an embodiment of this application;

[0067] Figure 3 A flowchart illustrating another management method based on multi-source heterogeneous data provided in this application embodiment;

[0068] Figure 4 A flowchart illustrating another management method based on multi-source heterogeneous data provided in this application embodiment;

[0069] Figure 5 This is a schematic diagram of the structure of a management device based on multi-source heterogeneous data provided in an embodiment of this application;

[0070] Figure 6 This is a schematic diagram of the hardware structure of a management device based on multi-source heterogeneous data provided in an embodiment of this application. Detailed Implementation

[0071] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0072] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0073] The management method based on multi-source heterogeneous data provided in this application can be applied to, for example... Figure 1 In the management system based on multi-source heterogeneous data shown, such as Figure 1 As shown, the communication system includes: a data acquisition module, a metadata registration center, a stream processing module, and a data storage module; wherein,

[0074] The data acquisition module is used to receive data streams from different systems via a message bus;

[0075] The metadata registry center is used to standardize the field structure of each data stream according to a unified business entity standard, obtain multi-dimensional standardized data, and generate a relationship graph based on the standardized data.

[0076] The stream processing module is used to perform quality verification and correction on multi-source standardized data based on a threshold rule base and a relationship graph.

[0077] The data storage module is used to write the verified data into the object storage layer and the real-time query layer according to business themes, and to provide a data query interface through the API gateway.

[0078] To address the technical problems of data silos and low access efficiency in existing technologies involving multi-source heterogeneous data, embodiments of this application provide a management method and system based on multi-source heterogeneous data. The method includes:

[0079] Receive data streams from different systems via the message bus;

[0080] The field structure of each data stream is standardized according to a unified business entity standard to obtain multi-source standardized data, and a relationship graph is generated based on the multi-source standardized data.

[0081] Based on a pre-built rule base and relationship graph, quality verification and correction are performed on multi-source standardized data;

[0082] The verified data is partitioned by business theme and written to the object storage layer and real-time query layer, and a data query interface is provided through the API gateway.

[0083] Based on this, this application achieves efficient management and utilization of multi-source heterogeneous data through a unified data flow, processing, storage and service architecture.

[0084] like Figure 2 As shown in the embodiments of this application, the management method based on multi-source heterogeneous data includes:

[0085] S1. Receive data streams from different systems via the message bus.

[0086] Among them, the message bus is a middleware used to transmit data between multiple independent systems, which can achieve decoupling and data interoperability between different systems.

[0087] In some implementations, a message bus can be built using open-source message queue tools (such as Apache Kafka and RabbitMQ) or commercial message middleware (such as IBMMQ). Data streams from different systems can be accessed through methods such as push via application programming interface (API), database log synchronization, and upload via file transfer protocol (FTP).

[0088] It should be noted that the message bus supports multiple data formats (such as JSON, XML, CSV, etc.) and adapts to different data source types of different systems (such as relational databases, non-relational databases, log files, etc.).

[0089] For example, the order system can push order data in JSON format to the message bus via a REST API, and the warehouse system can write inventory change records in CSV format to the message bus via a database log synchronization tool.

[0090] S2. Standardize the field structure of each data stream according to the unified business entity standard to obtain multi-source standardized data, and generate a relationship graph based on the multi-source standardized data.

[0091] The business entity standard specifies the basic attributes of each business entity (such as order, user, product, etc.), including field names, data types, and field meanings, to ensure that the same type of data from different sources has consistent semantics. The relationship graph is a graphical structure used to display the relationships between business entities, where nodes represent business entities and edges represent the relationships between entities, such as "order-contains-product" and "user-places order-order".

[0092] In some implementations, standardization can be performed using Extract-Transform-Load (ETL) tools (such as Talend and Kettle) to convert fields from different data streams into a format that conforms to the business entity standard based on preset field mapping rules; the relationship graph can be implemented using graph databases (such as Neo4j) or relational database association tables, and the association relationship can be built based on the primary key association between entities (such as order ID and user ID).

[0093] It should be noted that the business entity standard can be dynamically updated according to business needs, and the standardized processing process supports the marking of missing fields and the preliminary identification of abnormal fields.

[0094] For example, for "user" data from different systems, ETL tools are used to convert fields with different names such as "cust_id" and "user_no" into "user_id" and unify the data type to string. Based on "user_id", a relationship is established between order data and user data to generate a "user-order" relationship graph.

[0095] S3. Based on a pre-built rule base and relationship graph, perform quality verification and correction on multi-source standardized data.

[0096] The pre-built rule base is used to define multi-dimensional quality verification criteria to verify whether multi-source standardized data meets the requirements in terms of format, business logic and cross-source consistency, and to provide specific judgment basis for data quality verification.

[0097] In some implementations, quality verification can be performed using the Drools rule engine or the Python scripting language, checking standardized data item by item based on a pre-built rule base; correction methods include automatic correction (such as automatic conversion of incorrectly formatted dates) and manual correction (such as manual adjustment of complex business logic conflicts), and correction records can be stored in a log system for traceability.

[0098] It should be noted that relationship graphs can assist in cross-entity verification, such as verifying the validity of products in an order by linking "orders" and "products", thereby improving the comprehensiveness of the verification.

[0099] For example, based on the rule in the pre-built rule base that "order creation time format is YYYY-MM-DD", after verifying abnormal data in the format of "MM / DD / YYYY", it is corrected to the standard format by an automatic conversion tool; for business logic abnormalities such as "order amount is negative", it can be marked and pushed to the manual review interface for correction by business personnel.

[0100] S4. Write the verified data into the object storage layer and real-time query layer according to business theme partitions, and provide data query interface through API gateway.

[0101] The object storage layer is used to store massive amounts of historical data that are accessed infrequently, supporting high scalability and low-cost storage, such as object storage services like Amazon S3 and Alibaba Cloud OSS; the real-time query layer is used to store frequently accessed hot data, supporting fast response to query requests, such as storage systems like Redis and Elasticsearch.

[0102] In some implementations, partitioning by business theme (such as "order management" or "user operation") can be achieved through file directory partitioning or database partition tables; data can be written to the object storage layer through batch processing tools or to the real-time query layer through real-time synchronization tools; and the API gateway (such as Kong or Spring Cloud Gateway) serves as a unified interface entry point, receiving external query requests and routing them to the corresponding storage layer.

[0103] It should be noted that the data in the object storage layer and the real-time query layer can be kept consistent through a synchronization mechanism to ensure the accuracy of query results.

[0104] For example, the verified historical order data is written to the object storage layer in a directory structure of "order subject / year / month", and the order data for the past 7 days is written to the real-time query layer. When an external system queries data through the " / api / order" interface of the API gateway, the gateway automatically routes the historical data query request to the object storage layer and the real-time data query request to the real-time query layer.

[0105] Based on the above technical solutions, the management method for multi-source heterogeneous data provided in this application breaks down data silos between different systems through a message bus, realizing centralized data aggregation; standardized processing ensures the consistency of multi-source data, and the relationship graph clearly presents entity associations; quality verification and correction improve data accuracy; hierarchical storage and API gateway meet diverse data access needs and improve data utilization efficiency.

[0106] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3As shown, the above S1 can be implemented through the following S101 and S102, which are explained in detail below:

[0107] S101. Build and configure the message bus, and determine the topic and related parameters for data reception.

[0108] The message bus uses a distributed message queue system (such as Kafka) to receive and temporarily store data streams from different systems. Topics are classified storage units for data streams, and each topic corresponds to a type of business data, such as order data and inventory data.

[0109] In some implementations, the message bus parameters include the number of partitions and replicas for a topic. The number of partitions is used to split the data stream of the same topic into multiple parallel processing units, improving the concurrency of data writing and reading, avoiding bottlenecks in single-partition processing, and supporting horizontal scaling of data. More partitions result in a larger throughput of data streams that can be processed in parallel. The number of replicas is used to implement redundant data storage. When a node or partition fails, data can be recovered through replicas, ensuring data reliability and availability. More replicas mean stronger data resilience, but also increase storage and synchronization overhead. The number of partitions can be set based on the expected data stream concurrency; for example, the number of partitions = estimated maximum concurrent write volume / single partition processing capacity. The single partition processing capacity can be referenced from historical experience values ​​of similar systems. The number of replicas can be set according to data reliability requirements, such as 3 replicas to ensure no data loss.

[0110] It should be noted that the message bus should have high throughput and low latency to meet the real-time access requirements of multi-source heterogeneous data.

[0111] For example, a Kafka cluster with 3 nodes is set up. An order_topic is created for the order system data stream, with 8 partitions and 3 replicas. An inventory_topic is created for the warehouse system data stream, with 6 partitions and 3 replicas.

[0112] S102. Data streams from different systems are connected to the corresponding topics of the message bus in a preset manner.

[0113] The preset methods include real-time API push, CDC (Change Data Capture) log synchronization, and SFTP (Secure File Transfer Protocol) file retrieval. The specific access method is selected according to the system type and data characteristics.

[0114] In some implementations, for systems with high real-time requirements (such as order systems), REST APIs can be used to push data to the message bus in real time; for database storage systems (such as warehousing systems), database change logs can be captured and synchronized to the message bus using CDC tools such as Debezium; for systems that periodically generate files (such as supplier systems), files can be periodically pulled and converted into data streams and written to the message bus using the SFTP plugin of Kafka Connect.

[0115] It should be noted that initial data format compatibility should be ensured during the access process (such as conversion to JSON format) so that the message bus can process it uniformly.

[0116] For example, whenever a new order is generated, the Order Management System (OMS) calls Kafka's REST Proxy (Representational State Transfer Proxy) via a POST request to push the order data in JSON format to the `order_topic`. When the inventory table in the MySQL database of the Warehouse Management System (WMS) changes, Debezium captures the binary log (binlog), parses it into an inventory change event, and sends it to the `inventory_topic`. The daily user data CSV file generated by the Customer Relationship Management (CRM) system is uploaded to a designated server via the secure file transfer protocol SFTP. Kafka Connect periodically pulls this file, converts it into a JSON stream, and writes it to the `customer_topic`. Here, `order_topic` represents the order topic, `inventory_topic` represents the inventory topic, and `customer_topic` represents the customer topic.

[0117] Based on the above technical solutions, unified and efficient access to multi-source heterogeneous data can be achieved, laying the foundation for subsequent standardized processing, quality verification and other processes.

[0118] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S2 can be implemented through the following S201, S202 and S203, which are explained in detail below:

[0119] S201. Define a unified business entity standard through the metadata registry center and extract the original fields of the data flow.

[0120] The monitoring mechanism of the metadata registry is implemented through hook functions. When new data is written to a topic in the message bus, the hook function is automatically triggered to extract the raw fields of the data stream, including field names and data types. The business entity standard is a standardized definition of the field names, data types, data lengths, data formats, and business meanings of various business entities (such as orders, users, and inventory), used to unify the semantics of multi-source data; while the raw fields are unprocessed field information extracted from the data stream received from the message bus, also including field names and types.

[0121] In some implementations, hook functions can be integrated into the Kafka producer or consumer side. When data is written to a topic, the hook function parses the data's schema information (i.e., the field structure information contained in the data stream, specifically including field names, data types, field lengths, etc.), extracts the raw fields, and sends them to the metadata registry. For structured data (such as database table data), fields can be extracted directly from the data's metadata; for semi-structured data (such as JSON), fields can be extracted by parsing key-value pairs. Meanwhile, the metadata registry can use tools like Apache Atlas, allowing configuration of business entity standards through a visual interface. For example, defining the "order_id" of the "order" entity as a 10-character string and "amount" as a floating-point number (with two decimal places). Raw field extraction is achieved through this monitoring mechanism of the metadata registry, which is automatically triggered when the data stream connects to the message bus, parsing the data structure and recording field information.

[0122] It should be noted that the extraction of original fields should be performed in real time to ensure that subsequent standardization processing can be carried out in a timely manner based on the latest field information; at the same time, the business entity standard should cover core business scenarios and support dynamic updates according to business changes to ensure compatibility with newly added entities or fields.

[0123] For example, the metadata registry defines the unified standard for the "user" entity as follows: the user identifier "user_id" is an integer, the user contact information "phone" is an 11-digit numeric string, and the user registration time "register_time" is in the format "YYYY-MM-DD HH:MM:SS".

[0124] For example, when user data streams from a CRM system are received, the original fields "cust_no" (string), "mobile" (string with area code), and "reg_date" (formatted as "MM / DD / YYYY") are extracted. Similarly, when order_topic receives order data containing fields such as "order_id," "user_id," and "amount," the Kafka Atlas Hook is triggered, parsing the data schema, extracting the original fields "order_id" (string), "user_id" (integer), and "amount" (floating-point), and sending this information to the Apache Atlas metadata registry.

[0125] S202. Compare the original fields with the unified business entity standard, and perform standardization transformation on the mismatched fields to obtain multi-source standardized data.

[0126] The field comparison is achieved through a pre-trained field semantic embedding model, which is obtained by fine-tuning the pre-trained Transformer model based on historical multi-source datasets and unified business entity standard fields. It can convert fields into vectors to quantify semantic similarity. The standardization transformation refers to adjusting the mismatched original fields to a format that conforms to a unified standard.

[0127] In some implementations, the training process of a pre-trained field semantic embedding model includes: (clearly describing the model's input and output, the historical data obtained, and how the Transformer model was fine-tuned, etc.)

[0128] First, historical data is collected. Field metadata from historical multi-source datasets is extracted from the metadata registry, including the original fields from different systems and their corresponding descriptive information. At the same time, metadata for standardized fields of unified business entities is collected to ensure coverage of various fields of core business entities such as orders, users, and inventory. The data volume is no less than 100,000 field samples to ensure training effectiveness.

[0129] The model input is a structured text sequence of fields, which is composed of "field name + data type + business meaning description"; the output is a high-dimensional vector representation of the field, which is used to reflect the semantic features of the field, so that the vector distance of fields with similar semantics is closer.

[0130] Next, using a pre-trained Transformer model as the base model, some low-level parameters are frozen to retain general semantic understanding capabilities, while the top-level parameters are unfrozen for fine-tuning. Field pairs from historical data (field pairs consisting of the original field and its corresponding standardized field) are used as positive samples, and randomly paired fields with different business meanings are used as negative samples to construct the training set. The training objective is to minimize the vector cosine distance between positive sample pairs and maximize the vector cosine distance between negative sample pairs. The Adam optimizer is used, with an initial learning rate of 2e-5, and the iterations are performed for 5-10 epochs. The field matching accuracy is monitored using a validation set, and training stops when the accuracy stabilizes.

[0131] Once trained, the model is deployed to the model service module of the metadata registry, while saving the sample data and model parameters from the training process. The model is incrementally fine-tuned periodically using field samples from newly accessed data streams to adapt to new field types and changes in business scenarios, ensuring the model's semantic recognition capability for emerging fields.

[0132] In some implementations, the field comparison process includes:

[0133] First, the original field and the unified field standard are input into the pre-trained field semantic embedding model to output the original field vector and the standard field vector;

[0134] Subsequently, the cosine similarity score between the original field vector and the standard field vector is calculated. This score is obtained by dividing the vector dot product by the vector magnitude product and is used to quantify the degree of semantic matching between the two.

[0135] Meanwhile, a first threshold is set based on the business scenario: for example, in financial data scenarios, to ensure matching accuracy, the first threshold can be set to 0.85 (this threshold is obtained through verification of historical matching data, and when the threshold is 0.85, the field matching accuracy can reach more than 92%); in retail data scenarios, to improve matching coverage, the first threshold can be appropriately reduced to 0.75.

[0136] If the similarity score exceeds the first threshold, it is determined to be a field match. At this time, the original field is standardized and transformed according to the data type, data length, and data format in the unified field standard.

[0137] If the similarity score does not exceed the first threshold, the original field is marked as pending review, and a clustering algorithm is used to group and identify the original field pending review, so as to use the group identification results to assist manual verification.

[0138] In some implementation methods, the use of clustering algorithms to group and identify the original fields to be reviewed includes:

[0139] For clustering and grouping identification of the original fields to be reviewed, the vector set of the original fields to be reviewed is first extracted as the clustering input data. This vector set is generated by the field semantic embedding model after processing each field to be reviewed.

[0140] Next, the number of clusters K is determined using either the elbow method or the silhouette coefficient method: the elbow method calculates the sum of squared errors (SSE) under different K values ​​and takes the K corresponding to the point where the rate of decrease of SSE drops sharply as the optimal value; the silhouette coefficient method calculates the similarity between the sample and samples in the same cluster and the similarity between the sample and samples in other clusters, and takes the K value when the average silhouette coefficient is maximized.

[0141] Next, the vector set is grouped based on the K-means clustering algorithm: K cluster centers are initialized, and the distance between each vector and each center (such as Euclidean distance) is calculated iteratively. The vector is assigned to the nearest cluster center, and the center of each cluster is recalculated until the cluster centers are stable, or the number of iterations reaches a preset value or the center change is less than a threshold, and finally K cluster groups are obtained.

[0142] Simultaneously, feature labels are generated for each cluster group. The feature labels include high-frequency words and average vector values ​​of the fields within the group: high-frequency words are selected by statistically analyzing the frequency of words appearing in the field names and descriptions within the group, and words with a frequency higher than the average frequency of all words (such as "user" and "ID"); the average vector value is the arithmetic mean of the vectors of all fields within the group, which is used to characterize the overall semantic features of the group.

[0143] Finally, the clustering structure (including the field list of each cluster group) and feature labels are pushed to the human interface to help humans identify the business meaning of the fields within the group. For example, humans can quickly determine the "order number" field corresponding to a certain cluster group through high-frequency words such as "order" and "number". It also supports human confirmation of the rationality of the grouping or adjustment of the grouping boundary to improve the verification efficiency.

[0144] It should be noted that the conversion process needs to be logged, including the original value, the converted value, and the rules, to facilitate the tracking of data changes.

[0145] For example, the original field "cust_no" (string "C123456") has a similarity score of 0.88 with the standard field "user_id" (integer) (exceeding the first threshold of 0.85), and is automatically converted to the integer 123456; "mobile" (string "021-13800138000") is marked as pending review due to format mismatch, and is identified as a "user contact information field" after clustering and grouping, and is converted to "13800138000" after manual confirmation.

[0146] S203. Generate a relationship graph based on multi-source standardized data.

[0147] Among them, the relationship graph is a graphical structure that displays the relationships between business entities. Nodes are business entities, and edges are the relationships between entities, such as "order-include-product" and "user-place order-order". The relationships include explicit relationships identified based on preset rules and potential relationships mined based on data patterns.

[0148] In some implementations, the process of generating a relational graph includes:

[0149] First, the association rules between business entities are preset in the metadata registry. Then, the hook mechanism is used to obtain data snapshots of multi-source standardized data at the flow nodes. The data snapshots include entity primary keys, attribute information, data source identifiers, timestamps, and processing component identifiers. The flow nodes are preset key data processing nodes used to trigger hook functions to capture the state of data in key processing stages.

[0150] Next, explicit association relationships among multi-source standardized data are identified based on preset association rules.

[0151] Next, the data snapshots and flow nodes are input into the graph neural network model. By analyzing the co-occurrence patterns of the data (such as a user ID and a product ID frequently appearing at the same time) and the flow path (such as the path of data flowing from the order system to the payment system and then to the warehousing system), potential related fields are output.

[0152] The output potential related fields are then written into the metadata registry as new related rules, thereby enabling dynamic supplementation and updating of related rules to adapt to changes in business data.

[0153] Finally, a graph database is used to construct a relationship graph, where the nodes of the relationship graph are business entities, such as orders, users, and products; the edges of the relationship graph are associations, including explicit associations identified by preset rules and potential associations mined by graph neural network models, thus forming a complete network of business entity associations.

[0154] The following are examples of association rules in different business scenarios, covering both explicit and implicit associations:

[0155] (1) Explicit association rules:

[0156] Order and User Association: The "user_id" field of the order entity is the same as the "id" field of the user entity, which is used to indicate the association relationship of "user creating order".

[0157] Order and product association: The "sku_id" field in the order details matches the "sku_code" field of the product entity to indicate the association relationship that "the order contains products".

[0158] Inventory and warehouse association: The "warehouse_id" field of the inventory record corresponds to the "id" field of the warehouse entity, which is used to indicate the association relationship of "inventory belonging to warehouse".

[0159] Payment record association with order: The "order_no" field of the payment record is the same as the "order_id" field of the order entity, which is used to indicate the association relationship of "payment corresponding to order".

[0160] (2) Potential association rules:

[0161] User and Frequently Purchased Product Categories Association: The "region" field of the user entity and the "category" field of the product entity co-occur frequently. For example, "Frequently Purchased Home Appliance Categories by Users in East China" is used to represent the association relationship of "User's Regional Preference Product Categories".

[0162] Order and logistics timeliness correlation: The "shipping_type" field of the order entity and the "delivery_hours" field of the logistics record have statistical correlation. For example, "the average timeliness of expedited delivery orders is ≤24 hours" is used to indicate the correlation between "delivery method and logistics timeliness".

[0163] Inventory and promotional activity association: The "stock_quantity" field of the inventory entity and the "promotion_id" field of the marketing activity are associated in a time sequence. For example, "the inventory consumption rate of a certain product during the activity is 3 times that of the usual period" is used to indicate the relationship that "the promotional activity affects inventory consumption".

[0164] These association rules include both strong associations directly based on business primary keys and implicit associations mined through data patterns, together forming a complete relationship network between business entities.

[0165] It should be noted that the relationship graph needs to be updated in real time to ensure that the latest relationships between entities are reflected in a timely manner when new data is added or association rules are changed.

[0166] For example, based on preset rules, the explicit association of "order (order_id=O123) is associated with user (user_id=U456) through user_id=U456" is identified; through graph neural network model analysis, it is found that "user (U456)'s browsing history is highly correlated with the outbound frequency of inventory (sku_id=S789)", and "user-browsing-inventory" is written into the graph as a potential association to form a relationship network.

[0167] Based on the above technical solution, the traceability of the data access process is ensured through real-time monitoring by the metadata registry center. Through the standardized processing of multi-source data, semantic conflicts of fields are eliminated. At the same time, the generated relationship graph clearly presents the entity associations, providing consistent and associative basic data for subsequent quality verification and data application, effectively solving the problem of integrating multi-source heterogeneous data.

[0168] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S3 can be implemented through the following S301, S302 and S303, which are explained in detail below:

[0169] S301. Load the pre-set rule base and parse the verification rules to determine the quality verification dimensions of multi-source standardized data.

[0170] The pre-defined rule base is a collection of data format rules, business logic rules, and cross-source consistency rules, which are used to provide specific verification criteria; the relationship graph is used to support the association verification of cross-source data, and locates the attribute information of the same entity in multi-source data through entity nodes and association edges.

[0171] For example, a pre-built rule base can be as follows:

[0172]

[0173] In some implementations, the pre-built rule base is managed by a rule engine (such as Drools), and the rules are stored in the form of configurable scripts, such as "Date format validation: field='create_time' AND regex_match(value,'^\d{4}-\d{2}-\d{2}'" The relationship graph obtains entity relationships through the query interface of a graph database (such as Neo4j). For example, “MATCH(o:Order)-[r:BELONGS_TO]->(u:User)RETURNo.user_id,u.id” is used to locate the relationship fields between orders and users.

[0174] It should be noted that the rule base and relationship graph need to support dynamic updates. When the business entity standards or association rules change, they can be updated synchronously through the metadata registry interface to ensure that the verification logic is consistent with business requirements.

[0175] For example, the pre-defined rules for loading the "order" entity include: data format rules such as "order_id is a 10-character string" and "amount is a positive number with 2 decimal places"; business logic rules such as "pay_time > create_time"; cross-source consistency rules such as "order table user_id is consistent with user table id"; and the association edge information of "order-user" and "order-payment" is loaded from the relationship graph for cross-source verification.

[0176] S302. Based on the verification dimensions and rules, perform quality verification on multi-source standardized data and output verification anomaly results.

[0177] The quality verification is performed sequentially across three dimensions: data format, business logic, and cross-source consistency. Data format verification focuses on field-level syntax compliance, business logic verification focuses on whether the data conforms to business process constraints, and cross-source consistency verification verifies the attribute consistency of the same entity in multi-source data based on the relationship graph.

[0178] In some implementations, data format validation can be achieved through regular expression matching, such as mobile phone number validation: ^\d{11} The validation is implemented using format validation functions, such as value range judgment: value>=0 AND value<=10000; business logic validation is implemented through rule set inference, such as "inventory quantity = positive integer": is_integer(value) AND value>0, the rule set is dynamically loaded based on the business entity model of the metadata registry center; cross-source consistency validation is implemented by querying the attribute values ​​of the same entity in multi-source data through the relationship graph and calculating the deviation rate to judge consistency.

[0179] In some implementations, the rule set is dynamically loaded based on the business entity model of the metadata registry, which means that business entities are identified by extracting metadata, and the corresponding business logic rule base is loaded according to the business entities. Specifically, this includes:

[0180] Extract the metadata information of the data to be verified from the metadata registry, including the business entity type to which the data belongs (such as order, user, inventory, etc.), the fields contained in the entity and the field attributes (such as data type and business meaning).

[0181] Based on the extracted business entity type, the corresponding rule set category tag (such as "order entity rule set" or "inventory entity rule set") is matched in the rule base. Each tag is associated with the business logic rules unique to that entity. For example, the order entity is associated with the rule "order creation time is earlier than payment time", and the inventory entity is associated with the rule "inventory quantity is a positive integer".

[0182] Verify the compatibility between the rule set and the business entity model, ensuring that there is a mapping relationship between the rule fields in the rule set and the fields in the entity model (e.g., the "pay_time" field in the rule corresponds to the "Payment Time" field in the entity model). If there are unmatched rules, mark them as to be adapted and do not load them temporarily.

[0183] Load the successfully matched rule set into the rule engine to generate executable verification logic. At the same time, listen for business entity model change events in the metadata registration center (such as adding fields, modifying business meanings). When the model changes, automatically trigger the re - matching and loading of the rule set to ensure the real - time synchronization of the rules and the entity model.

[0184] Exemplarily, when the data to be verified is inventory data, extract its business entity type as "Inventory" from the metadata registration center, match the "Inventory Entity Rule Set" in the rule library, and load rules such as "The inventory quantity is a positive integer" and "The inventory update time is later than the warehousing time". If a new field "Safety Stock Threshold" is added to the "Inventory" entity in the metadata registration center, the rule set will automatically adapt and load the new rule "The actual inventory must not be lower than the safety stock threshold".

[0185] It should be noted that the verification process adopts a streaming processing mode (such as Flink) to verify each standardized data accessed in real - time, and batch data adopts a batch processing mode (such as Spark) for timed verification (once an hour), and record the entity identifier, abnormal field, belonging rule ID, and abnormal description of the abnormal data.

[0186] Exemplarily, perform verification on a certain order data: It is found that "create_time='2023 / 12 / 01'" does not conform to the "YYYY - MM - DD" format during data format verification, and it is marked as a format exception; it is found that "pay_time='2023 - 12 - 01 09:00'<create_time='2023 - 12 - 01 10:00'" during business logic verification, violating the rule "The payment time is later than the creation time", and it is marked as a logic exception; during cross - source consistency verification, it is found through the relationship graph query that the user "name='Zhang San'" in the user table, while "user_name='Zhang Can'" in the order table, and the deviation rate exceeds 5% (preset threshold), and it is marked as a cross - source inconsistency exception.

[0187] S303. According to the verification exception results, use the corresponding correction rules to correct the data and record the correction records.

[0188] Among them, the correction rules correspond one - to - one with the verification dimensions: Data format correction is used to repair field syntax errors, business logic correction is used to adjust data that does not conform to the business process, and cross - source consistency correction is used to unify the attribute values of the same entity in multi - source data.

[0189] In some implementations, data format correction is achieved through preset conversion functions. For example, the date format conversion function `convert_date(value,'MM / DD / YYYY','YYYY-MM-DD')` converts "12 / 01 / 2023" to "2023-12-01". Business logic correction uses rule sets to infer reasonable values. For example, based on the rule "inventory is a positive integer", "inventory quantity = -5" is corrected to "5", triggering manual review. Cross-source consistency correction uses the relationship graph to select attribute values ​​from verified data sources as a benchmark. For example, based on user table data, "user_name" in the order table is corrected to "Zhang San". Correction records include the original value, corrected value, correction rule ID, correction time, and operator, and are stored in the audit log database.

[0190] It should be noted that the corrected data needs to be re-executed through the S302 verification process to ensure that the correction results meet the quality requirements and form a closed loop of "verification-correction-re-verification". For anomalies that cannot be automatically corrected (such as missing key fields and no associated data), the alarm system is triggered to notify the data administrator for handling.

[0191] For example, the format conversion function is used to automatically correct "create_time='2023 / 12 / 01'" to "2023-12-01", and the second verification passes; the rule reasoning is used to correct "inventory quantity=-5" to "5", and it takes effect after manual review and confirmation; the "user_name" inconsistency problem is corrected to "Zhang San" in the order table based on the user table data, and the correction record is written to the audit log.

[0192] Based on the above technical solution, multi-dimensional verification of data quality is achieved by utilizing a pre-built rule base and relationship graph, realizing comprehensive quality control of multi-source standardized data. Specifically, the pre-built rule base provides standardized verification logic, the relationship graph supports cross-source data association verification, and targeted correction mechanisms and closed-loop verification processes ensure the effective repair of abnormal data. Ultimately, this improves the accuracy, consistency, and reliability of the data, laying a high-quality data foundation for subsequent hierarchical storage and query services.

[0193] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S4 specifically includes the following S401 to S403:

[0194] S401. Define multi-service theme partitions and predict hot topic partitions.

[0195] Among them, business theme partitioning divides the set of related business entities into different data partitions according to business scenarios, such as "order fulfillment theme" and "user management theme"; hot topic partitioning refers to the theme partitioning with high access volume predicted based on historical query frequency and entity association relationship, which is used to prioritize query efficiency.

[0196] In some implementations, business themes can be configured through a visual interface in the metadata registry. Each theme is associated with a corresponding set of business entities. For example, the "order fulfillment theme" is associated with order, payment, and logistics entities. Partition keys are defined by entity attributes, such as partitioning by "order creation date". When predicting hot topics, the relationships between entities are extracted from the relationship graph. Combined with the historical query logs of the past 30 days, a decision tree model is used to predict the hot topics for the next 7 days. The prediction results are sorted by query volume, and the top 20% are taken as the hot topic partitions.

[0197] In some implementations, the steps for using a decision tree model to predict trending topics for the next 7 days may include:

[0198] In some implementations, the steps for using a decision tree model to predict trending topics for the next 7 days may include:

[0199] (1) Data preparation: Extract the daily query volume, query time distribution, query source type and other features of each business theme in the past 30 days from the historical query log. At the same time, obtain the explicit relationship between the related entities of each theme from the relationship graph (such as the relationship between "order-payment" and "logistics-order"). Integrate this information into the model training dataset, where each sample contains feature variables and the corresponding label "whether the query volume of the next day is in the top 20%" (1 means yes, 0 means no).

[0200] (2) Feature engineering: The extracted features are preprocessed and the distribution of query time periods is converted into numerical features. For example, the proportion of the number of queries in different time periods to the total number of queries on the day is calculated. The entity association is encoded. Entity pairs with association are marked as 1, otherwise as 0. Features with high correlation to query volume are screened through Pearson correlation coefficient to reduce noise interference.

[0201] (3) Model training: The C4.5 decision tree algorithm was used to build the model. The preprocessed features were used as input and "whether the query volume of the next day is in the top 20%" was used as the output label. The training dataset was used for model training. During the training process, 10-fold cross-validation was used to adjust the parameters. The maximum tree depth could be set to 10 and the minimum number of split samples could be set to 20. Then, the training was stopped when the cross-validation accuracy was stable above 85%.

[0202] (4) Model prediction: Input the feature data of the topic to be predicted in the past 7 days into the trained decision tree model to obtain the probability that the daily query volume of the topic will enter the top 20% in the next 7 days. Take the average of the 7-day probability as the hot topic probability. Sort the topics from high to low according to the hot topic probability and select the top 20% of topics as the hot topic partitions for the next 7 days.

[0203] For example, when predicting the "order fulfillment" topic, the model inputs its features such as the average daily query volume over the past 7 days, the proportion of queries during the morning peak, and the association tags with "payment" and "logistics" entities. The model outputs that the daily hot topic probabilities for the next 7 days are 82%, 78%, 85%, 90%, 88%, 75%, and 80%, respectively, with an average of 82.5%, ranking in the top 15% of all topics. Therefore, it is determined to be a hot topic for the next 7 days.

[0204] It should be noted that business theme partitions support dynamic adjustments. When a new business entity is added or the entity relationship changes, the theme configuration can be updated through the metadata registry. Hot topic partitions are re-predicted every morning at midnight to ensure synchronization with actual access demand.

[0205] S402. Divide the data into multiple partitions, write them to the object storage layer and the real-time query layer, and ensure consistency.

[0206] Among them, multi-partition data refers to data that has been verified and distributed to the corresponding topic partition according to the business entity to which it belongs; the object storage layer is used to store the full historical data and supports high-capacity, low-cost storage; the real-time query layer is used to store hot topic data and supports low-latency query; the asynchronous dual-write mechanism refers to the simultaneous triggering of write tasks in the object storage layer and the real-time query layer through the stream processing module to ensure data consistency on both sides.

[0207] In some implementations, data partitioning is accomplished by comparing the entity identifiers of the data with the topic-entity mapping relationship (e.g., order data is automatically partitioned into the "order fulfillment" topic); the object storage layer uses a distributed file system (e.g., HDFS), and the data is serialized into Parquet columnar format and stored hierarchically according to "topic / partition key / entity", such as "order_fulfillment / 20231001 / order"; the real-time query layer uses an in-memory database (e.g., Redis) or a search engine (e.g., Elasticsearch), storing only the most recent 7 days of data for hot topics; asynchronous dual writes are implemented through the Flink stream processing module, triggering a retry mechanism when a write fails, and recording an exception log and triggering an alarm if the retry fails.

[0208] It should be noted that columnar format storage has a high compression rate, is suitable for batch reading, can reduce I / O overhead, and improve batch analysis efficiency; the data retention period of the real-time query layer can be adjusted according to business needs, such as extending it to 15 days during e-commerce promotions.

[0209] For example, the verified order data is divided into the "Order Fulfillment" topic, serialized into Parquet format by the partition key "20231001", and stored in the object storage layer directory "order_fulfillment / 20231001 / order". At the same time, because this topic is a hot topic, the data is synchronously written to the "order_fulfillment_hot" index in Elasticsearch, supporting real-time retrieval.

[0210] S403 provides a unified data query interface through the API gateway, linking topic partitions and storage layers.

[0211] The API gateway serves as a unified data access point, responsible for receiving query requests, routing them to the corresponding storage layer, and returning results. The query interface is designed according to business themes and supports queries based on conditions such as theme, entity, and time range.

[0212] In some implementations, the API gateway integrates identity authentication, access control, and request rate limiting (e.g., a maximum of 10 requests per user per second), and defines query interfaces through RESTful APIs. Internally, the interface automatically selects the data source through a topic-storage layer mapping relationship. For example, hot topics are routed to the real-time query layer, while non-hot topics are routed to the object storage layer. For cross-topic queries, multi-partition data is concatenated through a relationship graph and then returned.

[0213] For example, when a user calls the API interface to query order fulfillment data, the API gateway verifies the user's permissions and, because the topic is a hot topic, routes the query to the Elasticsearch engine and returns the result; if the user queries the non-hot topic "Supplier Management", the query is routed to the Parquet file in the object storage layer.

[0214] Based on the above technical solution, through steps S401 to S403, structured storage and efficient service of the verified data are achieved. Business theme partitioning ensures that data is aggregated according to business logic, hot topic prediction enables accurate allocation of storage resources, asynchronous dual-write mechanism guarantees data consistency, and API gateway provides a unified and convenient data access method. Ultimately, this meets the diverse needs of massive historical data storage and high-concurrency real-time querying, improving data availability and access efficiency.

[0215] The foregoing mainly describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as a management device based on multi-source heterogeneous data, includes at least one of the hardware structure and software module corresponding to each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0216] This application embodiment can divide the management device based on multi-source heterogeneous data into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0217] When using integrated units, Figure 5 A possible structural schematic diagram of the management device based on multi-source heterogeneous data (referred to as management device 50 based on multi-source heterogeneous data) involved in the above embodiments is shown. The management device 50 based on multi-source heterogeneous data includes a processing unit 501 and a communication unit 502, and may also include a storage unit 503. Figure 5 The structural diagram shown can be used to illustrate the structure of the management device based on multi-source heterogeneous data involved in the above embodiments.

[0218] when Figure 5 The schematic diagram shown illustrates the structure of the management device based on multi-source heterogeneous data involved in the above embodiments. The processing unit 501 is used to control and manage the operation of the management device based on multi-source heterogeneous data, the communication unit 502 is used for the management device based on multi-source heterogeneous data to communicate with other devices, and the storage unit 503 is used to store the program code and data of the management device based on multi-source heterogeneous data.

[0219] For example, it can be used to receive data streams from different systems via a message bus and provide data query interfaces to the outside world via an API gateway;

[0220] Processing unit 501 is used to standardize the field structure of each data stream according to a unified business entity standard to obtain multi-source standardized data, generate a relationship graph based on the multi-source standardized data, perform quality verification and correction on the multi-source standardized data based on a pre-set rule base and relationship graph, and write the verified data into the object storage layer and real-time query layer by business theme partitioning.

[0221] In one possible implementation, the processing unit 501 is also used to define a unified field standard for each business entity in the metadata registry center, trigger a monitoring mechanism to extract the original fields when the data stream is accessed, compare the original fields with the unified field standard and convert them, mark the missing required fields and trigger the field verification notification from the source system.

[0222] In one possible implementation, the communication unit 502 is further configured to obtain data snapshots of multi-source standardized data at the flow nodes through a hook mechanism during the generation of the relationship graph; the processing unit 501 is further configured to identify explicit associations based on association rules, input the data snapshots and flow nodes into a graph neural network model to obtain potential association fields and write them into a metadata registry, and construct the relationship graph using a graph database.

[0223] The processing unit 501 can be a processor or a controller, and the communication unit 502 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 503 can be a memory. When the management device 50 based on multi-source heterogeneous data is a chip, the processing unit 501 can be a processor or a controller, and the communication unit 502 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 503 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.)).

[0224] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the multi-source heterogeneous data management device 50 can be considered as the communication unit 502 of the multi-source heterogeneous data management device 50, and the processor with processing functions can be considered as the processing unit 501 of the multi-source heterogeneous data management device 50. Optionally, the device in the communication unit 502 used to implement the receiving function can be considered as a communication unit, which is used to execute the receiving steps in the embodiments of this application. The communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 502 used to implement the transmitting function can be considered as a transmitting unit, which is used to execute the transmitting steps in the embodiments of this application. The transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0225] Figure 5 If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0226] Figure 5 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0227] This application also provides a hardware structure diagram of a management device based on multi-source heterogeneous data (referred to as management device 60 based on multi-source heterogeneous data), see [link to relevant documentation]. Figure 6 The management device 60 based on multi-source heterogeneous data includes a processor 601, and optionally, a memory 602 connected to the processor 601.

[0228] In the first possible implementation, see Figure 6The management device 60 based on multi-source heterogeneous data also includes a transceiver 603. The processor 601, memory 602, and transceiver 603 are connected via a bus. The transceiver 603 is used to communicate with other devices or communication networks. Optionally, the transceiver 603 may include a transmitter and a receiver. The device in the transceiver 603 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 603 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0229] Based on the first possible implementation method Figure 6 The structural diagram shown can be used to illustrate the structure of the management device based on multi-source heterogeneous data involved in the above embodiments.

[0230] in, Figure 6 This can also be illustrated by a system chip in a management device based on multi-source heterogeneous data. In this case, the actions performed by the aforementioned management device based on multi-source heterogeneous data can be implemented by this system chip. The specific actions performed can be found above and will not be repeated here.

[0231] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0232] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a SoC (System-on-a-Chip), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0233] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.

[0234] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0235] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0236] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0237] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0238] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0239] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and modifications.

Claims

1. A management method based on multi-source heterogeneous data, characterized in that, The application relates to a data processing method and system. The method comprises: receiving data streams from different systems through a message bus; standardizing the field structure of each data stream according to a unified business entity standard to obtain multi-source standardized data, and generating a relationship graph based on the multi-source standardized data; based on a preset rule library and the relationship graph, performing quality checking and correction on the multi-source standardized data; writing the checked data into an object storage layer and a real-time query layer according to business themes; wherein the generation of the relationship graph based on the multi-source standardized data comprises: presetting association rules between business entities in a metadata registration center, and obtaining data snapshots of the multi-source standardized data at the preset key nodes through a hook mechanism; the data snapshots comprise entity primary keys, attribute information, data source identifiers and timestamps, and the key nodes are used to trigger a hook function; identifying the display association relationship of the multi-source standardized data based on the association rules; inputting the data snapshots and the key nodes into a graph neural network model to output potential association fields by analyzing the co-occurrence mode and the flow path of the data; writing the potential association fields as potential association rules into the metadata registration center; 2.The management method based on multi-source heterogeneous data according to claim 1, characterized in that, constructing a relationship graph using a graph database, wherein the nodes of the relationship graph are business entities, and the edges of the relationship graph are association relationships; wherein the association relationships comprise display association relationships and potential association relationships. The standardization of the field structure of each data stream according to the unified business entity standard comprises: defining unified field standards of each business entity through the metadata registration center, wherein the unified field standards comprise field names, data types, data lengths, data formats and business meanings; when the data stream is accessed through the message bus, triggering a monitoring mechanism of the metadata registration center to extract original fields of the data stream; comparing the original fields with the unified field standards, and converting the unmatched fields according to the unified field standards; 3.The management method based on multi-source heterogeneous data according to claim 2, characterized in that, marking the missing mandatory fields in the original fields as a to-be-supplemented state, and triggering a field checking notification of the source system of the missing mandatory fields. The comparison of the original fields with the unified field standards comprises: inputting the original fields and the unified field standards into a pre-trained field semantic embedding model to output original field vectors and standard field vectors; wherein the field semantic embedding model is obtained by fine-tuning a pre-trained Transformer model based on a historical metadata set and unified business entity standard fields; calculating the cosine similarity score of the original field vectors and the standard field vectors, and setting a first threshold value based on a business scenario; 4. The management method based on multi-source heterogeneous data according to claim 3, characterized in that, if the similarity score exceeds the first threshold value, it is determined that the fields are matched, and the original fields are converted according to the data types, data lengths and data formats in the unified field standards; otherwise, the original fields are marked as to-be-audited, and a clustering algorithm is used to group and identify the to-be-audited original fields to provide a checking basis for manual checking. The grouping and identification of the to-be-audited original fields using the clustering algorithm comprises: extracting a vector set of the to-be-audited original fields as clustering input data; determining the number of clusters K using the elbow method or the silhouette coefficient method; grouping the vector set based on a K-means clustering algorithm to obtain K cluster groups; generating a feature label for each cluster group, the feature label including a high-frequency word of a field in the group and an average vector value; the high-frequency word representing a word with a frequency higher than the average of all word frequencies; pushing the cluster structure and the feature label to an artificial interface.

5. The management method based on multi-source heterogeneous data according to claim 1, characterized in that, The quality checking and correction of the multi-source standardized data includes: checking the quality of the multi-source standardized data according to a relationship graph and a preset rule library; the preset rule library including data format rules, business logic rules and cross-source consistency rules; if there is data with a checking exception, using a preset correction rule to supplement an abnormal field and mark a correction record; the correction rule including data format correction, business logic correction and cross-source consistency correction.

6. The management method based on multi-source heterogeneous data according to claim 5, characterized in that, The data format rule is used to verify whether the field-level syntax structure conforms to a unified standard; the unified standard including: checking whether a date field is in the format of "YYYY-MM-DD", whether a mobile phone number format is correct, whether a numerical value type field is in a preset interval, whether a character type field length is less than or equal to a preset length threshold, and the data format rule is implemented through regular matching and format checking functions; The business logic rule is used to verify whether the data conforms to a business process constraint; the business process constraint including: "order creation time is earlier than payment time", "inventory quantity is a positive integer", "member user must contain a valid contact method", and the rule set of the business process constraint is dynamically loaded based on a business entity model of a metadata registration center, supporting switching of the rule set according to business scenarios; The cross-source consistency rule is used to verify the attribute consistency of the same entity between multi-source data according to a relationship graph.

7. The management method based on multi-source heterogeneous data according to claim 5, characterized in that, The data format correction corrects the abnormal field through a preset format conversion function; The business logic correction infers a value conforming to the business process constraint through a rule set, and corrects the abnormal field in combination with manual auditing; The cross-source consistency correction corrects the abnormal field through a relationship graph. 8.The management method based on multi-source heterogeneous data according to claim 1, characterized in that, The writing of the checked data into an object storage layer and a real-time query layer according to business themes includes: defining multi-business theme partitions, each theme being associated with a corresponding business entity set; querying the association relationship of each entity in the theme based on a relationship graph, and combining the historical query frequency of the entity to predict a hot theme through a machine learning model to obtain multiple hot theme partitions; dividing the checked data into multiple partition data according to the business theme partitions according to the business entities; writing the multiple partition data into the object storage layer, and serializing the partition data into a columnar format, and storing the columnar format data in a hierarchical directory structure; writing the data conforming to the hot theme partition in the multiple partition data into the real-time query layer, and using an asynchronous double-writing mechanism to ensure the data consistency of the object storage layer and the real-time query layer; the asynchronous double-writing mechanism means that two parallel writing tasks are triggered asynchronously by a stream processing module, one task writes data into the object storage layer, and the other task writes data into the real-time query layer.

9. A multi-source heterogeneous data-based management system, characterized in that, It includes: a data acquisition module, a metadata registration center, a stream processing module and a data storage module; wherein, The data acquisition module is configured to receive data streams from different systems through a message bus; The metadata registry is configured to standardize field structures of the data streams according to a unified business entity standard to obtain multi-source standardized data, and generate a relationship graph based on the multi-source standardized data; wherein The generating of the relationship graph based on the multi-source standardized data comprises: presetting association rules between business entities in the metadata registry, and obtaining data snapshots of the multi-source standardized data at flow nodes by using a hook mechanism; the data snapshots comprise entity primary keys, attribute information, data source identifiers, and timestamps, and the flow nodes represent preset key nodes for triggering a hook function; identifying display association relationships of the multi-source standardized data based on the association rules; inputting the data snapshots and the flow nodes into a graph neural network model, and outputting potential association fields by analyzing co-occurrence patterns and flow paths of the data; writing the potential association fields as potential association rules into the metadata registry; constructing a relationship graph by using a graph database, wherein nodes of the relationship graph are business entities, and edges of the relationship graph are association relationships; the association relationships comprise display association relationships and potential association relationships; The stream processing module is configured to perform quality checking and correction on the multi-source standardized data based on a threshold rule library and the relationship graph; The data storage module is configured to write the checked data into an object storage layer and a real-time query layer according to business topics, and provide a data query interface through an API gateway.

Images