Intelligent service association method based on data consanguinity

Abstract

The invention relates to the technical field of big data, in particular to an intelligent service association method based on data consanguinity, which comprises the following steps of: 1, acquiring and forming metadata; tracking and collecting a service activity link to obtain a data consanguinity and a metadata consanguinity; 2, generating a business log; step 3, collecting a business log, and establishing a business atlas based on metadata consanguinity; step 4, defining a service triggering rule, a service decision point and a response rule based on the service atlas; and 5, monitoring the data change in real time, mapping the data change into the associated service map according to a service triggering rule, and carrying out service early warning prompt or associated information change display. According to the invention, automatic tracking and intelligent management of the project business process can be realized, and the utilization rate of enterprise data and the enterprise management efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an intelligent business association method based on data lineage. Background Art

[0002] With the rapid development of information technology, enterprises are faced with the challenges of managing massive amounts of data and increasingly complex business processes. Data lineage, which describes the entire process from data generation to consumption, has gradually become a core tool for data governance. Although traditional data lineage analysis focuses on data tracking, impact analysis, and compliance auditing, it is insufficient in real-time business process monitoring, automated reminders, and intelligent association. The current industry is actively exploring the use of artificial intelligence and machine learning to enhance data lineage analysis capabilities, but the existing data lineage has a low correlation with business processes and cannot be applied to business processes to track and warn the corresponding business processes. Therefore, how to tightly integrate data lineage into business processes and realize intelligent business association has become a technical problem that needs to be solved in this field. Summary of the invention

[0003] In order to solve the technical problem in the prior art that the correlation between data lineage and business is low and data tracking and early warning cannot be achieved, the present invention proposes an intelligent business association method based on data lineage to achieve intelligent management of business processes.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a data lineage-based intelligent business association method, comprising the following steps:

[0005] Step 1: Collect the data attributes, field definitions, and data types of the original data source to form metadata; then track and collect the business activity links, and track the data changes generated during the business process to obtain the upstream and downstream associations of each business data, that is, the data lineage relationship. At the same time, based on the data lineage relationship, the metadata lineage is obtained;

[0006] Step 2: At the beginning of each business node, a corresponding business node code node_code is generated by embedding. The name and business entity name corresponding to the business node code are saved in the system. When the business occurs, the corresponding business node code node_code, global transaction ID and business occurrence time are recorded in the business log;

[0007] Step 3: Collect business logs, and obtain the data lineage, metadata lineage, and time series corresponding to business nodes based on the information in the business logs; segment the data lineage based on the time series of business nodes to obtain the corresponding relationship between data lineage and metadata lineage and business entities, and establish a business graph based on the corresponding relationship;

[0008] Step 4: Based on the business graph, define business trigger rules, business decision points, and response rules;

[0009] Step 5: Monitor data changes in the business database BinLog, data warehouse ChangeLog, and business logs in real time. According to the business trigger rules, map the data changes to the associated business graph to provide business warning prompts or display related information changes.

[0010] In step 1, the metadata collection channels include: business database, data warehouse, external system data and API interface.

[0011] In the step 1, the metadata formed includes: metadata name, metadata code, metadata type, field name, field type, and field length.

[0012] In step 1, the business activity links are tracked and collected through business logs, data warehouses, and business databases.

[0013] In step 1, the business activity links are tracked and collected through the global transaction ID.

[0014] The step 2 also includes the step of generating a global transaction ID at the initial entry of each business. In the process from the entry to the exit of each business, the entire business process is circulated through the global transaction ID.

[0015] In step 4, the defined service triggering rules include the following:

[0016] Metadata name, metadata code, metadata type, rule attributes, rule time, and downward triggering event.

[0017] In step 4, the defined business decision point is a business node for responding to a response rule corresponding to an upstream triggering event; the response rule includes the following contents:

[0018] Metadata name, metadata code, metadata type, source event, calculation rules, and downward triggering event;

[0019] The source event corresponds to a downward triggering event of an upstream service node.

[0020] In step 5, according to the business triggering rules, the data changes are mapped to the associated business graph, and the specific method for providing business warning prompts or displaying associated information changes is:

[0021] Monitor changes in data instance fields associated with metadata attributes bound to business trigger rules;

[0022] Determine whether the metadata change satisfies the triggering rules. If so, send the triggering information down along the business graph to the business decision point corresponding to the triggering event;

[0023] The business decision point responds to events triggered by upstream business entities according to response rules, and after calculation based on calculation rules, it issues business warnings or displays related changes. The business warnings and related changes are displayed in the corresponding business processes.

[0024] In step 5, the business decision point is also used to propagate downward the triggering event after performing calculation based on the calculation rule.

[0025] The two major innovations of this invention are: first, it realizes real-time monitoring and automated analysis of data changes in business processes, significantly improving the efficiency and response speed of business management; second, it reduces human intervention and effectively reduces business risks through an intelligent reminder mechanism. In addition, as the basis for realizing this function, the business graph depicts the relationship network between business entities, processes, and data. By connecting entity nodes and data flow lines, it clearly shows the overall picture of the business process, providing a visual basis for intelligent business association.

[0026] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes an intelligent business association method based on data lineage, which realizes automatic monitoring of data changes in various business processes through automatic association of data lineage and business processes, and realizes automatic reminder or display of relevant information through real-time analysis and identification of key business points.

[0027] Specifically, the present invention utilizes real-time data analysis technology to continuously monitor the dynamic changes of data in business processes, such as cost fluctuations, project progress updates, and supply chain status, and identifies key business points of data changes in each business process, such as cost overruns, project delays, material shortages, etc., and triggers early warnings or related business rules through data from key business points. Specifically, by judging whether the data at key business points reaches the early warning threshold or triggers the business rule conditions, automatic tracking and intelligent management of project business processes can be achieved, thereby improving the utilization rate of enterprise data and enterprise management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of an intelligent business association method based on data lineage provided by an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of the relationship between data lineage and metadata lineage in an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a business graph constructed in an embodiment of the present invention;

[0031] Figure 4 It is a schematic diagram of the propagation of a trigger event downward along the business graph in a business entity in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, an embodiment of the present invention provides a method for intelligent business association based on data lineage, comprising the following steps:

[0034] Step 1: Collect data attributes, field definitions, and data types of the original data source to form metadata; then track and collect business activity links, and track data changes generated during the business process to obtain the upstream and downstream associations of each business data, that is, data lineage relationships. At the same time, based on the data lineage relationships, metadata lineage is obtained.

[0035] Specifically, in step 1, the metadata collection channels include: business database, data warehouse, external system data and API interface.

[0036] Specifically, in step 1, the metadata formed includes: metadata name, metadata code, metadata type, field name, field type, and field length.

[0037] Specifically, in step 1, the method of tracking and collecting business activity links includes collecting through business logs, data warehouses, and business databases.

[0038] Specifically, in step 1, the business activity links are tracked and collected through the global transaction ID.

[0039] In this embodiment, metadata is a term used to describe data used to identify, describe, and locate data sets or information resources. Simply put, metadata is "data about data." It provides information about the content, quality, conditions, and other characteristics of the data. Metadata is used to describe various data generated in the business process. For example, through the MySQL table definition, the field attributes and field types defined in the table can be obtained. By collecting the dimension tables in the logarithmic warehouse, the attribute definition of a certain business indicator and its calculation logic can be obtained. By forming metadata, the definitions of all business data scattered throughout the business system can be obtained, thereby forming metadata for these data. In addition to the above methods, metadata can be defined by manual configuration for situations that cannot be automatically parsed.

[0040] In this embodiment, the metadata formed forms a metadata entity (MetaEntry), and each MetaEntry is encoded to obtain a metadata CODE. A metadata CODE represents a description definition of a set of data sets. For example, in a business scenario, there is a database table that stores material information. The table contains material names, material specifications, material inventory quantities, etc., and the metadata used to describe the table will store these field attributes and types of the table. For example, the material name is a string type, the material specification is a specific enumeration type, and the material inventory is a numeric type. The following is the data organization method of the metadata:

[0041] Table 1 Metadata data organization method table

[0042]

[0043] Furthermore, in this embodiment, business context tracking refers to the tracking and collection of business activity links, and the collection methods include business logs, data warehouse changeLog, database binLog, etc. Business activities track the entire business process through the global transaction ID (transaction_id), and track the data changes generated in the business process to establish the upstream and downstream relationship of data. The upstream and downstream association of data is called data lineage. If a business performs the following query operation on the database through SQL statements during execution:

[0044] [d7f45e26-6f39-7a0e-43c0-0c4131ad4770]-2024-5-28 16:23:15:[query]select a.*from material_inventory_info as a left join material_item b on a.id=b.pid

[0045] This query operation includes a join query on two business tables. When the system collects this query statement, it will consider that table A and table B have a relationship (bloodline relationship). Therefore, by collecting this log, the data bloodline relationship between the material_inventory_info and material_item tables can be collected, and the metadata bloodline (metadata bloodline) corresponding to the two tables can be established based on this. Figure 2 As shown, in this embodiment, the association relationship between the material_inventory_info table and the material_item table is data lineage, and the association relationship between the metadata therein is called metadata lineage.

[0046] Step 2: At the beginning of each business node, the corresponding business node code node_code is generated by embedding point method, and the name corresponding to the business node code and the business entity name are saved in the system; when the business occurs, the corresponding business node code node_code, global transaction ID and business occurrence time are recorded in the business log.

[0047] The step 2 also includes the step of generating a global transaction ID at the initial entry of each business. In the process from the entry to the exit of each business, the entire business process is circulated through the global transaction ID.

[0048] Step 3: Collect business logs, and obtain the data lineage, metadata lineage, and time series corresponding to the business nodes based on the information in the business logs; segment the data lineage based on the time series of the business nodes to obtain the correspondence between the data lineage and metadata lineage and the business entities, and establish a business graph based on the correspondence.

[0049] In this embodiment, a business entity refers to a data model with business meaning generated by each business node in the complete business activity process (transaction_id), and the relationship formed by these nodes is called a business graph. Business entity (BE) is a concept in software engineering and information system design. It represents the core object or concept involved in the execution of an enterprise's business process. The business entity contains the information model of all data related to the object during its life cycle, and also includes a life cycle model that describes how to call and execute tasks.

[0050] Business Graph is a conceptual model used to represent and understand various entities within an enterprise and their relationships. The main components of a business graph are nodes, edges, and properties. Business graphs are usually obtained based on the analysis of business systems, which requires the familiarity of business experts with the business system. Usually this method is top-down, such as first defining the business entity, then defining the metadata associated with the business entity and the data model corresponding to the metadata, and manually defining the relationship between business entities to form a business graph.

[0051] In this embodiment, through data lineage and metadata lineage, the scope of entities involved in a business activity is determined based on transaction_id, and a business graph is automatically constructed through time series analysis.

[0052] In this embodiment, the metadata model is divided into domains through business embedding points. For example, a certain material collection business process has "material item selection", "submit collection application form", "application form approval", "material out of warehouse", "material return", and "material settlement" nodes. The globally unique transaction ID generated by the initial entry of the business, the business node code generated at the beginning of the business node, and the time when the business occurred can form a business log, and then the data lineage, metadata lineage and time series corresponding to the node of the business can be obtained through the business log.

[0053] Aggregating the information in the business log can form a complete data set, which can be used for correlation analysis, such as Figure 3 As shown in the figure, the data lineage and metadata lineage generated in a time series are segmented based on the time segments corresponding to each business node, thereby obtaining the corresponding relationship between data lineage and metadata lineage and business entities. Among them, m1~m8 are nodes of metadata lineage, BE1~BE3 are nodes of each business entity, and node_code_1~node_code_3 correspond to each business node code. By segmenting each business metadata lineage through the time series, we can obtain the following: Figure 3 The data map is automatically constructed by the system based on business tracking without the need for human intervention, which greatly improves the application scope of the data.

[0054] Step 4: Based on the business graph, define business trigger rules, business decision points, and response rules.

[0055] In step 4, the defined business trigger rule includes the following contents: metadata name, metadata code, metadata type, rule attribute, rule time, and downward triggering event. As shown in Table 2, it is a form of a defined business trigger rule.

[0056] Table 2 Service trigger rules table

[0057]

[0058] In step 4, the defined business decision point is a business node for responding to a response rule corresponding to an upstream trigger event; the response rule includes the following contents: metadata name, metadata code, metadata type, source event, calculation rule, and downward trigger event; the source event corresponds to the downward trigger event of the upstream business node. As shown in Table 3, a form of a defined response rule is shown.

[0059] Table 3 Response rules table

[0060]

[0061] In this embodiment, the calculation rule is a calculation rule that continues to process the value of this node based on the event triggered by the upstream. For example: the current upstream inventory is 80, and this business node is a node for calculating costs. Each inventory unit is 1,000 yuan, then the calculation rule is to calculate the cost, and the calculation formula is 80*1000.

[0062] Step 5: Monitor data changes in the business database BinLog, data warehouse ChangeLog, and business logs in real time. According to the business trigger rules, map the data changes to the associated business graph to provide business warning prompts or display related information changes.

[0063] In step 5, according to the business triggering rules, the data changes are mapped to the associated business graph, and the specific method for providing business warning prompts or displaying associated information changes is:

[0064] Monitor changes in data instance fields associated with metadata attributes bound to business trigger rules;

[0065] Determine whether the metadata change satisfies the triggering rules. If so, send the triggering information down along the business graph to the business decision point corresponding to the triggering event;

[0066] The business decision point responds to events triggered by upstream business entities according to the response rules, and after performing calculations based on the calculation rules, it issues business warnings or displays related changes.

[0067] Furthermore, in step 5, the business decision point is also used to propagate the downward triggering event downward after performing calculation based on the calculation rules. Figure 4 As shown, after the business entity node BE-1 propagates the downward triggering event to the business entity node BE-2, the business entity node BE-2 can propagate the downward triggering event to the business entity node BE-3 and the business entity node BE-4 based on the response rules. Furthermore, the business entity node BE-3 can also continue to propagate to the downstream business entity node BE-5 based on its own response rules. The downstream business entity responds to the event triggered by the upstream business entity based on the business decision point definition, and performs business warnings or related change displays based on the calculation rules defined by itself. In addition to the warning and related information display of the business entity, the business entity that currently responds to the event can continue to pass the event downward. Intelligent business decision-making and related display can be achieved through the transmission of events in the business graph.

[0068] Combined with the business scenario case, for example, for a material inventory business entity, when the material quantity is less than 100, the INSUFFICIENT_RES (insufficient resources) event will be triggered. This event will be passed down and responded to by the material item business entity. When the entity responds to the event, it can issue an inventory warning, and continue to process the event based on the calculation rules and pass the INSUFFICIENT_STOCK_UP (replenishment of supply) event downward, and continue to pass the material settlement amount reduction event downward according to the replenishment of supply event. Therefore, the present invention can not only realize the construction of the business map, but also, based on the business map, define business trigger rules, business decision points and response rules, can realize the change and update of business warnings or related information, and the business warning information can be displayed in the corresponding business process.

[0069] Application scenarios of the service management method of the present invention include:

[0070] 1. Project management: In construction projects, monitoring of project progress is crucial. Using data lineage technology, you can track input and output data at each stage of the project, including design changes, construction progress, material usage, etc. Through intelligent business association methods, the system can automatically identify delay risks in project progress and automatically remind project managers to take measures at key nodes.

[0071] 2. Cost control: Cost control of construction projects is a complex process involving material costs, labor costs, equipment rental and other aspects. Data lineage technology can help project managers clearly understand the cost structure and monitor cost changes in real time. When the cost exceeds the budget threshold, the system will automatically issue an early warning to help managers adjust resource allocation in a timely manner.

[0072] 3. Supply chain optimization: Supply chain management of building materials is equally important for ensuring construction progress and cost control. Data lineage technology can be used to track the source, transportation process and inventory status of materials. Through intelligent business association, the system can predict material demand, optimize inventory management, and automatically alert the procurement department when materials are in short supply or surplus.

[0073] 4. Construction safety: Safety is a core concern of the construction industry. Data lineage technology can be applied to safety management of construction sites, such as monitoring the use of workers' safety equipment and the status of equipment on the construction site. The system can automatically detect potential safety risks based on the results of data lineage analysis and issue safety warnings when necessary.

[0074] In summary, the present invention proposes an intelligent business association method based on data lineage, the core of which is to automatically monitor data changes in business processes, identify key business nodes through real-time analysis, and automatically remind or display related businesses when necessary. Specifically, the present invention uses real-time data analysis technology to continuously monitor the dynamic changes of data in business processes, such as cost fluctuations, project progress updates, and supply chain status, and identify key business points in these changes, such as cost overruns, project delays, material shortages, etc., so as to determine whether the warning threshold is reached or the business rules are triggered. Once the data changes touch the preset conditions, the system will automatically display the business, such as displaying cost analysis reports, project schedules, or directly send business reminders to relevant persons in charge, such as cost overrun alerts and material procurement notifications. Therefore, the present invention provides a comprehensive solution for the construction industry, which not only improves the efficiency of project management, cost control, supply chain optimization, and construction safety, but also enhances the business agility and risk response capabilities of enterprises through real-time monitoring and intelligent reminders.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent business association based on data lineage, characterized in that: The following steps are involved: Step 1: Collect the data attributes, field definitions, and data types of the original data source to form metadata; then track and collect the business activity links, and track the data changes generated during the business process to obtain the upstream and downstream associations of each business data, that is, the data lineage relationship. At the same time, based on the data lineage relationship, the metadata lineage is obtained; Step 2: At the beginning of each business node, a corresponding business node code node_code is generated by embedding point method, and the name and business entity name corresponding to the business node code are saved in the system; When a business occurs, the corresponding business node code node_code, global transaction ID and business occurrence time are recorded in the business log; Step 3: Collect business logs and obtain the data lineage, metadata lineage, and time series corresponding to the business nodes based on the information in the business logs; Data lineage is segmented based on the time series of business nodes to obtain the correspondence between data lineage, metadata lineage and business entities, and a business graph is established based on the correspondence. Step 4: Based on the business graph, define business trigger rules, business decision points, and response rules; Step 5: Monitor data changes in the business database BinLog, data warehouse ChangeLog, and business logs in real time. According to the business trigger rules, map the data changes to the associated business graph to provide business warning prompts or display related information changes.

2. The method for intelligent business association based on data lineage according to claim 1, characterized in that: In step 1, the metadata collection channels include: business database, data warehouse, external system data and API interface.

3. The method for intelligent business association based on data lineage according to claim 1, characterized in that: In the step 1, the metadata formed includes: metadata name, metadata code, metadata type, field name, field type, and field length.

4. The method for intelligent business association based on data lineage according to claim 1, characterized in that: In step 1, the business activity links are tracked and collected through business logs, data warehouses, and business databases.

5. The method for intelligent business association based on data lineage according to claim 1, characterized in that: In step 1, the business activity links are tracked and collected through the global transaction ID.

6. The method for intelligent business association based on data lineage according to claim 1, characterized in that: The step 2 also includes the step of generating a global transaction ID at the initial entry of each business. In the process from the entry to the exit of each business, the entire business process is circulated through the global transaction ID.

7. The method for intelligent business association based on data lineage according to claim 1, characterized in that: In step 4, the defined service triggering rules include the following: Metadata name, metadata code, metadata type, rule attributes, rule time, and downward triggering event.

8. The method for intelligent business association based on data lineage according to claim 7, characterized in that: In step 4, the defined business decision point is a business node for responding to a response rule corresponding to an upstream triggering event; the response rule includes the following contents: Metadata name, metadata code, metadata type, source event, calculation rules, and downward triggering event; The source event corresponds to a downward triggering event of an upstream service node.

9. The method for intelligent business association based on data lineage according to claim 7, characterized in that: In step 5, according to the business triggering rules, the data changes are mapped to the associated business graph, and the specific method for providing business warning prompts or displaying associated information changes is: Monitor changes in data instance fields associated with metadata attributes bound to business trigger rules; Determine whether the metadata change satisfies the triggering rules. If so, send the triggering information down along the business graph to the business decision point corresponding to the triggering event; The business decision point responds to events triggered by upstream business entities according to response rules, and after calculation based on calculation rules, it issues business warnings or displays related changes. The business warnings and related changes are displayed in the corresponding business processes.

10. The method for intelligent business association based on data lineage according to claim 7, characterized in that: In step 5, the business decision point is also used to propagate downward the triggering event after performing calculation based on the calculation rule.

Images