Business data tracing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a business data tracing method and device, electronic equipment and a storage medium, belongs to the technical field of computers, and is applied to the field of financial science and technology and the field of health medical treatment. The method comprises the steps that data attributes, data sources, data processing logic and data use records are mapped into nodes, target nodes are obtained, and the target nodes comprise child nodes; obtaining a connection edge between any two sub-nodes and a semantic edge category of the connection edge; constructing a knowledge graph according to the child nodes, the connection edges and the semantic edge categories; in response to a traceability request for target business data, reversely traversing the connection edge in the knowledge graph to obtain a data traceability link; the target business data is traced according to the data traceability link, the business data can be traced, and the data circulation process and the data association relationship of the business data are clear.

Description

Technical Field

[0001] This application relates to the field of computer technology, and is applied to the fields of financial technology and healthcare, and particularly to a business data traceability method, apparatus, electronic device and storage medium. Background Technology

[0002] Data traceability refers to the recording and tracking of data throughout its entire lifecycle, from generation to use. By tracing the origins of business data, business compliance can be ensured. For example, in a fintech scenario, tracing transaction data allows us to visualize each step of the transaction process, from user authentication to execution, thus identifying abnormal transactions. Similarly, in a healthcare scenario, tracing medical diagnostic data allows us to visualize each step of the diagnostic process, from medical image input to diagnostic result output, thus identifying the causes of misdiagnosis.

[0003] Traditional data traceability often relies on simple log systems. These systems record data operation processes chronologically, such as data creation, modification, and access times. This only allows for basic data tracking, not comprehensive data tracing. Therefore, how to achieve data traceability for business data has become a pressing issue. Summary of the Invention

[0004] The main objective of this application is to propose a business data tracing method, apparatus, electronic device, and storage medium, which aims to achieve comprehensive and effective tracing of business data and clarify the data flow process and data association relationships of business data.

[0005] To achieve the above objectives, a first aspect of this application proposes a business data tracing method, the method comprising:

[0006] Obtain target metadata of the target business data; wherein, the target metadata includes data attributes, data source, data processing logic, and data usage records; The data attributes, data sources, data processing logic, and data usage records are mapped to nodes to obtain target nodes; wherein, the target node includes child nodes, and the child nodes include attribute nodes of the data attributes, source nodes of the data sources, processing nodes of the data processing logic, or record nodes of the data usage records; Obtain the connection edge between any two child nodes and the semantic edge category of the connection edge; Construct a knowledge graph based on the child nodes, the connecting edges, and the semantic edge categories; In response to a tracing request for the target business data, the connection edges are traversed in reverse in the knowledge graph to obtain a data tracing link; wherein, the data tracing link is a link composed of the attribute node, the source node, the processing node, the record node, the connection edge, and the semantic edge category; The target business data is traced based on the data tracing link.

[0007] In some embodiments, after constructing the knowledge graph based on the child nodes, the connection edges, and the semantic edge categories, the method further includes: Obtain the updated metadata and the data type of the updated metadata; wherein the updated metadata is obtained by updating the target metadata; Based on the data type, the updated metadata is subjected to field extraction to obtain key fields; The knowledge graph is updated based on the key fields.

[0008] In some embodiments, the data type includes structured, semi-structured, or unstructured types, and the step of extracting key fields from the updated metadata based on the data type includes: If the data type is the structured type, then a structured query statement is generated according to the preset field extraction table, and the structured query statement is executed to extract fields from the updated metadata to obtain the key fields; If the data type is the semi-structured type, then the target extraction tool is called to parse the fields of the updated metadata to obtain key-value pairs, and the key-value pairs are used as the key fields; If the data type is the unstructured type, then the large language model is invoked to extract fields from the updated metadata to obtain the key fields.

[0009] In some embodiments, the child node has a node field, and updating the knowledge graph based on the key field includes: Calculate the initial similarity between the key field and the node field; The highest initial similarity is selected as the reference similarity; The knowledge graph is updated based on the reference similarity and the key fields.

[0010] In some embodiments, the knowledge graph includes child nodes, connecting edges between any two child nodes, and semantic edge categories of the connecting edges. Updating the knowledge graph based on the reference similarity and the key fields includes: Compare the reference similarity with the preset similarity threshold; If the reference similarity is greater than or equal to the preset similarity threshold, then reference nodes are selected from the child nodes based on the reference similarity, reference edges are selected from the connection edges based on the reference similarity, and reference categories are selected from the semantic edge categories based on the reference similarity. Update the reference node, the reference edge, and the reference category based on the key fields.

[0011] In some embodiments, after comparing the reference similarity with a preset similarity threshold, the method further includes: If the reference similarity is less than the preset similarity threshold, then a new node, a new edge connecting the new node, and a new category of the new edge are created based on the key field. The knowledge graph is updated based on the newly added nodes, newly added edges, and newly added categories.

[0012] In some embodiments, the acquisition of update metadata and the data type of the update metadata include: Obtain reference metadata; wherein the reference metadata is obtained by updating the target metadata, and the reference metadata has a first timestamp; The reference metadata is written into a preset storage space to obtain a second timestamp; wherein the second timestamp is greater than the first timestamp; Calculate the difference between the first timestamp and the second timestamp to obtain the delay duration; The updated metadata is obtained by filtering the reference metadata located in the preset storage space according to the delay duration. Obtain the data type of the updated metadata.

[0013] To achieve the above objectives, a second aspect of this application provides a business data tracing device, the device comprising: The data acquisition module is used to acquire target metadata of the target business data; wherein, the target metadata includes data attributes, data source, data processing logic, and data usage records; A node mapping module is used to map the data attributes, data sources, data processing logic, and data usage records to nodes to obtain target nodes; wherein, the target node includes child nodes, and the child nodes include attribute nodes of the data attributes, source nodes of the data sources, processing nodes of the data processing logic, or record nodes of the data usage records; The edge acquisition module is used to acquire the connection edge between any two child nodes and the semantic edge category of the connection edge; The graph construction module is used to construct a knowledge graph based on the child nodes, the connecting edges, and the semantic edge categories. The query module is used to respond to a tracing request for the target business data by traversing the connection edges in reverse in the knowledge graph to obtain the data tracing link; wherein, the data tracing link is a link composed of the attribute node, the source node, the processing node, the record node, the connection edge, and the semantic edge category; The traceability module is used to trace the target business data according to the data traceability link.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0016] The business data tracing method, device, electronic device, and computer-readable storage medium proposed in this application acquire target metadata of the target business data. This target metadata includes data attributes, data sources, data processing logic, and data usage records. Based on these four dimensions of data information, the entire lifecycle of business data can be traced. The high coupling and deep hierarchical structure of business data necessitate multiple associations in traditional linear tracing to reconstruct the complete cross-system and cross-level link, resulting in low efficiency and susceptibility to chain breaks. To address this issue, this application introduces knowledge graph technology. By mapping data attributes, data sources, data processing logic, and data usage records to nodes, metadata entities are abstracted into nodes, resulting in target nodes. Target nodes include child nodes, which include attribute nodes for data attributes, source nodes for data sources, processing nodes for data processing logic, or record nodes for data usage records. The connection edges between any two child nodes and their semantic edge categories are acquired to abstract the cross-layer and cross-system dependencies, transformations, and usage relationships of business data into edges. Adding semantic edge categories further clarifies the data flow path. A knowledge graph is constructed based on child nodes, connecting edges, and semantic edge categories. This allows for the explicit storage of complex many-to-many and multi-level relationships between business data through node-edge-semantic edge category triples, enabling accurate, efficient, and automated end-to-end tracing in highly coupled and deep-level environments. In response to a tracing request for target business data, the connecting edges in the knowledge graph are traversed in reverse to extract the complete data tracing link in one go. The target business data is then traced based on this data tracing link, achieving efficient and unbroken end-to-end tracing. Attached Figure Description

[0017] Figure 1 This is a flowchart of the business data tracing method provided in the embodiments of this application; Figure 2 This is another flowchart of the business data tracing method provided in the embodiments of this application; Figure 3 yes Figure 2 The flowchart of step S210 in the process; Figure 4 yes Figure 2 The flowchart of step S220 in the text; Figure 5 yes Figure 2 The flowchart of step S230 in the text; Figure 6 yes Figure 5 The flowchart of step S530 in the text; Figure 7 This is another flowchart of the business data tracing method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the business data traceability device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Data traceability refers to the recording and tracking of data throughout its entire lifecycle, from generation to use. By tracing the origins of business data, business compliance can be ensured. For example, in a fintech scenario, tracing transaction data allows us to visualize each step of the transaction process, from user authentication to execution, thus identifying abnormal transactions. Similarly, in a healthcare scenario, tracing medical diagnostic data allows us to visualize each step of the diagnostic process, from medical image input to diagnostic result output, thus identifying the causes of misdiagnosis.

[0022] Traditional data traceability often relies on simple log systems. These systems record data operations chronologically, such as creation, modification, and access times. However, they lack effective presentation and management of crucial information like data sources, complex relationships between data, and the flow paths within different business processes. They can only achieve basic data tracking, not comprehensive data tracing. Therefore, how to trace the origins of business data has become a pressing issue.

[0023] Based on this, embodiments of this application provide a business data tracing method, a business data tracing device, an electronic device, and a computer-readable storage medium, aiming to comprehensively and effectively trace business data and clarify the data flow process and data association relationships of business data.

[0024] The business data tracing method, business data tracing device, electronic device, and computer-readable storage medium provided in this application are specifically described through the following embodiments. First, the business data tracing method in this application is described.

[0025] The business data tracing method provided in this application relates to the field of computer technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the business data tracing method, but is not limited to the above forms.

[0026] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0027] Figure 1 This is an optional flowchart of the business data tracing method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S160.

[0028] Step S110: Obtain the target metadata of the target business data; wherein, the target metadata includes data attributes, data source, data processing logic, and data usage records; Step S120: Map data attributes, data sources, data processing logic, and data usage records to nodes to obtain target nodes; wherein, the target node includes child nodes, and the child nodes include attribute nodes of data attributes, source nodes of data sources, processing nodes of data processing logic, or record nodes of data usage records; Step S130: Obtain the connection edge between any two child nodes and the semantic edge category of the connection edge; Step S140: Construct a knowledge graph based on child nodes, connecting edges, and semantic edge categories; Step S150: In response to the tracing request for the target business data, the connection edges are traversed in reverse in the knowledge graph to obtain the data tracing link; wherein, the data tracing link is a link composed of attribute nodes, source nodes, processing nodes, record nodes, connection edges and semantic edge categories; Step S160: Trace the target business data according to the data traceability link.

[0029] Steps S110 to S160 as shown in the embodiments of this application abstract business data and the many-to-many and multi-level flow relationships between business data into nodes and edges in one go through the knowledge graph, forming a flat and depth-traversable graph index. Through the graph index, all dependencies, transformations and usage relationships can be pre-stored as continuous edges, so that any data point can complete multi-hop traversal along the index, realizing efficient and unbroken full-link traceability.

[0030] In step S110 of some embodiments, target metadata of the target business data is obtained from channels such as databases and business systems. The target business data refers to the business data to be traced, and can be a single value, a record, a table, or an indicator. Taking a fintech scenario as an example, the target business data can be transaction data, user behavior data, or product operation data. Transaction data can be account transaction records, payment settlement information, etc.; user behavior data can be user login logs, browsing history, business processing preferences, etc.; and product operation data can be policy information, insurance claim information, etc. Taking a healthcare scenario as an example, the target business data can be clinical diagnosis and treatment data, medical equipment maintenance data, etc. Clinical diagnosis and treatment data can be medical history records, diagnostic results, surgical records, etc.; and medical equipment maintenance data can be equipment cleaning records, performance testing records, etc.

[0031] Target metadata refers to the target business data, used to describe its business meaning, technical characteristics, and management attributes. Unlike traditional log systems that only record a single dimension of operation time and operator, the target metadata in this embodiment includes four dimensions: data attributes, data source, data processing logic, and data usage records. Each dimension contains its own core attributes, ensuring no business data information is omitted. Data attributes record the essential characteristics of the target business data, such as file format, data dimension, creation time, and data size, resolving the problem of ambiguous basic data attributes. Data source clarifies the data source of the target business data, such as the data provider, data collection interface, and business system number, resolving the problem of untraceable data sources. Data processing logic refines the data processing logic for each step of the target business data, such as data processing algorithms (random forest), data cleaning rules (removing abnormal data), data processing results, and executors, resolving the problem of opaque data processing. Data usage records record the flow and scenario of each access, call, transmission, or reuse of the target business data, such as the user, application scenario, and usage time, resolving the problem of unclear data usage destinations.

[0032] In step S120 of some embodiments, traditional log recording systems lack the presentation and management of key information such as data sources and data relationships. Moreover, when faced with multi-source heterogeneous data, due to the huge differences in data formats and structures, they cannot effectively integrate various types of data information. The system can only achieve simple tracking of data and cannot achieve comprehensive tracking of data.

[0033] Taking the financial supply chain in a fintech scenario as an example, the financial supply chain involves data interaction among multiple entities such as suppliers, manufacturers, logistics providers, and financial institutions. Existing technologies cannot clearly display the complete chain of a single order, from raw material procurement, production and processing, logistics and transportation to payment receipt. Similarly, in the healthcare scenario, the data formats and structures generated by different medical institutions' medical record systems and testing systems vary significantly. Existing technologies cannot unify and integrate this scattered data, nor can they clearly demonstrate the flow and relationships of patient data within the entire healthcare system. To construct a comprehensive traceability system, this application's embodiments introduce knowledge graph technology and construct a knowledge graph comprehensively based on data attributes, data sources, data processing logic, and data usage records, rather than simply using a graph structure to display partial data relationships.

[0034] Specifically, data attributes, data sources, data processing logic, and data usage records are mapped to nodes, resulting in attribute nodes for data attributes, source nodes for data sources, processing nodes for data processing logic, and record nodes for data usage records. Based on the attribute nodes, source nodes, processing nodes, and record nodes, a target node is determined. The target node includes child nodes, which are one of the attribute nodes, source nodes, processing nodes, or record nodes.

[0035] In step S130 of some embodiments, the association between two child nodes is extracted from the target metadata, a connection edge is established between the two child nodes based on the association, and the semantic edge category of the connection edge is determined. The semantic edge category is used to indicate business logic, making the data flow path clearer. The semantic edge category includes at least one of time logic edges, operation logic edges, and permission logic edges. Time logic edges are used to indicate the time of data operation. For example, if there is a connection edge between business system data collection and database storage, it can be marked "Storage completed at 09:35 on 2024-02-20" to clarify the time sequence of data flow. Operation logic edges are used to indicate the business purpose of data processing. For example, if there is a connection edge between data cleaning and predictive model training, it can be marked "Training a financial risk prediction model based on compliant data" to explain the business purpose of data processing. Permission logic edges are used to indicate the permissions required to perform data operations. For example, if there is a connection edge between the prediction result output by the financial risk prediction model and the approval reference, it can be marked "Permission verification passed, approval permission granted" to associate the compliance requirements of data use.

[0036] In step S140 of some embodiments, a knowledge graph is constructed based on each child node, the connection edge connecting any two child nodes, and the semantic edge category of the connection edge. This knowledge graph enables complete and in-depth tracing and management of the entire lifecycle of business data, from generation, collection, storage, processing, transmission, use to destruction.

[0037] Please see Figure 2 In some embodiments, after step S140, the business data tracing method may also include, but is not limited to, steps S210 to S230: Step S210: Obtain the updated metadata and the data type of the updated metadata; wherein, the updated metadata is obtained by updating the target metadata; Step S220: Extract fields from the updated metadata based on the data type to obtain key fields; Step S230: Update the knowledge graph based on key fields.

[0038] In step S210 of some embodiments, the target business data changes as business activities continue to occur. To align the knowledge graph with real-world business scenarios, it is necessary to update the knowledge graph based on the target business data. This application embodiment employs an event-driven update mechanism to update the knowledge graph. When the target business data is updated, the knowledge graph update is automatically triggered. Update refers to operations such as adding, modifying, and deleting.

[0039] When the target business data is updated, the target metadata is also updated. Retrieving the updated target metadata reveals the updated metadata and its data type. With the diversification of data sources, data types include structured, semi-structured, and unstructured types. Structured metadata updates consist of structured database data, while semi-structured metadata updates consist of semi-structured XML or JSON data. Unstructured metadata updates consist of unstructured text, images, videos, and other data.

[0040] In step S220 of some embodiments, when dealing with multi-source heterogeneous data, fields are extracted from the updated metadata according to the data type to obtain key fields. These key fields are then mapped to a unified semantic that the knowledge graph can understand, thereby effectively integrating multi-source heterogeneous data. The key fields are those related to data attributes, data sources, data processing logic, and data usage records.

[0041] In step S230 of some embodiments, the knowledge graph includes child nodes, connecting edges, and semantic edge categories of connecting edges, and the child nodes, connecting edges, and semantic edge categories are updated according to key fields.

[0042] Taking insurance claims as an example, when the policy status changes to "claimed", a semantic edge is generated for policy review → policy claim, and the policy status in the data attributes is updated to "claimed", ensuring that the knowledge graph and actual business data are synchronized in real time.

[0043] Through steps S210 to S230 above, the knowledge graph can be updated in real time and dynamically when business data is updated, so that the knowledge graph is aligned with the real business scenario.

[0044] Please see Figure 3 In some embodiments, step S210 may include, but is not limited to, steps S310 to S350: Step S310: Obtain reference metadata; wherein the reference metadata is obtained by updating the target metadata, and the reference metadata has a first timestamp; Step S320: Write the reference metadata into a preset storage space to obtain a second timestamp; wherein the second timestamp is greater than the first timestamp; Step S330: Calculate the difference between the first timestamp and the second timestamp to obtain the delay duration; Step S340: Filter the reference metadata located in the preset storage space according to the delay duration to obtain the updated metadata; Step S350: Obtain the data type of the updated metadata.

[0045] In step S310 of some embodiments, reference metadata generated by real-time updating of the target metadata is obtained, and the reference metadata has a first timestamp. The first timestamp is the time point when the update event occurs.

[0046] In step S320 of some embodiments, reference metadata is written to a preset storage space to obtain a second timestamp. The preset storage space is a physical or logical storage area used to store data, such as a database or memory. The second timestamp is the time when the reference metadata is written to the preset storage space. The second timestamp is greater than the first timestamp.

[0047] In step S330 of some embodiments, the second timestamp is subtracted from the first timestamp to calculate the difference between the first and second timestamps, thus obtaining the delay duration. The delay duration is the time elapsed from the perception of a change in metadata to the recording of the changed metadata.

[0048] In step S340 of some embodiments, if the delay duration is less than or equal to a preset duration threshold, it indicates that the data write latency is relatively low, and the reference metadata located in the preset storage space is used as the updated metadata. If the delay duration is greater than the preset duration threshold, it indicates that the data write latency is relatively high, the performance of the preset storage space does not meet the requirements, and it cannot guarantee that the metadata provided to downstream applications is up-to-date. In this case, the reference metadata is retrieved again, and an alarm is issued. The preset duration threshold can be set according to actual conditions, such as 100 milliseconds.

[0049] In step S350 of some embodiments, the data type of the updated metadata is determined to be a structured type, a semi-structured type, or an unstructured type.

[0050] Through steps S310 to S350 above, the latest and most effective updated metadata can be obtained, ensuring the timely updating of the knowledge graph.

[0051] Please see Figure 4 In some embodiments, step S220 may include, but is not limited to, step S410, step S420, or step S430: Step S410: If the data type is a structured type, generate a structured query statement based on the preset field extraction table, execute the structured query statement to extract fields from the updated metadata, and obtain the key fields. Step S420: If the data type is a semi-structured type, call the target extraction tool to parse the fields of the updated metadata to obtain key-value pairs, and use the key-value pairs as key fields. In step S430, if the data type is unstructured, the large language model is called to extract fields from the updated metadata to obtain the key fields.

[0052] In step S410 of some embodiments, a preset field extraction table defines the fields to be extracted and field extraction rules. The field extraction rules include field names, data sources, filtering conditions, and field aliases. If the data type is a structured type, an executable structured query statement is generated based on the fields to be extracted and the field extraction rules. The structured query statement is an SQL statement. Executing the structured query statement extracts fields from the updated metadata to obtain key fields. Since the updated metadata is a structured type, the key field can be directly converted into a node.

[0053] In step S420 of some embodiments, the target extraction tool may use Python's built-in XML tool. If the data type is semi-structured, the XML tool is called to parse the fields of the updated metadata to obtain initial fields. The initial fields are then converted into a knowledge graph-compatible key-value pair format using a format converter to obtain key-value pairs, which are then used as key fields.

[0054] In step S430 of some embodiments, if the data type is unstructured, the large language model is invoked, the updated metadata is input into the large language model for field extraction, and key fields are obtained, thereby converting the unstructured information into structured graph node attributes.

[0055] Steps S410 to S430 above perform differentiated processing on the three types of metadata: structured, semi-structured, and unstructured, to ensure that all types of data can be seamlessly integrated into the knowledge graph, thereby promoting data collaboration between different fields and different types of data and breaking down data silos.

[0056] Please see Figure 5 In some embodiments, step S230 may include, but is not limited to, steps S510 to S530: Step S510: Calculate the initial similarity between the key fields and the node fields; Step S520: Select the largest initial similarity as the reference similarity; Step S530: Update the knowledge graph based on reference similarity and key fields.

[0057] In step S510 of some embodiments, the child node has a node field, which is a field related to data attributes, data source, data processing logic, or data usage records. Based on a similarity calculation metric, the similarity between the key field and the node field is calculated to obtain an initial similarity. The similarity calculation metric can be cosine distance, Euclidean distance, etc. A higher initial similarity indicates a greater similarity between the key field and the node field.

[0058] In step S520 of some embodiments, after obtaining the initial similarity between the key field and the node field of each child node, the largest initial similarity is selected as the reference similarity.

[0059] In step S530 of some embodiments, the child nodes in the knowledge graph, the connecting edges connecting two child nodes, and the semantic edge categories of the connecting edges are updated based on reference similarity and key fields.

[0060] Through the above steps S510 to S530, the knowledge graph can be dynamically updated.

[0061] Please see Figure 6 In some embodiments, step S530 may include, but is not limited to, steps S610 to S630: Step S610: Compare the reference similarity with the preset similarity threshold; Step S620: If the reference similarity is greater than or equal to the preset similarity threshold, then the reference nodes are selected from the child nodes based on the reference similarity, the reference edges are selected from the connection edges based on the reference similarity, and the reference categories are selected from the semantic edge categories based on the reference similarity. Step S630: Update the reference node, reference edge, and reference category based on the key fields.

[0062] In step S610 of some embodiments, the key fields are segmented to obtain a first field related to data attributes, a second field related to data source, a third field related to data processing logic, and a fourth field related to data usage records. The reference similarity includes a first similarity, a second similarity, a third similarity, and a fourth similarity. The first similarity is the maximum initial similarity between the first field and the node field; the second similarity is the maximum initial similarity between the second field and the node field; the third similarity is the maximum initial similarity between the third field and the node field; and the fourth similarity is the maximum initial similarity between the fourth field and the node field. The reference similarity includes sub-similarity, which can be the first similarity, the second similarity, the third similarity, or the fourth similarity. The preset similarity threshold includes a preset threshold for the sub-similarity.

[0063] For each sub-similarity, compare the sub-similarity with a preset threshold to determine the relationship between the two.

[0064] In step S620 of some embodiments, if the sub-similarity is greater than or equal to a preset threshold, it indicates that the existing parameters of the knowledge graph are being updated. In this case, the child nodes corresponding to the node fields are obtained as reference nodes based on the sub-similarity. Reference edges are selected from the connection edges based on the sub-similarity; these reference edges are the connection edges that connect to the reference nodes. Reference categories are selected from the semantic edge categories based on the sub-similarity; these reference categories are the semantic edge categories of the reference edges.

[0065] In step S630 of some embodiments, the key fields, namely the first field, the second field, the third field, or the fourth field, are written as new attributes to the reference nodes, reference edges, and reference categories to update the reference nodes, reference edges, and reference categories, thereby achieving incremental updates to the knowledge graph rather than overall replacement.

[0066] Through steps S610 to S630, the existing parameters in the knowledge graph can be dynamically updated.

[0067] Please see Figure 7 In some embodiments, after step S620, the business data tracing method may also include, but is not limited to, steps S710 to S720: Step S710: If the reference similarity is less than the preset similarity threshold, then create a new node, a new edge connecting the new node, and a new category for the new edge based on the key fields. Step S720: Update the knowledge graph based on the newly added nodes, edges, and categories.

[0068] In step S710 of some embodiments, if the reference similarity is less than a preset similarity threshold, it means that the knowledge graph does not have parameters related to the key field. Then, a new node, a new edge connecting the new node, and a new category of the new edge are created based on the key field.

[0069] In step S720 of some embodiments, new nodes, new edges, and new categories are added to the knowledge graph to update the knowledge graph.

[0070] Through steps S710 to S720 above, the knowledge graph can be dynamically updated based on the newly added data.

[0071] In step S150 of some embodiments, when the data volume is large and the data flow is complex, existing tracing methods based on log recording systems or simple indexes require traversing a large number of records, resulting in low query efficiency. For example, a banking business system generates a large amount of transaction data every day. If the source of a certain wealth management product evaluation data is to be traced, it involves multiple stages such as user registration information, product listing information, and product transactions, which is time-consuming and cannot meet the business scenarios with high real-time requirements.

[0072] This application embodiment utilizes a knowledge graph for intuitive and comprehensive display, enabling rapid location of data sources and flow paths, significantly shortening tracing time and improving data tracing efficiency. Specifically, in response to a tracing request for target business data, the knowledge graph starts with the child node containing the target business data and traverses the connecting edges in reverse until a child node without connecting edges is reached, thus obtaining the data tracing link. The data tracing link is a link composed of attribute nodes, source nodes, processing nodes, record nodes, connecting edges, and semantic edge categories.

[0073] In step S160 of some embodiments, the target business data is traced along the data tracing link. Taking a payment scenario as an example, starting from the "payment failure" node, the data can be traced along the path of "payment basis → credit limit verification → identity authentication → identity data collection" to clarify key information such as the collection time, collection personnel, and verification agency of identity data, thereby tracing the source compliance when payment is abnormal.

[0074] Please see Figure 8 This application also provides a business data tracing device that can implement the above-described business data tracing method. The business data tracing device includes: The data acquisition module 810 is used to acquire the target metadata of the target business data; wherein, the target metadata includes data attributes, data source, data processing logic and data usage records; The nodeization module 820 is used to map data attributes, data sources, data processing logic, and data usage records to nodes to obtain target nodes; wherein, the target node includes child nodes, and the child nodes include attribute nodes of data attributes, source nodes of data sources, processing nodes of data processing logic, or record nodes of data usage records; The edge acquisition module 830 is used to acquire the connection edge between any two child nodes and the semantic edge category of the connection edge; Graph construction module 840 is used to construct a knowledge graph based on child nodes, connection edges, and semantic edge categories; The query module 850 is used to respond to the source tracing request for the target business data by traversing the connection edges in reverse in the knowledge graph to obtain the data source tracing link; wherein, the data source tracing link is a link composed of attribute nodes, source nodes, processing nodes, record nodes, connection edges and semantic edge categories; The traceability module 860 is used to trace the source of target business data based on the data traceability link.

[0075] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned business data traceability method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0076] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910 using the business data tracing method of the embodiments of this application. The input / output interface 930 is used to implement information input and output; The communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940); The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0077] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described business data tracing method.

[0078] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] The business data tracing method, business data tracing device, electronic device, and computer storage medium provided in this application use a knowledge graph to abstract business data and the many-to-many and multi-level flow relationships between business data into nodes and edges at once, forming a flat, depth-traversable graph index. Through the graph index, all dependencies, transformations, and usage relationships can be pre-stored as continuous edges, allowing any data point to complete multi-hop traversal along the index, achieving efficient and unbroken end-to-end tracing.

[0080] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0081] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0084] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0085] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0087] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for tracing business data, characterized in that, The method includes: Obtain target metadata of the target business data; wherein, the target metadata includes data attributes, data source, data processing logic, and data usage records; The data attributes, data sources, data processing logic, and data usage records are mapped to nodes to obtain target nodes; wherein, the target node includes child nodes, and the child nodes include attribute nodes of the data attributes, source nodes of the data sources, processing nodes of the data processing logic, or record nodes of the data usage records; Obtain the connection edge between any two child nodes and the semantic edge category of the connection edge; Construct a knowledge graph based on the child nodes, the connecting edges, and the semantic edge categories; In response to a tracing request for the target business data, the connection edges are traversed in reverse in the knowledge graph to obtain a data tracing link; wherein, the data tracing link is a link composed of the attribute node, the source node, the processing node, the record node, the connection edge, and the semantic edge category; The target business data is traced based on the data tracing link.

2. The method according to claim 1, characterized in that, After constructing the knowledge graph based on the child nodes, the connecting edges, and the semantic edge categories, the method further includes: Obtain the updated metadata and the data type of the updated metadata; wherein the updated metadata is obtained by updating the target metadata; Based on the data type, the updated metadata is subjected to field extraction to obtain key fields; The knowledge graph is updated based on the key fields.

3. The method according to claim 2, characterized in that, The data type includes structured, semi-structured, or unstructured types. The step of extracting fields from the updated metadata based on the data type to obtain key fields includes: If the data type is the structured type, then a structured query statement is generated according to the preset field extraction table, and the structured query statement is executed to extract fields from the updated metadata to obtain the key fields; If the data type is the semi-structured type, then the target extraction tool is called to parse the fields of the updated metadata to obtain key-value pairs, and the key-value pairs are used as the key fields; If the data type is the unstructured type, then the large language model is invoked to extract fields from the updated metadata to obtain the key fields.

4. The method according to claim 2, characterized in that, The child node has a node field, and updating the knowledge graph based on the key field includes: Calculate the initial similarity between the key field and the node field; The highest initial similarity is selected as the reference similarity; The knowledge graph is updated based on the reference similarity and the key fields.

5. The method according to claim 4, characterized in that, The knowledge graph includes child nodes, connecting edges between any two child nodes, and semantic edge categories of the connecting edges. Updating the knowledge graph based on the reference similarity and the key fields includes: Compare the reference similarity with the preset similarity threshold; If the reference similarity is greater than or equal to the preset similarity threshold, then reference nodes are selected from the child nodes based on the reference similarity, reference edges are selected from the connection edges based on the reference similarity, and reference categories are selected from the semantic edge categories based on the reference similarity. Update the reference node, the reference edge, and the reference category based on the key fields.

6. The method according to claim 5, characterized in that, After comparing the reference similarity with the preset similarity threshold, the method further includes: If the reference similarity is less than the preset similarity threshold, then a new node, a new edge connecting the new node, and a new category of the new edge are created based on the key field. The knowledge graph is updated based on the newly added nodes, newly added edges, and newly added categories.

7. The method according to claim 2, characterized in that, The acquisition of updated metadata and the data type of the updated metadata include: Obtain reference metadata; wherein the reference metadata is obtained by updating the target metadata, and the reference metadata has a first timestamp; The reference metadata is written into a preset storage space to obtain a second timestamp; wherein the second timestamp is greater than the first timestamp; Calculate the difference between the first timestamp and the second timestamp to obtain the delay duration; The updated metadata is obtained by filtering the reference metadata located in the preset storage space according to the delay duration. Obtain the data type of the updated metadata.

8. A business data traceability device, characterized in that, The device includes: The data acquisition module is used to acquire target metadata of the target business data; wherein, the target metadata includes data attributes, data source, data processing logic, and data usage records; A node mapping module is used to map the data attributes, data sources, data processing logic, and data usage records to nodes to obtain target nodes; wherein, the target node includes child nodes, and the child nodes include attribute nodes of the data attributes, source nodes of the data sources, processing nodes of the data processing logic, or record nodes of the data usage records; The edge acquisition module is used to acquire the connection edge between any two child nodes and the semantic edge category of the connection edge; The graph construction module is used to construct a knowledge graph based on the child nodes, the connecting edges, and the semantic edge categories. The query module is used to respond to a tracing request for the target business data by traversing the connection edges in reverse in the knowledge graph to obtain the data tracing link; wherein, the data tracing link is a link composed of the attribute node, the source node, the processing node, the record node, the connection edge, and the semantic edge category; The traceability module is used to trace the target business data according to the data traceability link.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images