Data interface blood relationship determination method and system and server

Abstract

The invention provides a data interface blood relationship determination method and system and a server, relates to the technical field of data processing, and can identify all sensitive API calling relationships involved in the process of circulation, aggregation and conversion of sensitive data among different APIs in a cross-system and multi-jump scene. And the sensitive API blood relationship graph can be generated in real time, so that the flow path of sensitive data passing through the API can be quickly and visually displayed, and unauthorized sensitive access or abnormal conversion can be quickly positioned.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, and server for determining the lineage of data interfaces. Background Technology

[0002] Sensitive data in cross-system, multi-hop scenarios requires complex transfer, aggregation, or transformation between different business systems via APIs. For example, user data may be provided from the core database to the risk control system via API, and then flow to the reporting platform; third-party services may obtain sensitive data through APIs and then generate new datasets. This cross-system, multi-hop transmission makes the spread path of sensitive data hidden, and existing static monitoring technologies are unable to cover the entire data chain, especially in scenarios with incomplete log coverage, making it difficult to obtain a complete lineage graph of sensitive data. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, and server for determining the lineage of data interfaces. This method can identify all sensitive API call relationships involved in the flow, aggregation, and transformation of sensitive data between different APIs in cross-system and multi-hop scenarios. It can also generate a sensitive API lineage graph in real time, enabling rapid visualization of the flow path of sensitive data through APIs and quickly locating unauthorized sensitive access or abnormal transformations, thereby solving the aforementioned problems in the prior art.

[0004] In a first aspect, embodiments of the present invention provide a method for determining the lineage of a data interface, the method comprising: After collecting data on network traffic events using traffic probes, identify sensitive API events contained in the network traffic events and store the sensitive API events in the ClickHouse database; The similarity results of sensitive API events are obtained from the ClickHouse database. The similarity results are used to determine the sensitive API lineage data corresponding to the sensitive API events, and the sensitive API lineage data is saved to a preset MySQL database. After detecting the user's query command using the preset interactive page, the sensitive API lineage data corresponding to the user's query command is obtained through the MySQL database; Use the kinship data from the sensitive API to determine the phylogenetic data corresponding to the user's query command, and send the phylogenetic data to the interactive page; The control interface generates a kinship graph corresponding to the user's query command based on the kinship data, and displays the kinship graph on the control interface.

[0005] Optionally, the steps of collecting network traffic events using traffic probes, identifying sensitive API events within the network traffic events, and storing these sensitive API events in the ClickHouse database include: The Agent probes that connect to the internal network and the external network respectively are identified as traffic probes, and the traffic probes are used to collect the corresponding network traffic events of the internal network and the external network respectively. Control network traffic events to be stored in a preset Kafka messaging system, and then perform data cleaning on the data stream corresponding to the network traffic events based on the Flink stream processing system corresponding to the Kafka messaging system to obtain the sensitive API events contained in the network traffic events; Retrieve the database storage data corresponding to the sensitive API event and save the database storage data to the preset ClickHouse database.

[0006] Optionally, the steps of obtaining similarity results for sensitive API events based on the ClickHouse database, determining the sensitive API lineage data corresponding to the sensitive API events using the similarity results, and saving the sensitive API lineage data to a preset MySQL database include: Retrieve the ClickHouse database, build a temporary data table based on the sensitive API events, and synchronize the sensitive API events to the temporary data table; Use a temporary data table to identify the target events corresponding to sensitive API events, and add the target events to the initialized simList collection; The query condition directives for the ClickHouse database are determined by the event parameters corresponding to the target event, and the event datasets corresponding to the sensitive API events in the ClickHouse database are retrieved using the query condition directives. Calculate the similarity results between sensitive API events and event datasets, use the maximum value in the similarity results to determine the sensitive API lineage data corresponding to the sensitive API events from the event dataset, and save the sensitive API lineage data to the simList collection; Control the conversion of sensitive API lineage data in the simList collection into lineage storage data, and control the saving of the lineage storage data to a preset MySQL database.

[0007] Optionally, a temporary data table can be used to determine the target event corresponding to the sensitive API event, including: Determine the batch synchronization parameters corresponding to the temporary data table, and determine the batch synchronization instructions corresponding to the temporary data table based on the batch synchronization parameters; After using batch synchronization commands to save the temporary data table to a preset queue to be analyzed, the temporary data table is cleared. The target event is determined based on the sensitive API events contained in the queue to be analyzed.

[0008] Optionally, calculate the similarity results between the sensitive API event and the event dataset, including: Calculate the similarity between each event in the event dataset and the sensitive API event; the similarity result is calculated using the following formula: ;in, The common word segmentation items between each event data and the corresponding text of the sensitive API event are weighted by type. The result is a weighted sum of all unique word segments corresponding to the text of each event data point and the sensitive API event. Similarity results; If the maximum value corresponding to the similarity calculation result meets the preset similarity threshold condition, then the similarity calculation result corresponding to the maximum value is determined as the similarity result.

[0009] Optionally, after detecting a user query command using a preset interactive page, the step of retrieving sensitive API lineage data corresponding to the user query command from a MySQL database includes: The system detects user query commands in real time based on the corresponding query page in the preset interactive page. When a user query command is detected on the query page, the interactive server corresponding to the interactive page generates a data request command corresponding to the user query command, and uses the data request command to control the interactive server to make a data request to the MySQL database. The control interaction server outputs sensitive API lineage data after receiving the data request command from the MySQL database response.

[0010] Optionally, the steps of using sensitive API lineage data to determine the phylogenetic data corresponding to the user's query command and sending the phylogenetic data to the interactive page include: The control interaction server converts sensitive API lineage data into graph data and associates the graph data with user query commands; The control interaction server sends the map data to the interaction page according to the user's query command.

[0011] Optionally, the steps of controlling the interactive page to generate a kinship graph corresponding to the user's query command based on the graph data, and controlling the interactive page to display the kinship graph, include: Retrieve the corresponding graph display page and bloodline evidence display page from the interactive page; By utilizing time data, address data, path data, and sensitive data contained in the graph data, we can determine the kinship evidence data corresponding to the user's query command, and generate a kinship graph corresponding to the user's query command based on the graph data. The control interaction page displays the bloodline evidence data on the bloodline evidence display page, and the control interaction page also displays the bloodline relationship graph on the graph display page.

[0012] Secondly, the present invention provides a data interface bloodline determination system, the system comprising: The initialization storage module is used to identify sensitive API events contained in network traffic events after data collection using traffic probes, and to store the sensitive API events in the ClickHouse database. The lineage relationship storage module is used to obtain the similarity results of sensitive API events based on the ClickHouse database, determine the sensitive API lineage relationship data corresponding to the sensitive API events using the similarity results, and save the sensitive API lineage relationship data to a preset MySQL database; The bloodline determination module is used to obtain the sensitive API bloodline data corresponding to the user query command through the MySQL database after detecting the user query command using the preset interactive page; The graph data determination module is used to determine the graph data corresponding to the user's query command using sensitive API kinship data, and then send the graph data to the interactive page; The bloodline display module is used to control the interactive page to generate a bloodline graph corresponding to the user's query command based on the graph data, and to control the interactive page to display the bloodline graph.

[0013] Thirdly, embodiments of the present invention also provide a server, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the steps of the data interface lineage determination method provided in the first aspect.

[0014] Fourthly, embodiments of the present invention also provide a storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the steps of the data interface lineage determination method provided in the first aspect.

[0015] This invention provides a method, system, and server for determining the lineage of data interfaces. In acquiring a lineage graph of sensitive APIs, the method first uses a traffic probe to collect data from network traffic events, identifies sensitive API events within these events, and stores these events in a ClickHouse database. Then, based on the ClickHouse database, it obtains similarity results for the sensitive API events, uses these results to determine the corresponding sensitive API lineage data, and saves this data to a preset MySQL database. Subsequently, after detecting a user query command using a preset interactive page, it retrieves the corresponding sensitive API lineage data from the MySQL database. Then, it uses this sensitive API lineage data to determine the graph data corresponding to the user query command and sends this graph data to the interactive page. Finally, it controls the interactive page to generate the lineage graph corresponding to the user query command based on the graph data and displays the graph. This solution can identify all sensitive API call relationships involved in the flow, aggregation, and transformation of sensitive data between different APIs in cross-system and multi-hop scenarios. It can also generate a sensitive API lineage graph in real time, enabling rapid visualization of the flow path of sensitive data through APIs and quickly locating unauthorized sensitive access or abnormal transformations.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for determining blood relations via a data interface, as provided in an embodiment of the present invention; Figure 2 This is a flowchart of step S101 in a data interface bloodline determination method provided in an embodiment of the present invention; Figure 3 This is a flowchart of step S102 in a data interface bloodline determination method provided in an embodiment of the present invention; Figure 4 In step S302 of the data interface lineage determination method provided in this embodiment of the invention, a flowchart is shown below for determining the target event corresponding to a sensitive API event using a temporary data table. Figure 5 A flowchart illustrating step S304 of a data interface lineage determination method provided in this embodiment of the invention, which calculates the similarity result between a sensitive API event and an event dataset; Figure 6 This is a flowchart of step S103 in a data interface bloodline determination method provided in an embodiment of the present invention; Figure 7 This is a flowchart of step S104 in a data interface bloodline determination method provided in an embodiment of the present invention; Figure 8 This is a flowchart of step S105 in a data interface bloodline determination method provided in an embodiment of the present invention; Figure 9 A flowchart illustrating another data interface bloodline determination method provided in an embodiment of the present invention; Figure 10 A flowchart of step S102 in another data interface bloodline determination method provided in an embodiment of the present invention; Figure 11 A logical diagram of the similarity algorithm involved in step S102 of a data interface blood relationship determination method provided in an embodiment of the present invention; Figure 12 This is a diagram illustrating the effect of a bloodline map obtained using a data interface bloodline determination method, as provided in an embodiment of the present invention. Figure 13 This is a schematic diagram of the structure of a data interface bloodline determination system provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention.

[0020] icon: 1310 - Initialization storage module; 1320 - Bloodline storage module; 1330 - Bloodline determination module; 1340 - Graph data determination module; 1350 - Bloodline display module; 101 - Processor; 102 - Memory; 103 - Bus; 104 - Communication interface. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Sensitive data (such as user ID numbers, mobile phone numbers, and license plate numbers) in cross-system, multi-hop scenarios requires complex flow, aggregation, or transformation between different business systems via APIs. For example, user data may be provided from the core database to the risk control system via API, and then flow to the reporting platform; third-party services may obtain sensitive data through APIs and generate new datasets. This cross-system, multi-hop transmission makes the spread path of sensitive data hidden, and existing static monitoring technologies are unable to cover the entire data chain, especially in scenarios with incomplete log coverage, making it difficult to obtain a complete lineage graph of sensitive data. Based on this, this invention provides a method, system, and server for determining the lineage of data interfaces. It can identify all sensitive API call relationships involved in the flow, aggregation, and transformation of sensitive data between different APIs in cross-system, multi-hop scenarios, and can generate a sensitive API lineage graph in real time. This enables rapid visualization of the flow path of sensitive data through APIs and can quickly locate unauthorized sensitive access or abnormal transformations, thereby solving the above-mentioned problems existing in the prior art.

[0023] To facilitate understanding of this embodiment, a method for determining the lineage of a data interface disclosed in this embodiment will first be described in detail, such as... Figure 1 As shown, the method includes: Step S101: After collecting data from network traffic events using a traffic probe, identify the sensitive API events contained in the network traffic events and store the sensitive API events in the ClickHouse database.

[0024] By utilizing traffic probes (such as those based on NetFlow or Packet Capture technologies) deployed at key network nodes, real-time collection of all API call traffic events is achieved. This includes information such as the API's source IP, target service address, request parameters, return data structure, and call timestamp. Sensitive API events are then filtered from the collected traffic events using a pre-defined sensitive rule base (e.g., API identification rules containing user privacy fields and core business data fields). These events are then structured and stored in a ClickHouse database in a unified format (e.g., API unique identifier, call context, data field list), leveraging ClickHouse's efficient write and time-series data processing capabilities to support subsequent analysis.

[0025] Step S102: Obtain the similarity results of sensitive API events based on the ClickHouse database, use the similarity results to determine the sensitive API lineage data corresponding to the sensitive API events, and save the sensitive API lineage data to a preset MySQL database.

[0026] Based on sensitive API events stored in the ClickHouse database, the similarity between sensitive API events is calculated from multiple dimensions (such as the overlap of API call parameter fields, the correlation of returned data fields, the consistency of call chain timing, and the correlation of the business modules to which they belong). Based on the similarity results (e.g., events with similarity exceeding a preset threshold are judged to have data dependencies), a lineage data structure for sensitive APIs is constructed (including direct call relationships, indirect data flows, field-level data transmission paths, etc.). This structured lineage data (such as source API, target API, relationship type, associated fields, time range, etc.) is stored in a preset MySQL database, leveraging its transaction support and relationship query capabilities to ensure data consistency.

[0027] Step S103: After detecting the user's query command using the preset interactive page, retrieve the sensitive API lineage data corresponding to the user's query command from the MySQL database.

[0028] When a pre-defined interactive page (such as a web-based visual interface) detects a user's query command (such as specifying an API name, business field, time range, or other query conditions), the backend service of the interactive page parses the query command, generates a corresponding SQL query statement, and retrieves sensitive API lineage data that matches the query conditions by accessing the MySQL database, thereby achieving accurate data extraction.

[0029] Step S104: Use the sensitive API lineage data to determine the graph data corresponding to the user's query command, and send the graph data to the interactive page.

[0030] Based on sensitive API lineage data obtained from a MySQL database, the data is converted into a graph data structure (such as JSON format, containing arrays of nodes and edges) according to graph data specifications (e.g., nodes are defined as API interfaces / data fields, edges are defined as data flow / call relationships, and attributes include sensitivity level, system affiliation, etc.). The graph data is then sent to the interactive page front-end to provide a data foundation for visualization.

[0031] Step S105: Control the interactive page to generate a kinship graph corresponding to the user's query command according to the graph data, and control the interactive page to display the kinship graph.

[0032] After receiving the graph data, the interactive front-end calls a visualization engine (such as a graph rendering tool based on D3.js or ECharts) to generate a lineage graph according to preset layout rules (such as force-directed layout, which aggregates closely related nodes). It also supports interactive functions (such as displaying details when clicking on nodes, highlighting paths, zooming, and dragging), and controls the interactive page to visualize the generated lineage graph, helping users intuitively understand the data lineage links between sensitive APIs.

[0033] Optionally, step S101, which involves collecting network traffic events using a traffic probe, identifying sensitive API events within the network traffic events, and storing these sensitive API events in the ClickHouse database, is as follows: Figure 2 As shown, it includes: Step S201: The Agent probes that connect to the internal network and the external network respectively are identified as traffic probes, and the traffic probes are used to collect the corresponding network traffic events of the internal network and the external network respectively.

[0034] The probe deployment process employs a distributed probe deployment model, deploying high-performance hardware probes (such as Netflow detectors) on the core switches of the internal network and lightweight software probes (such as the eBPF-based XDP program) on the external network boundary firewall. The internal and external network probes transmit the collected data to the central processing node through a dedicated channel (such as a GRE tunnel), forming a "distributed collection, centralized processing" architecture.

[0035] The probes involved in the data acquisition process can employ Deep Packet Inspection (DPI) technology, which can extract application layer metadata in addition to basic network information, including: API call chain: Microservice call relationships are constructed by parsing TraceID / SpanID; User behavior characteristics: Extracting session identifiers such as cookies and OAuth tokens; Data fingerprinting: Calculates the hash value of sensitive fields in the request / response body (such as ID number masking); Protocol adaptive parsing: For different API gateways (such as Kong, APISIX) and RPC frameworks (such as gRPC, Thrift), the probe has a built-in protocol parsing plugin library that supports dynamic loading to adapt to new protocols.

[0036] Step S202: Control the network traffic events to be stored in the preset Kafka message system, and perform data cleaning processing on the data stream corresponding to the network traffic events based on the Flink stream processing system corresponding to the Kafka message system to obtain the sensitive API events contained in the network traffic events.

[0037] During Kafka cluster optimization, a Tiered Storage architecture can be adopted, where hot data is retained for 7 days and cold data is automatically archived to S3 objects. A compression strategy (LZ4) can also be configured to reduce storage costs, and dedicated Topic partitions can be set up (e.g., 128 partitions for external network traffic and 64 partitions for internal network traffic).

[0038] The backend for the state involved in the Flink job uses RocksDB to store operator states, which can support the management of billions of event states; the watermarking mechanism is based on event time processing and allows a 2-second out-of-order tolerance; the rule engine integrates the Drools rule engine and supports complex rule combinations (such as "API path contains / user and the request method is POST").

[0039] The data cleaning process can uniformly convert different API formats into JSON representation, the anomaly filtering process can discard requests with status codes 404 / 500, and the field extraction process can extract sensitive field paths (such as $.user.password) from nested JSON.

[0040] Step S203: Obtain the database storage data corresponding to the sensitive API event and save the database storage data to the preset ClickHouse database.

[0041] The ClickHouse database is primarily used to store cleaned sensitive API events. The table engines involved in ClickHouse can be selected from the core fact table, specifically using the ReplacingMergeTree engine, which supports automatic deduplication of primary key conflicts; a dictionary table using the Distributed engine can also be selected to achieve distributed queries; or a historical archive table can be selected, using SummingMergeTree pre-aggregation to improve the performance of statistical queries.

[0042] Optionally, step S102 involves obtaining similarity results for sensitive API events from the ClickHouse database, using the similarity results to determine the sensitive API lineage data corresponding to the sensitive API events, and saving the sensitive API lineage data to a preset MySQL database. Figure 3 As shown, it includes: Step S301: Obtain the ClickHouse database, construct a temporary data table according to the sensitive API events, and synchronize the sensitive API events to the temporary data table.

[0043] When designing the table structure, a memory-based temporary table using the MergeTree engine can be created. This table contains raw fields such as API usage time, request parameters, and response fields, as well as intermediate fields for similarity calculation (such as parameter fingerprints and call time windows). Data synchronization strategies can utilize ClickHouse's Materialized View mechanism to incrementally synchronize sensitive API events from the past 7 days to the temporary table. A TTL of 24 hours can also be set to automatically clean up historical data. During index optimization, adaptive indexes can be created for frequently queried fields (such as API paths and timestamps) to improve the efficiency of subsequent join queries.

[0044] Step S302: Use a temporary data table to determine the target event corresponding to the sensitive API event, and add the target event to the initialized simList collection.

[0045] API events are weighted and sorted based on business impact and data sensitivity level, prioritizing core business APIs (such as user information queries and payment interfaces) to achieve priority ranking. Specifically, relevant clustering algorithms can be used. The DBSCAN algorithm is used to initially cluster events in a temporary table, grouping events with parameter similarity exceeding 80% into the same cluster. The earliest event in each cluster is selected as the target event. An incremental update mechanism can also be set up, maintaining a sliding time window (e.g., the most recent hour) and recalculating the cluster center as the new target event every 5 minutes.

[0046] Step S303: Determine the query condition command corresponding to the ClickHouse database through the event parameters corresponding to the target event, and use the query condition command to query the event dataset corresponding to the sensitive API event in the ClickHouse database.

[0047] This step primarily involves constructing query conditions. Specifically, it extracts features from the request parameters of the target event to generate parameter fingerprints (e.g., converting user_id=123 to a wildcard pattern of user_id=*). For the time window calculation process, the query window (e.g., ±5 minutes) can be dynamically calculated based on the timestamp of the target event, and the window size can be adaptively adjusted according to the API call frequency. The query condition construction process can combine parameter fingerprints, time windows, API paths, and other conditions to generate SQL query statements, using ClickHouse's JSON functions for nested field matching.

[0048] Step S304: Calculate the similarity results between the sensitive API event and the event dataset, use the maximum value in the similarity results to determine the sensitive API lineage data corresponding to the sensitive API event from the event dataset, and save the sensitive API lineage data to the simList collection.

[0049] This step mainly implements the similarity calculation and kinship determination process. The parameter similarity involved can be achieved by calculating the Jaccard similarity of the parameter fields; the temporal similarity can be obtained by calculating the similarity of the calling time series based on the DTW algorithm; and the structural similarity can be calculated by using edit distance to calculate the similarity of the response JSON structure.

[0050] During the maximum value screening process, for each target event, the top 5 events with the highest similarity scores can be retained as a candidate set of lineage relationships. The lineage relationship confirmation process can recursively confirm multi-level call relationships using a transitive closure algorithm, constructing a directed acyclic graph.

[0051] Step S305: Control the sensitive API lineage data in the simList collection to be converted into lineage storage data, and control the lineage storage data to be saved to the preset MySQL database.

[0052] During implementation, MySQL's LOAD DATA INFILE function can be used to convert the simList collection into a CSV stream for batch import, improving write performance. Additionally, nested transaction mechanisms can be employed to ensure data consistency between the API node table and the relational tables.

[0053] Optionally, a temporary data table can be used to determine the target event corresponding to the sensitive API event, such as... Figure 4 As shown, it includes: Step S401: Determine the batch synchronization parameters corresponding to the temporary data table, and determine the batch synchronization instructions corresponding to the temporary data table based on the batch synchronization parameters. Step S402: After using the batch synchronization command to control the temporary data table to be saved to the preset queue to be analyzed, the temporary data table is cleared. Step S403: Determine the target event based on the sensitive API events contained in the queue to be analyzed.

[0054] The acquisition of the target event can be achieved using a temporary data table T2 and an analysis queue Q. Specifically, the ClickHouse database corresponds to T1. The latest sensitive API event data T1 in ClickHouse is synchronized to table T2. Then, the data in table T2 is synchronized to the analysis queue Q in batches. At the same time, the synchronized data in table T2 is deleted. Finally, the sensitive API events contained in the analysis queue Q are used to determine the target event.

[0055] Optionally, calculate the similarity results between the sensitive API event and the event dataset, such as Figure 5 As shown, it includes: Step S501: Calculate the similarity between each event data in the event dataset and the sensitive API event; Step S502: If the maximum value corresponding to the similarity calculation result satisfies the preset similarity threshold condition, then the similarity calculation result corresponding to the maximum value is determined as the similarity result.

[0056] The similarity result is calculated using the following formula: ;in, The common word segmentation items between each event data and the corresponding text of the sensitive API event are weighted by type. The result is a weighted sum of all unique word segments corresponding to the text of each event data point and the sensitive API event. This is the similarity result. The algorithm iterates through each event and analyzes its similarity to the current sensitive API event. After iteration, the events are sorted from highest to lowest similarity, and the event with the highest similarity is retrieved. Then, it is determined whether the event with the highest similarity meets the configured similarity threshold; if it does, it is added to the `simList` collection.

[0057] Optionally, after detecting a user query command using a preset interactive page, step S103 involves retrieving the sensitive API lineage data corresponding to the user query command from a MySQL database, as follows: Figure 6 As shown, it includes: Step S601: Real-time detection of user query commands based on the corresponding query page in the preset interactive page.

[0058] This step is mainly used to realize real-time detection of user query commands. The interactive page design can adopt a layered query interface. The upper layer provides preset query templates (such as "user data leakage paths in the past 7 days"), and the lower layer supports custom combination query conditions (API name, time range, sensitivity level, etc.).

[0059] The instruction detection mechanism can employ real-time listening, establishing a long connection via WebSocket, and immediately pushing changes to the server when client input changes. In addition, the semantic analysis process can use ANTLR to parse query statements and identify entities (such as APIs and users) and relationships (such as calls and dependencies).

[0060] Step S602: After a user query command is detected on the query page, a data request command corresponding to the user query command is generated using the interactive server corresponding to the interactive page, and the data request command is used to control the interactive server to make a data request to the MySQL database.

[0061] This step primarily handles the generation and sending of data request commands, triggered by the user. When a user makes a data request on the query page, the query page detects the user's query command, then the corresponding interactive server generates a data request command and uses this command to send a data request to the MySQL database. This entire process is implemented on the interactive server associated with the user's UI.

[0062] Step S603: Control the interactive server to obtain the sensitive API lineage data output after receiving the MySQL database response data request command.

[0063] This step mainly implements the response data acquisition and processing process. Specifically, it can convert MySQL query results into GraphSON format and perform partial masking on the returned parameter values ​​(such as displaying the first 6 digits and last 4 digits of the ID number) to achieve sensitive data desensitization.

[0064] Optionally, the phylogenetic data from the sensitive API can be used to determine the phylogenetic data corresponding to the user's query command, and the phylogenetic data can be sent to step S104 on the interactive page, such as... Figure 7 As shown, it includes: Step S701: Control the interactive server to convert sensitive API lineage data into graph data, and associate the graph data with the user's query command; Step S702: Control the interactive server to send the map data to the interactive page according to the user's query command.

[0065] The above interaction process is implemented based on the interaction server. Specifically, the interaction server converts sensitive API lineage data into graph data, associates the graph data with the user's query command, and then controls the interaction server to send the graph data to the interaction page according to the associated user query command.

[0066] Optionally, step S105 involves controlling the interactive page to generate a kinship graph corresponding to the user's query command based on the graph data, and controlling the interactive page to display the kinship graph. Figure 8 As shown, it includes: Step S801: Obtain the corresponding graph display page and bloodline evidence display page from the interactive page; Step S802: Use the time data, address data, path data and sensitive data contained in the graph data to determine the bloodline evidence data corresponding to the user query command, and generate the bloodline relationship graph corresponding to the user query command based on the graph data; Step S803: Control the interactive page to display the bloodline evidence data on the bloodline evidence display page, and control the interactive page to display the bloodline relationship map on the map display page.

[0067] In addition to the genealogy map display page, the interactive interface also includes a kinship evidence display page to show kinship evidence data. Kinship evidence data is determined using time, address, path, and sensitive data contained in the genealogy map. After obtaining the kinship evidence data and the kinship genealogy map, they are displayed on the kinship evidence display page and the genealogy map display page, respectively.

[0068] like Figure 9 The flowchart shown is for another data interface method for determining blood relations, and the components involved are as follows: Kafka: Stores raw API traffic events reported by the Agent (traffic collection probe); ClickHouse: Stores sensitive API events after cleaning; MySQL: Stores sensitive API relationship graphs and configures sensitive API lineage identification thresholds; ETL: Data Cleaning. This involves consuming raw API traffic event data from Kafka using Flink to normalize network traffic events and identify sensitive API events. Alarm: Collects sensitive API events and identifies sensitive API lineage events that meet a threshold similarity. Manager: Processes sensitive API lineage data into a display data format; UI: Displays the lineage graph of sensitive APIs.

[0069] Specifically, Agant acts as a traffic acquisition probe, collecting raw traffic events from both internal and external networks and reporting them to Kafka for storage. The ETL application consumes Kafka messages, normalizes the raw traffic events, and identifies sensitive API events using regular expressions. These sensitive API events are then stored in ClickHouse. Subsequently, the Alarm application retrieves the sensitive API events stored in ClickHouse and performs similarity analysis according to the identification algorithm. The Alarm application then queries ClickHouse based on the identification algorithm's query logic to find a batch of sensitive API events that meet the criteria. Furthermore, the Alarm application stores the identified sensitive API lineage data in a MySQL database. When a user queries the relevant UI of the sensitive API lineage system, the UI calls the Manager application to obtain the sensitive API lineage graph data. The Manager application retrieves the sensitive API lineage data stored in MySQL and processes it into displayable graph data.

[0070] Specifically, another flowchart for step S102, which involves obtaining similarity results of sensitive API events from the ClickHouse database, using these similarity results to determine the sensitive API lineage data corresponding to the sensitive API events, and saving the sensitive API lineage data to a preset MySQL database, is as follows: Figure 10 As shown. First, the latest sensitive API event data T1 in ClickHouse is synchronized to table T2; then, the data in table T2 is synchronized in batches to the analysis queue Q, while the synchronized data in that batch is deleted from table T2; if there is no data to synchronize in table T2, the identification program ends; if the analysis queue Q is not empty, a sensitive API event E1 is obtained and added to the simList collection. If the analysis queue Q is empty, the data in table T2 is synchronized in batches to the analysis queue Q again for cyclic processing.

[0071] Using the target IP of the current E1 event as the source IP, and the event time being greater than or equal to the E1 event time as the query condition, the system queries the ClickHouse sensitive API event data T1, with the query result being R. The system checks if the query result R is empty. If not, a modified "Jaccard" similarity algorithm is used to iterate through and analyze the similarity between each event content and the current sensitive API event E1. After iteration, the events are sorted from highest to lowest similarity, and the event with the highest similarity is retrieved. Then, it checks if the event with the highest similarity meets the configured similarity threshold. If it meets the threshold, it is added to the simList collection. If not, the simList collection performs count, duplication, and whitelist checks on similar events. If the checks pass, the simList data is converted into a saveList for storing lineage data. If the checks fail, the simList data is discarded. The saveList data is then stored in a MySQL database, and the simList and saveList collections are reset. A schematic diagram of the logic of the similarity algorithm is shown below. Figure 11 As shown, the specifics will not be elaborated further.

[0072] This method collects network traffic events using an agent (traffic probe), then cleans the data using an ETL application (traffic event normalization, sensitive event regular expression matching), and stores the cleaned sensitive API events in a ClickHouse database. The Alarm application then identifies the lineage of sensitive APIs based on sensitive API similarity rules and algorithms and stores this information in MySQL. The Manager and UI then display the identified sensitive API lineage graph to the user. This method can quickly identify sensitive API calls and generate a sensitive API lineage graph. The effect of the lineage graph is shown in the following figure. Figure 12 As shown.

[0073] As can be seen from the data interface lineage determination method in the above embodiments, this method can identify all sensitive API call relationships involved in the process of sensitive data flowing, aggregating, and transforming between different APIs in cross-system and multi-hop scenarios, and can generate a sensitive API lineage graph in real time, realizing a rapid visual display of the flow path of sensitive data through APIs, and can quickly locate unauthorized sensitive access or abnormal transformation.

[0074] Corresponding to the above-described method for determining the lineage of data interfaces, this embodiment of the invention also provides a system for determining the lineage of data interfaces, such as... Figure 13 As shown, the system includes: The storage module 1310 is initialized to identify sensitive API events contained in the network traffic events after the network traffic events are collected by the traffic probe, and to store the sensitive API events in the ClickHouse database. The lineage relationship storage module 1320 is used to obtain the similarity results of sensitive API events based on the ClickHouse database, determine the sensitive API lineage relationship data corresponding to the sensitive API events using the similarity results, and save the sensitive API lineage relationship data to a preset MySQL database; The bloodline determination module 1330 is used to obtain the sensitive API bloodline data corresponding to the user query command through the MySQL database after detecting the user query command using the preset interactive page; The graph data determination module 1340 is used to determine the graph data corresponding to the user's query command using sensitive API kinship data, and send the graph data to the interactive page; The bloodline display module 1350 is used to control the interactive page to generate a bloodline graph corresponding to the user's query command according to the graph data, and to control the interactive page to display the bloodline graph.

[0075] As can be seen from the above data interface lineage determination system, the system can identify all sensitive API call relationships involved in the flow, aggregation, and transformation of sensitive data between different APIs in cross-system and multi-hop scenarios, and can generate a sensitive API lineage graph in real time. It can quickly visualize the flow path of sensitive data through APIs and quickly locate unauthorized sensitive access or abnormal transformation.

[0076] The data interface lineage determination system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned data interface lineage determination method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned data interface lineage determination method embodiment.

[0077] This embodiment also provides a server, the structural diagram of which is shown below. Figure 14 As shown, the device includes a processor 101 and a memory 102; wherein, the memory 102 is used to store one or more computer instructions, which are executed by the processor to implement the steps of the above-described data interface lineage determination method.

[0078] Figure 14 The server shown also includes a bus 103 and a communication interface 104. The processor 101, the communication interface 104, and the memory 102 are connected via the bus 103.

[0079] The memory 102 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. The bus 103 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 14 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0080] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and to send encapsulated IPv4 packets or IPv4 packets to the user terminal through the network interface.

[0081] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. The processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 102. The processor 101 reads the information in memory 102 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0082] This invention also provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the data interface lineage determination method described in the foregoing embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, devices, and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0086] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data interface blood relationship determination method, characterized in that, The method comprises: After data collection on network traffic events by a traffic probe, sensitive API events contained in the network traffic events are determined, and the sensitive API events are stored in a ClickHouse database; Based on the ClickHouse database, a similarity result of the sensitive API events is obtained, the sensitive API blood relationship data corresponding to the sensitive API events is determined by using the similarity result, and the sensitive API blood relationship data is saved to a preset Mysql database; After detecting a user query instruction by using a preset interactive page, the sensitive API blood relationship data corresponding to the user query instruction is obtained through the Mysql database; The sensitive API blood relationship data is used to determine the graph data corresponding to the user query instruction, and the graph data is sent to the interactive page; The interactive page is controlled to generate a blood relationship graph corresponding to the user query instruction according to the graph data, and the interactive page is controlled to display the blood relationship graph.

2. The data interface kinship determination method of claim 1, wherein, After data collection on network traffic events by a traffic probe, sensitive API events contained in the network traffic events are determined, and the sensitive API events are stored in a ClickHouse database, comprising: The Agent probe connected to the internal network and the external network is determined as the traffic probe, and the network traffic events corresponding to the internal network and the external network are collected by using the traffic probe; The network traffic events are stored in a preset Kafka message system, and after data cleaning processing of the data stream corresponding to the network traffic events by a Flink stream processing system corresponding to the Kafka message system, the sensitive API events contained in the network traffic events are obtained; The database storage data corresponding to the sensitive API events is obtained, and the database storage data is saved to a preset ClickHouse database.

3. The data interface kinship determination method of claim 1, wherein, Based on the ClickHouse database, a similarity result of the sensitive API events is obtained, the sensitive API blood relationship data corresponding to the sensitive API events is determined by using the similarity result, and the sensitive API blood relationship data is saved to a preset Mysql database, comprising: A temporary data table is constructed in the ClickHouse database according to the sensitive API events, and the sensitive API events are synchronized to the temporary data table; The target event corresponding to the sensitive API events is determined by using the temporary data table, and the target event is added to an initialized simList set; The query condition instruction corresponding to the ClickHouse database is determined through the event parameters corresponding to the target event, and the event data set corresponding to the sensitive API events in the ClickHouse database is queried by using the query condition instruction; calculate a similarity result between the sensitive API event and the event data set, determine sensitive API blood relationship data corresponding to the sensitive API event from the event data set by using a maximum value corresponding to the similarity result, and save the sensitive API blood relationship data to the simList set; control the sensitive API blood relationship data in the simList set to be converted into blood relationship storage data, and control the blood relationship storage data to be saved to the preset Mysql database.

4. The data interface kinship determination method of claim 3, wherein, determine a target event corresponding to the sensitive API event by using the temporary data table, including: determine a batch synchronization parameter corresponding to the temporary data table, and determine a batch synchronization instruction corresponding to the temporary data table according to the batch synchronization parameter; after saving the temporary data table to the preset to-be-analyzed queue by using the batch synchronization instruction, clear the temporary data table; determine the target event according to the sensitive API event contained in the to-be-analyzed queue.

5. The data interface kinship determination method of claim 3, wherein, calculate a similarity result between the sensitive API event and the event data set, including: calculating a similarity result between each event data in the event data set and the sensitive API event; the similarity result is calculated by the following formula: ; wherein, is a weighting result of the common segmentation items of the corresponding text between each event data and the sensitive API event according to the type; is a weighted sum result of all unique segmentation items of the corresponding text between each event data and the sensitive API event; is the similarity result; if a maximum value corresponding to the similarity calculation result satisfies a preset similarity threshold condition, determine the similarity calculation result corresponding to the maximum value as the similarity result.

6. The data interface kinship determination method of claim 1, wherein, after detecting a user query instruction by using a preset interactive page, the step of obtaining the sensitive API blood relationship data corresponding to the user query instruction through the Mysql database, including: detect the user query instruction in real time according to a corresponding query page in the preset interactive page; after detecting the user query instruction in the query page, generate a data request instruction corresponding to the user query instruction by using an interactive service corresponding to the interactive page, and control the interactive service to perform data request on the Mysql database by using the data request instruction; control the interactive service to obtain the sensitive API blood relationship data output by the Mysql database in response to the data request instruction.

7. The data interface kinship determination method of claim 6, wherein, determine graph data corresponding to the user query instruction by using the sensitive API blood relationship data, and send the graph data to the interactive page, including: control the interactive service to convert the sensitive API blood relationship data into the graph data, and associate the graph data with the user query instruction; control the interactive service to send the graph data to the interactive page according to the user query instruction.

8. The data interface kinship determination method of claim 6, wherein, control the interactive page to generate a blood relationship graph corresponding to the user query instruction according to the graph data, and control the interactive page to display the blood relationship graph, including: obtain a graph display page and a blood evidence display page corresponding to the interactive page; determine blood evidence data corresponding to the user query instruction by using time data, address data, path data and sensitive data contained in the graph data, and generate a blood relationship graph corresponding to the user query instruction based on the graph data; The control page displays the bloodline forensics data in the bloodline forensics display page, and controls the interactive page to display the blood relationship graph in the graph display page.

9. A data interface blood relationship determination system, characterized by, The system comprises: An initialization storage module is configured to determine sensitive API events contained in network traffic events after data collection by a traffic probe, and store the sensitive API events in a ClickHouse database; A blood relationship storage module is configured to obtain a similarity result of the sensitive API events based on the ClickHouse database, determine sensitive API blood relationship data corresponding to the sensitive API events by using the similarity result, and save the sensitive API blood relationship data in a preset Mysql database; A blood relationship determination module is configured to obtain the sensitive API blood relationship data corresponding to a user query instruction through the Mysql database after detecting the user query instruction by using a preset interactive page; A graph data determination module is configured to determine graph data corresponding to the user query instruction by using the sensitive API blood relationship data, and send the graph data to the interactive page; A blood relationship display module is configured to control the interactive page to generate a blood relationship graph corresponding to the user query instruction according to the graph data, and control the interactive page to display the blood relationship graph.

10. A server, characterized by The processor executes the computer executable instructions to implement the steps of the data interface blood relationship determination method in any one of claims 1 to 8.

Images