Method, device and equipment for determining relationship of log data, medium and product

Abstract

The invention provides a log data relation determination method and device, equipment, a medium and a product. The method comprises the steps of obtaining a log operation data set; wherein the log operation data set comprises a plurality of operation data, and each operation data corresponds to a source path, a target path and context information; performing screening processing on the operation data in the log operation data set to obtain a plurality of corresponding target operation data; based on the context information, the multiple pieces of target operation data are analyzed, and an application program ID corresponding to each piece of target operation data is obtained; for any target operation data in the multiple pieces of target operation data, determining a source data table corresponding to a source path of the target operation data and a target data table corresponding to a target path; and generating a corresponding data relation graph based on the source data tables, the target data tables and the application program IDs corresponding to the multiple pieces of target operation data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, medium and product for determining the relationship of log data. Background Technology

[0002] During the data warehouse processing, a large number of historical processing logs are generated. In addition to helping troubleshoot problems in the processing chain, these processing logs can also uncover the upstream and downstream relationships between data in different fields and layers of the data warehouse.

[0003] Currently, the methods for extracting data relationships from historical processing logs are to control the data processing task submission platform or to embed tracking points in the processing task program. The data relationships are then extracted through secondary processing based on the platform's processing logs or the data obtained through tracking points. However, during secondary processing, the incomplete extraction of lineage relationships is easily caused by inconsistent log formats.

[0004] Therefore, this method cannot extract the full data relationships from the historical processing logs. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for determining the relationships in log data, so as to achieve the effect of extracting the full data relationships in historical processing logs.

[0006] In a first aspect, embodiments of this application provide a method for determining the relationship of log data, comprising: obtaining a log operation data set; wherein the log operation data set includes multiple operation data, each operation data corresponding to a source path, a target path, and context information; filtering the operation data within the log operation data set to obtain multiple target operation data; parsing the multiple target operation data based on the context information to obtain an application ID corresponding to each target operation data; for any one of the multiple target operation data, determining the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path; and generating a corresponding data relationship graph based on the source data table, target data table, and application ID corresponding to the multiple target operation data.

[0007] In one possible implementation, a corresponding data relationship graph is generated based on the source data table, target data table, and application ID corresponding to multiple target operation data. This includes: determining the corresponding application based on the application ID corresponding to the multiple target operation data; establishing a relationship mapping between the application ID and the data table based on the source data table, target data table, and application ID corresponding to the multiple target operation data, generating multiple corresponding table-level data relationship graphs; and generating the corresponding data relationship graph based on the corresponding application and the multiple corresponding table-level data relationship graphs.

[0008] In one possible implementation, a mapping between application IDs and data tables is established to generate multiple corresponding table-level data relationship graphs, including: for any application ID, constructing an input-output link based on the application ID and the corresponding target operation data; establishing a relationship mapping based on the input-output link, the source data table, and the target data table to generate a table-level data relationship graph corresponding to the application ID.

[0009] In one possible implementation, the log operation data set further includes: an operation type corresponding to each operation data; filtering the operation data within the log operation data set to obtain multiple target operation data, including: filtering the log operation data set based on the operation type corresponding to each operation data; removing operation data in the log operation data set that does not conform to a predefined operation type to obtain multiple target operation data.

[0010] In one possible implementation, determining the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path includes: simplifying the source path and target path of the target operation data to obtain simplified path information; wherein, the simplified path information is data table identification information from the source path to the target path; and extracting the simplified path information using regular expressions to obtain the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path.

[0011] In one possible implementation, determining the corresponding application based on the application ID corresponding to multiple target operation data includes: obtaining historical processing logs corresponding to multiple target operation data; for any one of the multiple target operation data, determining the application corresponding to the application ID based on the application ID corresponding to the target operation data and the historical processing logs.

[0012] Secondly, embodiments of this application provide an apparatus for determining the relationship of log data, comprising: an acquisition module for acquiring a log operation data set; wherein the log operation data set includes multiple operation data, each operation data corresponding to a source path, a target path, and context information; a processing module for filtering the operation data within the log operation data set to obtain multiple corresponding target operation data; the processing module is further configured to parse the multiple target operation data based on the context information to obtain an application ID corresponding to each target operation data; the processing module is further configured to determine, for any one of the multiple target operation data, the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path; and a generation module for generating a corresponding data relationship graph based on the source data table, target data table, and application ID corresponding to the multiple target operation data.

[0013] In one possible implementation, the generation module is further configured to: determine the corresponding application based on the application IDs corresponding to multiple target operation data; establish a relationship mapping between the application IDs and the data tables based on the source data tables, target data tables, and application IDs corresponding to multiple target operation data, and generate multiple corresponding table-level data relationship graphs; and generate a corresponding data relationship graph based on the corresponding application and the multiple corresponding table-level data relationship graphs.

[0014] In one possible implementation, the generation module is further configured to: construct an input / output link based on the application ID and the corresponding target operation data for any application ID; and establish a relationship mapping based on the input / output link, the source data table, and the target data table to generate a table-level data relationship graph corresponding to the application ID.

[0015] In one possible implementation, the processing module is further configured to: filter the log operation data set based on the operation type corresponding to each operation data; remove operation data in the log operation data set that does not conform to the predefined operation type, and obtain multiple target operation data.

[0016] In one possible implementation, the processing module is further configured to: simplify the source path and target path of the target operation data to obtain simplified path information; wherein the simplified path information is data table identification information from the source path to the target path; and extract the simplified path information using regular expressions to obtain the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path.

[0017] In one possible implementation, the processing module is further configured to: obtain historical processing logs corresponding to multiple target operation data; and for any one of the multiple target operation data, determine the application corresponding to the application ID based on the application ID corresponding to the target operation data and the historical processing logs.

[0018] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0019] The memory stores instructions that the computer executes;

[0020] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0022] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0023] This application provides a method, apparatus, device, medium, and product for determining the relationships of log data. It obtains multiple target operation data by acquiring and filtering log operation datasets; parses the target operation data based on context information to obtain the application ID corresponding to the operation data; processes the source and target paths of the target operation data to obtain corresponding source and target data tables; and generates a corresponding data relationship graph based on the obtained application ID, source data table, and target data table. This solution parses target operation data using context information, is compatible with operation data of different formats, and reduces the problems of missed and incorrect judgments compared to manual secondary processing of logs. It can extract all data relationships from historical processing logs. Simultaneously, based on the application ID and path-processed data tables obtained from context information parsing, a data relationship graph is constructed, enabling full-link visualization of the data's origin, processing process, and final destination, completely displaying the flow relationships between all data. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] Figure 1A flowchart illustrating a method for determining the relationship of log data provided in this application;

[0026] Figure 2 This application provides a table-level data relationship graph;

[0027] Figure 3 A data relationship graph provided for this application;

[0028] Figure 4 A schematic diagram of the structure of a device for determining the relationship of log data provided in this application;

[0029] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0030] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device. As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0034] During the data warehouse processing, a large number of historical processing logs are generated. In addition to helping troubleshoot problems in the processing chain, these processing logs can also uncover the upstream and downstream relationships between data in different fields and layers of the data warehouse.

[0035] Currently, the methods for extracting data relationships from historical processing logs rely on two "pre-constraints." The first is to control the submission platform of data processing tasks, which requires all data processing tasks to be submitted and executed through a unified scheduling platform. The platform will automatically record the metadata of input and output data during task execution and generate processing logs. The second is to pre-embed points in the processing task program, which involves embedding specific log output statements in the data processing code and obtaining data flow relationships by parsing these embedded logs.

[0036] However, extracting data relationships requires secondary processing of the generated processing logs. But for historical processing tasks prior to the implementation of a unified platform or event tracking mechanism, the generated processing logs lack a standardized format, making it impossible to parse out valid data relationships through secondary processing. Furthermore, if the processing logs generated based on these two pre-existing constraints have inconsistent formats, it's also impossible to extract all the data relationships present in the processing logs. Therefore, the current data relationship extraction method cannot extract all data relationships from historical processing logs.

[0037] To address the technical problem of existing technologies failing to extract full data relationships from historical processing logs, this application provides a method for determining log data relationships. This method involves acquiring and filtering log operation datasets to obtain multiple target operation data sets. The target operation data is then parsed based on contextual information to obtain the corresponding application ID. The source and target paths of the target operation data are processed to obtain corresponding source and target data tables. A corresponding data relationship graph is generated based on the obtained application ID, source data table, and target data table. This solution parses target operation data using contextual information, is compatible with operation data of different formats, and reduces the problems of missed or incorrect judgments compared to manual log reprocessing. It can extract full data relationships from historical processing logs. Furthermore, by constructing a data relationship graph based on the application ID and path-processed data tables obtained from contextual information parsing, the origin, processing, and final destination of the data are visualized throughout the entire process, fully demonstrating the flow relationships between all data.

[0038] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0039] Example 1

[0040] Figure 1 A flowchart illustrating a method for determining relationships in log data provided in this application is shown below. Figure 1 As shown, the method for determining the relationship of log data provided in this embodiment includes:

[0041] S101, Obtain the log operation data set; wherein, the log operation data set includes multiple operation data, each operation data corresponding to a source path, a target path and context information;

[0042] The log operation data set is obtained from the historical logs of the storage system. These historical logs are collections of operation and status information automatically recorded by storage devices (such as hard drives, disk arrays, distributed storage systems, and cloud storage services) during operation, encompassing the entire lifecycle of the storage system from initialization, data reading and writing, device interaction to failure. Operation data, including source path, destination path, and context information, is obtained from these historical logs.

[0043] Specifically, the source path refers to the source identifier of a data table, field, or query statement, used to locate "where the raw data is obtained"; the target path refers to the destination identifier of the target table, view, or stored procedure, used to locate "where the processing results are stored". The source path and target path form an "input-output" correspondence. Context information refers to all the environment, conditions, and related data surrounding the operation from the source path to the target path, used to supplement the background details of the operation to ensure that the complete logic of the operation can be understood.

[0044] S102, filter the operation data within the log operation data set to obtain multiple target operation data;

[0045] The data warehouse processing involves various types of operations, such as open, create, rename, delete, copy, and move operations. Therefore, the resulting log operation data set contains a large amount of operational data. However, during the extraction and processing of data relationships, a significant amount of irrelevant or secondary operational data is present. By filtering and processing this data, multiple target operational data relevant to data relationship extraction are obtained, thereby reducing data redundancy and improving processing efficiency.

[0046] S103, based on context information, parse and process multiple target operation data to obtain the application ID corresponding to each target operation data;

[0047] The application ID refers to a string or numeric code used to uniquely identify an application; the context information typically contains multi-dimensional clues related to the application. For example, for any target operation data 1, given source path 1 and target path 1, based on the context information, the processing from source path 1 to target path 1 is through application ID1=100 and application ID2=200; similarly, target operation data 2 corresponds to application ID1=100, target operation data 3 corresponds to application ID1=100 and application ID3=300, target operation data 4 corresponds to application ID2=200 and application ID4=400, and target operation data 5 corresponds to application ID5=500.

[0048] S104, for any one of the multiple target operation data, determine the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path;

[0049] The path contains key identification information of the data table; for example, if the target operation data is "to summarize the original order data according to the grouping of 'user ID + date'", the source path is "original order data" and the target path is "summary data", thus it can be determined that the source data table is "original order data table" and the target data table is "summary data table".

[0050] S105 generates a corresponding data relationship graph based on the source data table, target data table, and application ID corresponding to multiple target operation data.

[0051] This involves establishing a relationship link between "source data table - application ID - target data table" and generating a corresponding data relationship graph based on the relationship links of multiple target operation data.

[0052] In practical applications, the generated data relationship graph is stored in a distributed data warehouse. Through the distributed storage, structured query statement compatibility, and multi-scenario integration capabilities of the distributed data warehouse, the storage, efficient querying, and deep reuse of data relationships can be achieved.

[0053] This application provides a method for determining relationships in log data. It involves acquiring and filtering log operation datasets to obtain multiple target operation data sets. The target operation data is then parsed based on contextual information to obtain the corresponding application ID. The source and target paths of the target operation data are processed to obtain corresponding source and target data tables. A corresponding data relationship graph is generated based on the obtained application ID, source table, and target table. This solution parses target operation data using contextual information, is compatible with operation data of different formats, and reduces the problems of missed or incorrect judgments compared to manual log reprocessing. It can extract all data relationships from historical processing logs. Furthermore, the data relationship graph is constructed based on the application ID and path-processed data tables obtained from contextual information parsing, enabling full-link visualization of the data's origin, processing, and final destination, completely displaying the flow relationships between all data.

[0054] In one possible implementation, a corresponding data relationship graph is generated based on the source data table, target data table, and application ID corresponding to multiple target operation data, including:

[0055] The corresponding application is determined based on the application ID corresponding to multiple target operation data.

[0056] Based on the source data table, target data table and application ID corresponding to multiple target operation data, establish a relationship mapping between application ID and data table, and generate multiple corresponding table-level data relationship graphs.

[0057] Generate the corresponding data relationship graph based on the corresponding application and multiple corresponding table-level data relationship graphs.

[0058] An application comprises multiple applications, and each application ID contains characteristic information related to its respective application. Referring to the previous example, we can determine that application ID1, application ID3, and application ID5 belong to application A, while application ID2 and application ID4 belong to application B. Based on the source data table, target data table, and application IDs corresponding to multiple target operation data, a mapping between application IDs and data tables is established, generating multiple corresponding table-level data relationship graphs. For example, "Source Data Table 2 - Application ID1 - Target Data Table 2".

[0059] Application ID1, Application ID3, and Application ID5 belong to application A, while Application ID2 and Application ID4 belong to application B. Therefore, the application-table relationship graph can be generated from the application-table relationship graph. For example, Source Data Table 1 - Application A - Target Data Table 1.

[0060] In one possible implementation, a mapping between application IDs and data tables is established, generating multiple corresponding table-level data relationship graphs, including:

[0061] For any given application ID, construct an input / output chain based on the application ID and the corresponding target operation data;

[0062] Establish a relationship mapping based on the input / output links, source data tables, and target data tables to generate a table-level data relationship graph corresponding to the application ID.

[0063] Based on the above examples, the target operation data corresponding to application ID1 includes target operation data 1, target operation data 2, and target operation data 3; the target operation data corresponding to application ID2 includes target operation data 1 and target operation data 4; the target operation data corresponding to application ID3 includes target operation data 3; the target operation data corresponding to application ID4 includes target operation data 4; and the target operation data corresponding to application ID5 includes target operation data 5.

[0064] Figure 2 This application provides a table-level data relationship graph, such as... Figure 2 As shown, the input / output links constructed for application ID1 include: branch 1 "source data table A1-ID1-ID2", branch 2 "source data table B2-ID1-target data table b2", and branch 3 "source data table C3-ID1-ID3". Similarly, the links for application ID2 include: branch 1 "ID1-ID2-target data table a1", and branch 2 "source data table D4-ID2-ID4"; the links for application ID3 include: branch 1 "ID1-ID3-target data table C3"; the links for application ID4 include: branch 1 "ID2-ID4-target data table d4"; and the links for application ID5 include: branch 1 "source data table E5-ID5-target data table e5".

[0065] Figure 3 A data relationship graph provided for this application, such as Figure 3 As shown, since application ID1, application ID3, and application ID5 belong to application A, and application ID2 and application ID4 belong to application B, the input / output links constructed for application A include: main line 1 "source data table A1 - application A - application B", branch line 2 "source data table B2 - application A - target data table b2", branch line 3 "source data table C3 - application A - target data table c3", and branch line 4 "source data table E5 - application A - target data table e5". Similarly, the links for application B include: branch line 1 "application A - application B - target data table a1" and branch line 2 "source data table D4 - application B - target data table d4".

[0066] By constructing a complete input-output path chain with the application ID as the main thread, the transformation dependencies between multiple tables can be identified.

[0067] In one possible implementation, the log operation data set further includes: an operation type corresponding to each operation data; filtering the operation data within the log operation data set to obtain multiple corresponding target operation data, including:

[0068] Based on the operation type corresponding to each operation data, the log operation data set is filtered and processed;

[0069] Remove operation data from the log operation data set that does not conform to the predefined operation type to obtain multiple target operation data.

[0070] The log operation data obtained also includes the corresponding operation type.

[0071] Predefined operation types are set according to requirements; for example, predefined operation types can be set as open, create, and rename. Based on the operation type corresponding to each operation data, the log operation data set is filtered; operation data that does not conform to the predefined operation types are removed from the log operation data set, resulting in multiple target operation data.

[0072] For example, the log operation data set includes operation data 1, operation data 2, operation data 3, operation data 4, and operation data 5; among them, operation data 1 is a delete operation, operation data 2 is an open operation, operation data 3 is a create operation, operation data 4 is an update operation, and operation data 5 is a rename operation; based on the predefined operation types, operation data 1 and operation data 4 are removed, and the target operation data is operation data 2, operation data 3, and operation data 5.

[0073] In one possible implementation, determining the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path includes:

[0074] The source and target paths of the target operation data are simplified to obtain simplified path information; the simplified path information is the data table identification information from the source path to the target path.

[0075] Regular expressions are used to extract simplified path information, resulting in the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path.

[0076] The purpose of path simplification is to eliminate temporary paths, historical directory paths, trash can paths, etc., while retaining basic path information to avoid redundant paths that could prevent accurate location of the source and target data tables. For example, the simplified path information would be "[storage type] / [database name] / [table name] / [partition]".

[0077] Regular expressions are tools used to describe string patterns. Through special characters and syntax rules, they enable operations such as matching, searching, replacing, and extracting from text. Regular expressions can be used to extract simplified path information, yielding the source data table corresponding to the source path and the target data table corresponding to the target path. For example, if the simplified source path is "hive / ods / user_login / dt=2023-10-01", regular expressions can be used to capture and extract the key information "first group ods + second group user_login" by grouping, resulting in the source data table ods.user_login.

[0078] By simplifying paths and removing redundant information, the difficulty of extracting data tables is reduced; regular expressions can be used to process path information in batches, supporting large-scale data scenarios.

[0079] In one possible implementation, the corresponding application is determined based on the application ID corresponding to multiple target operation data, including:

[0080] Obtain historical processing logs corresponding to multiple target operation data;

[0081] For any one of the multiple target operation data, determine the application corresponding to the application ID based on the application ID and historical processing logs.

[0082] Historical processing logs are structured or unstructured records generated during data processing, such as cleaning, aggregation, and computation. They contain end-to-end processing information about the data from its original state to its target state. Historical processing logs record "who processed the data."

[0083] For each target operation data, the application corresponding to the application ID is determined through the steps of "application ID → historical processing log matching → application determination". Specifically, in the historical processing logs corresponding to multiple target operation data, log fragments related to the application ID of the current target operation data are filtered out, and the application corresponding to the application ID is matched according to the relevant log fragments.

[0084] Example 2

[0085] Figure 4A schematic diagram of a device for determining the relationship of log data provided in this application is shown below. Figure 4 As shown, the apparatus 400 for determining the relationship of log data provided in this embodiment includes...

[0086] The acquisition module 401 is used to acquire a log operation data set; wherein, the log operation data set includes multiple operation data, each operation data corresponding to a source path, a target path and context information;

[0087] Processing module 402 is used to filter and process the operation data within the log operation data set to obtain multiple target operation data.

[0088] The processing module 402 is also used to parse and process multiple target operation data based on context information to obtain the application ID corresponding to each target operation data;

[0089] The processing module 402 is also used to determine the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path for any one of the multiple target operation data.

[0090] The generation module 403 is used to generate a corresponding data relationship graph based on the source data table, target data table and application ID corresponding to multiple target operation data.

[0091] The apparatus for determining the relationship of log data provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0092] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0093] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0094] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0095] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0096] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0097] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0098] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0099] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0100] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0101] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0102] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0105] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0107] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for determining the relationship of log data, characterized in that, include: Obtain a log operation data set; wherein, the log operation data set includes multiple operation data, each operation data corresponding to a source path, a target path, and context information; The operation data within the log operation data set is filtered to obtain multiple corresponding target operation data. Based on the context information, the multiple target operation data are parsed and processed to obtain the application ID corresponding to each target operation data; For any one of the multiple target operation data, determine the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path. Based on the source data table, target data table, and application ID corresponding to multiple target operation data, a corresponding data relationship graph is generated.

2. The method according to claim 1, characterized in that, The process of generating a corresponding data relationship graph based on the source data table, target data table, and application ID corresponding to multiple target operation data includes: The corresponding application is determined based on the application ID corresponding to the multiple target operation data; Based on the source data table, target data table and application ID corresponding to the multiple target operation data, establish a relationship mapping between application ID and data table, and generate multiple corresponding table-level data relationship graphs. Based on the corresponding application and the multiple corresponding table-level data relationship graphs, a corresponding data relationship graph is generated.

3. The method according to claim 2, characterized in that, The process of establishing a mapping between application IDs and data tables generates multiple corresponding table-level data relationship graphs, including: For any given application ID, an input / output link is constructed based on the application ID and the corresponding target operation data; Based on the input / output link, the source data table, and the target data table, a relationship mapping is established to generate a table-level data relationship graph corresponding to the application ID.

4. The method according to claim 1, characterized in that, The log operation data set further includes: the operation type corresponding to each operation data; the filtering process of the operation data within the log operation data set to obtain multiple corresponding target operation data includes: The log operation data set is filtered based on the operation type corresponding to each operation data. The operation data that does not conform to the predefined operation type in the log operation data set is removed to obtain the corresponding multiple target operation data.

5. The method according to claim 1, characterized in that, Determining the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path includes: The source path and target path of the target operation data are simplified to obtain simplified path information; wherein, the simplified path information is the data table identification information from the source path to the target path; The simplified path information is extracted using regular expressions to obtain the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path.

6. The method according to claim 2, characterized in that, The step of determining the corresponding application based on the application ID corresponding to the multiple target operation data includes: Obtain the historical processing logs corresponding to the multiple target operation data; For any one of the multiple target operation data, the application corresponding to the application ID is determined based on the application ID and historical processing logs corresponding to the target operation data.

7. An apparatus for determining the relationship of log data, characterized in that, include: The acquisition module is used to acquire a log operation data set; wherein, the log operation data set includes multiple operation data, each operation data corresponding to a source path, a target path, and context information; The processing module is used to filter and process the operation data within the log operation data set to obtain multiple target operation data. The processing module is further configured to parse and process multiple target operation data based on the context information to obtain the application ID corresponding to each target operation data. The processing module is further configured to determine, for any one of the multiple target operation data, the source data table corresponding to the source path of the target operation data and the target data table corresponding to the target path; The generation module is used to generate corresponding data relationship graphs based on the source data tables, target data tables, and application IDs corresponding to multiple target operation data.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images