Data-level blood relationship analysis method and system based on system logs

Abstract

The invention provides a system log-based data-level blood relationship analysis method and system. The method comprises the following steps of: creating a service table structure; setting fields of data-level blood relationship analysis; designing a log table structure of data-level blood relationship analysis; recording an operation log; generating data-level blood relationship data according to the log table structure and the recorded operation log; performing data-level blood relationship analysis display on the data in the whole life cycle from creation to modification to deletion, and performing red marking processing on important operations to form a data-level blood relationship map; and performing data operation analysis to obtain whether illegal operation exists or not. According to the method, the log record of the service system is optimally designed, the data-level service field data change analysis logic is added, the data-level blood relationship analysis capability is formed, the data-level blood relationship is visually displayed through the system or the platform, and the data analysis efficiency is improved; the system sharing of data illegal use is reduced, the business data change process is recorded and analyzed, the whole data change is monitored, the data illegal operation is prevented, and the database security is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data-level lineage analysis; in particular, to a data-level lineage analysis method and system based on system logs. Background Art

[0002] In the development of daily business needs, it is often necessary to record the operation logs of key business functions, that is, a user operates a certain function at a certain time, and the data before and after the operation is recorded. Business system log records can record every business operation, including business processes, transaction records, system operations, etc.

[0003] Through in-depth analysis of business system log records, such as performance optimization, troubleshooting, or security audits, we can discover potential system problems, optimize business processes, and improve analysis efficiency. By deeply mining and utilizing the log data generated by the system, we can obtain valuable information, thereby optimizing system operation, improving system performance, and improving business efficiency.

[0004] However, conventional log records can only record single data, and there is no association between data, and the changes in data cannot be displayed in terms of time. The logs recorded by existing business systems generally have the following characteristics or problems: 1. Data recorded in business system logs (such as Figure 1 As shown in the figure), the records contain a lot of information (fields), and records with multiple fields do not have importance levels (such as priority, urgency, etc.) set for the fields. The fields have equal relationships, which will affect the way the data is processed, sorted, or displayed, especially in scenarios where key information needs to be quickly identified or prioritized.

[0005] 2. Each time the data operating system records data, it exists independently. The same data (field) does not form a change relationship at different times, making it difficult to systematically analyze the changes in a single field. If you want to view the changes in a certain field, it is difficult to analyze based on the existing log records.

[0006] Due to the above characteristics of conventional log records, it is difficult to analyze the business data change process, difficult to effectively monitor the changes of the entire data, and it is easy to have security issues such as illegal data operations. Summary of the invention

[0007] In view of this, the purpose of the present invention is to propose a data-level lineage analysis method and system based on system logs. By optimizing the log records of the business system, the data-level business field data change analysis logic is added to form a data-level lineage analysis capability. The data-level lineage relationship can be intuitively displayed through the system or platform to help system users analyze the use of data and reduce system sharing problems caused by illegal use of data. By designing a data-level lineage analysis method, the business data change process can be recorded and analyzed, the life cycle of business data can be understood, the entire data changes can be effectively monitored, and illegal data operations can be prevented.

[0008] The present invention provides a data-level lineage analysis method based on system logs, comprising the following steps: S1. Create a business table structure of a database, wherein the fields of the business table structure include a primary key, a user name, a user address, and a deletion flag; For example, the table name is user_info, as shown in Table 1: Table 1

[0009] S2. Set the fields that require data-level lineage analysis; Exemplarily, the address field of the user_info table is set as the field to be subjected to data-level lineage analysis, and is used for data-level lineage analysis in step S5; S3. Design a log table structure that needs to be subjected to data-level lineage analysis, wherein the log table structure for the data-level lineage analysis includes: a table name and table fields; For example, the table name of the data-level lineage analysis table is: user_info_xxxx (business table name + suffix), xxxx has no actual meaning, but is only used to indicate that this table is a user data-level lineage analysis record table, and user_info_ indicates that data-level lineage analysis is performed on the user_info table; S4, record operation logs to provide basic data for data-level lineage analysis; Taking the user_info table (user information module) as an example, the recorded operation log is shown in Table 2: Table 2

[0010] Preferably, the operation log includes the log primary key, business module, operation method, operation time, operation user, operation machine MAC, and operation content information; S5. Perform data-level blood relationship analysis based on the log table structure of step S3 and the operation log recorded in step S4 to generate data-level blood relationship data (as shown in Table 3). Table 3

[0011] Table 3 shows the monitoring of a certain data of the address field of the user_info table (primary key is BUSIPK10001); S6. Perform data-level lineage analysis and display of the entire life cycle of a single piece of data from creation to modification to deletion, and mark important operations in red to form a data-level lineage diagram; The present invention adopts data-level lineage analysis based on system logs to track the flow, conversion and dependency of data throughout its entire life cycle from source to final form.

[0012] S7. Analyze individual data operations based on the data-level lineage graph to determine whether there are any illegal operations.

[0013] Furthermore, before the S1 step, the following steps are also included: Do not use direct connection to the database for analysis. Only operate the database through the business system and strictly control the database permissions. Set primary keys for data tables that need to be monitored. Use logical deletion for data deletion. Monitor data addition, data modification, data deletion, and field name modification.

[0014] Furthermore, the design of the log table structure for data-level lineage analysis in step S3 includes: Design the table fields of the data-level lineage analysis table, which include: data-level lineage analysis table primary key, data primary key, business table field name, business table new field name, business table field description, business table data value, operation time, operation type, operator, and operation machine MAC.

[0015] Furthermore, the display content of the data-level lineage analysis display in step S6 includes: operation time, operation machine MAC, operation type, operator, business data value, field name, field description, primary key data and machine ID.

[0016] Furthermore, the method of designing the log table structure for data-level lineage analysis in step S3 also includes: Use the business table name + suffix method to design the table name.

[0017] The present invention also provides a data-level lineage analysis system based on system logs, which executes the data-level lineage analysis method based on system logs as described above, including: Business table structure creation module: used to create a business table structure of a database, wherein the fields of the business table structure include a primary key, a user name, a user address, and a deletion flag; Set analysis field module: used to set fields that require data-level lineage analysis; Log table structure design module: used to design the log table structure that needs to be analyzed at the data level, the log table structure of the data level analysis includes: table name, table fields; Operation log recording module: used to record operation logs and provide basic data for data-level lineage analysis; Data-level lineage analysis module: used to perform data-level lineage analysis based on the log table structure required by step S3 and the operation log recorded in step S4, generating data-level lineage relationship data; Data-level lineage analysis and display module: used to perform data-level lineage analysis and display of the entire life cycle of a single piece of data from creation to modification to deletion, and to mark important operations in red to form a data-level lineage diagram; Data analysis module: used to analyze a single data operation according to the data-level lineage diagram to determine whether there is any illegal operation.

[0018] Furthermore, the log table structure design module includes: Design table field unit: used to design table fields of data-level lineage analysis table, the table fields include: data-level lineage analysis table primary key, data primary key, business table field name, business table new field name, business table field description, business table data value, operation time, operation type, operator, and operation machine MAC; Design table name unit: used to design table name by using business table name + suffix.

[0019] Furthermore, the data-level blood relationship analysis display module includes: Display content unit: used for data-level lineage analysis display, display content includes: operation time, operation machine MAC, operation type, operator, business data value, field name, field description, primary key data and machine ID.

[0020] The preferred embodiment of the data-level lineage analysis method and system based on system logs provided by the present invention is as follows: through the sub-table design of the data-level lineage analysis table (since the amount of data will be very large, a sub-table design can be adopted), each business table in the data-level lineage analysis table that needs to perform field-level lineage analysis corresponds to a data-level lineage analysis data record table, which effectively reduces the problem of too much data. Configure the scope of the business table that needs to perform data-level lineage analysis, set the fields that need to be analyzed, and perform data-level lineage analysis of the business fields in a targeted manner, thereby reducing unnecessary server performance loss, reducing the amount of data in the data-level lineage analysis table, and improving analysis efficiency. For data deletion, the method of logical deletion is changed to display monitoring deleted data. By setting the operating permissions of the database and the method of operating the data only through the business system, the purpose and effect of data-level lineage analysis are achieved.

[0021] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the data-level lineage analysis method based on system logs as described above are implemented.

[0022] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data-level lineage analysis method based on the system log as described above are implemented.

[0023] Compared with the prior art, the present invention has the following beneficial effects: The data-level lineage analysis method and system based on system logs of the present invention optimize the log records of the business system, increase the data-level business field data change analysis logic, form data-level lineage analysis capabilities, and intuitively display data-level lineage relationships through the system or platform, so as to help system users analyze the use of data and improve data analysis efficiency; reduce system sharing problems caused by illegal use of data, can record and analyze business data change processes, understand the life cycle of business data, can effectively monitor changes in the entire data, prevent illegal data operations, and effectively improve the security of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0025] In the attached picture: Figure 1 An actual interface diagram of a log recorded by a business system of the prior art; Figure 2 This is a diagram showing the actual interface of the data-level lineage analysis of an embodiment of the present invention; Figure 3 It is a flow chart of the data-level lineage analysis method based on system logs of the present invention; Figure 4 The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of systems and products consistent with some aspects of the present disclosure as detailed in the appended claims.

[0027] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] The embodiments of the present invention are described in further detail below.

[0030] The data-level lineage analysis method based on system logs in the embodiment of the present invention does not use a direct connection to the database for analysis, but only operates the database through the business system and strictly controls the database permissions; sets a primary key for the data table that needs to be monitored; uses logical deletion for data deletion; and monitors data addition, data modification, data deletion, and field name modification.

[0031] The data level lineage analysis method based on system log provided by the embodiment of the present invention comprises the following steps (see Figure 3 shown): S1. Create a business table structure of a database, wherein the fields of the business table structure include a primary key, a user name, a user address, and a deletion flag; In this embodiment, the table name is user_info, as shown in Table 1: Table 1

[0032] S2. Set the fields that require data-level lineage analysis; In this embodiment, the address field of the user_info table is set as the field to be subjected to data-level lineage analysis, and is used for data-level lineage analysis in step S5; S3. Design a log table structure that needs to be subjected to data-level lineage analysis, wherein the log table structure for the data-level lineage analysis includes: a table name and table fields; In this embodiment, the table name of the data-level lineage analysis table is: user_info_xxxx (business table name + suffix), xxxx has no actual meaning, but is only used to indicate the user data-level lineage analysis record table of this table, and user_info_ indicates that data-level lineage analysis is performed on the user_info table; Specifically, the table name is designed in the form of business table name + suffix.

[0033] The table fields include: data-level lineage analysis table primary key, data primary key, business table field name, business table new field name, business table field description, business table data value, operation time, operation type, operator, and operation machine MAC.

[0034] S4, record the operation log (as shown in Table 2) to provide basic data for data-level lineage analysis; The operation log of the user_info table (user information module) recorded in this embodiment is shown in Table 2: Table 2

[0035] The operation log contains the log primary key, business module, operation method, operation time, operation user, operation machine MAC, and operation content information; S5. Perform data-level blood relationship analysis based on the log table structure of step S3 and the operation log recorded in step S4 to generate data-level blood relationship data (as shown in Table 3). Table 3

[0036] Table 3 shows the monitoring of a certain data of the address field of the user_info table (primary key is BUSIPK10001); S6. Perform data-level lineage analysis and display of the entire life cycle of a single piece of data from creation to modification to deletion, and mark important operations in red to form a data-level lineage diagram; Specifically, the display content of the data-level lineage analysis display includes: operation time, operation machine MAC, operation type, operator, business data value, field name, field description, primary key data and machine ID.

[0037] This embodiment uses data-level lineage analysis based on system logs to track the flow, conversion and dependency of data throughout its entire life cycle from source to final form. Figure 2 An overall picture of data-level lineage analysis is shown.

[0038] S7. Analyze individual data operations based on the data-level lineage graph to determine whether there are any illegal operations.

[0039] The present invention also provides a data-level lineage analysis system based on system logs, which executes the data-level lineage analysis method based on system logs as described above, including: Business table structure creation module: used to create a business table structure of a database, wherein the fields of the business table structure include a primary key, a user name, a user address, and a deletion flag; Set analysis field module: used to set fields that require data-level lineage analysis; Log table structure design module: used to design the log table structure that needs to be analyzed at the data level, the log table structure of the data level analysis includes: table name, table fields; Operation log recording module: used to record operation logs and provide basic data for data-level lineage analysis; Data-level lineage analysis module: used to perform data-level lineage analysis based on the log table structure required by step S3 and the operation log recorded in step S4, generating data-level lineage relationship data; Data-level lineage analysis and display module: used to perform data-level lineage analysis and display of the entire life cycle of a single piece of data from creation to modification to deletion, and to mark important operations in red to form a data-level lineage diagram; Data analysis module: used to analyze a single data operation according to the data-level lineage diagram to determine whether there is any illegal operation.

[0040] The log table structure design module includes: Design table field unit: used to design table fields of data-level lineage analysis table, the table fields include: data-level lineage analysis table primary key, data primary key, business table field name, business table new field name, business table field description, business table data value, operation time, operation type, operator, and operation machine MAC; Design table name unit: used to design table name by using business table name + suffix.

[0041] The data-level blood relationship analysis display module includes: Display content unit: used for data-level lineage analysis display, display content includes: operation time, operation machine MAC, operation type, operator, business data value, field name, field description, primary key data and machine ID.

[0042] This embodiment uses a sub-table design for the data-level lineage analysis table to correspond each business table that needs to perform field-level lineage analysis in the data-level lineage analysis table to a data-level lineage analysis data record table, effectively reducing the problem of too much data. Configure the scope of the business tables that need to perform data-level lineage analysis, set the fields that need to be analyzed, and perform data-level lineage analysis of the business fields in a targeted manner, thereby reducing unnecessary server performance loss, reducing the amount of data in the data-level lineage analysis table, and improving analysis efficiency. The deletion of data is changed to a logical deletion method, so that the deleted data can be displayed and monitored. By setting the operation permissions of the database and the operation mode that can only be used through the business system, the purpose and effect of data-level lineage analysis can be achieved.

[0043] An embodiment of the present invention further provides a computer device, Figure 4 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention; see the accompanying drawings Figure 4 As shown, the computer device includes: an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the data-level lineage analysis method based on the system log provided in the above embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 4 The example of connecting through bus is taken in the following.

[0044] The memory 22 is a readable and writable storage medium of a computing device, which can be used to store software programs and computer executable programs, such as the program instructions corresponding to the data-level blood relationship analysis method based on the system log described in the embodiment of the present invention; the memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc.; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some instances, the memory 22 can further include a memory remotely arranged relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The input system 23 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the device; the output system 24 may include display devices such as display screens.

[0046] The processor 21 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 22, that is, implements the above-mentioned data-level lineage analysis method based on system logs.

[0047] The computer device provided above can be used to execute the data-level lineage analysis method based on system logs provided in the above embodiment, and has corresponding functions and beneficial effects.

[0048] The embodiment of the present invention also provides a storage medium containing computer executable instructions, which are used to execute the data-level lineage analysis method based on system logs as provided in the above embodiment when executed by a computer processor. The storage medium is any of various types of memory devices or storage devices, and the storage medium includes: installation media, such as CD-ROM, floppy disk or tape system; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or a combination thereof; in addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, the second computer system is connected to the first computer system via a network (such as the Internet); the second computer system may provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected via a network). The storage medium can store program instructions (for example, specifically implemented as a computer program) that can be executed by one or more processors.

[0049] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the data-level lineage analysis method based on system logs as described in the above embodiment, but can also execute related operations in the data-level lineage analysis method based on system logs provided in any embodiment of the present invention. So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data-level lineage analysis method based on system logs, characterized in that: The following steps are involved: S1. Create a business table structure of a database, wherein the fields of the business table structure include a primary key, a user name, a user address, and a deletion flag; S2. Set the fields that require data-level lineage analysis; S3. Design a log table structure that needs to be subjected to data-level lineage analysis, wherein the log table structure for the data-level lineage analysis includes: a table name and table fields; S4, record operation logs to provide basic data for data-level lineage analysis; S5. Perform data-level lineage analysis based on the log table structure required by step S3 and the operation log recorded in step S4 to generate data-level lineage relationship data; S6. Perform data-level lineage analysis and display of the entire life cycle of a single piece of data from creation to modification to deletion, and mark important operations in red to form a data-level lineage diagram; S7. Analyze individual data operations based on the data-level lineage graph to determine whether there are any illegal operations.

2. The data-level lineage analysis method based on system logs according to claim 1 is characterized in that: The S1 step also includes: Do not use direct connection to the database for analysis. Only operate the database through the business system and strictly control the database permissions. Set primary keys for data tables that need to be monitored. Use logical deletion for data deletion. Monitor data addition, data modification, data deletion, and field name modification.

3. The data-level lineage analysis method based on system log according to claim 1 is characterized in that: The design of the S3 step requires a log table structure for data-level lineage analysis, including: Design the table fields of the data-level lineage analysis table, which include: data-level lineage analysis table primary key, data primary key, business table field name, business table new field name, business table field description, business table data value, operation time, operation type, operator, and operation machine MAC.

4. The data-level lineage analysis method based on system log according to claim 1 is characterized in that: The display content of the data-level lineage analysis display in step S6 includes: operation time, operation machine MAC, operation type, operator, business data value, field name, field description, primary key data and machine ID.

5. The data-level lineage analysis method based on system log according to claim 1 is characterized in that: The design of the S3 step requires a log table structure for data-level lineage analysis, including: Use the business table name + suffix method to design the table name.

6. A data-level lineage analysis system based on system logs, which executes a data-level lineage analysis method based on system logs as described in any one of claims 1 to 5, characterized in that: include: Business table structure creation module: used to create a business table structure of a database, wherein the fields of the business table structure include a primary key, a user name, a user address, and a deletion flag; Set analysis field module: used to set fields that require data-level lineage analysis; Log table structure design module: used to design the log table structure that needs to be analyzed at the data level, the log table structure of the data level analysis includes: table name, table fields; Operation log recording module: used to record operation logs and provide basic data for data-level lineage analysis; Data-level lineage analysis module: used to perform data-level lineage analysis based on the log table structure required by step S3 and the operation log recorded in step S4, generating data-level lineage relationship data; Data-level lineage analysis and display module: used to perform data-level lineage analysis and display of the entire life cycle of a single piece of data from creation to modification to deletion, and to mark important operations in red to form a data-level lineage diagram; Data analysis module: used to analyze a single data operation according to the data-level lineage diagram to determine whether there is any illegal operation.

7. The data-level blood relationship analysis system based on system log according to claim 6 is characterized in that: The log table structure design module includes: Design table field unit: used to design table fields of data-level lineage analysis table, the table fields include: data-level lineage analysis table primary key, data primary key, business table field name, business table new field name, business table field description, business table data value, operation time, operation type, operator, and operation machine MAC; Design table name unit: used to design table name by using business table name + suffix.

8. The data-level blood relationship analysis system based on system log according to claim 6 is characterized in that: The data-level blood relationship analysis display module includes: Display content unit: used for data-level lineage analysis display, display content includes: operation time, operation machine MAC, operation type, operator, business data value, field name, field description, primary key data and machine ID.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the data-level lineage analysis method based on system logs described in any one of claims 1 to 5 are implemented.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data-level lineage analysis method based on system logs are implemented as described in any one of claims 1-5.

Images