Data relationship updating method and device, storage medium and electronic equipment

Abstract

The invention discloses a data relation updating method and device, a storage medium and electronic equipment, and the data relation updating method comprises the steps: adding a real-time recording field in front of a first character of a corresponding operation field of a target database, and obtaining a target field; under the condition that data calling operation occurs in the updating target database of the data relation, collecting target information used for filling the updating target field of the data relation to obtain a target log file; the updated target log file of the data relation is analyzed, blood relationship change information corresponding to current data calling is obtained, and the data relation between different data in a blood relationship graph bound to the updated target database of the data relation is updated based on the updated blood relationship change information of the data relation. By adopting the technical scheme, the problem of how to efficiently maintain the data blood relationship in a big data environment so as to reduce the manual maintenance cost and improve the data management efficiency is solved.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a method, apparatus, storage medium, and electronic device for updating data relationships. Background Technology

[0002] With the advent of the big data era, the amount of data processed by enterprises and organizations is exploding. The diversification of data sources and the increasing complexity of processing processes make data management and governance particularly important. Data lineage analysis has emerged in this context. It aims to help enterprises understand and master the entire lifecycle of data, including its source, processing, flow path, and final destination, thereby ensuring data quality, security, and compliance. Because different data processing needs are implemented by different developers, each data processing need requires processing source data according to statistical standards into target data stored in the target database for analysis and use. Since only the corresponding developer understands the data processing process, data lineage is maintained and managed by each developer individually. Faced with increasingly numerous data processing needs, relying on developers to manually manage data lineage is time-consuming and prone to omissions.

[0003] Therefore, in related technologies, no effective solution has yet been proposed for how to efficiently maintain data lineage in a big data environment to reduce manual maintenance costs and improve data management efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, storage medium, and electronic device for updating data relationships, in order to at least solve the problem in the related art of how to efficiently maintain data lineage relationships in a big data environment, so as to reduce manual maintenance costs and improve data management efficiency.

[0005] According to one embodiment of this application, a method for updating data relationships is provided, comprising: adding a real-time record field before the first character of the corresponding operation field in the target database to obtain a target field, wherein the real-time record field is used to record key lineage information during the process of calling the target database, the key lineage information including: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database; when a data call operation occurs in the target database, collecting target information for filling the target field to obtain a target log file; parsing the target log file to obtain lineage change information corresponding to the current data call, and updating the data relationships between different data in the lineage relationship graph bound to the target database based on the lineage change information.

[0006] In an exemplary embodiment, before parsing the target log file to obtain the lineage change information corresponding to the current data call, the above-mentioned data relationship update method further includes: identifying the source name and target name corresponding to the data call operation recorded in the target log file, wherein the source name is used to locate the data source of the target data and the target name is used to locate the data flow of the target data; splitting the basic data bytes associated with the source name and the target name to determine the first field corresponding to the source name and the second field corresponding to the target name.

[0007] In an exemplary embodiment, splitting the basic data bytes associated with the source name and the target name includes: cleaning the basic data bytes to obtain a first string; splitting the character elements in the first string with spaces as the interval to obtain multiple second strings; identifying special characters in the multiple second strings; determining a first field based on the special characters and a preset name recognition rule; and determining a second field based on the special characters and a preset name query rule.

[0008] In an exemplary embodiment, after splitting the basic data bytes associated with the source name and the target name to determine the first field corresponding to the source name and the second field corresponding to the target name, the above data relationship update method further includes: capturing abnormal characters that occur during the splitting process and generating an abnormal character set; determining the normal character set corresponding to the abnormal character set; and establishing a verification character library based on the correspondence between the same character in the abnormal character set and the normal character set.

[0009] In an exemplary embodiment, after updating the data relationships between different data in the bloodline relationship graph bound to the target database based on bloodline change information, the above data relationship update method further includes: obtaining the update result of the bloodline relationship graph; if the update result indicates that the bloodline relationship graph update has failed, sending a prompt message to the management object corresponding to the bloodline relationship graph, wherein the prompt message is used to indicate that the bloodline change information acquisition has been abnormal and the bloodline change information of the target database needs to be collected again; if the update result indicates that the bloodline relationship graph update has been successful, recording the update time when the update is completed.

[0010] In an exemplary embodiment, after recording the update time when the update result indicates that the kinship diagram has been successfully updated, the above-mentioned data relationship update method further includes: upon receiving a display instruction input by the management object, determining the image content to be displayed in the kinship diagram, wherein the display instruction is used to set the content features to be displayed in the kinship diagram; highlighting the image content in the kinship diagram and blurring the non-highlighted parts in the kinship diagram to obtain visualized kinship data for display on the control terminal associated with the target database.

[0011] According to another aspect of the embodiments of this application, a data relationship updating apparatus is also provided, comprising: an adding module, configured to add a real-time recording field before the first character of the corresponding operation field in the target database to obtain a target field, wherein the real-time recording field is used to record key lineage information during the process of calling the target database, the key lineage information including: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database; a collecting module, configured to collect target information for filling the target field when a data call operation occurs in the target database to obtain a target log file; and an updating module, configured to parse the target log file to obtain lineage change information corresponding to the current data call, and update the data relationship between different data in the lineage relationship graph bound to the target database based on the lineage change information.

[0012] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described data relationship update method when running.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the data relationship update method through the computer program.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program and a method for updating the above-mentioned data relationship when the computer program is executed by a processor.

[0015] In this embodiment, a real-time recording field, or target field, is added before the operation field of the target database. This target field is specifically used to capture and store key lineage information during the process of calling the target database, including important details such as data lineage type, lineage relationship, and data flow direction. When the target database performs a data call operation, the relevant lineage information is automatically collected to form a target log file. Subsequently, these log files are processed using intelligent parsing methods to extract the lineage relationship changes caused by the data call. This change information is used to update the lineage relationship diagram associated with the target database in real time to reflect the latest relationship changes between different data in the target database. This scheme ensures that the performance of database operations is not affected, and through batch parsing of log files, the data flow path can be analyzed efficiently to determine the updated lineage relationship diagram, improving the efficiency of continuous tracking of target data. The above technical solution solves the problem of how to efficiently maintain data lineage relationships in a big data environment to reduce manual maintenance costs and improve data management efficiency. It achieves the goal of embedding additional recording logic into the database operation code, enabling the real-time capture and recording of key lineage information whenever a database operation occurs. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the hardware environment for a data relationship updating method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a data relationship update method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a process for automatically discovering data relationships according to an embodiment of this application;

[0021] Figure 4 This is a data processing flowchart according to an embodiment of this application;

[0022] Figure 5 This is a flowchart illustrating the processing of a UDF function according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram illustrating a data relationship according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of a visual tool interaction interface according to an embodiment of this application;

[0025] Figure 8 This is a structural block diagram of a data relationship updating device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, apparatus, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, apparatus, or devices.

[0028] According to one aspect of the embodiments of this application, a method for updating data relationships is provided. This method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned method for updating data relationships can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. Figure 1 This is a schematic diagram of the hardware environment for a data relationship update method according to an embodiment of this application, such as... Figure 1As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PCs, mobile phones, tablets, etc.

[0030] This embodiment provides a method for updating data relationships, applied to the aforementioned terminal device. Figure 2 This is a flowchart of a data relationship update method according to an embodiment of this application, which includes the following steps:

[0031] Step S202: Add a real-time record field before the first character of the corresponding operation field in the target database to obtain the target field. The real-time record field is used to record key lineage information during the process of calling the target database. The key lineage information includes: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database.

[0032] In simple terms, database operation statements (such as SQL) typically involve read and write operations on specific database tables or fields. The core of this step is to dynamically add a special real-time logging field before the first character of these operation fields (such as table names and field names). This real-time logging field is designed to capture and record key lineage information during the process of calling the target database (e.g., executing SQL operations). Lineage information here refers to the data's lineage type, the relationship between data points, and the specific flow of data. For example, whether the data is loaded from a file into the database, extracted from the database and sent to another data system, or involves the association processing between multiple data points.

[0033] Step S204: In the event of a data retrieval operation in the target database, collect the target information used to populate the target field to obtain the target log file;

[0034] Once a data call operation occurs in the target database, it triggers the collection of all target information related to this operation. This information includes the type of operation, the context of the operation (such as which code triggered the operation), and the specific content of the operation (such as which data was called and what processing was performed). All the collected information will be used to populate the real-time record field generated in step S202, that is, to record the actual lineage information in this field. Subsequently, this populated field information will be organized and a target log file will be generated. The log file records the lineage information of each database call operation in detail, providing a data foundation for subsequent data lineage analysis and updates.

[0035] Step S206: Parse the target log file to obtain the bloodline change information corresponding to the current data call, and update the data relationship between different data in the bloodline relationship graph bound to the target database based on the bloodline change information.

[0036] This involves intelligently parsing and processing the target log file generated in step S204. Specifically, it reads the records in the log file and analyzes changes in lineage information using preset algorithms or rules to identify changes in the relationships between data before and after data access, such as new data dependencies and changes in data flow. Based on this parsed lineage change information, the lineage diagram related to the target database is automatically updated. A lineage diagram is a graphical data relationship model that shows the dependencies and flow between data. The updated lineage diagram will more accurately reflect the actual lineage relationships of data in the database, helping data governance personnel to promptly identify potential problems in data processing and ensuring data quality and the healthy operation of the data system.

[0037] Through the above steps, a real-time recording field, namely the target field, is added before the operation field in the target database. This target field is specifically used to capture and store key lineage information during the process of calling the target database, including important details such as data lineage type, lineage relationship, and data flow. When the target database performs a data call operation, the relevant lineage information is automatically collected to form a target log file. Subsequently, these log files are processed using intelligent parsing methods to extract the changes in lineage relationships caused by the data call. This change information is used to update the lineage relationship diagram associated with the target database in real time to reflect the latest relationship changes between different data in the target database. This solution ensures that the performance of database operations is not affected, and through batch parsing of log files, the data flow path can be analyzed efficiently to determine the updated lineage relationship diagram, improving the efficiency of continuous tracking of target data. This technical solution solves the problem of how to efficiently maintain data lineage relationships in a big data environment to reduce manual maintenance costs and improve data management efficiency. It achieves the goal of embedding additional recording logic into the database operation code, enabling the real-time capture and recording of key lineage information whenever a database operation occurs.

[0038] In an exemplary embodiment, before parsing the target log file to obtain the lineage change information corresponding to the current data call, the above-mentioned data relationship update method further includes: identifying the source name and target name corresponding to the data call operation recorded in the target log file, wherein the source name is used to locate the data source of the target data and the target name is used to locate the data flow of the target data; splitting the basic data bytes associated with the source name and the target name to determine the first field corresponding to the source name and the second field corresponding to the target name.

[0039] Optionally, the system first scans each data retrieval operation recorded in the target log file, focusing on identifying the source and target names in each operation. The source name refers to the origin of the data retrieval operation, i.e., where the data initially came from, while the target name is the destination of the data retrieval operation, identifying the new location or entity the data will flow to. To ensure accurate extraction of the source and target names, the underlying data bytes containing these names are further meticulously split. This process involves analyzing the strings in the log records to identify the specific fields representing the source and target names. For example, for SQL statements, it may be necessary to determine the first field (source name field) and the second field (target name field) based on specific syntax structures (such as "FROM" followed by the source name, and "INTO" or "OVERWRITE" followed by the target name). This precise field splitting ensures that the source and target names used in subsequent parsing processes are accurate, thereby improving the accuracy and reliability of lineage analysis.

[0040] In summary, by pre-identifying and separating the source and target names, this method can more efficiently and accurately obtain key lineage information of data call operations when parsing target log files. This provides a solid data foundation for subsequent lineage graph updates, thereby achieving automated and intelligent updates of data lineage relationships and effectively improving transparency and efficiency in data governance and data quality monitoring. This preprocessing step is a crucial guarantee for ensuring the smooth operation of the entire update process and the accuracy of data lineage tracing.

[0041] In an exemplary embodiment, splitting the basic data bytes associated with the source name and the target name includes: cleaning the basic data bytes to obtain a first string; splitting the character elements in the first string with spaces as the interval to obtain multiple second strings; identifying special characters in the multiple second strings; determining a first field based on the special characters and a preset name recognition rule; and determining a second field based on the special characters and a preset name query rule.

[0042] Understandably, by cleaning the collected basic data bytes (i.e., raw log records) to remove noise and interference elements such as redundant spaces, comments, and special symbols, the raw data is transformed into a more standardized first string, laying a good foundation for subsequent processing. Next, using spaces as delimiters, the resulting first string is divided into a series of shorter second strings, each representing an independent character element or lexical unit. This is done to break down complex SQL statements or other data call formats into easily analyzable components. Then, by identifying special characters in these second strings (such as SQL keywords like FROM, JOIN, and INTO) and combining them with preset name recognition rules, it is determined which second strings belong to the source name and categorized them as the first field. Similarly, based on preset name query rules, second strings related to the target data flow are identified and marked as second fields. These rules are typically based on the syntactic characteristics of data call operations; for example, in SQL statements, the string immediately following the keywords "INTO" or "OVERWRITE" is likely the target name. This meticulous splitting and identification process allows for the accurate location of source and target names within log files, providing clear and accurate data points for further data lineage analysis.

[0043] In summary, by cleaning and segmenting the basic data bytes and using rule matching based on special characters, this embodiment provides an efficient and accurate method for identifying source and target names. This paves the way for the automated update process of data lineage relationships, ensuring the correct tracking and recording of data flow paths, thereby improving the efficiency of data governance and the level of data quality management. This series of string processing steps is an indispensable part of realizing automated lineage relationship update technology, and its accuracy and effectiveness directly affect the reliability and practicality of the final analysis results.

[0044] In an exemplary embodiment, after splitting the basic data bytes associated with the source name and the target name to determine the first field corresponding to the source name and the second field corresponding to the target name, the above data relationship update method further includes: capturing abnormal characters that occur during the splitting process and generating an abnormal character set; determining the normal character set corresponding to the abnormal character set; and establishing a verification character library based on the correspondence between the same character in the abnormal character set and the normal character set.

[0045] Optionally, a company is automating data lineage discovery in its sales data processing workflow. In log recording, data is loaded from the local file system into a Hadoop Hive table. First, the logs are parsed to obtain the data source and target names. During this process, special characters such as backslashes "\", single quotes "'", and backticks may be encountered. These characters can affect the accurate segmentation and identification of data in certain situations. Therefore, these characters are captured, generating an abnormal character set. The abnormal character set is analyzed to identify the usage scenarios and functions of these characters in normal database operations and log recording. For example, single quotes and backticks are typically used to enclose table names or file paths, while backslashes are used for escaping. A character validation library is established based on the above analysis results, containing rules for the correct use of single quotes, backticks, and backslashes. For example, single quotes and backticks should appear in pairs to enclose strings or identifiers; backslashes should be used for escaping, not arbitrarily. Through these steps, not only can data lineage discovery be effectively automated, but the accuracy of the parsing process and the integrity of the data lineage graph can also be ensured, greatly improving the efficiency of data management and analysis.

[0046] In an exemplary embodiment, after updating the data relationships between different data in the bloodline relationship graph bound to the target database based on bloodline change information, the above data relationship update method further includes: obtaining the update result of the bloodline relationship graph; if the update result indicates that the bloodline relationship graph update has failed, sending a prompt message to the management object corresponding to the bloodline relationship graph, wherein the prompt message is used to indicate that the bloodline change information acquisition has been abnormal and the bloodline change information of the target database needs to be collected again; if the update result indicates that the bloodline relationship graph update has been successful, recording the update time when the update is completed.

[0047] Optionally, an online platform is implementing automated data lineage discovery to track the flow of course data from the video upload service to the data analysis platform. The target log file containing the data flow information from the video upload service to the course data table is parsed to generate initial lineage information. Subsequently, the lineage graph is updated based on this information. If an update failure is detected, it may be due to an abnormal format of a field in the log, such as an unexpected timestamp format. For example, "timestamp=2023-03-15T10:00:00Z" might be mistakenly entered as "timestamp=2023-03-15T10;00:00Z", where the semicolon ";" is an abnormal character. Upon recognizing this anomaly, an email notification will be sent to the administrator responsible for maintaining the data lineage: "Lineage graph update failed; an anomaly occurred in the timestamp field." If the kinship graph is successfully updated, the update time will be recorded in the background, for example, "2023-03-15T10:15:00Z". This timestamp will be used for subsequent data auditing and kinship history tracking to ensure that the history of changes in data kinship is clearly visible. Through the above steps, not only can data kinship be automatically discovered and updated, but the accuracy and reliability of the data kinship graph can also be ensured. Once a problem is discovered, a rapid response can be made to optimize data management strategies and improve the quality and efficiency of data governance.

[0048] In an exemplary embodiment, after recording the update time when the update result indicates that the kinship diagram has been successfully updated, the above-mentioned data relationship update method further includes: upon receiving a display instruction input by the management object, determining the image content to be displayed in the kinship diagram, wherein the display instruction is used to set the content features to be displayed in the kinship diagram; highlighting the image content in the kinship diagram and blurring the non-highlighted parts in the kinship diagram to obtain visualized kinship data for display on the control terminal associated with the target database.

[0049] In other words, updating the data lineage graph is not limited to changes in the backend data structure, but also includes optimization and customization of the frontend display to enhance user experience and the intuitiveness of data visualization. Once the lineage graph update operation is completed and confirmed successfully, this update moment is automatically recorded, providing an accurate time reference for subsequent data status tracking and time series analysis.

[0050] Next, in scenarios where users input display commands through the management object, the system further responds to users' customized needs. These display commands refer to specific content features that users wish to highlight in the lineage diagram. These features could be a specific data flow path, data changes within a specific time period, or the relationship between a specific data source and a target. Based on the received commands, the system intelligently filters and determines the key image content to be displayed in the lineage diagram, ensuring that the display interface accurately reflects the data lineage details that the user cares about.

[0051] To enhance the visual appeal of the display and guide user attention to key information, this embodiment employs a display strategy combining highlighting and blurring. Specifically, certain image content is highlighted, making it prominently displayed in the kinship diagram for easy identification and understanding by users. Simultaneously, the non-highlighted portions of the diagram are blurred. This reduces visual interference and enhances the perception of the importance of the highlighted areas through contrast, helping users to more effectively browse and analyze the data kinship diagram.

[0052] Ultimately, the optimized lineage diagram will be presented as visualized lineage data, displayed through a control terminal (such as the user interface of a data governance platform) associated with the target database. Users will see not only the updated lineage structure on the control terminal, but also customized highlighting and blurring effects based on display commands. This intuitive and customized visualization method greatly facilitates data governance personnel's in-depth exploration of data flow history and relationships, while also improving the efficiency and accuracy of data quality management.

[0053] To better understand the process of the above data relationship update method, the following description of the data relationship update method flow is further illustrated with reference to optional embodiments, but it is not intended to limit the technical solutions of the embodiments of this application.

[0054] In related technologies, due to different data processing needs, data lineages need to be developed and maintained separately. This requires developers to spend a lot of energy sorting out data lineages in addition to data processing work itself. As data processing needs increase, sorting out data lineages takes more and more time and maintenance costs also increase.

[0055] Alternatively, data lineage, also known as data pedigree, data origin, or data genealogy, refers to the natural, blood-like relationships that form between data throughout their entire lifecycle—from generation, processing, integration, and flow to eventual disappearance. This relationship records the source, destination, and processing of the data, and is crucial for understanding the overall structure and logical relationships of the data.

[0056] To address the aforementioned issues, this application proposes an optional embodiment of an automated data lineage discovery method. This method automatically collects log information from various data processing processes through code embedding, then performs intelligent analysis on the logs to obtain data lineage results. Finally, it utilizes visualization tools and technologies to graphically display the data lineage, helping data managers intuitively understand the data flow path and relationships, thereby making it easier to discover and resolve data quality issues and improve overall data management efficiency.

[0057] Optionally, code-based embedding can be used to embed data into the database execution process. When database operations occur during routine program execution, corresponding code is automatically triggered, generating a log entry and saving it to a lineage relationship log file. This process is continuous and real-time. At the end of each day, the program reads the contents of the lineage relationship log file, performs a series of operations such as log parsing, data cleaning, and data analysis, and generates data stream lineage relationship information, which is saved to the daily lineage relationship record results for analysis. The lineage relationship analysis results can be viewed through a visual interface. Data governance personnel can use various filters to view the results, identify data processing issues, improve the efficiency of big data governance, and unlock the value of big data.

[0058] Optionally, Figure 3 This is a flowchart illustrating an automated data relationship discovery process according to an embodiment of this application, specifically including the following steps:

[0059] Step 3.1: Lineage Data Model Design. Determine the types of lineage data to be managed and their attributes to accurately represent the complex upstream and downstream dependencies between data. This includes defining dependencies between data tables, between data tables and files, and between files themselves. The lineage data model is shown in Table 1:

[0060] Table 1

[0061]

[0062] Optionally, source type and target type represent the type of data source, which can be a file or a database. Source name and target name are the specific file paths plus names or table names, reflecting the upstream and downstream dependencies between data. Job flow and job identifier are information about a specific job flow in the scheduling system, indicating which job flow this data flow is being processed in. In the data lineage, the relationship between target and source is many-to-many, so the same target may have multiple sources, resulting in multiple data entries, as shown in Table 2.

[0063] Table 2

[0064]

[0065] Step 3.2: Data processing and visualization of blood relations. For example... Figure 4 As shown, Figure 4 This is a data processing flowchart according to an embodiment of this application, specifically including the following steps:

[0066] Step 4.1: Data Collection. Genesis data is collected through code tracking. Genesis data is extracted from various data sources, including databases, data warehouses, and ETL tools. When database operations occur during routine program execution, corresponding code is automatically triggered to generate a log record, which is saved to a genesis relationship log file. This process is continuous and ongoing.

[0067] Optionally, the log format can be: job flow id001|@|job id01|@|collected lineage content (can be SQL, filename, or other agreed format: such as load data inpath ' / data / a.txt' overwrite into table e_hive_a)|@|file(source type)|@|hive(target type).

[0068] It should be noted that the files are generated daily in a specified directory on the NAS shared disk.

[0069] Step 4.2: Read and parse the data. This specifically includes:

[0070] (1) Read the log. At the end of the day, the program reads the contents of the lineage log file, extracts the key information reflecting the data flow of the lineage relationship, and saves it as the initial information of the lineage relationship. By reading the log file, the job flow ID, job ID, collected lineage relationship content (which can be SQL, file name, or other agreed format), source type, and target type can be extracted and saved to the Hive table.

[0071] (2) Analyze the bloodline data flow. Further analyze the initial bloodline data using analytical algorithms to generate a data flow of bloodline information, which is then saved to the daily bloodline record results for visualization. The collected bloodline content includes some information that directly records the source and target names, which can be directly entered into the bloodline data model. Other information requires parsing the source and target names. In this application, the source and target names are mainly obtained by parsing SQL using HiveUDF functions. For example... Figure 5 As shown, Figure 5 This is a flowchart of a UDF function processing according to an embodiment of this application, which mainly includes the following operations:

[0072] Data cleaning. This includes formatting and standardizing SQL statements, replacing two or more consecutive spaces with a single space, removing comments, replacing special characters such as "@" with commas, replacing newline characters like \t and \n with spaces, and converting SQL statements to uppercase or lowercase, etc.

[0073] The SQL is split by spaces. After formatting, each word in the SQL is separated by a space, so all elements in the array generated by splitting the SQL string by spaces contain only one word.

[0074] Iterate through the array to obtain the target name. In a single SQL statement, there is only one target name. Therefore, determine the key of the first element of the array. For SQL statements filtering specific keywords such as CREATE, DROP, ALTER, TURNCATE, etc., since the SQL syntax is `create table table_name`, `drop table table_name`, etc., the third element of the array must be the table name. When the key is UPDATE, since the SQL syntax is `update table_name set`, the second element of the array is the table name. When the key is INSERT, determine the second element of the array. If it is INTO, the third element is the table name; if it is OVERWAKE, the syntax is usually INSERT OVERWAKE table, and the fourth element is the table name.

[0075] Iterate through the array to obtain the source table name. In a single SQL query, there are one or more source table names (subqueries, joins, etc.). If the i-th element (i = 0, 1, 2... array length - 1) of the array is FROM or JOIN, then if the next (i+1) element is SELECT, it's considered a subquery or join, and skipped; otherwise, the next (i+1) element is the source table name. If the i-th element (i = 0, 1, 2... array length - 1) of the array is UNION, then determine the type of the (i+1)-th (UNION) or (i+2)-th (UNION ALL) element. If it's SELECT, it indicates a subquery, and skipped; otherwise, the next (i+1) element is the source table name. If the i-th element (i = 0, 1, 2... array length - 1) of the array is LOAD, it indicates a Hive load function (load data table name), and the (i+2)-th element is the source table name.

[0076] Exception handling. Since this involves large-scale parsing, it's crucial to catch exceptions to prevent errors during SQL parsing from affecting the normal processing of other data.

[0077] Step 4.3: Data Visualization. Save the analysis results to the lineage model table and visualize them. Use visualization tools and techniques to display the lineage diagram, providing an interactive interface for users to explore and analyze. Visualization tools can be used to display the data as charts, supporting filtering by analysis time, workflow ID, job ID, source type, target type, source name, and target name. The data can be displayed in tables or as graphs for data governance personnel to analyze. Figure 6 and Figure 7 As shown, Figure 6 This is a schematic diagram illustrating a data relationship according to an embodiment of this application. Figure 7 This is a schematic diagram of a visual tool interaction interface according to an embodiment of this application.

[0078] It should be noted that the method provided by this invention only requires embedding a piece of log printing code in a pre-defined format before the database operation code. When a database operation triggers the recording of key information about blood relations, it can record blood relations information in real time and has the characteristics of strong portability and low invasiveness, without affecting the database operation.

[0079] In summary, this application uses a day-end batch processing procedure to parse log files, ensuring that data lineage relationships reflect the latest state of data flow and changes in real time, reducing the time delay of data lineage updates, and ensuring the timeliness and accuracy of data lineage information. Through refined modeling, a more detailed lineage graph model is designed to accurately represent the complex relationships between data. This includes defining more lineage relationship types (such as parent-child relationships, dependency relationships, upstream and downstream relationships, etc.). Simultaneously, this application supports multiple data sources, ensuring that the automatic data lineage analysis function supports various data sources, including relational databases, non-relational databases, data warehouses, and data lakes. This helps to build a comprehensive view of data lineage, improving data integrity and consistency. Furthermore, by regularly monitoring the performance and effectiveness of the automatic data lineage analysis, its impact on data quality management is assessed. Based on the monitoring and assessment results, optimization strategies and methods are adjusted and optimized in a timely manner.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software device. This computer software device is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0081] This embodiment also provides a data relationship updating device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0082] Figure 8 This is a structural block diagram of a data relationship updating device 800 according to an embodiment of this application; as shown below. Figure 8 As shown, it includes:

[0083] Add module 82, used to add a real-time record field before the first character of the corresponding operation field in the target database to obtain the target field. The real-time record field is used to record key lineage information during the process of calling the target database. The key lineage information includes: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database.

[0084] Collection module 84 is used to collect target information for populating the target field when a data call operation occurs in the target database, and obtain a target log file;

[0085] The update module 86 is used to parse and process the target log file to obtain the bloodline change information corresponding to the current data call, and update the data relationship between different data in the bloodline relationship graph bound to the target database based on the bloodline change information.

[0086] The aforementioned data relationship update device adds a real-time recording field, the target field, before the operation field in the target database. This target field specifically captures and stores key lineage information during the process of calling the target database, including important details such as data lineage type, lineage relationship, and data flow. When the target database performs a data call operation, it automatically collects related lineage information, forming a target log file. Subsequently, these log files are processed using intelligent parsing methods to extract the lineage relationship changes caused by the data call. This change information is used to update the lineage relationship diagram associated with the target database in real time, reflecting the latest relationship changes between different data in the target database. This scheme ensures that the performance of database operations is not affected, and through batch parsing of log files, the data flow path can be efficiently analyzed to determine the updated lineage relationship diagram, improving the efficiency of continuous tracking of target data. This technical solution solves the problem of how to efficiently maintain data lineage relationships in a big data environment, reducing manual maintenance costs and improving data management efficiency. It achieves the goal of embedding additional recording logic into the database operation code, enabling the real-time capture and recording of key lineage information whenever a database operation occurs.

[0087] In an exemplary embodiment, the above apparatus further includes: an identification module, configured to identify the source name and target name corresponding to the data call operation recorded in the target log file before parsing the target log file to obtain the lineage change information corresponding to the current data call, wherein the source name is used to locate the data source of the target data and the target name is used to locate the data flow of the target data; and to split the basic data bytes associated with the source name and the target name to determine the first field corresponding to the source name and the second field corresponding to the target name.

[0088] In an exemplary embodiment, the aforementioned identification module is further configured to perform data cleaning on the basic data bytes to obtain a first string; divide the character elements in the first string by spaces to obtain multiple second strings; identify special characters in the multiple second strings; determine a first field based on the special characters and a preset name identification rule; and determine a second field based on the special characters and a preset name query rule.

[0089] In an exemplary embodiment, the above-mentioned identification module further includes: a capture unit, configured to, after splitting the basic data bytes associated with the source name and the target name, and determining the first field corresponding to the source name and the second field corresponding to the target name, capture abnormal characters that occur during the splitting process, generate an abnormal character set; determine the normal character set corresponding to the abnormal character set; and establish a verification character library based on the correspondence between the same character in the abnormal character set and the normal character set.

[0090] In one exemplary embodiment, the apparatus further includes: an acquisition module, configured to acquire the update result of the kinship graph after updating the data relationships between different data in the kinship graph bound to the target database based on kinship change information; if the update result indicates that the kinship graph update has failed, send a prompt message to the management object corresponding to the kinship graph, wherein the prompt message is used to indicate that an abnormality has occurred in the acquisition of kinship change information and that the kinship change information of the target database needs to be collected again; and if the update result indicates that the kinship graph update has been successful, record the update time when the update is completed.

[0091] In an exemplary embodiment, the above-described apparatus further includes: a determining module, configured to, after recording the update time when the update result indicates that the kinship diagram has been successfully updated, determine the image content to be displayed on the kinship diagram upon receiving a display instruction input by the managed object, wherein the display instruction is used to set the content features to be displayed on the kinship diagram; highlight the image content on the kinship diagram and blur the non-highlighted parts of the kinship diagram to obtain visualized kinship data for display on a control terminal associated with the target database.

[0092] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0093] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0094] S1, add a real-time record field before the first character of the corresponding operation field in the target database to obtain the target field. The real-time record field is used to record key lineage information during the process of calling the target database. The key lineage information includes: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database.

[0095] S2, when a data retrieval operation occurs in the target database, collect the target information used to populate the target field to obtain the target log file;

[0096] S3, parse the target log file to obtain the bloodline change information corresponding to the current data call, and update the data relationship between different data in the bloodline relationship graph bound to the target database based on the bloodline change information.

[0097] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0098] Embodiments of this application also provide a computer program product, including a computer program, wherein the computer program, when executed by a processor, performs the steps in any of the above method embodiments.

[0099] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0100] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0101] S1, add a real-time record field before the first character of the corresponding operation field in the target database to obtain the target field. The real-time record field is used to record key lineage information during the process of calling the target database. The key lineage information includes: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database.

[0102] S2, when a data retrieval operation occurs in the target database, collect the target information used to populate the target field to obtain the target log file;

[0103] S3, parse the target log file to obtain the bloodline change information corresponding to the current data call, and update the data relationship between different data in the bloodline relationship graph bound to the target database based on the bloodline change information.

[0104] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical discs.

[0105] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0106] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0107] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for updating data relationships, characterized in that, include: A real-time record field is added before the first character of the corresponding operation field in the target database to obtain the target field. The real-time record field is used to record key lineage information during the process of calling the target database. The key lineage information includes: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database. When a data retrieval operation occurs in the target database, target information used to populate the target fields is collected to obtain a target log file; The target log file is parsed to obtain the bloodline change information corresponding to the current data call, and the data relationship between different data in the bloodline relationship graph bound to the target database is updated based on the bloodline change information.

2. The data relationship updating method according to claim 1, characterized in that, Before parsing the target log file to obtain the lineage change information corresponding to the current data call, the method further includes: Identify the source name and target name corresponding to the data retrieval operation recorded in the target log file, wherein the source name is used to locate the data source of the target data, and the target name is used to locate the data flow direction of the target data; The basic data bytes associated with the source name and the target name are split to determine the first field corresponding to the source name and the second field corresponding to the target name.

3. The data relationship updating method according to claim 2, characterized in that, The basic data bytes associated with the source name and the target name are split, including: The basic data bytes are cleaned to obtain the first string; Separate the character elements in the first string using spaces as separators to obtain multiple second strings; Identify special characters in the plurality of second strings, and determine the first field based on the special characters and a preset name recognition rule; The second field is determined based on the special characters and the preset name query rules.

4. The data relationship updating method according to claim 3, characterized in that, After splitting the basic data bytes associated with the source name and the target name to determine the first field corresponding to the source name and the second field corresponding to the target name, the method further includes: Capture abnormal characters that occur during the splitting process and generate a set of abnormal characters; Determine the set of normal characters corresponding to the set of abnormal characters; A verification character library is established based on the correspondence between the abnormal character set and the normal character set for the same character.

5. The data relationship updating method according to claim 1, characterized in that, After updating the data relationships between different data in the kinship graph bound to the target database based on the kinship change information, the method further includes: Obtain the updated result of the bloodline diagram; If the update result indicates that the bloodline diagram update has failed, a prompt message is sent to the management object corresponding to the bloodline diagram. The prompt message is used to indicate that the bloodline change information acquisition has failed and the bloodline change information in the target database needs to be collected again. If the update result indicates that the kinship diagram has been successfully updated, the update time is recorded.

6. The data relationship updating method according to claim 5, characterized in that, If the update result indicates that the kinship diagram has been successfully updated, after recording the update time, the method further includes: Upon receiving a display instruction input by the managed object, the image content to be displayed in the kinship diagram is determined, wherein the display instruction is used to set the content features of the kinship diagram to be displayed; The image content is highlighted in the kinship diagram, and the non-highlighted parts of the kinship diagram are blurred to obtain visualized kinship data for display on the control terminal associated with the target database.

7. A data relationship updating device, characterized in that, include: An add module is used to add a real-time record field before the first character of the corresponding operation field in the target database to obtain the target field. The real-time record field is used to record key lineage information during the process of calling the target database. The key lineage information includes: lineage type information of the target data called from the target database, lineage relationship information of the target data called from the target database, and data flow direction of the target data called from the target database. The collection module is used to collect target information for populating the target fields when a data retrieval operation occurs in the target database, and obtain a target log file; The update module is used to parse and process the target log file to obtain the bloodline change information corresponding to the current data call, and update the data relationship between different data in the bloodline relationship graph bound to the target database based on the bloodline change information.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6.

Images