Data lake-oriented external data repair method and device, equipment and storage medium

Abstract

This application discloses an external data repair method, apparatus, device, and storage medium for data lakes, relating to the fields of financial technology and data engineering technology. The method includes: acquiring scheduling information and reference relationship information of external data in the data lake; determining corresponding downstream protection link information based on the scheduling and reference relationship information; monitoring corresponding data entry start operations and abnormal failure operations based on the downstream protection link information to determine the data entry start operation information and abnormal failure operation information; identifying emergency handling tasks corresponding to the affected downstream operation links based on the data entry start operation information and abnormal failure operation information to determine emergency handling task information; and controlling the system based on the emergency handling task information to update or repair the data in the data lake. This application significantly improves response efficiency and ensures data lake security by accurately identifying affected downstream operation links and automatically generating emergency task control data repair.

Description

Technical Field

[0001] This application relates to the fields of financial technology and data engineering technology, and in particular to external data repair methods, apparatus, devices and storage media for data lakes. Background Technology

[0002] With the widespread application of enterprise-level data lakes in industries such as finance, a large amount of external data is introduced to support business decisions and operations in order to make up for the lack of internal data. However, external data has significant uncertainties in terms of source, quality, and stability, which can bring significant security and risk challenges to the data link. Therefore, it is necessary to effectively manage the risks of using external data in the data lake and ensure the safe and reliable operation of the data link.

[0003] Currently, the existing practice involves using lineage analysis tools to assess the scope of impact and relying on manual investigation and communication to locate the source of the problem and promote remediation when data anomalies or interruptions occur. However, the existing practice cannot identify and avoid potential risks before data is introduced, relies on manual coordination, is time-consuming to locate the responsible party, has low collaboration efficiency, and requires rerunning the entire process after remediation. It lacks automated self-healing capabilities, has high operation and maintenance costs, and a long remediation cycle. At the same time, risk information is not transparent, downstream users cannot predict the risks of using external data, and there is a lack of effective data usage guidance and backup plans. Therefore, how to more accurately and effectively identify the security and risks of external data has become an urgent problem to be solved.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an external data repair method, apparatus, device, and storage medium for data lakes, aiming to solve the technical problem of how to more accurately and effectively identify the security and risks of external data.

[0006] To achieve the above objectives, this application proposes an external data repair method for data lakes, the method comprising:

[0007] Obtain scheduling and reference relationship information of external data within the data lake; The corresponding downstream support link information is determined based on the scheduling information and the reference relationship information; Based on the downstream support link information, the corresponding data entry start operation and abnormal failure operation are monitored to determine the data entry start operation information and abnormal failure operation information. Based on the data entry start operation information and the abnormal failure operation information, the emergency handling tasks corresponding to the downstream affected operation links are identified, and the emergency handling task information is determined. The emergency response task information control system updates or repairs the data in the data lake.

[0008] In one embodiment, the step of determining the corresponding downstream guarantee link information based on the scheduling information and the reference relationship information includes: Based on the scheduling information and the reference relationship information, the corresponding downstream support link operation increment is identified, and the downstream support operation increment information is determined; Based on the incremental information of downstream support operations and the predefined risk knowledge base, the corresponding source risks are matched to determine the source risk information; Based on the aforementioned source-end risk information, the corresponding contract information and compatibility modification information are determined; The downstream support operation incremental information, the source risk information, the contract information, and the compatibility modification information are used to obtain the downstream support link information.

[0009] In one embodiment, the step of identifying the job increment of the corresponding downstream support link based on the scheduling information and the reference relationship information, and determining the downstream support job increment information, includes: The link change status of the scheduling information and the reference relationship information within a predefined time period is identified by comparing the difference of snapshot data, and the link change information is determined, including the addition of dependency information and the deletion of dependency information. Based on the timeliness and quality tags of the link change information in the data lineage, the corresponding new downstream support operations are identified, and the information of the new downstream support operations is determined.

[0010] In one embodiment, the step of matching the corresponding source-end risks based on the downstream support operation incremental information and a predefined risk knowledge base to determine the source-end risk information includes: Obtain the results of the source-end risk investigation; The predefined risk knowledge base is updated based on the source-end risk survey results to determine the target risk knowledge base; Based on the matching of the newly added downstream support operation information with the source risk records in the target risk knowledge base, the source risk matching result is determined; Based on the source-end risk matching results, corresponding source-end risk information is generated.

[0011] In one embodiment, the step of identifying emergency response tasks corresponding to downstream affected operation links based on the data inflow initiation operation information and the abnormal failure operation information, and determining the emergency response task information, includes: Based on the data entry into the lake starting operation information and the predefined source-end contract knowledge base, the corresponding upstream anomaly root causes are identified, and the upstream anomaly root cause information is determined. The upstream anomaly root cause information includes source system person in charge information, object name information and extraction condition information. By using the lineage analysis of the data warehouse and the abnormal fault operation information, the root causes of the corresponding downstream abnormalities and downstream cross-layer abnormalities are identified, and the root cause information of the downstream abnormalities is determined. The root cause information of the downstream abnormalities includes the downstream person in charge information, the downstream operation name information, the downstream operation guarantee level information, and the downstream guarantee maximum timeliness requirement information. Based on the upstream anomaly root cause information and the downstream anomaly root cause information, the affected downstream operation links are pruned to determine the pruning operation link information; Based on the pruning operation link information, the corresponding emergency response tasks are identified to obtain emergency response task information.

[0012] In one embodiment, the step of identifying the corresponding emergency response task based on the pruning operation link information and obtaining emergency response task information includes: Obtain the emergency response group interface; Call the emergency response group interface to create an emergency response group corresponding to the downstream link impact information and the person in charge impact information in the pruning operation link information; Based on the emergency response group and the predefined large language model, the emergency response tasks of each person in the group will be identified to obtain emergency response task information.

[0013] In one embodiment, the step of updating or repairing data in the data lake based on the emergency response task information control system includes: Obtain the data lake table storage format, which includes sliced ​​tables and non-sliced ​​tables; Based on the emergency response task information, the data links corresponding to the data entry table storage format are configured, and the first link information and the second link information are determined. In response to the first link information and the second link information, the control system updates or repairs the data in the data lake using a predefined data self-healing scheme.

[0014] Furthermore, to achieve the above objectives, this application also proposes an external data repair device for data lakes, the external data repair device for data lakes comprising: The upstream and downstream contract signing module is used to obtain scheduling information and reference relationship information of external data in the data lake; The upstream and downstream contract signing module is also used to determine the corresponding downstream guarantee link information based on the scheduling information and the reference relationship information; The link analysis and collaborative processing module is used to monitor the corresponding data entry start operation and abnormal failure operation based on the downstream support link information, and determine the data entry start operation information and abnormal failure operation information. The link analysis and collaborative processing module is also used to identify emergency handling tasks corresponding to downstream affected operation links based on the data entry start operation information and the abnormal fault operation information, and to determine emergency handling task information. The data self-healing module is used to update or repair the data in the data lake based on the emergency response task information control system.

[0015] Furthermore, to achieve the above objectives, this application also proposes an external data repair device for data lakes, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the external data repair method for data lakes as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the external data repair method for data lakes as described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This embodiment proposes an external data repair method for data lakes. The method acquires scheduling and reference relationship information of external data within the data lake; determines corresponding downstream support link information based on the scheduling and reference relationship information; monitors corresponding data entry initiation operations and abnormal failure operations based on the downstream support link information, determining the data entry initiation operation information and abnormal failure operation information; identifies emergency handling tasks corresponding to the affected downstream operation links based on the data entry initiation operation information and abnormal failure operation information, determining emergency handling task information; and controls the system to update or repair the data within the data lake based on the emergency handling task information. This application, by accurately identifying downstream support links and monitoring data entry initiation operations and abnormal failure operations in real time, automatically identifies affected downstream links, generates corresponding emergency handling tasks, and controls the system to update or repair the data within the data lake. This significantly improves emergency response efficiency and collaboration levels, effectively reduces operation and maintenance costs and system resource consumption, and fundamentally ensures the security and business continuity of external data use within the data lake. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a system architecture diagram of the external data repair method for data lakes proposed in this application; Figure 2 This is a flowchart illustrating an embodiment of the external data repair method for data lakes provided in this application. Figure 3 This application provides a schematic diagram illustrating the incremental link and change identification for the external data repair method for data lakes. Figure 4 This is a schematic diagram of the emergency risk avoidance collaboration process for the external data repair method for data lakes in this application. Figure 5 This is a schematic diagram of the data slice self-healing scheme for external data repair methods for data lakes in this application. Figure 6 This is a flowchart illustrating Embodiment 2 of the external data repair method for data lakes provided in this application; Figure 7 This is a schematic diagram illustrating cross-organizational, one-layer object identification for the external data repair method for data lakes in this application. Figure 8 This is a schematic diagram of the module structure of an external data repair device for a data lake according to an embodiment of this application; Figure 9 This is a schematic diagram of the hardware operating environment involved in the external data repair method for data lakes in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: acquiring the scheduling information and reference relationship information of external data in the data lake; determining the corresponding downstream support link information based on the scheduling information and the reference relationship information; monitoring the corresponding data entry start operation and abnormal failure operation based on the downstream support link information, and determining the data entry start operation information and abnormal failure operation information; identifying the emergency handling tasks corresponding to the downstream affected operation links based on the data entry start operation information and abnormal failure operation information, and determining the emergency handling task information; and controlling the data in the data lake to update or repair based on the emergency handling task information.

[0025] In this embodiment, for ease of description, the following description will focus on identifying an external data repair device for the data lake as the execution subject.

[0026] Because existing technologies cannot identify and mitigate potential risks before data is introduced, they rely on manual coordination, which is time-consuming to locate the responsible party, resulting in low collaboration efficiency. Furthermore, the entire process needs to be restarted after repair, lacking automated self-healing capabilities, leading to high operation and maintenance costs and long repair cycles. At the same time, risk information is not transparent, downstream users cannot predict the risks of using external data, and there is a lack of effective data usage guidance and fallback solutions.

[0027] This application provides a solution, such as Figure 1 As shown, Figure 1 This is a system architecture diagram for the external data repair method for data lakes proposed in this application. The system obtains scheduling and reference information of external data within the data lake through upstream and downstream contracting modules, enabling accurate identification of downstream support links. Pre-emptive risk assessment and compatible design modifications are achieved through source-end risk surveys and downstream fallback contracts. Based on this, the automated link analysis and collaborative processing module can monitor the initiation of data entry into the data lake and abnormal failure operations in real time, automatically locating affected downstream links. It generates corresponding emergency processing tasks through automatic group creation and emergency collaborative processing mechanisms, ensuring rapid response and multi-party collaboration during incidents. The data self-healing module triggers differentiated self-healing solutions (daily repair, fixed window period repair, or manual triggering) based on emergency processing task information and data table storage format (e.g., sliced ​​or non-sliced ​​tables), controlling the system to automatically update or repair abnormal data within the data lake. This forms a closed-loop management system from risk identification and emergency response to automated repair, significantly improving emergency response efficiency, reducing manual maintenance costs, and ensuring the security and business continuity of external data usage.

[0028] As can be seen from the above embodiments, this application can accurately identify downstream support links and monitor the data entry start operation and abnormal failure operation in real time, thereby automatically identifying the downstream affected links, generating corresponding emergency handling tasks, and controlling the system to update or repair the data in the data lake, significantly improving emergency response efficiency and collaboration level, effectively reducing operation and maintenance costs and system resource consumption, and fundamentally ensuring the security and business continuity of external data use in the data lake.

[0029] Based on this, embodiments of this application provide an external data repair method for data lakes, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the external data repair method for data lakes in this application.

[0030] In this embodiment, the external data repair method for data lakes includes steps S10 to S50: Step S10: Obtain the scheduling information and reference relationship information of external data in the data lake; It should be noted that the scheduling information refers to the job operation configuration information recorded in the job scheduling system within the data lake, such as the job start time, execution cycle, job dependencies, and current running status. The reference relationship information refers to the lineage dependency relationship between data tables and between jobs and data tables in the data lake, such as the correspondence between the generation of a downstream data table and the output of an upstream data table, or the correspondence between the input source of a data processing job and an external data table.

[0031] It is understood that the external data mentioned above is relative to the enterprise's internally generated data. That is, it is data obtained from external suppliers, institutions or public channels through procurement, cooperation or API calls. This type of data can be used to make up for the deficiencies of the enterprise's internal data to support richer business decisions (such as marketing leads, risk rating, market analysis). However, since the source, format, quality and delivery stability of external data are not directly controlled by the enterprise, there will be high uncertainty and risk after it is introduced into the data lake.

[0032] Additionally, it should be noted that the data lake is a centralized repository capable of storing massive amounts of heterogeneous data from multiple sources in their raw format. It supports on-demand processing and large-scale analysis of the data in read-time mode, where data can be stored without prior modeling, thus enabling the rapid absorption of external data sources. However, this also places higher demands on data quality and stability management.

[0033] In a specific embodiment, as an optional implementation, the upstream and downstream contracting modules can periodically scan and obtain the scheduling information of upstream data access jobs corresponding to all external data tables by connecting to the underlying metadata repository of the data lake (such as Hive Metastore, metadata service of data lake tables) and job scheduling engine (such as Airflow, DolphinScheduler). At the same time, by parsing the job lineage graph generated by the SQL parser, it can obtain which downstream jobs or data tables reference the external data table, thereby constructing a complete set of scheduling information and reference relationship information. For example, an external data table "ext_market_data_di" is responsible for entering the lake by a daily scheduled job "job_ext_market_etl". The execution status and execution time of this job are the scheduling information. Meanwhile, the lineage analysis shows that the table is depended on by multiple downstream jobs (such as "job_dwd_sales" and "job_dws_report"). These dependencies are the reference relationship information.

[0034] In a specific embodiment, as another optional implementation, if the data lake platform itself provides a unified data asset management and lineage tracing API interface, then by calling these interfaces, using the unique identifier of the external data table (such as the table name or data source ID) as input parameters, it is possible to query and obtain its complete upstream scheduling information and downstream reference relationship information in real time without directly accessing the underlying storage.

[0035] In specific embodiments, to ensure the timeliness and accuracy of information, the acquisition process can be set to be triggered daily at set times, or dynamically triggered when a change event related to external data operations is received (such as adding dependencies, deleting dependencies, or modifying scheduling configuration). If the scheduling information or reference relationship information of a certain external data table cannot be obtained, it indicates that the data table has not yet been included in the standard management process of the data lake, or its lineage link has been interrupted. At this time, the system can record the anomaly and prompt the administrator to conduct manual verification in order to avoid blind spots in subsequent risk control.

[0036] Step S20: Determine the corresponding downstream guarantee link information based on the scheduling information and the reference relationship information; It should be noted that the downstream support link information refers to the set of specific downstream operations and their related attributes that are marked as requiring key support in the operation dependency links of the data lake.

[0037] It is understood that the downstream support link information may include the name of the data processing job, the person responsible for the job, the support level of the job (e.g., L1 to L5 levels divided according to business timeliness and importance, where L1 to L3 are support jobs), the timeliness requirements of the job (e.g., must be completed before 8:00 a.m. every day), and the upstream external data table information on which the job depends. This information is used to identify when the external data is abnormal and the corresponding downstream job will be affected, and to formulate risk response strategies in advance to ensure the continuity of critical business links.

[0038] In a specific embodiment, as an optional implementation, the link change status of the scheduling information and the reference relationship information within a predefined time period is identified by comparing snapshot data differences to determine the link change information, which includes added dependency information and deleted dependency information. Based on the timeliness and quality tags of the link change information in the data lineage, the corresponding downstream new support operations are identified to determine the downstream new support operation information, i.e., as shown below. Figure 3 As shown, Figure 3 This application illustrates the external data repair method for data lakes, ensuring incremental link identification and change recognition. The upstream and downstream contracting modules utilize downstream scheduling information and object reference relationships from external data within the data lake to accurately identify link change statuses within a predefined timeframe. These link change statuses refer to newly added dependencies or jobs downstream in the current period (Day T) compared to the previous day (T-1) or previous period. A snapshot data difference comparison method is used, performing a full snapshot of all job scheduling dependencies and reference relationships in the data lake daily. The snapshot from Day T is then compared item by item with the snapshot from Day T-1 to identify newly added dependencies or jobs. The system identifies dependencies (new dependency information) and deleted dependencies (deleted dependency information). New dependency information indicates that a downstream job has added a reference to an external data table, or a new job has been launched. Deleted dependency information indicates that a reference relationship no longer exists. Based on this, and combined with the timeliness and quality tags preset in the data lineage (where L1 to L3 are defined as guaranteed jobs, and L4 to L5 are non-guaranteed jobs), the identified link change information is filtered and mapped to determine the new downstream guaranteed job information, thereby realizing the automated identification of incremental changes and changes in external data guarantee links.

[0039] The process involves: obtaining the source-end risk survey results; updating the predefined risk knowledge base based on these results to determine the target risk knowledge base; matching the newly added downstream support operation information with the source-end risk records in the target risk knowledge base to determine the source-end risk matching result; and generating corresponding source-end risk information based on the matching result. Specifically, after identifying newly added downstream support operation information, the upstream and downstream contracting modules will automatically match it with the predefined risk knowledge base. If the matching result indicates that the operation is not yet associated with any risk records, a notification will be automatically pushed to the upstream table manager via a chatbot, triggering the source-end risk survey process, whereby the manager will identify and assess the source-end risks. The survey results are entered into the risk knowledge base for updating. If relevant risk records already exist in the knowledge base, the system directly pushes a reminder to the person in charge for risk confirmation. Based on this, the source-end risk survey results are obtained, and the predefined risk knowledge base is dynamically updated based on the survey results to determine the target risk knowledge base. Then, the source-end risk matching results are obtained by matching the newly added downstream protection operation information with the source-end risk records in the target risk knowledge base, and the corresponding source-end risk information is generated. The source-end risk information includes assessments of source-end stability, stop-throw risk, etc., and provides the person in charge of the data entry table with a summary basis for risk processing script identification and modification schemes in combination with the current downstream usage.

[0040] Based on the aforementioned source-end risk information, corresponding contract information and compatibility modification information are determined. Specifically, the upstream and downstream contract modules can further determine the corresponding contract information and compatibility modification information based on the identified source-end risk information. Regarding contract signing, for identified source-end risks, specific fallback strategies and emergency response responsibilities are agreed upon with downstream data users. The contract results are recorded in the downstream contract ledger. Simultaneously, the specific data quality requirements of downstream data usage scenarios are collected, summarized into basic quality verification rules to intercept problems, and targeted modification guidance is provided based on downstream script compatibility to obtain contract information. Regarding compatibility modification, the downstream user's responsible person modifies their own script based on the source-end risk information. Compatibility design can include emergency mitigation compatibility design and data self-healing compatibility design. Emergency mitigation compatibility design means that when there is a risk of data loss upstream and the data lake uses date slicing tables, the downstream script needs to dynamically determine whether the data exists. If the data on day T-1 is empty, the data on day T-2 will be automatically retrieved to ensure business continuity. Data self-healing compatibility design is designed to adapt to the automatic repair process after upstream data anomalies. For example, when the source throws the data from the end of last month every day, the downstream needs to determine whether to trigger the script to be re-executed based on the data update date. Since the repair methods of different data links vary depending on the data provision form, it is necessary to agree on them one by one and formulate personalized solutions to form compatibility transformation information.

[0041] The downstream support operation incremental information, the source risk information, the contract information, and the compatibility modification information are used to obtain the downstream support link information.

[0042] In a specific embodiment, as another optional implementation method, in addition to the incremental identification method, the system can also periodically (e.g., weekly) perform a full rescan of all existing downstream protection links to ensure that the risk knowledge base and contract information are consistent with the current situation. During the full scan, the system no longer only focuses on new dependencies, but matches all protection operations with the risk knowledge base and cleans up or prompts for updates to expired risks or changed contract information.

[0043] In a specific embodiment, if no risk information can be matched in the risk knowledge base and the source risk investigation has not been completed, the system will mark the protection link as pending investigation and continuously remind the user until the risk information is entered. This ensures that all newly added critical links have been risk-predicted before being put into production, avoiding risk blind spots. Thus, when the data link is established or changes occur, the system will automatically identify the critical protection link and promote relevant parties to complete risk identification, signing, and compatibility transformation.

[0044] In one feasible implementation, step S20 may include steps A11 to A14: Step A11: Based on the scheduling information and the reference relationship information, identify the corresponding downstream support link's job increment and determine the downstream support job increment information; It should be noted that the incremental information of downstream support operations is obtained by analyzing the scheduling dependencies and reference relationships of external data in the data lake, identifying newly added operations in the downstream links that require key support and their related attribute sets, which are used to mark newly established data processing tasks that have an impact on business continuity.

[0045] In one feasible implementation, step A11 may include steps B11-B12: Step B11: Identify the link change status of the scheduling information and the reference relationship information within a predefined time by comparing the difference of snapshot data, and determine the link change information, which includes added dependency information and deleted dependency information; It should be noted that the link change information is identified by comparing the scheduling dependencies and reference relationship snapshots of day T and day T-1, and the changes in dependencies are shown as newly added or deleted dependency edges.

[0046] It is understandable that adding dependency information means that a downstream job has added a reference to an upstream external data table or a new dependent job has been launched, while deleting dependency information means that the original reference relationship no longer exists.

[0047] Step B12: Based on the timeliness and quality tags of the link change information in the data lineage, identify the corresponding new downstream support operations and determine the new downstream support operation information.

[0048] It should be noted that the newly added downstream support operation information is selected from the link change information, combined with the timeliness and quality tags preset in the data lineage, and the newly added dependencies that belong to the high support level (such as L1-L3 level) and their related attributes, including operation name, responsible person, support level, timeliness requirements, etc., to clarify that the newly added links need to be included in the key control scope.

[0049] Understandably, the timeliness tag in the data lineage is a level divided according to the business importance of the operation, while the quality tag represents the data quality requirements of the operation. This automatically filters out non-critical link changes and retains only the incremental operations that have a substantial impact on business continuity, thereby achieving accurate risk identification and resource focus.

[0050] Step A12: Based on the incremental information of downstream support operations and the predefined risk knowledge base, match the corresponding source risks to determine the source risk information; Understandably, a risk knowledge base is a structured database that centrally stores and manages risk information from external data sources. It originates from the continuous accumulation and dynamic maintenance of risk survey results at the source, and records the risk items identified by each data source and corresponding handling suggestions.

[0051] In one feasible implementation, step A12 may include steps C11 to C14: Step C11: Obtain the results of the source-end risk survey; It should be noted that the source-end risk survey results are feedback information on the risk status of external data sources collected by conducting surveys with the person in charge of the data lake table. This includes assessments of the stability of the source system, the reliability of data delivery, historical anomaly records, potential risk points, as well as the person in charge's identification conclusions and preliminary modification suggestions for the risk processing script.

[0052] It is understandable that the process of obtaining the source-end risk survey results can be triggered by automated tools. When the system detects a new protection operation and there is no matching record in the risk knowledge base, it automatically pushes the survey task to the person in charge to ensure the timely collection and supplementation of risk information.

[0053] Step C12: Update the predefined risk knowledge base based on the source-end risk survey results to determine the target risk knowledge base; It should be noted that the target risk knowledge base is an updated version of the latest risk records formed by merging the latest source-end risk survey results into the original risk knowledge base. This knowledge base not only retains historical risk data, but also incorporates risk items identified by new surveys, ensuring the timeliness and completeness of risk information.

[0054] Step C13: Based on the newly added downstream support operation information, match it with the source-end risk records in the target risk knowledge base to determine the source-end risk matching result; It should be noted that the source-end risk matching result is the specific matching result obtained by comparing the external data source identifier on which the downstream new protection operation depends with the risk records in the target risk knowledge base. It directly determines whether the current new protection operation is associated with known risks and which risks are associated with it.

[0055] Step C14: Generate corresponding source-end risk information based on the source-end risk matching results.

[0056] It should be noted that the source-end risk information refers to the specific risk descriptions identified by matching the external data sources relied upon by the downstream new support operations with the risk knowledge base. These include risk types (such as insufficient system stability, data loss risk, update delays, format changes, etc.), risk levels, historical frequency of occurrence, potential impact scope, and suggested countermeasures.

[0057] Step A13: Determine the corresponding contract information and compatibility modification information based on the source-end risk information; It should be noted that the contract information is based on the emergency handling strategies and responsible person records agreed upon between the source-end risk and the downstream data user, including risk mitigation plans, response time requirements, and responsible persons, and is stored in the downstream contract ledger. The compatibility modification information is the technical adaptation plan of the downstream user's responsible person for their own script based on the source-end risk characteristics, including emergency risk avoidance compatibility design (such as automatically retrieving historical data when data is missing) and data self-healing compatibility design (such as triggering script rerun based on data update date), thereby enhancing the downstream link's fault tolerance and automated repair capabilities against upstream risks.

[0058] Step A14: Obtain downstream support link information based on the downstream support operation incremental information, the source risk information, the contract information, and the compatibility modification information.

[0059] Understandably, downstream support link information integrates and encapsulates newly identified support operations, matched source risks, agreed contractual strategies, and implemented compatibility modification schemes to form a complete description of each downstream support link. Through this integration, the scope of links that need to be focused on for control can be clearly identified.

[0060] Step S30: Based on the downstream support link information, monitor the corresponding data entry start operation and abnormal fault operation to determine the data entry start operation information and abnormal fault operation information. It should be noted that the data entry start-up job information refers to the relevant information of the specific job that first loads external data from the source system into the data lake. This includes the job's identifier (job ID), job name, scheduling configuration (such as execution frequency and start time), current running status (such as running, successful, failed, waiting), job responsible person, and the corresponding data table information (table name, partition information, etc.). This information is used to identify the entry point for external data into the data lake. When external data is abnormal, it can be traced back to the source job to analyze whether the problem is due to the source data not being delivered, the extraction process failing, or other issues.

[0061] It is understood that the abnormal fault job information refers to the relevant information of jobs in the data lake job chain that are captured by the monitoring system and are in an abnormal state (such as running failure, timeout failure, data quality verification failure). This includes the basic identifier of the abnormal job (job ID, name), the type of abnormality (such as scheduling failure, SQL execution error, empty data, duplicate primary key), the time of occurrence of the abnormality, the abnormality details log, and the dependency relationship between the abnormal job and its upstream and downstream operations, so as to determine which downstream support jobs will be affected.

[0062] In a specific embodiment, as an optional implementation, the link analysis and collaborative processing module can extract a list of all jobs marked as assurance jobs from the downstream assurance link information, as well as the upstream data inflow initiation jobs that these jobs depend on, and continuously track the running status of these jobs. Specifically, the system can periodically (e.g., every 5 minutes) call the API interface of the job scheduling system to query the latest running status of assurance jobs and their upstream data inflow initiation jobs, and obtain data inflow initiation job information. For jobs with a status of failure or running timeout, the system automatically captures the detailed abnormal logs of the job and, in conjunction with its scheduling history, generates abnormal failure job information. For example, if the data inflow initiation job "job_ext_market_data" is detected to have failed, the system will record the failure time and error information (such as "connection to source database timeout") of the job and mark it as an abnormal failure job.

[0063] In a specific embodiment, as another optional implementation, in addition to the failure status actively reported by the scheduling system, the system can also capture runtime anomalies of the job by listening to the real-time log stream. When the job throws a specific error code during operation, the log collection component pushes this anomaly information to the system's anomaly event center in real time. The event center determines whether it is an anomaly or fault that needs attention according to preset rules, and automatically associates the downstream support link information corresponding to the job to generate anomaly or fault job information.

[0064] In specific embodiments, the monitoring scope is not limited to the operations in the assurance link, but can be expanded according to actual needs. For example, system administrators can manually configure non-assistance operations that need to be monitored, or set a global scanning strategy to perform basic monitoring on all data entry operations to prevent potential risks.

[0065] Step S40: Based on the data entry start operation information and the abnormal failure operation information, identify the emergency handling tasks corresponding to the downstream affected operation links and determine the emergency handling task information. Understandably, the downstream affected operation chain starts from the abnormal failure operation and spreads downstream along the data lineage. It is a set of paths formed by all data processing operations that may be interrupted, have delayed output, or generate erroneous data due to this anomaly, as well as their dependencies. This chain includes not only the downstream layer of operations that directly depend on the abnormal failure operation, but also the downstream operations that are indirectly affected through layers of dependencies, until all the support chain nodes.

[0066] In a specific embodiment, as an optional implementation, the upstream anomaly root causes are identified based on the data inflow initiation operation information and a predefined source-end contract knowledge base to determine the upstream anomaly root cause information. The upstream anomaly root cause information includes source system manager information, object name information, and extraction condition information. Using data warehouse lineage analysis and the anomaly / failure operation information, the corresponding downstream anomalies and downstream cross-layer anomaly root causes are identified to determine the downstream anomaly root cause information. The downstream anomaly root cause information includes downstream manager information, downstream operation name information, downstream operation support level information, and downstream support maximum timeliness requirement information. Based on the upstream and downstream anomaly root cause information, the affected downstream operation links are pruned to determine the pruning operation link information. Based on the pruning operation link information, the corresponding emergency handling tasks are identified to obtain emergency handling task information, i.e., as shown below. Figure 4 As shown, Figure 4This diagram illustrates the emergency risk mitigation and collaboration process for the external data repair method for data lakes proposed in this application. The link analysis and collaborative processing module comprises three functional modules: an upstream / downstream identification module, an automatic group creation module, and an emergency collaborative processing module. The upstream / downstream identification module, based on the initial data entry information, calls a predefined source-end contractual knowledge base through the upstream identification interface to automatically identify and determine the upstream anomaly root cause information, including source system manager information, object name information, and extraction condition information, for preliminary anomaly root cause judgment. Simultaneously, based on the anomaly failure operation information, it uses the downstream identification interface combined with data warehouse lineage analysis to perform real-time downstream impact range assessment of the anomaly failure operation, identifying downstream anomaly root cause information, including downstream manager information, downstream operation name information, downstream operation support level information, and downstream support maximum timeliness requirement information. The system then prunes affected downstream work links to identify pruning link information. Based on this, the automatic group creation module, when confirming that the source cannot resolve the issue promptly and there is a risk of data delays downstream, calls the chat software API to create an emergency handling group corresponding to the downstream link impact information and responsible person impact information in the pruning work link information. Relevant upstream and downstream responsible persons are added to the group. The emergency collaborative handling module allows maintenance personnel to edit information such as the problem background and estimated processing time in the group. Combining this with the emergency plans recorded in the upstream and downstream contract modules, the module uses a large language model to analyze and infer the emergency handling tasks for each downstream work responsible person. Emergency tasks are then issued through a chatbot, and the progress of the tasks is tracked synchronously until the anomaly is repaired. This achieves accurate identification of emergency handling tasks corresponding to the pruning work link information and fully automated emergency collaboration throughout the process.

[0067] In a specific embodiment, as another optional implementation, the generation of emergency response tasks can also be based on a rule engine instead of a large language model. The system predefines templates for different anomaly types and contractual schemes. When an anomaly occurs, it directly fills in variables (such as the person in charge, task name, and time) according to the matching template to generate task information. This method is simple to implement but lacks flexibility. This embodiment preferably uses a large language model because it can better understand complex contexts and generate more natural and accurate guidance.

[0068] For example, consider the following scenario: the initial data ingestion job "job_ext_market" fails, causing the downstream L2 backup job "job_dws_market_agg" to be unable to run. In this situation, the system identifies the source-end responsible person as Zhang San through upstream identification and determines the downstream responsible person to be notified as Li Si (responsible for job_dws_market_agg) through downstream identification and pruning. The pre-contractual agreement shows that the emergency backup plan for this job is "to use T-2 day snapshot data replacement when upstream data is delayed." The system inputs the anomaly information (data delay, estimated repair time of 2 hours) and the contractual plan into the large model. The large model generates the following task description: "Li Si, due to the failure of the upstream market data source system, today's data is expected to be delayed by 2 hours. According to the pre-contractual agreement, your job should immediately use T-2 snapshot data replacement processing to ensure that the report is produced on time. Please check whether the risk avoidance logic in the job configuration has taken effect and confirm that the replacement data has been successfully written within 1 hour. If you encounter any problems, please report them in the group." This task information is then recorded and distributed.

[0069] In a specific embodiment, if it is found during the identification process that some affected work links lack prior contract information, the system will specifically mark "no contract" in the emergency handling task information and suggest that the person in charge supplement the contract as soon as possible after the emergency to improve risk control. At the same time, the system will record these operations as high-risk points and promote the prior contract module to conduct supplementary research.

[0070] Step S50: Update or repair the data in the data lake based on the emergency response task information control system.

[0071] Understandably, tables in a data lake are categorized into sliced ​​tables and non-sliced ​​tables based on their data organization. Sliced ​​tables are tables where data is partitioned and stored according to a time dimension (such as date or month), for example, one partition per day (ds=20231001). Each partition contains a full snapshot of the data for that day or incremental data. This type of table is very common in fields such as finance and log analysis. Non-sliced ​​tables, on the other hand, do not have time partitioning or only store the latest full data. Each update will overwrite or replace the old data based on the primary key. Therefore, the system needs to select the most appropriate repair path based on the type of data table, the specific circumstances of the anomaly, and the pre-agreed repair plan.

[0072] In a specific embodiment, as an optional implementation method, the storage form of the data lake table is obtained. The storage form of the data lake table includes sliced ​​tables and non-sliced ​​tables. That is, the system can determine the storage form of the target table by querying the metadata service of the data lake. If the metadata shows a clear partition field and the data writing logic is to replace by partition, it is determined to be a sliced ​​table; otherwise, it is determined to be a non-sliced ​​table.

[0073] Based on the emergency response task information, the data links corresponding to the data lake table storage format are configured, and first link information and second link information are determined. In response to the first link information and the second link information, the control system updates or repairs the data in the data lake using a predefined data self-healing scheme. Specifically, based on the anomaly range locked in the emergency response task information (e.g., the data date to be repaired, the affected data table), combined with the data lake table storage format, the corresponding data links are configured, and first link information and second link information are determined. The first link information is for the repair link of non-sliced ​​tables, and the repair logic is relatively simple. When the source end completes data repair and throws the repaired full data along with the normal data of T+1 day, the system recognizes the data packet and starts the repair link. This link does not perform incremental insertion but instead performs an "upsert" (update or insert) operation based on the data primary key, updating the historical error records in the target table to correct values ​​while simultaneously writing new normal data. Downstream jobs, running on T+1 day, process data directly based on this repaired full table, thus automatically "self-healing" in the next cycle. The second link information is the repair link for slice tables, with more refined repair logic. The system needs to maintain a daily batch processing link and a data repair link for the same physical table. The daily batch processing link is the batch processing link under normal data conditions, such as a job configuration that extracts T-1 data from the source and writes it to the target table daily. The data repair link is specifically used for handling anomaly repairs. Its configuration differs from the daily link, aiming to accurately correct data in specific historical batches or the current batch without affecting other normal data. For example, link A (daily batch processing link): responsible for inserting T-1 data into the partition with the corresponding date (e.g., 20231001) daily. Link B (data repair link): responsible for receiving the repair data packet sent by the source. This data packet contains the data content to be repaired and its corresponding target data date (e.g., the data packet is marked "target_date=20230915"). The system parses this tag, dynamically generates a write path, and accurately updates the historical partition 20230915 without overwriting or affecting data in other partitions (such as 20230916). In this way, even if the source discovers and corrects the data error of day T-30 on day T and pushes it down through the repair link, the data lake can automatically put it back into the correct historical slice. If the downstream job is designed to read a specific partition (such as reading the end-of-month partition to create a report), it will automatically obtain the correct data after the repair on the next execution, without the need for manual rerunning of the entire link.

[0074] Therefore, after determining the daily and repair links, the system responds to the configuration of these two links and controls the system to update or repair the data in the data lake using a predefined data self-healing scheme. This data self-healing scheme is a set of rules that automatically trigger the repair execution to ensure that the repair action can be completed on demand and on time.

[0075] In a specific embodiment, as another optional implementation, the repair modes corresponding to the predefined data self-healing scheme may include a daily repair mode, a fixed window period repair mode, and a manually triggered repair mode. The daily repair mode sets the repair chain to execute daily, but its execution time is usually slightly later than the daily chain. This mode is suitable for scenarios where downstream businesses do not have specific date restrictions on data, i.e., as long as the data is ultimately correct. The system checks the input source of the repair chain daily (such as a specific "data to be repaired" directory). Once a repair package is found to have been pushed from the source, an upsert or partition update operation is immediately performed. The fixed window period repair mode is... The scheduler for the repair link is set to run within a fixed time window each month, such as daily execution from the 3rd to the 15th of each month. This perfectly fits scenarios with strict requirements for data timeliness, such as monthly reports and financial reports (e.g., they must be finalized before the 15th). Even if the source discovers errors in the previous month's data at the beginning of the month, it can be automatically corrected through the repair link within the window period, ensuring that the data is accurate when it is released on the 15th. The manual trigger repair mode serves as a fallback mechanism. When special situations occur that cannot be covered by the automation rules (e.g., emergency regulatory reporting, large-scale data corrections notified temporarily by the source), operations and maintenance personnel can trigger the repair link with one click through the system interface to immediately perform data repair.

[0076] For example, such as Figure 5 As shown, Figure 5 This diagram illustrates the data slicing self-healing scheme for external data repair in this application for data lakes. Assume there's a slice table named "dws_monthly_report" storing monthly financial reports. Its data depends on upstream external data. On the 5th of a certain month, the business discovers errors in the previous month's (T-1 month) report data. This is because a batch of data dropped by the source on T-1 month 30 was incorrect. The source repaired the data on T month 5 and dropped it through the repair link (link B). Upon receiving the repair packet, the data lake system parses the data date as T-1 month 30 and automatically updates the corresponding T-1 month 30 partition in the "dws_monthly_report" table. The downstream job "job_monthly_finance," responsible for generating the monthly report, is set to run on the 10th of each month. When it reads the "dws_monthly_report" table on the 10th, it retrieves the automatically repaired and correct data. The resulting monthly report is accurate, and the entire process requires no manual intervention to rerun the monthly report link.

[0077] In one feasible implementation, step S50 may include steps D11-D13: Step D11: Obtain the data lake table storage format, which includes sliced ​​tables and non-sliced ​​tables; It should be noted that the storage format of the data lake table is the organization method of the target table determined by the system through querying the metadata service of the data lake, based on whether there is a clear partition field and whether the data writing logic is based on partition replacement.

[0078] Understandably, sliced ​​tables and non-sliced ​​tables are two basic table structure types in a data lake. Sliced ​​tables are partitioned and stored according to time dimensions (such as date and month), which facilitates the management of historical data and the implementation of fine-grained repairs. Non-sliced ​​tables, on the other hand, do not perform time partitioning or only store the latest full data. When updated, they are usually updated by overwriting or replacing the primary key.

[0079] Step D12: Based on the emergency response task information, configure the data link corresponding to the data entry table storage format and determine the first link information and the second link information. It should be noted that the first link information and the second link information are sets of repair link parameters configured for non-sliced ​​tables and sliced ​​tables based on the scope of the anomaly locked by the emergency handling task and the storage form of the table. The first link information corresponds to the repair logic of non-sliced ​​tables, and the second link information corresponds to the repair logic of sliced ​​tables.

[0080] In step D13, in response to the first link information and the second link information, the control system updates or repairs the data in the data lake using a predefined data self-healing scheme.

[0081] Understandably, the data self-healing solution is a set of rules that automatically trigger repair execution, including daily repair mode, fixed window period repair mode, and manual trigger repair mode. This ensures that data repair actions can be completed automatically on time and as needed according to business requirements, guaranteeing the consistency and integrity of data within the data lake and reducing manual operation and maintenance costs.

[0082] This embodiment proposes an external data repair method for data lakes. The method acquires the scheduling and reference relationship information of external data within the data lake; determines the corresponding downstream support link information based on the scheduling and reference relationship information; monitors the corresponding data entry initiation operations and abnormal failure operations based on the downstream support link information to determine the data entry initiation operation information and abnormal failure operation information; identifies the emergency handling tasks corresponding to the affected downstream operation links based on the data entry initiation operation information and the abnormal failure operation information to determine the emergency handling task information; and controls the system to update or repair the data within the data lake based on the emergency handling task information. This solves the technical problem of how to more accurately and effectively identify the security and risks of external data. Compared with existing technologies, this application, by accurately identifying downstream support links and monitoring the data entry initiation operations and abnormal failure operations in real time, automatically identifies the affected downstream links, generates corresponding emergency handling tasks, and controls the system to update or repair the data within the data lake. This significantly improves emergency response efficiency and collaboration levels, effectively reduces operation and maintenance costs and system resource consumption, and fundamentally ensures the security and business continuity of external data use within the data lake.

[0083] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter.

[0084] In this embodiment, refer to Figure 6 , Figure 6 This is a flowchart illustrating Embodiment 2 of the external data repair method for data lakes provided in this application. Step S40 specifically includes steps S41 to S44: Step S41: Based on the data entry start operation information and the predefined source-end contract knowledge base, identify the corresponding upstream anomaly root cause and determine the upstream anomaly root cause information. The upstream anomaly root cause information includes source system person in charge information, object name information and extraction condition information. It should be noted that the upstream anomaly root cause information refers to the specific information and set of responsible persons directly related to the source failure identified by the system when an anomaly occurs in external data. This is achieved by tracing back to the initial data entry operation and combining it with the risk and responsibility data pre-stored in the source-end contract knowledge base.

[0085] Understandably, the source system owner information is used to quickly contact the responsible party at the source to communicate the processing progress, the object name information specifies the specific data table or file where the anomaly occurred, and the extraction condition information provides the specific rules and parameters for data to be added to the lake.

[0086] In a specific embodiment, based on the data entry start operation information and combined with a predefined source-end contract knowledge base, upstream anomaly root cause information can be identified. When the system receives an anomaly fault operation information, it will trace upstream to locate the data entry start operation corresponding to the anomaly link. Then, using the start operation as input, it will query the source-end contract knowledge base to obtain upstream anomaly root cause information, including source system manager information, object name information, and extraction condition information, etc., to preliminarily determine whether the anomaly was caused by the source end (e.g., source system shutdown, data not generated on time).

[0087] Step S42: Using the lineage analysis of the data warehouse and the abnormal fault operation information, the root causes of the corresponding downstream abnormalities and downstream cross-layer abnormalities are identified, and the root cause information of the downstream abnormalities is determined. The root cause information of the downstream abnormalities includes the downstream person in charge information, the downstream operation name information, the downstream operation guarantee level information, and the downstream guarantee maximum timeliness requirement information. It should be noted that the downstream anomaly root cause information is a set of all the support operation nodes that may be affected by this anomaly and their key attributes, which are identified by the system starting from the faulty operation that caused the anomaly and traversing the operation dependencies downstream through the lineage analysis capability of the data warehouse. This information is used to accurately locate the scope of the anomaly propagation and the corresponding responsible party.

[0088] Understandably, downstream responsible person information is used to clarify the specific recipients of emergency notifications and coordination, downstream job name information identifies the affected job entities, downstream job assurance level information is used to differentiate the importance of business impact to determine response priorities, and downstream assurance maximum timeliness requirement information clarifies the latest time node that the job must be completed.

[0089] In a specific embodiment, the lineage analysis capability of the data warehouse can be utilized, combined with abnormal and faulty operation information, to identify the root cause information of downstream anomalies and downstream cross-layer anomalies. That is, the system starts with the abnormal and faulty operation, traverses the lineage relationship graph downwards, and finds all downstream operations that directly and indirectly depend on the operation. During this process, the system will pay special attention to those nodes marked as support operations and extract detailed information of each affected support operation to obtain the root cause information of downstream anomalies, including the downstream person in charge, the downstream operation name, the downstream operation support level (such as L1 to L3), and the maximum timeliness requirement for downstream support (such as it must be completed before 8 o'clock).

[0090] Step S43: Based on the upstream anomaly root cause information and the downstream anomaly root cause information, prune the downstream affected operation links to determine the pruning operation link information; It should be noted that the pruning operation link information is a set of downstream operation nodes that only need to be individually notified and closely monitored, along with their responsible persons and the scope of impact, after the downstream affected operation nodes obtained from lineage analysis have been merged and simplified according to the responsible person attributes.

[0091] Understandably, pruning is an optimization mechanism for the attribution of responsibilities in the data warehouse operation chain. Its purpose is to avoid redundancy in emergency notifications caused by the same person managing multiple consecutive operations. By retaining only the cross-organizational or cross-responsibility node closest to the abnormal or faulty operation as a representative, and recording the responsible person and all affected operations under their responsibility, a concise and efficient emergency notification list is generated. This ensures that the emergency coordination process can accurately reach every responsible person, while avoiding information overload and communication interference.

[0092] In a specific implementation, considering that job managers in a data warehouse often oversee multiple jobs, the system needs to prune the jobs in the identified downstream affected job chains. This pruning process involves identifying only downstream objects that cross organizational and individual responsibility levels, avoiding redundancy by including multiple jobs managed by the same person in the emergency notification scope. For example... Figure 7 As shown, Figure 7 This diagram illustrates the cross-organizational, one-layer object identification method for external data repair in this application for data lakes. After job A malfunctions, among its downstream jobs B, C, D, and E, B and A share the same responsible person. Therefore, only the responsible person needs to be notified about the impact on C, D, and E. Pruning can yield a simplified downstream affected job chain information, i.e., pruned job chain information, which only includes job nodes requiring individual notification and their responsible persons. For example, starting from the malfunctioning job, all downstream job nodes are obtained through lineage analysis. These nodes are then traversed and grouped according to the responsible person's attribute. For multiple consecutive nodes under the same responsible person, only the node closest to the malfunctioning job is retained as a representative, and the responsible person and the scope of all affected jobs under their responsibility are recorded. For example, if responsible person A is responsible for jobs C and D (C is upstream of D), then only C is used as the notification node, and the emergency task notes indicate that D is also affected.

[0093] Step S44: Based on the pruning operation link information, the corresponding emergency handling task is identified to obtain emergency handling task information.

[0094] It should be noted that the emergency response task information is a set of executable operation instructions automatically generated by the system to ensure the continuity of downstream critical business when external data anomalies occur. It may include multiple task items, each corresponding to an affected operation link node (such as a downstream support operation), and describing in detail the emergency operations that the person in charge of the node needs to perform. For example, whether to enable risk avoidance compatibility logic, whether to ignore the anomaly, whether to manually rerun, the expected data recovery time, and the reference contractual backup plan. This transforms the abstract anomaly event into a clear division of responsibilities and operation guidelines, ensuring that the emergency response process is orderly and efficient.

[0095] In a specific embodiment, an emergency handling group interface is obtained; the emergency handling group interface is called to create an emergency handling group corresponding to the downstream link impact information and the person in charge impact information in the pruning operation link information; based on the emergency handling group and a predefined large language model, the emergency handling tasks of each person in the group are identified to obtain emergency handling task information. That is, after determining the pruning operation link information, a specific emergency handling task needs to be generated for each link node (i.e., each person in charge who needs to be notified individually and their affected operations). At this time, the operation and maintenance personnel edit the background of the current problem, the estimated processing time, and other information, and combine the contract emergency plan recorded in the upstream and downstream contract modules to deduce the emergency handling task of each downstream operation person in charge through large language model analysis, including a brief description of the abnormal background, a reference to the contract plan, specific operation instructions, and task feedback requirements. Among them, the brief description of the abnormal background is... This document explains the source, cause, and expected duration of the anomaly. The contractually agreed-upon fallback strategy is retrieved from previous agreements signed by the responsible party, reminding them to adhere to the agreement. Specific operational instructions are generated based on the anomaly type and contractually agreed-upon strategy, such as "Please immediately check if the operation's risk mitigation and compatibility logic is effective" and "If the data is still not recovered, please manually trigger a backup data rerun before 10:00 AM." Task feedback requires the responsible party to confirm completion of the task in the system to track the overall emergency progress. This allows for obtaining emergency handling task information, and emergency tasks are distributed via chatbot. Operations personnel can process information in real-time through group chat and collect information on downstream emergency handling completion status through these tasks, further mitigating data usage risks in high-risk SLA assurance links. The progress of these tasks is tracked until the anomaly is resolved, thus achieving fully automated emergency collaboration from anomaly identification, responsibility allocation, automatic group creation to task distribution.

[0096] In one feasible implementation, step S44 may include steps E11 to E13: Step E11: Obtain the emergency response group interface; It should be noted that the emergency response group interface is an application programming interface pre-integrated into the system for interacting with external instant messaging platforms. This interface encapsulates the calling protocol and authentication mechanism for basic communication functions such as creating groups, adding members, and sending messages.

[0097] It is understandable that the purpose of the emergency response group interface is to enable the system to break through the limitations of the internal platform and directly call the functions of the chat software commonly used by enterprises, thereby efficiently connecting the technical system with human resources.

[0098] Step E12: Call the emergency response group interface to create an emergency response group corresponding to the downstream link impact information and the person in charge impact information in the pruning operation link information; It should be noted that the downstream link impact information clearly specifies the name of the affected operation, the protection level, and the degree of impact, while the responsible person impact information identifies the identity and contact information of each responsible person who needs to participate in the emergency response.

[0099] It is understood that the emergency response group is a temporary collaborative space dynamically created by calling the interface. This group brings together operations and maintenance personnel and all affected job managers in the same dialogue environment, providing a centralized communication platform for information synchronization, task allocation and progress tracking, thereby breaking down information silos and achieving efficient cross-organizational collaboration.

[0100] Step E13: Based on the emergency response group and the predefined large language model, the emergency response tasks of each person in the group will be identified to obtain emergency response task information.

[0101] It should be noted that the large language model is a deep learning model trained on massive amounts of data. It has powerful natural language understanding and generation capabilities and is used to comprehensively analyze abnormal backgrounds, group contexts, and fallback solutions in the pre-signed knowledge base to infer and generate personalized operation guidelines for each responsible person.

[0102] Understandably, each person's emergency response task in the group is a set of personalized operation instructions generated by the system for each downstream person in charge in the group, based on the specific characteristics of the abnormal event and the pre-agreed backup strategy. This instruction set may include a brief description of the abnormal background, the risk avoidance operation steps to be performed, the expected completion time limit, and references to the contracted plan. This is used to transform ambiguous abnormal events into clear and executable division of responsibilities, ensuring the orderly progress of the emergency response process.

[0103] This embodiment proposes an external data repair method for data lakes. Based on the data inflow initiation operation information and a predefined source-end contract knowledge base, the method identifies the corresponding upstream anomaly root causes, determining upstream anomaly root cause information, including source system manager information, object name information, and extraction condition information. Using data warehouse lineage analysis and the anomaly / failure operation information, the method identifies the corresponding downstream anomalies and downstream cross-layer anomaly root causes, determining downstream anomaly root cause information, including downstream manager information, downstream operation name information, downstream operation support level information, and downstream support maximum timeliness requirement information. Based on the upstream and downstream anomaly root causes, the method prunes the affected downstream operation links, determining pruning operation link information. Based on the pruning operation link information, the method identifies the corresponding emergency handling tasks, obtaining emergency handling task information. This application addresses the technical challenge of more accurately and effectively identifying the security and risks of external data. Compared to existing technologies, it identifies the root causes of anomalies and their cross-layer impacts in the upstream and downstream processes, pinpoints the responsible parties and operational attributes in the anomaly propagation chain, and prunes the affected downstream operational links. By consolidating the responsible parties, the scope of notification is streamlined, avoiding redundant communication. Based on the pruned link information, personalized emergency response tasks are automatically identified and generated. This achieves fully automated emergency collaboration from anomaly root cause location, accurate scope of impact definition, optimized notification recipients to intelligent task distribution, significantly improving response efficiency, reducing manual coordination costs, and ensuring the continuity of critical business links and the security of data usage.

[0104] This application also provides an external data repair device for data lakes, please refer to... Figure 8 The external data repair device for data lakes includes: The upstream and downstream contract signing module 10 is used to obtain scheduling information and reference relationship information of external data in the data lake; The upstream and downstream contract signing module 10 is also used to determine the corresponding downstream guarantee link information based on the scheduling information and the reference relationship information; The link analysis and collaborative processing module 20 is used to monitor the corresponding data entry start operation and abnormal fault operation based on the downstream support link information, and determine the data entry start operation information and abnormal fault operation information. The link analysis and collaborative processing module 20 is also used to identify emergency handling tasks corresponding to downstream affected operation links based on the data entry into the lake initiation operation information and the abnormal fault operation information, and to determine emergency handling task information. The data self-healing module 30 is used to update or repair the data in the data lake based on the emergency response task information control system.

[0105] The upstream and downstream contract signing module 10 is also used to identify the operation increment of the corresponding downstream support link based on the scheduling information and the reference relationship information, and determine the downstream support operation increment information. Based on the incremental information of downstream support operations and the predefined risk knowledge base, the corresponding source risks are matched to determine the source risk information; Based on the aforementioned source-end risk information, the corresponding contract information and compatibility modification information are determined; The downstream support operation incremental information, the source risk information, the contract information, and the compatibility modification information are used to obtain the downstream support link information.

[0106] The upstream and downstream contract module 10 is also used to identify the link change status of the scheduling information and the reference relationship information within a predefined time by comparing the difference of snapshot data, and to determine the link change information, which includes added dependency information and deleted dependency information. Based on the timeliness and quality tags of the link change information in the data lineage, the corresponding new downstream support operations are identified, and the information of the new downstream support operations is determined.

[0107] The upstream and downstream contract signing module 10 is also used to obtain the source end risk survey results; The predefined risk knowledge base is updated based on the source-end risk survey results to determine the target risk knowledge base; Based on the matching of the newly added downstream support operation information with the source risk records in the target risk knowledge base, the source risk matching result is determined; Based on the source-end risk matching results, corresponding source-end risk information is generated.

[0108] The link analysis and collaborative processing module 20 is also used to identify the corresponding upstream anomaly root cause based on the data entry start operation information and the predefined source end contract knowledge base, and determine the upstream anomaly root cause information, the upstream anomaly root cause information including source system person in charge information, object name information and extraction condition information; By using the lineage analysis of the data warehouse and the abnormal fault operation information, the root causes of the corresponding downstream abnormalities and downstream cross-layer abnormalities are identified, and the root cause information of the downstream abnormalities is determined. The root cause information of the downstream abnormalities includes the downstream person in charge information, the downstream operation name information, the downstream operation guarantee level information, and the downstream guarantee maximum timeliness requirement information. Based on the upstream anomaly root cause information and the downstream anomaly root cause information, the affected downstream operation links are pruned to determine the pruning operation link information; Based on the pruning operation link information, the corresponding emergency response tasks are identified to obtain emergency response task information.

[0109] The link analysis and collaborative processing module 20 is also used to obtain the emergency response group interface; Call the emergency response group interface to create an emergency response group corresponding to the downstream link impact information and the person in charge impact information in the pruning operation link information; Based on the emergency response group and the predefined large language model, the emergency response tasks of each person in the group will be identified to obtain emergency response task information.

[0110] The link analysis and collaborative processing module 20 is also used to obtain the data lake table storage format, which includes sliced ​​tables and non-sliced ​​tables. Based on the emergency response task information, the data links corresponding to the data entry table storage format are configured, and the first link information and the second link information are determined. In response to the first link information and the second link information, the control system updates or repairs the data in the data lake using a predefined data self-healing scheme.

[0111] The external data repair apparatus for data lakes provided in this application, employing the external data repair method for data lakes described in the above embodiments, can solve the technical problem of how to more accurately and effectively identify the security and risks of external data. Compared with the prior art, the beneficial effects of the external data repair apparatus for data lakes provided in this application are the same as those of the external data repair method for data lakes provided in the above embodiments, and other technical features in the external data repair apparatus for data lakes are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0112] This application provides an external data repair device for a data lake, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the external data repair method for a data lake as described in Embodiment 1 above.

[0113] The following is for reference. Figure 9This document illustrates a structural schematic diagram of an external data repair device suitable for implementing embodiments of this application for data lakes. The external data repair device for data lakes in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The external data repair device for data lakes shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0114] like Figure 9 As shown, the external data repair device for the data lake may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the external data repair device for the data lake. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows external data repair equipment for data lakes to exchange data wirelessly or via wired communication with other devices. Although the figure shows external data repair equipment for data lakes with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0115] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0116] The external data repair device for data lakes provided in this application, employing the external data repair method for data lakes described in the above embodiments, can solve the technical problem of how to more accurately and effectively identify the security and risks of external data. Compared with the prior art, the beneficial effects of the external data repair device for data lakes provided in this application are the same as those of the external data repair method for data lakes provided in the above embodiments, and other technical features of this external data repair device for data lakes are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0117] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0119] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the external data repair method for a data lake as described in the above embodiments.

[0120] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0121] The aforementioned computer-readable storage medium may be included in an external data repair device for a data lake; or it may exist independently and not be assembled into an external data repair device for a data lake.

[0122] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by an external data repair device targeting the data lake, the external data repair device causes the following: it acquires scheduling information and reference relationship information of external data in the data lake; determines the corresponding downstream support link information based on the scheduling information and the reference relationship information; monitors the corresponding data entry start operation and abnormal failure operation based on the downstream support link information, and determines the data entry start operation information and abnormal failure operation information; identifies the emergency handling tasks corresponding to the downstream affected operation links based on the data entry start operation information and the abnormal failure operation information, and determines the emergency handling task information; and controls the system to update or repair the data in the data lake based on the emergency handling task information.

[0123] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0126] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned external data repair method for data lakes. This addresses the technical problem of how to more accurately and effectively identify the security and risks of external data. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the external data repair method for data lakes provided in the above embodiments, and will not be elaborated upon here.

[0127] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An external data repair method for data lakes, characterized in that, The method includes: Obtain scheduling and reference relationship information of external data within the data lake; The corresponding downstream support link information is determined based on the scheduling information and the reference relationship information; Based on the downstream support link information, the corresponding data entry start operation and abnormal failure operation are monitored to determine the data entry start operation information and abnormal failure operation information. Based on the data entry start operation information and the abnormal failure operation information, the emergency handling tasks corresponding to the downstream affected operation links are identified, and the emergency handling task information is determined. The emergency response task information control system updates or repairs the data in the data lake.

2. The method as described in claim 1, characterized in that, The step of determining the corresponding downstream support link information based on the scheduling information and the reference relationship information includes: Based on the scheduling information and the reference relationship information, the corresponding downstream support link operation increment is identified, and the downstream support operation increment information is determined; Based on the incremental information of downstream support operations and the predefined risk knowledge base, the corresponding source risks are matched to determine the source risk information; Based on the aforementioned source-end risk information, the corresponding contract information and compatibility modification information are determined; The downstream support operation incremental information, the source risk information, the contract information, and the compatibility modification information are used to obtain the downstream support link information.

3. The method as described in claim 2, characterized in that, The step of identifying the corresponding downstream support link's job increment based on the scheduling information and the reference relationship information, and determining the downstream support job increment information, includes: The link change status of the scheduling information and the reference relationship information within a predefined time period is identified by comparing the difference of snapshot data, and the link change information is determined, including the addition of dependency information and the deletion of dependency information. Based on the timeliness and quality tags of the link change information in the data lineage, the corresponding new downstream support operations are identified, and the information of the new downstream support operations is determined.

4. The method as described in claim 2, characterized in that, The step of matching the corresponding source-end risks based on the downstream support operation incremental information and the predefined risk knowledge base to determine the source-end risk information includes: Obtain the results of the source-end risk investigation; The predefined risk knowledge base is updated based on the source-end risk survey results to determine the target risk knowledge base; Based on the matching of the newly added downstream support operation information with the source risk records in the target risk knowledge base, the source risk matching result is determined; Based on the source-end risk matching results, corresponding source-end risk information is generated.

5. The method as described in claim 1, characterized in that, The step of identifying emergency response tasks corresponding to downstream affected operation links based on the data entry start operation information and the abnormal failure operation information, and determining the emergency response task information includes: Based on the data entry into the lake starting operation information and the predefined source-end contract knowledge base, the corresponding upstream anomaly root causes are identified, and the upstream anomaly root cause information is determined. The upstream anomaly root cause information includes source system person in charge information, object name information and extraction condition information. By using the lineage analysis of the data warehouse and the abnormal fault operation information, the root causes of the corresponding downstream abnormalities and downstream cross-layer abnormalities are identified, and the root cause information of the downstream abnormalities is determined. The root cause information of the downstream abnormalities includes the downstream person in charge information, the downstream operation name information, the downstream operation guarantee level information, and the downstream guarantee maximum timeliness requirement information. Based on the upstream anomaly root cause information and the downstream anomaly root cause information, the affected downstream operation links are pruned to determine the pruning operation link information; Based on the pruning operation link information, the corresponding emergency response tasks are identified to obtain emergency response task information.

6. The method as described in claim 5, characterized in that, The step of identifying the corresponding emergency response task based on the pruning operation link information and obtaining the emergency response task information includes: Obtain the emergency response group interface; Call the emergency response group interface to create an emergency response group corresponding to the downstream link impact information and the person in charge impact information in the pruning operation link information; Based on the emergency response group and the predefined large language model, the emergency response tasks of each person in the group will be identified to obtain emergency response task information.

7. The method as described in claim 1, characterized in that, The steps of updating or repairing the data in the data lake based on the emergency response task information control system include: Obtain the data lake table storage format, which includes sliced ​​tables and non-sliced ​​tables; Based on the emergency response task information, the data links corresponding to the data entry table storage format are configured, and the first link information and the second link information are determined. In response to the first link information and the second link information, the control system updates or repairs the data in the data lake using a predefined data self-healing scheme.

8. An external data repair device for data lakes, characterized in that, The device includes: The upstream and downstream contract signing module is used to obtain scheduling information and reference relationship information of external data in the data lake; The upstream and downstream contract signing module is also used to determine the corresponding downstream guarantee link information based on the scheduling information and the reference relationship information; The link analysis and collaborative processing module is used to monitor the corresponding data entry start operation and abnormal failure operation based on the downstream support link information, and determine the data entry start operation information and abnormal failure operation information. The link analysis and collaborative processing module is also used to identify emergency handling tasks corresponding to downstream affected operation links based on the data entry start operation information and the abnormal fault operation information, and to determine emergency handling task information. The data self-healing module is used to update or repair the data in the data lake based on the emergency response task information control system.

9. An external data repair device for data lakes, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the external data repair method for a data lake as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the external data repair method for a data lake as described in any one of claims 1 to 7.

Images