All-link influence analysis method and early warning system for data in civil aircraft manufacturing industry data

Abstract

The invention discloses a full-link influence analysis method and early warning system for data in civil aircraft manufacturing industry data, and the system comprises a data source module which comprises a data service composed of a data interface, a data report and a data index, a database service, ETL operation, and servers and server monitoring software which all services belong to, a data collection module, a data processing module, a data processing module and a data processing module. The method is used for collecting metadata information from a business system, a database and an ETL tool and comprises data service definition, table structure definition and job task definition. Full-link connection and unified modeling of a business layer, a logic layer and a physical layer are achieved, the problem of data link splitting is solved, the system supports second-level influence range analysis and bidirectional visual tracing based on a graph database and a search engine, the troubleshooting efficiency is greatly improved, multi-dimensional intelligent early warning is achieved through dynamic threshold and cross-layer association, and the system is suitable for large-scale popularization and application. Abnormal influence is accurately positioned, the system provides data support for resource optimization and capacity planning, and objectivity and foresight of operation and maintenance decisions are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of data analysis technology, specifically a method and early warning system for analyzing the full-link impact of data in the civil aircraft manufacturing industry data platform. Background Technology

[0002] In today's enterprise digital transformation, data has become a core asset, and data analytics applications are deeply integrated into all business lines. To ensure the reliability and maintainability of the data chain, data lineage analysis and system monitoring tools have become crucial. Currently, the industry typically employs two main types of technologies: one is data lineage analysis tools, used to trace dependencies between data tables; the other is system monitoring tools, used to monitor hardware resource status and task execution.

[0003] However, existing technical solutions have significant drawbacks. First, data lineage analysis is often limited to the physical or logical layers (such as table-level dependencies), failing to connect data services at the business layer, processing tasks at the logical layer, and underlying resources at the physical layer. This results in gaps in the data chain, making end-to-end tracing from business requirements to underlying operations difficult. Second, when data anomalies or system changes occur, there is a lack of effective cross-layer impact analysis methods. Technical personnel must manually investigate layer by layer, which is inefficient and prone to missing key impact areas. Third, existing monitoring and early warning mechanisms are relatively simplistic, typically focusing on task status or hardware metrics in isolation, failing to correlate business impacts with resource anomalies. This leads to inaccurate early warning information that cannot directly guide operational decisions. Furthermore, traditional tools lack visualization and retrieval capabilities, typically providing only static lineage diagrams and lacking interactive, multi-perspective end-to-end displays and flexible search functions, making it difficult to support rapid root cause analysis and link understanding.

[0004] In summary, existing technologies are insufficient to meet the operational and maintenance needs of complex enterprise data environments for knowing, analyzing, and controlling the entire data chain. There is an urgent need for a solution that can connect business, logical, and physical layers to achieve correlation analysis, intelligent early warning, and visualization. Summary of the Invention

[0005] The technical solution adopted in this invention is as follows: a data middleware system for the civil aircraft manufacturing industry with full-link impact analysis and early warning, comprising:

[0006] The data source module includes data services consisting of data interfaces, data reports, and data metrics, database services, ETL jobs, and the servers and server monitoring software to which all services belong.

[0007] The data acquisition module is used to collect metadata information from business systems, databases, and ETL tools, including data service definitions, table structures, and job task definitions.

[0008] The data storage module is used to divide the data into business layer, logical layer and physical layer, including graph database and Elasticsearch search engine. It constructs the collected metadata into a bidirectional network containing four types of nodes and their relationships, and stores it in graph database. At the same time, it synchronizes the graph database data with the Elasticsearch search engine in real time.

[0009] The functional modules are used for impact range analysis, real-time monitoring and early warning, end-to-end visualization and knowledge base management based on graph databases.

[0010] The data source module and the data acquisition module are connected by a signal, the data acquisition module and the data storage module are connected by a signal, and the data storage module and the functional module are connected by a signal.

[0011] Furthermore, the four types of nodes constructed in the data storage module include:

[0012] The first type of node (N1) represents data services.

[0013] The second type of node (N2) represents a data table.

[0014] The third type of node (N3) represents the job task.

[0015] The fourth type of node (N4) represents the hardware and software dependency environment.

[0016] Furthermore, the data storage module uses the Neo4j graph database to store nodes and relationships.

[0017] Furthermore, the functional modules specifically include:

[0018] The monitoring and early warning engine is used to monitor the execution status of job tasks and the status of hardware resources in real time, trigger an early warning mechanism based on preset thresholds, and push messages to mobile devices.

[0019] The visualization submodule is used to provide a search interface through Elasticsearch and combine it with front-end technologies to achieve full-link visualization.

[0020] The knowledge base submodule is used to store common problem patterns, record historical problem solutions, and support case retrieval and recommendation.

[0021] Furthermore, the monitoring and early warning engine sets dynamic thresholds based on historical data of job execution. When the actual execution exceeds the threshold, it performs impact range analysis based on the graph database and issues an early warning notification.

[0022] The method for analyzing the full-link impact of data from a data platform in the civil aircraft manufacturing industry includes the following steps:

[0023] S1: Conduct multi-layered reverse influencing factor analysis, and carry out data flow analysis from three levels: business layer, logic layer and physical layer.

[0024] S2: Extract and establish relationships of business layer elements, determine the data services of the business layer as the first type of node (N1) and the related tables as the second type of node (N2), and record the association relationship between N1 and N2.

[0025] S3: Extract and establish relationships of logical layer elements, determine the tasks of the logical layer as the third type of node (N3) and the related tables as the second type of node (N2), and record the association between the third type of node (N3) and the second type of node (N2).

[0026] S4: Physical layer relies on environmental information recording. It records the hardware and software environment information where data services, tables, and job tasks are located as the fourth type of node (N4), and records the association relationship between the fourth type of node (N4) and other nodes.

[0027] S5: Graph database storage, based on Neo4j graph database, stores the above four types of nodes (N1, N2, N3, N4) and the relationships between nodes, and constructs a complete full-link relationship graph.

[0028] S6: Work task execution early warning. Thresholds are set based on historical data of work task execution. When the actual execution exceeds the threshold, the impact range analysis is carried out based on the graph database, and an early warning notification is issued to the relevant responsible persons.

[0029] S7: Software and hardware status early warning. Set performance thresholds based on software and hardware execution status. When the actual indicators are detected to exceed the thresholds, conduct impact range analysis based on graph database and issue early warning notifications to relevant personnel.

[0030] S8: Full-link visualization, providing search functionality through Elasticsearch, visually displaying the entire data link, supporting forward data flow display and reverse problem node location.

[0031] S9: Knowledge base establishment, storage of common problem patterns, recording of historical problem solutions, and support for case retrieval and recommendation.

[0032] Furthermore, in step S5, the nodes and relationships are stored in the Neo4j graph database, and the relationship types include "dependency", "use", "run on", and "call".

[0033] Furthermore, in step S6, the threshold is dynamically set based on statistical indicators calculated from historical execution data.

[0034] Furthermore, in step S8, the second type of nodes (N2) and the first type of nodes (N1) are displayed in the direction of the forward data flow, and the bidirectional influence range is determined and analyzed for the third type of nodes (N3) and the fourth type of nodes (N4).

[0035] Furthermore, the knowledge base in step S9 is updated after each warning processing.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] (1) In this invention, the entire link between the business layer, the logic layer and the physical layer is realized. Through multi-layer reverse analysis and unified node modeling, data services, data tables, job tasks and hardware and software environments are associated into a complete relationship graph, which completely solves the problem of data link fragmentation and difficulty in cross-layer tracing in traditional solutions, enabling technical personnel to grasp the complete path of data from business requirements to physical implementation from a global perspective.

[0038] (2) This invention significantly improves the efficiency and accuracy of impact range analysis. Based on the storage and traversal capabilities of the Neo4j graph database, the system can quickly perform bidirectional relationship queries and impact range positioning when data is abnormal or changed, achieving a response time in seconds. This replaces the inefficient traditional manual layer-by-layer investigation mode, greatly shortening the time for fault investigation and impact assessment.

[0039] (3) In this invention, a multi-dimensional and interconnected intelligent early warning mechanism is established. The system not only monitors the task execution status and hardware resource indicators, but also realizes cross-layer linkage early warning through dynamic threshold settings and graph correlation analysis. For example, hardware failures are directly associated with the affected upper-layer data services and reports, thereby providing more accurate and actionable early warning information.

[0040] (4) This invention provides flexible and efficient visualization and search capabilities. Combining Elasticsearch's index retrieval and front-end visualization technology, the system supports forward display and reverse problem tracing according to data flow, while also providing a quick keyword search function, enabling the intuitive and interactive exploration and analysis of complex data lineages and dependencies.

[0041] (5) In this invention, data support is provided for system resource optimization and capacity planning. Through the association between environmental nodes and upper-layer tasks and data tables, the system can identify physical resource nodes that carry critical business links, thereby providing an objective basis for server resource assurance and expansion priority judgment, and realizing data-driven operation and maintenance decisions. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall system architecture of the present invention.

[0043] In the diagram: 1. Data source module; 2. Data acquisition module; 3. Data storage module; 4. Functional module. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] Example 1

[0046] Reference Figure 1 :

[0047] A data middle platform system for the civil aircraft manufacturing industry, encompassing end-to-end data impact analysis and early warning, including:

[0048] Data source module 1 includes data services consisting of data interfaces, data reports, and data metrics; database services; ETL jobs; and servers and server monitoring software for all services. Data acquisition module 2 is used to collect metadata information from business systems, databases, and ETL tools, including data service definitions, table structures, and job task definitions. Data storage module 3 is used to construct a bidirectional network containing four types of nodes and their relationships, based on the business layer, logical layer, and physical layer, and store it in a graph database. Simultaneously, it synchronizes the graph database data with the Elasticsearch search engine in real time. Functional module 4 is used for impact scope analysis, real-time monitoring and early warning, end-to-end visualization, and knowledge base management based on the graph database. Data source module 1 and data acquisition module 2 are connected by signals; data acquisition module 2 and data storage module 3 are connected by signals; and data storage module 3 and functional module 4 are connected by signals. The four types of nodes constructed in Block 3 include: the first type of node (N1), representing data services; the second type of node (N2), representing data tables; the third type of node (N3), representing job tasks; and the fourth type of node (N4), representing hardware and software dependencies. The data storage module 3 uses the Neo4j graph database to store nodes and relationships. The functional module 4 specifically includes: a monitoring and early warning engine, used to monitor the execution status of job tasks and hardware resource status in real time, trigger an early warning mechanism based on preset thresholds, and push messages to mobile devices; a visualization display submodule, used to provide a search interface through Elasticsearch and combine front-end technology to achieve full-link visualization display; and a knowledge base submodule, used to store common problem patterns, record historical problem solutions, and support case retrieval and recommendation. The monitoring and early warning engine sets dynamic thresholds based on historical data of job task execution. When the actual execution status is detected to exceed the threshold, it conducts impact range analysis based on the graph database and issues an early warning notification.

[0049] The method for analyzing the full-link impact of data from a data platform in the civil aircraft manufacturing industry includes the following steps:

[0050] Step S1: Multi-layered reverse influencing factor analysis. First, clarify the scope of analysis: business layer (including data services such as APIs and reports), logic layer (including data processing logic such as ETL and stored procedures), and physical layer (including physical resources such as database tables, views, and servers). Conduct reverse data flow analysis from these three layers.

[0051] Step S2: Extraction and Relationship Establishment of Business Layer Elements. Identify all data services and define them as first-type nodes (N1); parse the implementation of each data service, determine its dependent data tables (physical tables, views, etc.), and define them as second-type nodes (N2); record the "dependency" and other association relationships between N1 and N2.

[0052] Step S3: Extraction and Relationship Establishment of Logical Layer Elements. Scan the ETL tool and scheduling system to identify all job tasks and define them as third-type nodes (N3); parse the input and output of each job and any custom SQL that may be included to determine the data tables involved (N2); record the "use" or "production" relationships between N3 and N2.

[0053] Step S4: Physical layer dependent environment information recording. Collect hardware information such as servers and storage, and define them as fourth type nodes (N4); record which software and hardware environment (N4) the data service (N1), table (N2), and job task (N3) are specifically deployed or running on, and establish the "running on" relationship.

[0054] Step S5: Graph Database Storage. Store the four types of nodes (N1, N2, N3, N4) and their relationships (such as "dependency," "use," "run on," and "call") in the Neo4j graph database to construct a complete end-to-end relationship graph. Simultaneously, synchronize the key attribute information of the nodes in this graph to the Elasticsearch search engine in real time.

[0055] Step S6: Job Execution Early Warning. Collect historical execution data of job tasks (such as duration), calculate statistical indicators such as average and standard deviation, and dynamically set thresholds based on these (e.g., average plus 3 times the standard deviation). Monitor the real-time execution of tasks. When an indicator is detected to exceed the threshold, immediately start from the abnormal task node (N3), traverse its downstream dependent data tables (N2) and the ultimately affected data services (N1) in the graph, determine the scope of impact, and issue an early warning notification to the relevant responsible persons.

[0056] Step S7: Hardware and Software Status Early Warning. Monitor server hardware metrics such as CPU, memory, and disk, and set absolute thresholds (e.g., CPU > 90%) and relative change thresholds. When abnormal metrics are detected, identify all job tasks (N3), data tables (N2), and data services (N1) "running" in the faulty environment (N4) based on the graph, conduct impact range analysis, and issue an early warning.

[0057] Step S8: Full-link visualization. Elasticsearch provides a keyword search interface (e.g., searching by table name or service name). For the retrieved nodes, their related subgraphs are obtained from the Neo4j graph for visualization: for data tables (N2) and data services (N1), the forward data flow direction can be displayed; for job tasks (N3) and the environment (N4), a bidirectional impact range analysis view can be displayed, supporting reverse location of problem nodes.

[0058] Step S9: Knowledge Base Establishment. The system automatically archives and stores the root cause, scope of impact, and solution for each alert event. Once a case library is formed, operations and maintenance personnel can retrieve data based on problem characteristics. The knowledge base is updated after each alert is processed.

[0059] Through the above implementation, this system has realized a knowledge loop from multi-source data access, full-link map construction, intelligent early warning to visualization analysis, providing end-to-end observable and proactive operation and maintenance capabilities for the data platform of the civil aircraft manufacturing industry.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data mid-platform data end-to-end impact analysis and early warning system for the civil aircraft manufacturing industry, characterized in that: include: The data source module (1) includes data services consisting of data interfaces, data reports, and data indicators, database services, ETL jobs, and the servers and server monitoring software to which all services belong. The data acquisition module (2) is used to collect metadata information from business systems, databases and ETL tools, including data service definitions, table structures and job task definitions; The data storage module (3) is used to construct the collected metadata into a bidirectional network containing four types of nodes and their relationships according to the division of business layer, logical layer and physical layer, including graph database and Elasticsearch search engine, and store it in graph database. At the same time, the graph database data is synchronized with the Elasticsearch search engine in real time. Functional module (4) is used for impact range analysis, real-time monitoring and early warning, full-link visualization and knowledge base management based on graph database; The data source module (1) is connected to the data acquisition module (2) by signal, the data acquisition module (2) is connected to the data storage module (3) by signal, and the data storage module (3) is connected to the functional module (4) by signal.

2. The data full-link impact analysis and early warning system for the civil aircraft manufacturing data platform according to claim 1, characterized in that, The four types of nodes constructed in the data storage module (3) include: The first type of node represents data services; The second type of node represents a data table; The third type of node represents the task / job; The fourth type of node represents the software and hardware dependency environment.

3. The data full-link impact analysis and early warning system for the civil aircraft manufacturing data platform according to claim 1, characterized in that, The data storage module (3) uses the Neo4j graph database to store nodes and relationships.

4. The data full-link impact analysis and early warning system for the civil aircraft manufacturing data platform according to claim 1, characterized in that, The functional module (4) specifically includes: The monitoring and early warning engine is used to monitor the execution status of job tasks and the status of hardware resources in real time, trigger an early warning mechanism based on preset thresholds, and push messages to mobile devices; The visualization submodule is used to provide a search interface through Elasticsearch and combine it with front-end technologies to achieve full-link visualization. The knowledge base submodule is used to store common problem patterns, record historical problem solutions, and support case retrieval and recommendation.

5. The data full-link impact analysis and early warning system for the civil aircraft manufacturing data platform according to claim 4, characterized in that, The monitoring and early warning engine sets dynamic thresholds based on historical data of job execution. When the actual execution exceeds the threshold, it performs impact range analysis based on the graph database and issues an early warning notification.

6. A method for analyzing the full-link impact of data from a data platform in the civil aircraft manufacturing industry, characterized in that... Includes the following steps: S1: Conduct multi-layered reverse influencing factor analysis, and carry out data flow analysis from three levels: business layer, logic layer and physical layer; S2: Extracting and establishing relationships from business layer elements, identifying data services in the business layer as the first type of node and related tables as the second type of node, and recording the association between the first type of node and the second type of node; S3: Extract and establish relationships of logical layer elements, determine the tasks of the logical layer as the third type of nodes and the related tables as the second type of nodes, and record the association between the third type of nodes and the second type of nodes; S4: Physical layer relies on environmental information recording, which records the software and hardware environment information where data services, tables, and job tasks are located as the fourth type of node, and records the association relationship between the fourth type of node and other nodes; S5: Graph database storage, based on Neo4j graph database, stores the above four types of nodes and the relationships between nodes, and constructs a complete full-link relationship graph; S6: Work task execution early warning. Thresholds are set based on historical data of work task execution. When the actual execution exceeds the threshold, the impact range analysis is carried out based on the graph database, and an early warning notification is issued to the relevant responsible persons. S7: Software and hardware status early warning. Set performance thresholds based on the execution status of software and hardware. When the actual indicators are detected to exceed the thresholds, conduct impact range analysis based on graph database and issue early warning notifications to relevant personnel. S8: Full-link visualization, providing search functionality through Elasticsearch, visually displaying the entire data link, supporting forward data flow display and reverse problem node location; S9: Knowledge base establishment, storage of common problem patterns, recording of historical problem solutions, and support for case retrieval and recommendation.

7. The method for analyzing the full-link impact of data in the civil aircraft manufacturing industry data platform according to claim 6, characterized in that, In step S5, nodes and relationships are stored in the Neo4j graph database, and the relationship types include "dependency", "use", "run on", and "call".

8. The method for analyzing the full-link impact of data in the civil aircraft manufacturing industry data platform according to claim 6, characterized in that, In step S6, the threshold is dynamically set based on statistical indicators calculated from historical execution data.

9. The method for analyzing the full-link impact of data in the civil aircraft manufacturing industry data platform according to claim 6, characterized in that, In step S8, the second type of nodes and the first type of nodes are displayed in the direction of the forward data flow, and the bidirectional influence range of the third type of nodes and the fourth type of nodes is determined and analyzed.

10. The method for analyzing the full-link impact of data in the civil aircraft manufacturing industry data platform according to claim 6, characterized in that, The knowledge base in step S9 is updated after each warning is processed.

Images