Multi-source heterogeneous data dynamic braiding and unified service generation method and device

Abstract

The invention relates to a multi-source heterogeneous data dynamic braiding and unified service generation method and device, and belongs to the technical field of big data processing. The method comprises a dynamic data weaving method and a unified service generation method. The dynamic data knitting method comprises the steps of multi-source heterogeneous data source butt joint, dynamic data knitting configuration, multi-source data dynamic knitting execution, view authorization and authority control and system performance monitoring. The unified service generation method is based on a data view and comprises the steps of data service configuration, data service generation and data service management and monitoring. According to the method, the suitability of special data sources in the military field is effectively improved, the data service construction and calling access cost is effectively reduced, and the data service compatibility and usability are improved.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, specifically relating to a method and apparatus for dynamic weaving and unified service generation of multi-source heterogeneous data. Background Technology

[0002] Military decision-making processes require the integration of multi-source heterogeneous data, including entity data such as personnel and materials stored in relational databases, satellite imagery stored in object databases, and command data from graph databases. Existing technologies have the following shortcomings: Data source compatibility limitations: There is a lack of connector systems for dedicated data sources in the military field (such as radar data, command stream data, command data, etc.); Rigid weaving strategy: Unable to dynamically adjust sharding and transaction strategies according to military business scenarios (real-time rapid scheduling / data situation review and analysis), unable to quickly support data query, data analysis and processing and unified data service generation in high real-time scenarios in the military field; Fragmented service generation: Various business libraries cannot develop data interfaces based on a unified view, resulting in poor standardization and compatibility of data services; Insufficient monitoring targeting: Lack of the ability to quickly locate performance issues in high real-time scenarios in the military field.

[0003] Therefore, there is an urgent need for a solution that can dynamically integrate and generate unified services from multi-source heterogeneous data in the military field. Summary of the Invention

[0004] (a) Technical problems to be solved The technical problem to be solved by the present invention is how to provide a method and apparatus for dynamic weaving and unified service generation of multi-source heterogeneous data, so as to solve the problems of low integration efficiency of multi-source heterogeneous data, inconsistent service generation, and insufficient performance monitoring in the military field, which cannot meet the requirements of near real-time and high-performance scenarios.

[0005] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a method for dynamic weaving and unified service generation of multi-source heterogeneous data. This method includes a dynamic data weaving method and a unified service generation method; wherein... Dynamic data weaving methods include: S1: Multi-source heterogeneous data source integration, establishing connections with military relational databases, object storage, and graph databases based on the connector system, synchronizing the data source directory through the data source interface, and using message push to update adaptation information in real time; S2: Dynamic data weaving configuration, based on data virtualization and federated query capabilities, configures virtual view creation rules, data weaving strategies and routing rules; S3: Multi-source data dynamic weaving execution, starts the data weaving engine to access cross-source data through federated queries, supports database dialect adaptation, extracts and merges data after parsing and optimizing query statements and collects metadata information, realizes dynamic view updates through metadata collection, and generates a unified military domain data view. S4: View authorization and permission control, based on a three-tier role system of super administrator - topic administrator - ordinary user, create authorization policies and assign view permissions; S5: System performance monitoring, collects performance data, optimizes storage through sharding, compression, and indexing, and generates diagnostic reports through the analysis engine; The unified service generation method is based on a data view and includes: T1: Data service configuration, configuring service generation rules, interface protocols and encryption rules; T2: Data service generation, encrypts sensitive data and encapsulates it into a standardized service, supporting API interface and JDBC access; T3: Data service management and monitoring, enabling full lifecycle management of services, collecting operational performance data and tracing SQL links, linking with system monitoring to perform service performance diagnosis and generate monitoring reports.

[0006] The present invention also provides a device for dynamic weaving and unified service generation of multi-source heterogeneous data, the device comprising: Data source connection module: Based on the connector system, it connects to multiple data sources, synchronizes the catalog, and updates adaptation information; Data weaving configuration module: Configure virtual views, weaving strategies, operator orchestration, routing, and encryption rules; Data weaving execution module: Enables cross-source data weaving and dynamic view updates through federated queries and dual engines; View authorization module: Manages view permissions based on a three-tier role system; Performance monitoring module: Collects and stores performance data and generates diagnostic reports; Encryption module: encrypts and decrypts sensitive data according to rules; Service configuration module: Configures unified data service generation and data encryption rules; Service generation module: encapsulates the encrypted data view into a standardized data service; Service Management Module: Enables full lifecycle management of data services; Service monitoring module: Collects performance data and performs diagnostics accordingly.

[0007] (III) Beneficial Effects This invention proposes a method and apparatus for dynamic weaving and unified service generation of multi-source heterogeneous data. Through technologies such as dual-engine (Shardingsphere + Trino) dynamic switching, data virtualization, federated queries, data sharding, distributed transactions, and read / write separation, it separates the underlying heterogeneous data sources from the military domain and cross-organizational applications. An abstraction layer is created to generate a unified data virtual view and standardized data service APIs, enabling applications to access data through a unified data view and unified API. This effectively improves the adaptability of military-specific data sources. The method and apparatus can be adapted to specific business scenarios in the military domain (real-time rapid scheduling / (Situation review and analysis) Dynamically adjust the weaving strategy to solve the problems of cross-source data query and multi-dimensional data analysis and processing in high real-time scenarios in the military field while taking into account system performance; furthermore, develop API interfaces based on the formed unified view to standardize data service generation, effectively reduce the cost of data service construction and access, and improve data service compatibility and ease of use; finally, this invention integrates the functions of Metrics, Tracing, and Logging into the Agent through plugins, and performs performance monitoring for specific scenarios such as data source connection, dynamic routing, and cross-source query, realizing high-performance monitoring and diagnosis and rapid problem location capabilities in near real-time scenarios.

[0008] The beneficial effects of this invention are: High adaptability: Optimize connection strategies to form a connector system for heterogeneous data sources and business decision-making scenarios in the military field; Efficiency Improvement: The dual-engine architecture balances performance and query analysis capabilities, dynamically switching based on the processing scenario to improve query response speed and ensure multi-dimensional analysis capabilities across data sources and high-concurrency real-time query efficiency. High security: Access control is implemented based on a three-level role system. Access control and encryption mechanisms work together to ensure that data is accessed according to the rights and prevent the leakage of sensitive data. High ease of use: Standardized data service interface generation forms a unified service, reducing the cost of building and accessing data services and improving the ease of use of data services.

[0009] Highly targeted: Performance monitoring uses a Java agent to collect performance data such as connection time, number of connection timeouts, routing time, cross-database query time, and SQL execution time. It performs performance monitoring for specific scenarios such as data source connection, dynamic routing, and cross-source queries, achieving high-performance monitoring and diagnosis in near real-time scenarios. Attached Figure Description

[0010] Figure 1 This is an overall architecture diagram of the multi-source heterogeneous data dynamic compilation and unified data service generation device of the present invention. Figure 2 is a technical architecture design diagram of the device for dynamic compilation of multi-source heterogeneous data and unified data service generation according to the present invention; Figure 3 is a flowchart of the method for dynamic compilation of multi-source heterogeneous data and generation of unified data services according to the present invention. Figure 4 This is a schematic diagram of the functional structure of the device for dynamic compilation of multi-source heterogeneous data and generation of unified data services according to the present invention. Detailed Implementation

[0011] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0012] This invention discloses a method and apparatus for dynamic weaving and unified service generation of multi-source heterogeneous data, relating to big data processing technology and the military field. The method includes: S1 Multi-source heterogeneous data source connection, establishing connections with various heterogeneous data sources such as relational databases and object storage based on connector systems such as JDBC Connector, object storage Connector, and graph database Connector, synchronizing data source directories and updating them in real time; S2 Dynamic data weaving configuration, configuring virtual view rules, sharding / transaction / encryption strategies, and routing rules; S3 Multi-source data dynamic weaving execution, achieving cross-source access and dynamic updates through federated queries and dual engines (Trino / Shardingsphere) to generate a unified view; S4 View authorization and permission control, implementing view authorization management through flexible user permission policies; S5 Performance monitoring, collecting, storing, and analyzing system performance data and generating diagnostic reports through a Java agent proxy probe mode; S6 Unified service generation, supporting the construction and application of data services for various business scenarios in the military field through an independent service proxy layer and related rule configuration. The device includes modules for data source connection, data weaving configuration, view authorization management, and data services. This invention addresses the challenges faced in the military field, such as the difficulty of integrating cross-source heterogeneous data, the low real-time generation of required decision data, and the lack of unified data services. It improves the efficiency and security of heterogeneous data access, supporting rapid generation of data services and quick business decision-making in the military field.

[0013] Dynamic data weaving method: The technical solution of this invention is: This invention provides a method for dynamic weaving and unified service generation of multi-source heterogeneous data, the method comprising: S1 data source integration: Connect to databases such as Oracle, MySQL, and DM through JDBC Connector, Object Storage Connector, and Graph Database Connector, synchronize data source directories, and update via push notifications (e.g., when a new Kafka real-time data source is added, notifications are sent and updates are made in real time via push notifications).

[0014] S2 weaving configuration: Virtual view creation rules: Integrate and process data across sources by dragging and dropping operators (data cleaning, association, aggregation, transformation, filtering, etc.); Weaving strategy: Vertical sharding splits fields according to military domain entity types (such as personnel and materials), and horizontal sharding splits data according to region / time; distributed transactions are selected as XA (strong consistency) or flexible transactions (high concurrency) depending on the scenario. Routing rules: Real-time data query requests are routed to the production database, while situation review and analysis requests are routed to the virtual database.

[0015] S3 Dynamic Weaving Execution: Parses SQL statements and dynamically switches between two engines based on the processing requirements. For OLAP scenarios, it calls the Trino engine (supporting cross-source multi-dimensional analysis requirements), while for OLTP scenarios, it calls the Shardingsphere engine (supporting real-time query requirements). Data synchronization enables data collaboration between the two engines. Finally, the returned data is extracted and merged to generate a view, and near real-time dynamic updates of the view are achieved through metadata collection.

[0016] S4 Access Control: A three-tier role system is built, with super administrators managing all views and roles, topic administrators managing and authorizing topic views, and ordinary users only being able to access authorized views.

[0017] S5 Performance Monitoring: The Java agent collects performance data such as connection time, connection timeout count, routing time, cross-database query time, and SQL execution time. The data is stored in time-sharded format and compressed using LZ4. Anomalies are detected using the 3σ principle, and optimization suggestions are generated.

[0018] Unified service generation method: The unified service generation method includes: T1 Service Configuration: Configure data service generation rules, including data source information, mapping rules, parameter rules, request types, HTTP / JDBC protocols, etc., and define encryption rules for sensitive data.

[0019] T2 Service Generation: After integrating the service generation rules configured in T1, it is encapsulated into API service (HTTP) and database proxy service (JDBC), which support service calls and data access after authorization by third-party systems.

[0020] T3 Service Management: Enables full lifecycle management of data services, including online debugging, publishing, editing, deletion, and authorization. It integrates Metrics / Tracing / Logging through a Java agent to trace SQL links and dynamically monitor data service performance such as response time, number of successful / failed calls, and error rate.

[0021] Device structure: A device for dynamic weaving and unified service generation of multi-source heterogeneous data includes: The system consists of two main subsystems: dynamic data weaving and unified service generation. These subsystems include the following models, and the modules communicate with each other via HTTP / HTTPS interfaces (e.g., the data weaving execution module provides a view access interface to the service generation module).

[0022] Data source connection module: Based on the connector system, it connects to multiple data sources, synchronizes the catalog, and updates adaptation information; Data weaving configuration module: Configure virtual views, weaving strategies, operator orchestration, routing, and encryption rules; Data weaving execution module: Enables cross-source data weaving and dynamic view updates through federated queries and dual engines; View authorization module: Manages view permissions based on a three-tier role system; Performance monitoring module: Collects and stores performance data and generates diagnostic reports; Encryption module: Encrypts and decrypts sensitive data according to rules.

[0023] A method and apparatus for dynamic weaving and unified service generation of multi-source heterogeneous data, characterized in that it includes: Service configuration module: Configures unified data service generation and data encryption rules; Service generation module: encapsulates the encrypted data view into a standardized data service; Service Management Module: Enables full lifecycle management of data services; Service monitoring module: Collects performance data and performs diagnostics accordingly; Example 1: Figure 1 , Figure 2 These are the overall architecture diagram and technical architecture design diagram of the multi-source heterogeneous data dynamic weaving and unified data service generation device of the present invention, including: data layer, connection layer, service layer and interface layer; The data layer supports heterogeneous data from multiple sources, including relational databases, object storage, and graph databases. It performs data mapping from various data sources, obtains metadata information from the database through hybrid indexes, and performs data weaving for the display of woven views.

[0024] The connection layer provides different connectors to enable the connection of multi-source heterogeneous data sources, including JDBC Connector, object storage Connector, graph database Connector and other connectors.

[0025] The service layer is the core business implementation layer. It uses Spring Cloud, MyBatis Plus, and Redis Template as the system framework and tools, supports efficient and flexible operation of multiple data sources, supports distributed data access and cross-data source federated queries, and uses Trino and Shardingsphere dual engines to support access to various databases. It provides functions such as data federation, read-write separation, data sharding, transaction processing, and data encryption.

[0026] The interface layer provides diverse data interfaces, supporting external systems to access data through different protocols or methods. It primarily provides APIs, implements data transmission based on the HTTP protocol, and also supports virtual view access to the data source using JDBC.

[0027] Figure 3 The flowchart of the multi-source heterogeneous data dynamic weaving and unified data service generation method of the present invention is as follows: First, the data source to be connected is connected, the data source connection information is registered, and virtual view information and user permission rules are configured. Then, the data weaving process begins, parsing SQL statements and dynamically selecting different query engines according to different parsing requirements. Trino is used for OLAP, and Shardingsphere is used for OLTP. The engine is started to query data according to the configured virtual view creation rules (data source and operator orchestration rules), data weaving strategy (data sharding / transaction / read-write separation / encryption) and routing rules. After the query, metadata is collected, and the returned data is merged and aggregated. According to the configured data service generation rules, encryption rules, etc., an API interface is encapsulated and generated, or JDBC is used to obtain the dataset through virtual view access. In the above process, a Java agent probe design pattern is adopted to collect, store, and analyze observable performance data during the process, realizing performance monitoring and diagnosis of the overall database application system. The functions of Metrics, Tracing, and Logging are integrated into the Agent through plug-ins. The collected performance data is stored through data sharding, data compression, index optimization, etc. By using a proxy and tracing approach, the entire SQL execution process can be traced, enabling rapid problem location and diagnosis.

[0028] Figure 4 This is a functional structure diagram of the multi-source heterogeneous data dynamic weaving and unified data service generation device of the present invention, including data source management, data weaving, view authorization management, performance monitoring and data service sub-modules; The data source management module provides connector registration and management functions to manage data source connections; Data weaving, based on data virtualization technology, provides cross-data source access capabilities through data federation queries and support for heterogeneous database dialects. It offers backend capabilities for data integration, transformation, and weaving, weaving data from different relational data sources into a unified data view interface according to certain rules. Key functions include data federation, data virtualization, data views, operator management, topic management, data services, data sharding, distributed transactions, read / write separation, and data encryption. View authorization management provides unified management of permissions for data weaving views. It allows for the configuration of corresponding policies for authorization as needed, including functions such as user role management and authorization management. Performance monitoring includes performance data acquisition, performance data management, and monitoring report management functions. It collects, stores, and analyzes system performance data to perform overall database system performance monitoring and diagnosis. The following section combines... Figure 3 The process description explains the specific implementation steps.

[0029] Step 1: Prepare the hardware and software environment for deployment; Hardware: Server CPU 16 cores, 64GB memory, storage ≥2TB; Terminal Windows 7 64-bit operating system; 8GB RAM, quad-core CPU Software: CentOS 8 or later, Chrome browser version 60 or later, Trino 425, Shardingsphere version 5.4.0.

[0030] Step 2: Data source connection configuration; Deploy the JDBC Connector to connect to the DM database (personnel and material data), the Object Storage Connector to connect to MinIO (satellite imagery), and the Graph Database Connector to connect to Neo4j (command scheduling data). Test the connection to ensure that the connected data sources are in a normal connectivity state. Then configure the message push interface to synchronize the data sources to update dynamic information.

[0031] Step 3: Configure weaving strategy; Virtual view rule configuration: Select the data source and orchestration operators to complete the creation of data view weaving rules; Vertical segmentation strategy: Split the material data table according to "Material Basic Information - Material Parameters - Maintenance Records"; Horizontal partitioning strategy: Split job geographic data according to "scheduling region – scheduling time"; Distributed transaction strategy: Real-time command issuance adopts the XA protocol, while situation review data statistical analysis adopts flexible transactions; Routing rule configuration: Configure the "Real-time Scheduling" tag to route to the production database, and the "Situation Review and Analysis" tag to route to the virtual database; Dynamic switching between dual engines: Real-time command issuance automatically selects and calls the Shardingsphere engine, while situation review data statistical analysis automatically selects and calls the Trino engine. After querying, metadata is collected and the returned data is merged and aggregated to form a unified virtual data view (material data situation analysis view and a material dispatch command data view for a certain area). The virtual data view is dynamically updated through metadata.

[0032] Step 4: Service configuration generation; Theme configuration: Organize the generated virtual data view into various themes based on multi-dimensional business attributes; User role configuration: Based on a three-tier role permission system, the super administrator configures the roles of topic administrator and topic view consumer; Topic View Authorization: After the topic administrator authorizes topic view permissions to topic view consumers, topic view consumers can obtain the material data situation analysis view and the material dispatch instruction data view for a certain region. They can view and browse the data returned by the material data situation analysis view and the material dispatch instruction data view for a certain region online. Service rule configuration: The topic view consumer selects the view configuration for the data service to be generated, including request type (get / post), service name, encryption rules (configure AES-256 encryption algorithm to encrypt the core parameter column of the material), etc.

[0033] Data service generation: Data view consumers debug and publish the configured services online, thus completing the standardized and unified service generation.

[0034] In this implementation example, system performance data is collected and tracked for all business processes, including data source connection, data weaving, service generation, and call consumption. This results in a performance monitoring report and diagnostic suggestions. The monitoring report includes data source connection timeouts, connection collection time, routing time, cross-database query time, SQL execution time, API call timeouts, and API call success / failure counts. Specifically, the Java agent collects and diagnoses the "data source connection time > 2s" anomaly, and the diagnostic report suggests optimizing the Connector timeout configuration.

[0035] Example 2: A method for dynamically weaving and generating unified services from multi-source heterogeneous data includes: S1: Multi-source heterogeneous data source integration, establishing connections with military relational databases, object storage, and graph databases based on a connector system. The connector system includes a JDBC Connector, an object storage Connector, and a graph database Connector. Each connector handles communication, data transformation, and SQL parsing and execution. The data source directory is synchronized through the data source interface, and the adaptation information is updated in real time using message push. S2: Dynamic data weaving configuration, based on data virtualization and federated query capabilities, configures virtual view creation rules (supports cross-source data retrieval and operator calls), data weaving strategies (data sharding / transactions / read-write separation / encryption) and routing rules (routing to the production database / virtual database according to the scenario). S3: Multi-source data dynamic weaving execution, starts the data weaving engine to access cross-source data through federated queries, supports database dialect adaptation, extracts and merges data after parsing and optimizing query statements and collects metadata information, realizes dynamic view updates through metadata collection, and generates a unified military domain data view. S4: View authorization and permission control, based on a three-tier role system of super administrator - topic administrator - ordinary user, create authorization policies and assign view permissions; S5: System performance monitoring, collecting performance data such as connection time, number of connection timeouts, routing time, cross-database query time, and SQL execution time, optimizing storage through sharding / compression / indexing, and generating diagnostic reports through the analysis engine.

[0036] A method for dynamic weaving and unified service generation of multi-source heterogeneous data. The unified service generation method, based on a data view, includes: T1: Data service configuration, configuring service generation rules (selecting data source, call type, etc.), interface protocol (HTTP / JDBC) and encryption rules; T2: Data service generation, encrypts sensitive data and encapsulates it into a standardized service, supporting API interface and JDBC access; T3: Data service management and monitoring, realizing full lifecycle management of services, collecting operational performance data and tracing SQL links, linking with system monitoring to perform service performance diagnosis and generate monitoring reports; A method for dynamic weaving and unified service generation of multi-source heterogeneous data, comprising a multi-source heterogeneous data dynamic weaving device, including: Data source connection module: Based on the connector system, it connects to multiple data sources, synchronizes the catalog, and updates adaptation information; Data weaving configuration module: Configure virtual views, weaving strategies, operator orchestration, routing, and encryption rules; Data weaving execution module: Enables cross-source data weaving and dynamic view updates through federated queries and dual engines; View authorization module: Manages view permissions based on a three-tier role system; Performance monitoring module: Collects and stores performance data and generates diagnostic reports; Encryption module: Encrypts and decrypts sensitive data according to rules.

[0037] A method and apparatus for dynamic weaving and unified service generation of multi-source heterogeneous data, characterized in that it includes: Service configuration module: Configures unified data service generation and data encryption rules; Service generation module: encapsulates the encrypted data view into a standardized data service; Service Management Module: Enables full lifecycle management of data services; Service monitoring module: Collects performance data and performs diagnostics accordingly; Furthermore, in step S1, the relational databases include Oracle, MySQL, PostgreSQL, and DM, and the message push is implemented based on a RESTful interface.

[0038] Furthermore, the commonly used operators in step S2 include algorithms such as data association, aggregation, transformation, and filtering, and custom operators can be generated using SQL statements.

[0039] Furthermore, in step S2, the distributed transaction strategy supports local transactions, XA protocol two-phase transactions, and flexible transactions (based on RocketMQ and compensation mechanisms).

[0040] Furthermore, in step S2, read / write separation supports one master and multiple slaves (load balancing strategies: round-robin, random, weighted) and multiple master and multiple slaves (master databases perform self-synchronization, master databases serve as backups for each other, and master-slave latency optimization is supported) configurations.

[0041] Furthermore, in step S2, the encryption strategy supports AES-256, SM4 algorithms, and custom algorithms, and the encryption column and corresponding algorithm are configured.

[0042] Furthermore, the performance diagnostic report in step S5 includes trend charts, anomaly lists, etc., and supports visual display.

[0043] Furthermore, in step T3, performance data collection uses a Java agent integrated with Metrics / Tracing / Logging, and employs data sharding, data compression, and index optimization for storage.

[0044] Furthermore, the data source connection module also connects to streaming and unstructured data via ODBC, HTTP, and Kafka.

[0045] Furthermore, the data weaving execution module automatically selects either the Trino (OLAP) or Shardingsphere (OLTP) engine based on SQL complexity and data volume thresholds to achieve high-performance, near real-time data query and analysis.

[0046] Furthermore, the service configuration module supports full lifecycle management of data services, and allows for online debugging, editing, viewing, and deleting of data services as well as browsing of specific data content.

[0047] Furthermore, the service generation module supports configuring custom encryption and desensitization rules to quickly generate a unified data service interface after encrypting the data view.

[0048] This invention provides a method and apparatus for dynamically weaving and generating unified services from multi-source heterogeneous data. Through technologies such as dual-engine (Shardingsphere + Trino) dynamic switching, data virtualization, federated queries, data sharding, distributed transactions, and read / write separation, it separates the underlying heterogeneous data sources from military and cross-organizational data sources from the upper-layer applications. An abstraction layer is created to generate a unified data virtual view and standardized data service APIs, enabling applications to access data through a unified data view and unified API. This effectively improves the adaptability of military-specific data sources. The method and apparatus can be tailored to specific business scenarios in the military field (real-time rapid scheduling / (Situation review and analysis) Dynamically adjust the weaving strategy to solve the problems of cross-source data query and multi-dimensional data analysis and processing in high real-time scenarios in the military field while taking into account system performance; furthermore, develop API interfaces based on the formed unified view to standardize data service generation, effectively reduce the cost of data service construction and access, and improve data service compatibility and ease of use; finally, this invention integrates the functions of Metrics, Tracing, and Logging into the Agent through plugins, and performs performance monitoring for specific scenarios such as data source connection, dynamic routing, and cross-source query, realizing high-performance monitoring and diagnosis and rapid problem location capabilities in near real-time scenarios.

[0049] The beneficial effects of this invention are: High adaptability: Optimize connection strategies to form a connector system for heterogeneous data sources and business decision-making scenarios in the military field; Efficiency Improvement: The dual-engine architecture balances performance and query analysis capabilities, dynamically switching based on the processing scenario to improve query response speed and ensure multi-dimensional analysis capabilities across data sources and high-concurrency real-time query efficiency. High security: Access control is implemented based on a three-level role system. Access control and encryption mechanisms work together to ensure that data is accessed according to the rights and prevent the leakage of sensitive data. High ease of use: Standardized data service interface generation forms a unified service, reducing the cost of building and accessing data services and improving the ease of use of data services.

[0050] Highly targeted: Performance monitoring uses a Java agent to collect performance data such as connection time, number of connection timeouts, routing time, cross-database query time, and SQL execution time. It performs performance monitoring for specific scenarios such as data source connection, dynamic routing, and cross-source queries, achieving high-performance monitoring and diagnosis in near real-time scenarios.

[0051] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-source heterogeneous data dynamic weaving and unified service generation method, characterized in that, This method includes a dynamic data weaving method and a unified service generation method; among them, Dynamic data weaving methods include: S1: Multi-source heterogeneous data source integration, establishing connections with military relational databases, object storage, and graph databases based on the connector system, synchronizing the data source directory through the data source interface, and using message push to update adaptation information in real time; S2: Dynamic data weaving configuration, based on data virtualization and federated query capabilities, configures virtual view creation rules, data weaving strategies and routing rules; S3: Multi-source data dynamic weaving execution, starts the data weaving engine to access cross-source data through federated queries, supports database dialect adaptation, extracts and merges data after parsing and optimizing query statements and collects metadata information, realizes dynamic view updates through metadata collection, and generates a unified military domain data view. S4: View authorization and permission control, based on a three-tier role system of super administrator - topic administrator - ordinary user, create authorization policies and assign view permissions; S5: System performance monitoring, collects performance data, optimizes storage through sharding, compression, and indexing, and generates diagnostic reports through the analysis engine; The unified service generation method is based on a data view and includes: T1: Data service configuration, configuring service generation rules, interface protocols and encryption rules; T2: Data service generation, encrypts sensitive data and encapsulates it into a standardized service, supporting API interface and JDBC access; T3: Data service management and monitoring, enabling full lifecycle management of services, collecting operational performance data and tracing SQL links, linking with system monitoring to perform service performance diagnosis and generate monitoring reports.

2. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 1, characterized in that, In S1, the connector system includes: JDBC Connector, Object Storage Connector, and Graph Database Connector. Each connector handles communication, data transformation, and SQL parsing and execution.

3. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 1, characterized in that, In S2, the virtual view creation rules include: cross-source integration and processing of data through drag-and-drop operators; the weaving strategy includes: vertical sharding splitting fields according to military domain entity types, horizontal sharding splitting data according to region and time, and distributed transactions selecting strong consistency or flexible transactions according to the scenario; the routing rules include: real-time data query requests are routed to the production database, and situation review and analysis requests are routed to the virtual database.

4. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 1, characterized in that, The S3 includes: parsing SQL statements, dynamically switching between the two engines based on the processing requirements of the parsing, calling the Trino engine in OLAP scenarios to support cross-source multi-dimensional analysis requirements, calling the Shardingsphere engine in OLTP scenarios to support real-time query requirements, achieving data collaboration between the two engines through data synchronization, and finally extracting and merging the returned data to generate a view, and achieving near real-time dynamic updates of the view through metadata collection.

5. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 4, characterized in that, The S4 includes: building a three-level role system, where the super administrator manages all views and roles, the topic administrator manages and authorizes topic views, and ordinary users can only access authorized views.

6. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 5, characterized in that, The S5 includes: a Java agent collecting data on connection time, connection timeout count, routing time, cross-database query time, and SQL execution time, storing the data in time-sharded chunks and using LZ4 compression, detecting anomalies using the 3σ principle, and generating optimization suggestions.

7. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 1, characterized in that, The T1 includes: configuring data service generation rules, including data source information, mapping rules, parameter rules, request types, HTTP / JDBC protocol, and defining encryption rules for sensitive data.

8. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 7, characterized in that, The T2 includes: integrating the service generation rules configured in T1 and encapsulating them into API services and database proxy services JDBC, which support service calls and data access after authorization by third-party systems.

9. The method for dynamic weaving and unified service generation of multi-source heterogeneous data as described in claim 8, characterized in that, The T3 includes: enabling full lifecycle management of online service debugging, publishing, editing, deletion, and authorization; integrating Metrics, Tracing, and Logging through JavaAgent to trace SQL links; and dynamically monitoring data service response time, number of successful calls / failed calls, and error rate performance.

10. A device for dynamic weaving and unified service generation of multi-source heterogeneous data based on the method described in any one of claims 1-9, characterized in that, The device includes: Data source connection module: Based on the connector system, it connects to multiple data sources, synchronizes the catalog, and updates adaptation information; Data weaving configuration module: Configure virtual views, weaving strategies, operator orchestration, routing, and encryption rules; Data weaving execution module: Enables cross-source data weaving and dynamic view updates through federated queries and dual engines; View authorization module: Manages view permissions based on a three-tier role system; Performance monitoring module: Collects and stores performance data and generates diagnostic reports; Encryption module: encrypts and decrypts sensitive data according to rules; Service configuration module: Configures unified data service generation and data encryption rules; Service generation module: encapsulates the encrypted data view into a standardized data service; Service Management Module: Enables full lifecycle management of data services; Service monitoring module: Collects performance data and performs diagnostics accordingly.

Images