A network data weaving all-in-one machine system architecture

Abstract

This invention discloses a network-based data weaving integrated machine system architecture, belonging to the field of data management and integration technology. It includes a hierarchically deployed basic node layer, composed of multiple physical or virtual device nodes, each node comprising a computing unit, storage unit, caching unit, network communication unit, and query execution engine; a node control layer, configured to uniformly manage all nodes, including a node registration module, a node discovery module, a task scheduling module, and a routing control module; a data weaving capability layer, configured to provide data management functions, including an active metadata management module, a federated query module, a semantic layer module, and an access control module; a network collaboration layer, configured to realize multi-node collaboration, including a cross-node query collaboration module, a metadata synchronization module, and an access control mapping module; and an intelligent collaboration layer, configured to provide intelligent computing interfaces. This invention employs the above system architecture to construct a scalable, networkable, and collaborative network-based data infrastructure.

Description

Technical Field

[0001] This invention relates to the technical field of data management and integration, and in particular to a network-based data weaving machine system architecture. Background Technology

[0002] With the rapid development of artificial intelligence, large-scale models, and the marketization of data elements, enterprises and industry users are placing higher demands on data infrastructure. Data Fabric, as a modern data architecture, aims to achieve seamless integration, management, and application of data through automation and intelligence, supporting cross-platform and cross-system data access and processing. International companies such as IBM, Informatica, and Talend have already launched data fabric-related solutions, which are widely used in industries such as finance, healthcare, and manufacturing. Domestic data fabric technology started later in China, and mature products and solutions remain relatively scarce.

[0003] However, existing data management and integration technologies still face several prominent challenges in practical applications. First, traditional data platforms often employ single-machine, single-cluster, or single-center deployment models. Typical examples include data warehouses, data middleware, federated query platforms, and metadata platforms. These systems typically operate independently in local environments, hindering cross-regional, cross-organizational, and cross-node collaborative work and preventing the formation of a network-based data infrastructure. Second, existing software platforms typically rely on software instance expansion for scaling, making it difficult to achieve horizontal scaling at the device level, vertical scaling of functional modules, and elastic scaling of network nodes. This hinders their ability to flexibly adapt to the demands of high concurrency, cross-domain collaboration, and rapid business changes. Third, traditional data architectures primarily focus on the storage and retrieval of structured data, making it difficult to effectively support emerging intelligent application scenarios such as federated learning, AI agent collaboration, model sharing, and vector retrieval. Furthermore, they exhibit significant shortcomings in areas such as automatic multimodal data identification and association, deep learning-driven data fusion, and real-time semantic understanding, failing to meet the requirements of deep integration and dynamic scheduling of multi-source heterogeneous data for large-scale model training and inference. Summary of the Invention

[0004] The purpose of this invention is to provide a network-based integrated data weaving machine system architecture that solves the problems of isolated deployment, poor scalability, and lack of network collaboration capabilities in the existing technology.

[0005] To achieve the above objectives, the present invention provides a network-based data weaving integrated machine system architecture, comprising a basic node layer, a node control layer, a data weaving capability layer, a network collaboration layer, and an intelligent collaboration layer deployed in sequence. The basic node layer consists of multiple physical or virtual device nodes, each of which includes a computing unit, a storage unit, a cache unit, a network communication unit, and a query execution engine. The node control layer is configured to manage all nodes in a unified manner, including a node registration module, a node discovery module, a health monitoring module, a task scheduling module, and a routing control module. The data weaving capability layer is configured to provide data management functions, including an active metadata management module, a federated query module, a semantic layer module, a data security module, an access control module, and a data asset module. The network collaboration layer is configured to enable multi-node collaboration, including a cross-node query collaboration module, a metadata synchronization module, a permission mapping module, a task distribution module, and a model synchronization module. The intelligent collaboration layer is configured to provide intelligent computing interfaces, including a federated learning engine interface, an AI Agent interface, a vector retrieval interface, and an inference acceleration module.

[0006] Preferably, each node in the basic node layer includes an independent storage unit and a cache unit. The cache unit uses a distributed consistent hashing algorithm to store the routing information of the data shards. The routing control module forwards the query request directly to the node storing the target data shard based on the routing information.

[0007] Preferably, each node in the basic node layer encapsulates data weaving capabilities into network node infrastructure devices, with each node operating independently or in a network with other nodes; the node control layer automatically discovers new nodes in the network through the node discovery module and completes node registration through the node registration module, thereby realizing automatic networking of multiple nodes.

[0008] Preferably, the network collaboration layer and the node control layer collaborate to construct a virtual-physical integrated data network: the physical network includes physical connections between headquarters nodes, regional nodes, edge nodes, and customer nodes; the logical network includes a metadata network, a permission network, a semantic network, and a model network; the virtual-physical integrated data network realizes the association mapping between physical nodes and the logical network through a metadata synchronization module and a permission mapping module.

[0009] Preferably, the proactive metadata management module includes a unified metadata graph model, supports a metadata ingestion framework for multiple data source types, a global metadata service, a visualization service platform, and sharing and consumption APIs; the metadata synchronization module synchronizes metadata changes across multiple nodes based on the global metadata service.

[0010] Preferably, the federated query module adopts a storage-computation separation mechanism based on a connector architecture, including a data source connector framework and a distributed SQL query engine; the data source connector framework converts relational databases, object storage, or key-value stores into unified tables, columns, rows, and data types; the distributed SQL query engine adopts a massively parallel processing architecture, horizontally distributed across multiple nodes to form a cluster, supporting federated queries on physically distributed HDFS, RDBMS, and other data sources.

[0011] Preferably, the federated learning engine interface in the intelligent collaboration layer is configured to aggregate model updates without exchanging raw data among local nodes; the AI ​​Agent interface is configured to receive task requests from external AI agents and return inference results; the vector retrieval interface is configured to support approximate nearest neighbor search for high-dimensional vectors; and the inference acceleration module includes FPGA or GPU hardware acceleration units.

[0012] Preferably, the query execution engine supports cross-node query collaboration, specifically including: when a query request is received that spans multiple nodes, the cross-node query collaboration module decomposes the query into subqueries, distributes them to the query execution engines of the relevant nodes through the task distribution module, and obtains the latest metadata of each node through the metadata synchronization module to optimize the execution plan.

[0013] Preferably, the system architecture supports unified scheduling of horizontal expansion, vertical expansion, and network expansion. Specifically, horizontal expansion enhances computing and storage capabilities by adding device nodes in the basic node layer; vertical expansion enhances business capabilities by adding functional modules to the data weaving capability layer or intelligent collaboration layer; network expansion expands network coverage by adding business nodes; and the task scheduling module dynamically allocates data shards and computing tasks according to the expansion method.

[0014] Therefore, the present invention adopts the above-described network-based data weaving machine system architecture, which has the following advantages: (1) In this invention, through the hierarchical collaboration of the basic node layer, the node control layer and the network collaboration layer, the data weaving capability is encapsulated into a network node infrastructure device. Each node can operate independently or in a network. The node control layer automatically discovers and registers new nodes, which solves the problems of isolated deployment of single nodes, lack of cross-node metadata synchronization, permission collaboration and query collaboration in the prior art, and realizes a network-type data infrastructure that is cross-regional, cross-organizational and cross-node.

[0015] (2) In this invention, a unified scheduling mechanism is used to add device nodes by horizontal expansion, add functional modules by vertical expansion, and add business nodes by network expansion. Combined with the use of distributed consistent hashing algorithm to store data fragment routing information in the cache unit, the routing control module directly forwards the query request to the target node. This solves the problem that the existing technology has a single expansion method and cannot achieve device-level elastic expansion and dynamic load balancing. It realizes the on-demand improvement of computing and storage capabilities and flexible adaptation in high-concurrency scenarios.

[0016] (3) In this invention, the federated learning engine interface, AI Agent interface, vector retrieval interface and inference acceleration module are set by the intelligent collaboration layer. The federated learning engine interface aggregates model updates without exchanging original data among local nodes. The inference acceleration module includes FPGA or GPU hardware acceleration units. Combined with the cross-node query collaboration module, the query is decomposed into subqueries and the latest metadata is obtained to optimize the execution plan. This solves the problem that the existing technology is difficult to support federated learning, multimodal data fusion and large model intelligent application. It realizes privacy protection collaboration without data leaving the domain, cross-node intelligent query acceleration and efficient utilization of heterogeneous computing resources.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the framework of a network-type data weaving machine system architecture according to the present invention; Figure 2 This is a schematic diagram of the vertical expansion provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the lateral expansion provided in Embodiment 1 of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Specific model specifications need to be selected and determined according to the actual specifications of the device, etc. The specific selection calculation method adopts existing technology in the art, and therefore will not be described in detail.

[0020] Example 1 like Figures 1-3 As shown, the present invention provides a network-based data weaving integrated machine system architecture, including a basic node layer, a node control layer, a data weaving capability layer, a network collaboration layer, and an intelligent collaboration layer deployed in sequence; and a user access layer that calls the intelligent collaboration layer and other lower-level capabilities. The basic node layer consists of multiple physical or virtual device nodes, each of which includes a computing unit, a storage unit, a cache unit, a network communication unit, and a query execution engine. The node control layer is configured to manage all nodes in a unified manner, including a node registration module, a node discovery module, a health monitoring module, a task scheduling module, and a routing control module. The data weaving capability layer is configured to provide data management functions, including an active metadata management module, a federated query module, a semantic layer module, a data security module, an access control module, and a data asset module; the semantic layer module is based on natural language processing technology to extract semantic information from the data and supports intelligent search and recommendation of distributed multi-source heterogeneous data; The network collaboration layer is configured to enable multi-node collaboration, including a cross-node query collaboration module, a metadata synchronization module, a permission mapping module, a task distribution module, and a model synchronization module. The intelligent collaboration layer is configured to provide intelligent computing interfaces, including a federated learning engine interface, an AI Agent interface, a vector retrieval interface, and an inference acceleration module.

[0021] Employing a graph-based distributed data weaving model, this system dynamically connects heterogeneous data from multiple sources (such as relational databases, NoSQL databases, and file systems) to construct a unified logical data layer. Dynamic data connectivity breaks down data silos through a lightweight data weaving protocol, enabling seamless data flow across systems and platforms. Data virtualization virtualizes dispersed data sources into a unified view, freeing users from concern for data storage location and format. Real-time data synchronization supports real-time data synchronization and updates, ensuring data consistency and timeliness.

[0022] Each node in the basic node layer includes an independent storage unit and a cache unit. The cache unit uses a distributed consistent hashing algorithm to store the routing information of the data shards. Based on this routing information, the routing control module directly forwards the query request to the node storing the target data shard.

[0023] Each node in the basic node layer encapsulates data weaving capabilities into network node infrastructure devices. Each node can operate independently or in a network with other nodes. The node control layer automatically discovers new nodes in the network through the node discovery module and completes node registration through the node registration module, thereby realizing automatic networking of multiple nodes.

[0024] The network collaboration layer and the node control layer work together to build a virtual-physical integrated data network: the physical network includes physical connections between headquarters nodes, regional nodes, edge nodes and customer nodes; the logical network includes metadata network, permission network, semantic network and model network; the virtual-physical integrated data network realizes the association mapping between physical nodes and logical network through metadata synchronization module and permission mapping module.

[0025] The proactive metadata management module includes a unified metadata graph model, supports a metadata ingestion framework for various data source types, a global metadata service, a visualization service platform, and sharing and consumption APIs; the metadata synchronization module synchronizes metadata changes across multiple nodes based on the global metadata service.

[0026] Proactive metadata management is an enhanced active data catalog that leverages a unified metadata graph model to achieve end-to-end metadata discovery, observability, and governance. The metadata ingestion framework is an extensible framework supporting plugins for data ingestion, transformation, and processing. Global metadata services: Unified management and maintenance of metadata information across the entire data ecosystem, including models and storage. Visualization service platform: Provides intuitive visualization services for data discovery, data lineage, analysis and statistics, and data impact. Sharing and consumption APIs: Provides standardized interfaces for data sharing and consumption, including GraphQL, OpenAPI, Rest.liAPI, and Java & Python SDKs, facilitating data consumption and integration.

[0027] The federated query module adopts a storage-computation separation mechanism based on a connector architecture, including a data source connector framework and a distributed SQL query engine. The data source connector framework transforms relational databases, object storage, or key-value stores into unified tables, columns, rows, and data types. The distributed SQL query engine adopts a massively parallel processing architecture, horizontally distributed across multiple nodes to form a cluster, supporting federated queries on physically distributed HDFS, RDBMS, and other data sources.

[0028] The federated query module can solve the analysis and insight problems of physically distributed data by using methods such as join queries, parallel queries, and horizontal cluster scaling. Data federation does not require data replication and migration to a centralized location; it allows querying data in physically distributed HDFS and other distributed object storage systems, as well as RDBMS and other data sources. It supports over 40 data application clients and over 30 data source types. The core of data federation's storage and compute separation is based on a connector-based architecture. Connectors provide interfaces for data federation to access any data source. The connector framework is user-definable for converting underlying data sources (such as relational database management systems (RDBMS), object storage, or key-value stores) into tables, columns, rows, and data types for data federation. This conversion can be a simple SQL-to-SQL conversion and mapping, or a more complex conversion from SQL to object storage or NoSQL systems.

[0029] The federated learning engine interface in the intelligent collaboration layer is configured to aggregate model updates without exchanging raw data among local nodes; the AI ​​Agent interface is configured to receive task requests from external AI agents and return inference results; the vector retrieval interface is configured to support approximate nearest neighbor search for high-dimensional vectors; and the inference acceleration module includes FPGA or GPU hardware acceleration units.

[0030] The query execution engine supports cross-node query collaboration, specifically including: when a query request is received that spans multiple nodes, the cross-node query collaboration module decomposes the query into subqueries, distributes them to the query execution engines of the relevant nodes through the task distribution module, and obtains the latest metadata of each node through the metadata synchronization module to optimize the execution plan.

[0031] The system architecture supports unified scheduling for horizontal, vertical, and network expansion. Specifically, horizontal expansion enhances computing and storage capabilities by adding device nodes to the basic node layer. Horizontal expansion supports clustered deployment of multiple all-in-one machines, automatically allocating data shards and computing tasks based on a consistent hashing algorithm to avoid single-point bottlenecks. Vertical expansion enhances business capabilities by adding functional modules to the data weaving capability layer or intelligent collaboration layer. It can also achieve on-demand upgrades of single-machine performance through modular hardware design (such as pluggable GPU cards and NVMe storage expansion slots) to meet bursty computing demands. Network expansion extends network coverage by adding business nodes. The task scheduling module dynamically allocates data shards and computing tasks according to the expansion method.

[0032] Example 2 This embodiment takes a large manufacturing enterprise as an example to describe the specific application of the system architecture of the present invention in a network scenario of enterprise headquarters and multiple branches.

[0033] Deployment method: One master control node is deployed in the headquarters data center, and one edge node is deployed in each branch office (East China, South China, and North China factories). The master control node and edge nodes are interconnected via dedicated lines or VPNs to form a three-tier physical network of headquarters-regions-edge.

[0034] Metadata Synchronization: The master node's proactive metadata management module is configured with a global metadata service. Based on this global metadata service, the metadata synchronization modules of each edge node periodically synchronize metadata changes from local data sources (such as table structure changes or new data files) to the master node. The master node maintains a unified metadata graph model, recording the physical location, lineage, and access permissions of each data asset.

[0035] Cross-node query: When a headquarters business staff member initiates a query to summarize order data across three factories, the cross-node query collaboration module on the master node receives the query request, breaks it down into three sub-queries, and distributes them to the query execution engines of the three edge nodes through the task distribution module. After each edge node executes the query locally, it returns the results to the master node, which then aggregates the results and returns them to the user. During the query process, the cross-node query collaboration module obtains the latest metadata from each edge node through the metadata synchronization module to optimize the execution plan (for example, if it finds that the data of a certain factory has not been updated recently, it skips the query for that node).

[0036] Access Control Coordination: The headquarters' access control module centrally manages user permissions, and the access mapping module maps headquarters role permissions to the local access control systems of each edge node. When an edge node receives a query request, the access mapping module verifies whether the user has permission to access the data assets on that node.

[0037] This embodiment enables cross-domain collaborative querying without data leaving the factory (raw data from each edge node is not transmitted to headquarters), solving the network bandwidth pressure and data security issues caused by data aggregation in traditional solutions.

[0038] Example 3 This embodiment uses a medical consortium composed of multiple hospitals as an example to describe the application of the present invention in a privacy-protected data collaboration scenario.

[0039] Deployment method: Each hospital deploys one integrated machine node in its on-site data center, and the medical consortium central node deploys one master control node. Each hospital node is connected to the central node via a dedicated healthcare network.

[0040] Federated learning collaboration: Each hospital node deploys an intelligent collaboration layer. The federated learning engine interface is configured to aggregate model updates without exchanging original patient data among local nodes. The specific process is as follows: The central node distributes the initial pneumonia CT image recognition model (initial parameters) to each hospital node; Each hospital node uses local desensitized CT image data to train the model and update the model parameters, but the original data never leaves the hospital. Each hospital node encrypts and uploads model parameter updates (rather than raw data) to the central node through the federated learning engine interface; The central node's federated learning engine interface aggregates parameter updates from each hospital node to generate a new round of global model; The above steps are repeated multiple times until the model converges. Throughout the process, no single hospital can access the original patient data from other hospitals, thus meeting medical data privacy compliance requirements.

[0041] Inference acceleration: Each hospital node is equipped with an inference acceleration module. When a doctor uploads CT images of a suspected pneumonia patient for auxiliary diagnosis, the inference acceleration module of the local node calls the pre-trained model to perform inference and returns diagnostic suggestions in milliseconds, without having to upload sensitive image data to the center or the cloud.

[0042] This embodiment resolves the contradiction between the requirement that medical data "cannot leave the domain" and the need for "cross-institutional collaborative training of models".

[0043] Example 4 This embodiment takes the automotive parts supply chain as an example to describe a scenario in which multiple enterprise nodes automatically form a network to create a trusted data space for the industry.

[0044] Deployment method: OEMs, Tier 1 suppliers, and Tier 2 suppliers each deploy integrated machine nodes internally. Each node connects via the Internet and VPN. The node control layer automatically discovers other companies' nodes through a node discovery module (based on a DHT distributed hash table), completes node registration after two-way authentication, and achieves automatic networking of multiple enterprise nodes.

[0045] Data Asset Catalog: Each enterprise node registers its own data assets (such as parts inventory, delivery cycle, and quality inspection pass rate) that it is willing to share into the global metadata service through the proactive metadata management module, and sets access permissions (for example, second-tier suppliers can only view the procurement plans published by the OEM and cannot view quotations from other suppliers).

[0046] Federated Query: The OEM node initiates a query for "Inventory and Delivery Cycle of a Specific Part Model from All Suppliers in East China". The cross-node query collaboration module decomposes this query and sends it to each Tier 1 supplier node via the task distribution module. Each Tier 1 supplier node further distributes sub-queries involving Tier 2 suppliers to the Tier 2 supplier nodes. After each node executes the query locally, it returns the results along the original path, which are finally aggregated and presented by the OEM node.

[0047] Data security: Each node deploys a data security module to encrypt query results transmitted across enterprises at the field level (e.g., supplier names are displayed in encrypted form, and only the OEM has the decryption key).

[0048] This embodiment achieves reliable data flow among multi-level enterprises in the supply chain without establishing a central data warehouse, allowing each enterprise to maintain complete control over its own data.

[0049] Therefore, the present invention adopts the above-mentioned network-type data weaving integrated machine system architecture, realizing a unified and elastic expansion of a data network with automatic multi-node networking and virtual-physical integration, as well as at the device level, functional level, and network level, thereby constructing a scalable, networkable, and collaborative network-type data infrastructure.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A network-based data weaving integrated machine system architecture, characterized in that: It includes a hierarchical deployment of a basic node layer, a node control layer, a data weaving capability layer, a network collaboration layer, and an intelligent collaboration layer; The basic node layer consists of multiple physical or virtual device nodes, each of which includes a computing unit, a storage unit, a cache unit, a network communication unit, and a query execution engine. The node control layer is configured to manage all nodes in a unified manner, including a node registration module, a node discovery module, a health monitoring module, a task scheduling module, and a routing control module. The data weaving capability layer is configured to provide data management functions, including an active metadata management module, a federated query module, a semantic layer module, a data security module, an access control module, and a data asset module. The network collaboration layer is configured to enable multi-node collaboration, including a cross-node query collaboration module, a metadata synchronization module, a permission mapping module, a task distribution module, and a model synchronization module. The intelligent collaboration layer is configured to provide intelligent computing interfaces, including a federated learning engine interface, an AI Agent interface, a vector retrieval interface, and an inference acceleration module.

2. The network-type data weaving integrated machine system architecture according to claim 1, characterized in that: Each node in the basic node layer includes an independent storage unit and a cache unit. The cache unit uses a distributed consistent hashing algorithm to store the routing information of the data shards. Based on this routing information, the routing control module directly forwards the query request to the node storing the target data shard.

3. The network data weaving integrated machine system architecture of claim 2, wherein: Each node in the basic node layer encapsulates data weaving capabilities into network node infrastructure devices. Each node can operate independently or in a network with other nodes. The node control layer automatically discovers new nodes in the network through the node discovery module and completes node registration through the node registration module, thereby realizing automatic networking of multiple nodes.

4. The network data weaving integrated machine system architecture of claim 3, wherein: The network collaboration layer and the node control layer work together to build a virtual-physical integrated data network: the physical network includes physical connections between headquarters nodes, regional nodes, edge nodes and customer nodes; the logical network includes metadata network, permission network, semantic network and model network; the virtual-physical integrated data network realizes the association mapping between physical nodes and logical network through metadata synchronization module and permission mapping module.

5. The network-type data weaving integrated machine system architecture according to claim 4, characterized in that: The proactive metadata management module includes a unified metadata graph model, supports a metadata ingestion framework for multiple data source types, a global metadata service, a visualization service platform, and sharing and consumption APIs; The metadata synchronization module synchronizes metadata changes across multiple nodes based on the global metadata service.

6. The network data weaving integrated machine system architecture of claim 5, wherein: The federated query module adopts a storage and computation separation mechanism based on a connector architecture, including a data source connector framework and a distributed SQL query engine. The data source connector framework transforms relational databases, object stores, or key-value stores into unified tables, columns, rows, and data types; The distributed SQL query engine adopts a massively parallel processing architecture, which is horizontally distributed across multiple nodes to form a cluster, supporting joint queries on physically distributed HDFS, RDBMS and other data sources.

7. The network data weaving integrated machine system architecture of claim 6, wherein: The federated learning engine interface in the intelligent collaboration layer is configured to aggregate model updates without exchanging raw data among local nodes; the AIAgent interface is configured to receive task requests from external AI agents and return inference results; the vector retrieval interface is configured to support approximate nearest neighbor search for high-dimensional vectors; and the inference acceleration module includes FPGA or GPU hardware acceleration units.

8. The network data weaving integrated machine system architecture of claim 7, wherein: The query execution engine supports cross-node query collaboration, specifically including: when a query request is received that spans multiple nodes, the cross-node query collaboration module decomposes the query into subqueries, distributes them to the query execution engines of the relevant nodes through the task distribution module, and obtains the latest metadata of each node through the metadata synchronization module to optimize the execution plan.

9. The network data weaving integrated machine system architecture of claim 8, wherein: The system architecture supports unified scheduling of horizontal scaling, vertical scaling, and network expansion. Specifically, horizontal scaling enhances computing and storage capabilities by adding device nodes in the basic node layer; vertical scaling enhances business capabilities by adding functional modules to the data weaving capability layer or intelligent collaboration layer; network expansion expands network coverage by adding business nodes; and the task scheduling module dynamically allocates data shards and computing tasks according to the scaling method.

Images