Intelligent data management method based on knowledge graph and semantic reasoning

Abstract

The invention relates to the technical field of data processing, and particularly discloses an intelligent data governance method based on a knowledge graph and semantic reasoning. The method comprises the following steps: constructing a dynamically evolved domain knowledge graph and performing real-time updating based on a time sequence perception graph neural network; executing depth semantic matching fusing an attention mechanism and fuzzy description logic consistency verification to detect conflicts; establishing a differentiable mapping relation between the data quality index and the graph node embedding, and realizing closed-loop optimization through gradient back propagation; generating a governance strategy based on the context, and automatically executing cleaning, fusion and authority control; the treatment effect is evaluated and fed back through reinforcement learning, and the system is driven to evolve continuously. According to the technical scheme, the automation level of data management, decision reliability and adaptability to dynamic change of business semantics are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to an intelligent data governance method based on a knowledge graph and semantic reasoning. BACKGROUND

[0002] With the deep integration of artificial intelligence and big data technology, data governance has become the core link to guarantee the quality, safety and value release of enterprise data assets. Traditional data governance methods mainly rely on rule engines, metadata management and manual review processes, and their core goal is to achieve standardized, consistent and compliant control of data. However, when facing massive, high-dimensional and dynamically evolving business data, such methods expose systematic defects such as weak semantic understanding ability, missing context association and insufficient automated reasoning level, making it difficult to support deep mining and intelligent control of data connotation in complex business scenarios.

[0003] Among them, the intelligent data governance method based on the knowledge graph and semantic reasoning has become a research hotspot in recent years. This direction aims to explicitly express scattered data entities and their relationships as structured semantic networks by building domain ontology models, and automatically identify data anomalies, conflicts and redundancies using logical reasoning mechanisms.

[0004] The prior art generally adopts a static ontology modeling strategy when building a knowledge graph, which cannot adapt to the dynamic evolution of business semantics, resulting in a disconnection between the graph and the actual data context. At the same time, the semantic reasoning process relies mainly on predefined rule sets, lacks quantitative processing capabilities for uncertainty and fuzzy semantics, and is difficult to cope with the widespread ambiguity and noise in real scenarios. In addition, the existing governance framework does not establish a differentiable association mechanism between data quality indicators and the topology of the knowledge graph, making the reasoning results unable to optimize the graph construction process in a closed loop, resulting in a lack of closed-loop governance capability. The above problems are particularly prominent in high-sensitivity fields such as financial risk control, medical health and intelligent manufacturing, severely restricting the intelligent level and decision reliability of the data governance system, so there is an urgent need for an intelligent data governance method that can integrate dynamic knowledge modeling, probabilistic semantic reasoning and adaptive optimization mechanisms. SUMMARY

[0005] The purpose of the present application is to provide an intelligent data governance method based on a knowledge graph and semantic reasoning, which can effectively solve the problems in the background art.

[0006] In order to achieve the above object, the technical scheme adopted by the present application is: An intelligent data governance method based on a knowledge graph and semantic reasoning, comprising the following specific steps: Step 1: constructing a dynamically evolving domain knowledge graph, extracting entities, attributes and relationships from multiple source heterogeneous data, and dynamically updating the graph structure and semantic representation based on a time sequence perception graph neural network model, wherein the number of entity types is not less than 50, the number of relationship types is not less than 30, and the graph update cycle is not more than 24 hours; Step 2: performing probabilistic semantic reasoning and conflict detection, calculating the semantic similarity between entities based on the knowledge graph using a deep semantic matching model with a fusion attention mechanism, and performing consistency checking using fuzzy description logic to identify potential conflicts and redundancies in the data, the semantic similarity calculation dimension is not less than 512 dimensions, and the conflict detection accuracy is greater than or equal to 95%; Step 3: implementing joint optimization of data quality indicators and graph topology, establishing a differentiable mapping relationship between data quality evaluation indicators and knowledge graph node embedding vectors, adaptively adjusting the graph construction parameters through the gradient back propagation mechanism, realizing the closed loop of the governance process, and setting the number of joint optimization iterations to 1000 times and the initial value of the learning rate to 0.001; Step 4: performing context-aware intelligent decision support, generating data governance strategy suggestions based on the reasoning results and the optimized knowledge graph, and automatically implementing data cleaning, fusion and permission control operations through a strategy execution engine, and the strategy generation response time is less than 2 seconds; Step 5: realizing governance effect evaluation and feedback learning, collecting the data state changes after the strategy execution, calculating the governance effect quantitative indicators, and updating the semantic reasoning model and graph optimization strategy using a reinforcement learning algorithm to ensure the continuous adaptive evolution of the system, and the effect evaluation indicators cover three dimensions of integrity, consistency and timeliness, with a total of 15 sub-indicators.

[0007] Preferably, the step 1 of constructing a dynamically evolving domain knowledge graph specifically comprises: using a sequence labeling model based on a bidirectional long short-term memory network to jointly extract entities and relationships from structured tables, semi-structured logs and unstructured documents; the entity extraction precision is not less than 92%, and the relationship extraction recall rate is not less than 88%; the time sequence perception graph neural network model adopts a combination structure of graph attention network and gated recurrent unit, the hidden layer dimension is 256, the number of attention heads is 8, and the dynamic evolution law of the relationship between entities can be captured.

[0008] Preferably, the probabilistic semantic reasoning in the step 2 adopts a probabilistic graph model based on a variational autoencoder, the latent variable dimension is 128, an uncertainty quantification module is introduced in the reasoning process, and the confidence score of each reasoning conclusion is output; the membership function of the fuzzy description logic adopts a Gaussian kernel function, and the bandwidth parameter is adaptively determined by maximum likelihood estimation, supporting accurate processing of boundary fuzzy semantics.

[0009] Preferably, the differentiable mapping relationship in step 3 is implemented by a multi-layer perception machine, the network structure is 256 dimensions for the input layer, 512 dimensions for the hidden layer, and 128 dimensions for the output layer, the activation function adopts a rectified linear unit, the gradient back propagation mechanism adopts a self-adaptive matrix estimation algorithm for parameter optimization, and a gradient clipping technology is introduced to prevent divergence in the training process.

[0010] Preferably, the strategy generation module in step 4 integrates a rule engine and a machine learning model, the rule engine supports more than 1000 pre-defined governance rules, the machine learning model adopts an extreme gradient boosting algorithm, the number of trees is 1000, and the maximum depth is 8; the strategy execution engine supports transactional operations to ensure the atomicity and consistency of data governance operations.

[0011] Preferably, the reinforcement learning algorithm in step 5 adopts a proximal policy optimization algorithm, the policy network and the value network are both 3-layer fully connected neural networks, and the number of neurons in each layer is 256, 128, and 64 respectively; the weights of the effect evaluation indicators are dynamically adjusted by an analytic hierarchy process to ensure the consistency of the evaluation results and business objectives.

[0012] Preferably, the method further includes a knowledge graph version management mechanism, a distributed storage system based on content addressing is adopted to save graph historical versions, the graph state at any time point can be traced back and compared and analyzed, the version storage capacity is not less than 100TB, and the data retrieval delay is less than 100 milliseconds.

[0013] Preferably, the semantic reasoning module integrates knowledge graph embedding and neural network reasoning, adopts a TransE algorithm to generate low-dimensional vector representations of entities and relationships, the vector dimension is 200, the scoring function adopts an L1 norm distance, and the number of negative samples is 50; the neural network reasoning part adopts a graph convolution network, the number of layers is 3, and the output dimension of each layer is 512, 256, and 128 respectively.

[0014] Preferably, the data quality evaluation indicators include statistical feature analysis of numerical indicators, semantic consistency detection of text indicators, and structure integrity verification of associated indicators, a total of 3 categories and 20 specific indicators, each indicator defines a quantitative calculation formula and a threshold range.

[0015] Preferably, the method is deployed in a distributed computing framework, adopts a master-slave architecture, the master node is responsible for task scheduling and state monitoring, the slave node executes specific data processing and calculation tasks, the cluster size can be elastically expanded to 1000 nodes, and the daily average processing data volume is not less than 10TB.

[0016] Compared with existing technologies, this invention has the following advantages: By constructing a dynamically evolving knowledge graph and integrating a probabilistic semantic reasoning mechanism, it achieves a deep understanding and accurate characterization of data semantics; by establishing a differentiable association between data quality indicators and graph topology, it forms a closed-loop optimization capability, significantly improving the automation level and decision reliability of data governance; by adopting a context-aware intelligent decision support and reinforcement learning feedback mechanism, it ensures that the system can continuously adapt to the dynamic changes in business semantics, effectively solving the systematic defects of traditional methods in semantic understanding, association analysis and adaptive optimization, and providing a reliable data governance solution for highly sensitive fields. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall technical architecture of the intelligent data governance method based on knowledge graphs and semantic reasoning proposed in this invention. Detailed Implementation

[0018] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] This invention proposes an intelligent data governance method based on knowledge graphs and semantic reasoning.

[0020] Reference Appendix Figure 1 The overall technical architecture of this invention includes a dynamic knowledge graph construction module, a probabilistic semantic reasoning and conflict detection module, a data quality and graph topology joint optimization module, a context-aware intelligent decision support module, and a governance effect evaluation and feedback learning module. These modules work collaboratively to form a complete closed-loop governance process from data input to policy execution and effect feedback.

[0021] In the aforementioned intelligent data governance method based on knowledge graphs and semantic reasoning, step (1), which constructs a dynamically evolving domain knowledge graph, aims to address the problem that traditional static ontology models cannot adapt to the dynamic evolution of business semantics. Specifically, step (1) extracts entities, attributes, and relationships from multi-source heterogeneous data sources, including structured tables, semi-structured logs, and unstructured documents. There are no fewer than 50 entity types, covering core business objects such as customers, products, transactions, contracts, equipment, sensors, events, locations, organizations, and personnel; and no fewer than 30 relationship types, including semantic relationships such as "belongs to," "associated with," "triggered," "dependent on," "affected," "contains," "located in," and "occurred in." The graph update cycle does not exceed 24 hours to ensure that knowledge representation is synchronized with business reality.

[0022] The specific implementation process of step (1) includes the following sub-operations: first, a sequence labeling model based on a bidirectional long short-term memory network is used to perform joint entity and relationship extraction on the original data. The model uses character-level and word-level dual-channel input, captures local and global semantic features through a shared bottom embedding layer, and uses a conditional random field decoder in the output layer to ensure the global optimality of the label sequence. The entity extraction accuracy is not less than 92%, and the relationship extraction recall rate is not less than 88%. Second, the extraction results are mapped to a unified domain ontology framework to generate an initial knowledge graph triple set. Subsequently, a graph neural network model based on time sequence perception is used to dynamically update the graph. The model uses a combination structure of graph attention network and gated recurrent unit, wherein the graph attention network is responsible for aggregating neighbor node information at each time step, the number of attention heads is 8, and the hidden layer dimension is 256; the gated recurrent unit maintains the historical memory of the node state and captures the dynamic evolution law of the relationship between entities. The model receives an incremental data stream in a sliding time window manner, completes full graph reconstruction and incremental update fusion every 24 hours, and ensures the timeliness and accuracy of the graph structure and semantic representation.

[0023] In the above intelligent data governance method based on knowledge graph and semantic reasoning, step (2) performs probabilistic semantic reasoning and conflict detection, aiming to overcome the defects of traditional rule engines in handling fuzzy semantics and uncertainty. Specifically, step (2) uses a deep semantic matching model with a fusion attention mechanism to calculate the semantic similarity between any two entities based on the knowledge graph constructed in step (1). The model encodes entity names, attribute values, and context descriptions into 512-dimensional vectors, interacts and compares semantic features through a multi-head cross-attention mechanism, and finally outputs a normalized similarity score. The semantic similarity calculation dimension is not less than 512 dimensions, ensuring fine-grained discrimination in high-dimensional semantic space.

[0024] In addition, step (2) uses fuzzy description logic for consistency checking. The membership function of fuzzy description logic uses a Gaussian kernel function, and the bandwidth parameter is adaptively determined by maximum likelihood estimation, thereby supporting accurate quantitative processing of boundary fuzzy semantics (such as "high-risk customers" and "slight abnormalities"). A probabilistic graph model based on a variational autoencoder is introduced in the reasoning process, with a latent variable dimension of 128, used to model the joint probability distribution of entity attributes and relationships. The uncertainty quantification module calculates the confidence score of each reasoning conclusion based on the posterior distribution variance. When the confidence is lower than a preset threshold (e.g., 0.7), it is marked as a pending manual review item. The conflict detection accuracy is greater than or equal to 95%, which can effectively identify potential conflicts (such as the contradiction between customer age and birth date) and redundancies (such as multiple IDs for the same device in different systems) in the data.

[0025] The core principle framework of probabilistic semantic reasoning and conflict detection includes a semantic embedding layer, an attention matching layer, a fuzzy logic verification layer, and an uncertainty quantification layer. The semantic embedding layer generates low-dimensional vector representations of entities and relationships using the TransE algorithm, with a vector dimension of 200, an L1 norm distance for the scoring function, and a negative sampling number of 50. The neural network reasoning part uses a 3-layer graph convolutional network with output dimensions of 512, 256, and 128 respectively for each layer, to propagate semantic information on the graph structure and enhance node representations. This integrated architecture balances the interpretability of symbolic reasoning and the generalization ability of neural reasoning.

[0026] In the above intelligent data governance method based on knowledge graph and semantic reasoning, step (3) implements joint optimization of data quality indicators and graph topology, aiming to establish a differentiable correlation between data quality and knowledge representation, and realize closed-loop adaptive optimization. Specifically, step (3) first defines a data quality evaluation index system, including statistical feature analysis of numerical indicators (such as mean shift and variance anomaly), semantic consistency detection of text indicators (such as term standardization and context coherence), and structure integrity verification of associated indicators (such as foreign key absence and loop anomaly), totaling 3 categories and 20 specific indicators. Each indicator defines a quantitative calculation formula and threshold range.

[0027] Subsequently, a differentiable mapping relationship is established between the data quality evaluation indicators and the knowledge graph node embedding vectors. This mapping relationship is implemented through a multi-layer perceptron, with a network structure of 256 dimensions for the input layer (corresponding to node embedding), 512 dimensions for the hidden layer, and 128 dimensions for the output layer (corresponding to quality indicator prediction values), and a rectified linear unit activation function. Through the gradient backpropagation mechanism, the gradient of the quality indicator loss with respect to the graph construction parameters (such as entity extraction model weights and graph neural network parameters) is calculated, and the adaptive moment estimation algorithm is used for parameter optimization with an initial learning rate of 0.001. To prevent divergence during training, gradient clipping technology is introduced to limit the gradient norm to within 1.0. The number of joint optimization iterations is set to 1000, and the graph construction model is updated each iteration to make the generated knowledge graph more consistent with the internal structure of high-quality data.

[0028] In the above intelligent data governance method based on knowledge graph and semantic reasoning, the step (4) is to perform intelligent decision support for context awareness, aiming to realize automatic conversion from reasoning results to governance actions. Specifically, the step (4) generates data governance strategy suggestions based on the reasoning results of step (2) and the optimized knowledge graph of step (3) by a strategy generation module. This module integrates a rule engine and a machine learning model: the rule engine supports more than 1000 pre-defined governance rules, covering data cleaning, standardization, desensitization, fusion and other scenarios; the machine learning model uses the extreme gradient boosting algorithm, with 1000 trees and a maximum depth of 8, to handle complex or edge cases not covered by rules. The response time of strategy generation is less than 2 seconds, meeting the real-time governance needs.

[0029] The generated strategy is automatically implemented by a strategy execution engine, and the operation types include data cleaning (such as correcting incorrect values, filling missing values), data fusion (such as entity disambiguation, record linking) and permission control (such as dynamically adjusting data access levels). The strategy execution engine supports transactional operations.

[0030] In the above intelligent data governance method based on knowledge graph and semantic reasoning, the step (5) is to realize governance effect evaluation and feedback learning, aiming to drive the system to continuously adapt and evolve. Specifically, the step (5) collects the data state changes after the strategy execution, and calculates the governance effect quantitative indicators. The effect evaluation indicators cover three dimensions of integrity (such as missing rate reduction ratio), consistency (such as conflict number reduction rate) and timeliness (such as data delay shortening degree), with a total of 15 sub-indicators. The indicator weights are dynamically adjusted by the analytic hierarchy process, and the importance of each dimension is redistributed according to the current business objectives (such as quarterly risk control focus), ensuring that the evaluation results are consistent with the business strategy.

[0031] Subsequently, the semantic reasoning model and the graph optimization strategy are updated using a reinforcement learning algorithm. The reinforcement learning algorithm uses the proximal policy optimization algorithm, and the policy network and value network are both 3-layer fully connected neural networks with 256, 128 and 64 neurons per layer, respectively. The state space includes the current graph topology features, data quality indicator values and historical strategy execution effects; the action space is the model hyperparameter adjustment instruction (such as learning rate scaling factor, attention head number increase / decrease); the reward function is defined as the weighted sum of governance effect indicators. Through interaction with the environment, the system continuously optimizes its internal models and improves long-term governance efficiency.

[0032] In addition, the present application also includes a knowledge graph version management mechanism. The mechanism uses a content-based distributed storage system (such as IPFS or similar architecture) to save the history versions of the graph, and each version is uniquely identified by a content hash value. The system supports graph state rollback and comparative analysis at any time point, such as comparing the trend of the number of "high-risk customer" nodes in this month and last month's graph. The version storage capacity is not less than 100TB, and the data retrieval delay is less than 100ms, meeting the large-scale historical tracing demand.

[0033] The present application is deployed in a distributed computing framework and adopts a master-slave architecture. The master node is responsible for task scheduling, state monitoring and resource coordination; the slave node executes specific data processing and computing tasks, including entity extraction, graph neural network training, semantic reasoning, etc. The cluster size can be flexibly expanded to 1000 nodes, and the daily processing data volume is not less than 10TB. The communication layer uses gRPC protocol, and the data serialization uses Protocol Buffers format, ensuring high throughput and low delay.

[0034] To specifically illustrate the implementation effect of the present application, an application example in a financial risk control scene is constructed. A certain bank generates about 5TB of transaction logs, customer files and external public opinion data every day. The system starts step (1) every morning, extracts entities and relationships from the core bank system, anti-fraud platform and news API, and constructs a dynamic knowledge graph containing 62 entity types (such as "credit card account", "POS machine", "merchant category code") and 35 relationship types. The graph update period is 22 hours.

[0035] In step (2), the system detects that the "residential address" and "frequent transaction location" of customer A have significant geographical deviation, the semantic similarity score is 0.32 (threshold 0.6), and the fuzzy logic verification shows that the "out-of-place transaction" membership degree is 0.85 and the confidence score is 0.91, which is judged as a potential fraud risk. Step (3) calculates the quality indicators of the customer's related data and finds that the "address update frequency" indicator is abnormal (lower than the historical mean by 2 standard deviations), and adjusts the attention weight of the address extraction model through gradient back propagation.

[0036] Step (4) generates a strategy: temporarily freeze the non-face-to-face transaction permission of the account, and trigger a manual review work order. The strategy is generated and executed within 1.3 seconds. Step (5) evaluates the effect after 24 hours: no fraud event occurs, customer satisfaction does not decrease, integrity index improves by 3.2%, and consistency index improves by 5.7%. The reinforcement learning module rewards the current strategy accordingly and fine-tunes the response threshold for similar scenarios in the future.

[0037] In another healthcare scenario, the hospital information system includes patient electronic medical records, laboratory reports, and drug inventory data. The system constructs a knowledge graph containing 58 entities (such as "diagnosis code," "generic drug name," and "laboratory test item"). Step (2) discovers that "warfarin" and "aspirin" are prescribed simultaneously in a patient's prescription. Although the semantic similarity is low, fuzzy logic verification shows that the risk membership degree of drug interaction reaches 0.93, and the conflict detection accuracy rate is 96.4%. The system automatically generates a medication conflict warning and suggests alternative solutions. After the intervention, the medication error rate decreased by 42%, and the timeliness index improved due to automatic interception.

[0038] The above examples demonstrate that this invention effectively addresses the systemic deficiencies of traditional data governance methods in semantic understanding, association analysis, and adaptive optimization through dynamic knowledge graphs, probabilistic semantic reasoning, and closed-loop optimization mechanisms.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent data governance method based on a knowledge graph and semantic reasoning, characterized in that: Comprise the following specific steps: Step 1: Construct a dynamically evolving domain knowledge graph, extract entities, attributes and relationships from multi-source heterogeneous data, and dynamically update the graph structure and semantic representation based on a time series perception graph neural network model; Step 2: Perform probabilistic semantic reasoning and conflict detection, calculate the semantic similarity between entities based on the knowledge graph using a deep semantic matching model with a fusion attention mechanism, and perform consistency checking using fuzzy description logic to identify potential conflicts and redundancies in the data; Step 3: Implement joint optimization of data quality indicators and graph topology, establish a differentiable mapping relationship between data quality evaluation indicators and knowledge graph node embedding vectors, and adaptively adjust the graph construction parameters through the gradient back propagation mechanism; Step 4: Perform context-aware intelligent decision support, generate data governance strategy recommendations based on the reasoning results and the optimized knowledge graph, and automatically implement data cleaning, fusion and permission control operations through a strategy execution engine; Step 5: Realize the evaluation and feedback learning of the governance effect, collect the data state changes after the strategy execution, calculate the governance effect quantitative indicators, and update the semantic reasoning model and graph optimization strategy using reinforcement learning algorithm to ensure the continuous adaptive evolution of the system.

2. The intelligent data governance method based on knowledge graph and semantic reasoning according to claim 1, characterized in that: The step 1 of constructing a dynamically evolving domain knowledge graph specifically comprises: using a sequence labeling model based on a bidirectional long short-term memory network to jointly extract entities and relationships from structured tables, semi-structured logs and unstructured documents.

3. The intelligent data governance method based on knowledge graph and semantic reasoning according to claim 1, characterized in that: The probabilistic semantic reasoning in step 2 uses a probabilistic graph model based on a variational autoencoder, and introduces an uncertainty quantification module in the reasoning process to output the confidence score of each reasoning conclusion; The membership function of the fuzzy description logic uses a Gaussian kernel function, and the bandwidth parameter is adaptively determined by maximum likelihood estimation. 4.The intelligent data governance method based on knowledge graph and semantic reasoning of claim 1, characterized in that: The differentiable mapping relationship in step 3 is realized by a multilayer perceptron with an input layer of 256 dimensions, a hidden layer of 512 dimensions and an output layer of 128 dimensions, and the activation function uses a rectified linear unit; The gradient back propagation mechanism uses the adaptive moment estimation algorithm for parameter optimization, and introduces the gradient clipping technology to prevent divergence in the training process. 5.The intelligent data governance method based on knowledge graph and semantic reasoning of claim 1, characterized in that: The strategy generation module in step 4 integrates a rule engine and a machine learning model, and the machine learning model uses an extreme gradient boosting algorithm. 6.The intelligent data governance method based on knowledge graph and semantic reasoning according to claim 1, characterized in that: The reinforcement learning algorithm in step 5 uses a proximal policy optimization algorithm, and the policy network and value network are both 3-layer fully connected neural networks with 256, 128 and 64 neurons respectively in each layer. 7.The intelligent data governance method based on knowledge graph and semantic reasoning according to claim 1, characterized in that: It also includes a knowledge graph version management mechanism, which uses a content-addressed distributed storage system to save the history versions of the graph, and supports graph state backtracking and comparative analysis at any time point. 8.The intelligent data governance method based on knowledge graph and semantic reasoning of claim 1, wherein: The semantic reasoning module integrates knowledge graph embedding and neural network reasoning, and uses the TransE algorithm to generate low-dimensional vector representations of entities and relationships. 9.The intelligent data governance method based on knowledge graph and semantic reasoning of claim 1, characterized in that: The data quality evaluation index includes statistical feature analysis of numerical index, semantic consistency detection of text index and structure integrity verification of correlation index. 10.The intelligent data governance method based on knowledge graph and semantic reasoning of claim 1, characterized in that: The method is deployed in a distributed computing framework, adopts master-slave architecture, the master node is responsible for task scheduling and state monitoring, and the slave node executes specific data processing and computing tasks.

Images