System design method and system based on distributed causal inference
By integrating the central control node, computing node and data storage module in the causal inference system, and using the causal forest algorithm for parallel calculation, the problem of high consumption and low efficiency of the traditional causal inference method is solved, and efficient and accurate causal inference is achieved .
Patent Information
- Application Number
- CN202411406956.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-02-11
AI Technical Summary
The traditional causal inference method relies on a centralized computing model, resulting in high consumption of computing resources and low efficiency, and high computational complexity and poor model adaptability when processing complex data structures and high-dimensional data.
Design a system based on distributed causal inference, and adopts causal forest by integrating the central control node, computing node and data storage module into a system to realize the automated process of task submission, distribution, data sharding, parallel computing and result summary. The algorithm performs causal inference calculations.
It significantly improves the computational efficiency of causal inference, can complete the causal inference task of large-scale data sets more quickly, reduces the computational pressure of a single node, improves the accuracy and stability of causal inference, and improves the scalability of the system and Adaptability.
Smart Images
Figure CN119415228A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed computing technology, and in particular to a system design method and system based on distributed causal inference. Background Art
[0002] Causal inference is an important topic in data science and is widely used in fields such as medicine, finance, and social sciences. However, traditional causal inference methods usually rely on centralized computing models, which not only require a lot of computing resources and storage space, but also have low efficiency when processing large-scale data sets. In addition, existing causal inference methods, such as structural equation modeling and Granger causality, often have problems such as high computational complexity and poor model adaptability when faced with complex data structures and high-dimensional data.
[0003] In recent years, with the development of big data technology, distributed computing methods have shown great advantages in processing large-scale data sets. However, effectively distributing causal inference tasks to multiple computing nodes for parallel computing still faces many challenges. Therefore, there is an urgent need for a new distributed causal inference system to improve the efficiency and accuracy of causal inference and to adapt to the needs of large-scale data sets. Summary of the invention
[0004] The purpose of the present invention is to provide a system design method and system based on distributed causal inference to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solution: a system design method based on distributed causal inference, the method comprising the following steps:
[0006] System integration: Integrate the central control node, computing node and data storage module into a complete system to realize the automated process of task submission, task distribution, data sharding, parallel computing and result aggregation;
[0007] Users submit causal inference tasks through the front-end interface. The system automatically distributes tasks to multiple computing nodes for parallel processing based on task characteristics. The computing nodes use the causal forest algorithm to perform causal inference calculations and return the results to the central control node for aggregation and integration, ultimately generating a causal inference report.
[0008] Preferably, the central control node is used to receive causal inference tasks submitted by users, distribute the tasks to multiple computing nodes for parallel processing according to the characteristics of the tasks, and summarize the calculation results of each node to obtain the final inference result;
[0009] Computing nodes are used to receive tasks and data distributed by the central control node, execute causal inference algorithms, and return results to the central control node;
[0010] Data storage is integrated into the central control node to manage data sets, including data reading, writing, sharding, and distribution to optimize storage efficiency and computing performance.
[0011] Preferably, task management is also included to achieve the following functions:
[0012] Task reception: receiving causal inference tasks submitted by users through the front-end interface;
[0013] Task distribution: Intelligently distribute tasks to multiple computing nodes for parallel processing based on the computational complexity and data volume of the tasks;
[0014] Task summary: Collect the calculation results returned by each computing node, summarize and integrate them, and generate the final causal inference report.
[0015] Preferably, it also includes computing nodes and algorithm execution, specifically realizing the following functions:
[0016] Data reception: receiving data shards and causal inference tasks from the central control node;
[0017] Causal inference: Use the causal forest algorithm to perform causal inference calculations on the received data shards;
[0018] Result return: The calculated causal inference results are returned to the central control node.
[0019] Preferably, the causal forest algorithm is also implemented, and the specific steps include:
[0020] Data preprocessing: Perform data cleaning and feature selection to prepare datasets for causal inference;
[0021] Tree model training: build multiple decision trees based on random sampling and decision tree algorithm, and calculate the average causal effect on the leaf nodes of each tree;
[0022] Model integration: Integrate the causal effect results of multiple decision trees to obtain the final causal effect estimate and calculate its confidence interval.
[0023] A system design system based on distributed causal inference, the system comprising:
[0024] A central control node to manage the task distribution, data storage and result aggregation of the entire system;
[0025] Multiple computing nodes communicate with the central control node to perform parallel computing tasks;
[0026] Among them, the central control node contains a task management module and a data storage module to support causal inference tasks submitted by users and realize effective management and distribution of data.
[0027] Preferably, the task management module has the following functions:
[0028] Task reception: receiving causal inference tasks submitted by users through the front-end interface, including data sets and causal inference model parameters;
[0029] Task distribution: Intelligently decompose tasks and distribute them to multiple computing nodes for parallel processing based on the computing requirements and data size of the tasks;
[0030] Task summary: Collect the calculation results from each computing node, summarize and integrate them to form the final causal inference report.
[0031] Preferably, the data storage module implements the following functions:
[0032] Data management: Store and manage the data sets required for causal inference, and provide interfaces for reading and writing data;
[0033] Data Sharding: Intelligently shard large-scale data sets to ensure that each computing node can process data evenly, reduce storage pressure, and improve computing efficiency.
[0034] Preferably, a calculation module is also included, specifically including:
[0035] Receive data sharding and causal inference tasks distributed by the central control node;
[0036] The causal forest algorithm is used to perform causal inference calculations on data shards, where the causal forest algorithm includes steps such as data preprocessing, tree model training, and model integration;
[0037] The calculation results are returned to the central control node for subsequent aggregation and integration.
[0038] Preferably, the following contents are also included:
[0039] After receiving the task, the central control node starts the task management module to distribute the task and partition the data;
[0040] Each computing node receives tasks and data shards, starts the computing module, and uses the causal forest algorithm for parallel computing;
[0041] After the calculation is completed, each computing node returns the result to the central control node;
[0042] The central control node collects all the results, summarizes and integrates them to form the final causal inference report, and displays it to the user through the front-end interface.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] The system design method and system based on distributed causal inference proposed in the present invention adopts distributed computing technology to distribute causal inference tasks to multiple computing nodes for parallel computing, which significantly improves the computing efficiency of causal inference. Compared with traditional centralized computing, it can complete causal inference tasks of large-scale data sets more quickly.
[0045] By distributing computing tasks to multiple computing nodes, we can fully utilize distributed computing resources, reduce the computing pressure of a single node, and reduce the consumption of computing resources.
[0046] The use of the causal forest algorithm for causal inference calculations can effectively process complex data structures and high-dimensional data, and improve the accuracy and stability of causal inference.
[0047] By adopting a distributed computing architecture, the system can flexibly expand computing nodes according to actual needs, thereby improving the scalability and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 This is a system architecture diagram of the present invention;
[0049] Figure 2 The flowchart of the computing node of the present invention;
[0050] Figure 3 This is the flow chart of the causal forest algorithm of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] For example, see Figures 1 to 3 The present invention provides a technical solution: a system design method based on distributed causal inference, the method comprising the following steps:
[0053] S1. System architecture design
[0054] S101, central control node
[0055] Provides task management module
[0056] Task reception: Receive causal inference tasks submitted by users, including data sets and causal inference models.
[0057] Task distribution: Distribute tasks to multiple computing nodes for parallel computing based on the computational complexity and data volume of the tasks.
[0058] Task summary: Collect the calculation results of each computing node, summarize and integrate them, and obtain the final causal inference results.
[0059] Provide data storage module
[0060] Data management: Stores the data sets required for causal inference and provides data reading and writing interfaces.
[0061] Data sharding: Shard large-scale data sets and distribute them to various computing nodes to reduce the storage pressure of a single node.
[0062] S102, computing node
[0063] Provide computing module
[0064] Data reception: Receive data shards and causal inference tasks distributed by the central control node.
[0065] Data processing: The Causal Forest algorithm is used to perform causal inference calculations on data shards.
[0066] Result return: The calculation results are returned to the central control node for summary.
[0067] S2. Causal Inference Algorithm Design
[0068] S201, Causal Forest Algorithm
[0069] The causal forest algorithm is a causal inference method based on random forest.
[0070] The main steps are as follows:
[0071] S20101. Data preprocessing
[0072] Data cleaning: remove missing values and outliers from the data.
[0073] Feature selection: A correlation-based feature selection method is used to select the features most relevant to the causal relationship.
[0074] S20102, tree model training
[0075] Random sampling: Randomly extract samples from the data set to construct a training set.
[0076] Decision tree construction: Use the decision tree algorithm to build multiple decision trees, and each tree is trained based on different samples and features.
[0077] Causal effect estimation: At the leaf node of each decision tree, the average causal effect (ATE) of the treatment group and the control group is calculated using the following formula:
[0078]
[0079] Among them, n t and n c are the number of samples in the treatment group and the control group, respectively, i and Y j These are the sample results for the treatment group and the control group, respectively.
[0080] S20103. Model Integration
[0081] Multi-tree ensemble: Integrate the causal effect results of multiple decision trees to obtain the final causal effect estimate.
[0082] Confidence interval calculation: The Bootstrap method is used to calculate the confidence interval of the causal effect estimate. The formula is as follows:
[0083]
[0084] in, is the estimated average causal effect, is the standard deviation, n is the sample size, is the critical value of the normal distribution.
[0085] S3. System workflow
[0086] S301, task submission: the user submits the causal inference task through the front-end interface, including the data set and causal inference model parameters.
[0087] S302, task distribution: the central control node distributes the task to multiple computing nodes for parallel computing according to the computational complexity and data volume of the task.
[0088] S303, data sharding: the central control node shards the large-scale data set and distributes it to each computing node to reduce the storage pressure of a single node.
[0089] S304, parallel computing, each computing node receives data shards and causal inference tasks, and uses the causal forest algorithm to perform parallel computing.
[0090] S305, result aggregation: the central control node collects the calculation results of each computing node, aggregates and integrates them, and obtains the final causal inference result.
[0091] Embodiment 2, based on embodiment 1, proposes a system design system based on distributed causal inference, the system comprising:
[0092] It includes a central control node and multiple computing nodes. The central control node is used to receive and manage causal inference tasks, distribute tasks to multiple computing nodes for parallel computing, and summarize and integrate the computing results. The multiple computing nodes are used to receive and process causal inference tasks, use the causal forest algorithm to perform causal inference calculations, and return the calculation results to the central control node for summary, so as to finally obtain the causal inference results.
[0093] The central control node also includes a data storage module for storing the data sets required for causal inference tasks and providing data reading and writing interfaces; multiple computing nodes receive the data shards and causal inference tasks distributed by the central control node, use the causal forest algorithm for parallel computing, and return the calculation results to the central control node for aggregation.
[0094] The causal forest algorithm includes data preprocessing steps, tree model training steps and model integration steps. Data preprocessing includes data cleaning and feature selection, tree model training includes random sampling and decision tree construction, and model integration includes multi-tree integration and confidence interval calculation.
[0095] The feature selection in data preprocessing adopts the feature selection method based on correlation. The random sampling in tree model training randomly extracts samples from the data set to construct a training set. The decision tree construction adopts the decision tree algorithm to construct multiple decision trees. The multi-tree integration in model integration integrates the causal effect results of multiple decision trees. The confidence interval calculation adopts the Bootstrap method to calculate the confidence interval of the causal effect estimate.
[0096] The central control node and multiple computing nodes communicate through a network connection. The central control node is responsible for task management and data storage, and the multiple computing nodes are responsible for parallel computing.
[0097] The decision tree used by the causal forest algorithm is a regression tree. The model is integrated according to the confidence interval of the causal effect estimate to obtain the final causal inference result.
[0098] The decision tree used by the causal forest algorithm is a classification tree. Causal effect estimation and model integration are performed based on the classification results to obtain the final causal inference result.
[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A system design method based on distributed causal inference, characterized by: The method comprises the following steps: System integration: Integrate the central control node, computing node and data storage module into a complete system to realize the automated process of task submission, task distribution, data sharding, parallel computing and result aggregation; Users submit causal inference tasks through the front-end interface. The system automatically distributes tasks to multiple computing nodes for parallel processing based on task characteristics. The computing nodes use the causal forest algorithm to perform causal inference calculations and return the results to the central control node for aggregation and integration, ultimately generating a causal inference report.
2. The system design method based on distributed causal inference according to claim 1, characterized in that: The central control node is used to receive causal inference tasks submitted by users, distribute tasks to multiple computing nodes for parallel processing according to task characteristics, and summarize the calculation results of each node to obtain the final inference result; Computing nodes are used to receive tasks and data distributed by the central control node, execute causal inference algorithms, and return results to the central control node; Data storage is integrated into the central control node to manage data sets, including data reading, writing, sharding, and distribution to optimize storage efficiency and computing performance.
3. The system design method based on distributed causal inference according to claim 2, characterized in that: It also includes task management, which enables the following functions: Task reception: receiving causal inference tasks submitted by users through the front-end interface; Task distribution: Intelligently distribute tasks to multiple computing nodes for parallel processing based on the computational complexity and data volume of the tasks; Task summary: Collect the calculation results returned by each computing node, summarize and integrate them, and generate the final causal inference report.
4. The system design method based on distributed causal inference according to claim 3 is characterized in that: It also includes computing nodes and algorithm execution, which specifically implement the following functions: Data reception: receiving data shards and causal inference tasks from the central control node; Causal inference: Use the causal forest algorithm to perform causal inference calculations on the received data shards; Result return: The calculated causal inference results are returned to the central control node.
5. The system design method based on distributed causal inference according to claim 1, characterized in that: It also includes the implementation of the causal forest algorithm, and the specific steps include: Data preprocessing: Perform data cleaning and feature selection to prepare datasets for causal inference; Tree model training: build multiple decision trees based on random sampling and decision tree algorithm, and calculate the average causal effect on the leaf nodes of each tree; Model integration: Integrate the causal effect results of multiple decision trees to obtain the final causal effect estimate and calculate its confidence interval.
6. A system design system based on distributed causal inference according to the system design method based on distributed causal inference according to any one of claims 1 to 5, characterized in that: The system comprises: A central control node to manage the task distribution, data storage and result aggregation of the entire system; Multiple computing nodes communicate with the central control node to perform parallel computing tasks; Among them, the central control node contains a task management module and a data storage module to support causal inference tasks submitted by users and realize effective management and distribution of data.
7. The system design system based on distributed causal inference according to claim 6, characterized in that: The task management module has the following functions: Task reception: receiving causal inference tasks submitted by users through the front-end interface, including data sets and causal inference model parameters; Task distribution: Intelligently decompose tasks and distribute them to multiple computing nodes for parallel processing based on the computing requirements and data size of the tasks; Task summary: Collect the calculation results from each computing node, summarize and integrate them to form the final causal inference report.
8. The system design system based on distributed causal inference according to claim 6, characterized in that: The data storage module implements the following functions: Data management: Store and manage the data sets required for causal inference, and provide interfaces for reading and writing data; Data Sharding: Intelligently shard large-scale data sets to ensure that each computing node can process data evenly, reduce storage pressure, and improve computing efficiency.
9. The system design system based on distributed causal inference according to claim 6, characterized in that: It also includes a computing module, including: Receive data sharding and causal inference tasks distributed by the central control node; The causal forest algorithm is used to perform causal inference calculations on data shards, where the causal forest algorithm includes steps such as data preprocessing, tree model training, and model integration; The calculation results are returned to the central control node for subsequent aggregation and integration.
10. A system design system based on distributed causal inference according to claim 9, characterized in that: Also included are the following: After receiving the task, the central control node starts the task management module to distribute the task and partition the data; Each computing node receives tasks and data shards, starts the computing module, and uses the causal forest algorithm for parallel computing; After the calculation is completed, each computing node returns the result to the central control node; The central control node collects all the results, summarizes and integrates them to form the final causal inference report, and displays it to the user through the front-end interface.