Interpretable Sparse Multi-Expert System
By introducing a multi-layer linear expert model and feature selection gating network into the multi-expert system, feature selection and normalization processing are realized, and the problem of complex and difficult to explain the existing multi-expert system structure is solved, and the interpretability and transparency of the system are improved.
Patent Information
- Application Number
- CN202411834675.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-08
AI Technical Summary
The existing multi-expert systems perform well in natural language processing, computer vision and other fields, but their structure is complex and difficult to explain, and it is difficult for ordinary users to understand the working principle of the system.
A multi-expert system of interpretability sparse multi-experts is designed to realize feature selection and normalization processing through multi-layer linear expert model and feature selection gating network, reducing model complexity and improving model interpretability.
The input features are selected and normalized through feature selection gating network, so that the importance of each feature can be quantified and intuitively explained, improving the transparency of the system and user trust, and reducing model complexity and computing resource consumption.
Smart Images
Figure CN119783727A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of pattern recognition, and in particular to an interpretable sparse multi-expert system. Background Art
[0002] A Mixture-of-Experts (MoE) system is an intelligent system composed of multiple "experts", each of which is responsible for processing a specific type of data or task. When input information enters the system, the system will select the most appropriate expert to process it based on the characteristics of the information, thereby improving accuracy.
[0003] The existing MoE system is useful in many areas, but there is a problem:
[0004] The construction of multi-expert technology is based on the idea of Divide-and-Conquer. The entire problem sample space is divided in the multi-expert model, where each expert is responsible for the local sample space. After the MoE training is completed, for new samples, the gating network decides which expert is used to solve the problem, and finally obtains the output through the weighted average strategy.
[0005] MoEs perform well in natural language processing, computer vision, and multimodal fields, and have promoted the development of computer vision and multimodal fields. However, they are usually complex in structure and difficult to explain, making it difficult for ordinary users to understand how the system works. Summary of the invention
[0006] The purpose of the present invention is to design a new multi-expert system with feature selection function, so that it can effectively select key features and eliminate useless features, making the system simpler and easier to understand.
[0007] The technical solution of the present invention is: providing an interpretable sparse multi-expert system, the multi-expert system comprising: a multi-layer linear expert model, a feature selection gating network and an output node;
[0008] The multi-layer linear expert model is used to receive multi-layer input samples and calculate the output of each input sample. Each input sample is processed by multiple separate expert systems to obtain the corresponding output result. The multi-layer linear expert model includes multiple input nodes and the same number of middle-layer nodes. All input nodes of the multi-layer linear expert model are connected to all middle-layer nodes respectively, and each middle-layer node represents a separate expert system. The output result of the middle-layer node is calculated based on the linear weight and bias term of the input sample.
[0009] The feature selection gating network has the same number of input nodes and feature selection layer nodes as the input nodes of the multi-layer linear expert model, all input nodes of the feature selection gating network are respectively connected to all feature selection layer nodes, the feature selection layer nodes of the feature selection gating network are connected to the middle layer nodes with the same number in the multi-layer linear expert model, and the feature selection gating network receives the same multi-layer input sample and calculates the normalized weights of different dimensions of the multi-layer input sample;
[0010] The output node is connected to the feature selection gating network and the multi-layer linear expert model, and is used to calculate the final output of the multi-expert system according to the output and normalized weights of the multi-layer input signal;
[0011] The multi-expert system adjusts the feature selection parameters by iteratively optimizing the error function to reduce the error of the final output and eliminate irrelevant features, thereby reducing the complexity of the model and improving the interpretability of the model.
[0012] In any of the above technical solutions, further, the calculation formula of the final output y(x) of the multi-expert system is:
[0013]
[0014] Where x=(x1,x2,…,x t ) T ∈R t Represents multiple layers of input samples, x1,x2,…,x t Represents the different dimensions of multi-layer input samples, y i (x) represents the output result of the i-th (i=1,2,…,n) individual expert system for multi-layer input samples in the multi-layer linear expert model. is the normalized weight corresponding to the i-th individual expert system.
[0015] In any of the above technical solutions, further, the output result y of the i-th individual expert system for the multi-layer input sample in the multi-layer linear expert model is i The expression of (x) is:
[0016]
[0017] Among them, w i,l is the dimension x of the input sample l The linear weight (l=1,2,…,t) in the ith individual expert system is determined by each individual expert system, w i,0 is the bias term of the i-th individual expert system.
[0018] In any of the above technical solutions, further, the normalized weight of the i-th individual expert system The expression is:
[0019]
[0020] Among them, f i (x) is the output of the i-th feature selection layer node of the feature selection gating network;
[0021]
[0022] c i,l represents the lth element of the i-th cluster center, b l Represents the feature selection parameters, which are used to construct the feature selection error function.
[0023] In any of the above technical solutions, further, the expression of the error function E is:
[0024]
[0025] Among them, j represents the jth input sample, J is the total number of training samples, o represents the corresponding ideal output, and λ>0 represents the penalty factor coefficient.
[0026] In any of the above technical solutions, further, the iterative optimization step of the error function includes:
[0027] The system initializes the features of the input data through a clustering algorithm and assumes that all features are "bad features", that is, their feature selection parameters are close to 0; the clustering algorithm is used to initialize the center value and width value of the Gaussian function. By default, all features are useless. With iterative updates, the width value gradually increases to identify important features;
[0028] Then, the feature selection parameters are gradually updated through an iterative optimization method; as the optimization proceeds: for important features, the values of their corresponding feature selection parameters will gradually increase and stabilize at a non-zero value; for irrelevant features, the values of their feature selection parameters will be gradually compressed to 0 by the regularization term.
[0029] The beneficial effects of the present invention are:
[0030] The technical solution of the present invention selects and normalizes input features through a feature selection gating network, so that the importance of each feature can be quantified and intuitively explained, thereby effectively solving problems that are difficult to explain in existing multi-expert systems. Users can clearly understand the operation process of the system, the contribution of each feature to the output results, and the basis for decision-making, which greatly improves the transparency of the system and user trust.
[0031] By introducing a feature selection error function, the present invention enables the system to automatically eliminate useless or irrelevant features, thereby reducing model complexity, optimizing computational efficiency, and improving the stability and generalization ability of the model when processing high-dimensional data; and by optimizing the error function, the system not only reduces the prediction error but also maintains the integrity of important features, so that the final output result is closer to the ideal value, which helps to achieve higher prediction performance in complex tasks.
[0032] The combination of the gating network and the multi-layer linear expert model reduces unnecessary computing operations through a sparse feature selection strategy, effectively reduces system resource consumption, and speeds up reasoning.
[0033] In the preferred implementation of the present invention, the design proposed by the present invention can be adjusted and optimized according to specific application requirements, including feature selection methods, error function forms and types of expert models, and has good adaptability and scalability; it can be applied to natural language processing, computer vision, multimodal analysis and other fields, especially in application scenarios with high requirements for model interpretation capabilities, it has significant advantages, such as medical diagnosis, financial risk control and legal analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The advantages of the above and additional aspects of the present invention will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:
[0035] Figure 1 is a schematic diagram of the structure of an interpretable sparse multi-expert system according to an embodiment of the present invention;
[0036] Figure 2 Schematic diagram of the structure of a feature selection gating network of an interpretable sparse multi-expert system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0039] like Figure 1 and Figure 2 As shown, this embodiment provides an interpretable sparse multi-expert system, which includes: a multi-layer linear expert model, a feature selection gating network and an output node.
[0040] The multi-layer linear expert model includes multiple input nodes and intermediate layer nodes with the same number as the input nodes. All input nodes of the multi-layer linear expert model are respectively connected to all intermediate layer nodes, and each intermediate layer node represents a separate expert system.
[0041] The feature selection gating network has the same number of input nodes and feature selection layer nodes as the input nodes of the multi-layer linear expert model, and all input nodes of the feature selection gating network are respectively connected to all feature selection layer nodes.
[0042] The feature selection gating network and the multi-layer linear expert model both receive the same multi-layer input samples. The feature selection layer nodes of the feature selection gating network are sequentially connected to the middle layer nodes with the same number in the multi-layer linear expert model, and all the feature selection layer nodes of the feature selection gating network are connected to the output nodes.
[0043] The multi-layer linear expert model calculates the outputs of different dimensions of multi-layer input samples, the feature selection gating network calculates the normalized weights of different dimensions of multi-layer input samples, and the output node obtains the outputs and normalized weights of the multi-layer input signals and calculates the final output of the multi-expert system.
[0044] The calculation formula of the final output y(x) of the multi-expert system is:
[0045]
[0046] Where x=(x1,x2,…,x t ) T ∈R t Represents multiple layers of input samples, x1,x2,…,x t Represents the different dimensions of multi-layer input samples, y i (x) represents the output result of the i-th (i=1,2,…,n) individual expert system for multi-layer input samples in the multi-layer linear expert model:
[0047]
[0048] Among them, w i,l is the dimension x of the input sample l The linear weight (l=1,2,…,t) in the ith individual expert system is determined by each individual expert system, w i,0 is the bias term of the i-th individual expert system.
[0049] is the normalized weight corresponding to the i-th individual expert system, and its expression is as follows:
[0050]
[0051] In order to reduce the complexity of the model, the present invention improves the gating network, and the output of the i-th feature selection layer node of the feature selection gating network is c i,l represents the lth element of the i-th cluster center, b l Represents the feature selection parameters, which are used to construct the feature selection error function.
[0052] The present invention also designs the following error function E to adjust the final output y(x) of the multi-expert system:
[0053]
[0054] Among them, j represents the jth input sample, J is the total number of training samples, o represents the corresponding ideal output, and λ>0 represents the penalty factor coefficient.
[0055] By optimizing the error function E, the system can achieve the following two goals at the same time:
[0056] Reduce the prediction error and make the final output y(x) of the multi-expert system closer to the ideal output;
[0057] Automatically remove irrelevant features to improve the sparsity and interpretability of the system.
[0058] First, the system initializes the features of the input data through a clustering algorithm, and assumes that all features are "bad features" (that is, their feature selection parameters are close to 0). The clustering algorithm is used to initialize the center value and width value of the Gaussian function. By default, all features are useless. With iterative updates, the width value gradually increases to identify important features.
[0059] Then, the feature selection parameters are gradually updated through the iterative optimization method; as the optimization proceeds: for important features, the values of their corresponding feature selection parameters will gradually increase and stabilize at a non-zero value; for irrelevant features, the values of their feature selection parameters will be gradually compressed to 0 by the regularization term.
[0060] The design of the entire system makes it no longer a "black box" when it operates, but an explainable tool. Users can understand the operation process of each step, the importance of each feature, and why the system makes a certain decision. This transparency is particularly suitable for application scenarios that require precise explanation and judgment.
[0061] In summary, the present invention proposes an interpretable sparse multi-expert system, including: a multi-layer linear expert model, a feature selection gating network and an output node.
[0062] The multi-layer linear expert model is used to receive multi-layer input samples and calculate the output of each input sample. Each input sample is processed by multiple separate expert systems to obtain the corresponding output result; the multi-layer linear expert model includes multiple input nodes and the same number of intermediate layer nodes. All input nodes of the multi-layer linear expert model are respectively connected to all intermediate layer nodes, and each intermediate layer node represents a separate expert system; the output result of the intermediate layer node is calculated based on the linear weight and bias term of the input sample.
[0063] The feature selection gating network has the same number of input nodes and feature selection layer nodes as the input nodes of the multi-layer linear expert model. All input nodes of the feature selection gating network are respectively connected to all feature selection layer nodes. The feature selection layer nodes of the feature selection gating network are connected to the middle layer nodes with the same number in the multi-layer linear expert model. The feature selection gating network receives the same multi-layer input sample and calculates the normalized weights of different dimensions of the multi-layer input sample.
[0064] The output node is connected to the feature selection gating network and the multi-layer linear expert model to calculate the final output of the multi-expert system according to the output and normalized weights of the multi-layer input signal.
[0065] The multi-expert system adjusts the feature selection parameters by iteratively optimizing the error function to reduce the error of the final output and eliminate irrelevant features, thereby reducing the complexity of the model and improving the interpretability of the model.
[0066] The steps in the present invention can be adjusted in order, combined or deleted according to actual needs.
[0067] The units in the device of the present invention can be combined, divided and deleted according to actual needs.
[0068] Although the present invention is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present invention. The scope of protection of the present invention is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present invention.
Claims
1. Interpretable sparse multi-expert system, characterized by: The multi-expert system includes: a multi-layer linear expert model, a feature selection gating network and an output node; The multi-layer linear expert model is used to receive multi-layer input samples and calculate the output of each input sample, and each input sample is processed by multiple separate expert systems to obtain a corresponding output result; the multi-layer linear expert model includes multiple input nodes and the same number of middle-layer nodes, all input nodes of the multi-layer linear expert model are respectively connected to all middle-layer nodes, and each middle-layer node represents a separate expert system; the result output by the middle-layer node is calculated based on the linear weight and bias term of the input sample; The feature selection gating network has the same number of input nodes and feature selection layer nodes as the input nodes of the multi-layer linear expert model, all the input nodes of the feature selection gating network are respectively connected to all the feature selection layer nodes, the feature selection layer nodes of the feature selection gating network are connected to the middle layer nodes with the same number in the multi-layer linear expert model, and the feature selection gating network receives the same multi-layer input sample and calculates the normalized weights of different dimensions of the multi-layer input sample; The output node is connected to the feature selection gating network and the multi-layer linear expert model, and is used to calculate the final output of the multi-expert system according to the output of the multi-layer input signal and the normalized weight; The multi-expert system adjusts each feature selection parameter by iteratively optimizing the error function to reduce the error of the final output and eliminate irrelevant features, thereby reducing the complexity of the model and improving the interpretability of the model.
2. The interpretable sparse multi-expert system according to claim 1, characterized in that: The calculation formula of the final output y(x) of the multi-expert system is: Where x=(x1,x2,…,x t ) T ∈R t Represents multiple layers of input samples, x1,x2,…,x t Represents the different dimensions of multi-layer input samples, y i (x) represents the output result of the i-th (i=1,2,…,n) individual expert system for multi-layer input samples in the multi-layer linear expert model. is the normalized weight corresponding to the i-th individual expert system.
3. The interpretable sparse multi-expert system according to claim 2, characterized in that: The output result y of the i-th individual expert system for the multi-layer input samples in the multi-layer linear expert model i The expression of (x) is: Among them, w i,l is the dimension x of the input sample l The linear weight (l=1,2,…,t) in the ith individual expert system is determined by each individual expert system, w i,0 is the bias term of the i-th individual expert system.
4. The interpretable sparse multi-expert system according to claim 2, characterized in that: The normalized weight of the i-th individual expert system The expression is: Among them, f i (x) is the output of the i-th feature selection layer node of the feature selection gating network; c i,l represents the lth element of the i-th cluster center, b l Represents the feature selection parameters, which are used to construct the feature selection error function.
5. The interpretable sparse multi-expert system according to claim 4, characterized in that: The expression of the error function E is: Among them, j represents the jth input sample, J is the total number of training samples, o represents the corresponding ideal output, and λ>0 represents the penalty factor coefficient.
6. The interpretable sparse multi-expert system according to claim 5, characterized in that: The iterative optimization step of the error function comprises: The system initializes the features of the input data through a clustering algorithm and assumes that all features are "bad features", that is, their feature selection parameters are close to 0; the clustering algorithm is used to initialize the center value and width value of the Gaussian function. By default, all features are useless. With iterative updates, the width value gradually increases to identify important features; Then, the feature selection parameters are gradually updated through an iterative optimization method; as the optimization proceeds: for important features, the values of their corresponding feature selection parameters will gradually increase and stabilize at a non-zero value; for irrelevant features, the values of their feature selection parameters will be gradually compressed to 0 by the regularization term.