Key factor screening method based on attention mechanism and Gaussian process regression mixing
By combining attention mechanism with Gaussian process regression and combined with Sobol method, the problem of accurate and low efficiency of extraction of key experimental factors in the prior art is solved, and the accuracy of sensitivity analysis under different data volume conditions is achieved. and efficiency improvement.
Patent Information
- Application Number
- CN202411378528.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, the extraction of key experimental factors is low accuracy and low efficiency, especially when the data volume is insufficient, the overfitting of neural networks leads to large errors in the sensitivity calculation results, and the calculation efficiency of Sobol analysis method is low.
A key factor screening method based on the mixing of attention mechanism and Gaussian process regression is proposed. By constructing a support vector set and an attention mechanism-based neural network, combining Gaussian process regression and Sobol method, the best sensitivity analysis method is selected to improve extraction. Efficiency and accuracy.
The best sensitivity analysis method can be selected based on the amount of experimental data, which improves the accuracy and efficiency of sensitivity analysis under small sample conditions, and screens out key experimental factors.
Smart Images
Figure CN119442166A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data analysis, and in particular relates to a key factor screening method based on a mixture of attention mechanism and Gaussian process regression. Background Art
[0002] The factors that can affect the experimental results in the experiment are called experimental factors. Among all the experimental factors, the factors that have a significant impact on the experimental results are called experimental key factors. The purpose of extracting experimental key factors is to screen out the experimental factors that have a greater impact on the experimental results from many experimental factors, optimize the experimental design, reduce the number of experiments, and improve the experimental efficiency. When designing the experimental plan, non-critical factors are usually not considered or kept constant. For key factors, their values should be traversed as much as possible to fully obtain the experimental result data.
[0003] The extraction of experimental key factors is essentially to analyze the sensitivity of experimental results to various experimental factors, and to classify experimental factors with higher sensitivity as key factors. The sensitivity analysis method based on neural network is a commonly used sensitivity analysis method. The principle of this method is to use the ability of neural network nonlinear function approximation to fit the mapping relationship between experimental results and experimental factors, and to obtain the partial derivative of network output to input as the sensitivity of experimental factors. The advantage of this method is that the sensitivity calculation is simple, and the sensitivity of experimental factors can be obtained more accurately under the condition of sufficient data. However, under the condition of insufficient data, the overfitting of neural network leads to large errors in sensitivity calculation results. Sobol analysis is another commonly used sensitivity analysis method. This method calculates the ratio of the variance of experimental results to the total variance of experimental results when each experimental factor is changed separately. This method can comprehensively reflect the influence of each experimental factor on the experimental results, but this method has the problem of low computational efficiency. Summary of the invention
[0004] Aiming at the technical problems of low accuracy and low efficiency in extracting key factors of experiments in the prior art, the present invention proposes a key factor screening method based on attention mechanism and Gaussian process regression.
[0005] The first aspect of the present invention proposes a key factor screening method based on a mixture of attention mechanism and Gaussian process regression, the method comprising:
[0006] Step S1, constructing an experimental observation data set and setting a data volume threshold;
[0007] Step S2, selecting a sensitivity analysis method according to the amount of experimental observation data;
[0008] Step S3, constructing a support vector set based on the experimental observation data set;
[0009] Step S4: constructing a neural network based on an attention mechanism using a support vector set, training the neural network and calculating sensitivity;
[0010] Step S5: construct a Gaussian process regression model using experimental observation data and perform sample expansion;
[0011] Step S6, calculating the sensitivity using the Sobol method based on the expanded experimental observation data;
[0012] Step S7: setting a sensitivity threshold and screening out key experimental factors.
[0013] In step S1, the values of all experimental factors in the experiment and the corresponding experimental results are taken as one piece of observation data, and all experimental data constitute the experimental observation data set The data volume threshold is set to N thres .
[0014] In step S2, the sample size N of the time observation data in the statistical experimental observation data set is obs , and the data volume threshold is N thres For comparison; if the sample size N obs Greater than the data volume threshold N thres , then select the sensitivity analysis method based on the attention mechanism; otherwise, select the sensitivity analysis method based on Gaussian process regression.
[0015] In step S3, the number of support vectors n is set c , use K-means algorithm to analyze the experimental observation data set Clustering is performed, the number of clusters is the number of support vectors, and the experimental data closest to the cluster center in each cluster is selected as the support vector. All support vectors constitute the support vector set
[0016] In step S4:
[0017] The support vector is concentrated on the experimental factor value part X s As the key vector, the support vector set [X s , Y s ] as the value vector, construct a neural network based on the attention mechanism, and calculate the forward propagation result of the neural network:
[0018]
[0019] Where: MLP(·) represents a feedforward neural network; MHA(·) represents a multi-head attention mechanism;
[0020] The calculation result of the multi-head attention mechanism is:
[0021]
[0022] Among them: || represents the splicing operation; W o represents the weight matrix; denote the query vector weight, key vector weight, and value vector weight of the kth attention head respectively; d K represents the dimension of the key vector space; represents the data Z after splicing the experimental factor values and experimental results in the experimental observation data s =[X s , Y s ];
[0023] The calculation result of the feedforward neural network is:
[0024]
[0025] in: represents the network weight; represents the network bias;
[0026] Using experimental observation dataset The neural network is trained, and the partial derivative of the neural network output to the input is calculated as the sensitivity S of each experimental factor:
[0027]
[0028]
[0029] In step S5:
[0030] Using experimental observation datasets Construct a Gaussian process regression model and set the number of experiments to be expanded to n a , generate experimental factor values according to the sampling strategy Using Gaussian process regression model to predict experimental results The prediction process is:
[0031]
[0032] A=K(X s ,X s )+σ 2 I
[0033]
[0034] The experimental data Expanding to experimental observation datasets The expanded experimental observation data
[0035] In step S6, for the expanded experimental observation data Combined with the Gaussian process regression model, the Sobol method is used to calculate the sensitivity of each experimental factor. The implementation steps of the Sobol method include:
[0036] The expanded experimental observation data Divided equally into two groups, And calculate the overall variance of the experimental results and the first-order partial variance of each experimental factor:
[0037]
[0038] Among them: f(·) represents the nonlinear mapping relationship between experimental results and experimental factors, that is,
[0039] The sensitivity of each experimental factor is:
[0040]
[0041] In step S7, a sensitivity threshold α is set, and experimental factors with a sensitivity greater than the threshold α are selected as key factors.
[0042] The second aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the key factor screening method based on the mixture of attention mechanism and Gaussian process regression described in the first aspect of the present disclosure are implemented.
[0043] The third aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the key factor screening method based on the mixture of attention mechanism and Gaussian process regression described in the first aspect of the present disclosure are implemented.
[0044] In summary, the beneficial technical effects brought about by the technical solution proposed in the present invention include: (1) being able to select the best sensitivity analysis method based on the amount of experimental data; and (2) being able to provide relatively accurate sensitivity analysis results under small sample conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0046] Figure 1Schematic diagram of the process of a key factor screening method based on attention mechanism and Gaussian process regression according to an embodiment of the present invention.
[0047] Figure 2 A neural network structure diagram based on an attention mechanism according to an embodiment of the present invention
[0048] Figure 3 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0050] The present invention proposes a key factor screening scheme based on attention mechanism and Gaussian process regression. Figure 1 As shown in the figure, first, a suitable sensitivity analysis method is selected according to the amount of experimental data obtained. When the amount of data is sufficient, a deep neural network based on the attention mechanism is constructed. Then, the neural network is trained using the experimental data and the sensitivity of the experimental factors is calculated by taking partial derivatives. When the amount of data is insufficient, a Gaussian process regression model is constructed. Then, the Gaussian process regression model is used to expand the experimental data. Then, the Sobol sensitivity analysis method is used to calculate the sensitivity of the experimental factors. Finally, the manually set threshold is used to screen out the key factors of the experiment according to the sensitivity of the experimental factors.
[0051] The first aspect of the present invention proposes a key factor screening method based on a mixture of attention mechanism and Gaussian process regression, the method comprising:
[0052] The first step is to construct an experimental observation data set and set a data volume threshold.
[0053] In the experiment, the values of all experimental factors and the corresponding experimental results are taken as one piece of observation data. All experimental data constitute the experimental observation data set. Manually set the data volume threshold N thres .
[0054] The second step is to select an appropriate sensitivity analysis method based on the amount of experimental observation data.
[0055] Sample size N of statistical experimental observation data set obs and the manually set number threshold N thresFor comparison, if the sample size N obs Greater than the threshold N thres Then select the sensitivity analysis method based on the attention mechanism. If N obs Less than the threshold N thres Then the sensitivity analysis method based on Gaussian process regression is selected.
[0056] The third step is to construct a support vector set based on the experimental observation dataset.
[0057] Set the number of support vectors n c , use K-means algorithm to analyze the experimental observation data set Clustering is performed, and the number of clusters is the number of support vectors set manually. The experimental data closest to the cluster center in each cluster is selected as the support vector. All support vectors constitute the support vector set
[0058] In the fourth step, the support vector set is used to build a neural network based on the attention mechanism, train the neural network and calculate the sensitivity.
[0059] The support vector is concentrated on the experimental factor value part X s As the key vector, the support vector set [X s , Y s ] as the value vector, construct a neural network based on the attention mechanism as follows Figure 2 As shown, the forward propagation result of the neural network is calculated according to formula (1).
[0060]
[0061] Where: MLP(·) represents a feedforward neural network; MHA(·) represents a multi-head attention mechanism.
[0062] According to formula (2) and formula (3), we can get the calculation results of the multi-head attention mechanism.
[0063]
[0064]
[0065] Among them: || represents the splicing operation; W o represents the weight matrix; denote the query vector weight, key vector weight, and value vector weight of the kth attention head respectively; d K represents the dimension of the key vector space; represents the data Z after splicing the experimental factor values and experimental results in the experimental observation data s =[X s , Y s ].
[0066] According to formula (4), the calculation result of the feedforward neural network can be obtained:
[0067]
[0068] in: represents the network weight; Indicates the network bias.
[0069] Using Observational Data Train the neural network and calculate the partial derivative of the neural network output with respect to the input according to equations (5) and (6) as the sensitivity S of each experimental factor.
[0070]
[0071] The fifth step is to use the experimental observation data to build a Gaussian process regression model and perform sample expansion.
[0072] Using experimental observation dataset Construct a Gaussian process regression model and set the number of experiments to be expanded n a , using a certain sampling strategy to generate experimental factor values Then use the Gaussian process regression model to predict the experimental results The prediction process can be calculated according to equations (7) to (11).
[0073]
[0074]
[0075] A=K(X s ,X s )+σ 2 I (10)
[0076]
[0077] The experimental data Expand to The experimental observation data after expansion is formed
[0078] The sixth step is to calculate the sensitivity using the Sobol method based on the expanded experimental observation data.
[0079] For experimental observation data Combined with the Gaussian process regression model, the Sobol method is used to calculate the sensitivity of each experimental factor. The steps of the Sobol method are as follows:
[0080] Will Divided into 2 groups Calculate the overall variance of the experimental results and the first-order partial variance of each experimental factor (Formula (12)-(13)):
[0081]
[0082] Among them: f(·) represents the nonlinear mapping relationship between experimental results and experimental factors, that is, The sensitivity of each experimental factor is:
[0083]
[0084] Step 7: Set the sensitivity threshold to screen out the key factors of the experiment.
[0085] The sensitivity threshold α is set manually, and the experimental factors with sensitivity greater than the threshold α are selected as key factors.
[0086] Specific examples
[0087] like Figure 1 As shown, taking the sensitivity analysis of a simulation experiment result and the screening of key factors as an example, the specific implementation steps and specific operation process of the present invention are introduced:
[0088] Step 100: construct an experimental observation data set and set a data volume threshold;
[0089] A simulation experiment contains 8 experimental factors x1, x2, ..., x8. The value of each experimental factor is The sample size N was constructed obs =100 and N obs =1000 Experimental observation sample sets under two conditions and Part of the data of the two experimental observation sample sets are shown in Table 1 and Table 2. Set the data volume threshold N thres =200.
[0090] Table 1: Some experimental observation data
[0091]
[0092] Table 2: Some experimental observation data
[0093]
[0094]
[0095] Step 200: Select an appropriate sensitivity analysis method according to the amount of experimental observation data;
[0096] Experimental observation data set The number of samples is greater than N thres Therefore, for the experimental observation dataset Use the key factor extraction method based on the attention mechanism; experimental observation data set The number of samples is less than N thres Therefore, for the experimental observation dataset A key factor extraction method based on Gaussian process regression was used.
[0097] Step 300: construct a support vector set based on the experimental observation data set;
[0098] Set the number of support vectors n c = 10, the experimental observation data set is analyzed by Kmeans algorithm The support vector set obtained by clustering is shown in Table 3.
[0099] Table 3: Support vector set
[0100]
[0101] Step 400: Use the support vector set to construct a neural network based on the attention mechanism, train the neural network and calculate the sensitivity;
[0102] For the attention-based neural network, the key vector and value vector space dimensions are set to d K =10,d V = 12, and the number of attention heads is set to h = 1. Using the experimental observation dataset Train the neural network. After the training is completed, the sensitivity of each experimental factor is calculated according to formula (5) and formula (6), as shown in Table 4.
[0103] Table 4: Sensitivity calculation results of experimental observation data set
[0104]
[0105] Step 500: construct a Gaussian process regression model using experimental observation data and perform sample expansion;
[0106] Based on experimental observation data set Construct a Gaussian process regression model, in which the Gaussian kernel scale factor γ = 2 and the variance σ = 0.01. Set the data augmentation amount n a = 100, using the Gaussian process regression model to perform data expansion to obtain the expanded experimental observation data set
[0107] Step 600: Calculate sensitivity using the Sobol method based on the expanded experimental observation data;
[0108] The sensitivity of each experimental factor is calculated according to formula (12) and formula (13), as shown in Table 5.
[0109] Table 5: Sensitivity calculation results of experimental observation data set
[0110]
[0111] Step 700: Setting a sensitivity threshold to screen out key factors of the experiment.
[0112] Set the sensitivity threshold α = 0.1, then from the results in Table 4, we can see that based on The key experimental factors selected from the experimental observation data set are factor 1, factor 2, and factor 3. It can be seen from Table 5 that based on The key experimental factors screened out from the experimental observation data set are factor 1, factor 2, and factor 3.
[0113] The second aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the key factor screening method based on the mixture of attention mechanism and Gaussian process regression described in the first aspect of the present disclosure are implemented.
[0114] Figure 3 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 3 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.
[0115] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0116] The third aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the key factor screening method based on the mixture of attention mechanism and Gaussian process regression described in the first aspect of the present disclosure are implemented.
[0117] In summary, the beneficial technical effects brought about by the technical solution proposed in the present invention include: (1) being able to select the best sensitivity analysis method based on the amount of experimental data; and (2) being able to provide relatively accurate sensitivity analysis results under small sample conditions.
[0118] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.
Claims
1. A key factor screening method based on a mixture of attention mechanism and Gaussian process regression, characterized in that: The method comprises: Step S1, constructing an experimental observation data set and setting a data volume threshold; Step S2, selecting a sensitivity analysis method according to the amount of experimental observation data; Step S3, constructing a support vector set based on the experimental observation data set; Step S4: constructing a neural network based on an attention mechanism using a support vector set, training the neural network and calculating sensitivity; Step S5: construct a Gaussian process regression model using experimental observation data and perform sample expansion; Step S6, calculating the sensitivity using the Sobol method based on the expanded experimental observation data; Step S7: setting a sensitivity threshold and screening out key experimental factors.
2. According to claim 1, a key factor screening method based on a mixture of attention mechanism and Gaussian process regression is characterized in that: In step S1, the values of all experimental factors in the experiment and the corresponding experimental results are taken as one piece of observation data, and all experimental data constitute the experimental observation data set The data volume threshold is set to N thres .
3. According to claim 2, a key factor screening method based on a mixture of attention mechanism and Gaussian process regression is characterized in that: In step S2, the sample size N of the time observation data in the statistical experimental observation data set is obs , and the data volume threshold is N thres For comparison; if the sample size N obs Greater than the data volume threshold N thres , then select the sensitivity analysis method based on the attention mechanism; otherwise, select the sensitivity analysis method based on Gaussian process regression.
4. According to claim 3, a key factor screening method based on a mixture of attention mechanism and Gaussian process regression is characterized in that: In step S3, the number of support vectors n is set c , use K-means algorithm to analyze the experimental observation data set Clustering is performed, the number of clusters is the number of support vectors, and the experimental data closest to the cluster center in each cluster is selected as the support vector. All support vectors constitute the support vector set 5. According to claim 4, a key factor screening method based on a mixture of attention mechanism and Gaussian process regression is characterized in that: In step S4: The support vector is concentrated on the experimental factor value part X s As the key vector, the support vector set [X s , Y s ] as the value vector, construct a neural network based on the attention mechanism, and calculate the forward propagation result of the neural network: Where: MLP(·) represents a feedforward neural network; MHA(·) represents a multi-head attention mechanism; The calculation result of the multi-head attention mechanism is: Among them: || represents the splicing operation; W o represents the weight matrix; denote the query vector weight, key vector weight, and value vector weight of the kth attention head respectively; d K represents the dimension of the key vector space; represents the data Z after splicing the experimental factor values and experimental results in the experimental observation data s =[X s , Y s ]; The calculation result of the feedforward neural network is: in: represents the network weight; represents the network bias; Using experimental observation dataset The neural network is trained, and the partial derivative of the neural network output to the input is calculated as the sensitivity S of each experimental factor:
6. The key factor screening method based on a mixture of attention mechanism and Gaussian process regression according to claim 5 is characterized in that: In step S5: Using experimental observation datasets Construct a Gaussian process regression model and set the number of experiments to be expanded to n a , generate experimental factor values according to the sampling strategy Using Gaussian process regression model to predict experimental results The prediction process is: A=K(X s ,X s )+s 2 I The experimental data Expanding to experimental observation datasets The expanded experimental observation data 7. The key factor screening method based on a mixture of attention mechanism and Gaussian process regression according to claim 6 is characterized in that: In step S6, for the expanded experimental observation data Combined with the Gaussian process regression model, the Sobol method is used to calculate the sensitivity of each experimental factor. The implementation steps of the Sobol method include: The expanded experimental observation data Divided equally into two groups, And calculate the overall variance of the experimental results and the first-order partial variance of each experimental factor: Among them: f(·) represents the nonlinear mapping relationship between experimental results and experimental factors, that is, The sensitivity of each experimental factor is:
8. The key factor screening method based on a mixture of attention mechanism and Gaussian process regression according to claim 6 is characterized in that: In step S7, a sensitivity threshold α is set, and experimental factors with a sensitivity greater than the threshold α are selected as key factors.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the key factor screening method based on a mixture of attention mechanism and Gaussian process regression described in any one of claims 1-8 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the key factor screening method based on a mixture of attention mechanism and Gaussian process regression described in any one of claims 1-8 are implemented.