Membership function approximation method based on strategy gradient algorithm

By converting the Gaussian membership function into a polynomial form and optimizing coefficients using the strategy gradient algorithm, the problem of computational complexity in continuous action space is solved, efficient membership function approximation is achieved, and system performance and stability are improved.

CN118897465APending Publication Date: 2024-11-05YANSHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410940440.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

In continuous action space, the value function method is highly computationally complex when used for optimal control of complex systems, and the Gaussian membership function is difficult to handle in fast calculations or resource-constrained environments.

Method used

The Gaussian membership function is converted into a polynomial form based on the strategy gradient algorithm, and the polynomial coefficients are optimized through gradient descent, and the mean square error is used as the reward function to gradually approximate the optimal membership function and reduce the computational complexity.

Benefits of technology

It significantly reduces the computational complexity, improves system performance and stability, retains the excellent characteristics of Gaussian membership function, and is suitable for optimal control of complex systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897465A_ABST
    Figure CN118897465A_ABST
Patent Text Reader

Abstract

The invention discloses a membership function approximation method based on a strategy gradient algorithm, and belongs to the technical field of artificial intelligence and optimal control, and the method comprises the following steps: S1, representing a conventional Gaussian membership function as a polynomial form, and setting an initial coefficient value of the polynomial to enable the polynomial to preliminarily approach the shape of a target membership function; s2, according to the performance index of the fuzzy control system, constructing a state-action value function used for evaluating the influence of different membership functions on the system performance; s3, optimizing polynomial coefficient performance through a gradient descent algorithm; and S4, the optimization process is iterated until the mean square error converges, and an optimal polynomial membership function is obtained to replace the Gaussian membership function. According to the method, the excellent characteristics of the Gaussian membership function can be reserved, the calculation complexity can be remarkably reduced, and the system performance and stability can be more effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence and optimal control, and in particular relates to a membership function approximation method based on a policy gradient algorithm. Background Art

[0002] Inspired by biological learning mechanisms, the adaptive dynamic programming (ADP) method simulates biological feedback learning and uses the concept of reinforcement learning to design approximate optimal controllers for complex systems when the system collects input and output data offline or online. However, many reinforcement learning algorithms rely on value function methods. Although value function methods perform well in discrete action spaces, when the action space is continuous, it becomes impractical to directly apply value function methods. In continuous action spaces, the choice of actions is infinite, which makes value function estimation and optimization very complicated. In order to solve the above problems, the policy method is introduced. The policy method directly parameterizes the decision-making strategy and maximizes the cumulative reward by optimizing these parameters.

[0003] In engineering practice and scientific research, fuzzy systems are widely used in the control and modeling of complex systems. Membership function is the core concept in fuzzy logic, which is used to express the degree of membership of an element to a fuzzy set. Gaussian membership function is widely used in various application scenarios due to its continuity and nonlinear characteristics. However, the expression of Gaussian function is relatively complex and is not as good as the polynomial used in most cases. This becomes a difficult problem to solve in environments that require fast calculations or have limited resources. Therefore, a membership function approximation method based on a policy gradient algorithm is needed that can convert the Gaussian membership function into a polynomial form with higher computational efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a membership function approximation method based on a policy gradient algorithm, which improves the expression and calculation of the membership function. It facilitates subsequent operations by converting the traditional Gaussian membership function into a polynomial form that is easy to calculate and express. The mean square error of the true value and the predicted value is used as a reward function, and the gradient descent method is used for continuous adjustment to ultimately obtain the optimal parameter value. This method can not only retain the excellent characteristics of the Gaussian membership function, but also significantly reduce the computational complexity, and can more effectively improve system performance and stability.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A membership function approximation method based on a policy gradient algorithm comprises the following steps:

[0007] Step S1: The traditional Gaussian membership function is expressed as a polynomial form, and a suitable polynomial form is selected to replace the membership function, and the initial coefficient value of the polynomial is set according to the requirements of the fuzzy control system and preliminary analysis so that the polynomial preliminarily approximates the shape of the target membership function;

[0008] Step S2: According to the performance indicators of the fuzzy control system, a state-action value function is constructed to evaluate the impact of different membership functions on the system performance, and a reward function based on the mean square error is constructed, and the mean square error between the true value and the predicted value is used as the optimization target to guide the adjustment of the polynomial coefficients;

[0009] Step S3: Apply the policy gradient method to optimize the performance of the polynomial coefficients through the gradient descent algorithm, calculate the gradient of the mean square error, continuously adjust the polynomial parameters to minimize the mean square error, and gradually approach the optimal membership function;

[0010] Step S4: Iterate the optimization process until the mean square error converges, and obtain the optimal polynomial membership function to replace the Gaussian membership function.

[0011] A further improvement of the technical solution of the present invention is that in step S1, the traditional Gaussian membership function is expressed in a polynomial form, as shown in the following formula:

[0012] P(x,c)=c0+c1x+c2x 2 +...+c n x n

[0013] Where (c=[c0,c1,...,c n ]) are polynomial coefficients, and n represents the polynomial order.

[0014] A further improvement of the technical solution of the present invention is that step S2 includes the following steps:

[0015] Step S201: Using the membership function of the polynomial fuzzy model, define a policy network, as shown in the following formula:

[0016] π θ (c i |x t )

[0017] This formula indicates that the coefficient c is selected under state x. i The probability of the policy network is θ, the input is the state x, and the output is the coefficient c of the polynomial fuzzy model. i ;

[0018] Step S202: using the policy network in step S201, sampling a number of trajectories in the environment as an initial data set;

[0019] Step S203: For a given training data set, the mean square error (MSE) is used to measure the gap between the actual output and the expected output, and the reward function is set to the negative value of the mean square error:

[0020]

[0021] Where L represents the mean square error, μ(x) i represents the output of the network, Represents the true value.

[0022] A further improvement of the technical solution of the present invention is that in step S203, according to the policy gradient theorem, the gradient of the expected return is expressed as shown in the following formula:

[0023]

[0024] In the formula, τ represents a trajectory, that is, a polynomial function under a coefficient, pθ(τ) represents the trajectory distribution obtained according to the policy network parameter θ, R(τ) represents the return of trajectory τ, and π θ (c i |x t ) means in state x t Lower generation coefficient c i probability.

[0025] A further improvement of the technical solution of the present invention is that: in step S3, the ADAM optimization algorithm is used to adaptively adjust the learning rate, and the learning rate is dynamically adjusted according to the first-order moment estimation and the second-order moment estimation of the gradient in each iteration.

[0026] A further improvement of the technical solution of the present invention is that the learning rate is adaptively adjusted using the ADAM optimization algorithm, comprising the following steps:

[0027] Step S301: Initialize ADAM algorithm parameters and initialize polynomial coefficients c i , initialize the momentum term m and the second-order moment term v to 0, set the hyperparameter β1 and momentum decay rate β2 and the numerical stability parameter ∈;

[0028] Step S302: Update the iterations. For each time step t, calculate the gradient ▽ θ L(c t ), and update the ADAM parameters:

[0029] The loss function is calculated by the back propagation algorithm for each coefficient c i The gradient of , whose gradient is calculated by the chain rule, is shown in the following formula:

[0030]

[0031] In the formula, Represents the derivative of the loss function L with respect to the model output μ(x);

[0032] Update the coefficient c using the gradient descent algorithm i , and use the ADAM optimization algorithm to dynamically adjust the learning rate to adapt to the update of different parameters, and use the mean square error as the optimization target to guide the adjustment of the polynomial coefficients:

[0033] Compute the first moment estimate:

[0034] m t =β1m t-1 +(1-β1)▽ θ L(c t )

[0035] in,

[0036]

[0037] Compute the second moment estimate:

[0038] v t =β2v t-1 +(1-β2)(▽ θ L(c t )) 2

[0039] Further correction of deviations:

[0040]

[0041] Last updated parameters:

[0042]

[0043] In the formula, α represents the learning rate, which controls the step size of weight update;

[0044] Step S303: Repeat step S302 until the reward function converges or reaches a preset stop condition.

[0045] A further improvement of the technical solution of the present invention is that in step S4, by continuously iteratively updating the coefficient ci, the loss function is gradually reduced to optimize the strategy network, and the negative value of the mean square error is defined as the reward function to maximize the reward function and make the polynomial membership function approach the true value.

[0046] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

[0047] The present invention improves the expression and calculation of the membership function based on the membership function approximation method of the policy gradient algorithm, and facilitates subsequent operations by converting the traditional Gaussian membership function into a polynomial form that is easy to calculate and express. The mean square error of the true value and the predicted value is used as the reward function, and the gradient descent method is used for continuous adjustment to finally obtain the optimal parameter value. This not only retains the excellent characteristics of the Gaussian membership function, but also significantly reduces the calculation complexity, and can more effectively improve the system performance and stability.

[0048] The present invention is based on the experience replay mechanism and fully utilizes online data and offline data in the strategy learning process, breaks the correlation between data and improves data utilization efficiency.

[0049] The present invention uses the ADAM optimization algorithm to adaptively adjust the learning rate, thereby improving the optimization efficiency and convergence speed and improving the stability of the system.

[0050] The policy gradient-based algorithm of the present invention performs well in reinforcement learning and optimal control. When the model is unknown or incompletely known, it can approximate the optimal control solution by continuously optimizing the strategy. The policy gradient algorithm is applied to the polynomial approximation of the membership function in the fuzzy system, which can not only retain the excellent characteristics of the Gaussian membership function, but also significantly reduce the computational complexity.

[0051] The present invention combines reinforcement learning and proposes a new method using policy gradient to make its approximation error smaller and more reasonable, thereby introducing more membership function information into the stability condition, improving the accuracy of the controller, and achieving better membership function polynomial approximation regardless of whether the controlled system is a single variable or a double variable. This means that the present invention has a wider applicability. In practical applications, the accuracy of this approximation is crucial to the performance of the control system because it directly affects the controller's ability to regulate the system state. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the effect of polynomial approximation of the univariate piecewise membership function in the present invention;

[0053] Figure 2 It is a schematic diagram of the effect of polynomial approximation of the bivariate membership function in the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with embodiments:

[0055] The present invention provides a membership function approximation method based on a policy gradient algorithm, which specifically comprises the following steps:

[0056] Step S1: The traditional Gaussian membership function is expressed as a polynomial form, by selecting a suitable polynomial form to replace the membership function, and setting the initial coefficient value of the polynomial according to the requirements of the fuzzy control system and preliminary analysis, to ensure that the polynomial can preliminarily approximate the shape of the target membership function;

[0057] The traditional Gaussian membership function is expressed as a polynomial form, as shown below:

[0058] P(x,c)=c0+c1x+c2x 2 +...+c n x n

[0059] Where (c=[c0,c1,...,c n ]) are polynomial coefficients, and n represents the polynomial order;

[0060] Step S2: According to the performance index of the fuzzy control system, a corresponding state-action value function is constructed. The value function is used to evaluate the impact of different membership functions on system performance, and a reward function based on mean square error is constructed, so that the mean square error between the true value and the predicted value is used as the optimization target to guide the adjustment of the polynomial coefficients. Specifically, the following steps are included:

[0061] Step S201: Using the membership function of the polynomial fuzzy model, define a policy network, as shown in the following formula:

[0062] π θ (c i |x t )

[0063] This formula indicates that the coefficient c is selected under state x. i The probability of the policy network is θ, the input is the state x, and the output is the coefficient c of the polynomial fuzzy model. i ;

[0064] Step S202: using the policy network in step S201, sampling a number of trajectories in the environment as an initial data set;

[0065] Step S203: Since it is an approximation of the polynomial coefficients, each coefficient is combined into a trajectory, and the entire polynomial function is called a strategy. Due to the particularity of the problem, in the discussion of this problem, the strategy network only needs to be given one, that is, a suitable initial value of the polynomial parameter. Therefore, there is no need to continue to update the strategy network parameters. It is only necessary to find the trajectory where the maximum reward function exists through the gradient method;

[0066] Put the initially given trajectories into a set, called a training data set (a given training data set is a lot of trajectories, that is, various strategies, that is, various initial polynomial coefficients). For a given training data set, use the mean square error (MSE) to measure the gap between the true output and the expected output, and set the reward function to the negative value of the mean square error:

[0067]

[0068] Where L represents the mean square error, μ(x) i represents the output of the network, represents the true value;

[0069] According to the policy gradient theorem, the gradient of the expected return can be expressed as follows:

[0070]

[0071] In the formula, τ represents a trajectory, that is, a polynomial function under a coefficient, pθ(τ) represents the trajectory distribution obtained according to the policy network parameter θ, R(τ) represents the return of trajectory τ, and π θ (c i |x t ) means in state x t Lower generation coefficient c i probability;

[0072] Step S3: Apply the policy gradient method to optimize the performance of the polynomial coefficients through the gradient descent algorithm, calculate the gradient of the mean square error, and continuously adjust the polynomial parameters to minimize the mean square error, thereby gradually approaching the optimal membership function;

[0073] In order to further improve the optimization efficiency and convergence speed, the ADAM optimization algorithm is used to adaptively adjust the learning rate. The ADAM optimization algorithm combines the characteristics of the momentum method and the adaptive learning rate. In each iteration, the learning rate is dynamically adjusted according to the first-order moment estimation and the second-order moment estimation of the gradient. Specifically, the following steps are included:

[0074] Step S301: Initialize ADAM algorithm parameters and initialize polynomial coefficients c i , which can be initialized randomly or empirically, initialize the momentum term m and the second-order moment term v to 0, set the hyperparameter β1 and momentum decay rate β2 and the numerical stability parameter ∈;

[0075] Step S302: Update the iterations. For each time step t, calculate the gradient ▽ θ L(c t ), and update the ADAM parameters:

[0076] The loss function is calculated by the back propagation algorithm for each coefficient c i The gradient of , the calculation of which can be obtained by the chain rule, as shown in the following formula:

[0077]

[0078] In the formula, Represents the derivative of the loss function L with respect to the model output μ(x);

[0079] Update the coefficient c using the gradient descent algorithm i , and use the ADAM optimization algorithm to dynamically adjust the learning rate to adapt to the update of different parameters, and use the mean square error as the optimization target to guide the adjustment of the polynomial coefficients:

[0080] Compute the first moment estimate:

[0081] m t =β1m t-1 +(1-β1)▽ θ L(c t )

[0082] in,

[0083]

[0084] Compute the second moment estimate:

[0085] v t =β2v t-1 +(1-β2)(▽ θ L(c t )) 2

[0086] Further correction of deviations:

[0087]

[0088] Last updated parameters:

[0089]

[0090] In the formula, α represents the learning rate, which controls the step size of weight update;

[0091] Step S303: Repeat step S302 until the reward function converges or reaches a preset stop condition;

[0092] Step S4: iterate the optimization process until the mean square error converges, and obtain the optimal polynomial membership function to replace the Gaussian membership function;

[0093] By continuously iteratively updating the coefficient ci, the loss function is gradually reduced, the policy network is continuously optimized, and the negative value of the mean square error is defined as the reward function, thereby maximizing the reward function and making the polynomial membership function closer to the true value. That is, the parameter gradient descent method is used to ensure that the optimized polynomial membership function can effectively replace the Gaussian membership function, thereby improving the system's computational efficiency and membership expression ability.

[0094] The present invention is verified by two examples below:

[0095] Example 1: Membership function approximation based on a single variable:

[0096] Set up a simple membership function and choose the standard Gaussian function as the approximation object:

[0097] Step S1: The traditional Gaussian membership function is expressed as a polynomial form, and a suitable polynomial form is selected to replace the membership function, and the initial polynomial coefficient c is used to calculate the membership function. i , calculate the predicted value of the input variable x:

[0098]

[0099] Wherein, (c=[c0,c1,c2,c3,c4]) is the polynomial coefficient, and the polynomial order is selected as 4th order;

[0100] Step S2: Set the initial values ​​of the polynomial parameters and define them as the policy network. Then, construct a reward function based on the mean square error, taking the mean square error between the true value and the predicted value as the optimization target to guide the adjustment of the polynomial coefficients. Specifically, the following steps are included:

[0101] Step S201: Using the membership function of the polynomial fuzzy model, define a policy network, as shown in the following formula:

[0102] π θ (c i |x t )

[0103] This formula indicates that the coefficient c is selected under state x. i The probability of the policy network is θ, the input is the state x, and the output is the coefficient c of the polynomial fuzzy model. i ;

[0104] Step S202: using the policy network in step S201, sampling a number of trajectories in the environment as an initial data set;

[0105] Step S203: Since it is an approximation of the polynomial coefficients, the composition of each coefficient each time is called a trajectory, and the entire polynomial function is called a strategy. Due to the particularity of the problem, in the discussion of this problem, the strategy network only needs to be given one, that is, a suitable initial value of the polynomial parameter. Therefore, there is no need to continue to update the strategy network parameters. It is only necessary to find the trajectory where the maximum reward function exists through the gradient method;

[0106] Since the membership function only needs to look at the positive part in most cases, we discuss the interval of 1-7, and divide the interval into 3 parts with an interval of 2. Initialize the initial strategy of each part, that is, give the initial values ​​of the polynomial coefficients of these three parts respectively, and perform polynomial approximation respectively;

[0107] For a given training data set, the mean square error is used to measure the difference between the true output value y and the expected output value. The gap between:

[0108]

[0109] In the formula, represents the mean square error, represents the output of the network, and y represents the true value;

[0110] And set the reward function to the negative of the mean square error:

[0111]

[0112] According to the policy gradient theorem, the gradient of the expected return can be expressed as follows:

[0113]

[0114] In the formula, τ represents a trajectory, that is, a polynomial function under a coefficient, pθ(τ) represents the trajectory distribution obtained according to the policy network parameter θ, R(τ) represents the return of trajectory τ, and π θ (c i |x t ) means in state x t Lower generation coefficient c i probability;

[0115] Step S3: In order to further improve the optimization efficiency and convergence speed, the ADAM optimization algorithm is used to adaptively adjust the learning rate. The ADAM optimization algorithm combines the characteristics of the momentum method and the adaptive learning rate. In each iteration, the learning rate is dynamically adjusted according to the first-order moment estimation and the second-order moment estimation of the gradient. Specifically, the following steps are included:

[0116] Step S301: Initialize ADAM algorithm parameters: α = 0.001, β1 = 0.9, β2 = 0.999, ∈ = 10-8 ;

[0117] Step S302: Update the iterations. For each time step t, calculate the gradient ▽ θ L(c t ), and update the ADAM parameters:

[0118] The loss function is calculated by the back propagation algorithm for each coefficient c i The gradient of , the calculation of which can be obtained by the chain rule, as shown in the following formula:

[0119]

[0120] In the formula, Represents the loss function L for the model output The derivative of

[0121] Update the coefficient c using the gradient descent algorithm i , even if the mean square error is used as the optimization target, it guides the adjustment of polynomial coefficients:

[0122] Compute the first moment estimate:

[0123] m t =β1m t-1 +(1-β1)▽ θ L(c t )

[0124] in,

[0125]

[0126] Compute the second moment estimate:

[0127] v t =β2v t-1 +(1-β2)(▽ θ L(c t )) 2

[0128] Further correction of deviations:

[0129]

[0130] Last updated parameters:

[0131]

[0132] In the formula, α represents the learning rate, which controls the step size of weight update;

[0133] Step S303: Repeat step S302 until the reward function converges or reaches a preset stop condition;

[0134] Step S4: By continuously iteratively updating the coefficient ci, the loss function is gradually reduced, and the policy network is continuously optimized, thereby maximizing the reward function and making the polynomial membership function closer to the true value.

[0135] As shown in Table 1 and Table 2, they are the initial polynomial coefficients of each segment of the single variable and the final polynomial coefficients of each segment of the single variable respectively:

[0136] Table 1 Single variable initial polynomial coefficients for each segment:

[0137] Number of groups 1-3 3-5 5-7 <![CDATA[c0]]> 0.1 0.0001 9e-09 <![CDATA[c1]]> 0.5 -0.0010 -2e-09 <![CDATA[c2]]> 1.0 0.0010 2e-09 <![CDATA[c3]]> -2.0 -0.0200 -9e-09 <![CDATA[c4]]> 2.0 0.0200 1e-09

[0138] Table 2 Final polynomial coefficients of each segment of the univariate:

[0139] Number of groups 1-3 3-5 5-7 <![CDATA[c0]]> 0.0300 0.0001 1e-11 <![CDATA[c1]]> -0.3497 -0.0013 -2.40e-10 <![CDATA[c2]]> 1.5153 0.0080 2.25e-09 <![CDATA[c3]]> -2.9017 -0.0223 -9.18e-09 <![CDATA[c4]]> 2.0776 0.0231 1.40e-08 Mean square error 5.4592e-07 1.0136e-11 1.9685e-18

[0140] It can be seen from Tables 1 and 2 that after the membership function is approximated by the policy gradient, the mean square error of the function corresponding to the optimal coefficient is very small, and the final approximation effect is as follows: Figure 1 shown.

[0141] Example 2: Polynomial approximation of bivariate membership functions:

[0142] Set up a simple membership function and choose the standard Gaussian function as the approximation object:

[0143] Step S1: The traditional Gaussian membership function is expressed as a polynomial form, and a suitable polynomial form is selected to replace the membership function, and the initial polynomial coefficient c is used to calculate the membership function. i , calculate the predicted value of the input variable x:

[0144]

[0145] There are 15 variables corresponding to the dual variable at the fourth order, among which (c=[c1,c2,...,c 15 ]) are polynomial coefficients, the polynomial order is selected as 4, and the approximation range is [-0.5, 0.5];

[0146] Step S2: Similar to Example 1, the policy network only needs to be given one appropriate initial value of the polynomial parameter. There is no need to update the policy network parameters in the future. It is only necessary to find the trajectory with the maximum reward function through the gradient method.

[0147] For a given training data set, the mean square error is used to measure the difference between the true output value y and the expected output value. The gap between:

[0148]

[0149] In the formula, represents the mean square error, represents the output of the network, and y represents the true value;

[0150] And set the reward function to the negative of the mean square error:

[0151]

[0152] According to the policy gradient theorem, the gradient of the expected return can be expressed as follows:

[0153]

[0154] In the formula, τ represents a trajectory, that is, a polynomial function under a coefficient, pθ(τ) represents the trajectory distribution obtained according to the policy network parameter θ, R(τ) represents the return of trajectory τ, and π θ (c i |x t ) means in state x t Lower generation coefficient c i probability;

[0155] Step S3: In order to further improve the optimization efficiency and convergence speed, the ADAM optimization algorithm is used to adaptively adjust the learning rate. The ADAM optimization algorithm combines the characteristics of the momentum method and the adaptive learning rate. In each iteration, the learning rate is dynamically adjusted according to the first-order moment estimation and the second-order moment estimation of the gradient. Specifically, the following steps are included:

[0156] Step S301: Initialize ADAM algorithm parameters: α = 0.001, β1 = 0.9, β2 = 0.999, ∈ = 10 -8 ;

[0157] Step S302: Update the iterations. For each time step t, calculate the gradient ▽ θ L(c t ), and update the ADAM parameters:

[0158] The loss function is calculated by the back propagation algorithm for each coefficient c i The gradient of , the calculation of which can be obtained by the chain rule, as shown in the following formula:

[0159]

[0160] In the formula, Represents the loss function L for the model output The derivative of

[0161] Update the coefficient c using the gradient descent algorithm i , even if the mean square error is used as the optimization target, it guides the adjustment of polynomial coefficients:

[0162] Compute the first moment estimate:

[0163] m t =β1m t-1 +(1-β1)▽ θ L(c t )

[0164] in,

[0165]

[0166] Compute the second moment estimate:

[0167] v t =β2v t-1 +(1-β2)(▽ θ L(c t )) 2

[0168] Further correction of deviations:

[0169]

[0170] Last updated parameters:

[0171]

[0172] In the formula, α represents the learning rate, which controls the step size of weight update;

[0173] Step S303: Repeat step S302 until the reward function converges or reaches a preset stop condition;

[0174] Step S4: By continuously iteratively updating the coefficient ci, the loss function is gradually reduced, and the policy network is continuously optimized to maximize the reward function, making the bivariate polynomial membership function closer to the true value.

[0175] As shown in Table 3 and Table 4, they are the initial polynomial coefficients of each segment of the two variables and the final polynomial coefficients of each segment of the two variables respectively:

[0176] Table 3. Coefficients of bivariate initial polynomials:

[0177] <![CDATA[c1]]> 0.9 <![CDATA[c6]]> -0.9 <![CDATA[c 11 ]]> 0.2 <![CDATA[c2]]> 0 <![CDATA[c7]]> 0 <![CDATA[c 12 ]]> 0 <![CDATA[c3]]> 0.2 <![CDATA[c8]]> 0 <![CDATA[c 13 ]]> 0.4 <![CDATA[c4]]> -0.9 <![CDATA[c9]]> 0 <![CDATA[c 14 ]]> 0 <![CDATA[c5]]> 0 <![CDATA[c 10 ]]> -0.3 <![CDATA[c 15 ]]> 0.2

[0178] Table 4 Bivariate final polynomial coefficients:

[0179]

[0180] It can be seen from Tables 3 and 4 that after the policy gradient is used to perform polynomial approximation on the membership function of the two variables, the mean square error of the function corresponding to the optimal coefficient is very small, indicating that the approximation effect is excellent. The final approximation effect is as follows Figure 2 As shown in Example 1 and Example 2, the present invention uses the policy gradient to replace the complex Gaussian function with a simple polynomial form for expression. In the field of fuzzy control, it is extremely important to relax the stability condition by introducing the membership function information, but the membership function in the form of a Gaussian function is difficult to introduce. Therefore, by approximating it to a polynomial form and then introducing it, the stability condition can be relaxed and a more reasonable controller can be obtained.

[0181] The present invention can improve computational efficiency while maintaining approximation accuracy. First, the mathematical model of the Gaussian membership function is defined, and the form of the target polynomial approximation is set. Then, a policy gradient-based optimization framework is constructed to minimize the approximation error by iteratively updating the polynomial coefficients. In particular, the present invention uses a policy gradient optimization algorithm to find the optimal polynomial coefficients to achieve the mapping of the Gaussian function to the polynomial. Compared with the prior art, the method of the present invention has the following advantages: high computational efficiency and easy implementation on hardware; can effectively reduce complexity, especially in resource-constrained environments; maintains a high degree of approximation accuracy and is suitable for various systems that require fuzzy processing. Therefore, the present invention is expected to be widely used in many fields such as automatic control, machine learning, and intelligent system design.

[0182] Polynomial fuzzy control can be used in industrial process control, robot control, drone control, medical diagnosis, drug delivery systems, power grid dispatching, and generator control. Among them, solving the controller is a problem worth exploring and studying.

[0183] Polynomial fuzzy control can effectively solve technical problems that are difficult to deal with by traditional control methods. By combining polynomial approximation and fuzzy logic, it can handle uncertainty and fuzziness in complex nonlinear systems and improve the robustness and response speed of control systems. However, the stability conditions of fuzzy control are often too stringent to obtain a controller. Therefore, it is necessary to introduce the membership function information unique to fuzzy control into the stability conditions through the relaxation matrix, thereby relaxing the conditions so that the entire system can obtain a controller.

[0184] However, since the membership function information is mostly Gaussian function, it is difficult to introduce, so a reasonable approximation method is needed to approximate the Gaussian function to a polynomial function, so that it is easier to introduce into the stability condition. Many approximation methods have been provided in existing research, such as Taylor series approximation method, Lagrange approximation method, and Chebyshev approximation method. The present invention combines reinforcement learning and proposes a new method using policy gradient to make the approximation error smaller and more reasonable, thereby introducing more membership function information into the stability condition and improving the accuracy of the controller. Whether the controlled system is a single variable or a double variable, a good membership function polynomial approximation can be achieved, which means that the present invention has a wide range of applicability. In practical applications, the accuracy of this approximation is crucial to the performance of the control system because it directly affects the controller's ability to adjust the system state.

[0185] For example, in the field of drone control, drones are subject to various uncertainties and environmental interferences (such as wind force, sensor errors, etc.) during actual flight, which seriously affects the normal operation of drones. Polynomial fuzzy control is robust and can maintain stability and performance under these uncertainties and interferences. And when we further introduce the membership function information unique to fuzzy control into the stability condition, the robustness of the system can be further improved. In the present invention, by using policy gradients to perform polynomial approximation on the membership function, it can be better introduced into the stability condition, and a controller that is more in line with the actual situation is obtained, thereby improving the ability of drones to deal with emergencies in actual work.

[0186] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A membership function approximation method based on a policy gradient algorithm, characterized in that The following steps are involved: Step S1: The traditional Gaussian membership function is expressed as a polynomial form, and a suitable polynomial form is selected to replace the membership function, and the initial coefficient value of the polynomial is set according to the requirements of the fuzzy control system and preliminary analysis so that the polynomial preliminarily approximates the shape of the target membership function; Step S2: According to the performance indicators of the fuzzy control system, a state-action value function is constructed to evaluate the impact of different membership functions on the system performance, and a reward function based on the mean square error is constructed, and the mean square error between the true value and the predicted value is used as the optimization target to guide the adjustment of the polynomial coefficients; Step S3: Apply the policy gradient method to optimize the performance of the polynomial coefficients through the gradient descent algorithm, calculate the gradient of the mean square error, continuously adjust the polynomial parameters to minimize the mean square error, and gradually approach the optimal membership function; Step S4: Iterate the optimization process until the mean square error converges, and obtain the optimal polynomial membership function to replace the Gaussian membership function.

2. The method for membership function approximation based on a policy gradient algorithm according to claim 1, characterized in that: In step S1, the traditional Gaussian membership function is expressed as a polynomial form, as shown in the following formula: P(x,c)=c0+c1x+c2x 2 +...+c n x n Where (c=[c0,c1,...,c n ]) are polynomial coefficients, and n represents the polynomial order.

3. The method for membership function approximation based on a policy gradient algorithm according to claim 2, characterized in that: The step S2 comprises the following steps: Step S201: Using the membership function of the polynomial fuzzy model, define a policy network, as shown in the following formula: p θ (c i |x t ) This formula indicates that the coefficient c is selected under state x. i The probability of the policy network is θ, the input is the state x, and the output is the coefficient c of the polynomial fuzzy model. i ; Step S202: using the policy network in step S201, sampling a number of trajectories in the environment as an initial data set; Step S203: For a given training data set, the mean square error (MSE) is used to measure the gap between the actual output and the expected output, and the reward function is set to the negative value of the mean square error: Where L represents the mean square error, μ(x) i represents the output of the network, Represents the true value.

4. The method for membership function approximation based on a policy gradient algorithm according to claim 3, characterized in that: In step S203, according to the policy gradient theorem, the gradient of the expected return is expressed as follows: In the formula, τ represents a trajectory, that is, a polynomial function under a coefficient, pθ(τ) represents the trajectory distribution obtained according to the policy network parameter θ, R(τ) represents the return of trajectory τ, and π θ (c i |x t ) means in state x t Lower generation coefficient c i probability.

5. The method for membership function approximation based on a policy gradient algorithm according to claim 4, characterized in that: In step S3, the ADAM optimization algorithm is used to adaptively adjust the learning rate, and the learning rate is dynamically adjusted according to the first-order moment estimation and the second-order moment estimation of the gradient in each iteration.

6. The method for membership function approximation based on a policy gradient algorithm according to claim 5, characterized in that: The ADAM optimization algorithm is used to adaptively adjust the learning rate, including the following steps: Step S301: Initialize ADAM algorithm parameters and initialize polynomial coefficients c i , initialize the momentum term m and the second-order moment term v to 0, set the hyperparameter β1 and momentum decay rate β2 and the numerical stability parameter ∈; Step S302: Update the iterations. For each time step t, calculate the gradient And update the ADAM parameters: The loss function is calculated by the back propagation algorithm for each coefficient c i The gradient of , whose gradient is calculated by the chain rule, is shown in the following formula: In the formula, Represents the derivative of the loss function L with respect to the model output μ(x); Update the coefficient c using the gradient descent algorithm i , and use the ADAM optimization algorithm to dynamically adjust the learning rate to adapt to the update of different parameters, and use the mean square error as the optimization target to guide the adjustment of the polynomial coefficients: Compute the first moment estimate: in, Compute the second moment estimate: Further correction of deviations: Last updated parameters: In the formula, α represents the learning rate, which controls the step size of weight update; Step S303: Repeat step S302 until the reward function converges or reaches a preset stop condition.

7. The method for membership function approximation based on a policy gradient algorithm according to claim 6, characterized in that: In step S4, the coefficient ci is updated iteratively and continuously, so that the loss function is gradually reduced to optimize the strategy network, and the negative value of the mean square error is defined as the reward function, so that the reward function is maximized and the polynomial membership function is close to the true value.