Construction method of zero-center nonlinear activation function and numerical solution method of teetertottering

Abstract

This invention provides a method for constructing a zero-center nonlinear activation function. First, a zero-center, monotonically increasing nonlinear odd function is constructed by rotating the general hyperbolic tangent function, thus creating an activation function for constructing neurons. Then, a teetertottering numerical algorithm is established to determine the learning and training depth of a deep neural network, solving the problem of calculating the output and derivative of this activation function. Because this activation function is a zero-center neuron activation function with a slope close to 1 and monotonically increasing, it can overcome problems such as gradient explosion, gradient vanishing, and neuron death during neural network training. Furthermore, since this activation function is zero-center symmetric, the neural network maintains a balanced learning speed regardless of whether the input is greater than or less than 0 in the learning direction, thereby improving the efficiency of neural network learning and training.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for constructing a zero-center nonlinear activation function and a numerical solution method for teetertottering. Background Technology

[0002] Activation functions are a crucial component of artificial neural networks. Increasingly, nonlinear functions such as Sigmoid, Tanh, and ReLU are used to help artificial neural networks learn from massive amounts of data. By creating complex mappings between output and input, they can fit any complex function and provide accurate predictions. The derivatives of Sigmoid and Tanh functions are both no greater than 1, and their outputs saturate when the input approaches zero, easily causing gradient vanishing in deep neural networks, leading to learning stagnation in shallower neurons. The introduction of the ReLU (Rectified Linear Unit) function effectively solves the gradient vanishing problem in deep neural networks with long-term dependencies. When the input is greater than 0, its slope is always equal to 1, thus avoiding gradient vanishing and gradient explosion. However, when the input is less than 0, the ReLU output is 0. During training, if a neuron's input is zero, it can lead to neuron starvation and death, negatively impacting the structure and robustness of the entire neural network. To address the problems of the ReLU function, various variant activation functions have been developed, including Leaky ReLU, PReLU (Parametric ReLU), RReLU (Randomized Leaky ELU), ELU (Exponential Linear Unit), and SReLU (S-shaped ReLU), which have overcome some of the shortcomings of the ReLU function. Except for SReLU, the other ReLU variants are not zero-center functions, leading to uneven learning speeds in different learning directions during training, objectively affecting the learning and training effects. SReLU is composed of three different linear functions connected together, which can be expressed as a zero-center function. However, if the slopes of the three linear functions are the same, they degenerate into a single straight line; if the slopes are different, slopes greater than 1 may cause gradient explosion during backpropagation in deep neural networks, while slopes less than 1 will lead to gradient vanishing. Summary of the Invention

[0003] This invention addresses the aforementioned problems, aiming to provide an activation function that overcomes gradient explosion, gradient vanishing, and neuron death, and improves the learning and training efficiency of neural networks. The invention employs the following technical solution:

[0004] This invention provides a method for constructing a zero-center nonlinear activation function, characterized by comprising the following steps:

[0005] Step S1-1: Establish a rotation function based on the general hyperbolic function.

[0006] The hyperbolic function is expressed as:

[0007]

[0008] In the formula, For input variables, For output variables, c is a constant, and k is a non-zero constant.

[0009] The rotation function is a zero-centered, monotonically increasing odd function, expressed as:

[0010]

[0011] The rotation function is then rearranged to obtain:

[0012] y = f(x, c, k, β)

[0013] In the formula, x and y are the input and output variables after rotation, respectively, and β is the angle of rotation;

[0014] Step S1-2: Based on the rotation function and its properties, establish the neuron's nonlinear activation function, i.e., the zero-center nonlinear activation function, whose general form is expressed as:

[0015]

[0016] In the formula, f(*) represents the expression derived from... Rotation yields the function, k0 = k 0- or k 0+ k ∞ =k -∞ or k +∞ k 0- and k 0+ Let k be the specific numerical values ​​that k approaches 0- and 0+, respectively. -∞ and k +∞ These are the specific numerical values ​​of k as it approaches -∞ and +∞, respectively. and With k0 and k respectively ∞ correspond, and All in Within its neighborhood.

[0017] Based on the general form of the zero-center nonlinear activation function, according to k0 = k 0- or k 0+ k∞ =k -∞ or k +∞ This yields four specific zero-center nonlinear activation functions.

[0018] The method for constructing a zero-center nonlinear activation function provided by this invention may also have the following technical feature: the zero-center nonlinear activation function is used to construct a deep neural network.

[0019] The depth of the deep neural network is determined according to the following formula:

[0020]

[0021] In the formula, ξ is an arbitrarily small positive number, and N is the number of layers in the neural network.

[0022] The method for constructing a zero-center nonlinear activation function provided by this invention may also have the following technical features, wherein, taking for and The maximum absolute value, The threshold x can be determined accordingly. T ,make The hyperbolic function Rotating by angle β will change the derivative. The value is determined in and between.

[0023] This invention provides a numerical solution method for teetertottering, used to solve the zero-center nonlinear activation function constructed using the above-described method, characterized by comprising the following steps:

[0024] Step S2-1: Initialize the data by selecting parameters c, k0, and k. ∞ Determine the specific zero-center nonlinear activation function;

[0025] Step S2-2, select ξ and θ max Determine the number of layers N and the threshold x of the deep neural network. T The value;

[0026] Step S2-3, let β in Within a small neighborhood, that is, satisfying

[0027] Step S2-4: Determine the hyperbolic function derivative The calculation formula is as follows:

[0028]

[0029] Step S2-5, will and Substituting the derivative In the calculation formula, the derivative is solved. The estimated value;

[0030] Step S2-6, determine the derivative Does the estimated value meet the conditions? When it is determined that the derivative is accepted If the estimated value is not found, return to step S2-3 to reselect β;

[0031] Steps S2-7: Using the expression and derivative of the hyperbolic function The estimated value was calculated. The estimated value:

[0032]

[0033] Step S2-8: Using the expression of the rotation function, calculate the value of the output y of the zero-center nonlinear activation function:

[0034]

[0035] Step S2-9, based on the expression of the rotation function, we can obtain:

[0036]

[0037] Calculate using the above formula and The value, to achieve the and The correction is performed, and the derivative is recalculated using the expression of the hyperbolic function.

[0038] Step S2-10, take the result obtained in step S2-9 Substitute the input x into the calculation formula of step S2-8, recalculate the output y of the zero-center nonlinear activation function, and calculate the derivative.

[0039]

[0040] Will or and the results calculated in steps 2-9 Substituting the value into the above formula, we can calculate the derivative. The value of .

[0041] Invention Function and Effect

[0042] According to the present invention, the method for constructing the zero-center nonlinear activation function and the teetertottering numerical solution method firstly utilizes the general hyperbolic tangent function to construct a nonlinear odd function with a zero center and monotonically increasing through rotation, thereby constructing an activation function for constructing neurons. Then, addressing the difficulty of solving the transcendental equations contained in this zero-center nonlinear activation function, a teetertottering numerical solution algorithm is established to determine the learning and training depth of deep neural networks. This solves the problem of solving the output and derivative of the zero-center nonlinear activation function, creating favorable conditions for optimizing the forward propagation solution and error backpropagation update of neural network weight coefficients based on this activation function. Because this zero-center nonlinear activation function is zero-center symmetric, has a slope of nearly 1, and is monotonically increasing, using it as the activation function for neurons can overcome the problems of gradient explosion and gradient vanishing during neural network training. Moreover, the activation function has the characteristics of an odd function, which means it cannot cause neuron death. Furthermore, since the activation function is zero-center symmetric, the neural network has a balanced learning speed during training, regardless of whether the input is greater than 0 or less than 0, thereby improving the efficiency of neural network learning and training. Attached Figure Description

[0043] Figure 1 This is a structural diagram of the deep neural network for optimizing vehicle following behavior in an embodiment of the present invention.

[0044] Figure 2 This is a flowchart of the vehicle following control method based on neural networks in this embodiment.

[0045] Figure 3 This is a flowchart of the method for constructing the zero-center nonlinear activation function in an embodiment of the present invention.

[0046] Figure 4 This is an embodiment of the present invention. And the curve of the rotation function when k>0.

[0047] Figure 5 This is an embodiment of the present invention. And the curve of the rotation function when k < 0.

[0048] Figure 6 These are the output y and derivative of each neuron in the embodiments of the present invention. The calculation method flowchart.

[0049] Figure 7 Simulation diagram of the teetertottering algorithm in this embodiment of the invention (when c=1, k=k) 0+ and k = k +∞ ),in, Figure 7(a) is the coordinate curve of the activation function. Figure 7 (b) is The curve;

[0050] Figure 8 This is a simulation diagram of the following vehicle behavior control of the vehicle following system in an embodiment of the present invention, wherein, Figure 8 (a) is the distance-time curve. Figure 8 (b) shows the velocity-time curve. Figure 8 (c) is the acceleration-time curve. Detailed Implementation

[0051] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following describes in detail the construction method of the zero-center nonlinear activation function and the teetertottering numerical solution method of this invention with reference to embodiments and accompanying drawings.

[0052] <Example>

[0053] Taking vehicle car-following behavior optimization as an example, this paper explains in detail how to implement the construction method of the zero-center nonlinear activation function and the teetertottering numerical solution method of the present invention.

[0054] Assuming the vehicles use non-regenerative braking, m is a natural number, and the initial positions of the front and rear vehicles are s respectively. p (0), s f (0), initial velocities are v p (0), v f (0), the time-series variable of the position of the preceding vehicle during its operation is s. p (1),s p (2),…,s p (m) can be obtained through a positioning detection device, with equation (1) as the optimization objective:

[0055]

[0056] In the formula, d actual (k) and d safe (k) represents the actual vehicle distance and the safe vehicle distance during the kth sampling period, respectively.

[0057] The temporal variable s of the position of the following vehicle when it optimizes its behavior. f (1),s f (2),…,s f (m), and thus the speed time series v of the following vehicle can be determined. f (1),v f (2),…,v f (m) and acceleration time series a f(1),a f (2),…,a f (m) is obtained.

[0058] Figure 1 This is a structural diagram of the deep neural network for optimizing vehicle following behavior in this embodiment.

[0059] like Figure 1 As shown, the activation functions of each neuron from layer 1 to the output layer in this neural network adopt the zero-center nonlinear activation function proposed in this invention, x i (j) represents the input of the i-th neuron in the i-th layer, y i (j) represents the output of the i-th neuron in the j-th layer, 1≤i≤n, 1≤j≤N-1, the output layer is the N-th layer, and the forward and backward propagation adopts the teetertottering numerical solution algorithm proposed in this invention.

[0060] Figure 2 This is a flowchart of the vehicle following control method based on neural networks in this embodiment.

[0061] like Figure 2 As shown, this embodiment provides a vehicle car-following control method based on neural networks, including the following steps:

[0062] Step S1: Construct the zero-center nonlinear activation function for neurons, and determine the structure, number of layers, and number of neurons in each layer of the deep neural network. See [link to step S1]. Figure 1 As shown, the neural network weights are initialized.

[0063] Figure 3 This is a flowchart of the construction method of the zero-center nonlinear activation function in this embodiment.

[0064] like Figure 3 As shown, the construction method of the zero-center nonlinear activation function includes the following steps:

[0065] Step S1-1: Establish a rotation function based on a general hyperbolic function, wherein the hyperbolic function is expressed as:

[0066]

[0067] In the formula, For input variables, For output variables, c is a constant, and k is a non-zero constant.

[0068] The rotation function is a zero-centered, monotonically increasing odd function, expressed as:

[0069]

[0070] The rotation function is then rearranged to obtain:

[0071] y = f(x, c, k, β)

[0072] In the formula, x and y are the input and output variables after rotation, respectively, and β is the angle of rotation.

[0073] Figure 4 When k>0 Rotation The curve graph, Figure 5 When k < 0 Rotation The curve after that.

[0074] Step S1-2: Based on the rotation function and its properties, establish the neuron's nonlinear activation function, i.e., the zero-center nonlinear activation function, whose general form is expressed as:

[0075]

[0076] In the formula, f(*) represents the expression derived from... Rotation yields the function, k0 = k 0- or k 0+ k ∞ =k -∞ or k +∞ k 0- and k 0+ Let k be the specific numerical values ​​that k approaches 0- and 0+, respectively. -∞ and k +∞ These are the specific numerical values ​​of k as it approaches -∞ and +∞, respectively. and With k0 and k respectively ∞ correspond, and All in Within its neighborhood.

[0077] Based on the general form of the zero-center nonlinear activation function, according to k0 = k 0- or k 0+ k ∞ =k -∞ or k +∞ This yields four specific zero-center nonlinear activation functions.

[0078] As described above, this zero-center nonlinear activation function is used to construct a deep neural network, the depth of which is determined according to the following formula:

[0079]

[0080] In the formula, ξ is an arbitrarily small positive number, and N is the number of layers in the neural network.

[0081] In the above method for constructing the zero-center nonlinear activation function, θ is taken as... max for and The maximum absolute value, The threshold x can be determined accordingly. T x can be determined accordingly. T ,make The hyperbolic function Rotating by angle β will change the derivative. The value is determined in and Between, obviously Very close to 1.

[0082] Step S2, with J = min{d ds E f To optimize the target, the neural network is trained using massive sample data from the vehicle following system. The forward propagation of the neural network output and the backward propagation of the optimization error are solved using the teetertottering numerical algorithm, and the weights are updated accordingly.

[0083] The forward propagation of a deep neural network proceeds from left to right. The calculation method for the output of each neuron can be obtained by following the calculation steps of the output y in the teetertottering numerical solution algorithm.

[0084] The formula for updating weights during backpropagation in a deep neural network is:

[0085]

[0086] In the formula, ΔW(k) is the weight increment at the k-th digit, and η is the learning rate. The teetertottering numerical algorithm can be used to solve this problem. To optimize the target's partial derivative with respect to the position of the following vehicle, It is a Jacobian matrix.

[0087] In this way, during backpropagation, the deep neural network can utilize the teetertottering numerical solution algorithm. The calculation steps determine the partial derivatives of neurons in each layer, thereby determining the weight increment ΔW(k) for weight updates in each layer, so as to realize weight updates.

[0088] Figure 6 The output y and derivative of each neuron in a deep neural network The flowchart of the calculation method, also known as the flowchart of the teetertottering numerical solution algorithm, includes the following steps:

[0089] Step S2-1: Initialize the data by selecting parameters c, k0, and k. ∞ Determine the specific zero-center nonlinear activation function;

[0090] Step S2-2, select ξ and θ max Determine the number of layers N and the threshold x of the deep neural network. T The value;

[0091] Step S2-3, let β in Within a small neighborhood, that is, satisfying

[0092] Step S2-4: Determine the hyperbolic function derivative The calculation formula is as follows:

[0093]

[0094] Step S2-5, will and Substituting the derivative In the calculation formula, the derivative is solved. The estimated value;

[0095] Step S2-6, determine the derivative Does the estimated value meet the conditions? When it is determined that the derivative is accepted If the estimated value is not found, return to step S2-3 to reselect β;

[0096] Steps S2-7: Using the expression and derivative of the hyperbolic function The estimated value was calculated. The estimated value:

[0097]

[0098] Step S2-8: Using the expression of the rotation function, calculate the value of the output y of the zero-center nonlinear activation function:

[0099]

[0100] Step S2-9, based on the expression of the rotation function, we can obtain:

[0101]

[0102] Calculate using the above formula and The value, to achieve the and The correction is performed, and the derivative is recalculated using the expression of the hyperbolic function.

[0103] Step S2-10, take the result obtained in step S2-9 Substitute the input x into the calculation formula of step S2-8, recalculate the output y of the zero-center nonlinear activation function, and calculate the derivative.

[0104]

[0105] Will or and the results calculated in steps 2-9 Substituting the value into the above formula, we can calculate the derivative. The value of .

[0106] Figure 7 This is a simulation diagram of the teetertottering algorithm in this embodiment, where c = 1 and k is assumed to be... 0+ =10 -10 k +∞ =10 10 ξ = 5.002 * 10 -4 θ max =10 -8 Therefore, N = 25000. Figure 7 (a) is the coordinate curve of the activation function. Figure 7 (b) is The simulation curve.

[0107] Depend on Figure 7 As can be seen, the constructed activation function consists of three straight lines with a slope of approximately 1.

[0108] Step S3, based on the initial data s of the positions and speeds of the front and rear vehicles. p (0),s f (0),v p (0),v f (0), and the time-series position data s of the preceding vehicle. p (1),s p (2),…,s p (m), using a trained deep neural network, calculates the position-time sequence s of the optimal behavior of the following vehicle. f (1),s f (2),…,s f (m);

[0109] Step S4, based on the initial value s of the position and speed of the following vehicle. f (0), v f (0) and the obtained position-time series s of the optimal behavior of the following vehicle f (1),s f (2),…,sf (m), calculate the speed time series v of the following vehicle. f (1),v f (2),…,v f (m) and acceleration time series a f (1),a f (2),…,a f (m);

[0110] Step S5, based on the acceleration time sequence a of the rear vehicle f (1),a f (2),…,a f (m) Perform follow-up vehicle behavior control.

[0111] Figure 8 This is a simulation diagram of the following vehicle behavior control of the vehicle following system in this embodiment.

[0112] like Figure 8 As shown in the simulation, constructing a deep neural network using a zero-center nonlinear activation function and employing the teetertottering numerical solution method for forward and backward propagation calculations to optimize the following behavior of the rear vehicle can achieve good tracking results.

[0113] Functions and effects of the embodiments

[0114] According to the vehicle following control method based on neural networks provided in this embodiment, a zero-center nonlinear activation function and a teetertottering numerical solution method are adopted. First, a nonlinear odd function with zero center and monotonically increasing is constructed by rotating the general hyperbolic tangent function, and then an activation function for constructing neurons is built. Then, to address the problem that the zero-center nonlinear activation function contains transcendental equations that are difficult to solve, a teetertottering numerical solution algorithm is established to determine the learning and training depth of the deep neural network. This solves the problem of solving the output and derivative of the zero-center nonlinear activation function, and creates favorable conditions for the forward propagation solution and error backpropagation update of the neural network weight coefficients based on this activation function to achieve the optimization purpose. Since this zero-center nonlinear activation function is a neuron activation function with a zero center, a slope close to 1, and monotonically increasing, it can overcome the problems of gradient explosion and gradient vanishing during neural network training. Moreover, the activation function has the characteristics of an odd function, which means it cannot cause neuron death. Furthermore, since the activation function is zero-center symmetric, the neural network has a balanced learning speed during training, regardless of whether the input is greater than 0 or less than 0, thereby improving the efficiency of neural network learning and training.

[0115] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

[0116] In the above embodiments, the zero-center nonlinear activation function and the teetertottering numerical solution method are applied to the field of vehicle car-following control. In fact, the zero-center nonlinear activation function and the teetertottering numerical solution method can also be widely applied to other fields, such as text recognition, speech recognition, video signal processing, etc., and can also achieve the corresponding technical effects.

Claims

1. A neural network-based car-following control method, characterized in that, The method comprises the following steps: Step S1, constructing a zero-centered nonlinear activation function for neurons, determining the structure, number of layers and number of neurons of each layer of a deep neural network; Step S2, training the deep neural network using sample data of a vehicle following system; Step S3, obtaining a position time sequence of an optimal behavior of a rear vehicle using the trained deep neural network according to initial data of positions and speeds of the front vehicle and the rear vehicle and time sequence position data of the front vehicle and the rear vehicle; Step S4, calculating a speed time sequence and an acceleration time sequence of the rear vehicle according to initial values of positions and speeds of the rear vehicle and the position time sequence of the optimal behavior; Step S5, controlling the behavior of the rear vehicle according to the acceleration time sequence of the rear vehicle, Wherein, step S1 comprises the following sub-steps: Step S1-1, establishing a rotation function based on a general hyperbolic function, The hyperbolic function is expressed as: wherein is an input variable, is an output variable, is a constant, is a non-zero constant, The rotation function is a zero-centered, monotonically increasing odd function, expressed as: The rotation function is arranged to obtain: wherein and are the rotated input and output variables, respectively, is the angle of rotation, Step S1-2, establishing a neuron nonlinear activation function, i.e. a zero-centered nonlinear activation function, according to the rotation function and the properties of the rotation function, which is generally expressed as: wherein denotes the function obtained by rotating , or , = or ; and are respectively tending to and a specific value, and are respectively tending to and a specific value; and correspond respectively to and , and are both in the neighborhood of , Based on the general form of the zero-centered nonlinear activation function, four specific zero-centered nonlinear activation functions are obtained according to = or , = or .

2. The neural network-based vehicle following control method according to claim 1, characterized in that: wherein The zero-centered nonlinear activation function is used to construct a deep neural network, The depth of the deep neural network is determined according to the following formula: In the formula, is any small positive number, is the number of layers of the neural network.

3. The neural network-based vehicle following control method according to claim 1, characterized in that: wherein Take For And The maximum value of the absolute value, The value of the derivative , let The hyperbolic function Rotates The angle, the value of the derivative Is determined between And .

4. The neural network-based vehicle following control method of claim 1, characterized in that: In step S2, the forward propagation of the deep neural network and the backward propagation of the optimization error are solved using a teeterottering numerical solution algorithm, and the weights are updated, The teeterottering numerical solution algorithm comprises the following steps: Step S2-1, data initialization is performed, parameters are selected , and determining a specific said zero-centered non-linear activation function; Step S2-2, selecting and determining the number of layers of the deep neural network and the value of the threshold . Step S2-3, let In In a small neighborhood of the point, i.e. satisfying ; Step S2-4, determining the derivative of the hyperbolic function the calculation formula of​ ； Step S2-5, substituting the derivative and into the calculation formula of the derivative to solve the estimated value of the derivative . Step S2-6, determine the derivative Does the estimated value meet the conditions? When it is determined that the derivative is accepted... If the estimated value is not found, return to step S2-3 to select again. ; Step S2-7, the estimated value of is calculated using the expression and the derivative of the hyperbolic function ​ ； Step S2-8, calculating the output of the zero-centered non-linear activation function using the expression of the rotation function of the value: ； Step S2-9, according to the expression of the rotation function, it can be obtained that: The values of and are calculated using the above formulas and are corrected, and the derivative is recalculated using the expression of the hyperbolic function Step S2-10, re-computing the output of the zero-centered non-linear activation function and input into the formula of Step S2-8, re-computing the output of the zero-centered non-linear activation function and computing the derivative : Substitute the values of or and the value of calculated in step 2-8 into the above equation to calculate the value of the derivative . Wherein, the zero-centered nonlinear activation function is constructed using the construction method of the zero-centered nonlinear activation function in any one of claims 1-3.

Images