Method for optimizing HardSwsh activation function curve in neural network
By optimizing the HardSwish activation function into a piecewise linear function, the problem of high computational cost is solved, the efficiency and resource utilization of the neural network are improved, and a significant performance improvement is achieved.
Patent Information
- Application Number
- CN202411089038.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2026-02-10
AI Technical Summary
The existing HardSwish activation function has a high computational cost, which affects the efficiency and resource utilization of neural networks.
By using a custom activation function, the HardSwish activation function is optimized into a piecewise linear function, replacing the original exponential function. This retains the curve trend and special point design, while using straight lines instead of curves to reduce computational costs.
While maintaining accuracy, the performance of the HardSwish activation function has been significantly improved, while reducing computational costs and resource consumption.
Smart Images
Figure CN121503548A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of intelligent video processing, and particularly relates to a method for optimizing a HardSwish activation function curve in a neural network. BACKGROUND
[0002] In the prior art, in a video / image processing process, including image recognition, semantic recognition, etc., a neural network learning method is often used, wherein each neuron node in the neural network accepts an output value of a previous layer of neurons as an input value of the neuron, and transmits the input value to a next layer, and an input layer neuron node directly transmits an input attribute value to a next layer (a hidden layer or an output layer). In a multi-layer neural network, there is a function relationship between an output of an upper layer node and an input of a lower layer node, and the function is called an activation function. The activation function is used to add a nonlinear factor, and a nonlinear function is introduced as the activation function, so that the expression ability of the deep neural network is more powerful.
[0003] In the prior art, the HardSwish activation function is used to replace the ReLU activation function, and the accuracy of the neural network is significantly improved, however, the HardSwish function is composed of an exponential function, and the ReLU is a segmented first-order function, and the calculation cost is much higher than that of the ReLU. SUMMARY
[0004] In order to solve the above problems, the purpose of the present application is to provide a method for reducing the calculation cost of HardSwish, which specifically represents speed improvement and reduction of resource usage, such as CPU, memory, etc.
[0005] Specifically, the present application provides a method for optimizing a HardSwish activation function curve in a neural network, which comprises the following steps:
[0006] S1, training a HardSwish activation function human shape detection model:
[0007] The training framework is PaddleDetection, the network is picodet_s_320_coco_lcnet, the custom activation CustomActivation function is used to replace the original activation function HardSwish function, and the optimized HardSwish activation function is no longer an exponential function, but a segmented first-order function like ReLU,
[0008] That is, the image of the optimized CustomActivation function becomes a segmented function; the curve trend and special point design of HardSwish are retained, including the lowest point of the quadratic function, the origin, and the segmented straight lines connected by the special points;
[0009] S2, training data coco dataset, precision map value 36;
[0010] Replace the original activation function HardSwish with the activation function CustomActivation of the application, and the precision map value is 35.4.
[0011] In the step S1, the optimization custom activation CustomActivation function is:
[0012]
[0013] Where x represents the function variable, which is the feature intermediate feature layer when put into the deep network.
[0014] The optimization custom activation CustomActivation function can be represented by Python code as follows:
[0015] Suppose class CustomActivation: class CustomActivation3(nn.Module):
[0016] Define def__init__(self): super(CustomActivation3,
[0017] self).__init__()
[0018] Define def forward(self,x):
[0019] Three conditions are defined:
[0020] cond1=x<-1.5
[0021] cond2=(x>=-1.5)&(x<=0)
[0022] cond3=x>0;
[0023] Apply the corresponding function:
[0024] out1=-0.25*x-0.75
[0025] out2=0.25*x
[0026] out3=x;
[0027] Use the where function of torch to select the output according to the condition:
[0028] output=torch.where(cond1,out1,x)
[0029] output = torch.where(cond2, out2, output)
[0030] output = torch.where(cond3, out3, output)
[0031] Return output: return output.
[0032] Therefore, the advantages of the present application are: under the premise of ensuring accuracy, the performance problem of the original HardSwish activation function is improved. BRIEF DESCRIPTION OF DRAWINGS
[0033] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application.
[0034] Figure 1 is a schematic diagram of HardSwish and its derivative image in the example.
[0035] Figure 2 is Figure 1 The image after the custom optimization activation function in the present method is a schematic diagram of the curve of the image.
[0036] Figure 3 is a schematic diagram of the flow of the present method. DETAILED DESCRIPTION
[0037] In order to more clearly understand the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the drawings.
[0038] The present application relates to a method for optimizing the curve of a HardSwish activation function, first, let's understand the HardSwish activation function:
[0039] 1. HardSwish formula
[0040] The specific formula is:
[0041]
[0042] Hardsigmoid formula:
[0043]
[0044] The derivative function is:
[0045]
[0046] 2. HardSwish and its derivative image
[0047] like Figure 1 As shown, the solid curve is the HardSwish graph, and the dashed line is its derivative function.
[0048] Next, this invention relates to an optimization method for the activation function, specifically a method for optimizing HardSwish activation:
[0049] 1. Optimize the CustomActivation function;
[0050] The optimized HardSwish activation function is no longer an exponential function, but a piecewise linear function like ReLU, which fundamentally reduces computational costs.
[0051] The activation function formula is optimized because curve activation functions have poor performance, i.e., high computational cost and long processing time. This application uses a straight line instead of a curve to improve performance, as follows:
[0052]
[0053] The graph of the optimized CustomActivation function is transformed into a piecewise function; the curve trend and special point design of HardSwish are preserved, including the lowest point of the quadratic function, the origin, etc., and these special points are connected to form a piecewise straight line.
[0054] The Python code is as follows:
[0055] class CustomActivation3(nn.Module):
[0056] def__init__(self):
[0057] super(CustomActivation3,self).__init__()
[0058] def forward(self,x):
[0059] # Define conditions
[0060] cond1 = x < -1.5
[0061] cond2 = (x >= -1.5) & (x <= 0)
[0062] cond3=x>0
[0063] # Apply the corresponding function
[0064] out1 = -0.25 * x - 0.75
[0065] out2 = 0.25 * x
[0066] out3 = x
[0067] #Use the where function in torch to select output based on conditions
[0068] output=torch.where(cond1,out1,x)
[0069] output=torch.where(cond2,out2,output)
[0070] output=torch.where(cond3,out3,output)
[0071] The output of the CustomActivation function is shown in the image. Figure 2 As shown, the bold lines of the arrows represent optimization.
[0072] II. The effect of customizing and optimizing activation functions
[0073] The training framework is PaddleDetection, and the model is picodet_s_320_coco_lcnet. The human detection accuracy in the coco dataset trained with HardSwish and a custom optimized activation function is less than 1 point, which is within an acceptable range.
[0074] Therefore, as Figure 3 As shown, this application relates to a method for optimizing HardSwish activation in a neural network, which can be taken as an example of human detection, and includes the following steps:
[0075] S1, Train the HardSwish activation function human detection model:
[0076] The training framework is PaddleDetection, and the network is pigcodet_s_320_coco_lcnet. The original activation function, HardSwish, is replaced with an optimized CustomActivation function. The optimized HardSwish activation function is no longer an exponential function, but a piecewise linear function similar to ReLU.
[0077] The optimized CustomActivation function is as follows:
[0078] as follows:
[0079]
[0080] Here, x represents a function variable, which in a deep network would be the intermediate feature layer;
[0081] S2, training data COCO dataset, precision map value 36;
[0082] The original activation function HardSwish is replaced with the activation function CustomActivation of this application, with a precision map value of 35.4.
[0083] The optimized activation function is applied to neural networks to introduce nonlinear transformations, enabling the neural network to learn and represent more complex functional relationships, thereby improving the efficiency of the entire system.
[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for optimizing the HardSwish activation function curve in a neural network, characterized in that, The method includes the following steps: S1, Train the HardSwish activation function human detection model: The training framework is PaddleDetection, and the network is pigcodet_s_320_coco_lcnet. The original activation function, HardSwish, is replaced with an optimized CustomActivation function. The optimized HardSwish activation function is no longer an exponential function, but a piecewise linear function similar to ReLU. That is, the graph of the optimized CustomActivation function is transformed into a piecewise function; the curve trend and special point design of HardSwish are preserved, including the lowest point and origin of the quadratic function, and these special points are connected to form a piecewise straight line. S2, training data Cocoa dataset, precision map value 36; The original activation function HardSwish is replaced with the activation function CustomActivation of this application, with a precision map value of 35.
4.
2. The method for optimizing the HardSwish activation function curve in a neural network according to claim 1, characterized in that, In step S1, the optimized custom activation function is: Here, x represents a function variable, which in a deep network would be the intermediate feature layer.
3. The method for optimizing the HardSwish activation function curve in a neural network according to claim 1, characterized in that, The optimized custom activation function, CustomActivation, can be represented in Python code as follows: Let the class CustomActivation: class CustomActivation3(nn.Module): Define `def__init__(self):super(CustomActivation3,` self).__init__() Define `def forward(self, x):` Three conditions are defined: cond1 = x < -1.5 cond2 = (x >= -1.5) & (x <= 0) cond3 = x > 0; Apply the corresponding function: out1 = -0.25 * x - 0.75 out2 = 0.25 * x out3 = x; Use torch's where function to select output based on conditions: output=torch.where(cond1,out1,x) output=torch.where(cond2,out2,output) output=torch.where(cond3,out3,output) Return output.